Phones, NPUs and embedded devices
Running LLMs on iOS, Android and single-board computers: memory budgets, bandwidth-based speed estimates, thermals and battery, runtime choices, what NPUs really help with, and a shipping checklist.
Modern phones have 8 to 16 GB of RAM, fast unified memory and neural processing units (NPUs). That makes small on-device language models practical for private assistants, offline features and low-latency tasks. But a phone is a hostile environment for a language model: your app gets only a slice of memory, the chip throttles within minutes, and the battery is shared with everything else the user cares about.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.