On-Device LLM: Running Large Language Models on Mobile Hardware

For three years, mobile AI meant sending a prompt to the cloud. That model is breaking. With Gemini Nano, Apple Intelligence, and Snapdragon’s on-device Llama, flagship SoCs now execute generative AI entirely within the memory and power envelope of a phone — typically under 4W sustained and 6GB of available DRAM.

This shift is not just software. It’s a hardware-software co-design problem that touches quantization, KV-cache management, heterogeneous compute scheduling, and thermal budgeting. Understanding it is essential for anyone building AI-first mobile experiences.

1. Why On-Device? The Physics of Cloud Inference

Cloud inference has three structural liabilities on mobile:

• Latency: Round-trip 4G/5G adds 80–350ms even before model compute. On-device decode can deliver first token in 180–250ms.
• Privacy & compliance: Health, messaging and enterprise data cannot leave the device in many jurisdictions. On-device keeps embeddings local.
• Availability & cost: Offline mode is mandatory for automotive and field use, and per-token cloud pricing scales poorly at millions of devices.

The tradeoff is capability. You cannot fit GPT-4 (1T+ parameters) on a phone. On-device LLMs target assistive tasks — summarization, smart reply, grammar, entity extraction, and tool-use — where a 1.8B–3.8B model fine-tuned for the task outperforms a distant giant.

2. The Hardware Stack: NPU, Memory, and the Bottleneck

Modern mobile SoCs, from Google’s Tensor G4 detailed in our Pixel 9 Pro teardown to Snapdragon 8 Gen 3, share a similar hierarchy:

Key insight: TOPS is misleading. Prefill scales with TOPS, but decode — where the user waits for streamed tokens — is bound by how fast you can move weights and KV-cache from DRAM into the NPU. This is why unified memory architecture and larger system-level cache, as seen in newer chiplet architectures for mobile SoCs, matter more than headline NPU numbers.

3. Making Models Fit: Quantization, Distillation & KV-Cache

A 3B model at FP16 needs 6 GB just for weights — impossible on a phone. Production on-device models rely on three techniques:

4-bit quantization (AWQ, GPTQ, QLoRA)

Weights are compressed to INT4 with per-group scaling (group size 128). Activation stays INT8/FP16. A 3.8B Phi-3 Mini drops from 7.6 GB (FP16) to ~2.1 GB (INT4 + overhead) with <2% accuracy loss on summarization. NPUs now have native INT4 datapaths to avoid dequantization on the fly.

Knowledge distillation & pruning

Gemini Nano (1.8B / 3.25B variants) is distilled from Gemini Pro with instruction tuning for on-device domains. Structured pruning removes 15–25% of attention heads while keeping latency stable.

KV-cache optimization

The KV-cache grows linearly with context length: 2 × layers × dim × tokens × bytes. At 4K context, a 3B model cache can exceed 600 MB. Techniques like PagedAttention, 8-bit KV quantization, and sliding-window attention (as in Gemma 2B) cut this by 50%.

# Quantize with llama.cpp — produce Q4_K_M for NPU runtime
./llama-quantize phi-3-mini-f16.gguf phi-3-mini-q4.gguf Q4_K_M

# ExecuTorch / QNN export for Snapdragon NPU (INT4 weights, INT8 act)
import torch
from executorch import exir
model = exir.capture(model).to_edge()
 quantized = exir.quantize(model, "qnn-8a4w") # 8-bit act, 4-bit weight
exir.save(quantized, "model.pte")

4. Platform Comparison: How the Vendors Do It

* Apple offloads larger models to Private Cloud Compute; on-device model is limited to ~3B via Core ML + ANE. Speeds are Q4 quantized, averaged across 512-token context, batch size 1.

Qualcomm’s QNN and MediaTek’s NeuroPilot push quantization-aware execution closest to the metal, while Google’s AICore abstracts Gemini Nano behind a system-level API with LoRA adapter support — allowing apps to fine-tune without shipping a full model. For power budgeting, the same thermal constraints that affect wireless charging coil design — skin temperature limits of 43–45°C — cap sustained NPU power to ~3.5W, throttling decode speed after ~20 seconds.

5. The Software Runtime: Where Fragmentation Lives

Hardware is only half the battle. The runtime determines if your model runs at all:

• Android: NNAPI is deprecated; AICore (Gemini Nano), QNN (Snapdragon), and MLC-LLM / llama.cpp (GPU via OpenCL/Vulkan) compete. ExecuTorch is emerging as a unified path.
• iOS: Core ML + MLX + ANE via MIL. Apple requires model conversion to Core ML format and enforces strict memory limits (~50% of RAM per app).
• Cross-platform: ONNX Runtime, MNN, and TFLite still used for smaller SLMs, but lack advanced KV-cache paging.

// Android AICore — Gemini Nano system service (API level 34+)
val model = GenerativeModel.aicore()
val response = model.generateContent("Summarize this email: ...")
// Runs on Edge TPU, no network, LoRA adapters auto-mixed

// llama.cpp on-device via GPU delegate (fallback)
./llama-cli -m phi-3-q4.gguf -p "Rewrite:" --threads 6 --gpu-layers 28

Security is non-trivial: prompt injection and weight tampering are new attack surfaces. This is why vendors isolate model weights in TEE and enforce mobile kernel exploit mitigations around the NPU driver — a compromised NPU DMA can read app memory.

6. Practical Limits & What’s Next

Today’s ceiling: 2K–4K context is practical; 8K is possible with sliding window but increases KV pressure. Multi-turn chat sustains ~15 tok/s on flagship, dropping to 6–8 tok/s on mid-tier (Snapdragon 7+ Gen 3). Power draw during decode is 2.8–4.2W — about 60% of peak gaming.

Next 18 months:

• Mixture-of-Experts (MoE) for mobile: 7B MoE with 2 active experts gives 1.5B active compute with 7B knowledge, ideal for heterogeneous NPUs.
• Speculative decoding: A tiny 0.5B draft model proposes 3 tokens, the main model verifies in one pass — 1.8× speedup in decode.
• Unified memory & LLC: Chiplet SoCs with 16–24 MB system cache will halve DRAM traffic, directly improving tokens/second.
• On-device RAG: Vector DB + embeddings stored on-device (via SQLite + NPU embedding model) to ground the SLM without the cloud.

On-device LLM is no longer research — it’s a hardware-guaranteed API. The winners will be apps that treat the NPU like the camera ISP: a constrained, always-available coprocessor that enables private, instant intelligence without a network round trip.