Mobile NPU and AI Accelerator: How On-Device AI Processing Works

For a decade, mobile AI meant sending audio to the cloud. Today your phone transcribes live speech, removes video backgrounds at 60fps, and runs a 7B-parameter LLM offline — all without leaving the SoC. The enabler is not a faster CPU. It's the NPU: a domain-specific accelerator built to do one thing intolerably inefficient on general-purpose cores — dense tensor math at extreme energy efficiency.

Apple's Neural Engine (38 TOPS in A18 Pro), Qualcomm's Hexagon NPU (45 TOPS in Snapdragon 8 Gen 4), and Google's Edge TPU v5 share the same first principles. Understanding them matters not only for performance, but for privacy: on-device inference keeps biometric, location, and conversation data out of the network path entirely.

1. What an NPU Actually Is

Think of a mobile SoC as a heterogeneous work crew. The CPU handles branching, OS, and control logic. The GPU excels at massively parallel but irregular floating-point work. The Neural Processing Unit (or NPU / AI Accelerator / Neural Engine) is a hardwired dataflow engine for multiply-accumulate (MAC) operations — the core of convolutions, attention, and fully-connected layers.

Unlike a GPU, an NPU has minimal caches, no graphics pipeline, and no need to handle divergent warps. Its silicon budget goes to three things: thousands of tiny MAC units, a programmer-managed scratchpad/SRAM hierarchy, and a deterministic interconnect that keeps weights stationary while activations flow through.

2. Architecture: Inside the Tile

Most mobile NPUs are tiled. A typical 35 TOPS design uses 2-4 identical tiles, each able to run an independent model or cooperate on one large model.

The systolic array is the heart. Weights are pre-loaded into each MAC unit and stay stationary. Activations are then pumped from the left, partial sums flow downward — like a grid of assembly-line workers each adding one product before passing the result. This weight-stationary dataflow cuts DRAM bandwidth by 10-20× versus a CPU loop.

Around it: a local SRAM (typically 2-8 MB per tile) holds the working set, a DMA engine double-buffers the next layer from DRAM while the current one computes, and vector units handle non-MAC ops — activations (ReLU, GELU, Softmax), normalization, and quantize/dequantize. The control core sequences layers without waking the main CPU.

3. The Precision Ladder: From FP32 to INT4

TOPS are meaningless without precision. An NPU's trick is doing good enough math at 4 or 8 bits instead of 32.

• FP32 → FP16: Half the memory, ~2× throughput. Used for training fallback and high-accuracy attention.
• INT8: The workhorse. 4× smaller than FP32, 4× more MACs per mm². Most camera, voice, and NLP models run here with <1% accuracy loss after quantization-aware training.
• INT4 & FP4/NF4: The new frontier for on-device LLMs. A 7B model at INT4 fits in ~3.5 GB vs 14 GB at FP16, enabling it to stay resident in LPDDR5 and stream at SRAM speed. Dedicated INT4 MACs are 2× denser than INT8.
• Mixed precision & sparsity: Modern accelerators skip zero-valued weights (common after pruning) and compute attention at INT8 while keeping Softmax in FP16.

Quantization isn't just truncation. The toolchain inserts per-channel scale and zero-point values and calibrates clipping ranges. At inference, the NPU fuses dequantization into the MAC: real_value = scale * (int_value - zero_point) — computed in hardware after the accumulate.

# Post-training quantization to INT8 with TFLite (Android NNAPI / Hexagon)
import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model("./bert_mobile")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_gen  # calibration set
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_int8 = converter.convert()  # ~4x smaller, NPU-ready
# For LLM INT4: use GPTQ/AWQ externally, then compile with Core ML or QNN

As we detailed in our analysis of the mobile camera module assembly, modern ISP + NPU pipelines keep RAW frames on-chip; the NPU runs segmentation and super-resolution at INT8 before the image ever hits the AP's DRAM — critical for both latency and privacy.

4. How Inference Actually Flows

1. Compile ahead of time: Frameworks (Core ML, TensorFlow Lite + QNN, ONNX Runtime) compile the graph into NPU microcode. The compiler fuses ops (Conv+BN+ReLU), tiles tensors to SRAM size, and schedules DMA.
2. Dispatch: The app calls NNAPI / Core ML / ANE. The driver queues the job without involving the CPU cluster.
3. Double-buffered execution: While Tile 0 computes layer N from SRAM, DMA prefetches layer N+1 weights from LPDDR5. Zero bubble.
4. Writeback: Final logits/embeddings land in a shared buffer. The CPU only sees the result — not the intermediate activations.

This is why on-device transcription feels instant: a 120M-parameter Conformer runs at ~3ms/30ms chunk vs 400ms round-trip to the cloud.

5. NPU vs GPU vs DSP vs CPU — When to Use What

Compilers increasingly heterogeneously schedule: attention on NPU, Softmax and sampling on GPU, decoding on CPU. You don't choose one chip — the runtime does.

Software Stack Reality Check

On iOS, Core ML automatically prefers the Neural Engine; if an op is unsupported it silently falls back to GPU/ANE. On Android, NNAPI and vendor delegates (Qualcomm QNN, MediaTek NeuroPilot, Samsung ENN) compile .tflite/.onnx graphs. PyTorch Mobile now exports to ExecuTorch with XNNPACK+NPU delegation.

Security review matters here. Our review of the App Store security review process found that on-device model encryption is often the weakest link — models shipped as plain .tflite files can be extracted from APKs. Leverage hardware-backed model protection via the NFC secure element architecture pattern: store model decryption keys in the StrongBox/Secure Enclave and decrypt only into NPU SRAM.

6. Power, Thermal, and Privacy

An NPU can sustain 1–1.8 W at full tilt before thermal throttling — about the heat of a video call. Compare that to 5–7 W for the GPU at similar token rates. System designers therefore budget NPU bursts in 500 ms windows: wake, infer, return to retention. Always-on uses like keyword spotting run on a micro-NPU (or DSP) at 3–5 mW.

From a privacy lens, on-device inference collapses the attack surface. No audio, keystrokes, or face embeddings transit TLS, CDN, or vendor logs. For sensitive domains like mobile digital car key with UWB, the same principle applies: keep ranging and identity classification local to the Secure Processing Unit rather than the main AP — the NPU + secure element boundary is becoming the standard trusted path.

Threat model to watch: side-channel leakage via power or timing of NPU executions. Mitigate by batching sensitive inferences, disabling performance counters for untrusted apps, and using constant-time Softmax kernels where secret-dependent logits exist.

7. What's Next

Three trends are converging in 2026–2027: INT4-native LLMs with on-chip KV-cache, unified SRAM shared between NPU and ISP for zero-copy vision, and on-device training adapters (LoRA fine-tuning at FP8/INT8). The NPU is evolving from an inference engine into a personal learning core — one that must remain isolated, auditable, and off the network by default.

If you're building for mobile, profile first: measure end-to-end latency including pre/post-processing, not just TOPS. A 40 TOPS NPU is wasted if your pipeline spends 18 ms converting YUV to RGB on the CPU.