Mobile AI Agents: On-Device Autonomous Task Execution Architecture

For a decade, “assistant” on mobile meant a remote procedure call. You spoke, audio was streamed to a data center, an intent classifier fired, and a canned response returned. The device was a thin client. That model is now inverted. With 3 to 8-billion parameter models quantized to 4-bit and accelerated on 35+ TOPS NPUs, the phone can run a full agentic loop locally: observe, plan, use tools, verify, and retry — all under 900ms per reasoning step.

This is not just a smaller ChatGPT on your phone. An autonomous mobile agent is an event-driven control system. It closes the loop around the operating system itself. Understanding its architecture is critical for anyone building, securing, or evaluating the next wave of mobile software.

1. From Chatbot to Agent: What Changes

A traditional assistant is stateless and single-turn: user utterance → NLU → API call → TTS. An autonomous agent is stateful, multi-step, and goal-directed. Give it “Rebook my missed flight for tomorrow morning and message my hotel” and it will not ask you for a confirmation after every sub-step. It will decompose the goal, sequence dependencies, and recover from failures.

Three properties define an on-device agent:

• Autonomy: Operates for minutes without user intervention, maintaining a memory of prior actions and observations.
• Tool-use as the interface: Language is internal reasoning; the external interface is structured function calling (calendars, intents, UI automation, filesystem, sensors).
• Grounded perception: Reads the actual device — Accessibility Tree, screen pixels via lightweight VLM, app intents inventory, sensor streams — not just the chat history.

On Android, this builds on App Actions and the Accessibility Framework; on iOS, on App Intents, Shortcuts, and the new App Intents-driven Screen Understanding APIs introduced in iOS 19. The crucial shift is that the LLM is no longer the product — the orchestrator is.

2. The Perception–Reasoning–Action Loop

Every mobile agent, from research prototypes to Samsung Gauss Agent and Apple’s on-device Siri Agent, implements the same core loop, typically running at 2–5 Hz:

Perception

Two parallel perception streams feed the agent. The symbolic stream extracts a structured view hierarchy (view IDs, bounds, text, clickability, scroll state) via Accessibility or Semantic Tree APIs. This is cheap, deterministic, and privacy-preserving. The visual stream runs a 300–400M parameter vision encoder on downscaled screenshots (224–448px) to resolve ambiguous UI where the view tree is incomplete — custom canvases, games, or WebViews. The fusion layer produces a compact UI graph: ~1,500 tokens vs. 8,000+ for raw HTML.

Reasoning & Planning

The core LLM (typically a 3B–7B instruction-tuned model quantized to INT4 with AWQ/GPTQ) runs with a ReAct-style prompt: Thought → Action (tool_call) → Observation. The loop is bounded by a planner that maintains a task graph. Modern agents use hierarchical planning: a 7B planner decomposes “plan a commute” into subtasks, while a distilled 1.5B executor handles atomic tool invocations, saving 40–60% NPU cycles.

Context management is decisive. With 8K–16K context windows on-device, the trajectory must be compressed. Implementations use rolling summarization and tool-output distillation: only the last 3 observations are kept verbatim; earlier steps are summarized to 300 tokens.

Action & Verification

Actions are typed commands against a Tool Registry. After each action, a verifier model (often the same LLM in judge mode) checks the new UI state against the expected post-condition. If verification fails — e.g., tap missed, or login wall appeared — the agent re-plans rather than hallucinating success. This is what separates reliable agents from brittle RPA scripts. Critically, this verification must also be on-device to avoid leaking screen content.

3. The On-Device Runtime Stack

Building this loop in a production phone is less about model size and more about determinism, power, and latency.

Model Runtime

Most vendors now ship with an on-device inference framework: Apple’s ANE + MLX, Qualcomm’s QNN, MediaTek NeuroPilot, and Google’s AICore. Models are delivered as 4-bit quantized weights with separate 8-bit activations (W4A8) and use flash attention-2 kernels optimized for the NPU. A 3B model occupies ~1.8 GB after quantization, leaving room for the vision encoder and KV cache in 12 GB devices. Speculative decoding with a 0.5B draft model lifts throughput from ~22 to 35 tokens/s without quality loss.

Orchestration sits above the runtime. Instead of a monolithic prompt, the agent is configured via a signed manifest:

A single reasoning step then looks like a tool call:

This structure is why supply-chain security for agent tooling matters: each tool is an attack surface. A compromised calendar provider could inject prompt instructions into an observation. Leading runtimes now isolate tool outputs as untrusted data and apply instruction hierarchy — system > developer > user > tool — so tool content can never override user goals.

4. Security: Capability Isolation & Enclave Attestation

Giving an LLM the ability to tap your screen and send messages is the most privileged software boundary on the phone. The OS must treat the agent as an untrusted principal, even when the model is first-party.

Best practice (see our analysis of secure enclave architecture on modern SoCs) is three-tier isolation:

• Sandboxed runtime: The agent process has no direct access to app data. All tool invocations go through a broker that enforces per-tool entitlements and rate limits. UI automation is gated by foreground consent and blocked for banking apps unless explicitly allowed.
• Attested execution: Sensitive tools (payments, identity, health) require biometric consent that is bound to a hardware key in the Secure Enclave/StrongBox. The tool broker verifies the attestation token before executing. This prevents a prompt-injected agent from silently exfiltrating funds.
• Auditable trajectory: Every Thought/Action/Observation tuple is logged to an append-only, on-device store encrypted with a device key. Users can review, revoke, and roll back. Enterprises can enforce DLP policies that flag when an agent accesses classified documents.

Privacy benefits are real but conditional. On-device execution avoids the DNS and network metadata leakage inherent to cloud assistants, but the perception layer still processes sensitive screen content. Apple and Google both mandate that screenshots used for grounding never leave the NPU SRAM and are discarded after fusion — a claim that must be verified via binary attestation, not just policy.

5. Cloud vs. On-Device vs. Hybrid

The pragmatic path for 2026–2027 is hybrid. The device handles perception and low-risk actions locally; the planner may call into a Private Cloud Compute enclave for multi-step decomposition when confidence is low. User policy — “never send health data off device” — is enforced at the broker, not by prompting.

6. Performance, Thermals & Battery Reality

On-device agents are thermally constrained. A Snapdragon 8 Gen 4 NPU can sustain ~35 TOPS for only 4–6 seconds before throttling to ~18 TOPS. An agentic task that takes 60–90 seconds will see mid-task throttling unless the scheduler interleaves.

Vendors mitigate this with three techniques: (1) Chunked execution — the loop sleeps 250ms between actions to let the SoC cool and to wait for UI animations; (2) Model cascade — a 1.5B model handles trivial taps, waking the 7B planner only for decisions; (3) KV-cache reuse — the prompt prefix (system + tool schemas) is cached in SRAM, saving 30% prefill time. Measured on a 2026 flagship, a 12-step “organize my photos and create an album” task consumed 4.2% battery and peaked at 42°C — acceptable for occasional use, not for background daemons.

Developers should also consider deterministic fallbacks. The sensor-to-software pipeline in camera modules already shows how tightly coupled hardware and post-processing must be. Agent-driven camera tasks (e.g., “scan receipts and file expenses”) need similar co-design: use the ISP’s own document detection before waking the VLM.

7. Outlook: The Agentic OS

The end state is not a better assistant app. It is an OS where every App Intent is an agent-addressable tool and the shell is agent-orchestrated. Users will express goals, not taps: “prep me for my 9am with briefing docs and a quiet notification profile.” The agent compiler will turn that into a verified plan, execute under enclave attestation, and present a diff for approval.

For engineers, this demands a new discipline: treat prompts as code, tools as syscalls, and trajectories as logs. Version your agent manifests, test them with simulated UI mutations, and assume the model will be jailbroken. The phone that can act for you must also be able to say no.

On-device autonomy is ready for narrow, high-value tasks today — triage, rebooking, organization, accessibility — and will widen as NPUs approach 60 TOPS and 4-bit 7B models close the reasoning gap with their cloud counterparts. The architecture above is the blueprint to get there without trading privacy for convenience.