Mobile DSP Architecture: Audio Processing and Always-On Sensing

Your phone is listening even when its screen is off — but not in the way headlines suggest. A sub-2mW island buried inside the SoC keeps a few kilobytes of neural network alive on dedicated audio DSP hardware, waiting for a wake word while the application processor and DRAM remain in deep sleep. That island is where modern mobile audio architecture lives or dies on power, latency, and privacy.

In flagship SoCs from Qualcomm, MediaTek, Google, and Apple, the audio and sensing DSP is no longer a peripheral. It is a self-contained, always-on compute subsystem with its own clock tree, power rail, SRAM, DMA, and instruction set tuned for multiply-accumulate density. Understanding its structure explains why voice unlock feels instantaneous today yet cost less than 2% battery per day — and why getting it wrong means hot pockets, missed triggers, and failed certification.

Why Audio Needs Its Own Processor

Audio is uniquely hostile to general-purpose cores. A 48kHz stereo stream demands deterministic 20.8µs sample deadlines, heavy FIR/IIR filtering, and continuous operation for hours — the exact opposite of bursty, latency-tolerant mobile workloads. Running this on a Cortex-A or Apple P-core wastes energy on caches, out-of-order logic, and DRAM accesses that contribute nothing to SNR.

Dedicated DSPs solve this with architectural choices the AP cannot afford to replicate:

• VLIW / Vector MAC clusters: Cadence HiFi 5 delivers 4x 32-bit MACs per cycle with zero-overhead loops and circular buffers. Hexagon HVX provides 1024-bit vectors (32x 32-bit MACs) for simultaneous beamforming and neural inference.
• Tightly Coupled Memory (TCM): 256-512KB of single-cycle SRAM partitioned into instruction and data banks. No cache misses, no DRAM wake-ups. All wake-word buffers (∼150ms) reside in TCM.
• Deterministic DMA + SoundWire: Audio DMA engines move 16/24-bit PCM with sample-accurate scheduling, while the DSP sleeps between interrupts. Contrast this with AP-driven ALSA, which must wake the kernel every period.
• Separate voltage island: The LPAD runs at 0.55-0.65V on FDSOI or compact SRAM bitcells, allowing 4-20MHz operation at microwatt leakage. This links directly to power strategies discussed in our analysis of touch sampling and display power co-optimization — display and audio must not share a power rail if always-on is required.

Result: 15-40 GOPS/W for audio DSP vs. ~3 GOPS/W for an A-core on the same workload. That efficiency pays for active noise cancellation, ultrasonic sensing, and 24/7 context awareness that users now expect.

Anatomy of the Always-On Audio Island

Below is a simplified block diagram of a current-gen LPAD subsystem, representative of Snapdragon 8 Gen 3, Dimensity 9300, and Tensor G4 approaches. The dashed line marks the always-on power domain.

The Audio Processing Pipeline — From PDM to PCM

The signal path is rigidly pipelined to guarantee latency. Modern phones use digital PDM microphones clocked at 1.5-3.072MHz, allowing the mic itself to remain analog-simple while the DSP handles decimation.

1. PDM Decimation (CIC + FIR): A 64x decimation filter converts 3.072MHz PDM to 48kHz PCM. This runs in hardwired logic or DSP microcode to avoid waking the core on every sample.
2. Voice Activity Detection (VAD): An energy + zero-crossing detector with adaptive noise floor runs at 10ms frames. Power: ~80µW. If VAD is negative for 500ms, the pipeline clock-gates downstream stages.
3. Beamforming & AEC-Lite: Two-mic delay-and-sum improves SNR by 6-8dB. A lightweight acoustic echo canceller suppresses speaker leakage for barge-in (“Hey Siri” while music plays).
4. Feature Extraction: 25ms windowing → 40-dim Mel filterbank → log-Mel → MFCC / LFBE. Implemented with fixed-point FFT (128-point) reused from the DSP’s accelerator. Memory stays in TCM to avoid SRAM thrashing.
5. First-Stage Keyword Spotting (KWS): A depthwise-separable CNN or tiny Transformer (60-120KB, 3-5 layers, int8) scores every 20ms. Threshold tuned for <0.5 false accepts / hour at 92% recall.

If the first stage triggers, the LP DSP raises an interrupt to the main ADSP, streaming 600-900ms of buffered PCM via a 16KB ping-pong SRAM mailbox. The ADSP then runs a larger model (400KB-1.2MB) with speaker verification before waking the AP. This cascade keeps the AP asleep for 99.92% of audio frames — critical for devices also maintaining secure enclaves for tasks like local password manager vault encryption, where waking the AP risks exposing secure memory.

Always-On Sensing Beyond Voice

The same island doubles as a contextual sensor hub. Because the LP DSP is already powered, adding inertial and ultrasonic sensing costs almost nothing:

• Ultrasonic presence: 18-24kHz chirps from the earpiece, captured by the top mic, detect near-field motion for raise-to-wake or pocket rejection. The DSP runs a matched filter at 48kHz without AP involvement.
• In-pocket / on-table detection: Fused accel + mic coherence tells the system to lower KWS sensitivity in a noisy pocket, cutting false triggers by 30%.
• Cough / baby cry detection: Emerging on-device models for accessibility reuse the LFBE front-end, gated by user opt-in and hardware privacy switches.

Isolation matters. Sensor data crosses to the AP only via a hardware mailbox with a trusted audio path. Raw PCM is never mapped into Linux userspace. Apple and Qualcomm both implement eFuses that burn if non-secure firmware tries to DMA from mic SRAM — a requirement for certification in enterprise environments where acoustic leakage is treated like RF leakage. That philosophy mirrors hardened RF isolation seen in reconfigurable intelligent surface designs for 5G — separate the sensitive analog front end physically and contractually.

Durability also intersects. Unlike sealed peripherals covered in our IP68 waterproofing teardown, the mic port is an intentional acoustic leak. MEMS mics use hydrophobic meshes and LCP diaphragms, but the DSP compensates for water-induced frequency shifts with adaptive EQ rather than assuming a hermetic seal.

Power, Clocking, and Firmware Realities

Achieving true always-on means designing for leakage, not just active power. Leading implementations use:

• Dual-rail SRAM: 0.55V retention for TCM during VAD-only phases, with bitline assist to avoid failures at low voltage.
• DCXO with spread-spectrum: A 38.4MHz digitally-compensated crystal oscillator stays on; PLLs remain off. Jitter <2ps RMS to avoid intermodulation in PDM clocks.
• Adaptive duty cycling: When ambient SPL <35dBA for >10s, the DSP drops to 4MHz and runs VAD every 30ms instead of 10ms, saving ~35% power with negligible detection loss.
• Firmware partitioning: A 40KB ROM bootloader + 200KB signed patch loaded into TCM. No XIP from flash — flash reads would cost 20x energy. OTA updates are delta-patched via the AP but verified inside the island.

Latency budgets are contractual: VAD to AP interrupt <180ms, AP resume to UI <400ms. OEMs tune this against false-accept rate; a 5ms reduction in VAD window often buys 2% recall at the cost of 15% more power. Thermal is negligible, but acoustic coupling is not — vibration from haptics or speakers can saturate PDm inputs, requiring the DSP to blank 10-20ms post-haptic.

Architecture

Firmware Snapshot: Configuring the Pipeline

Most vendors expose configuration via a binary topology file, but low-level validation often requires register-level C. The snippet below shows a representative LP-DSP initialization for 2-mic PDM + VAD + first-stage KWS — simplified from a Tensilica HiFi4 BSP.

/* lpad_config.c — LP Audio DSP boot, 48kHz 2x PDM, TCM-only */
#include "hifi4_lpad.h"
#define TCM_BASE  0x1E000000
#define VAD_THRESH_DB  -42  // adaptive floor + 6dB

void lpad_audio_init(void) {
  /* 1. Clock: stay on DCXO 38.4MHz, div 8 for 4.8MHz idle */
  LPAD_CCR = DCXO_EN | CLK_DIV(8) | SRAM_RETEN_0P55V;

  /* 2. PDM: 3.072MHz, stereo, CIC order 4 */
  PDM_CTRL = PDM_EN | PDM_RATE_3072K | CIC_ORDER_4;
  DMA_CFG  = DMA_PINGPONG | DMA_TCM_ONLY | DMA_WIDTH_24B;

  /* 3. VAD: 10ms frames, gate downstream if no speech >500ms */
  vad_config_t vad = {
    .frame_ms = 10, .thresh_db = VAD_THRESH_DB,
    .hangover_ms = 500, .clock_gate = true
  };
  vad_init(&vad, (int16_t*)TCM_BASE);

  /* 4. KWS stage-1: 100KB int8 D-S CNN, TCM-resident */
  kws_model_t *m = kws_load((void*)(TCM_BASE + 0x20000), "kws_s_v2.int8");
  m->threshold = 720; // tuned for <0.5 FA/hr
  kws_set_callback(m, &adsp_mailbox_wake);
}
// Power: ~1.4mW @ 4.8MHz idle, ~2.1mW during KWS scoring

The key detail is DMA_TCM_ONLY — any spill to DRAM would force the memory controller on, adding ~30mW. Production firmware also enables BCLK gating between PDM samples, and the adsp_mailbox_wake path is rate-limited to once per second to prevent adversarial audio from hammering the AP.

Integration Pitfalls and What’s Next

Even with clean hardware, teams stumble on: clock-domain crosstalk causing PDM bit errors, insufficient TCM for new 90KB models after feature creep, and defeat of AEC when speakers are driven into non-linear compression. Validation requires an acoustic chamber that replays calibrated SPS (speech + noise) for 100+ hours while monitoring battery drain — not an emulator.

Next-gen islands are converging DSP and NPU: Hexagon v75 and HiFi 1.5 both add 8-bit vector MACs for Transformer-based KWS without waking the big NPU. Expect 2027 SoCs to run full speaker diarization and on-device LLM rescoring (<500M params quantized) entirely within the 15mW audio domain, triggered only by the LP stage. Combined with multi-modal sensing — lidar-assisted beamforming and display-aware mic muting — the DSP becomes the phone’s subconscious, filtering reality before the AP ever wakes.

For platform architects, the lesson is simple: budget milliwatts, not megahertz. The competitive edge in mobile audio is no longer codec quality alone but how silently, securely, and persistently the device can sense.