LiDAR and ToF Sensors in Mobile: 3D Scanning and AR Applications

Apple's LiDAR scanner and Android's direct Time-of-Flight modules look identical — a small black dot next to the cameras. Underneath, they represent two fundamentally different approaches to measuring depth with light. Understanding dToF vs iToF, VCSEL arrays, and SPAD detectors explains why some phones can scan a room in 3D while others only blur the background.

Depth sensing is the most underappreciated subsystem in modern smartphones. While image sensors chase megapixels, a tiny optoelectronic module firing invisible infrared pulses quietly enables portrait mode, autofocus in darkness, AR furniture placement, and real-time 3D mesh generation. From the iPhone 15 Pro to the Galaxy S24 Ultra, depth hardware has shifted from simple laser autofocus to true 3D perception. This article dissects how mobile LiDAR and ToF actually work, where each technology wins, and what software unlocks when depth leaves the camera app.

1. How Mobile Depth Sensing Actually Works

All Time-of-Flight systems share one principle: emit light, measure its return time, calculate distance using d = (c × Δt) / 2, where c is the speed of light and Δt is the round-trip time. At one meter, that round trip is just 6.6 nanoseconds. To resolve depth to ±1 cm at 5 meters, the sensor must discriminate time differences below 67 picoseconds. That precision is not achieved with conventional CMOS pixels.

Mobile implementations overcome this with two distinct architectures:

Indirect ToF (iToF): Phase Measurement

iToF modules, common in phones from 2019-2022 such as the Samsung Galaxy S20 Ultra and Huawei P30 Pro, use a continuous-wave modulated infrared signal, typically at 940nm. A VCSEL illuminator emits amplitude-modulated light (often 20–100 MHz). The sensor pixels perform homodyne mixing, measuring phase shift (φ) between emitted and reflected signals. Distance is derived as d = (c × φ) / (4π × fmod). Four phase samples (0°, 90°, 180°, 270°) are used to reject ambient light and compute both phase and amplitude.

Advantages are low peak power and use of standard CIS processes with specialized photonic mixer devices. Limitations are critical: multipath interference (light bouncing off two surfaces corrupts the phase), limited unambiguous range inversely proportional to modulation frequency, and motion artifacts because four sequential exposures are needed. That is why early iToF phones struggled with edge bleeding around hair and glasses.

Direct ToF (dToF): Photon Counting

Modern mobile LiDAR — including Apple’s LiDAR Scanner and Sony’s IMX570 depth sensor in flagships — is true dToF. It fires sub-nanosecond VCSEL pulses (typically 1–3 ns, 1–2W peak but <500mW average) and measures the exact arrival time of individual photons with Single-Photon Avalanche Diodes (SPADs). Each SPAD feeds a Time-to-Digital Converter (TDC) that timestamps photon arrivals into a histogram with bins of 50–125 picoseconds. The histogram peak corresponds to the object distance.

Because it uses time-correlated single photon counting (TCSPC), dToF is inherently robust to ambient light and multipath. It needs only one exposure, enabling motion-robust operation at 30–60 fps. The trade-off is die complexity: SPADs require high-voltage guard rings, in-pixel TDCs, and histograms that increase silicon area. This is where node advantage matters — see how TSMC 3nm vs Samsung 2nm foundry competition is enabling smaller, more power-efficient SPAD SoCs.

2. Mobile LiDAR Is Not Automotive LiDAR

The term “LiDAR” on phones confuses consumers. Automotive LiDAR uses mechanical scanning, 1550nm lasers, and APDs to reach 200 meters. Mobile “LiDAR” is a solid-state flash dToF array with no moving parts. Apple’s module projects a diffractive optical element (DOE) that splits a single VCSEL array into ~500–1000 infrared dots across a 63° FoV, flood-illuminating the scene. An 8×8 or 16×16 zone SPAD array with microlenses collects returns. Effectively it is a low-resolution depth camera (e.g., 8×16 to 10×16 points per zone × 64 zones = 5k–10k points), interpolated to match RGB resolution (256×192 depth map on iOS).

Android dToF modules from STMicroelectronics (VL53L8) and Sony take a different approach: a single VCSEL flood without DOE, paired with a 64-zone multi-zone sensor. The Samsung Galaxy S26 Ultra teardown confirms Samsung’s latest shift to an 8×8 zoned dToF for faster laser autofocus rather than full LiDAR-style scanning — a cost/power decision. Full LiDAR draws 300–600 mW active; single-zone AF dToF draws <60 mW. That matters for thermal and battery.

3. Hardware Architecture Under the Glass

A complete dToF stack contains five elements stacked under the rear glass cover (which must pass 940nm with >92% transmission and AR coating):

1. VCSEL Array + Driver: 3–8 junction VCSELs driven by a GaAs or CMOS pulse driver with <500 ps rise time. Peak pulse current hits 2–5 A, requiring a low-ESL path and decoupling capacitor microns from the emitter.

2. Optics: Collimating lens, DOE for dot projection (LiDAR) or diffuser for flood (ToF AF), and a narrow bandpass filter (10–25 nm FWHM at 940±5nm) in front of the receiver to suppress ambient light to <1%.

3. SPAD Receiver SoC: Pixel pitch ~16–20 µm, Photon Detection Probability ~20–35% at 940nm, Dark Count Rate <50 cps/µm². Each macro-pixel contains 4–16 SPADs binned to one TDC to improve dynamic range.

4. Histogram & DSP: On-chip histogramming (128–512 bins) with Ambient subtraction, pile-up correction, and first/last peak extraction. Some chips output raw histograms for AP-side processing; others output calibrated distance per zone.

5. ISP & Fusion: The Application Processor fuses RGB, IMU, and depth into depth maps via temporal filtering and confidence maps. Apple’s Photonic Engine and Qualcomm’s Spectra ISP perform guided upsampling using RGB edges.

4. AR Applications: Where Depth Changes Reality

Portrait blur was the first win, but the real platform shift is spatial anchoring. ARKit and ARCore use depth in three ways: initial plane detection, raycast hit-testing, and occlusion.

Instant Plane Detection: Without depth, VIO (visual-inertial odometry) needs to move the phone to triangulate a plane, taking 1–3 seconds and failing on textureless surfaces like white tables. With dToF, a single frame provides a metric point cloud to fit a plane via RANSAC immediately. Hit-testing latency drops from ~800 ms to <200 ms. The user experience changes from “wave your phone” to “tap to place.”

Occlusion and Physics: A dense depth map lets virtual objects disappear behind real ones pixel-accurately. For a 1920×1440 RGB frame, the ISP upscales the 256×192 depth map using a joint bilateral filter guided by RGB edges, then runs a depth test per pixel in the GPU. This is computationally heavy — 4–8 ms on A17 Pro — but it enables believable room-scale AR. Without per-pixel depth, virtual furniture floats.

Room Scanning: iOS RoomPlan, Matterport, and PolyCam use LiDAR + IMU SLAM to build a 3D mesh. The phone streams ~10k points/frame at 30 Hz; a truncated signed distance function (TSDF) fuses points into a voxel mesh on-device (Apple Neural Engine handles TSDF updates). A 4×5 m room can be captured in ~45 seconds with ±2 cm wall accuracy, sufficient for floor plans, renovation estimates, and real estate. Android’s Depth API in ARCore now offers raw depth for similar reconstruction, though fewer devices ship with zoned sensors.

Behind the scenes, these scans raise familiar questions about sensor data exfiltration. Unlike camera images, mesh data reveals room layout, object sizes, and potentially sensitive geometry. Developers handling ARDepthData should review mobile DNS over HTTPS privacy protections and on-device processing patterns to avoid leaking spatial maps to third-party analytics.

5. 3D Scanning, Photogrammetry, and Point Clouds

LiDAR alone does not make a good 3D model. It provides metric scale; photogrammetry provides texture and sub-mm detail. The best mobile scanners fuse both.

A point cloud from mobile dToF is sparse and noisy by metrology standards. Expect 300–800 points per square meter at 1.5 m distance, with range noise ~0.8% of distance and systematic “wiggling” error from temperature drift of the VCSEL wavelength. Professional apps mitigate this with temporal averaging (accumulating 8–16 frames while the user holds steady), voxel downsampling, and plane smoothing. Exported PLY or USDZ files contain XYZ plus confidence per point (0–255). ARKit’s ARDepthData.confidenceMap lets developers filter low-confidence points (reflective surfaces, edges, sunlight).

For object scanning (shoes, faces, parts), Apple’s Object Capture API uses LiDAR only for bounding box scale estimation, then relies on 30–60 photogrammetry images around the object to generate a high-fidelity mesh via Neural Radiance Fields. Pure LiDAR mesh from a phone is unsuitable for manufacturing — but it is excellent for rapid inventory, crash documentation, or Volumetric video. One research workflow pioneered by mobile studios is night-mode LiDAR-assisted NeRF: depth priors reduce training iterations by ~40%.

// ARKit: depth map + confidence per frame
func session(_ session: ARSession, didUpdate frame: ARFrame) {
  guard let depth = frame.sceneDepth else { return }
  // depthMap: CVPixelBuffer 256x192 Float32 (meters)
  // confidenceMap: CVPixelBuffer UInt8 (0=low, 2=high)
  let width = CVPixelBufferGetWidth(depth.depthMap)   // 256
  let height = CVPixelBufferGetHeight(depth.depthMap) // 192
  
  // Example spec per pixel
  // depth 1.247 m | confidence 2 (high) | at (128,96)
  
  // Filter low-confidence before TSDF fusion
  // Only fuse where confidence == 2 and 0.2 < depth < 5.0
}

// Android ARCore equivalent
// frame.acquireDepthImage16Bits() // 160x120 UINT16 mm
// depthImage.getPlanes()[0].getBuffer()

Technology:        Stacked SPAD dToF (65nm + 40nm)
Resolution:        100k SPADs, 48 zones (8x6)
Wavelength:        940nm VCSEL, 10k dots via DOE (LiDAR variant)
Histogram bins:    128 x 100ps (12.8ns window ≈ 1.92m per sub-range)
Range:             0.1 - 5.0 m, dual-range stitching
Accuracy:          ±1.5% (>1m), 30 fps full histogram
Interface:         MIPI CSI-2 2-lane + I2C
Power:             450 mW typical (depth mode), 58 mW AF-only
Cover glass:       940nm AR, 2-layer bandpass

6. Limitations: When Depth Fails

Marketed demos hide failure modes every developer discovers. Specular surfaces (mirrors, polished metal, glass) reflect the VCSEL away or create multipath doubling, producing holes or phantom depths behind glass. Black fabrics and hair absorb 940nm, returning too few photons — confidence drops to zero and depth collapses to invalid. Sunlight saturates SPADs despite the bandpass filter; outdoors, usable range often halves and noise triples. Designers should provide fallback UI for sunlight and always cross-check depth against VIO.

Power and thermal throttling are equally real. Continuous 60 fps LiDAR heats the camera bump by 6–9 °C. After ~7 minutes, iOS reduces depth frame rate to 30 fps, then pauses session. Apps that need sustained scanning (Matterport-style tours) must duty-cycle or collect depth intermittently. Under-display depth modules, which Apple and Samsung are prototyping for 2027, will face additional 30–40% transmission loss through OLED — requiring higher VCSEL power or computational super-resolution to compensate.

Finally, eye safety constrains everything. Mobile VCSELs are IEC 60825 Class 1; maximum permissible exposure at 940nm limits peak power and pulse width. Extending range beyond 5 m either requires larger apertures (conflicts with thin phones) or MEMS scanning rather than flash — an approach some Android vendors reserve for tablet-sized devices.

7. What’s Next: From Depth Sensor to World Understanding

Depth hardware is converging. The next generation replaces histogram峰 detection with learned, per-pixel time-of-flight using histogram-as-input CNNs running on the ISP/NPU. STM’s VD55H1 already integrates a small neural engine for ambient rejection. Apple’s patents point to dual-frequency dToF with two pulse widths for short and long range without stitching. And Qualcomm’s Snapdragon 8 Gen 4 spectra integrates depth-guided super-resolution, borrowing the 200MP RGB stream to hallucinate 1024×768 depth via edge-aware upsampling.

For developers, the strategy is clear: treat depth as a signal with confidence, not ground truth. Validate with temporal consistency, fuse with VIO, and degrade gracefully. The phones that ship multi-zone dToF in 2026 — even if only for autofocus — are future-proofing for AR glasses tethering, where the phone becomes a world-mesher for the wearable. In that topology, low-latency depth over ultra-wideband and Wi-Fi 7, converged with satellite–5G NTN links, will matter more than raw range.

If you are building these features, test under the worst light, on black velvet, and through a window. The physics does not care about your demo video.

Related coverage: How 3nm/2nm nodes shrink SPAD SoCs · Privacy implications of spatial data · S26 Ultra dToF teardown