Deployed VIO
The highest-volume SLAM systems in the world are not research codebases — they are the VIO stacks inside consumer XR and mobile devices. Meta Quest headsets, Apple’s ARKit (iPhone/iPad, Vision Pro), and Google’s ARCore each run visual-inertial tracking on hundreds of millions of devices. They are worth studying as case studies because their constraints differ sharply from the benchmarks that academic systems optimize for.
What a commercial XR stack looks like
The common recipe is inside-out tracking:
- Sensors: multiple wide-FoV grayscale cameras (headsets typically use several; phones use one) plus one or more IMUs, hardware-synchronized.
- Estimator: a tightly-coupled visual-inertial estimator — sliding-window optimization or filtering — running continuously on-device.
- Extras on headsets: tracking of hand-held controllers and (on recent devices) hands, layered on top of the head-pose estimate.
- Developer surface: phone AR frameworks like ARKit and ARCore expose the VIO output as 6-DoF camera poses plus sparse feature points, plane detections, tracking-quality state, and persistent anchors.
Public technical detail is limited (these are proprietary systems), but the broad architecture — multi-camera + IMU tightly-coupled estimation with a relocalization/mapping layer — matches the open literature this level covers.
Benchmark priorities vs product priorities
| Academic benchmark | Shipped device | |
|---|---|---|
| Primary metric | trajectory RMSE | perceived tracking failures per hour |
| Environment | curated datasets | every bedroom, office, train, and mirror |
| Compute | desktop CPU/GPU | shared mobile SoC, hard power budget |
| Calibration | done offline, assumed correct | drifts with temperature and drops; refined online |
| Initialization | expert-recorded excitation | whatever motion an untrained user provides |
| Failure handling | sequence excluded / reported | must degrade gracefully, recover invisibly |
What deployment changes about the problem
- Robustness beats accuracy. A benchmark cares about RMSE; a headset cares that tracking never visibly fails — across dark bedrooms, blank walls, mirrors, moving trains, and users who shake their heads violently. Enormous engineering goes into degraded-mode behavior, fast relocalization, and graceful fallback (e.g., dropping to 3-DoF rotational tracking when visual features vanish) rather than peak accuracy. Failure taxonomies — low light, low texture, dynamic scenes, sensor saturation — drive the roadmap more than benchmark leaderboards.
- Latency and power are hard constraints. Motion-to-photon latency must stay low enough (combined with pose prediction and late-stage reprojection of rendered frames) to avoid motion sickness, and the whole stack must run continuously within a mobile power budget, sharing the SoC with rendering. This drives lightweight front-ends, fixed-point/DSP implementations, dedicated co-processors, and careful thread scheduling — and is why MSCKF-class efficiency (accuracy per CPU cycle) matters more here than anywhere else.
- Calibration at scale. Millions of units mean per-device factory calibration plus online refinement of camera-IMU extrinsics, intrinsics, and time offset — devices deform with temperature and drops. Online calibration, a research topic in OpenVINS-era papers, is table stakes in production.
- Mapping is a product feature. Persistent anchors, shared/cloud anchors for multi-user AR, and headset “roomscale” boundary systems are map reuse and relocalization problems — the deployed cousins of loop closure and multi-session mapping (the problem maplab tackled in the open literature).
- Privacy and safety constraints shape what can be stored or uploaded: maps are typically abstracted (feature descriptors and geometry, not photos) and processed on-device where possible.
- The user is part of the system. Initialization must work with whatever motion an untrained user provides (no “excite all axes” instructions), and observability-driven degradations (hover-like stillness, pure rotation) must be handled invisibly — a sharp contrast with datasets recorded by experts who know what VIO needs.
Why study them without source code?
Three reasons. First, the constraint set explains why the open literature looks the way it does: efficiency-obsessed filters, online calibration, fast relocalization, and consistency machinery are exactly what shipping products need. Second, the public artifacts — developer APIs (anchors, plane detection, tracking-state notifications), device teardowns, and published talks/papers from these teams — are informative about architecture even when implementations stay closed. Third, many open systems in this roadmap were written by people who came from or went to these teams; the design DNA is shared.
What to take away
Treat deployed VIO as the requirements document for this level: any open system you study can be graded against the deployment checklist — What happens in the dark? How fast is relocalization? Does it estimate its own calibration? Does it degrade gracefully? How many milliwatts? — and the gaps you find are the research problems industry actually pays for.
Why it matters for SLAM
Deployed VIO is proof that the material in this level is not academic: MSCKF-style filtering, preintegration, online calibration, and observability-aware design are exactly what ships in consumer devices. Studying the deployment constraints (robustness, latency, power, calibration drift) tells you which research problems industry actually pays for — and explains design choices in open systems written by authors who came from or went to these teams.
Related
- Tightly-coupled vs Loosely-coupled — the architecture deployed stacks use.
- MSCKF — the efficient filtering approach suited to embedded compute budgets.
- OpenVINS — open-source system with the online calibration machinery production stacks need.
- OKVIS — keyframe sliding-window VIO from an era of early AR/MAV deployment work.
- maplab — multi-session mapping and relocalization, the research analogue of persistent anchors.
- Observability — the theory behind graceful handling of degenerate user motion.