ORB-SLAM3

Campos 2020 · Paper

One-line summary — The first SLAM library to support visual, visual-inertial, and multi-map SLAM across monocular, stereo, and RGB-D cameras with pinhole and fisheye models, substantially improving accuracy over prior systems.

Problem

ORB-SLAM2 lacked IMU integration and could not recover from tracking failures: once tracking was lost, the map was effectively gone. Real-world deployments — AR headsets, drones, long robot missions — require surviving temporary occlusion, degenerate motion, and long periods of poor visual information. ORB-SLAM3 (IEEE TRO, University of Zaragoza) addresses both gaps: a tightly-integrated visual-inertial system “that fully relies on Maximum-a-Posteriori (MAP) estimation, even during the IMU initialization phase”, and an Atlas multi-map mechanism that turns tracking loss from a catastrophe into a recoverable event.

Method & architecture

The system keeps ORB-SLAM2’s thread layout, extended for multi-map operation: a tracking thread localizes each frame against the active map (adding body velocity and IMU biases to the optimized state in inertial modes); a local mapping thread grows and refines the active map with visual or visual-inertial BA over a sliding window of keyframes, and runs IMU initialization; and a loop & map merging thread searches the whole Atlas at keyframe rate — a match in the active map triggers loop closure, a match in another map triggers a seamless merge (welding-window BA, then essential-graph pose optimization), with full BA afterwards in a separate thread. The camera model (projection, unprojection, Jacobians) is abstracted into a module, so pinhole and Kannala-Brandt fisheye work throughout, including MLPnP-based relocalization and non-rectified stereo treated as two monocular cameras with a fixed extrinsic SE(3)\mathrm{SE}(3) constraint.

In visual-inertial mode the per-keyframe state is Si≐{Ti,vi,big,bia}\mathcal{S}_{i}\doteq\{\mathbf{T}_{i},\mathbf{v}_{i},\mathbf{b}^{g}_{i},\mathbf{b}^{a}_{i}\} (pose, velocity, gyro/accelerometer biases). IMU preintegration between keyframes gives ΔRi,i+1,Δvi,i+1,Δpi,i+1\Delta\mathbf{R}_{i,i+1},\Delta\mathbf{v}_{i,i+1},\Delta\mathbf{p}_{i,i+1} and inertial residuals rIi,i+1\mathbf{r}_{\mathcal{I}_{i,i+1}}, e.g. the rotation term rΔRi,i+1=Log(ΔRi,i+1TRiTRi+1)\mathbf{r}_{\Delta\mathrm{R}_{i,i+1}}=\mathrm{Log}\left(\Delta\mathbf{R}_{i,i+1}^{\mathrm{T}}\mathbf{R}_{i}^{\mathrm{T}}\mathbf{R}_{i+1}\right). With reprojection residuals rij=uij−Π(TCBTi−1⊕xj)\mathbf{r}_{ij}=\mathbf{u}_{ij}-\Pi\left(\mathbf{T}_{\mathrm{CB}}\mathbf{T}_{i}^{-1}\oplus\mathbf{x}_{j}\right), visual-inertial SLAM is the keyframe-based MAP problem

min⁡Sˉk,X(∑i=1k∥rIi−1,i∥ΣIi,i+1−12+∑j=0l−1∑i∈KjρHub(∥rij∥Σij−1))\min_{\bar{\mathcal{S}}_{k},\mathcal{X}}\left(\sum_{i=1}^{k}\left\lVert\mathbf{r}_{\mathcal{I}_{i-1,i}}\right\rVert_{\Sigma_{\mathcal{I}_{i,i+1}}^{-1}}^{2}+\sum_{j=0}^{l-1}\sum_{i\in\mathcal{K}^{j}}\rho_{\mathrm{Hub}}\left(\left\lVert\mathbf{r}_{ij}\right\rVert_{\Sigma_{ij}^{-1}}\right)\right)

with a Huber kernel on visual terms only, since inertial measurements have no mis-associations.

MAP-based IMU initialization proceeds in three steps: (1) vision-only — 2 s of pure monocular SLAM gives an accurate up-to-scale trajectory; (2) inertial-only — the state Yk={s,Rwg,b,vˉ0:k}\mathcal{Y}_{k}=\{s,\mathbf{R}_{\mathrm{wg}},\mathbf{b},\bar{\mathbf{v}}_{0:k}\} (scale, gravity direction, biases, velocities) is solved by

Yk∗=arg⁡min⁡Yk(∥b∥Σb−12+∑i=1k∥rIi−1,i∥ΣIi−1,i−12)\mathcal{Y}_{k}^{*}=\arg\min_{\mathcal{Y}_{k}}\left(\|\mathbf{b}\|_{\Sigma_{b}^{-1}}^{2}+\sum_{i=1}^{k}\|\mathbf{r}_{\mathcal{I}_{i-1,i}}\|_{\Sigma_{\mathcal{I}_{i-1,i}}^{-1}}^{2}\right)

holding the visual trajectory fixed; (3) joint visual-inertial BA. Tracking losses are handled in two stages: short-term (pose from IMU, wide-window map-point re-matching) and long-term (start a fresh active map in the Atlas). The improved place recognition raises DBoW2’s recall by verifying candidates geometrically against covisible keyframes already in the map instead of waiting for three consecutive keyframe detections, and by intensively searching mid-term matches in a local window around the matched keyframe.

Results

All experiments ran on an Intel Core i7-7700 CPU. On EuRoC (median of 10 runs, RMS ATE), ORB-SLAM3 beats prior systems in all four configurations: stereo-inertial averages 0.035 m (0.6% scale error) across all 11 sequences, monocular-inertial 0.043 m, stereo 0.084 m, monocular 0.041 m on completed sequences. Monocular-inertial is “five to ten times more accurate than MCSKF, OKVIS and ROVIO, and more than doubles the accuracy of VI-DSO and VINS-Mono”; stereo-inertial is three to four times more accurate than Kimera and VINS-Fusion. The IMU initialization reaches 5% scale error within 2 seconds and about 1% after refinement. On the fisheye TUM-VI benchmark it outperforms VINS-Mono and Basalt overall; in the AR/VR-like room sequences average ATE is 0.009 m stereo-inertial and 0.011 m monocular-inertial. In EuRoC multi-session experiments the Atlas merging more than doubles the accuracy of CCM-SLAM and VINS-Mono (advantage over VINS-Mono grows from 2.6× single-session to 3.2× multi-session). Overall the abstract claims “two to ten times more accurate than previous approaches”; the open-source release became the standard baseline in hundreds of subsequent papers.

Why it matters for SLAM

ORB-SLAM3 is the culmination of the feature-based SLAM lineage that began with PTAM and ORB-SLAM, and on release it was the most complete and accurate open-source SLAM library available — the first to exploit short-term, mid-term, long-term, and multi-map data association in all algorithm stages. The multi-map Atlas made lifelong, failure-tolerant operation practical, and its MAP-based visual-inertial initialization set a new standard for VIO systems. It remains one of the most common baselines and production starting points in the field.