DSO

Engel 2016 · Paper

One-line summary — Direct Sparse Odometry performs windowed photometric bundle adjustment over a sparse, evenly distributed set of high-gradient pixels, combining a fully direct probabilistic model with consistent joint optimisation of all model parameters.

Problem

Before DSO, the direct and sparse worlds each kept only half the prize. LSD-SLAM tracked photometrically but refined geometry with filtering and a pose graph, never jointly optimising structure and motion; feature-based methods (ORB-SLAM) did full bundle adjustment but reduced images to keypoints. DSO combines “a fully direct probabilistic model (minimizing a photometric error) with consistent, joint optimization of all model parameters” — poses, inverse depths, intrinsics, and per-frame brightness — in real time, made possible “by omitting the smoothness prior used in other direct methods and instead sampling pixels evenly throughout the images.”

Method & architecture

• Photometric calibration: image formation is modelled as $I_i(\mathbf{x}) = G\big(t_i\, V(\mathbf{x})\, B_i(\mathbf{x})\big)$ with response function $G$, vignette $V$, exposure $t_i$, irradiance $B_i$; frames are corrected up-front via $I_i'(\mathbf{x}) := t_i B_i(\mathbf{x}) = G^{-1}(I_i(\mathbf{x}))/V(\mathbf{x})$.
• Photometric error: a point $\mathbf{p}$ hosted in frame $I_i$ and observed in $I_j$ contributes a weighted SSD over an 8-pixel residual pattern $\mathcal{N}_{\mathbf{p}}$:

$$E_{\mathbf{p}j} := \sum_{\mathbf{p}\in\mathcal{N}_{\mathbf{p}}} w_{\mathbf{p}} \left\| (I_j[\mathbf{p}'] - b_j) - \frac{t_j e^{a_j}}{t_i e^{a_i}} (I_i[\mathbf{p}] - b_i) \right\|_{\gamma},$$

where $\mathbf{p}' = \Pi_{\mathbf{c}}\big(\mathbf{R}\, \Pi_{\mathbf{c}}^{-1}(\mathbf{p}, d_{\mathbf{p}}) + \mathbf{t}\big)$ is the reprojection with inverse depth $d_{\mathbf{p}}$ and relative pose $\mathbf{T}_j \mathbf{T}_i^{-1}$; $a, b$ are affine brightness parameters (logarithmic parametrisation prevents drift), $\|\cdot\|_\gamma$ the Huber norm, and $w_{\mathbf{p}} = c^2 / (c^2 + \|\nabla I_i(\mathbf{p})\|_2^2)$ down-weights high-gradient pixels (implicit geometric noise). The total energy is

$$E_{\text{photo}} := \sum_{i\in\mathcal{F}} \sum_{\mathbf{p}\in\mathcal{P}_i} \sum_{j\in\mathrm{obs}(\mathbf{p})} E_{\mathbf{p}j},$$

optimised by Gauss–Newton over a sliding window; each point has just one unknown (inverse depth in its host frame), so after the Schur complement the system solves like classical sparse BA.

• Marginalisation: old frames and points leave the window via the Schur complement, $\widehat{\mathbf{H}_{\alpha\alpha}} = \mathbf{H}_{\alpha\alpha} - \mathbf{H}_{\alpha\beta}\mathbf{H}_{\beta\beta}^{-1}\mathbf{H}_{\beta\alpha}$, leaving a quadratic prior on the remaining variables; the linearisation (tangent) point of marginalised-connected variables stays fixed thereafter, and residuals that would fill in the Hessian are dropped.
• Front-end: a window of $N_f = 7$ active keyframes and $N_p = 2000$ active points. New frames are tracked against the latest keyframe by two-frame direct alignment with a constant-motion model; keyframes are created when a weighted score of mean optical flow, translation-only flow, and brightness change exceeds a threshold (initially 5–10 keyframes/s, then redundant ones are marginalised early — always keeping the latest two, dropping frames with under 5% visibility, otherwise by a spatial distance score). Candidate points are picked with a region-adaptive gradient threshold (median block gradient $\bar g + g_{\text{th}}$, $g_{\text{th}}=7$) over $d \times d$ blocks — so edges and smooth intensity variations on white walls are sampled too — tracked along the epipolar line for a depth initialisation, then activated to maximise spatial spread.

Results

Evaluated on three datasets, every sequence run forwards and backwards (500 runs on TUM monoVO — 50 photometrically calibrated sequences, 105 min; 220 runs on EuRoC MAV — 11 sequences, 19 min; 80 runs on ICL-NUIM — 8 sequences, 4.5 min), reported as cumulative error plots capturing both accuracy and robustness:

• TUM monoVO and ICL-NUIM: “the direct, sparse approach clearly outperforms ORB-SLAM in accuracy and robustness” (alignment/rotation/scale-drift error on monoVO; ATE on ICL-NUIM). LSD-SLAM and SVO “consistently fail on most of the sequences” due to their brightness-constancy assumption under real exposure changes.
• EuRoC MAV: ORB-SLAM is more accurate (no photometric calibration available, and its local mapping implicitly closes the datasets’ many small loops, which pure odometry cannot); when ORB-SLAM is restricted to map points seen within the last 10 s it performs similar to DSO in accuracy but is less robust.
• Efficiency: real-time on an i7-4910MQ; at reduced settings ($N_p = 800$, $N_f = 6$, 424×320) DSO still achieves very good accuracy and robustness while running at 5× real-time speed.
• Ablations: full photometric calibration gives the best results and a naïve brightness-constancy assumption performs clearly worst; using more than $N_p = 500$ points or $N_f = 7$ frames has only marginal accuracy impact (2000 points are kept mainly for denser maps).

Why it matters for SLAM

DSO is the definitive direct sparse method, closing the era opened by DTAM and LSD-SLAM and standing alongside ORB-SLAM as one of the two canonical classical baselines (direct vs feature-based). Its formulation spawned an entire family — LDSO, Stereo DSO, VI-DSO, DVSO, D3VO, DM-VIO — and its photometric-calibration insights, fixed-tangent marginalisation, and sliding-window design are still standard practice in odometry systems today.

Related

• LSD-SLAM
• LDSO
• DVSO
• D3VO
• Stereo DSO
• Marginalization