DVSO

Yang 2018 · Paper

One-line summary — Deep Virtual Stereo Odometry feeds CNN disparity predictions into DSO as “virtual stereo” measurements, achieving stereo-level accuracy and metric scale from a single camera.

Problem

Monocular VO that relies purely on geometric cues is “prone to scale drift and require[s] sufficient motion parallax in successive frames for motion estimation and 3D reconstruction.” Stereo rigs fix both problems but cost hardware, calibration, and baseline constraints. DVSO (“Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry”) asks whether a depth-prediction network can stand in for the second camera: keep DSO’s windowed photometric bundle adjustment, but add the constraints a stereo camera would have provided.

Method & architecture

StackNet — a stacked disparity network: SimpleNet (ResNet-50 encoder-decoder from DispNet, skip connections, resize-convolution upsampling) predicts left and right disparity maps at 4 scales from the left image alone; ResidualNet (12 residual blocks) takes SimpleNet’s output plus warped reconstructions and the ℓ1\ell_1 reconstruction error ele_l, and learns additive residuals: disps=dispsimple,s⊕dispres,s\mathit{disp}_s=\mathit{disp}_{\mathit{simple},s}\oplus\mathit{disp}_{\mathit{res},s}. Training is semi-supervised on stereo pairs with per-scale loss combining a self-supervised photometric term, a supervised term against Stereo DSO’s sparse reconstructions (no LiDAR needed), left-right consistency, second-order smoothness, and occlusion regularisation. The self-supervised term is

LUleft=1N∑x,yα 1−SSIM(Ileft,Ireconsleft)2+(1−α)∥Ileft−Ireconsleft∥1,α=0.84\mathcal{L}_{U}^{\mathit{left}}=\frac{1}{N}\sum_{x,y}\alpha\,\frac{1-\mathrm{SSIM}\big(I^{\mathit{left}},I^{\mathit{left}}_{\mathit{recons}}\big)}{2}+(1-\alpha)\big\lVert I^{\mathit{left}}-I^{\mathit{left}}_{\mathit{recons}}\big\rVert_1,\qquad \alpha=0.84

and the supervised term uses the reverse Huber (berHu) norm βϵ\beta_{\epsilon} on displeft−dispDSOleft\mathit{disp}^{\mathit{left}}-\mathit{disp}^{\mathit{left}}_{\mathit{DSO}} over the sparse pixel set ΩDSO\Omega_{\mathit{DSO}} — classical stereo geometry distilled into a monocular network.

Odometry — DVSO builds on monocular DSO’s windowed direct bundle adjustment (N=7N=7 keyframes, marginalisation via Schur complement) and uses the predictions in two ways. (1) Initialisation: each new point’s inverse depth is set from the left disparity, dp=DL(p)/(fxb)d_{\mathbf{p}}=D^{L}(\mathbf{p})/(f_x b), giving stable, metric initialisation; points failing a left-right consistency check elr=∣DL(p)−DR(p′)∣>1e_{lr}=|D^{L}(\mathbf{p})-D^{R}(\mathbf{p}')|>1 are rejected as likely occlusions. (2) Virtual stereo term: alongside DSO’s temporal photometric energy

Eijp=ωp∥(Ij[p~]−bj)−eajeai(Ii[p]−bi)∥γE_{ij}^{\mathbf{p}}=\omega_{\mathbf{p}}\left\lVert (I_j[\tilde{\mathbf{p}}]-b_j)-\frac{e^{a_j}}{e^{a_i}}(I_i[\mathbf{p}]-b_i)\right\rVert_{\gamma}

(with affine brightness parameters a,ba,b, gradient-dependent weight ωp\omega_{\mathbf{p}}, Huber norm ∥⋅∥γ\lVert\cdot\rVert_{\gamma}), each point gets a residual against a virtual right image synthesised from the predicted right disparity DRD^R:

Ei†p=ωp∥Ii†[p†]−Ii[p]∥γ,Ii†[p†]=Ii[p†−(DR(p†),0)⊤]E_i^{\dagger\mathbf{p}}=\omega_{\mathbf{p}}\left\lVert I_i^{\dagger}[\mathbf{p}^{\dagger}]-I_i[\mathbf{p}]\right\rVert_{\gamma},\qquad I_i^{\dagger}[\mathbf{p}^{\dagger}]=I_i\big[\mathbf{p}^{\dagger}-(D^{R}(\mathbf{p}^{\dagger}),0)^{\top}\big]

where p†=Πc(Πc−1(p,dp)+tb)\mathbf{p}^{\dagger}=\Pi_c(\Pi_c^{-1}(\mathbf{p},d_{\mathbf{p}})+\mathbf{t}_b) projects through the known virtual baseline tb\mathbf{t}_b. The total energy Ephoto=∑i∈F∑p∈Pi(λEi†p+∑j∈obs(p)Eijp)E_{photo}=\sum_{i\in\mathcal{F}}\sum_{\mathbf{p}\in\mathcal{P}_i}\big(\lambda E_i^{\dagger\mathbf{p}}+\sum_{j\in\mathrm{obs}(\mathbf{p})}E_{ij}^{\mathbf{p}}\big) is minimised by Gauss-Newton, so triangulated and predicted depth are traded off with robust norms inside the optimisation — the right image itself is never used at runtime.

Results

Depth (KITTI, Eigen split): StackNet reaches RMSE 4.442 m (0–80 m range), beating the self-supervised state of the art of Godard et al. (4.935) and the LiDAR-semi-supervised Kuznietsov et al. (4.621) on most metrics; at 1–50 m it scores 3.390 vs 3.518/3.729. Inference takes under 40 ms at 512×256. Odometry (KITTI): monocular DSO averages 65.6% translational drift (trelt_{rel}) over sequences 00–10; DVSO without baseline tuning reaches 1.06%, and the full system (in,vs,lr,tbin{,}vs{,}lr{,}tb) 0.77% / 0.20°, better than Stereo DSO (0.84/0.20), stereo ORB-SLAM2 without loop closure (0.81/0.26), and Stereo LSD-VO (1.14/0.40) — from one camera. It also outperforms end-to-end methods (DeepVO, UnDeepVO, SfMLearner) on all available sequences, and swapping StackNet for Godard’s depth degrades the mean to 1.51%, confirming the depth network matters.

Why it matters for SLAM

DVSO showed that learned depth can close the accuracy gap between monocular and stereo visual odometry, and its “CNN as a virtual stereo camera” framing — expressing the depth prior as a photometric residual with the same units as the geometric terms — became an influential pattern for integrating networks into direct pipelines. It is the middle step of the CNN-SLAM → DVSO → D3VO lineage, which D3VO completed by adding learned pose and uncertainty on top of learned depth; the geometry-teaches-network loop (Stereo DSO supervising the net that upgrades monocular DSO) also prefigured later self-improving systems.