UndeepVO

Li 2018 · Paper

One-line summary — A monocular visual odometry system trained with unsupervised deep learning that recovers absolute metric scale by training on stereo image pairs while running on monocular input at test time.

Problem

Supervised learned VO (DeepVO) needs ground-truth 6-DoF poses, which are expensive to collect at scale, while the first self-supervised alternative (SfM-Learner) trained from monocular video alone and therefore inherited the fundamental monocular limitation: depth and trajectory are recovered only up to an unknown scale. UnDeepVO’s answer: let training-time stereo geometry, not runtime sensors or post-processing, supply the scale — its two claimed “salient features” are the unsupervised training scheme and absolute scale recovery.

Method & architecture

• Pose estimator. A VGG-based CNN takes two consecutive monocular frames and regresses the 6-DoF transform, with decoupled fully-connected branches for translation and Euler-angle rotation so the rotation can be weighted separately (rotation is highly nonlinear and harder to train).
• Depth estimator. An encoder-decoder that predicts depth maps directly rather than disparity — the authors report the whole system converges more easily this way.
• Stereo training, monocular testing. During training both left and right image sequences are fed through the networks; the losses tie the predictions to the known, fixed baseline $B$. At test time only consecutive monocular images are used.
• Spatial losses (within a stereo pair) exploit that corresponding pixels are displaced by the disparity

$$D_{p}=Bf/D_{dep},$$

($f$ focal length, $D_{dep}$ predicted depth) — this is where metric scale enters. One image is synthesized from the other via a spatial transformer and penalized with a combined SSIM + L1 photometric loss, e.g.

$$L_{pho}^{l}=\lambda_{s}L^{SSIM}(I_{l},I^{\prime}_{l})+(1-\lambda_{s})L^{l_{1}}(I_{l},I^{\prime}_{l}),$$

plus a disparity-map consistency loss and a pose consistency loss $L_{pos}=\lambda_{p}L^{l_{1}}(\mathbf{x}^{\prime}_{l},\mathbf{x}^{\prime}_{r})+\lambda_{o}L^{l_{1}}(\varphi^{\prime}_{l},\varphi^{\prime}_{r})$ forcing left- and right-sequence pose predictions to agree.

• Temporal losses (between consecutive frames) recover the motion: pixels are reprojected with the predicted depth and pose,

$$p_{k+1}=KT_{k,k+1}D_{dep}K^{-1}p_{k},$$

and penalized with the same SSIM+L1 photometric loss, plus an ICP-like 3D geometric registration loss $L_{geo}^{k}=L^{l_{1}}(P_{k},P^{\prime}_{k})$ aligning the point clouds of the two frames under $T_{k,k+1}$.

• Training. Adam ($\beta_1=0.9$, $\beta_2=0.99$), lr 0.001 halved every 1/5 of iterations, 20–30 epochs, 416×128 inputs, with color/rotational/left-right augmentation. Inference: under 400 MB GPU memory at 40 Hz on a GTX 980M.

Results

• KITTI odometry (trained on sequences 00–08, the same data as SfMLearner; no loop closure in any method): mean over sequences 00/02/05/07/08 — UnDeepVO $t_{rel}$ 4.07%, $r_{rel}$ 2.02°/100m, vs SfMLearner 36.23% / 4.56 (after 7-DoF alignment), VISO2-M 17.93% / 2.80 and ORB-SLAM-M (VO-only) 27.05% / 10.23, both of which required full 1242×376 images vs UnDeepVO’s 416×128. (These sequences overlap the training set — the paper frames this as measuring goodness of network fit; on GT-less sequences 11–21 its trajectories are comparable to stereo VISO2-S.)
• Scale. Poses and depths come out metrically scaled with no scale post-processing — SfMLearner and ORB-SLAM-M needed 7-DoF alignment to ground truth, VISO2-M needed the fixed-camera-height prior.
• Depth (KITTI Eigen split): Abs Rel 0.183, Sq Rel 1.73, RMSE 6.57, RMSE log 0.268 — better than supervised Eigen et al. (0.214) and scale-free SfMLearner (0.208), behind stereo-trained MonoDepth (0.148, which used higher resolution, a ResNet architecture, and the full KITTI raw set).

Why it matters for SLAM

UndeepVO was one of the first works to show that the monocular scale problem can be attacked with training data geometry rather than extra sensors at runtime: use stereo supervision once, deploy monocular forever. This stereo-supervised self-supervision recipe became a standard ingredient in later self-supervised depth and VO methods (MonoDepth-style photometric losses combined with pose networks, later refined and integrated into direct VO by D3VO). It is a useful case study of how classical multi-view geometry constraints can serve as free supervisory signals for learning.

Related

• SfM-Learner — the purely monocular self-supervised depth+pose predecessor (scale-ambiguous).
• MonoDepth — stereo-supervised self-supervised depth estimation.
• DeepVO — supervised end-to-end learned VO counterpart.
• D3VO — later system integrating self-supervised depth, pose, and uncertainty into direct VO.
• Self-supervised depth — the general concept.
• Scale ambiguity — the monocular limitation UnDeepVO trains its way around.