SfM-Learner

Zhou 2017 · Paper

One-line summary — Jointly trains a single-view depth network and a multi-view pose network from unlabeled monocular video using photometric view synthesis as the only supervision, founding the self-supervised depth + ego-motion field.

Problem

Supervised depth and pose estimation need expensive ground truth — LiDAR scans for depth, motion capture or high-grade GPS/INS for poses — which limits training data to a handful of instrumented datasets. Yet unstructured monocular video is essentially free and unlimited.

The insight: a view synthesis system “only performs consistently well when its intermediate predictions of the scene geometry and the camera poses correspond to the physical ground-truth”. So train networks on the ‘meta’-task of view synthesis, and depth and ego-motion must emerge as intermediate predictions.

Method & architecture

• Two networks, coupled only by the loss. The depth network (DispNet-style encoder-decoder with skip connections and multi-scale side predictions; depth output $1/(\alpha \cdot \mathrm{sigmoid}(x)+\beta)$ with $\alpha=10,\beta=0.01$) sees a single target frame. The pose network (target + source frames concatenated; 7 stride-2 convolutions, then a 1×1 conv with $6(N-1)$ channels — 3 Euler angles + 3D translation per source view — and global average pooling) predicts relative poses $\hat{T}_{t\rightarrow s}$. They train jointly but run independently at test time.
• View synthesis objective. With target $I_t$ and sources $I_s$:

$$\mathcal{L}_{vs}=\sum_{s}\sum_{p}\left|I_{t}(p)-\hat{I}_{s}(p)\right|,$$

where $\hat{I}_{s}$ is $I_s$ warped into the target frame. Each target pixel $p_t$ is projected into the source view via

$$p_{s}\sim K\hat{T}_{t\rightarrow s}\hat{D}_{t}(p_{t})K^{-1}p_{t},$$

($K$ intrinsics, $\hat{D}_t$ predicted depth) and $\hat{I}_s(p_t)$ is filled by differentiable bilinear sampling of the 4-pixel neighbours (spatial transformer networks). Only correct depth and pose make the warp photo-consistent.

• Explainability mask. A third head (sharing the pose encoder, 5 deconv layers, per-pair softmax) predicts a soft mask $\hat{E}_{s}$ weighting the loss, $\mathcal{L}_{vs}=\sum_{s}\sum_{p}\hat{E}_{s}(p)\,|I_{t}(p)-\hat{I}_{s}(p)|$, discounting moving objects, occlusion, and non-Lambertian effects; a cross-entropy regularizer toward 1 prevents the trivial all-zero mask.
• Gradient locality fix. Bilinear-sampling gradients are local, so the full objective is applied over an image pyramid with an L1 penalty on second-order depth gradients for smoothness:

$$\mathcal{L}_{final}=\sum_{l}\mathcal{L}_{vs}^{l}+\lambda_{s}\mathcal{L}_{smooth}^{l}+\lambda_{e}\sum_{s}\mathcal{L}_{reg}(\hat{E}_{s}^{l}).$$

• Training. TensorFlow; $\lambda_s=0.5/l$, $\lambda_e=0.2$; Adam (lr 0.0002, batch 4), ~150K iterations; 3-frame snippets at 128×416 (44,540 KITTI sequences: 40,109 train / 4,431 val).

Results

• KITTI depth (Eigen split, 697 test images, median-scaled): Abs Rel 0.208 trained on KITTI, 0.198 with Cityscapes pre-training — comparable to depth-supervised Eigen et al. (0.203 Fine) and close to pose-supervised Garg et al. (cap 50 m: 0.201 vs 0.169), though behind concurrent stereo-trained Godard et al. (0.148). Remarkable given zero labels.
• Explainability ablation is modest (0.221 → 0.208 Abs Rel) — KITTI is mostly static over 3-frame snippets.
• Pose (KITTI odometry 09/10, ATE over 5-frame snippets): 0.021±0.017 / 0.020±0.015 m — better than ORB-SLAM (short) run on the same 5-frame snippets (0.064±0.141 / 0.064±0.130) and the mean-odometry baseline, behind only ORB-SLAM (full), which uses whole sequences with loop closure and relocalization (0.014±0.008 / 0.012±0.011).
• Make3D zero-shot: Abs Rel 0.383 without any Make3D training — captures global layout, with a gap to Make3D-supervised methods.
• Inherited limits stated in the paper: scale ambiguity remains, intrinsics must be known, and static-scene/photo-consistency assumptions bound accuracy — the same assumptions that limit direct SLAM.

Why it matters for SLAM

SfM-Learner showed that the classical SfM/SLAM objective — photometric consistency across views — can serve as a training signal instead of a runtime cost, letting networks learn depth and ego-motion from raw video. Its view-synthesis loss became the template for Monodepth2, D3VO, and dozens of self-supervised VO/depth systems that now feed learned depth priors back into SLAM pipelines. It marks the conceptual bridge between direct SLAM and end-to-end learned odometry.

Related

• Self-supervised depth — the concept this paper launched
• MonoDepth — stereo-supervised sibling approach
• DeepVO — supervised learned odometry counterpart
• UndeepVO — follow-up recovering absolute scale from stereo training
• DeMoN — concurrent supervised two-view depth + motion
• D3VO — self-supervised predictions integrated into direct VO