DeepFactors

Czarnowski 2020 · Paper

One-line summary — DeepFactors (RA-L 2020) turns CodeSLAM’s learned compact depth codes into the first real-time probabilistic dense monocular SLAM system, using photometric, reprojection, and sparse geometric errors simultaneously as factors in standard factor-graph software.

Problem

Monocular SLAM had fragmented along three axes: scene geometry representation (sparse landmarks vs dense maps), the consistency metric used to optimize the multi-view problem (photometric vs reprojection vs geometric), and whether learned priors are used. Sparse methods allow real-time joint probabilistic inference but yield maps useless for interaction; dense methods discard cross-correlations and alternate tracking/mapping; CodeSLAM proved the compact-code idea but “lacked features of full SLAM, was not capable of real-time performance and did not generalise to real handheld camera scenes”. DeepFactors unifies these paradigms “in a probabilistic framework while still maintaining real-time performance”.

Method & architecture

Each keyframe carries a pose variable $p_i$ and a code variable $\mathbf{c}_i$; depth decodes linearly from the code so factors never need costly relinearization:

$$D_i = f(\mathbf{c}_i, I_i) = D_i^0 + J(I_i)\,\mathbf{c}_i ,$$

where $D_i^0$ is the zero-code depth and $J(I_i) = \partial D_i / \partial \mathbf{c}_i$ is an image-conditioned Jacobian. The network (U-Net feature extractor + linear-decoder VAE, plus an explicit code-prediction encoder for initialization and a per-pixel uncertainty $b$) is trained on ~1.4M ScanNet images at 256×192 with code size 32.

Three pairwise consistency errors become factors, all built on the warp $\omega_{ji}(\mathbf{x}, \mathbf{c}_i, I_i) = \pi(T_{ji}\,\pi^{-1}(\mathbf{x}, D_i(\mathbf{x})))$ with $T_{ji} \in SE(3)$:

$$e_{pho}^{ij} = \sum_{\mathbf{x}\in\Omega_i} \| I_i(\mathbf{x}) - I_j(\omega_{ji}(\mathbf{x}, \mathbf{c}_i, I_i)) \|^2 \qquad \text{(dense, direct)}$$

$$e_{rep}^{ij} = \sum_{(\mathbf{x},\mathbf{y})\in M_{ij}} \| \omega_{ji}(\mathbf{x}, \mathbf{c}_i, I_i) - \mathbf{y} \|^2 \qquad \text{(BRISK matches, Cauchy robust cost)}$$

$$e_{geo}^{ij} = \sum_{\mathbf{x}\in\Omega_i} \| \left[ T_{ji}\,\pi^{-1}(\mathbf{x}, D_i(\mathbf{x})) \right]_z - D_j(\hat{\mathbf{x}}) \|^2 \qquad \text{(depth consistency, Huber, sparse pixel samples)}$$

Zero-code prior factors keep codes inside the VAE’s Gaussian latent region. The system flow: incoming frames are tracked against the nearest keyframe with a GPGPU SE(3) Lucas–Kanade aligner (~250 Hz); new keyframes are initialized from the code-prediction network and connected to the last N keyframes with pairwise factors; the whole map is optimized as a batch MAP problem with iSAM2’s incremental Bayes-tree updates (GTSAM). Lightweight “one-way” frames (no own depth) feed extra photometric evidence into the latest keyframe and are then marginalized out. Local loops (pose criteria within the last 10 keyframes) and global loops (bag-of-words + tracking verification, reprojection factors only) are closed online.

Results

• Factor ablation on ScanNet validation scenes (Table I): every factor helps, reprojection mainly trajectory, geometric mainly reconstruction; on scene0084_00 ATE-RMSE drops from 0.131 m (photometric only) to 0.061 m combined and pc110 rises 69.14% → 73.66%.
• Reconstruction (ICL-NUIM + TUM, % of depth within 10% of ground truth, whole estimated trajectory): average 27.10 vs CNN-SLAM 19.77, Laina 14.62, LSD-BS 3.44 (Table II).
• Trajectory (TUM fr1, ATE m): fr1/360 0.142 vs CNN-SLAM 0.500; fr1/rpy 0.047 vs 0.261; beats CodeSLAM on all sequences (fr1/desk 0.119 vs 0.654) and is comparable to non-real-time DeepTAM (fr1/desk 0.078) while running in real time (Table III).
• Timing on a single GTX 1080: tracking ~250 Hz; per-keyframe network pass ~340 ms, of which only 16 ms is the forward pass (rest is the code Jacobian via tf.gradients). Released open source.

Why it matters for SLAM

DeepFactors is the bridge between the latent-code mapping line (CodeSLAM) and mainstream factor-graph SLAM engineering: it showed learned dense geometry can live inside the same probabilistic back-end as classical constraints rather than in a bespoke optimizer. Practically, it demonstrated that combining direct, feature-based, and learned-prior cues makes dense monocular SLAM both more robust and uncertainty-aware — and its standard-factor-graph design makes adding other sensor modalities straightforward, a design philosophy echoed by later hybrid systems.

Related

• CodeSLAM
• SceneCode
• CodeMapping
• CNN-SLAM — the real-time learned-prior baseline it compares against
• DeepTAM — learned dense tracking/mapping baseline
• Factor graph
• Marginalization