DPVO

Teed 2023 · Paper

One-line summary — A patch-based, lightweight variant of DROID-SLAM showing that sparse patch tracking plus differentiable bundle adjustment matches or beats dense flow at a fraction of the memory and compute.

Problem

Deep VO built on dense optical flow (DROID-SLAM) raised accuracy dramatically, but “using dense flow incurs a large computational cost, making these previous methods impractical for many use cases”: DROID’s VO frontend needs 8.7 GB of GPU memory and its frame rate collapses during fast motion. The field assumed the cost was necessary — that dense flow “provides additional redundancy against incorrect matches.” Deep Patch Visual Odometry (DPVO) set out to test, and ultimately disprove, that assumption.

Method & architecture

Patch representation. The scene is a set of poses $\mathbf{T} \in \mathbb{SE}(3)^N$ and square $p \times p$ image patches, each stored as a $4 \times p^2$ homogeneous array with pixel coordinates $(\mathbf{x}, \mathbf{y})$ and a single inverse depth $\mathbf{d}$ shared by the patch (a fronto-parallel plane, as in DSO). A patch $k$ from frame $i$ reprojects into frame $j$ as

$$\mathbf{P}'_{kj} \sim \mathbf{K} \mathbf{T}_j \mathbf{T}_i^{-1} \mathbf{K}^{-1} \mathbf{P}_k$$

with $\mathbf{K}$ the calibration matrix. A bipartite patch graph links each patch to every frame within distance $r$ of its source frame; patch trajectories are the sets of these reprojections, and the graph grows/shrinks as frames arrive.

Features and patches. Twin residual networks give matching and context features at 1/4 resolution (2-level pyramid). Patches are cropped at random pixel locations — 96 per frame (Default) or 48 (Fast) — which ablations show works better than SIFT, ORB, SuperPoint, or gradient-based selection.

Recurrent update operator. Operating on the patch graph with a hidden state per edge, each iteration runs: (1) correlation — for each pixel of the reprojected patch, inner products against a $7\times 7$ grid of frame features, $\mathbf{C}_{uv\alpha\beta} = \langle \mathbf{g}_{uv},\ \mathbf{f}(\mathbf{P}'_{kj}(u,v) + \Delta_{\alpha\beta}) \rangle$; (2) 1D temporal convolution along each patch trajectory; (3) softmax aggregation — message passing among edges sharing a patch or a frame, replacing the spatial convolutions dense methods get for free; (4) a transition block (gated residual units + LayerNorm); (5) a factor head predicting a 2D trajectory revision $\delta_{kj}$ and confidence $\Sigma_{kj} \in (0,1)^2$ per edge.

Differentiable bundle adjustment. Poses and patch inverse depths are updated by two Gauss-Newton iterations (Schur complement, gradients backpropagated through) on

$$\sum_{(k,j)\in\mathcal{E}} \left\lVert \hat{\omega}_{ij}(\mathbf{T}, \mathbf{P}_k) - [\hat{\mathbf{P}}'_{kj} + \delta_{kj}] \right\rVert^2_{\Sigma_{kj}}$$

the Mahalanobis distance between induced and predicted patch-center reprojections — DROID-SLAM’s DBA on a radically smaller problem.

Training and system. Trained end-to-end on TartanAir (240k iterations, one RTX-3090, 3.5 days) with loss $\mathcal{L} = 10\mathcal{L}_{pose} + 0.1\mathcal{L}_{flow}$, where $\mathcal{L}_{pose}$ compares relative poses after Umeyama scale alignment. At inference: 8-frame initialization, constant-velocity pose prediction, one update + two BA iterations per frame over a 10-keyframe window (7 for Fast), flow-based keyframe removal. No loop closure or global BA — that gap is filled by DPV-SLAM.

Results

Median of 5 runs; two configs: Default (60 FPS, 4.9 GB) and Fast (120 FPS, 2.5 GB) on an RTX-3090, vs DROID-VO at 40 FPS / 8.7 GB.

• TartanAir (ECCV 2020 SLAM competition test split): avg ATE 0.21 — 40% lower than full DROID-SLAM (0.33) and 64% lower than DROID-VO (0.58); classical DSO averages 7.32 and ORB-SLAM3 14.38. On the validation split, AUC 0.80 vs DROID-SLAM’s 0.71 while running 4x faster.
• EuRoC: avg ATE 0.105 vs DROID-VO 0.186 (43% lower); even the 120 FPS Fast config (0.129) beats DROID-VO on most sequences.
• TUM-RGBD (freiburg1, mono): avg ATE 0.089 vs DROID-VO 0.098 (9% lower), with zero catastrophic failures where ORB-SLAM3 and DSO fail on multiple sequences.
• Stability: frame rate is nearly constant (above 48 FPS for 95% of frames; Fast stays above 98 FPS) while DROID-VO drops to 11 FPS in the worst case — an 8.9x gap.

Why it matters for SLAM

DPVO made differentiable-BA-based visual odometry practical for real-time robotics rather than an offline GPU-heavy affair, and its recurrent-update-on-sparse-patches design was adopted by followers such as MAC-VO and DPV-SLAM (and adapted to event cameras in DEVO). It is the standard modern baseline for learned monocular VO. It appears in both Level 3 (as a monocular system) and Level 5 (as a deep-learning method).

Related

• DROID-SLAM
• DPV-SLAM
• MAC-VO
• DSO — source of the patch representation
• RAFT
• DEVO