DPV-SLAM

Lipson 2024 · Paper

One-line summary — Extended DPVO into a full SLAM system (ECCV 2024) by adding efficient loop-closure and global-correction mechanisms, keeping real-time operation on a single GPU.

Problem

Deep-network SLAM backbones deliver excellent accuracy but “such approaches are often expensive to run or do not generalize well zero-shot. Their runtime can also fluctuate wildly while their frontend and backend fight for access to GPU resources.” Concretely: two CUDA workloads on one device run sequentially, so existing deep SLAM systems periodically drop from ~30 Hz to <1 Hz while a backend iteration runs — consistent real-time needs two GPUs — and flow-based backends must retain dense feature maps for every frame, so memory grows with video length. DPVO solved efficiency but is odometry-only, so drift grows without bound. DPV-SLAM completes the design: monocular deep SLAM with loop closure, high minimum framerate, and 5-7 GB memory on a single GPU.

Method & architecture

DPVO base. The frontend keeps DPVO’s patch graph: patches Pik\mathbf{P}_{ik} (pixel coordinates + inverse depth d\mathbf{d}) reproject into frame jj via Pikj′=Π[Gj−1⋅Gi⋅Π−1(Pik)]\mathbf{P}'_{ikj} = \Pi[G_j^{-1} \cdot G_i \cdot \Pi^{-1}(\mathbf{P}_{ik})]; a recurrent operator predicts residuals Δikj\Delta_{ikj} and confidences wikjw_{ikj}, and bundle adjustment aligns reprojections to the “ideal” targets Iikj=Pikj′+Δikj\mathcal{I}_{ikj} = \mathbf{P}'_{ikj} + \Delta_{ikj}:

arg min⁡G,d∑i∑k∑j∥Π[Gj−1⋅Gi⋅Π−1(Pik)]−Iikj∥Σikj2,Σikj=diag⁡(wikj)\operatorname*{arg\,min}_{G, \mathbf{d}} \sum_i \sum_{k} \sum_{j} \left\lVert \Pi[G_j^{-1} \cdot G_i \cdot \Pi^{-1}(\mathbf{P}_{ik})] - \mathcal{I}_{ikj} \right\rVert^2_{\Sigma_{ikj}}, \qquad \Sigma_{ikj} = \operatorname{diag}(w_{ikj})

Proximity loop closure (mid-term). The key observation: for each directed edge, the correlation operator only needs dense features of the destination frame, while the BA factor constrains both poses regardless of edge direction — so edges can be flipped arbitrarily to control which frames pay the memory cost. DPV-SLAM therefore permanently stores only patch features for past frames (~0.6 GB per 1000 frames) and inserts uni-directional edges from old patches into recent frames whenever the camera passes near a previously visited pose. Odometry and loop-closure factors are mixed in one shared optimization, run by a new CUDA block-sparse bundle adjustment built for sparse, variable-sized patch graphs. Everything runs in a single process on a single GPU; one proximity global BA takes 0.1-0.18 s on EuRoC vs 0.5-5 s for DROID-SLAM’s backend.

Classical loop closure (long-term, “DPV-SLAM++”). A complementary CPU backend corrects scale drift: dBoW2 image retrieval over ORB features detects loop candidates; off-the-shelf detectors/matchers plus structure-only BA triangulate 3D keypoints around each retrieved pair, whose alignment via RANSAC + Umeyama gives the drift ΔSjkloop∈Sim(3)\Delta S^{loop}_{jk} \in Sim(3). Keyframe similarities SiS_i are then optimized by Levenberg-Marquardt over a pose graph with smoothness and loop residuals

ri=log⁡Sim(3)(ΔS(i,i+1)−1⋅Si−1⋅Si+1),rjk=log⁡Sim(3)(ΔSjkloop⋅Sj−1⋅Sk)r_i = \log_{Sim(3)}\big(\Delta S_{(i,i+1)}^{-1} \cdot S_i^{-1} \cdot S_{i+1}\big), \qquad r_{jk} = \log_{Sim(3)}\big(\Delta S^{loop}_{jk} \cdot S_j^{-1} \cdot S_k\big)

after which poses and depths are rescaled (di←di/sid_i \leftarrow d_i / s_i). Retrieval and PGO run in parallel processes, adding virtually zero runtime.

Results

Median of 5 runs, timings on an RTX-3090; same TartanAir-trained weights everywhere (zero-shot):

Why it matters for SLAM

DPV-SLAM completes the DROID-SLAM → DPVO arc: differentiable-BA visual odometry, made sparse and fast, finally equipped with loop closure to be a genuine SLAM system — and it is a clean case study of how learned frontends combine with classical global machinery (dBoW2 retrieval, Sim(3)Sim(3) pose-graph optimization) in one budget. Its edge-flipping memory trick and single-GPU frontend/backend co-existence address the systems problems that keep deep SLAM out of robots.