TANDEM

Koestler 2021 · Paper

One-line summary — Real-time monocular dense SLAM that pairs DSO-style photometric bundle adjustment with a learned multi-view stereo network (CVA-MVSNet) and TSDF fusion, and tracks new frames against depth rendered from the global model.

Problem

Real-time dense reconstruction from a monocular camera is much harder than RGB-D mapping, where depth can simply be read out and fused. Classical direct methods (DSO) track accurately but reconstruct only sparse/semi-dense points; accurate multi-view stereo traditionally ran offline, and fully learned dense SLAM lagged classical methods in tracking accuracy. TANDEM asks how to get real-time tracking and dense, globally consistent mapping from a single moving camera — the first system to integrate learning-based MVS into a traditional optimization-based VO.

Method & architecture

Three components run in a tight loop (VO on CPU threads; MVS inference and TSDF fusion asynchronously on GPU):

1. Visual odometry with dense front-end. The back-end is DSO’s direct sparse windowed photometric bundle adjustment. The front-end tracks each new frame by dense direct image alignment against the last keyframe, using a combined depth buffer: sparse depths DnDSOD_{n}^{\text{DSO}} from the active window’s points where available, otherwise dense depth DnTSDFD_{n}^{\text{TSDF}} rendered from the incrementally built global TSDF model — the first monocular dense front-end tracking against a global model, and the key robustness gain over sparse-only tracking.

2. CVA-MVSNet. Given the active keyframes {(Ii,Ti)}i=1n\{(I_{i},\mathbf{T}_{i})\}_{i=1}^{n}, shared-weight 2D U-Nets extract multi-scale features Fis\mathbf{F}_{i}^{s}; depth for the reference keyframe is predicted in 3 cascaded stages of increasing resolution. Each stage warps features onto per-pixel depth hypotheses Dhyps\mathbf{D}_{hyp}^{s} to form feature volumes Vis\mathbf{V}_{i}^{s}. Because sliding-window keyframes have very uneven baselines (heavy occlusion / non-overlap), the standard variance cost metric that weighs views equally is replaced by self-adaptive view aggregation:

Cs=∑i=1,i≠jn(1+Wis) ⊙ (Vis−Vjs)2n−1\mathbf{C}^{s}=\frac{\sum_{i=1,i\neq j}^{n}(1+\mathbf{W}_{i}^{s})\,\odot\,(\mathbf{V}_{i}^{s}-\mathbf{V}_{j}^{s})^{2}}{n-1}

where Wis\mathbf{W}_{i}^{s} are per-voxel weights predicted by a shallow 3D CNN from (Vis−Vjs)2(\mathbf{V}_{i}^{s}-\mathbf{V}_{j}^{s})^{2}, letting the network down-weight erroneous views. The cost volume is regularized by a 3D U-Net and softmaxed into a probability volume Ps\mathbf{P}^{s}; depth is its expectation

Ds[h,w]=∑d=1Ds  Ps[d,h,w]⋅Dhyps[d,h,w]D^{s}[h,w]=\sum_{d=1}^{D^{s}}\;\mathbf{P}^{s}[d,h,w]\cdot\mathbf{D}_{hyp}^{s}[d,h,w]

Later stages sample few hypotheses around the upsampled previous-stage depth (D1=48D^{1}=48 planes over the full range, then fewer), trained with an L1L_1 loss summed over all three stages.

3. TSDF fusion. Predicted depth maps are fused into a voxel-hashed truncated signed distance function grid, giving the consistent global model that both the visualized mesh and the dense tracking front-end draw from.

Results

Why it matters for SLAM

TANDEM is a clean demonstration of the classical-plus-learned hybrid design pattern: keep the well-understood geometric estimator for poses, insert learning exactly where classical methods are weakest (dense depth in textureless areas), and let the dense map feed back into tracking. It showed monocular cameras can produce dense TSDF maps online without a depth sensor, standing as the pragmatic alternative to fully learned pipelines like DROID-SLAM and to neural-implicit systems like iMAP/NICE-SLAM.