BundleFusion

Dai 2016 · Paper

One-line summary — Globally consistent real-time RGB-D reconstruction via hierarchical local-to-global pose optimization, with on-the-fly TSDF de-integration and re-integration whenever poses are corrected.

Problem

Scalable real-time 3D scanning suffers from pose drift that accumulates significant errors in the reconstructed model; global correction traditionally required hours of offline processing. The online methods that existed either needed minutes to perform correction (not truly real-time), used brittle frame-to-frame or frame-to-model tracking that failed catastrophically without recovery, or supported only unstructured point representations that limit scan quality. BundleFusion set out to solve all three at once: globally optimized (i.e. bundle-adjusted) poses, robust tracking with relocalization, and a high-quality volumetric model — all in real time.

Method & architecture

Input is a 30 Hz, 640×480 RGB-D stream $S = \{f_i = (C_i, D_i)\}$; the goal is rigid transforms $T_i(p) = R_i p + t_i$ mapping each frame into world space. Instead of temporal tracking, every new frame is localized against the complete history of input — loop closure is implicit and continuous, and relocalization comes for free.

Sparse correspondence search. SIFT keypoints are detected (4–5 ms/frame) and matched on GPU against all previous frames (≈0.05 ms per pair, feasible against 20K+ frames). Matches pass three filters: a Kabsch-based keypoint filter (rigid fit with condition analysis; correspondences pruned until max residual < 0.02 m), a surface-area filter (spanned area must exceed 0.032 m²), and a dense two-sided geometric/photometric verification on 80×60 downsampled frames (pair invalidated if reprojection error > 0.075 m). A frame pair needs $N_{\min} = 5$ valid matches.

Hierarchical local-to-global optimization. Consecutive frames are grouped into chunks of $N_{\mathrm{chunk}} = 11$ (overlapping by 1). Chunks are bundle-adjusted internally; each chunk contributes one keyframe with an aggregated feature set to a second-level global optimization over all keyframes. Both levels minimize the same energy over stacked pose parameters $X$:

$$E_{\mathrm{align}}(X) = w_{\mathrm{sparse}} E_{\mathrm{sparse}}(X) + w_{\mathrm{dense}} E_{\mathrm{dense}}(X)$$

$$E_{\mathrm{sparse}}(X) = \sum_{i}\sum_{j}\sum_{(k,l) \in C(i,j)} \left\| T_i\, p_{i,k} - T_j\, p_{j,l} \right\|_2^2$$

where $p_{i,k}$ is the $k$-th feature point of frame $i$ and $C(i,j)$ its correspondences with frame $j$. The dense term $E_{\mathrm{dense}} = w_{\mathrm{photo}} E_{\mathrm{photo}} + w_{\mathrm{geo}} E_{\mathrm{geo}}$ combines photometric alignment on luminance gradients $I_i$ and a point-to-plane geometric term, evaluated on frame pairs $E$ with similar viewing angle (within 60°) and overlap:

$$E_{\mathrm{photo}}(X) = \sum_{(i,j) \in E}\sum_{k} \left\| I_i\big(\pi(d_{i,k})\big) - I_j\big(\pi(T_j^{-1} T_i\, d_{i,k})\big) \right\|_2^2$$

$$E_{\mathrm{geo}}(X) = \sum_{(i,j) \in E}\sum_{k} \Big[ n_{i,k}^{\top} \Big( d_{i,k} - T_i^{-1} T_j\, \pi^{-1}\big( D_j\big(\pi(T_j^{-1} T_i\, d_{i,k})\big) \big) \Big) \Big]^2$$

with $\pi$ the perspective projection, $d_{i,k}$ the 3D point of pixel $k$ and $n_{i,k}$ its normal. $w_{\mathrm{dense}}$ is increased linearly so sparse terms fix global structure first, then dense terms refine within their basin of convergence. The non-linear least squares is solved by Gauss-Newton with a data-parallel GPU PCG solver (Jacobi preconditioner, $J_F$ and $J_F^{\top}$ applied in separate kernels). After each solve, if the maximum residual exceeds 0.05 m all correspondences of the offending frame pair are removed — a final outlier safeguard.

On-the-fly re-integration. Geometry is fused into a voxel-hashed TSDF (8³ voxel blocks). Integration of depth sample $d_i(v)$ with weight $w_i(v)$, and its exact inverse (de-integration), are:

$$D'(v) = \frac{D(v)\,W(v) + w_i(v)\,d_i(v)}{W(v) + w_i(v)}, \qquad W'(v) = W(v) + w_i(v)$$

with subtraction in place of addition to de-integrate. Each frame stores its integrated and optimized pose; per new input frame, the $N_{\mathrm{fix}} = 10$ frames with the largest pose discrepancy are de-integrated at the old pose and re-integrated at the new one, so the dense model continuously converges to the optimized trajectory — no map deformation needed.

Results

Live scanning used a Structure Sensor on an iPad streaming to a desktop; the pipeline runs at well beyond 30 Hz split across two GPUs (GTX Titan X + Titan Black), with 4 mm default voxels. Captured scenes — 4 offices, 2 apartments, 1 copyroom, trajectories up to 95 m and over 20,000 frames — show drift-free alignment and completeness on par with offline approaches. On the synthetic ICL-NUIM benchmark, ATE RMSE is kt0 0.6 cm, kt1 0.4 cm, kt2 0.6 cm, kt3 1.1 cm — outperforming online (e.g. ElasticFusion: 0.9/0.9/1.4/10.6 cm) and offline (Redwood rigid: 25.6/3.0/3.3/6.1 cm) state of the art. On TUM RGB-D: fr1/desk 1.6 cm, fr2/xyz 1.1 cm, fr3/office 2.2 cm, fr3/nst 1.2 cm, on par with or better than prior systems. Ablations (sparse-only vs sparse+local-dense vs full) confirm each stage of the sparse-to-dense design adds accuracy. The system recovers from tracking failure (sensor occlusion, featureless regions) by continuing to localize against global keyframes, and was later used to capture the ScanNet dataset.

Why it matters for SLAM

BundleFusion solved the problem that had haunted volumetric SLAM since KinectFusion: how to correct the dense map after loop closure without deformation artifacts. The de-integration/re-integration mechanism is the TSDF-world answer to ElasticFusion’s map deformation, defining one side of the fundamental design fork in dense SLAM. Whenever a modern dense system needs to reconcile “poses change after optimization” with “the map is already fused”, it is reusing this playbook.

Related

• KinectFusion — the fixed-volume TSDF fusion ancestor
• ElasticFusion — the competing surfel-deformation approach to global consistency
• InfiniTAM v3 — voxel-hashing framework for scalable TSDF storage
• BAD SLAM — later direct BA over dense geometry
• TSDF vs Surfel maps — the design fork BundleFusion sits on