InfiniTAM v3

Prisacariu 2017 · Paper

One-line summary — A modular, cross-device open-source RGB-D reconstruction framework combining voxel-hashed TSDF (or surfel) mapping, robust ICP/RGB tracking, random-fern relocalization, and submap-based globally consistent reconstruction.

Problem

Representing a reconstruction volumetrically as a TSDF gives the simplicity and efficiency that GPU implementations of KinectFusion-style systems enjoy — but a dense uniform grid is memory-intensive and limits scale, and the research community lacked a single fast, flexible pipeline in which camera tracking, scene representation, and data integration could be swapped and adapted. InfiniTAM is that framework; the v3 technical report documents its third iteration, whose headline additions are a robust tracker with failure detection, a random-ferns relocalizer, globally consistent TSDF reconstruction via submaps, and a surfel backend.

Method & architecture

Engine architecture. A chain-of-responsibility design: stateless processing engines (tracking, allocation, integration, raycast, swapping) pass state objects, each engine split into an Abstract, a Device-Specific (CPU/CUDA/Metal), and a Device-Agnostic layer of shared inline C code. Two pipelines exist: ITMBasicEngine (standard fusion) and ITMMultiEngine (globally consistent, with loop closure).

Voxel hashing. Voxels (TSDF value, weight, optional RGB) are grouped into 8×8×88\times 8\times 8 blocks stored in a contiguous voxel block array (2182^{18} entries); a hash table maps a block’s corner coordinates b\mathbf{b} to its storage index via

h(b)=((bx⋅73856093)⊕(by⋅19349669)⊕(bz⋅83492791)) mod nh(\mathbf{b}) = \big( (b_x \cdot 73856093) \oplus (b_y \cdot 19349669) \oplus (b_z \cdot 83492791) \big) \bmod n

with collisions handled by an unordered excess list. Allocation backprojects, for each depth pixel dd, the segment from d−μd-\mu to d+μd+\mu and allocates the intersected blocks in three non-blocking stages; integration then updates each visible voxel with the running weighted TSDF average as in KinectFusion.

Tracking. The classic ITMDepthTracker minimizes point-to-plane distances against a raycast of the model, d=(Rp+t−V(pˉ))⊤N(pˉ)d = (\mathbf{R}\mathbf{p} + \mathbf{t} - \mathcal{V}(\bar{\mathbf{p}}))^{\top} \mathcal{N}(\bar{\mathbf{p}}), over a resolution hierarchy; ITMColorTracker instead minimizes color differences d=∥I(π(R V(i)+t))−C(i)∥2d = \| I(\pi(\mathbf{R}\,\mathcal{V}(i) + \mathbf{t})) - \mathcal{C}(i) \|_2. New in v3, the default ITMExtendedTracker makes ICP robust: a Huber norm on per-pixel errors, depth-dependent down-weighting of far (noisier) measurements, outlier gating by distance threshold, and an optional frame-to-frame photometric term on intensities I=0.299R+0.587G+0.114BI = 0.299R + 0.587G + 0.114B (Tukey loss, scaled by 0.3), all minimized with Levenberg-Marquardt coarse-to-fine. An SVM classifier on the ICP statistics (inlier percentage, Hessian determinant, residual) detects tracking failure and triggers relocalization.

Relocalization (random ferns). Each RGB-D image is encoded as mm binary code blocks of nn feature tests; similarity between images is the block-wise Hamming distance BlockHD(bCI,bCJ)=1m∑k=1m(bFkI≡bFkJ)\mathrm{BlockHD}(b_C^I, b_C^J) = \frac{1}{m}\sum_{k=1}^{m} (b_{F_k}^I \equiv b_{F_k}^J). Code tables map codes to keyframe IDs, so the nearest stored keyframe (and its pose) is retrieved in constant time — used both for pose recovery and loop-closure detection.

Globally consistent reconstruction. The scene is divided into rigid submaps (active ones tracked each frame, passive ones dormant; new submaps spawned as the camera leaves the current one). Inter-submap constraints accumulate from tracking and fern-detected loop closures, and a submap pose graph is optimized on a background thread. Rendering raycasts an implicit combined TSDF fused on the fly,

F^(X)=∑iFw(PiX) F(PiX)\hat{F}(\mathbf{X}) = \sum_i F_w(\mathbf{P}_i \mathbf{X})\, F(\mathbf{P}_i \mathbf{X})

where Pi\mathbf{P}_i is submap ii‘s pose and F,FwF, F_w its TSDF and weight — the global map is a view over submaps, never explicitly built.

Surfel backend & swapping. A beta implementation of Keller et al.’s point-based fusion updates a matched surfel by confidence-weighted averaging, vˉk←(cˉkvˉk+αvg)/(cˉk+α)\bar{\mathbf{v}}_k \leftarrow (\bar{c}_k \bar{\mathbf{v}}_k + \alpha \mathbf{v}^g) / (\bar{c}_k + \alpha), up to 5M surfels. A swapping engine with fixed-size host/device transfer buffers pages voxel blocks out of GPU memory and re-fuses them on return, so maps can exceed GPU capacity.

Results

The v3 paper is a technical report on the framework’s implementation, not a benchmark study — it contains no quantitative evaluation tables; the underlying globally-consistent submap method is evaluated in its companion publication (Kähler et al., ECCV 2016). The report’s own claims are engineering ones: the pipeline runs in real time on GPU (the less-optimized surfel backend is “still real-time” on a good GPU); an off-the-shelf graphics card holds roughly a single room at 4 mm voxel resolution in active memory even with hashing, which the swapping subsystem extends to larger scenes; and the whole stack runs across CPU, CUDA, and Metal device layers with numerous sensors (Kinect, PrimeSense, RealSense, Structure). See the companion papers for full tracking-accuracy evaluation. Its lasting impact is as one of the most widely used, hackable open-source RGB-D reconstruction codebases.

Why it matters for SLAM

InfiniTAM v3 packaged the post-KinectFusion state of the art — voxel hashing, robust frame-to-model tracking with failure detection, fern relocalization, submap-based loop closure — into one modular framework. If you want to understand how a production-quality dense SLAM pipeline is engineered (memory management, GPU kernels, swappable map backends, host-device paging), reading InfiniTAM’s code is one of the best exercises at this level; it is also the reference implementation of the voxel-hashing storage scheme that scalable TSDF systems like BundleFusion build on.