Co-SLAM

Wang 2023 · Paper

One-line summary — Combined Instant-NGP-style multi-resolution hash grids with smooth one-blob coordinate encoding for neural SLAM, running at 10-17 Hz — far faster than NICE-SLAM — while keeping coherent surfaces.

Problem

Coordinate-encoding MLPs carry coherence and smoothness priors that give high-fidelity, hole-filling reconstruction, but “suffer from slow convergence and catastrophic forgetting” when optimised sequentially; parametric encodings (feature grids) are fast but “fall short of hole filling and smoothness”. NICE-SLAM was scalable but far from real-time, and its local grids cannot complete unobserved regions. Co-SLAM (“Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM”) asks for both properties at once in a single real-time RGB-D SLAM system.

Method & architecture

Given an RGB-D stream with known intrinsics, Co-SLAM jointly optimises camera poses {ξt}\{\xi_t\} and a neural field fθ(x)↦(c,s)f_\theta(\mathbf{x})\mapsto(\mathbf{c},s) mapping world coordinates to colour and truncated signed distance (TSDF). The joint encoding concatenates a smooth One-blob coordinate encoding γ(x)\gamma(\mathbf{x}) with features from an Instant-NGP multi-resolution hash grid Vα\mathcal{V}_\alpha (levels spanning RminR_{min} to RmaxR_{max}, tri-linearly interpolated). Two tiny MLPs decode:

fτ(γ(x),Vα(x))↦(h,s),fϕ(γ(x),h)↦c,f_\tau(\gamma(\mathbf{x}),\mathcal{V}_\alpha(\mathbf{x}))\mapsto(\mathbf{h},s),\qquad f_\phi(\gamma(\mathbf{x}),\mathbf{h})\mapsto\mathbf{c},

with learnable parameters θ={α,ϕ,τ}\theta=\{\alpha,\phi,\tau\} — “fast convergence, efficient memory use, and hole filling needed for online SLAM”. Colour and depth are rendered along rays xi=o+dir\mathbf{x}_i=\mathbf{o}+d_i\mathbf{r} as normalised weighted sums c^=1∑iwi∑iwici\hat{\mathbf{c}}=\tfrac{1}{\sum_i w_i}\sum_i w_i\mathbf{c}_i, d^=1∑iwi∑iwidi\hat{d}=\tfrac{1}{\sum_i w_i}\sum_i w_i d_i, with a simple bell-shaped SDF-to-weight conversion

wi=σ ⁣(sitr)σ ⁣(−sitr),w_i=\sigma\!\left(\frac{s_i}{tr}\right)\sigma\!\left(-\frac{s_i}{tr}\right),

where trtr is the truncation distance (10 cm) and σ\sigma the sigmoid. Sampling is depth-guided: McM_c uniform samples plus MfM_f near-surface samples around the measured depth.

Losses: ℓ2\ell_2 colour/depth rendering losses; an approximate SDF loss inside the truncation region, Lsdf\mathcal{L}_{sdf} pulling predictions toward D[u,v]−dD[u,v]-d; a free-space loss forcing sp=trs_p=tr far from surfaces; and a smoothness regulariser on feature-metric differences of adjacent hash-grid vertices, Lsmooth=1∣G∣∑x∈GΔx2+Δy2+Δz2\mathcal{L}_{smooth}=\tfrac{1}{|\mathcal{G}|}\sum_{\mathbf{x}\in\mathcal{G}}\Delta_x^2+\Delta_y^2+\Delta_z^2, computed on small random regions to suppress hash-collision noise in unobserved space.

Tracking initialises each frame with a constant-speed motion model Tt=Tt−1Tt−2−1Tt−1\mathbf{T}_t=\mathbf{T}_{t-1}\mathbf{T}_{t-2}^{-1}\mathbf{T}_{t-1}, then optimises ξt\xi_t over NtN_t sampled pixels. Global bundle adjustment is the second key idea: instead of storing full keyframe images and selecting ~10 of them (iMAP/NICE-SLAM), Co-SLAM stores only ~5% of pixels per keyframe, inserts keyframes frequently (every 5th frame), and samples NgN_g rays from the entire keyframe database to jointly optimise the map and all poses, alternating kmk_m map steps with pose updates from accumulated gradients.

Results

Why it matters for SLAM

Co-SLAM brought neural implicit SLAM to real-time rates, making the NeRF-SLAM family practical for interactive use, and showed that Instant-NGP’s hash grids are viable for online mapping when paired with a smooth coordinate encoding that restores hole filling. Its joint parametric + coordinate encoding and its sparse-pixel global bundle adjustment became influential design patterns in the NICE-SLAM lineage, alongside ESLAM’s tri-planes and Point-SLAM’s neural point clouds.