SplaTAM

Keetha 2024 · Paper

One-line summary — Among the first SLAM systems to use 3D Gaussian Splatting as the map (concurrent with GS-SLAM and MonoGS): an RGB-D splat-track-map loop through a differentiable rasteriser, with a rendered silhouette guiding both pose optimisation and Gaussian densification.

Problem

Dense SLAM methods were “often hampered by the non-volumetric or implicit way they represent a scene”: handcrafted explicit maps (points, surfels, SDFs) track reliably only with rich 3D features and explain only observed surfaces, while implicit radiance-field SLAM (NICE-SLAM, Point-SLAM) needs expensive per-ray volumetric sampling, forcing losses over a sparse set of pixels. 3D Gaussian Splatting rasterises at up to 400 FPS but had always required known poses. SplaTAM (CVPR 2024, CMU/MIT) shows “for the first time that representing a scene by 3D Gaussians can enable dense SLAM using a single unposed monocular RGB-D camera”.

Method & architecture

Simplified Gaussian map. The scene is a set of isotropic, view-independent Gaussians — 8 parameters each (RGB colour $\mathbf{c}$, centre $\boldsymbol{\mu}\in\mathbb{R}^3$, radius $r$, opacity $o$) — each influencing space as $f(\mathbf{x}) = o\exp\bigl(-\tfrac{\|\mathbf{x}-\boldsymbol{\mu}\|^{2}}{2r^{2}}\bigr)$. Colour, depth, and a silhouette are all rendered by sorting Gaussians front-to-back and alpha-compositing their 2D splats:

$$C(\mathbf{p})=\sum_{i=1}^{n}\mathbf{c}_{i}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}\bigl(1-f_{j}(\mathbf{p})\bigr), \quad D(\mathbf{p})=\sum_{i=1}^{n}d_{i}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}\bigl(1-f_{j}(\mathbf{p})\bigr), \quad S(\mathbf{p})=\sum_{i=1}^{n}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}\bigl(1-f_{j}(\mathbf{p})\bigr),$$

where $f_i(\mathbf{p})$ uses the projected centre $\boldsymbol{\mu}^{2D} = K\,E_{t}\boldsymbol{\mu}/d$ and radius $r^{2D} = fr/d$ with $d=(E_{t}\boldsymbol{\mu})_{z}$. The silhouette $S$ says how much map evidence each pixel has — the map’s epistemic uncertainty.

Each frame runs three steps:

1. Camera tracking. The new pose is initialised by constant-velocity propagation, $E_{t+1}=E_{t}+(E_{t}-E_{t\text{-}1})$, then refined by gradient descent through the rasteriser with Gaussians frozen, using only well-mapped pixels: $$L_{t}=\sum_{\mathbf{p}}\Bigl(S(\mathbf{p})>0.99\Bigr)\Bigl(\mathrm{L}_{1}\bigl(D(\mathbf{p})\bigr)+0.5\,\mathrm{L}_{1}\bigl(C(\mathbf{p})\bigr)\Bigr).$$
2. Gaussian densification. A mask picks pixels the map does not yet explain — low silhouette, or true geometry in front of the rendered geometry: $$M(\mathbf{p})=\Bigl(S(\mathbf{p})<0.5\Bigr)+\Bigl(D_{\mathrm{GT}}(\mathbf{p})<D(\mathbf{p})\Bigr)\Bigl(\mathrm{L}_{1}\bigl(D(\mathbf{p})\bigr)>50\,\mathrm{MDE}\Bigr),$$ with MDE the median depth error. Masked pixels each spawn a Gaussian with the pixel’s colour, centre at the unprojected depth, opacity 0.5, and one-pixel radius $r = D_{\mathrm{GT}}/f$.
3. Map update. With poses frozen, Gaussian parameters are optimised over $k$ keyframes (current frame, latest keyframe, and the $k-2$ keyframes with highest frustum overlap with the current depth point cloud), warm-started from the existing map, using the colour+depth loss without the silhouette mask plus an SSIM term; near-transparent or oversized Gaussians are culled.

Results

• Replica (ATE RMSE avg, 8 scenes): 0.36 cm — more than 30% below the prior SOTA Point-SLAM (0.52) and well below ESLAM (0.63), NICE-SLAM (1.06), Vox-Fusion (3.09).
• TUM-RGBD: 5.48 cm avg, cutting Point-SLAM’s 8.92 by almost 40% (NICE-SLAM 15.87); feature-based ORB-SLAM2 (1.98) still wins among sparse methods. On the similarly low-quality original ScanNet, 11.88 cm is comparable to Point-SLAM (12.19) and NICE-SLAM (10.70).
• ScanNet++ (high-quality captures but huge inter-frame motion, ~30 Replica frames per step): SplaTAM tracks both sequences at 1.2 cm average error while Point-SLAM and RGB-D ORB-SLAM3 fail completely; novel-view synthesis reaches 24.41 dB PSNR (27.98 on train views) with ~2 cm novel-view depth L1.
• Rendering: Replica train-view PSNR 34.11 dB — about 10 dB above NICE-SLAM (24.42) and Vox-Fusion (24.41), on par with Point-SLAM (35.17, which uses GT depth to place samples); the map renders at 400 FPS at 876x584.
• Runtime (RTX 3080 Ti, Replica R0): 25 ms/tracking and 24 ms/mapping iteration while rendering the full ~1.2M-pixel image per iteration — versus baselines optimising over only 200-1000 sampled pixels; SplaTAM-S (fewer iterations) runs 5x faster (0.19 s + 0.33 s per frame) at 0.39 cm ATE.
• Ablations (Room 0): remove the silhouette mask and tracking collapses (ATE 115.8 cm); a 0.99 threshold instead of 0.5 gives 5x lower error (0.27 vs 1.30); no velocity propagation is >10x worse; depth-only loss fails entirely (86.03 cm). Stated limitations: sensitivity to motion blur, large depth noise, and aggressive rotation.

Why it matters for SLAM

SplaTAM helped launch the 3DGS SLAM research line, showing an explicit, differentiable, volumetric map can beat NeRF-style SLAM on tracking, mapping, and view synthesis at interactive rates — and that dense per-pixel losses become affordable when rendering is rasterisation instead of ray marching. Its silhouette mask became a standard densification and uncertainty-gating tool, and its alternating track-then-map loop through a differentiable rasteriser is the template that Photo-SLAM, RTG-SLAM, GS-ICP SLAM, and many follow-ups build on.

Related

• MonoGS
• NICE-SLAM
• Point-SLAM
• Photo-SLAM
• RTG-SLAM
• GS-ICP SLAM