SemanticFusion

McCormac 2016 · Paper

One-line summary — The first real-time dense semantic SLAM system, fusing per-frame CNN segmentation predictions into an ElasticFusion surfel map via recursive Bayesian updates to produce a coherent 3D semantic map.

Problem

Dense geometric SLAM produces accurate 3D reconstructions, but the map does not know what anything is — for the next level of robot intelligence and intuitive user interaction, maps need to extend beyond geometry and appearance to contain semantics. Single-frame CNN semantic segmentation, meanwhile, is noisy and view-dependent: the same surface can be labeled differently from different viewpoints. The missing piece was a mechanism to accumulate many per-frame 2D predictions into one persistent, consistent 3D labeling.

Method & architecture

The pipeline has three units running together, plus an optional regularizer:

SLAM backbone (ElasticFusion): for each frame $k$ , ElasticFusion tracks the camera by combined ICP + RGB alignment, yielding pose $T_{WC}$ , and fuses depth into a surfel map. Its deformation graph lets probability distributions be “carried along” with surfels through small and large loop closures, so surfels stay persistently associated with real-world entities — exactly the long-term correspondence semantic fusion needs. (Default parameters, except depth cutoff extended from 3 m to 8 m.)
CNN front-end: a Deconvolutional Semantic Segmentation network (Noh et al., VGG-16 based, in Caffe) is run on every 10th frame, returning per-pixel class probabilities $P(O_{u} = l_i \mid I_k)$ over 13 NYUv2 classes. A fourth depth channel is added by initializing depth filters from the average intensity of the pretrained RGB filters, rescaled ( $\approx 32\times$ ) from the 0–255 color range to the 0–8 m depth range. Inputs are rescaled to 224×224; outputs upsampled to 640×480.
Bayesian label fusion: each surfel $s$ stores a discrete distribution $P(L_s = l_i)$ over labels, initialized uniform. Using the tracked pose, each visible surfel at 3D position $x(s)$ is associated with pixel $u(s,k) = \pi\big(T_{CW}(k)\, x(s)\big)$ and updated recursively:

$P(l_i \mid I_{1,\dots,k}) = \frac{1}{Z}\, P(l_i \mid I_{1,\dots,k-1})\; P\big(O_{u(s,k)} = l_i \mid I_k\big)$

with $Z$ a normalizing constant. The SLAM correspondences are what allow label hypotheses from many viewpoints to be combined in a Bayesian way.

Optional CRF regularization: a fully-connected CRF with Gaussian edge potentials treats each surfel as a node and incrementally updates the distributions by approximately minimizing $E(\mathbf{x}) = \sum_s \psi_u(x_s) + \sum_{s<s'} \psi_p(x_s, x_{s'})$ , with unary $\psi_u(x_s) = -\log P(L_s = x_s \mid I_{1,\dots,k})$ and Potts-weighted pairwise kernels — a bilateral appearance kernel over surfel positions $\mathbf{p}$ and colors $\mathbf{c}$ and a smoothness kernel over normals $\mathbf{n}$ :

$k^1 = \exp\Big(-\frac{|\mathbf{p}_s-\mathbf{p}_{s'}|^2}{2\theta_\alpha^2} - \frac{|\mathbf{c}_s-\mathbf{c}_{s'}|^2}{2\theta_\beta^2}\Big), \qquad k^2 = \exp\Big(-\frac{|\mathbf{p}_s-\mathbf{p}_{s'}|^2}{2\theta_\alpha^2} - \frac{|\mathbf{n}_s-\mathbf{n}_{s'}|^2}{2\theta_\gamma^2}\Big)$

with $\theta_\alpha = 0.05$ m, $\theta_\beta = 20$ , $\theta_\gamma = 0.1$ rad, applied every 500 frames.

Per-frame cost (i7-5820K + Titan Black): SLAM 29.3 ms, probability-table bookkeeping 1.0 ms; CNN forward pass 51.2 ms and Bayesian update 41.1 ms every 10 frames — an average frame rate of 25.3 Hz.

Results

On a purpose-built office reconstruction dataset (loopy trajectory, 49 annotated test frames, 13 NYUv2 classes), fusing predictions raised the RGBD-CNN from 43.6% to 48.3% class-average accuracy, and the state-of-the-art Eigen et al. CNN from 57.1% to 60.0%. On the NYUv2 test set (140 usable sequences, 360 labeled images), SemanticFusion improved the RGBD-CNN from 55.6% to 58.9% class average (62.0% to 67.5% pixel average) and Eigen et al. from 59.9% to 63.2% (+3.3%); adding the CRF gave small further gains (63.6%). The improvement is roughly double on the office dataset versus NYUv2’s mostly rotational trajectories — multi-view fusion is worth most exactly when viewpoints vary. Predicting on every frame gives 52.5% accuracy at 8.2 Hz; every 10th frame gives 49–51% at 25.3 Hz.

Why it matters for SLAM

SemanticFusion pioneered “deep semantic SLAM”: it demonstrated that CNN perception and dense SLAM are complementary, with SLAM providing the correspondences that turn per-frame 2D predictions into a persistent, consistent 3D semantic map — and that the fusion improves even the 2D labeling itself. It directly inspired the semantic mapping line that leads to Fusion++, PanopticFusion, Kimera, and today’s 3D scene graph systems.

SemanticFusion

Problem

Method & architecture

Results

Why it matters for SLAM

Related