MoreFusion

Wada 2020 · Paper

One-line summary — Fuses RGB-D video into per-object occupancy volumes and jointly refines the 6D poses of multiple known objects with differentiable collision checking, enabling accurate pose estimation for objects in contact and under occlusion.

Problem

6D object pose estimation from a single RGB-D image is fundamentally limited by occlusion and ambiguity: an object partially hidden behind another cannot be fully localized from one view, and symmetric objects admit multiple valid poses. Single-view methods therefore fail precisely in the most practically important scenario — cluttered scenes where objects are in contact and occlude each other, the norm in robot manipulation. Multi-view information is the natural remedy, but naively accumulating per-frame pose proposals produces physically impossible states where objects interpenetrate; a joint, physics-aware optimization is needed.

Method & architecture

The system runs four stages over a live RGB-D stream (camera pose from robot forward kinematics, or ORB-SLAM2 when hand-held):

  1. Object-level volumetric fusion — Mask R-CNN produces instance masks each frame; masked depth is fused into a per-object octree occupancy map (OctoMap), one volume per detected object and per unrecognized background structure; instances are tracked across frames by IoU between detected and rendered masks. This accumulates each object’s geometry from multiple views, plus free/unknown-space information.
  2. Volumetric pose prediction — each target object gets a 32×32×3232\times32\times32 occupancy grid whose voxels are labeled self / other-object / free / unknown (gself,gother,gfree,gunknowng^{\mathtt{self}}, g^{\mathtt{other}}, g^{\mathtt{free}}, g^{\mathtt{unknown}}). A network combines ResNet-18 features from the masked RGB, point-wise encodings of the masked point cloud, voxelization of those features, and 3D CNNs over the concatenated feature+occupancy grid, then predicts point-wise poses with confidences. Training uses the DenseFusion-style pose loss over model points pq∈Xp_q \in X, Li=1∣X∣∑q∥(Rpq+t)−(R^ipq+t^i)∥,L=1N∑i(Lici−λlog⁡ci)L_{i}=\frac{1}{|X|}\sum_{q}\lVert(\mathtt{R}p_{q}+\mathbf{t})-(\hat{\mathtt{R}}_{i}p_{q}+\hat{\mathbf{t}}_{i})\rVert, \qquad L=\frac{1}{N}\sum_{i}\big(L_{i}c_{i}-\lambda\log c_{i}\big) with λ=0.015\lambda=0.015; for symmetric objects the correspondence becomes a nearest-neighbor min over pq′p_{q'}, and a warm-up epoch with the standard loss avoids the symmetric loss’s local minima on non-convex shapes.
  3. Collision-based pose refinement (ICC) — poses of all objects are refined jointly by gradient descent through a differentiable occupancy voxelization: a CAD-model point pqp_q transformed by the pose hypothesis maps to voxel coordinates uq=(pq−l)/su_{q}=(p_{q}-l)/s, and voxel kk gets occupancy ok=1−δk/δto_{k}=1-\delta_{k}/\delta^{t} with δk=min⁡(δt,min⁡q∥uq−vk∥)\delta_{k}=\min(\delta^{t},\min_{q}\lVert u_{q}-v_{k}\rVert). Surrounding-object hypotheses are aggregated with an element-wise max and unioned with fused impenetrable space gmimpen=gmother∪gmfreeg^{\mathtt{impen}}_{m}=g^{\mathtt{other}}_{m}\cup g^{\mathtt{free}}_{m}; the loss penalizes intersection with impenetrable space while rewarding overlap with the object’s own fused surface: L=1N∑m(Lmc+−Lmc−),Lmc+=∑kgmtarget⊙gmtarget−∑kgmtarget,Lmc−=∑kgmtarget⊙gmself∑kgmselfL=\frac{1}{N}\sum_{m}\big(L_{m}^{\mathtt{c+}}-L_{m}^{\mathtt{c-}}\big), \quad L_{m}^{\mathtt{c+}}=\frac{\sum_{k} g^{\mathtt{target}}_{m}\odot g^{\mathtt{target-}}_{m}}{\sum_{k}g^{\mathtt{target}}_{m}}, \quad L_{m}^{\mathtt{c-}}=\frac{\sum_{k} g_{m}^{\mathtt{target}}\odot g_{m}^{\mathtt{self}}}{\sum_{k}g_{m}^{\mathtt{self}}} computed batched on GPU — objects in contact constrain each other.
  4. CAD alignment — once the pairwise pose losses among the NN most recent per-view hypotheses agree under a threshold, the CAD model is spawned into the map at the agreed pose, replacing the intermediate volumetric representation.

Results

Evaluated with ADD(-S)/ADD-S AUC (10 cm max threshold) on YCB-Video and the authors’ harder physics-simulated Cluttered YCB dataset (1200 scenes × 15 frames). The reimplemented baseline DenseFusion* (warm-up loss + point-cloud centralization) already lifts the official DenseFusion from 83.9/90.9 to 89.1/93.3 on YCB-Video. Trained on the combined set, MoreFusion reaches 91.0/95.7 on YCB-Video vs 88.4/94.9 for DenseFusion*, and 83.4/92.3 vs 81.7/91.7 on Cluttered YCB; the gap widens under heavy occlusion (visibility < 30 %): 63.5/85.1 vs 59.7/83.8. Feeding richer occupancy context at test time (full non-target + background reconstructions) improves further to 85.5/93.8 — exactly the information the incremental mapping system accumulates. For refinement, Iterative Collision Check beats ICP under heavy occlusion and the combination ICC+ICP is consistently best (ICC resolves collisions in the discretized grid, ICP then aligns surfaces precisely). The headline demonstration is a real-time robot arm that disassembles complicated piles of objects — moving obstructing objects aside to reach a target — using only on-board RGB-D vision.

Why it matters for SLAM

MoreFusion connects semantic SLAM to robot manipulation: cluttered tabletop scenes with touching, occluding objects are exactly where single-view 6D pose estimators fail and where a SLAM-style multi-view, map-centric approach shines. It extends the Fusion++ per-object volume idea from reconstruction to precise 6D pose estimation, and its collision-aware joint optimization is a principled treatment of mutual occlusion that later object-level and manipulation-oriented systems build on.