SpatialLM
Mao 2025 · Paper
One-line summary — Fine-tunes an off-the-shelf LLM to consume 3D point clouds and emit structured indoor models — walls, doors, windows, and oriented object boxes — as Python-dataclass scripts, grounding LLM reasoning in 3D spatial structure.
Problem
Structured indoor modeling — recovering the architectural layout (walls, doors, windows) plus 3D object bounding boxes from raw RGBD scans — has been the domain of task-specific networks (RoomFormer’s polygon queries, V-DETR’s detection heads) or bespoke autoregressive decoders with domain-specific tokens (SceneScript). Meanwhile LLMs excel at reasoning and code generation but have no grounding in 3D space. Two things blocked simply fine-tuning an LLM for the task: no large, high-quality dataset pairing point clouds with structured 3D annotations, and no empirical understanding of how to align point-cloud input with an LLM. SpatialLM (tech report: arXiv 2506.07491, NeurIPS 2025) tackles both.
Method & architecture
- Standard “Encoder–MLP–LLM” pipeline — no specialized decoder. A point-cloud encoder compresses the scene into a short sequence of visual tokens for the LLM:
where each point carries XYZ + RGB. The chosen encoder is Sonata (an encoder-only, self-supervised Point Transformer V3 variant), a two-layer MLP projector, and Qwen2.5-0.5B as the base LLM (603.5M params total; released checkpoints also include a Llama-1B variant).
- Scene description as Python code. The output is plain text: dataclass-style scripts (e.g.
class Wall: a_x, a_y, a_z, b_x, b_y, b_z, height) for walls/doors/windows and oriented bounding boxes over 59 object categories. Code is human-editable, extensible to new classes, and exploits the LLM’s built-in coding ability; unlike SceneScript’s domain-specific tokens, no custom vocabulary is needed. - SpatialLM dataset. 12,328 synthetic scenes (54,778 rooms) parsed from professionally authored interior designs, rendered photo-realistically along simulated camera trajectories (images every 0.5 m); split 11,328/500/500 train/val/test. This dwarfs real annotated datasets (SceneCAD: 1,151 rooms with layout labels).
- Empirical alignment study. Naive token reduction of per-point DINOv2 features (voxelization or farthest-point sampling) collapses to near-zero object F1 — too much spatial information is lost; learned encoders (sparse 3DCNN, Sonata) work, with Sonata best (layout F1 84.6 vs 79.4 for 3DCNN at [email protected]). Doubling encoder spatial resolution (K ≈ 510 tokens) helps (object F1 49.4 → 61.1 at [email protected]); 4× hurts. Single-stage training with encoder, MLP, and LLM all trainable beats every two- and three-stage freezing schedule.
- Evaluation metrics: Hungarian-matched F1 at IoU thresholds 0.25/0.5 — 2D IoU of projected plane segments for layout, 3D IoU for object boxes (F1 rather than mAP because autoregressive models emit no confidence scores).
Results
- Layout estimation (Structured3D): SpatialLM pre-trained on its own dataset then fine-tuned on Structured3D reaches F1 94.3 / 93.5 ([email protected]/0.5), beating the specialist RoomFormer (83.4/81.4) and SceneScript (90.4/89.2). Training on Structured3D alone gives only 32.6/18.1 — the large pre-training dataset is what makes LLM fine-tuning viable.
- 3D object detection (ScanNet, 18 classes, F1): pre-train + ScanNet fine-tune scores 65.6 / 52.6 vs SceneScript 49.1/36.8 and the specialist V-DETR 65.1/56.8 — ahead at [email protected], behind at 0.5, with most misses on the smallest objects (picture, sink). ScanNet-only training collapses (2.9/0.7).
- Zero-shot on video reconstructions: on 107 indoor-tour videos reconstructed with MASt3R-SLAM — noisy, occluded, artifact-ridden point clouds — SpatialLM produces consistent layouts and full object boxes without fine-tuning, completing unseen regions (beds extended to the floor, plausible balcony/dining layouts) from context.
Why it matters for SLAM
SpatialLM illustrates where semantic mapping is heading: from label-annotated point clouds (OpenScene, ConceptFusion) and scene graphs (ConceptGraphs) toward representations an LLM can directly parse and manipulate. A SLAM system provides the point cloud — the paper’s own demo pipeline runs MASt3R-SLAM on monocular video — and SpatialLM turns it into a compact, editable, machine-readable description of the space, useful for scene editing, AR, robot navigation, digital twins, and embodied AI.