SAM

Kirillov 2023 · Paper

One-line summary — Segment Anything: a promptable segmentation foundation model trained on 1.1B masks from 11M images that segments arbitrary objects zero-shot from point, box, or mask prompts.

Problem

Segmentation research was fragmented into task-specific models — interactive, semantic, instance, panoptic — each requiring its own labeled dataset and training run, and none transferring zero-shot to new image distributions or new tasks. NLP had shown that a foundation model trained at sufficient scale on a promptable task can generalize to tasks it was never explicitly trained for. The Segment Anything project asks what the equivalent task, model, and dataset are for segmentation — the catch being that masks are not naturally abundant on the internet, so billion-scale annotation had to be manufactured.

Method & architecture

The project has three interconnected components: a promptable segmentation task (return a valid mask for any prompt — point, box, mask, or exploratory free-form text), the model (SAM), and a data engine producing SA-1B.

SAM has three modules, designed so the expensive part is amortized per image:

• Image encoder: an MAE pre-trained ViT-H/16 (14×14 windowed attention with four global-attention blocks) on 1024×1024 input, producing a 16x-downscaled image embedding. Runs once per image, before any prompt arrives.
• Prompt encoder: sparse prompts (points, boxes) become positional encodings summed with learned per-type embeddings; text goes through an off-the-shelf CLIP text encoder; dense prompts (masks) are embedded by convolutions and summed element-wise with the image embedding.
• Mask decoder: two modified Transformer decoder blocks with prompt self-attention and two-directional cross-attention (prompt-to-image and image-to-prompt), then the image embedding is upsampled and an MLP maps an output token to a dynamic linear classifier that scores mask foreground probability per location. Given a precomputed embedding, prompt encoder + decoder run in ~50 ms in a web browser on CPU — real-time interactive prompting.

Ambiguity: a single click can validly mean whole/part/subpart, so SAM predicts 3 output masks per prompt with a predicted IoU confidence each; training backpropagates only the minimum loss over the three. Mask supervision is a linear combination of focal loss and dice loss, with an interactive setup simulated by sampling prompts over 11 rounds per mask.

Data engine (model-in-the-loop, three stages): (1) assisted-manual — annotators click with SAM in the loop, 4.3M masks from 120k images, per-mask time falling from 34 s to 14 s as the model was retrained 6 times; (2) semi-automatic — SAM pre-fills confident masks, annotators add the rest, +5.9M masks from 180k images; (3) fully automatic — prompt SAM with a 32×32 point grid, keep masks that are confident (predicted IoU) and stable (thresholding the probability map at $0.5-\delta$ and $0.5+\delta$ yields similar masks), deduplicate with NMS. Applied to all 11M images, this yielded SA-1B: 1.1B masks, ~100 masks/image, 11x more images and 400x more masks than Open Images; released images downsampled to 1500 px shortest side.

Results

• Single-point zero-shot segmentation on a new 23-dataset suite: SAM beats the strongest interactive baseline RITM on 16 of 23 datasets (by up to ~47 IoU); with an oracle picking the best of its 3 masks, it wins on all 23. In a human study, SAM’s masks rate 7–9 out of 10, consistently above RITM — even on datasets where automatic mIoU favors RITM.
• Zero-shot instance segmentation (prompting with ViTDet-H boxes): 46.5 mask AP on COCO vs 51.0 for fully supervised ViTDet-H, and 44.7 vs 46.6 on LVIS — close behind without any COCO/LVIS mask training, and rated higher than ViTDet by human annotators (ViTDet exploits dataset-specific mask biases).
• Zero-shot object proposals on LVIS: outperforms ViTDet-H on medium/large and rare/common objects, trailing only on small and frequent ones.
• Zero-shot edge detection on BSDS500: sensible edge maps despite never training for edges; high recall (R50) at lower precision.
• Ablations: training on automatic masks alone loses only ~0.5 mIoU vs all data; 1M images (~10% of SA-1B) nearly matches the full dataset; ViT-H clearly beats ViT-B but only marginally beats ViT-L.
• Stated limits: can miss fine structures, hallucinate small components, not real-time end-to-end with the heavy encoder; text-to-mask is exploratory.

Why it matters for SLAM

SAM gave SLAM systems class-agnostic object masks on demand — the missing ingredient for open-vocabulary semantic mapping. Paired with a text-prompted detector (Grounded SAM = Grounding DINO + SAM), it powers 3D scene-graph and open-vocabulary mapping systems like ConceptGraphs and Clio, and provides clean masks for dynamic-object removal. Its per-image amortized design (heavy encoder once, cheap prompts many times) fits SLAM keyframe pipelines, and its video successor SAM 2 adds the temporal consistency SLAM actually needs across frames.

Related

• SAM 2 — video extension with streaming memory
• Grounding DINO — text-prompted boxes that feed SAM
• ConceptGraphs — open-vocabulary 3D scene graphs built on SAM masks
• Clio — task-driven open-vocabulary mapping consumer of SAM-style masks
• ConceptFusion — open-set multimodal 3D mapping