SAM 2

Meta 2024 · Paper

One-line summary — Extends Segment Anything to video with a streaming memory-attention architecture that tracks and segments objects consistently across frames, while also being more accurate and 6x faster than SAM on still images.

Problem

SAM segments each image independently, so running it on video loses object identity across frames — yet robotics, AR, and video editing all need the same object segmented consistently over time, through motion, deformation, occlusion, and lighting change, and often at lower per-frame quality (blur, camera motion). SAM 2 poses Promptable Visual Segmentation (PVS): given clicks/boxes/masks on any frame of a video, predict the object’s spatio-temporal mask (a “masklet”) across the entire video, refinable with further prompts on any frame. Existing VOS models were not promptable in this sense, and video mask annotations were scarce — so the model and the dataset had to be built together.

Method & architecture

A streaming generalization of SAM — frames are consumed one at a time, and when the memory is empty the model is SAM on an image:

• Image encoder: an MAE pre-trained Hiera hierarchical ViT runs once per frame, producing unconditioned multi-scale embeddings; hierarchy enables high-resolution skip connections into the decoder (bypassing memory attention) for crisp masks.
• Memory attention: $L$ transformer blocks condition the current frame’s features on the past — each block does self-attention, then cross-attention to the memory bank’s entries (spatial memories of prompted and unprompted frames, plus object pointers), then an MLP; vanilla attention so efficient kernels apply.
• Prompt encoder + mask decoder: as in SAM (points/boxes/masks; ambiguity handled by multiple candidate masks), plus a new occlusion head predicting whether the object is even present on the current frame — in video, a valid prompt can have no visible target.
• Memory encoder: downsamples the predicted mask with a convolutional module, sums it element-wise with the unconditioned frame embedding, and fuses with light conv layers.
• Memory bank: a FIFO queue of up to $N$ recent-frame memories plus up to $M$ prompted-frame memories (e.g., in VOS the first frame’s memory is always retained), stored as spatial feature maps, alongside a list of object pointers — lightweight vectors from the mask decoder’s output tokens carrying high-level object semantics.

Training is joint on images and video: 8-frame sequences with up to 2 prompted frames, corrective clicks sampled from ground truth and model predictions; the initial prompt is a ground-truth mask (p=0.5), a positive click (p=0.25), or a box (p=0.25).

Data engine (three phases): Phase 1, per-frame SAM annotation at 6 FPS — 37.8 s/frame, 16K masklets; Phase 2, SAM 2 Mask propagates first-frame masks — 7.4 s/frame (5.1x speedup), 63.5K masklets; Phase 3, full SAM 2 in the loop with occasional refinement clicks — 4.5 s/frame (8.4x speedup at comparable quality), 197K masklets, with a separate verification pass and automatic grid-prompted masklets for coverage. Result: SA-V — 50.9K videos, 642.6K masklets, 35.5M masks, 53x more masks (15x without auto) than any prior VOS dataset, released under CC-BY 4.0.

Results

• Interactive video segmentation: over 9 datasets, SAM 2 beats SAM+XMem++ and SAM+Cutie in $\mathcal{J}\&\mathcal{F}$ across offline and online settings, achieving better accuracy with more than 3x fewer interactions.
• Semi-supervised VOS (first-frame mask prompt), $\mathcal{J}\&\mathcal{F}$: SAM 2 (Hiera-L) scores 77.9 MOSE val, 90.7 DAVIS 2017 val, 78.0 LVOS val, 77.9/78.4 SA-V val/test, 89.3 YouTube-VOS 2019 — vs the best prior method Cutie-base+ at 71.7 MOSE and ~61/63 on SA-V, showing how far prior VOS was from “segment anything in videos”.
• Speed: 43.8 FPS (Hiera-B+) / 30.2 FPS (Hiera-L) on a single A100 — real-time streaming.
• Image segmentation: on the 37-dataset SA benchmark, SAM 2 gets 58.9 1-click mIoU on SA-23 vs SAM’s 58.1 without extra data while being 6x faster (smaller but more effective Hiera encoder); training on the SA-1B + video mix lifts it to 61.4.
• Data ablations: adding data-engine data gives +12.1 $\mathcal{J}\&\mathcal{F}$ average on 9 zero-shot VOS datasets over training on DAVIS/MOSE/YouTube-VOS alone; accuracy follows a power law in SA-V training data size.

Why it matters for SLAM

Frame-by-frame masks are not enough for SLAM — the system must know it is seeing the same object over time to build semantic maps and handle dynamic scenes. SAM 2’s streaming memory design matches the sequential nature of SLAM pipelines: temporally consistent masklets enable dynamic-object removal, object-level mapping, and persistent instance identity for data association, and the occlusion head explicitly models disappearance/re-appearance that map maintenance must handle. Spatio-temporal semantic mapping systems (e.g., Khronos-style) are natural consumers of this capability.

Related

• SAM — the image-only predecessor
• Grounding DINO — text prompts for what to track
• Khronos — spatio-temporal metric-semantic mapping
• DynaSLAM — dynamic-object masking in classical SLAM