RoMa

Edstedt 2024 · Paper

One-line summary — Robust dense feature matching that fuses frozen DINOv2 foundation-model features (robust but coarse) with specialized VGG19 fine features (precise but brittle) in a Transformer match decoder predicting anchor probabilities, trained with regression-by-classification then robust regression.

Problem

Dense feature matching — estimating a dense warp $W^{\mathcal{A}\to\mathcal{B}}$ plus a matchability score $p(x^{\mathcal{A}})$ for every pixel between two images — must survive extreme real-world changes in scale, illumination, viewpoint, and texture. Features trained from scratch on matching data (DKM’s ResNet50) are spatially precise but overfit to the training set; frozen DINOv2 features are dramatically more robust (the paper measures 27.1 px EPE / 85.6% robustness vs 60.2 / 57.5% for ResNet50 and 87.6 / 43.2% for VGG19 on a frozen-feature coarse-matching probe) but exist only at coarse stride 14. RoMa asks how to get both, and how to train each stage with a loss matched to its error regime.

Method & architecture

Two-stage dense pipeline (DKM skeleton). Decoupled encoders extract coarse and fine features; a global matcher $G_\theta = D_\theta(E_\theta(\varphi^{\mathcal A}_{\text{coarse}}, \varphi^{\mathcal B}_{\text{coarse}}))$ produces a coarse warp and certainty, then refiners $R_{\theta,i}$ at strides $\{1,2,4,8\}$ recursively predict residual warp and certainty-logit offsets using stacked feature maps and a local correlation volume around the previous estimate, with gradients detached between stages.

Robust + localizable features. $F_{\text{coarse},\theta}=\text{DINOv2}$ (frozen throughout training — fixing the representation reduces overfitting and cuts compute), while $F_{\text{fine},\theta}=\text{VGG19}$: the ablation shows VGG19 makes poor coarse features but the best fine features, revealing “an inherent tension between fine localizability and coarse robustness.”

Transformer match decoder with anchor probabilities. Instead of regressing coordinates, the decoder (5 ViT blocks, 8 heads, hidden size 1024, no position encodings — propagating only by feature similarity to avoid resolution overfitting and oversmoothing) outputs a discretized conditional distribution over $K = 64\times 64$ uniform anchors:

$$p_{\text{coarse},\theta}(x^{\mathcal{B}}|x^{\mathcal{A}})=\sum_{k=1}^{K}\pi_k(x^{\mathcal{A}})\,\mathcal{B}_{m_k},$$

with $\pi_k$ anchor probabilities and $m_k$ anchor coordinates — so multimodal ambiguity (repetitive structure, motion boundaries) is represented instead of averaged. The warp is decoded by argmax over anchors followed by a local softargmax over the 4-neighborhood $N_4(k^*)$.

Loss matched to each stage. Modeling matchability at scale $s$ as a blurred joint distribution $q(x^{\mathcal{A}},x^{\mathcal{B}};s)=\mathcal{N}(0,s^2\mathbf{I}) \ast p(x^{\mathcal{A}},x^{\mathcal{B}};0)$ shows the coarse conditional is multimodal near motion boundaries while refinement (conditioned on the previous warp) is locally unimodal. Hence $\mathcal{L}_{\text{coarse}}$ is regression-by-classification — NLL of the anchor closest to the ground truth, $k^{\dagger}(x)=\operatorname{argmin}_k \lVert m_k - x\rVert$ — and $\mathcal{L}_{\text{fine}}$ is a robust generalized Charbonnier regression ($\alpha=0.5$), whose log-density is $-(\lVert\mu_\theta - x_i^{\mathcal{B}}\rVert^2 + s)^{1/4}$: locally L2-like gradients that decay toward zero for outliers. Total loss $\mathcal{L}=\mathcal{L}_{\text{coarse}}+\mathcal{L}_{\text{fine}}$ with no cross-stage weighting needed. Trained on MegaDepth (+ScanNet model for indoor eval) at 560×560.

Results

• Ablation (100−PCK@5px on MegaDepth validation, lower better): DKM baseline 5.8 → decoupled encoders 4.5 → +DINOv2 coarse 3.2 → +regression-by-classification 2.8 → +robust refinement loss 2.7 (full RoMa); swapping the Transformer decoder back to a ConvNet degrades to 3.5.
• WxBS (extreme wide multi-nuisance baselines): 80.1 mAA@10px vs DKM 58.9 and LoFTR 55.4 — a 36% gain over the prior state of the art.
• IMC2022: 88.0 mAA@10 vs DKM 83.1 — a 26% relative error reduction.
• MegaDepth-1500 pose: 62.6 / 76.7 / 86.3 AUC@5°/10°/20° (DKM 60.4/74.9/85.1); MegaDepth-8-Scenes: 62.2/75.9/85.3.
• ScanNet-1500 pose: 31.8 / 53.4 / 70.9 — the first method over 70 AUC@20°.
• InLoc visual localization: DUC1 60.6/79.3/89.9, DUC2 66.4/83.2/87.8 — state of the art.
• Runtime: only 7% slower than DKM (186.3 → 198.8 ms per pair at 560×560, batch 8, RTX 6000).

Why it matters for SLAM

RoMa demonstrated that frozen foundation-model features dramatically improve matching robustness — establishing the “foundation features for coarse anchors + specialized features for precision” paradigm. For SLAM this matters most in relocalization and loop closure under severe appearance change (day/night, seasons), where sparse hand-crafted or even learned keypoints fail. Dense RoMa-style matchers now back several modern reconstruction and localization pipelines.

Related

• RoMa v2 — the harder-better-faster-denser successor
• LoFTR — earlier detector-free Transformer matching
• DeDoDe — same group; decoupled detection/description
• Foundation models — why frozen pre-trained features generalize
• MASt3R — dense matching fused with 3D reconstruction