LoFTR

Sun 2021 · Paper

One-line summary — Detector-free dense feature matching with Transformers: self- and cross-attention condition features on both images, producing reliable matches even in texture-poor regions where keypoint detectors fail.

Problem

The classical pipeline performs feature detection, description, and matching sequentially, so everything hinges on the detector producing repeatable interest points in both images — which detectors notoriously fail to do in low-texture areas (blank walls, floors) and under repetitive patterns. Existing dense alternatives searched correspondences through cost volumes, which is expensive and still relies on local evidence. LoFTR skips detection entirely: establish pixel-wise dense matches at a coarse level first, then refine the good ones — with a Transformer’s global receptive field deciding what matches, conditioned on both images at once.

Method & architecture

Backbone. A ResNet-18 + FPN CNN extracts coarse features $\tilde{F}^A, \tilde{F}^B$ at 1/8 resolution and fine features $\hat{F}^A, \hat{F}^B$ at 1/2 resolution.

LoFTR module. 2D sinusoidal positional encodings (added once) make features position-dependent — crucial for matching indistinctive regions. Then $N_c = 4$ interleaved self- and cross-attention layers transform the coarse features into $\tilde{F}^A_{tr}, \tilde{F}^B_{tr}$. Vanilla attention $\mathrm{Attention}(Q,K,V) = \mathrm{softmax}(QK^T)\,V$ costs $O(N^2)$, so LoFTR uses the Linear Transformer kernel

$$\mathrm{sim}(Q,K) = \phi(Q)\cdot\phi(K)^{T}, \quad \phi(\cdot) = \mathrm{elu}(\cdot) + 1$$

which, by associativity of matrix products (compute $\phi(K)^T V$ first, feature dim $D \ll N$), reduces the cost to $O(N)$.

Coarse matching. A score matrix $\mathcal{S}(i,j) = \frac{1}{\tau} \langle \tilde{F}^A_{tr}(i), \tilde{F}^B_{tr}(j) \rangle$ feeds either an optimal-transport layer (as in SuperGlue) or dual-softmax:

$$\mathcal{P}_c(i,j) = \mathrm{softmax}\big(\mathcal{S}(i,\cdot)\big)_j \cdot \mathrm{softmax}\big(\mathcal{S}(\cdot,j)\big)_i$$

Coarse matches are pairs that are mutual nearest neighbours in $\mathcal{P}_c$ with confidence $\geq \theta_c = 0.2$.

Coarse-to-fine refinement. For every coarse match, $w \times w = 5 \times 5$ local windows are cropped from the fine feature maps and transformed by a smaller LoFTR module ($N_f = 1$); correlating the centre vector of the query window against the other window yields a matching-probability heatmap whose expectation gives the sub-pixel position $\hat{j}'$.

Supervision. $\mathcal{L} = \mathcal{L}_c + \mathcal{L}_f$: negative log-likelihood over ground-truth coarse grid matches (from poses + depth, as in SuperGlue), plus a heatmap-variance-weighted $\ell_2$ loss on the fine offsets:

$$\mathcal{L}_f = \frac{1}{|\mathcal{M}_f|} \sum_{(\hat{i},\hat{j}')\in\mathcal{M}_f} \frac{1}{\sigma^2(\hat{i})} \big\lVert \hat{j}' - \hat{j}'_{gt} \big\rVert_2$$

Cost. 116 ms per 640×480 pair on an RTX 2080Ti with dual-softmax (130 ms with optimal transport); trained end-to-end from scratch, 24 h on 64 GTX 1080Ti GPUs for the indoor model.

Results

• HPatches homographies: AUC@3px 65.9 vs 53.9 for SuperPoint+SuperGlue, 50.6 for DRC-Net; the margin grows at stricter thresholds.
• Indoor pose (ScanNet, 1500 test pairs): LoFTR-DS AUC@5°/10°/20° = 22.06/40.8/57.62 vs 16.16/33.81/51.84 for SuperPoint+SuperGlue and 7.69/17.93/30.49 for DRC-Net — the biggest gains exactly in low-texture, wide-baseline indoor scenes.
• Outdoor pose (MegaDepth): LoFTR-DS 52.8/69.19/81.18 vs 42.18/61.16/75.96 for SuperPoint+SuperGlue (13% better at AUC@10°) and 61% better than detector-free DRC-Net at AUC@10°.
• Visual localization: at publication, ranked first among published methods on two tracks of the Long-Term Visual Localization benchmark — best on the Aachen v1.1 night-time local-feature track (LoFTR-DS 72.8/88.5/99.0) and best published on InLoc (DUC1 47.5/72.2/84.8, DUC2 54.2/74.8/85.5 with hloc).
• Ablation: replacing the LoFTR module with convolutions of comparable parameters drops AUC substantially (14.98 vs 22.06 @5°); DETR-style per-layer positional encoding also hurts.

Why it matters for SLAM

Indoor SLAM constantly fails where there is nothing to detect — blank walls, floors, repetitive surfaces. LoFTR showed that a global-context matcher can produce correspondences there anyway, and it established the detector-free paradigm that RoMa, EfficientLoFTR, and many others build on. In practice it is a go-to choice for indoor reconstruction, wide-baseline relocalization, and loop-closure verification when sparse matching is too brittle, at the cost of more compute than sparse matchers.

Related

• SuperGlue — the sparse learned-matching counterpart
• LightGlue — fast sparse matcher; the efficiency-focused alternative
• RoMa — dense matching with foundation-model features
• SuperPoint — the detector-based paradigm LoFTR sidesteps
• HF-Net — visual localization pipeline where detector-free matchers slot in