SuperGlue

Sarlin 2020 · Paper

One-line summary — Graph Neural Network feature matcher that uses self- and cross-attention plus differentiable Sinkhorn optimal transport (with a dustbin for unmatched points) to replace brittle nearest-neighbor matching.

Problem

Classical feature matching is a pipeline of hand-designed heuristics: nearest-neighbor search in descriptor space, ratio test, mutual check, then RANSAC to clean up. Each descriptor is compared independently — no reasoning about the other keypoints, the scene’s geometry, or which points are simply not visible in the other image. Under strong viewpoint change, repetitive structure, or partial overlap, this collapses. SuperGlue reframes matching itself as a learnable optimization problem: jointly find correspondences and reject non-matchable points, exploiting two physical constraints — a keypoint has at most one correspondence, and some keypoints are unmatched due to occlusion or detector failure.

Method & architecture

Given images $A, B$ with $M$ and $N$ local features (position $\mathbf{p}_i := (x, y, c)_i$ with detection confidence $c$, and descriptor $\mathbf{d}_i \in \mathbb{R}^D$, e.g. SuperPoint or SIFT), SuperGlue predicts a partial soft assignment $\mathbf{P} \in [0,1]^{M \times N}$ with $\mathbf{P}\mathbf{1}_N \leq \mathbf{1}_M$ and $\mathbf{P}^\top \mathbf{1}_M \leq \mathbf{1}_N$. Two blocks:

1. Attentional Graph Neural Network. A keypoint encoder embeds position into the descriptor so appearance and layout are reasoned about jointly:

$${}^{(0)}\mathbf{x}_i = \mathbf{d}_i + \mathrm{MLP}_{\mathrm{enc}}(\mathbf{p}_i)$$

All keypoints of both images form one complete multiplex graph with self edges (within an image) and cross edges (across images). A residual message-passing update runs for $L$ layers, alternating self and cross edges:

$${}^{(\ell+1)}\mathbf{x}_i^A = {}^{(\ell)}\mathbf{x}_i^A + \mathrm{MLP}\big(\big[{}^{(\ell)}\mathbf{x}_i^A \,\Vert\, \mathbf{m}_{\mathcal{E}\rightarrow i}\big]\big)$$

The message is attentional aggregation, $\mathbf{m}_{\mathcal{E}\rightarrow i} = \sum_{j} \alpha_{ij} \mathbf{v}_j$ with weights $\alpha_{ij} = \mathrm{Softmax}_j(\mathbf{q}_i^\top \mathbf{k}_j)$ over the edge set — self-attention lets a keypoint attend to salient points in its own image, cross-attention to candidate matches in the other image. Final matching descriptors are linear projections $\mathbf{f}_i^A = \mathbf{W}\,{}^{(L)}\mathbf{x}_i^A + \mathbf{b}$.

2. Optimal matching layer. Pairwise scores are inner products $\mathbf{S}_{i,j} = \langle \mathbf{f}_i^A, \mathbf{f}_j^B \rangle$. The score matrix is augmented with a dustbin row and column filled with a single learnable scalar $z$, so occluded/undetected points are explicitly assigned. The entropy-regularized optimal transport problem is solved with $T$ differentiable Sinkhorn iterations (iterative row/column normalization of $\exp(\bar{\mathbf{S}})$), yielding $\bar{\mathbf{P}}$; dropping the dustbins recovers $\mathbf{P}$.

Supervision. Negative log-likelihood over ground-truth matches $\mathcal{M}$ (from poses + depth or homographies) and unmatched sets $\mathcal{I}, \mathcal{J}$:

$$\mathrm{Loss} = -\sum_{(i,j)\in\mathcal{M}} \log \bar{\mathbf{P}}_{i,j} - \sum_{i\in\mathcal{I}} \log \bar{\mathbf{P}}_{i,N+1} - \sum_{j\in\mathcal{J}} \log \bar{\mathbf{P}}_{M+1,j}$$

Details: $D = 256$, $L = 9$ layers of 4-head attention, $T = 100$ Sinkhorn iterations, 12M parameters; a forward pass averages 69 ms (15 FPS) per indoor pair on a GTX 1080 GPU. Match confidence threshold 0.2 at test time.

Results

• Homography estimation (synthetic homographies over the Oxford/Paris 1M distractor images): 98.3% recall and 90.7% precision; AUC 65.85 with plain DLT vs 53.67 with RANSAC — correspondences so clean that a non-robust least-squares solver beats RANSAC. NN matching gets 0.00 DLT AUC; OANet 52.29.
• Indoor pose (ScanNet, 1500 wide-baseline test pairs): SuperPoint+SuperGlue pose AUC@5°/10°/20° = 16.16/33.81/51.84 vs 11.76/26.90/43.85 for SuperPoint+OANet and 9.43/21.53/36.40 for NN+mutual; precision 84.4%. With SIFT: 6.71/15.70/28.67, up to 10× more correct matches than ratio-test matching.
• Outdoor pose (PhotoTourism): SuperPoint+SuperGlue AUC@5°/10°/20° = 34.18/50.32/64.16 vs 21.03/34.08/46.88 for OANet; precision 84.9%. SIFT+SuperGlue 23.68/36.44/49.44 vs 15.19/24.72/35.30 for ratio test.
• Ablation: the GNN explains most of the gains; backpropagating into SuperPoint descriptors lifts AUC@20° from 51.84 to 53.38, showing the path toward end-to-end learning.

Why it matters for SLAM

SuperGlue changed the front-end recipe for hard association problems: SuperPoint + SuperGlue became the dominant baseline for visual localization, wide-baseline loop closure, and mapping via the hloc pipeline. For SLAM specifically, it made relocalization work across day/night and strong viewpoint changes where descriptor-distance matching collapses. The paper itself frames this learnable middle-end as “a major milestone towards end-to-end deep SLAM”. Its cost — full attention over all keypoints every frame — motivated LightGlue, the efficient successor now standard in real-time settings.

Related

• SuperPoint — the detector/descriptor it usually matches
• LightGlue — faster adaptive successor
• LoFTR — detector-free dense alternative
• HF-Net — the hierarchical localization pipeline built around it
• hloc — the localization toolbox where it is the standard matcher