R2D2

Revaud 2019 · Paper

One-line summary — Jointly learns explicit repeatability and reliability maps alongside dense descriptors, so keypoints are only detected where they can actually be matched.

Problem

Classical local features follow a detect-then-describe paradigm: hand-crafted detectors find repeatable keypoints, then a separate descriptor represents them. Learned methods had caught up — repeatable saliency maps for detection, metric-learning descriptors at those locations — but R2D2 argues this conflates two different properties. Salient regions are not necessarily discriminative: every square of a checkerboard, every window on a facade, tree leafage, sea waves are detected consistently yet match ambiguously due to self-similarity. Descriptors should only be trusted (and keypoints only detected) where matching can be performed with high confidence.

Method & architecture

Three dense outputs from one FCN. For an H×WH \times W image the network predicts: dense L2-normalized 128-D descriptors X\boldsymbol{X}, a repeatability heatmap S∈[0,1]H×W\boldsymbol{S}\in[0,1]^{H\times W} whose local maxima are keypoint candidates, and a reliability map R∈[0,1]H×W\boldsymbol{R}\in[0,1]^{H\times W} estimating each descriptor’s discriminativeness. Backbone is L2-Net with the final 8x8 convolution replaced by three 2x2 convolutions (5x fewer weights); S\boldsymbol{S} and R\boldsymbol{R} come from an element-wise square, a 1x1 convolution and a softmax on the 128-D output.

Repeatability: self-supervised cosine similarity. Given images I,I′I, I' with dense ground-truth correspondences UU and SU′\boldsymbol{S}'_U the second heatmap warped by UU, the maps are aligned patchwise over all N×NN\times N patches P\mathcal{P}:

Lcosim(I,I′,U)=1−1∣P∣∑p∈Pcosim(S[p],SU′[p])\mathcal{L}_{cosim}(I,I',U)=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}cosim\bigl(\boldsymbol{S}\left[p\right],\boldsymbol{S}'_{U}\left[p\right]\bigr)

A peakiness loss prevents the trivial constant solution and sets keypoint density via NN:

Lpeaky(I)=1−1∣P∣∑p∈P(max⁡(i,j)∈pSij−mean(i,j)∈pSij)\mathcal{L}_{peaky}(I)=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\Bigl(\max_{(i,j)\in p}\boldsymbol{S}_{ij}-\mathrm{mean}_{(i,j)\in p}\boldsymbol{S}_{ij}\Bigr)

combined as Lrep=Lcosim+λ(Lpeaky(I)+Lpeaky(I′))\mathcal{L}_{rep}=\mathcal{L}_{cosim}+\lambda\left(\mathcal{L}_{peaky}(I)+\mathcal{L}_{peaky}(I')\right).

Reliability: AP ranking loss with a learned confidence. Descriptor matching is treated as a ranking problem: for each query pixel, a differentiable approximation of Average Precision over candidates in the second image is optimized directly, rather than a triplet/contrastive surrogate. Crucially, the network may opt out on undistinctive regions:

LAPκ(i,j)=1−[AP(i,j) Rij+κ (1−Rij)]\mathcal{L}_{AP\kappa}(i,j)=1-\bigl[AP(i,j)\,\boldsymbol{R}_{ij}+\kappa\,(1-\boldsymbol{R}_{ij})\bigr]

where κ=0.5\kappa=0.5 is the minimum expected AP: the optimum is Rij=0\boldsymbol{R}_{ij}=0 where AP(i,j)<κAP(i,j)<\kappa and 1 otherwise, so R\boldsymbol{R} becomes an interpretable matchability probability.

Training data & inference. Dense correspondences come free from random homographies on web images, and from an optical-flow pipeline (EpicFlow with epipolar-constrained DeepMatching, masked by match-density) on SfM-verified Aachen image pairs — no manual labels. At test time the network runs over a scale pyramid (downsampling by 21/42^{1/4}), keypoints are local maxima of S\boldsymbol{S}, and the top KK are kept by the combined score SijRij\boldsymbol{S}_{ij}\boldsymbol{R}_{ij}.

Results

Why it matters for SLAM

Wrong matches are the dominant failure mode of feature-based SLAM under repetitive structure (corridors, facades, vegetation). R2D2 established the principle of reliability-aware keypoint selection — quality over quantity — which carried into later detector/descriptor designs and into how modern pipelines score and filter correspondences for localization and 3D reconstruction.

Hands-on