Patch NetVLAD

Hausler 2021 · Paper

One-line summary — Extracts multi-scale patch-level VLAD descriptors from NetVLAD residuals and re-ranks retrieval candidates with spatial verification, making visual place recognition far more robust to viewpoint change and perceptual aliasing.

Problem

Visual place recognition must survive the twin problems of appearance change (season, structure, illumination) and viewpoint change in an always-changing world. Global descriptors such as NetVLAD compress the whole image into a single vector, discarding spatial layout — fast to search but vulnerable to viewpoint shifts, partial occlusion, and perceptual aliasing. Full local-feature matching (e.g., SuperPoint + SuperGlue) preserves layout but is far too slow against a large database. Patch-NetVLAD combines the advantages of both local and global descriptor methods in one configurable pipeline.

Method & architecture

Two-stage retrieval. Vanilla NetVLAD first retrieves the top-$k$ ($k{=}100$) database candidates for a query; patch-level matching then re-ranks that shortlist with a spatial-consistency score, so cross-matching cost is paid only on 100 images, never the whole database.

Patch-level VLAD descriptors. NetVLAD’s aggregation layer sums soft-assigned residuals between CNN features $\mathbf{x}_i$ and $K$ learned cluster centres $\mathbf{c}_k$:

$$f_{\mathrm{VLAD}}(F)(j,k) = \sum_{i=1}^{N} \bar{a}_k(\mathbf{x}_i)\,\big(x_i(j) - c_k(j)\big)$$

Instead of aggregating the entire $H \times W \times D$ feature map ($N = H \times W$, the global NetVLAD), Patch-NetVLAD applies the same aggregation + projection $\mathbf{f}_i = f_{\mathrm{proj}}(f_{\mathrm{VLAD}}(P_i))$ to a dense set of $d_x \times d_y$ patches with stride $s_p$ over the feature-space grid, with

$$n_p = \Big\lfloor \tfrac{H-d_y}{s_p} + 1 \Big\rfloor \cdot \Big\lfloor \tfrac{W-d_x}{s_p} + 1 \Big\rfloor$$

patches per image — “locally-global” descriptors tied to spatial locations, requiring no keypoint detection.

Mutual nearest neighbours + spatial scoring. Query/reference patch descriptors are exhaustively cross-matched; the mutual-NN pair set $\mathcal{P}$ is scored either by RANSAC (inlier count of a fitted homography, inlier tolerance $s_p$, normalized by $n_p$) or by rapid spatial scoring over horizontal/vertical displacements $x_d, y_d$ of matched patches:

$$s_{\mathrm{spatial}} = \frac{1}{n_p} \sum_{i \in \mathcal{P}} \Big( \big|\max_j x_{d,j}\big| - \big|x_{d,i} - \bar{x}_d\big| \Big)^2 + \Big( \big|\max_j y_{d,j}\big| - \big|y_{d,i} - \bar{y}_d\big| \Big)^2$$

which penalizes matches whose offsets deviate from the mean motion — spatial verification without sampling.

Multi-scale fusion via IntegralVLAD. Scores from $n_s$ patch sizes are fused by a convex combination $s_{\mathrm{spatial}} = \sum_i w_i\, s_{i,\mathrm{spatial}}$ (used config: square patch sizes 2, 5, 8 with $w_i = 0.45, 0.15, 0.4$; a size-5 patch covers 228×228 px of a 640×480 image). An integral feature map $\mathcal{I}(i,j) = \sum_{i' < i, j' < j} \mathbf{f}^1_{i',j'}$ of $1{\times}1$ patch VLADs lets arbitrary patch sizes be recovered with four look-ups, implemented as dilated depth-wise convolution with kernel $K = \begin{pmatrix} 1 & -1 \\ -1 & 1 \end{pmatrix}$. PCA dimensionality reduction on patch descriptors gives a configurable speed/accuracy trade-off.

Results

Evaluated on ~300,000 images across six benchmarks (Nordland, Pittsburgh 30k, Tokyo 24/7, Mapillary MSLS, RobotCar Seasons v2, Extended CMU Seasons) with a single configuration tuned once on RobotCar Seasons v2 training data:

• Outperforms the global-descriptor methods NetVLAD, DenseVLAD and AP-GEM by 17.5%, 14.8% and 22.3% absolute R@1 on average; on Nordland’s extreme seasonal change the margin over NetVLAD is 34.5% (R@1 44.9 vs 10.4).
• Beats an optimistic NetVLAD→SuperPoint+SuperGlue re-ranking baseline by 3.1% absolute R@1 on average (6.0% relative), including 15.8% absolute on Nordland (44.9 vs 29.1) — while SuperGlue edges ahead on parts of Tokyo 24/7 and Pittsburgh.
• Won the Facebook Mapillary Visual Place Recognition Challenge at ECCV 2020: 48.1% R@1 on the withheld test set, +13.0% absolute over the NetVLAD baseline (35.1%).
• Rapid spatial scoring on multiple patch sizes is 3.1× faster than multi-RANSAC scoring with only 1.1% R@1 degradation; the speed-optimized configuration runs over an order of magnitude faster than the previous state of the art.

Why it matters for SLAM

Loop closure and relocalization in SLAM are exactly a place-recognition problem: a wrong match corrupts the pose graph, so robustness to perceptual aliasing matters more than raw retrieval speed. Patch-NetVLAD’s global-retrieval-then-spatial-re-ranking recipe is a practical drop-in for SLAM loop-closure front-ends — the paper explicitly targets enhancing “the overall performance of SLAM systems” — and it influenced the hierarchical retrieval designs used in long-term localization systems.

Related

• NetVLAD — the global descriptor this method builds on
• HF-Net — hierarchical (coarse-to-fine) localization pipeline
• hloc — the retrieval-then-match localization toolbox this recipe plugs into
• SuperGlue — full local feature matching used when accuracy trumps speed
• Visual Place Recognition (VPR) — the task this paper targets