HF-Net

Sarlin 2019 · Paper

One-line summary — Coarse-to-fine hierarchical localization: global retrieval narrows the search to candidate places, local feature matching then yields a precise 6-DoF pose — with HF-Net computing both feature types in a single CNN forward pass.

Problem

Robust and accurate visual localization is fundamental for autonomous driving, mobile robotics, and AR, but remains hard at large scale and under strong appearance change (day-night, seasons). Direct 2D-3D matching methods (Active Search, CSL) are accurate but become ambiguous and slow as models grow and appearance changes; image-retrieval methods are robust but only give a pose up to the database discretization. Hand-crafted features (SIFT) limit robustness, learned dense features are intractable on mobile — and computing a retrieval network plus a local-feature network separately is redundant, since both start from the same low-level image cues.

Method & architecture

Hierarchical localization pipeline (following Sarlin 2018, upgraded with learned features):

1. Prior retrieval: match the query’s global descriptor against database images; the k-nearest neighbors are candidate locations.
2. Covisibility clustering: group prior frames into places — connected components of the covisibility graph linking database images to the SfM model’s 3D points.
3. Local matching: per place, match query 2D keypoints to the place’s 3D points and estimate a 6-DoF pose with PnP + RANSAC, stopping at the first valid pose. A modified ratio test only rejects a match if the two nearest neighbors belong to different 3D points, keeping matches in highly covisible areas.

The strongest variant, NV+SP, uses NetVLAD for retrieval and SuperPoint for local features; SfM models are re-triangulated with SuperPoint keypoints via COLMAP under ground-truth reference poses.

HF-Net. To make this run on mobile, a single MobileNetV2 encoder (depth multiplier 0.75) feeds three heads: keypoint scores and dense local descriptors (SuperPoint’s parameter-free decoding, branching at layer 7 where spatial resolution is still high) and a NetVLAD global-descriptor head at layer 18 — local features are lower-level than image-wide ones, so the split points differ.

Multi-task distillation. Ground-truth local correspondences and globally diverse imagery don’t coexist in any dataset, so HF-Net is instead trained to copy three teachers — NetVLAD ($t_1$, global), SuperPoint ($t_2$ descriptors, $t_3$ keypoints) — with self-balancing loss weights $w_{1,2,3}$ (Kendall et al.):

$$L=e^{-w_{1}}\|\mathbf{d}^{g}_{s}-\mathbf{d}^{g}_{t_{1}}\|_{2}^{2}+e^{-w_{2}}\|\mathbf{d}^{l}_{s}-\mathbf{d}^{l}_{t_{2}}\|_{2}^{2}+2e^{-w_{3}}\,\mathrm{CrossEntropy}(\mathbf{p}_{s},\mathbf{p}_{t_{3}})+\sum_{i}w_{i}$$

where $\mathbf{d}^{g},\mathbf{d}^{l}$ are global/local descriptors and $\mathbf{p}$ keypoint scores. Training uses 185k Google Landmarks day-time images plus 37k night/dawn Berkeley Deep Drive images (night data proved critical for night retrieval), with photometric augmentation but teacher targets predicted on clean images.

Results

• Benchmarks (recall within distance/orientation thresholds): on Aachen Day-Night, NV+SP localizes 40.8 / 56.1 / 74.5% of night queries at (0.5m,2°)/(1m,5°)/(5m,10°) vs 30.6/43.9/58.2 for NV+SIFT and 19.4/30.6/43.9 for Active Search. On CMU Seasons urban, NV+SP reaches 91.7/94.6/97.7 vs 75.0/82.1/87.8 for the semantic SMC baseline. On RobotCar Seasons night, 6.6/17.1/32.2 vs 0.5/1.1/3.4 for AS. HF-Net tracks its NV+SP upper bound within 2.6% average recall (its distilled global descriptor is the limiting factor on blurry RobotCar night queries).
• Distillation can beat the teacher: swapping HF-Net’s local features into the NV+SP pipeline (NV+HF-Net) slightly outperforms SuperPoint itself on Aachen (e.g., 81.2 vs 79.7 at 0.25m day).
• Leaner maps: the SuperPoint/HF-Net Aachen model has 685k 3D points vs 1,899k for SIFT and 2,576 vs 10,230 keypoints per image, with a higher fraction of keypoints matched (33.8% vs 18.8%) — sparser models that localize better.
• Runtime (GTX 1080): full localization in 45 ms on Aachen day vs 148 ms for NV+SP, 375 ms for Active Search and 1356 ms for NV+SIFT — HF-Net inference is 7x faster than running NetVLAD+SuperPoint separately, and the whole system runs at over 20 FPS, ~10x faster than AS.

Why it matters for SLAM

The coarse-to-fine paradigm HF-Net established is now the universal design for relocalization and loop-closure verification in SLAM: essentially every modern system retrieves places with a global descriptor and confirms them with local feature matching. Its companion toolbox hloc became the standard visual localization pipeline used in competitions and research, and the same recipe powers large-scale AR localization services.

Related

• hloc — the open-source toolbox implementing this hierarchical pipeline
• NetVLAD — the global retrieval descriptor
• SuperPoint — the local features used for fine matching
• SuperGlue — the learned matcher that later upgraded the fine stage
• Visual Place Recognition (VPR) — the coarse retrieval problem in general