PaperlayerView exact revision on arXiv

Source-linked AI summary

From Coarse to Fine: Robust Hierarchical Localization at Large Scale

Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, Marcin Dymczyk

arXiv:1812.03506v2cs.CV

TL;DR

Large-scale visual localization must provide accurate 6-DoF poses despite appearance changes and constrained resources. HF-Net uses coarse-to-fine retrieval and local matching with a monolithic CNN that jointly predicts global and local features. The resulting systems retain strong localization performance, exceed 20 FPS at large scale, and show robustness on challenging benchmarks.

  • Problem

    Large-scale localization remains difficult under substantial appearance changes, while existing approaches can be too resource intensive for real-time mobile operation.

  • Method

    HF-Net performs global image retrieval followed by local feature matching and uses a monolithic CNN with multitask distillation to compute hierarchical features efficiently.

  • Results

    The resulting localization systems run at more than 20 FPS at large scale and outperform state-of-the-art approaches across challenging benchmarks with substantial appearance variations.

  • Takeaways & Limitations

    Learned global and local features enable a real-time hierarchical localization system with robustness across challenging large-scale conditions.

  • Takeaways & Limitations

    On large, self-similar environments, HF-Net’s capacity can limit performance, and failure of global retrieval directly causes hierarchical localization failure.

Abstract

from arXiv · show

Robust and accurate visual localization is a fundamental capability for numerous applications, such as autonomous driving, mobile robotics, or augmented reality. It remains, however, a challenging task, particularly for large-scale environments and in presence of significant appearance changes. State-of-the-art methods not only struggle with such scenarios, but are often too resource intensive for certain real-time applications. In this paper we propose HF-Net, a hierarchical localization approach based on a monolithic CNN that simultaneously predicts local features and global descriptors for accurate 6-DoF localization. We exploit the coarse-to-fine localization paradigm: we first perform a global retrieval to obtain location hypotheses and only later match local features within those candidate places. This hierarchical approach incurs significant runtime savings and makes our system suitable for real-time operation. By leveraging learned descriptors, our method achieves remarkable localization robustness across large variations of appearance and sets a new state-of-the-art on two challenging benchmarks for large-scale localization.

1. Introduction

Large-scale visual localization must remain accurate and robust across appearance changes while operating under constrained computational resources. HF-Net addresses this with hierarchical learned features and reports state-of-the-art robustness and efficiency.

  • Motivation: Visual localization supports applications requiring precise 6-DoF camera poses, including GPS-denied driving, augmented reality, SfM, and SLAM.Reliable operation is needed across indoor and outdoor conditions, weather, illumination, and seasonal changes.
  • Problem: Large appearance variations and limited computational resources make accurate localization in changing environments difficult, especially on mobile devices.The paper specifically targets estimating a query image’s 6-DoF pose relative to a 3D model under these constraints.
  • Problem: Direct 2D-3D matching is either robust but intractable on mobile or efficient but fragile, while retrieval methods trade accuracy for robustness and scalability.Hand-crafted local features also provide limited invariance to challenging appearance changes.
  • Approach: HF-Net combines global descriptors and local features in a monolithic CNN to support coarse-to-fine localization while sharing computation.The network jointly predicts hierarchical features, and multitask distillation trains heterogeneous predictors into a compact model.
  • Results: The approach sets a new state-of-the-art on public benchmarks and achieves outstanding robustness in particularly challenging large-scale conditions.The introduction reports the result as a contribution without specifying a numeric metric in the supplied passages.

2. Related Work

Prior localization methods balance accuracy, robustness, scalability, and computational cost imperfectly. HF-Net combines hierarchical search, learned local features, and shared multitask computation to target both efficiency and robustness.

  • Visual localization: Structure-based localization can estimate accurate poses but becomes compute intensive and ambiguous as models grow and appearance changes increase.Perceptual aliasing particularly impairs robustness under conditions such as day and night.
  • Visual localization: Image retrieval is more robust because it uses global image information, but provides only approximate poses and relies on computationally large deep models.Retrieval alone is therefore insufficiently precise for many applications.
  • Scalability: Hierarchical localization divides the task into a global coarse search followed by fine pose estimation, addressing scalability differently from inexpensive direct-matching features.Earlier approaches still faced limitations in scalability and robustness.
  • Feature learning: Learned sparse local features offer a faster alternative to dense matching and can outperform handcrafted descriptors in repeatability and descriptor matching.Their sparsity reduces the number of keypoints that must be matched.
  • Deep learning on mobile: Mobile deployment motivates sharing computation across tasks and distilling larger trained models into smaller networks.The paper applies these ideas to combine efficiency and robustness in large-scale localization.
  • HF-Net: HF-Net is presented as a simpler hierarchical localization approach that is more robust, accurate, and efficient than concurrent approaches.The figure caption gives the comparison without supplying numerical values.

3. Hierarchical Localization

Hierarchical localization narrows a large map through global retrieval before estimating a precise pose with local feature matching. This coarse-to-fine design reduces the search space, but prior systems face scalability, computational, and robustness limitations.

  • Hierarchical pipeline: Hierarchical localization first retrieves candidate database images using global descriptors, then refines pose by matching local features within selected places.The pipeline clusters retrieved frames by shared 3D structure before local matching.
  • Hierarchical pipeline: Covisibility clustering groups prior frames into connected places based on the 3D points they jointly observe.These places define the bounded regions used for subsequent local search.
  • Hierarchical pipeline: Local matching estimates a 6-DoF pose by associating query keypoints with place-specific 3D points and applying PnP with RANSAC.The algorithm stops once a valid pose is estimated, while considering fewer 3D points than the full model.
  • Limitations: Prior systems can be robust but intractable on mobile, or efficient but fragile, while hand-crafted features provide limited robustness under appearance changes.SIFT-based local matching is also expensive and generates many features, limiting scalability to larger, denser models.
  • Limitations: Using heterogeneous SIFT-based local features and CNN-based retrieval introduces redundant computation because both rely on image low-level cues.This redundancy is computationally suboptimal for resource-constrained platforms.

4. Proposed Approach

The proposed HF-Net replaces heterogeneous feature extraction with a single CNN that jointly predicts global and local representations for coarse-to-fine localization. Multitask distillation addresses scarce supervision while sharing computation across predictions.

  • HF-Net architecture: HF-Net detects keypoints and computes local and global descriptors in one forward pass, reducing shared-computation overhead while retaining the performance of a larger baseline network.It is designed for efficient online localization on mobile devices.
  • HF-Net architecture: A single encoder with three heads predicts keypoint scores, dense local descriptors, and a global image-wide descriptor.The architecture exploits CNN hierarchy: local features branch earlier at higher spatial resolution than the global head.
  • HF-Net architecture: HF-Net uses a MobileNet backbone and combines NetVLAD-based global descriptors with SuperPoint-style decoding for efficient local features.Dense descriptors can be sampled bilinearly, making runtime independent of the number of detected keypoints.
  • Training: Multitask distillation learns local and global representations from different teacher networks, enabling supervision from arbitrary datasets without requiring ground-truth correspondences for every task.Teacher inference provides labeled data, while directly predicting teacher outputs simplifies learning.
  • Training: The distillation formulation can support multiple computationally efficient predictions when collecting ground-truth data for all tasks is expensive.The authors also identify potential applications to hand-crafted descriptors that are too compute-intensive.

5. Experiments

The experiments evaluate learned local features, hierarchical localization, and HF-Net across challenging large-scale datasets, emphasizing robustness, accuracy, and runtime. HF-Net approaches the performance of its upper bound while substantially reducing inference time, but distilled global descriptors remain a limitation in blurry, self-similar scenes.

  • 5.1. Local Features Evaluation: SuperPoint offers the best trade-off between keypoint repeatability and localization error among evaluated detectors.
  • 5.1. Local Features Evaluation: Learned descriptors outperform hand-crafted features, while SuperPoint descriptors depend strongly on the detections from which they are extracted.
  • 5.3. Large-scale Localization: HF-Net models contain fewer points and keypoints than SIFT models but achieve a higher matched-keypoint ratio, producing models more suitable for localization.
  • 5.3. Large-scale Localization: NV+SP achieves state-of-the-art localization on CMU and challenging Aachen and RobotCar sequences, outperforming baselines in fine- and coarse-precision regimes.
  • 5.3. Large-scale Localization: HF-Net performs similarly to NV+SP on most sequences, with an average recall drop of 2.6%.
  • 5.3. Large-scale Localization: On RobotCar night sequences, HF-Net is significantly worse than NV+SP because distilled global descriptors perform poorly on blurry, low-quality images.
  • 5.4. Runtime Evaluation: HF-Net inference is 7 times faster than NetVLAD and SuperPoint, whose feature extraction accounts for 75% of NV+SP runtime.
  • 5.4. Runtime Evaluation: HF-Net runs at 20 FPS on very large-scale environments and is 10 times faster than Active Search while achieving higher accuracy across all datasets.

6. Conclusion

The method combines global retrieval with local 2D-3D matching in a coarse-to-fine pipeline for robust, accurate, real-time 6-DoF visual localization. HF-Net further unifies keypoint and descriptor computation, achieving over 20 FPS at large scale under challenging appearance changes.

  • Global image retrieval first identifies candidate database images, which are clustered into places using a 3D SfM model's covisibility graph.
  • Local 2D-3D matching within candidate places then produces an accurate 6-DoF camera pose estimate.
  • HF-Net computes keypoints, global descriptors, and local descriptors in a single CNN, using multitask distillation while retaining the original performance.
  • More than 20 FPS at large scale is achieved, alongside robustness on benchmarks containing day-night queries and substantial weather and seasonal appearance changes.

A.1. Network Architecture

HF-Net uses a MobileNetV2 encoder with separate local and global heads, while its training combines descriptor regression and keypoint classification losses. The evaluation defines correctness through reprojection and descriptor-nearest-neighbor criteria across HPatches and SfM.

  • HF-Net uses a MobileNetV2 encoder with depth multiplier 0.75; SuperPoint-like local heads branch at layer 7, while a NetVLAD-based global head branches at layer 18.The global head also includes dimensionality reduction to match the teacher descriptor dimension.
  • The architecture predicts a global descriptor, a dense local descriptor map, and keypoint scores through three heads.
  • Input images are resized to 640×480, converted to grayscale, and augmented with Gaussian noise, motion blur, brightness, and contrast changes.
  • Training applies L2 losses to global and local descriptors and cross-entropy to soft keypoint probabilities, aggregating all three with multitask learning.
  • A keypoint is correct when its reprojection lies within threshold ϵ of a detected keypoint, while a descriptor match additionally requires nearest-neighbor status.
  • HPatches and SfM use dataset-specific keypoint counts and correctness thresholds, with homography or 6-DoF pose evaluation procedures.HPatches uses 300 keypoints and ϵ = 3 pixels; SfM uses 1000 keypoints with thresholds 3 and 5.

B.2. Qualitative Results

The qualitative-results section reports visualizations of detected keypoints and matches on HPatches and SfM, alongside extended 3D-model statistics for three localization datasets.

  • Figures 7 and 8 show detected keypoints and corresponding matches on the HPatches and SfM datasets.
  • Table 6 provides statistics for SIFT- and HF-Net-built 3D models across Aachen Day-Night, RobotCar Seasons, and CMU Seasons.
  • The reported model statistics include track length, and CMU metrics aggregate urban and suburban slices.Some SIFT metrics cannot be computed on CMU because unmatched keypoints were not provided.

C.2. Implementation Details

The hierarchical pipeline reduces both NetVLAD and HF-Net global descriptors to 1024 dimensions with dataset-specific PCA, retrieves ten prior frames, and clusters them before feature extraction under GPU-memory limits.

  • Global descriptors from NetVLAD and HF-Net are independently reduced to 1024 dimensions per dataset using PCA learned on reference images.
  • The pipeline retrieves and clusters a total of 10 prior frames for all datasets.
  • GPU-memory limits require feature extraction on downsampled images.

C.3. Evaluation Process

The evaluation combines benchmark-author-reported results with prior-paper evaluations, while excluding direct SMC comparisons because its assumptions and training data differ. HF-Net uses one trained model across all three datasets without test-data exposure.

  • Benchmark authors evaluated HF-Net and the introduced baselines on all three datasets and generated the main plots.
  • Active Search, City Scale Localization, DenseVLAD, and NetVLAD use evaluations reported in their respective benchmark papers.
  • SMC is not directly compared because it assumes known camera height and uses a semantic segmentation CNN trained on RobotCar’s evaluation dataset.
  • HF-Net encountered no test data during training and was evaluated on all three datasets using the same trained model.

C.4. Qualitative Results

Qualitative HF-Net results are presented for three challenging seasonal or day-night localization datasets, with an additional comparison against NV+SIFT.

  • Figures 9, 10, and 11 show HF-Net results on Aachen Day-Night, RobotCar Seasons, and CMU Seasons, respectively.
  • Figure 12 provides an additional comparison between HF-Net and NV+SIFT.
Loading 1812.03506v2…