PaperlayerView exact revision on arXiv

Source-linked AI summary

Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition

Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, Tobias Fischer

arXiv:2103.01486v1cs.CV

TL;DR

Visual place recognition must remain reliable despite appearance and viewpoint changes, while existing local and global descriptors offer complementary strengths. Patch-NetVLAD derives locally-global patch descriptors from NetVLAD residuals, fuses multiple patch sizes, and matches them for retrieval. Across key benchmarks, it reports higher average performance than both original NetVLAD and a SuperPoint/SuperGlue VPR pipeline, with configurable faster variants.

  • Problem

    Visual place recognition is difficult under appearance, illumination, and viewpoint changes, while local and global descriptor methods have complementary strengths that prior work has rarely combined.

  • Method

    Patch-NetVLAD extracts locally-global descriptors from feature-space patches, matches them across multiple scales, and uses integral feature computation for multi-scale fusion.

  • Results

    Patch-NetVLAD’s average performance across key benchmarks is 17.5% higher than original NetVLAD and 3.1% higher absolute recall than the SuperPoint and SuperGlue-enabled VPR pipeline.

  • Takeaways & Limitations

    The configurable system provides performance-focused and compute-constrained options, including rapid spatial scoring with small reported performance reductions.

  • Takeaways & Limitations

    The paper identifies image-space patch crops as computationally prohibitive because overlapping patches require separate VGG forward passes and are important for viewpoint invariance.

Abstract

from arXiv · show

Visual Place Recognition is a challenging task for robotics and autonomous systems, which must deal with the twin problems of appearance and viewpoint change in an always changing world. This paper introduces Patch-NetVLAD, which provides a novel formulation for combining the advantages of both local and global descriptor methods by deriving patch-level features from NetVLAD residuals. Unlike the fixed spatial neighborhood regime of existing local keypoint features, our method enables aggregation and matching of deep-learned local features defined over the feature-space grid. We further introduce a multi-scale fusion of patch features that have complementary scales (i.e. patch sizes) via an integral feature space and show that the fused features are highly invariant to both condition (season, structure, and illumination) and viewpoint (translation and rotation) changes. Patch-NetVLAD outperforms both global and local feature descriptor-based methods with comparable compute, achieving state-of-the-art visual place recognition results on a range of challenging real-world datasets, including winning the Facebook Mapillary Visual Place Recognition Challenge at ECCV2020. It is also adaptable to user requirements, with a speed-optimised version operating over an order of magnitude faster than the state-of-the-art. By combining superior performance with improved computational efficiency in a configurable framework, Patch-NetVLAD is well suited to enhance both stand-alone place recognition capabilities and the overall performance of SLAM systems.

1. Introduction

Patch-NetVLAD addresses visual place recognition under appearance and viewpoint changes by combining locally-global patch descriptors with multi-scale fusion. It is evaluated across challenging datasets against global and local descriptor baselines.

  • Visual Place Recognition supports stand-alone positioning and SLAM but is challenged by changes in appearance, illumination, and viewpoint.
  • Patch-NetVLAD combines local and global approaches by matching densely sampled locally-global descriptors extracted from feature-space patches.
  • Multi-scale fusion combines hybrid descriptors from different patch sizes using an integral feature space to limit computational growth.
  • Patch-NetVLAD is evaluated on diverse datasets capturing viewpoint and appearance changes, including variations in acquisition method, time, weather, and season.
  • Patch-NetVLAD outperforms global descriptor methods by 6% to 330% relative increase across all evaluated datasets.

2. Related Work

Related work spans global image descriptors, local keypoint features, and region or patch descriptors. Patch-NetVLAD differs by performing multi-scale fusion at the final scoring stage rather than fusing descriptors earlier.

  • Global Image Descriptors: Global image descriptors aggregate local or dense image features using methods including BoW, Fisher Vectors, VLAD, and learned variants such as NetVLAD.
  • Local Keypoint Descriptors: Local keypoint descriptors support place recognition, visual SLAM, and 6-DoF localization, but many learning-based methods target keypoint-level pose estimation.
  • Local Region/Patch Descriptors: Prior region and patch approaches represent image regions using CNN activations, region matching, attention, or separately aggregated VLAD representations.
  • Local Region/Patch Descriptors: Patch-NetVLAD generates region-level VLAD descriptors through NetVLAD, reusing VPR-relevant learned cluster membership rather than separately aggregating region features.
  • Multi-Scale Approaches: Unlike descriptor-level fusion, Patch-NetVLAD fuses multiple scales at the final scoring stage, enabling parallel processing with associated speed benefits.

3. Methodology

Patch-NetVLAD first retrieves candidates with NetVLAD, then extracts and matches locally-global patch descriptors across scales, using spatial consistency to refine image retrievals. Integral feature computation and rapid scoring support configurable efficiency-performance trade-offs.

  • Pipeline: The hierarchical pipeline uses NetVLAD to retrieve top-k candidates, then reorders them using locally-global patch-descriptor matching.The experiments use k = 100 initial candidates.
  • Original NetVLAD Architecture: NetVLAD aggregates CNN feature maps into a K × D matrix using soft cluster assignments and residuals relative to cluster centers.
  • Patch-Level Global Features: Patch descriptors apply NetVLAD aggregation and projection to densely sampled feature-space patches rather than aggregating the entire feature map globally.
  • Patch-Level Global Features: Patch features are sampled with stride sp, and patch coordinates identify each patch center within the feature map.
  • Patch-Level Global Features: Patch descriptors implicitly contain scene semantics because each covers a larger area than comparatively small local-feature regions.
  • Mutual Nearest Neighbours: Mutual nearest-neighbor matching exhaustively compares query and reference patch descriptors to obtain matched descriptor pairs.
  • Spatial Scoring: Spatial scoring compares matched patch locations, using RANSAC homography inliers or a faster direct score computed from matched feature pairs.
  • Multiple Patch Sizes: Multiple patch sizes are fused by taking a convex combination of their spatial matching scores, with nonnegative weights summing to one.

4. Experimental Results

Patch-NetVLAD is evaluated across challenging place-recognition datasets, benchmark methods, and configurable accuracy–compute trade-offs. Multi-scale fusion improves performance, while rapid-scoring and speed-oriented variants reduce computation with limited recall loss.

  • Evaluation setup: Patch-NetVLAD is evaluated on six benchmark datasets spanning substantial viewpoint and appearance changes, including weather, illumination, time-of-day, and seasonal variation.The datasets include Nordland, Pittsburgh, Tokyo24/7, Mapillary Streets, RobotCar Seasons v2, and Extended CMU Seasons.
  • Evaluation setup: The evaluation compares Patch-NetVLAD with AP-GEM, DenseVLAD, NetVLAD, and SuperGlue using recall-based localization metrics and dataset-specific error tolerances.Recall@N is used for most datasets; RobotCar Seasons v2 and Extended CMU Seasons use default translational and rotational error thresholds.
  • Comparison with state-of-the-art methods: 17.5%, 14.8%, and 22.3% absolute R@1 gains over NetVLAD, DenseVLAD, and AP-GEM respectively are reported on average across the evaluated datasets.The largest differences occur on datasets with substantial appearance variation, including Nordland, Extended CMU Seasons, Tokyo24/7, RobotCar Seasons, and Mapillary.
  • Comparison with state-of-the-art methods: 15.8% absolute performance improvement over SuperGlue is reported on Nordland, although SuperGlue leads slightly on selected Tokyo24/7 and Pittsburgh recall metrics.On Mapillary, the Patch-NetVLAD–SuperGlue difference increases from 1.1% at R@1 to 5.1% at R@25.
  • Comparison with state-of-the-art methods: 13.0% absolute R@1 improvement over NetVLAD is reported on the withheld Mapillary test set, where Patch-NetVLAD won the ECCV2020 Mapillary Challenge.The passage also reports no fine-tuning on any dataset.
  • Ablation studies: 2.0% average R@1 improvement from multi-scale fusion is reported over single-scale RANSAC scoring, while rapid spatial verification is 2.9–3.1 times faster with 0.6–1.1% degradation.The single-scale and multi-scale rapid-verification variants provide explicit accuracy–compute alternatives.
  • Ablation studies: 15 times faster processing than SuperGlue is achieved by the speed-oriented configuration, at the expense of 0.6% recall relative to SuperGlue and 1.7% relative to the performance-focused configuration.The balanced configuration is more than three times faster than SuperGlue with comparable performance, while the storage-focused variant largely outperforms NetVLAD with similar memory requirements to a SIFT-like descriptor.
  • Ablation studies: Patch-NetVLAD remains robust to patch-size selection and to the convex combination of patch sizes used for multi-patch fusion.The reported single-patch peak occurs at dp = 4, while the multi-patch results show robustness to fusion choices.

5. Discussion and Conclusion

Patch-NetVLAD combines global descriptor techniques with local patch features and multi-scale fusion to improve place-recognition robustness and performance. The reported results show gains over NetVLAD and SuperPoint/SuperGlue, while qualitative examples include difficult condition and viewpoint changes.

  • Patch-NetVLAD is a locally-global feature descriptor that applies global descriptor techniques to improve the appearance robustness of local descriptors.It considers visual content within larger image patches rather than relying on prior keypoint-based local descriptors.
  • Patch-NetVLAD retrieved the correct reference in a Tokyo 24/7 example involving day-night change, severe viewpoint shift, and occlusions, unlike NetVLAD and SuperGlue.
  • 17.5% higher average performance than original NetVLAD was achieved across key benchmarks.
  • 3.1% absolute recall improvement over the SuperPoint and SuperGlue-enabled VPR pipeline was reported.
  • Fusing multiple patch sizes produced higher recall than any single patch size with minimal computational penalty compared to single-scale techniques.Performance remained high across a broad range of increasing patch-size combinations.
  • A potential extension is replacing mutual-nearest-neighbor matching with a learned matcher using Patch-NetVLAD features.The paper suggests this could improve global re-localization performance and robustness when combined with outlier rejection.

for Place Recognition

The paper is authored by Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, and Tobias Fischer at QUT Centre for Robotics, Queensland University of Technology.

  • Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, and Tobias Fischer are listed as authors.
  • The authors are affiliated with the QUT Centre for Robotics, Queensland University of Technology.
  • The paper is identified as arXiv:2103.01486v1, dated 2 March 2021.

Overview

The supplementary material provides additional condition-specific results, dataset evaluations, computation-time analyses, and ablation studies.

  • Section 1 reports RobotCar Seasons v2 and Extended CMU Seasons results split by query condition.
  • Section 2 adds quantitative results for Pittsburgh 30k and Tokyo 24/7 and separates computation time into feature extraction and matching.
  • Section 3 contains additional ablation studies, including studies that further demonstrate the reported findings.

1. Results Split by Condition on RobotCar Seasons v2 and Extended CMU Seasons

The supplementary results examine RobotCar Seasons v2 and Extended CMU Seasons by query condition and error threshold. Patch-NetVLAD generally leads the comparisons, with limited exceptions at looser thresholds or particular environments.

  • RobotCar Seasons v2: RobotCar Seasons v2 main-paper results are summary statistics weighted by the number of images in each condition.
  • RobotCar Seasons v2: 1.3% absolute recall improvement over SuperGlue was reported at .25m translational and 2 degrees orientation error thresholds.
  • RobotCar Seasons v2: SuperGlue slightly outperformed Patch-NetVLAD in some night traverses at looser error thresholds.
  • A learned matcher using Patch-NetVLAD features is proposed as a future direction for potentially more robust matching.The suggested matcher would be similar to SuperGlue and could be combined with outlier rejection.
  • Extended CMU Seasons: Patch-NetVLAD consistently outperformed all comparison methods across Extended CMU Seasons conditions and error thresholds, except Park at the largest error threshold.SuperGlue was slightly better in that single Park case.

2. Additional Quantitative Results

Additional results provide recall plots for Pittsburgh and Tokyo 24/7, and separate computational-time analyses for Mapillary.

  • Supplementary Figure 1 reports Recall@N performance on Pittsburgh 30k and Tokyo 24/7.
  • Supplementary Figure 2 separates Mapillary query processing time into feature extraction, feature matching, and combined time.

3. Further Ablation Studies

The ablations examine fusion choices, patch-size matching, pooling strategies, and feature-space versus image-space patch processing. Results favor post-processing fusion, same-size matching, VLAD pooling, and overlapping feature-space patches.

  • Ablation of Multi-Scale Fusion Weights and Patch Sizes: Fusion weights and patch sizes are evaluated for robustness on Mapillary, with weights selected by grid search on RobotCar Seasons v2 training data.
  • Ablation of Multi-Scale Fusion Weights and Patch Sizes: Increasing fused patch sizes from three to five does not improve recall beyond the best three-size combination.The reported inference is that the span between the smallest and largest patch sizes matters more than the number of sizes.
  • Early Match Fusion: 79.5% R@1, 86.2% R@5, and 87.7% R@10 from post-processing fusion exceed early fusion on Mapillary.Early fusion achieves 77.2% R@1, 85.3% R@5, and 87.3% R@10.
  • Other Pooling Strategies: Patch-level VLAD pooling reaches 79.5% R@1 on Mapillary, versus 73.6% for average pooling and 74.5% for max-pooling.The comparison keeps other settings equal and uses vanilla NetVLAD as a 60.8% R@1 reference.
  • Patch Crops in the Image Space Instead of Feature Space: Overlapping feature-space patches are retained because they are crucial for viewpoint invariance, whereas image-space patch crops impose prohibitive compute and storage costs.Each image-space patch crop would require a separate VGG forward pass.
  • Matching Across Different Patch Sizes: 79.5% R@1 with same-size matching falls to 78.1% when patches of different sizes are matched.The proposed method matches patches with other patches of the same size.

4. Additional Qualitative Results

Qualitative results show Patch-NetVLAD successes and failures across several datasets, including cases where patch scale provides complementary correspondences and where competing methods fail.

  • Supplementary figures show correct and incorrect matches for Mapillary, Nordland, Pittsburgh, and Tokyo 24/7, including Patch-NetVLAD successes against failed methods.SuperGlue also finds correct matches in some cases where Patch-NetVLAD does not localize correctly.
  • Patch-NetVLAD often succeeds on Pittsburgh images with substantial sky because larger patches include clouds together with ground-level features.SuperGlue may instead match points corresponding to clouds.
  • Small, medium, and large patches provide complementary correspondences, with small patches helping near boundaries and larger patches supplying broader contextual cues.Supplementary Figure 3 uses dp = 2, 5, and 8 and includes examples where all scales match distinct image regions.
  • Supplementary Figure 8 contains cases where all methods fail, while Supplementary Figure 9 includes Patch-NetVLAD matches judged outside the error tolerance.The latter cases suggest tight tolerances or slight ground-truth errors.

5. Detailed Dataset Description

The evaluation spans six benchmark datasets covering diverse viewpoint, appearance, acquisition, and geographic conditions. Each dataset contributes distinct environmental or imaging challenges to the assessment.

  • Nordland: Nordland provides a 728 km train traverse recorded in four seasons, with summer references and winter queries selected for high appearance dissimilarity.The video is subsampled at 1 fps across the entire traverse.
  • RobotCar Seasons v2: RobotCar Seasons v2 uses overcast reference images and queries spanning dawn, dusk, sun, rain, seasonal, snow, night, and night-rain conditions.The dataset mainly captures appearance changes, while viewpoint changes are relatively minor.
  • Extended CMU Seasons: Extended CMU Seasons covers an 8.8 km Pittsburgh route with multiple seasonal query traverses over one year but no nighttime images.
  • Pittsburgh: Pittsburgh uses Street View images captured at different times and years, with 24 perspectives per place producing substantial viewpoint and appearance variation.The evaluated Pitts 30k subset has 10,000 reference images and 6,816 query images.
  • Tokyo 24/7: Tokyo 24/7 contains 125 locations photographed by smartphones from three viewing directions and at three times of day, including nighttime images.
  • Mapillary Street Level Sequences: Mapillary contains over 1.6 million images from 30 major cities across seven years, with geographic, seasonal, temporal, viewpoint, and weather variation.The evaluation uses the image-to-image task and distinguishes public validation from withheld test data.
Loading 2103.01486v1…