PaperlayerView exact revision on arXiv

Source-linked AI summary

MixVPR: Feature Mixing for Visual Place Recognition

Amar Ali-bey, Brahim Chaib-draa, Philippe Giguère

arXiv:2303.02190v1cs.CV

TL;DR

Visual place recognition needs compact descriptors that remain robust to changing appearance while supporting practical retrieval latency. MixVPR uses cascaded all-MLP feature mixing on pretrained-backbone feature maps, and the paper reports broad benchmark gains plus substantially faster two-stage retrieval comparisons.

  • Problem

    VPR must produce descriptors invariant to illumination, weather, seasonal, and other appearance changes while remaining practical for real-world latency.

  • Method

    MixVPR applies cascaded Feature-Mixer MLP blocks to flattened feature maps from a pretrained backbone, then projects them into compact global descriptors.

  • Results

    MixVPR outperforms existing state-of-the-art methods across every tested benchmark and is over 500× faster than Patch-NetVLAD and TransVPR.

  • Takeaways & Limitations

    The reported results support holistic all-MLP aggregation as a lightweight alternative to local, regional, or two-stage VPR retrieval techniques.

Abstract

from arXiv · show

Visual Place Recognition (VPR) is a crucial part of mobile robotics and autonomous driving as well as other computer vision tasks. It refers to the process of identifying a place depicted in a query image using only computer vision. At large scale, repetitive structures, weather and illumination changes pose a real challenge, as appearances can drastically change over time. Along with tackling these challenges, an efficient VPR technique must also be practical in real-world scenarios where latency matters. To address this, we introduce MixVPR, a new holistic feature aggregation technique that takes feature maps from pre-trained backbones as a set of global features. Then, it incorporates a global relationship between elements in each feature map in a cascade of feature mixing, eliminating the need for local or pyramidal aggregation as done in NetVLAD or TransVPR. We demonstrate the effectiveness of our technique through extensive experiments on multiple large-scale benchmarks. Our method outperforms all existing techniques by a large margin while having less than half the number of parameters compared to CosPlace and NetVLAD. We achieve a new all-time high recall@1 score of 94.6% on Pitts250k-test, 88.0% on MapillarySLS, and more importantly, 58.4% on Nordland. Finally, our method outperforms two-stage retrieval techniques such as Patch-NetVLAD, TransVPR and SuperGLUE all while being orders of magnitude faster. Our code and trained models are available at https://github.com/amaralibey/MixVPR.

1. Introduction

Visual place recognition retrieves locations from images, but appearance changes make robust, invariant descriptors difficult. MixVPR addresses this challenge with holistic feature aggregation and reports strong performance across benchmarks.

  • VPR retrieves a query image’s location by matching its compact descriptor against reference images with known geolocations.
  • Illumination, occlusion, weather, seasonal changes, construction, and vegetation make descriptors difficult to keep invariant.
  • Most state-of-the-art methods emphasize local-feature aggregation, although local features can fail under severe illumination and seasonal changes.
  • Region-pooling methods such as MAC, R-MAC, and GeM have generally underperformed NetVLAD for VPR, while CosPlace reports strong benchmark performance.
  • MixVPR uses feature maps from a pretrained backbone and iteratively incorporates global relationships through Feature-Mixer blocks.

2. Related Works

Related VPR work aggregates learned backbone features using local descriptors, regions, or multi-stage retrieval. MixVPR instead uses an all-MLP holistic aggregator without self-attention or regional pooling.

  • NetVLAD assigns local features softly to learned clusters, motivating variants that add contextual, spatial, regional, or pyramid information.
  • MAC, R-MAC, and GeM aggregate regions or pooled activations; CosPlace combines GeM with a linear projection and reports state-of-the-art VPR results.
  • Two-stage retrieval first selects top-k candidates globally, then reranks them using computationally heavy local-feature processing.
  • MixVPR’s architecture uses flattened intermediate feature maps, Feature-Mixer blocks, and projection into a compact global descriptor.
  • Unlike TransVPR and Patch-NetVLAD, MixVPR uses all-MLP aggregation without self-attention or regional feature pooling.

3. Methodology

MixVPR treats CNN activation maps as global features, mixes spatial relationships with cascaded MLP blocks, and projects the result into a compact normalized descriptor.

  • The method extracts intermediate CNN feature maps and treats each activation map as a two-dimensional global feature before flattening it.
  • Feature-Mixer blocks process flattened feature maps through cascaded MLPs that incorporate global relationships into each map.
  • Each MLP uses two fully connected layers, a nonlinearity, and a skip connection that adds the input back to the projection.
  • The cascade preserves the input shape across L Feature-Mixer blocks before dimensionality reduction.
  • Two successive fully connected projections reduce channel depth and spatial rows, acting as weighted pooling that controls descriptor size.
  • The final d×r output is flattened and L2-normalized, while matrix multiplications avoid self-attention’s quadratic complexity and intermediate cropping reduces parameters.

4. Experiments

The experiments evaluate MixVPR against existing VPR methods on multiple challenging benchmarks using comparisons, implementation details, and ablation studies.

  • The evaluation covers implementation details, datasets, metrics, performance comparisons, and ablation studies across multiple challenging benchmarks.

4.1. Implementation details

MixVPR is evaluated alongside established aggregation methods using controlled backbone and dataset settings, with implementation choices differing in backbone cropping.

  • MixVPR uses a backbone cropped in the middle, whereas comparison techniques use the last convolutional layer as recommended by their authors.
  • All six techniques are trained on the same dataset using the same ResNet-50 backbone for fair comparison.
  • The reported retrieval criterion counts a reference as correct when it lies within d = 25 meters of the query.

4.2. Comparison to the state of the art

Across four challenging VPR benchmarks, MixVPR outperforms existing techniques by large margins, including under viewpoint, seasonal, illumination, and severe appearance changes.

  • 94.6% recall@1 on Pitts250k-test is 3.1 percentage points above CosPlace and over 4.1 points above NetVLAD.
  • 88.0% recall@1 on MSLS is 3.5 points above CosPlace and 5.4 points above NetVLAD.
  • 85.2% recall@1 on SPED exceeds NetVLAD by 7.5 points under seasonal and day-night illumination changes.
  • 58.4% recall@1 on Nordland represents 69% and 79% relative improvements over CosPlace and NetVLAD, respectively.
  • MixVPR obtains by far the best performance on all benchmarks with big margins in the controlled comparison.

4.3. Comparing against two-stage techniques

MixVPR is compared with two-stage retrieval methods that rerank candidates using local-feature matching, emphasizing the latency cost of refinement.

  • Two-stage methods first retrieve M candidates globally, then rerank them through geometric verification on local features.
  • TransVPR takes 45 milliseconds as the second-fastest method, while re-ranking makes such techniques unusable for real-time applications.

4.4. Ablation studies

The ablations examine Feature-Mixer depth, descriptor dimensionality, and backbone choices. Results indicate that MixVPR maintains strong performance across configurations while enabling substantial parameter and computation reductions.

  • Descriptor dimensionality: MixVPR outperforms all other techniques across the tested descriptor dimensionality configurations on Pitts30k-test.NetVLAD, GeM, and AVG use PCA reduction, whereas CosPlace and MixVPR are trained for each output-dimensionality configuration.
  • Backbone choice: Cropping the backbone at the fourth residual layer halves backbone parameters, accelerating computation and reducing memory use.With ResNet-18, MixVPR uses 3.5M parameters, reported as 15% of CosPlace or NetVLAD, while retaining competitive results.

4.5. Qualitative Results

Qualitative retrieval examples show MixVPR succeeding on challenging queries involving repetitive structures, viewpoint changes, skylines, illumination changes, and occlusions. Learned Feature-Mixer weights select both localized regions and whole inputs.

  • Challenging retrieval scenarios: MixVPR retrieves the correct references in five challenging scenarios where other techniques struggle, including repetitive structures, viewpoint changes, skylines, illumination changes, and occlusions.The Figure 4 caption describes these comparisons on MSLS and Pitts30k and reports that other techniques fail on all shown challenging queries.
  • Repetitive structures: Only MixVPR retrieved the right references for repetitive structures, avoiding visually similar images from different places.Repetitive layouts or textures can produce false positives for other VPR techniques.
  • Viewpoint change: Only MixVPR retrieved the correct references under the illustrated extreme viewpoint changes.The examples contrast MixVPR with techniques that focus on local features, such as NetVLAD.
  • Learned weights: Feature-Mixer neurons learn regional selections ranging from small image spots to the entire input.The illustrated subset contains 24 of 400 neurons, with weights reshaped to 20×20; the authors suggest their combination can replace attention and pyramidal schemes.
  • Illumination change: Under night-to-day illumination changes, MixVPR retrieved the correct reference while CosPlace, NetVLAD, and Gated NetVLAD retrieved nighttime locations.The authors identify illumination variation as an important aspect of robust VPR and describe the example as difficult even for human observers.

5. Conclusion

MixVPR is an all-MLP aggregation technique that mixes feature maps from pretrained networks to learn robust representations. Across tested benchmarks, it outperforms existing methods and is reported as over 500× faster than two-stage retrieval approaches.

  • Method: MixVPR uses stacked Feature-Mixer blocks to incorporate global relationships between individual feature maps in a cascade of feature mixing.The method is an all-MLP aggregation technique operating on feature maps from pretrained networks.
  • Results: MixVPR outperforms existing state-of-the-art methods by a wide margin on every tested benchmark.The conclusion also reports ablation studies validating the feature-mixing design.
  • Efficiency: Over 500× faster than Patch-NetVLAD and TransVPR, MixVPR is superior to the compared two-stage retrieval techniques.The comparison includes Patch-NetVLAD and TransVPR as examples of two-stage retrieval methods.
Loading 2303.02190v1…