Source-linked AI summary
EigenPlaces: Training Viewpoint Robust Models for Visual Place Recognition
Gabriele Berton, Gabriele Trivigno, Barbara Caputo, Carlo Masone
TL;DR
Visual Place Recognition must retrieve the same place despite substantial viewpoint changes, which can defeat existing retrieval systems. EigenPlaces trains descriptors using classes that contain multiple viewpoints of inferred points of interest, and broad benchmarks report state-of-the-art performance on many datasets with lower resource requirements. Its point-of-interest construction can misplace points in settings such as crossroads, although that variability is reported as potentially useful for robustness.
Problem
Heavy viewpoint shifts can prevent state-of-the-art retrieval systems from finding relevant database images among their highest-ranked candidates.
Method
EigenPlaces clusters training data into classes containing multiple viewpoints of the same scene, using geographical principal components to estimate points of interest.
Results
EigenPlaces outperforms previous state of the art on numerous datasets while using 50% smaller descriptors and requiring 60% less GPU memory for training.
Takeaways & Limitations
Viewpoint robustness is embedded at the retrieval stage, making the learned global descriptors better suited to perspective changes.
Takeaways & Limitations
At crossroads, estimated eigenvectors may place points of interest away from buildings, although the resulting variability may improve robustness.
Abstract
from arXiv · showhide
Visual Place Recognition is a task that aims to predict the place of an image (called query) based solely on its visual features. This is typically done through image retrieval, where the query is matched to the most similar images from a large database of geotagged photos, using learned global descriptors. A major challenge in this task is recognizing places seen from different viewpoints. To overcome this limitation, we propose a new method, called EigenPlaces, to train our neural network on images from different point of views, which embeds viewpoint robustness into the learned global descriptors. The underlying idea is to cluster the training data so as to explicitly present the model with different views of the same points of interest. The selection of this points of interest is done without the need for extra supervision. We then present experiments on the most comprehensive set of datasets in literature, finding that EigenPlaces is able to outperform previous state of the art on the majority of datasets, while requiring 60\% less GPU memory for training and using 50\% smaller descriptors. The code and trained models for EigenPlaces are available at {\small{\url{https://github.com/gmberton/EigenPlaces}}}, while results with any other baseline can be computed with the codebase at {\small{\url{https://github.com/gmberton/auto_VPR}}}.
1 Politecnico di Torino
The passage provides an institutional contact email for the paper.
- The listed address uses the polito.it institutional domain.
- The contact address begins with multiple author names.
1. Introduction
The introduction frames viewpoint shifts as a persistent VPR challenge and presents EigenPlaces as a training strategy that learns viewpoint-robust descriptors. Across broad benchmarks, it reports stronger results on many datasets with substantially lower training memory and descriptor size.
- Motivation: Visual Place Recognition predicts where a query photo was taken from visual features using retrieval against georeferenced images.
- Motivation: Heavy viewpoint shifts remain difficult because modern retrieval systems may fail to place any relevant database image among the highest-ranked candidates.
- Approach: EigenPlaces clusters training images so each class contains multiple viewpoints of the same scene, encouraging viewpoint-robust global descriptors.
- Approach: EigenPlaces estimates places from training-data geography and finds principal components within each class without extra supervision.
- Results: EigenPlaces outperforms previous state of the art on numerous datasets while using 50% smaller descriptors and requiring 60% less GPU memory for training.
- Evaluation: The work contributes a broad VPR benchmark intended to expose both strengths and weaknesses across EigenPlaces and prior methods.
2. Related Work
Prior VPR methods use local or learned global features, but their training protocols generally do not explicitly address viewpoint invariance. Separate local-feature methods can refine retrieved candidates, yet depend on retrieval first producing a positive match.
- Visual place recognition: Early VPR methods matched queries with local features, while deep learning enabled CNN-derived global features for landmark retrieval.
- Visual place recognition: NetVLAD-based methods use similar database images as positives, while MixVPR and CosPlace group training images with similar or identical orientations.
- Viewpoint invariant matching: Local-feature refinement methods seek viewpoint-invariant landmark matching after retrieval supplies a shortlist of candidates.
- Viewpoint invariant matching: These refinement methods assume the retrieved shortlist contains at least one positive match and are computationally expensive, making robust initial retrieval important.
3. Method
EigenPlaces constructs training classes that represent the same place from different viewpoints, using geographical structure within map cells without extra supervision. It combines lateral and frontal viewpoint-oriented losses to train descriptors robust to viewpoint shifts.
- Class construction: The map is partitioned into 15-meter cells and non-neighboring cells are grouped into subsets so each subset can serve as a classification task without visual overlap.This follows a CosPlace-inspired spatial split.
- Class construction: EigenPlaces addresses viewpoint variability by selecting images that face the same place from different perspectives within each training cell.The method uses geographical image distributions rather than annotations of buildings or other points of interest.
- Focal-point selection: SVD of centered image coordinates identifies a road-aligned first component and a perpendicular second component used to place a focal point toward likely points of interest.The focal point is c_i = E[X_i] + D × V_1, where D controls its distance from the cell’s center of mass.
- Loss design: EigenPlaces adds a second frontal focal point for datasets collected mainly with front-facing cameras, complementing the lateral construction.The frontal procedure uses the road-aligned first eigenvector, while the lateral procedure uses the second component.
- Focal-point selection: Images are grouped by orientations close to the angle from each image toward the focal point, enabling one class to contain multiple views of the same place.The angle varies across images because it depends on their relative positions and the focal point.
- Loss design: The network is trained end-to-end with CosFace using separate lateral and frontal classifiers and corresponding loss components.An ablation reports that lateral loss is sufficient for satisfactory multi-view results but fails to produce robust frontal-view embeddings.
4. Experiments
The experiments evaluate EigenPlaces across diverse multi-view and frontal-view datasets, using a simple, configurable architecture and extensive baseline comparisons. Results show strong viewpoint robustness, competitive performance across dataset types, and substantial efficiency advantages, while ablations clarify the roles of the two loss components and focal distance.
- 4.2. Implementation details: The architecture uses a standard VGG-16 or ResNet-50 backbone, GeM pooling, and a fully connected layer whose width determines descriptor dimensionality.This design is simpler than NetVLAD- and MLP-Mixer-based alternatives and makes descriptor size straightforward to change.
- 4.2. Implementation details: EigenPlaces is trained for 200k iterations with 128-image batches, Adam, a 1e−5 learning rate, SFRS augmentations, 15-meter cells, N = 3, and focal distance D = 10 meters.Experiments also report that D = 20 meters achieves higher average results on several datasets.
- 4.3. Comparison with previous work: EigenPlaces achieves the best overall performance on frontal-view datasets, although it uses descriptors twice as large.Newer compact-descriptor methods are generally robust, while older NetVLAD and SFRS models are less robust to domain shifts.
- 4.4. Efficiency: EigenPlaces trains with less than 7 GB of GPU memory, versus more than 18 GB for MixVPR, while its top model is twice as fast and uses half the descriptor memory.Descriptor dimensionality is linearly related to retrieval memory footprint and nearest-neighbor matching time.
- 4.5. Ablations: Combining the lateral and frontal-view losses provides robust performance across datasets, whereas either component alone favors different dataset types.The lateral loss performs well on multi-view datasets but fails on frontal-view datasets; the frontal-view loss shows the opposite pattern.
- 4.5. Ablations: A focal distance of 0 meters performs better than expected despite including images facing opposite directions.The analysis also examines focal points extending from cell centers toward likely building facades.
- 4.5. Ablations: EigenPlaces maintains higher embedding similarity across large viewpoint changes than CosPlace and MixVPR in the confusion-matrix analysis.Images are ordered along the first principal component, and similarity is evaluated among views oriented toward the focal point.
5. Conclusions
EigenPlaces trains VPR models to recognize the same point of interest from multiple viewpoints using data-derived classes and a viewpoint-focused loss. Across diverse datasets, it achieves state-of-the-art results in most cases while using lighter descriptors.
- 5. Conclusions: EigenPlaces divides the map into fine-grained cells, infers points of interest from the data, and forms classes containing views of those points from multiple perspectives.The method uses a loss that encourages recognition of the same point across viewpoints without extra supervision.
- 5. Conclusions: EigenPlaces is designed to address perspective shifts directly in the learned global descriptors rather than relying only on later retrieval refinement.Its training classes are constructed from geographic data to expose the model to viewpoint variation.
- 5. Conclusions: The method achieves state-of-the-art results in the majority of evaluated cases while using lighter descriptors than previous works.The conclusion emphasizes performance across datasets with diverse characteristics and challenges.
A. Data for Different Training Methods
Existing training methods generally provide little viewpoint variation, whereas EigenPlaces explicitly constructs training data with large viewpoint shifts to improve viewpoint robustness.
- NetVLAD query-positive pairs have very little viewpoint shift because positives are selected as the most similar database images.This positive-mining strategy is also used by most subsequent methods listed in the passage.
- CosPlace groups images with the same orientation and small spatial separation, while Conv-AP and MixVPR use classes with little intraclass viewpoint variation.
- EigenPlaces instead creates training data with large viewpoint shifts between images, unlike the preceding methods.The paper presents this construction as making the trained model more robust.
B. Datasets
The evaluation spans diverse visual place-recognition datasets covering viewpoint, temporal, modality, weather, illumination, and domain changes, with figures illustrating their query and database imagery.
- The benchmark uses many datasets to examine the strengths and weaknesses of different methods across varied conditions.The authors emphasize broad evaluation and reproducible downloading for several datasets.
- AmsterTime pairs historical grayscale queries with modern reference images, combining viewpoint, temporal, modality, and camera changes.
- SF-XL covers San Francisco with over 41M database images and uses challenging Flickr queries plus queries shared with San Francisco Landmark.
- MSLS contains more than 1M images from multiple cities, with evaluation performed on its validation set because test labels are unavailable.
- Nordland uses winter queries and summer database images from the same train route, with localization accepted within 10 frames.
- SVOX evaluates cross-domain retrieval using a Google StreetView database and RobotCar queries spanning overcast, rainy, sunny, snowy, and night conditions.
C.1. Further results
Further results report recall values for the same datasets as the main paper, separating multi-view from frontal-view evaluations.
- Recall results are reported separately for multi-view and frontal-view datasets.
C.2. Qualitative results
The qualitative evaluation uses example retrievals and recall tables to examine EigenPlaces across multi-view and frontal-view settings. Visual examples indicate stronger handling of difficult viewpoints, including sidewalk photographs.
- The qualitative examples show EigenPlaces handling difficult viewpoints, including photos taken from the sidewalk.
- Tables 7 and 8 report recalls as R@1, R@5, R@10, and R@20 for multi-view and frontal-view datasets, respectively.Results are split by backbone and descriptor dimension, with overall and within-group best results marked separately.
- EigenPlaces qualitative results include queries and their first predicted database images for comparison with popular methods.Each figure column represents one query and its top prediction.