PaperlayerView exact revision on arXiv

Source-linked AI summary

VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change

Mubariz Zaffar, Sourav Garg, Michael Milford, Julian Kooij, David Flynn, Klaus McDonald-Maier, Shoaib Ehsan

arXiv:2005.08135v2cs.CV

TL;DR

VPR evaluation is fragmented across communities, with inconsistent metrics and largely qualitative treatment of viewpoint and illumination invariance. The paper introduces VPR-Bench, an open framework integrating broad datasets, techniques, metrics, and quantified variation analysis. Results show that performance is application- and metric-dependent, with no universal state-of-the-art technique and distinct effects from viewpoint and illumination changes.

  • Problem

    VPR evaluation lacks standardization across computer vision and robotics, while viewpoint and illumination invariance have often been assessed qualitatively and ambiguously.

  • Method

    VPR-Bench integrates 12 datasets, 10 techniques, cross-community evaluation metrics, and variation-quantified datasets to assess performance and invariance.

  • Results

    No universal state-of-the-art technique exists: 8 of 10 techniques achieve state-of-the-art performance on at least one dataset, while metric utility depends on application requirements.

  • Takeaways & Limitations

    VPR evaluation should consider application-dependent metrics and quantified variation, while ensemble-based approaches may offer value for generic VPR systems.

  • Takeaways & Limitations

    The Point Features dataset uses synthetic scenes, while real-world variation-quantified datasets provide less detailed quantification.

Abstract

from arXiv · show

Visual Place Recognition (VPR) is the process of recognising a previously visited place using visual information, often under varying appearance conditions and viewpoint changes and with computational constraints. VPR is related to the concepts of localisation, loop closure, image retrieval and is a critical component of many autonomous navigation systems ranging from autonomous vehicles to drones and computer vision systems. While the concept of place recognition has been around for many years, VPR research has grown rapidly as a field over the past decade due to improving camera hardware and its potential for deep learning-based techniques, and has become a widely studied topic in both the computer vision and robotics communities. This growth however has led to fragmentation and a lack of standardisation in the field, especially concerning performance evaluation. Moreover, the notion of viewpoint and illumination invariance of VPR techniques has largely been assessed qualitatively and hence ambiguously in the past. In this paper, we address these gaps through a new comprehensive open-source framework for assessing the performance of VPR techniques, dubbed "VPR-Bench". VPR-Bench (Open-sourced at: https://github.com/MubarizZaffar/VPR-Bench) introduces two much-needed capabilities for VPR researchers: firstly, it contains a benchmark of 12 fully-integrated datasets and 10 VPR techniques, and secondly, it integrates a comprehensive variation-quantified dataset for quantifying viewpoint and illumination invariance. We apply and analyse popular evaluation metrics for VPR from both the computer vision and robotics communities, and discuss how these different metrics complement and/or replace each other, depending upon the underlying applications and system requirements.

1 Introduction

VPR research spans computer vision and robotics applications, but differing datasets, metrics, and requirements have produced fragmentation and inconsistent state-of-the-art results. The paper addresses this with an open, integrated evaluation framework that quantifies viewpoint and illumination variation.

  • Motivation: VPR matches previously visited places from visual input despite perceptual aliasing and seasonal, illumination, and viewpoint changes.Applications include SLAM loop closure, image search, vehicular navigation, aerial asset management, and 3D-model creation.
  • Research gap: Computer vision and robotics communities remain separated, and state-of-the-art performance is not temporally consistent across techniques and datasets.The paper attributes this divide partly to differing application requirements, datasets, and evaluation metrics.
  • Research gap: Viewpoint and illumination invariance have commonly been assessed using subjective categories such as ‘mild’, ‘moderate’, ‘high’ and ‘extreme’.The paper identifies viewpoint and illumination as variations that can be modelled with quantitative metrics, unlike seasonal variation, which is difficult to quantify.
  • Contributions: VPR-Bench combines techniques, datasets, and evaluation metrics from both communities in a systematic, open-source framework covering diverse environments and variation levels.The framework re-implements techniques, standardizes datasets and ground truths, and provides a common strategy for future evaluations.
  • Contributions: The framework quantifies viewpoint and illumination invariance using the Point Features dataset and extends the analysis to QUT multi-lane and MIT multi-illumination datasets.It also supports analyses of metrics, ground-truth manipulation, hardware trade-offs, descriptor size, retrieval time, and viewpoint variance.

2 Literature Review

The literature treats VPR as visual retrieval under changing viewpoints and conditions, using handcrafted, global, local, region-focused, semantic, sequence-based, and deep-learning approaches. Evaluation spans direct retrieval metrics and indirect system-level metrics, while existing benchmarks emphasize different application settings.

  • VPR approaches: VPR generally retrieves a correct reference match among top-ranked candidates for a query image under varying viewpoints and conditions.Local-feature matching can be added for more accurate localization at increased computational cost.
  • VPR approaches: Handcrafted VPR approaches include local descriptors such as SIFT and SURF, global descriptors such as Gist, and sequence matching such as Seq-SLAM.These methods span feature-level, whole-image, and sequential matching strategies.
  • VPR approaches: Deep-learning methods use CNN-derived representations and have improved invariance to conditional changes in VPR.The review also covers region-of-interest methods, semantic segmentation, object proposals, and synthetic views.
  • Benchmarks: Existing visual-localisation benchmarks and associated metrics often focus on distinctive landmark datasets rather than continuous, varied traversals central to robotics-oriented VPR.VPR-Bench focuses on direct VPR evaluation and integrates techniques and descriptors through a modular framework.
  • Evaluation: Direct metrics evaluate retrieved images, whereas indirect metrics evaluate a larger pipeline in which VPR is only one component.Examples of indirect metrics include Absolute-Trajectory-Error and Relative-Pose-Error in visual-SLAM.

3 VPR-Bench Framework

VPR-Bench is a modular framework for evaluating VPR tasks, datasets, techniques, metrics, and invariance quantification. It integrates broad indoor and outdoor coverage with variation-quantified datasets for analysing viewpoint and illumination effects.

  • Framework overview: VPR-Bench introduces an evaluation framework covering task formulation, datasets, techniques, evaluation metrics, and invariance quantification.These components define the framework’s evaluation setup.

3.1 VPR Task Formulation

The paper formalizes a VPR system as computing query and map features, performing VPR matching, and storing the resulting outputs. The formulation accommodates images, sequences, and pre- or post-processed query candidates.

  • System formulation: A VPR system receives a query Q and reference map M_R, computes query and map descriptors F_Q and F_M, and returns P, S, and C.The formulation includes ROI locations, descriptors, and salience within the query features when ROI extraction is used.
  • Generic algorithm: The generic algorithm separates query-description, map-feature, and VPR-processing functions, each with preprocessing and postprocessing steps.The main procedure stores the returned P, S, and C outputs.

3.2 Evaluation Datasets

VPR-Bench combines diverse indoor and outdoor datasets while exposing challenges in ground-truth definition, viewpoint tolerance, and coverage of real-world conditions. Its unified templates and varied viewpoint tolerances support more consistent evaluation across applications.

  • Dataset considerations: VPR datasets contain multiple views of environments captured under seasonal, viewpoint, and illumination changes, using traversal or non-traversal collection mechanisms.
  • Ground-truth considerations: Ground-truth matching is difficult because geographically nearby images may represent the same place, requiring match ranges rather than a single correct reference.
  • Application requirements: VPR systems trade pose accuracy against viewpoint invariance, with robotics prioritising confident localisation and computer vision prioritising retrieval of many matches.
  • Benchmark design: The benchmark integrates datasets spanning minimally acceptable to large viewpoint variation, and analyses how changing viewpoint tolerances affects evaluation.
  • Scope boundary: The dataset collection excludes extreme weather, aerial and underwater platforms, opposing views, and motion blur from high-speed platforms.
  • Dataset coverage: The 12 datasets cover varied environments, including cities, natural scenery, train-lines, rooms, offices, corridors, buildings, and busy streets.
  • Indoor and outdoor environments: Indoor datasets add dynamic objects, changing configurations, less-informative content, and perceptual aliasing, complementing the seasonal challenges represented outdoors.
  • Standardisation: VPR-Bench reformats dataset ground truths into unified templates and plans to re-release datasets with associated ground-truth information for consistent reporting.

3.3 VPR Techniques

VPR-Bench evaluates 10 VPR techniques spanning handcrafted descriptors, CNN-based methods, aggregation approaches, and supervised or semi-supervised learning. Implementations are adapted to a common template while retaining method-specific feature extraction and matching designs.

  • Technique set: The framework evaluates 10 VPR techniques and documents implementation details needed to interpret the experiments and results.
  • Handcrafted descriptors: HOG provides a traditional handcrafted descriptor, while CoHOG extracts entropy-based regions and convolutionally matches regional HOG descriptors for lateral viewpoint invariance.
  • CNN descriptors: AlexNet uses conv3 activations with Gaussian random projections and cosine distance, reflecting the reported robustness of conv3 to conditional variations.
  • Aggregation methods: DenseVLAD aggregates densely sampled multi-scale SIFT keypoints into a VLAD descriptor using a 128-word dictionary, followed by PCA compression and whitening.
  • Learned descriptors: AP-GeM uses a ResNet101-based GeM model trained with a ranking loss based on mean-Average-Precision and produces 2048-dimensional descriptors.
  • Learned aggregation: NetVLAD evaluates a VGG-16 model trained end-to-end on Pittsburgh 30K with a 64-word dictionary and whitening of final descriptors.
  • SPED-trained CNN methods: AMOSNet and HybridNet use conv5 features from models trained from scratch or through transfer learning on SPED, with L1 matching.
  • Region-based aggregation: RegionVLAD combines AlexNet Places365 features, 400 regions of interest, a 256-word VLAD dictionary, and cosine similarity for matching.

3.4 Evaluation Metrics

VPR-Bench argues that no single evaluation metric serves all VPR applications and system requirements, so it combines metrics covering matching, false-positive handling, computational needs, and retrieval distribution.

  • Metric selection: A single universal VPR metric does not exist because application requirements differ, including precision, retrieval breadth, real-time operation, and computational constraints.High precision may be acceptable for offline loop closure but unsuitable for real-time topological navigation when retrieval is slow.
  • System requirements: VPR-Bench also evaluates computational and spatial factors, including retrieval-time requirements and the distribution of true positives along a trajectory.A moderately precise technique with uniform true-positive spacing may be more useful for localisation than a highly precise technique with uneven coverage.
  • Metric taxonomy: VPR-Bench classifies evaluation metrics across primary localisation, loop closure, and image retrieval, while also distinguishing matching and computational-intensity perspectives.The taxonomy identifies suitable use cases rather than restricting metrics to particular communities.
  • AUC-PR: AUC-PR measures the precision–recall trade-off across confidence thresholds and is suited to applications requiring one highly precise correct match, such as loop closure.It does not reward additional correct matches beyond the single match considered per query.
  • RecallRate@N: RecallRate@N counts queries with a correct reference among the top N results, supporting systems whose later filtering can re-rank candidate matches.Unlike AUC-PR, it explicitly models the possibility that the correct image need not be ranked first.
  • ROC curves: RecallRate@N and AUC-PR reward only one correct match per query, whereas AUC-ROC adds true-negative analysis for identifying new places.ROC curves are less useful when true-positive and true-negative classes are imbalanced or when a fixed map makes new-place detection unnecessary.

3.5 Invariance Quantification Setup

VPR-Bench quantifies viewpoint and illumination invariance using controlled datasets spanning synthetic and semi-quantified real-world variations. Its evaluation compares confidence scores for same-but-varied and different places across numerically specified conditions.

  • Point Features Dataset: The Point Features dataset provides 119 viewpoints across three arcs and 19 single-LED illumination conditions for controlled invariance analysis.The framework uses two synthetic places from this dataset and focuses on viewpoint and illumination variations.
  • Dataset Scope and Limitations: Point Features uses synthetic scenes, while QUT and MIT provide less detailed real-world or semi-quantified variation to partially bridge the sim-to-real gap.The authors identify synthetic content as a limitation of Point Features and note that real-world illumination is difficult to control.
  • QUT Multi-lane Dataset: The QUT multi-lane dataset captures forward/backward and lateral viewpoint changes across five laterally shifted viewpoints.The evaluation uses two scenes, 15 viewpoints per scene, approximately 1.2-meter lateral shifts, and approximately 3.5-meter forward/backward shifts.
  • MIT Multi-illumination Dataset: The MIT multi-illumination dataset contributes indoor scenes captured under 25 predetermined lighting directions sampled over the upper hemisphere.The framework manually selects two scenes representing indoor places from the dataset.
  • Evaluation Mechanism: The evaluation measures how confidence scores decline as viewpoint or illumination changes from a keyframe, comparing same-but-varied places against different places.Identical images receive a matching score of 1, while quantified variation produces analytically and visually interpretable decreases in confidence.

4 Results and Analysis

VPR-Bench evaluates 10 techniques across 12 datasets using precision-recall, RecallRate@N, computational, and qualitative analyses. Results show metric- and dataset-dependent performance, with no single technique consistently dominating and challenging environments affecting all methods.

  • Precision-Recall Results: Deep-learning techniques generally provide higher place-matching precision, while CoHOG and DenseVLAD outperform AlexNet and CALC in the reported comparisons.NetVLAD and DenseVLAD handle 3D viewpoint variation better than other techniques, while HybridNet and AMOSNet perform well under conditional variation.
  • Dataset-Dependent Performance: All techniques perform poorly on Nordland and 17 Places, while HOG achieves state-of-the-art performance on the small-scale Living Room dataset.Nordland combines significant perceptual aliasing with a large reference database; Living Room contains high-quality indoor images under illumination variation.
  • RecallRate@N: Increasing N from 1 to 20 increases RecallRate for all techniques, and the technique ordering can change across N values and datasets.DenseVLAD leads Tokyo 24/7 on AUC-PR, whereas NetVLAD and AP-GeM outperform it for most RecallRate values.
  • Computational Performance: Descriptor matching time depends on the similarity function, descriptor dimensions, dimension length, and data type, while deep-learning methods generally require more encoding time.For fixed similarity and float-64 descriptors, matching time increases almost linearly with descriptor dimension.
  • Precision-Recall Results: State-of-the-art AUC-PR performance is distributed across techniques and datasets rather than consistently achieved by NetVLAD.The table reports the corresponding state-of-the-art technique for each dataset, and the paper emphasizes broad distribution across all techniques.
  • Metric Interpretation: AUC-PR should be reported with RecallRate@N because a perfect AUC-PR score does not ensure that every query retrieves a correct match.A perfect AUC-PR only indicates that true positives rank above false positives collectively.

4.2 ROC Curves: Finding New Places

ROC analysis evaluates VPR techniques’ ability to distinguish previously visited places from new places, complementing precision–recall analysis. The broader evaluation also examines computational platform, descriptor size, and true-positive distribution as application-dependent factors.

  • ROC performance: Learning-based techniques outperform handcrafted techniques in ROC class separation on modified Gardens Point, although all methods generally achieve sub-ideal AUC-ROC.CALC has stronger class-separation capacity than most other learning-based techniques despite weaker PR-curve performance.
  • Computational performance: GPU and CPU method orderings are mostly similar, but computation times vary noticeably across platforms.RegionVLAD versus NetVLAD is a notable encoding-time exception because RegionVLAD performs compute-intensive CPU region extraction and VLAD description.
  • Descriptor size: Increasing descriptor dimension generally increases descriptor matching time and memory footprint, while AUC-PR varies by technique and descriptor design.The experiments vary HOG cell size or PCA output dimensions while holding other parameters constant.
  • Trajectory distribution: True-positive distributions usually produce ripple effects at small inter-frame distances rather than a single ideal peak.This can support periodic loop closure for SLAM but may be problematic for VPR-only topological or primary localization; the analysis excludes non-trajectory datasets.

4.6 Acceptable Ground-truth Manipulation

VPR performance depends on ground-truth construction, viewpoint and illumination conditions, and application-specific retrieval requirements. VPR-Bench therefore evaluates these factors through ground-truth manipulation, quantified variation analysis, and platform-aware performance measures.

  • Acceptable Ground-truth Manipulation: Interchanging query and reference traverses changes matching performance and can produce different top-performing techniques on the same dataset.The magnitude and direction of performance changes differ across techniques, especially on conditionally variant datasets.
  • Acceptable Ground-truth Manipulation: Accepting overlapping frames as true positives benefits viewpoint-invariant techniques, whereas one-to-one frame ground truth may not.Traversal datasets commonly contain visual overlap between consecutive frames, so the chosen true-positive range affects measured performance.
  • Retrieval Time vs Platform Speed: Retrieval-time requirements depend on map size and platform speed rather than a fixed real-time FPS threshold.The framework models required retrieval rate from application sampling density and platform velocity, while VPR retrieval rate also depends on reference-image count.
  • Invariance Analysis: Lateral viewpoint changes reduce matching confidence more than forward/backward changes under the evaluated scene-depth conditions.Very large forward/backward movement can nevertheless reduce confidence more than a small lateral change.
  • Invariance Analysis: Directional illumination changes that partition scenes into highly and weakly illuminated regions have the strongest effect on matching confidence.HOG-based handcrafted techniques distinguish same-place illumination variants from different places across the evaluated scenarios, while contemporary deep-learning techniques struggle in these conditions.

4.9 Variance vs Invariance

Viewpoint invariance is not universally beneficial: some precise localization applications require viewpoint variance to distinguish viewpoint-specific images. The paper therefore identifies controllable viewpoint variance as an open research direction alongside invariance.

  • Variance vs Invariance: Viewpoint variance can support metrically precise localization when an accurate appearance-based or topometric map is available.In such settings, distinguishing viewpoint-varied images of the same place may be preferable to matching them invariantly.
  • Variance vs Invariance: The framework estimates viewpoint invariance, but current techniques lack a control parameter for tuning invariance to viewpoint changes.The authors propose controlled viewpoint variance as an open VPR research challenge.

5 Conclusions and Future Work

VPR-Bench is a comprehensive, open-source framework combining diverse datasets, techniques, metrics, and quantified variation analysis. Its evaluations show that VPR performance is metric-, dataset-, and variation-dependent, while several computational and environmental questions remain open.

  • Conclusions: VPR-Bench integrates 12 indoor and outdoor datasets, 10 contemporary VPR techniques, and evaluation metrics from computer vision and robotics.Its modular design supports future integration of additional datasets, techniques, and metrics.
  • Conclusions: 8 out of 10 techniques achieve state-of-the-art AUC-PR on at least one dataset, while DenseVLAD achieves state-of-the-art AUC-PR on 6 out of 12 datasets.These results indicate that no universal state-of-the-art technique exists and suggest value in ensemble-based approaches.
  • Conclusions: VPR technique utility depends strongly on the evaluation metric and application, with different metrics producing different state-of-the-art techniques.RecallRate assumes a false-positive rejection scheme, whereas AUC-PR does not make that same assumption.
  • Conclusions: 3D viewpoint change harms matching confidence more than lateral change, while directional illumination change is more challenging than uniform illumination change.Deep-learning techniques generally suffer less from 3D viewpoint change than handcrafted descriptors.
  • Future Work: Future work includes relating matching performance to encoding and matching-time bottlenecks and evaluating more challenging conditions such as aerial, underwater, extreme-weather, motion-blur, and opposing-viewpoint environments.The current study discusses only selected comparisons despite supporting many more dataset, technique, and metric comparisons.

A VPR-Bench Design

VPR-Bench is designed as a holistic, modular framework whose datasets, techniques, and evaluation modes can be configured through a common entry point. Its templates support flexible integration and future updates, while the implementation targets researchers and developers rather than GUI-based users.

  • Framework Design: VPR-Bench focuses on holistic evaluation, modularity, and convenient templates for regular updates and future consistency.The framework's code structure is documented separately in Fig. 22.
  • Framework Design: The main file lets users select evaluation datasets, VPR techniques, and one of two modes: VPR Performance Evaluation or Invariance Analysis.Performance evaluation reports place-matching performance using precision-related and computational metrics.
  • Implementation: The framework has no dedicated GUI because it targets developers and researchers with basic domain knowledge, and it is designed to remain flexible for future improvements.The code was tested on Ubuntu 20.04.1 LTS, and evaluations do not require a GPU by default.
  • Integration: New datasets can be integrated without framework-code changes by following the unified dataset template and setting the dataset path in the main file.The template is intended to accommodate variations among dataset formats.
  • Integration: New techniques can be integrated either by re-implementing them within the framework template or by supplying pre-computed matching information from an external implementation.The pre-computed-data path accepts matching scores, feature-encoding time, and descriptor-matching time; DenseVLAD and GeM use this path.

B VPR-Bench Datasets Template

The dataset template standardizes how VPR-Bench represents datasets so varied formats can be handled consistently. Query and reference traversals are organized into dedicated subfolders with integer-named images.

  • Dataset Template: Each dataset's query and reference traversals are represented by dedicated subfolders within a generic template.The template is designed to accommodate variations among available and newly integrated dataset formats.
  • Dataset Template: Images within each query or reference folder must be named as integers, motivated by representing traversal-based datasets through a graph structure.

C VPR-Bench Techniques Template

The technique template standardizes the interface between VPR methods and the framework while preserving room for method-specific preprocessing, postprocessing, and descriptor representations. It also supports techniques supplied through pre-computed matching information.

  • Technique Template: Technique templates accommodate neural or traditional descriptors, ROI extraction, image preprocessing, descriptor adaptation, and sequential or geometric priors.The pipeline also provides a parallel path for integrating pre-computed place-matching information.
  • Technique Template: A query image Q and reference map M_R are converted into query and reference feature descriptors F_Q and F_M.The query may also be a sequence or another pre-processed or post-processed query candidate.
  • Technique Template: The framework represents matching scores against all reference images as a confusion matrix C and requires three consistent technique functions.Technique-aware selective re-imports preserve consistency and ease of integration.
  • Technique Template: The required functions compute a query descriptor, compute map features, and perform VPR using the query and map features.The template returns F_Q, F_M, and the place predictions, scores, and confusion matrix represented as P, S, and C.
Loading 2005.08135v2…