PaperlayerView exact revision on arXiv

Source-linked AI summary

VPR-Space: A Benchmark for Visual Place Recognition in Space Analog Domains

Richard Fechner, Peer Neubert

arXiv:2610.35642v1cs.ROcs.CV

TL;DR

Feature-poor extraterrestrial-like environments challenge visual place recognition, despite strong performance in urban and feature-rich natural settings. VPR-Space benchmarks space-analog sequence pairs using ground truth tied to relative-pose-estimation value for downstream SLAM, and its evaluation finds substantial room for improvement across current methods.

  • Problem

    VPR methods that excel in urban and feature-rich natural environments face severe challenges on vast, feature-poor extraterrestrial-like surfaces.

    Source: §I · p. 1

  • Method

    VPR-Space curates 16 space-analog database-query pairs and uses an ensemble of relative pose estimators to construct ground truth for valuable downstream matches.

    Source: §I · p. 2

  • Results

    Significant headroom remains; averaged across datasets, the best approach achieves AUPRC 0.47 and recall@1 0.64.

    Source: A. Quantative Analysis · p. 6

  • Takeaways & Limitations

    The benchmark aims to set new goals for VPR development through challenging space-analog cases and qualitative analysis of current methods’ failure modes.

    Source: §VI · p. 7

    RPEs and VPR methods may share visual-overlap failure modes, potentially exaggerating performance on space analog domains.

    Source: §V · p. 6

Abstract

from arXiv · show

Visual Place Recognition (VPR) is an important means for loop closure detection in visual Simultaneous Localization and Mapping (SLAM). While state-of-the-art VPR and SLAM approaches excel in urban and feature-rich natural environments, the vast and feature-poor environments of extraterrestrial surfaces like on the Moon or Mars pose severe challenges for existing approaches. This paper presents a novel benchmark, VPR-Space, to evaluate VPR capabilities on extraterrestrial surfaces. There are three contributions: (1) We collected sixteen VPR database-query datasets based on existing data from Space Analog Domains (SAD), terrestrial environments that resemble properties of the Moon or Mars. The collection includes a range of challenges regarding environment, viewpoint change, and illumination conditions. (2) To address the ill-posed nature of VPR, we propose a novel approach to generate the ground-truth information in VPR benchmarks. Instead of defining that a query and datbase image show the same place if and only if their camera poses are within fixed position and viewing direction thresholds, we propose to make the decision dependent on whether the match would be valuable in the downstream SLAM task. To avoid the overhead and restrictions of evaluating VPR in-the-loop within SLAM systems, we provide VPR ground-truth (GT) matrices based on the performance of an ensemble of state-of-the-art Relative Pose Estimators (RPE). Under this GT we require a VPR method to identify exactly those image pairs for which a subsequently applied RPE is expected to provide a valuable pose estimate in a SLAM context. (3) We evaluated nine state-of-the-art VPR approaches. The results show considerable room for improvement and indicate directions for future research. We provide a pre-release version of the dataset on github: $\texttt{https://github.com/rfechner/VPR-Space}$

I. INTRODUCTION

Feature-poor, homogeneous extraterrestrial-like landscapes and harsh illumination challenge visual place recognition (VPR), motivating evaluation under planetary-like conditions. VPR-Space pairs space-analog datasets with ground truth based on whether relative pose estimators can provide valuable downstream pose estimates.

I. INTRODUCTION

  • Space analog landscapes combine homogeneous, feature-poor terrain and harsh illumination, making image descriptors difficult to match reliably.Descriptor-similarity distributions are even less separable in the space-analog example than in the challenging urban example.Source: §I · p. 1, Figure 1 · p. 1
  • The RPE criterion retains a 20 m-apart MADMAX pair with low-error relative pose change that pose thresholds or frustum overlap miss.This pair is treated as a desired VPR match.Source: Figure 2 · p. 2, §I · p. 2
  • Its ground truth uses an ensemble of relative pose estimators to identify valuable matches without evaluating VPR in the SLAM loop.The match decision is tied to expected relative-pose-estimation quality for the downstream task.Source: §I · p. 2
  • It rejects same-location SPICE-HL³ dawn–night images when minimum RPE rotation error exceeds 31°, unlike pose or frustum criteria.The paper argues that matching this pair is unlikely to produce a correct pose estimate.Source: Figure 3 · p. 2, §I · p. 2
  • VPR-Space contributes 16 database-query sequence pairs from existing space-analog SLAM datasets and evaluates nine state-of-the-art VPR methods.The benchmark also introduces an application-driven ground-truth method using relative pose estimators.Source: §I · p. 2

A. VPR Datasets and Benchmarks

Existing VPR benchmarks cover diverse terrestrial challenges, but predominantly reflect terrestrial features. Space-analog SLAM datasets provide varied desert, volcanic, simulated lunar, and illumination-change settings for benchmark construction.

A. VPR Datasets and Benchmarks

  • Existing VPR datasets cover urban, seasonal, and cross-modal challenges, but benchmarks predominantly reflect terrestrial features such as roads, vegetation, and man-made structures.The cited examples include Oxford RobotCar, Tokyo 24/7, Nordland, WildCross [20], and VPR-Bench [40].Source: A. VPR Datasets and Benchmarks · pp. 2–3

B. SLAM Space Datasets

  • MADMAX, S3LI Etna, LunarLoc, and SPICE-HL³ offer desert, volcanic, simulated-lunar, and repeat-traverse settings with varied illumination.MADMAX includes dunes and stone desert; S3LI Etna imagery depicts volcanic rock and ash; LunarLoc uses a simulated lunar landscape; SPICE-HL³ includes differently illuminated sequences along a fixed trajectory.Source: B. SLAM Space Datasets · p. 3

C. Approaches to Construct a VPR-GT

Existing VPR ground truth uses varied matching rules, while point-cloud or frustum-overlap approaches leave some decision rules open. VPR-Space is designed around viewpoint and illumination changes, diverse analog environments, application-relevant ground truth, and challenge to current methods.

C. Approaches to Construct a VPR-GT

  • VPR ground truth has no unique correct matching rule; common benchmarks use pose distance or frame index, while benthic VPR uses seafloor footprint overlap.Some prior work also discusses point-cloud overlap and shared visible information in a frustum, but leaves the decision rule for future work when LiDAR or depth maps are available.Source: C. Approaches to Construct a VPR-GT · p. 3

III. BENCHMARK CONSTRUCTION

  • VPR-Space targets viewpoint and illumination changes, diverse Mars- and Moon-analog environments, application-relevant ground truth, and challenge to state-of-the-art methods.The dataset table defines viewpoint change and illumination change as hard changes within or between sequences, respectively.Source: §III · p. 3, Table I · p. 3

A. Choosing Database and Query Sequences

The benchmark selects sequence pairs for viewpoint and illumination changes, then manually checks data because pose and imagery inconsistencies would affect relative-pose-based ground truth.

A. Choosing Database and Query Sequences

  • The benchmark selects 16 pairs: ten viewpoint-change pairs and six SPICE-HL³ illumination-change setups across three lighting pairings.The viewpoint-change pairs come from MADMAX, S3LI, and LunarLoc; the selection balances Mars- and Moon-analogs, trajectory topologies, and geological environments.Source: A. Choosing Database and Query Sequences · pp. 3, 5

B. Data Cleaning

  • Three annotators checked imagery and poses, and every frame flagged by any annotator for visual corruption or apparent error beyond 1 m or 10° was discarded.For about 9,200 flagged data points, annotators agreed on about 72%, and at least two agreed on 81%.Source: B. Data Cleaning · p. 5

C. Construction of the VPR Ground Truth

VPR-Space defines a match by whether an ensemble of relative pose estimators (RPEs) can produce a valuable pose estimate for downstream SLAM, rather than by pose distance alone. Voting and adjustable error thresholds produce hard and soft ground-truth matrices.

C. Construction of the VPR Ground Truth

  • The ground-truth matrix labels an image pair positive when RPE estimates agree sufficiently with its ground-truth relative pose transformation, making the criterion reflect downstream SLAM value.The oracle uses ground-truth poses and is run once per image pair, rather than once for every evaluated VPR method.Source: C. Construction of the VPR Ground Truth · p. 5
  • A majority vote combines multiple RPEs, so a match need not depend on any single estimator.The ensemble includes DUSt3R, MUSt3R, MapAnything, MADPose, and SuperPoint with LightGlue.Source: C. Construction of the VPR Ground Truth · p. 5
  • An RPE counts as correct only when both input orderings meet the rotation and translation error thresholds; at least n estimators must do so, and pairs beyond dmax are excluded.Translation error is minimized over the estimated translation direction, reducing the requirement to five degrees of freedom.Source: C. Construction of the VPR Ground Truth · p. 5
  • Varying the error thresholds and vote requirement yields hard and soft correspondence matrices; soft-GT matches outside the hard GT are not penalized.With positive matches equalized, established GT constructions tend to request harder matches, a pattern reported across the dataset.Source: Figure 5 · p. 5, C. Construction of the VPR Ground Truth · p. 6

A. Quantative Analysis

VPR methods leave substantial room for improvement on space analog domains, with performance varying across datasets and illumination conditions. Bag-of-Queries and MegaLoc show some domain adaptation, while large lighting differences remain difficult for current methods.

A. Quantative Analysis

  • Averaged across datasets, the best VPR approach achieves AUPRC 0.47 and recall@1 0.64; on the most challenging dataset, these are 0.18 and 0.38, respectively.In the worst case, R@100p is near zero across all methods.Source: A. Quantative Analysis · p. 6
  • Bag-of-Queries and MegaLoc demonstrate domain adaptation on the MADMAX A0–A2, MADMAX F0–F3, and SPICE-HL³ dawn–noon pairs.The reported pattern is not uniform across illumination conditions: current methods struggle when lighting differences become too large, except where outlines and shadows are quite similar.Source: A. Quantative Analysis · p. 6
  • DenseVLAD, a non-deep-learning method, performs best on SPICE-HL³-SPAD dawn–noon.Source: A. Quantative Analysis · p. 6

B. Qualitative Analysis: What are hard queries that the SotA cannot solve yet?

The qualitative analysis identifies shared-horizon imagery in open space as a hard case: it can score highly despite being a false match and may pass geometric verification. The discussion qualifies the benchmark’s scope and ground-truth interpretation, including its RPE dependence and descriptor coverage.

B. Qualitative Analysis: What are hard queries that the SotA cannot solve yet?

V. DISCUSSION

  • Because VPR and RPE both rely on visual overlap, the RPE-based ground truth may exclude coincident-pose pairs when a valuable pose estimate cannot be made.The authors frame this as a possible limitation while emphasizing that the GT is tailored to the downstream SLAM domain.Source: §V · p. 6
  • The GT uses voting to go beyond any one RPE, and its matrices can be updated as RPEs or downstream-task expectations change; modified GTs should be identified.Source: §V · p. 6
  • False positives matter under this GT because they are known to produce large relative pose errors and may pass spatial verification, which uses methods similar to RPEs.The authors note that this makes false-positive-sensitive measures such as AUPRC and R@100p relevant again.Source: §V · p. 6
  • The evaluation covers holistic descriptors only, although VPR-Space can also evaluate local-descriptor and hierarchical pipelines.Source: §V · p. 7
  • RPE-based GT generation requires ground-truth pose information and therefore cannot itself solve the VPR problem.Source: §V · p. 7

VI. CONCLUSION

VPR-Space connects VPR evaluation in space analog domains to downstream SLAM by using relative pose estimators to define ground truth. The evaluation finds substantial room for improvement in current methods.

VI. CONCLUSION

  • The evaluation of state-of-the-art VPR methods shows significant headroom for improvement on VPR-Space.Qualitative analysis also identified concrete failure modes of the evaluated methods.Source: §VI · p. 7
  • VPR-Space uses curated sequence pairs from existing space analog SLAM datasets to represent viewpoint and illumination changes.Source: §VI · p. 7
  • An ensemble of state-of-the-art relative pose estimators defines VPR ground truth by rewarding retrieval only when a low-error pose estimate can be made.Source: §VI · p. 7

APPENDIX

The appendix describes subsampling for sequences with repeated views and gives the reported scope of additional dataset statistics.

APPENDIX

  • For sequences with multiple images of the same place, subsampling uses a rolling window, except for SPICE-HL³ sequences.The passage attributes repeated views to differences in sensor-platform velocities and sampling rates.Source: APPENDIX · p. 7
  • A new image is retained when its position changes by more than 1 m or its camera yaw changes by more than 10° from the previously retained image.Source: APPENDIX · p. 7
  • Table IV reports image counts, the percentage of queries with at least one match, and the percentage of positive matches for each database/query pair under VPR-GT-RPE-0.Source: Table IV · p. 7

Related papers

Loading 2610.35642v1…