PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyLoc: Towards Universal Visual Place Recognition

Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, Sourav Garg

arXiv:2308.00688v2cs.CVcs.AIcs.RO

TL;DR

Existing VPR systems are highly effective in training-like urban settings but generalize poorly across unstructured environments and major distribution shifts. AnyLoc combines off-the-shelf self-supervised foundation-model features with unsupervised local aggregation, achieving up to 4× higher performance than prior approaches and supporting broad environmental, temporal, and viewpoint diversity.

  • Problem

    VPR methods trained on large urban datasets perform well in similar settings but do not generalize reliably to indoor and unstructured environments.

  • Method

    AnyLoc uses per-pixel features from off-the-shelf foundation models and combines them with unsupervised aggregation methods such as VLAD and GeM.

  • Results

    AnyLoc outperforms previous state-of-the-art approaches by up to 4× across a diverse suite of VPR datasets and environments.

  • Takeaways & Limitations

    AnyLoc provides a baseline for VPR spanning urban, indoor, aerial, underwater, and subterranean environments under varied temporal and viewpoint conditions.

  • Takeaways & Limitations

    Existing VPR approaches rely on abundant domain- or map-specific data and task-specific training, particularly for urban settings.

Abstract

from arXiv · show

Visual Place Recognition (VPR) is vital for robot localization. To date, the most performant VPR approaches are environment- and task-specific: while they exhibit strong performance in structured environments (predominantly urban driving), their performance degrades severely in unstructured environments, rendering most approaches brittle to robust real-world deployment. In this work, we develop a universal solution to VPR -- a technique that works across a broad range of structured and unstructured environments (urban, outdoors, indoors, aerial, underwater, and subterranean environments) without any re-training or fine-tuning. We demonstrate that general-purpose feature representations derived from off-the-shelf self-supervised models with no VPR-specific training are the right substrate upon which to build such a universal VPR solution. Combining these derived features with unsupervised feature aggregation enables our suite of methods, AnyLoc, to achieve up to 4X significantly higher performance than existing approaches. We further obtain a 6% improvement in performance by characterizing the semantic properties of these features, uncovering unique domains which encapsulate datasets from similar environments. Our detailed experiments and analysis lay a foundation for building VPR solutions that may be deployed anywhere, anytime, and across anyview. We encourage the readers to explore our project page and interactive demos: https://anyloc.github.io/.

I. INTRODUCTION

VPR supports robot state estimation but existing high-performing methods are largely specialized for urban, structured environments. AnyLoc instead combines foundation-model features with unsupervised aggregation to target universal operation across diverse environments and conditions.

  • VPR is fundamental for robot state estimation and is used in autonomous cars, uncrewed vehicles, and wearable devices.
  • AnyLoc rethinks VPR around general-purpose visual features from large-scale pretrained foundation models rather than VPR-specific training.
  • The approach blends foundation-model features with established local-aggregation methods, including VLAD and GeM.
  • AnyLoc is presented as a baseline evaluated across 12 datasets spanning major diversity in place, time, and perspective.
  • Per-pixel self-supervised features combined with unsupervised aggregation outperform direct per-image features, while semantic domain characterization further improves VLAD vocabulary construction.
  • Existing VPR methods perform strongly on environments resembling their training data but drop sharply under substantial feature distribution shifts, such as underwater or aerial deployment.

III. ANYLOC: TOWARDS UNIVERSAL VPR

AnyLoc investigates how foundation models, local feature extraction, aggregation, and vocabulary design can support universal VPR. The analysis favors self-supervised models and dense intermediate-layer features whose invariances can be transferred to image-level place representations.

  • AnyLoc targets VPR that operates across environments, temporal conditions, and viewpoints without relying on task-specific deployment training.
  • The paper asks which foundation models, local features, aggregation methods, and vocabularies are best suited to universal VPR.
  • A. Choice of Foundation Model: Self-supervised joint-embedding models such as DINO and DINOv2 outperform contrastive CLIP and masked-autoencoding MAE in the authors’ initial experiments.
  • B. Choice of Features: Per-pixel ViT features enable fine-grained matching, so AnyLoc extracts intermediate-layer facets and discards the CLS token.
  • B. Choice of Features: The value facet provides the strongest contrast between matched points and background, while deeper-layer value features are most robust to distractors.
  • B. Choice of Features: A CLS token summarizes an entire image, whereas AnyLoc investigates transferring dense per-pixel invariances to image-level place recognition.

C. Choice of Aggregation Technique

AnyLoc constructs global place descriptors by aggregating per-pixel features, exploring generalized-mean pooling and VLAD variants. VLAD forms a database-derived vocabulary and summarizes feature residuals relative to cluster centers.

  • The aggregation technique groups local features to describe image sections and ultimately an environment, instead of directly using the CLS token.
  • For an H × W image with per-pixel features f_i ∈ R^D, the global descriptor is defined through generalized-mean pooling.
  • The pooling exponent p yields GAP at p = 1, GeM at p = 3, and GMP as p →∞.
  • VLAD variants cluster database features into N centers that form a vocabulary, then sum residuals assigned to each center.
  • VLAD assignment can be hard, with α_k(f_i) equal to 0 or 1, or soft, with α_k(f_i) representing an assignment probability between 0 and 1.

D. Choice of Vocabulary

AnyLoc constructs VLAD vocabularies around semantic domains identified from foundation-model features, rather than relying only on global or map-specific sources. PCA projections of global descriptors reveal distinct environment groups that guide vocabulary selection.

  • D. Choice of Vocabulary: Domain-specific vocabulary construction is motivated by the open-set semantic attributes encoded in foundation-model features.This approach is presented as more scalable than relying on abundant domain- or map-specific data and task-specific training.
  • D. Choice of Vocabulary: Foundation-model features and unsupervised PCA identify distinct semantic domains across the evaluated environments.The domains include Urban, Indoor, Aerial, SubT, Degraded, and Underwater.
  • D. Choice of Vocabulary: The evaluation spans structured and unstructured environments, including urban, indoor, aerial, underwater, visually degraded, and subterranean settings.These datasets introduce distribution shifts, degraded corridors, low illumination, seasonal variation, and varied aerial imagery.
  • D. Choice of Vocabulary: PCA is computed solely from database images, excluding query images for fair analysis.

1) Structured Environments:

The experiments benchmark AnyLoc across structured and unstructured datasets using Recall@K and standardized hardware and random seeds. The comparison includes specialized VPR methods and foundation-model CLS baselines.

  • Datasets: The evaluation covers indoor, outdoor, aerial, underwater, visually degraded, and subterranean environments with substantial viewpoint and appearance changes.Structured benchmarks include six indoor and outdoor datasets, while unstructured benchmarks target challenging distribution shifts and degraded imagery.
  • Evaluation protocol: Recall@K is the evaluation metric, with higher recall indicating better performance.All experiments use random seed 42 and NVIDIA RTX 3090 hardware for consistency and reproducibility.
  • Baselines: AnyLoc is compared with specialized VPR baselines and foundation-model CLS descriptors across training, representation, supervision, backbone, and data-scale variations.
  • AnyLoc nomenclature: AnyLoc variants combine VLAD or GeM aggregation with DINO or DINOv2 features.The specified configurations use selected ViT layers and facets, with 128 or 32 VLAD clusters depending on the variant.

V. EXPERIMENTS, RESULTS, AND ANALYSES

The experiments test AnyLoc across structured and unstructured environments, temporal changes, viewpoint shifts, and representation variants. Results show strong performance from aggregated foundation-model features, especially in unstructured settings.

  • Experimental scope: AnyLoc is evaluated against state-of-the-art VPR techniques across structured and unstructured environments, viewpoint shifts, and temporal appearance variations.The analysis also compares specialized baselines, CLS-token variants, vocabulary designs, and self-supervised versus VPR-trained ViTs.
  • Structured Environments: AnyLoc-VLAD-DINOv2 achieves the highest recall across all Indoor datasets, outperforming MixVPR and CosPlace by 5% and 20% on average R@1.
  • Structured Environments: AnyLoc-VLAD beats all prior approaches on the reported structured datasets.On daytime Pitts30k and St Lucia, it is inferior by 3-4%, but it achieves state-of-the-art on Oxford day-night variations.
  • Unstructured Environments: In unstructured environments, AnyLoc-VLAD exceeds the best specialized and CLS baselines by 32% and 18% on average R@1.DINOv2-CLS also exceeds MixVPR by 41% on Nardo-Air and 35% on VP-Air under strong viewpoint variations.

3) Temporal & Viewpoint Changes:

AnyLoc-VLAD improves retrieval under temporal and viewpoint changes, while domain-specific vocabularies improve recall over alternative vocabulary sources. Local aggregation also produces large gains over CLS descriptors.

  • Temporal Changes: AnyLoc-VLAD gains 5/11% on day-night cycles, 9/8% on seasonal shifts, and 21/28% on long-period jumps over MixVPR/CosPlace.
  • Viewpoint Changes: AnyLoc-VLAD gains 21/30% on orientation-based shifts and 39/49% on extreme 90°/180° shifts over MixVPR/CosPlace.
  • CLS vs. Aggregation: DINOv2-based AnyLoc-GeM and AnyLoc-VLAD outperform DINOv2-CLS by 9%/2% and 23%/18% on structured/unstructured environments.
  • Vocabulary Analysis: Domain-specific vocabulary achieves 13% higher recall than map-specific vocabulary and 19% higher recall than global vocabulary in the aerial domain.
  • Vocabulary Analysis: Similar colors across reference-query images indicate matched VLAD clusters within a domain.

2) Consistency:

AnyLoc’s domain-specific vocabularies exploit consistent local-feature assignments across environments and places, improving transfer to target maps with limited reference information.

  • Consistency: Domain-specific vocabularies consistently assign similar structures across changing conditions and places, including roads, vegetation, buildings, floors, ceilings, signs, and furniture.Urban and aerial scenes show cross-image consistency, while indoor signs and furniture show intra-place consistency.
  • Vocabulary transfer: 7–18% higher R@1 is achieved in Aerial and Urban domains when vocabularies come from larger source maps rather than each target dataset’s smaller map.This demonstrates transferability of vocabularies between datasets within the same domain.
  • Vocabulary transfer: Unified diverse Indoor-domain vocabularies achieve better overall recall than map-specific vocabularies.Indoor performance can drop when the largest source map is limited or datasets differ substantially, such as shops versus offices.
  • Design choices: Performance scales with DINOv2 backbone size but tends to saturate at ViT-L, while ViT-G is used because it performs better on average.ViT-L has 300 million parameters.

2) ViT Layers & Facets:

AnyLoc’s performance depends on selecting suitable transformer layers and feature facets, while unsupervised aggregation and self-supervised representations provide the core design advantages over VPR-supervised alternatives.

  • ViT Layers & Facets: Peak performance occurs in deeper intermediate layers rather than consistently in the final layer, motivating operating-layer choices for DINO and DINOv2.For DINO ViT-S on Oxford, middle layers benefit from stronger positional encoding under unchanged viewpoints.
  • ViT Layers & Facets: Key and value facets consistently achieve high recall for DINO and DINOv2 respectively, while positional bias makes query and key less robust to large viewpoint shifts.The query–value performance ordering reverses between Baidu and Oxford.
  • Aggregation: Hard-assignment VLAD performs best among compared unsupervised aggregators, while GeM offers a favorable performance–storage trade-off.Hard assignment is typically 1.4 times faster than soft assignment.
  • Self-supervised vs VPR-supervised ViT: AnyLoc-VLAD-DINOv2 outperforms ViT-B CosPlace-VLAD by 8–13%, and ViT-S AnyLoc-VLAD exceeds it by 4–18% using 4× fewer parameters.The exception is the urban domain, where CosPlace-VLAD performs better than ViT-S- and ViT-B-based AnyLoc-VLAD.
  • Method: AnyLoc combines per-pixel foundation-model features with unsupervised VLAD and GeM aggregation to extend VPR across diverse environments.The conclusion frames this combination as a step toward deployment across anytime, anywhere, and anyview conditions.

APPENDIX

The appendix documents evaluation resources, dataset construction, and the diverse structured-environment benchmarks used to assess AnyLoc.

  • Vocabulary construction: Urban vocabulary construction uses all Oxford and St Lucia images but only every fourth Pitts-30k image to balance reference-image frequencies.Aerial vocabulary construction similarly subsamples VP-Air while retaining all Nardo-Air images.
  • Evaluation setup: The evaluation uses 12 diverse datasets spanning varied places, times, viewpoints, and environmental conditions.The appendix introduces detailed descriptions of these datasets.
  • Indoor datasets: Baidu Mall provides 2292 query images and 689 reference images with varying camera poses and perceptual aliasing challenges.Its ground-truth location and 3D pose support both 6-DoF localization and VPR testing.
  • Indoor datasets: Gardens Point contains 200 reference and 200 query images across day and night traverses, with drastic lighting changes between indoor and outdoor scenes.These conditions make the dataset challenging for VPR.
  • Indoor datasets: 17 Places includes 406-image reference and query traverses across buildings, with clutter, lighting changes, and semantically rich content.The dataset covers York University and Coast Capri Hotel.
  • Urban datasets: Pittsburgh-30k uses 10,000 database images and 6816 query images, challenging retrieval with viewpoint shifts, varied structures, and distractors.The evaluation uses the test split.

B. Unstructured Environments

The unstructured-environment benchmarks cover visually degraded indoor, subterranean, aerial, and underwater settings with strong illumination, viewpoint, and distribution shifts.

  • Visually degraded indoor: Hawkins evaluates a 282 m abandoned-hospital corridor with visually degraded features and opposing forward and backward viewpoints.The database and query sets contain 65 and 101 images, respectively.
  • Subterranean: Laurel Caverns presents low illumination, sparse visual features, and opposing viewpoints across a 102 m subterranean trajectory.The database and query sets contain 141 and 112 images, respectively.
  • Aerial: Nardo-Air uses GNSS-denied drone imagery at 50 m altitude against satellite references, combining perceptual aliasing, unusual vegetation, and a long time shift.The trajectory spans 700 m across a square-kilometer area.
  • Aerial: VP-Air contains 2706 database-query pairs and 10,000 distractors from 300 m altitude across urban regions, farmlands, and forests.The dataset spans over 100 km.
  • Underwater: Mid-Atlantic Ridge captures underwater place recognition under low illumination, seabed objects, and appearance shifts between 2015 and 2020.The dataset contains 65 database and 101 query images over an 18 m trajectory.
Loading 2308.00688v2…