PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceive to Hypothesize, Verify to Ground: An Agentic Reasoning Framework for Open-World Geo-Localization

Yutian Jiang, Ruijie Li, Sisuo Lyu, Xixuan Hao, Qingxiang Liu, Yongzi Yu, Yuxuan Liang

arXiv:2608.29880v1cs.AIcs.MM

TL;DR

Open-world geo-localization remains difficult because ambiguous visual cues and unconstrained retrieval can produce hallucinated or context-inconsistent conclusions. GeoPAVE addresses this with single-pass uncertainty-aware hypotheses followed by evidence-grounded support, refutation, and refinement, and evaluates the approach with PAVED. The paper reports complementary perception and verification components, while noting rule-based routing, latency, and urban sampling limitations.

  • Problem

    Ambiguous visual evidence and unconstrained tool use leave existing geo-localization systems vulnerable to perceptual shortcuts and verification inconsistency.

  • Method

    GeoPAVE uses single-pass uncertainty-aware hypothesis generation and conditional tool-based verification that supports, refutes, or refines candidates.

  • Results

    Perception and verification are complementary: removing verification lowers Acc@500m from 16.87% to 12.72%, while Kmax = 4 improves some thresholds but worsens others.

  • Takeaways & Limitations

    The perceive-then-verify formulation combines broad candidate coverage with evidence-based refinement for fine-grained geo-localization.

  • Takeaways & Limitations

    The rule-based uncertainty-to-skill routing may not generalize to ambiguous edge cases, and sequential tool invocations add latency.

Abstract

from arXiv · show

Open-world geo-localization requires models to reason over ambiguous visual cues through multi-step reasoning and external knowledge grounding. While recent large vision-language models exhibit strong multimodal reasoning capabilities, existing approaches still suffer from perceptual hallucination and context drift due to the lack of explicit evidence-grounded verification. In this work, we reformulate geo-localization as a human-like perceive-then-verify reasoning problem and propose GeoPAVE (Geo-localization Perception-and-Verification-Engine), a bi-level agentic framework that contains perception-based hypothesis generation via single-pass rollouts and verification-based evidence grounding for decision actions: support, refute, and refine. To support rigorous evaluation, we further introduce PAVED, a novel dataset derived from real-world user check-in data, equipped with comprehensive reasoning trajectories featuring multi-hop queries, multi-round tool invocations, and structured perception-verification traces. The dataset and code are available at https://github.com/Arandinglv/GeoPAVE.

1 Introduction

Open-world geo-localization must integrate ambiguous visual cues with broad geographic knowledge and iterative external verification. GeoPAVE addresses perceptual shortcuts and verification inconsistency through a perceive-then-verify framework with uncertainty-aware hypotheses and evidence-grounded refinement.

  • Open-world geo-localization predicts image coordinates without a retrieval gallery, requiring fine-grained visual evidence, geographic knowledge, multi-hop retrieval, and fact verification.
  • Existing approaches struggle when ambiguous perceptual evidence and unaudited reasoning steps make conclusions unreliable.
  • Perceptual shortcuts map partial cues to frequent regions, while parallel rollouts increase inference cost without resolving single-pass unreliability.
  • Unconstrained tool use can mix relevant and irrelevant retrievals, causing models to follow plausible but visually inconsistent locations.
  • The framework and PAVED dataset operationalize a human-like loop that forms ranked hypotheses and cross-checks them against external geographic context.
  • GeoPAVE generates uncertainty-aware hypotheses in one pass, then verifies and refines them through strategic tool invocation and evidence grounding.

2 Preliminaries

The paper formulates open-world geo-localization as evidence-conditioned selection among visual hypotheses rather than closed-world classification or retrieval. This framing motivates tool-augmented agentic reasoning beyond fixed reference galleries.

  • Geo-localization predicts a coordinate from an image by generating candidate hypotheses, retrieving external evidence, and selecting the highest-scoring refined candidate.
  • Open-world geo-localization seeks precise global locations, whereas traditional classification and retrieval depend on predefined cells or large visual databases.
  • Agentic multimodal systems use external tools to ground perception and extend beyond parametric knowledge.
  • Existing agentic and test-time-scaling methods deepen reasoning through multi-agent exploration or reinforcement learning but require expensive computation.

3 Methodology

GeoPAVE separates single-pass hypothesis generation from conditional, evidence-grounded verification. Uncertainty routes specialized tools, whose support, refutation, or refinement judgments update candidate scores and preserve consistency.

  • GeoPAVE decouples hypothesis generation from validation through single-pass perception and non-parametric refinement using heterogeneous external evidence.
  • Perception Module: The perception agent generates multiple candidate outcomes in one pass, each containing a coordinate, confidence, rationale, uncertainty tags, and verification actions.
  • Perception Module: Conflicting visual, textual, or semantic cues lower adjusted confidence and record unresolved ambiguity as structured routing signals.
  • Verification Module: GeoPAVE maps uncertainty tags to three verification skills: linguistic grounding, scene verification, and contextual validation.
  • Verification Module: Each evidence record labels a target hypothesis, source, observation, decision, score update, and possible hard veto.
  • Verification Module: Support raises confidence, refutation lowers or rejects candidates, and refinement updates locations or supporting context when evidence is grounded.
  • Hypothesis Refinement: The refinement pass reassesses initial hypotheses with the evidence pool, and the highest-scoring valid candidate is selected deterministically.
  • Hypothesis Refinement: Evidence-anchored second-pass reasoning reduces context drift and preserves consistency in ambiguous, long-horizon visual scenarios.

4 Datasets

PAVED is a globally sourced geo-localization dataset combining human check-in imagery with structured, tool-grounded reasoning trajectories. Its curation process audits evidence consistency and localization precision.

  • Dataset Construction: PAVED contains 2,524 manually filtered street-view images from worldwide Foursquare check-ins, emphasizing commercially active, public, and residential locations.
  • Dataset Construction: The dataset jointly supports global coverage, geo-localizability, high-resolution imagery, nuanced evaluation, and paired tool-grounded reasoning trajectories.
  • Structured CoT: Structured CoT separates each sample into hypothesis generation, evidence verification, and candidate refinement.
  • Trajectory Curation: Trajectory curation first generates traces of candidate generation and tool interactions, then uses reviewer rejection sampling to enforce valid calls, evidence consistency, and d(p_hat,p*) < 5km.
  • Evaluation: Table 2 compares accuracy across PAVED, MAPBench-V2, and IM2GPS3K under multiple error thresholds, with Gain reporting absolute improvement over Gemini-3.0-Flash.

5 Experiments

GeoPAVE is evaluated on complementary global, Flickr-based, and China-centered benchmarks using coordinate accuracy and trace-quality metrics. Results show gains from the combined perception-verification design, evidence skills, and calibrated hypothesis budgeting, with correction diagnostics explaining how candidates are retained and refined.

  • Experiment Setup: GeoPAVE is evaluated on PAVED, IM2GPS3K, and MAPBenchV2, covering global open-world, legacy Internet-photo, and China-specific urban geo-localization.PAVED contains 2,524 human-check-in street-view images, while IM2GPS3K contains 2,997 Flickr images and MAPBench-V2 uses a 2.4K-image test split.
  • Experiment Setup: Coordinate accuracy uses Acc@r across thresholds from street-level to regional localization, while trace metrics measure candidate coverage, correction, and logarithmic error reduction.The reported Acc@r thresholds include 500m, 2km, 25km, 200km, and 750km; COR, CR, and LCG expose the correction process behind final predictions.
  • Overall Performance (RQ1): On PAVED, GeoPAVE achieves the best performance across all thresholds, improving Gemini-3.1-Pro from 11.10% to 16.87% at Acc@500m and from 17.00% to 26.89% at Acc@2km.The gains extend from fine-grained to coarse-grained thresholds on samples requiring text, POI, and contextual cues.
  • Overall Performance (RQ1): GeoPAVE outperforms Thinking with Map from Acc@25km onward on MAPBenchV2, although the four-rollout baseline is stronger for exact fine-grained localization.The comparison indicates different strengths across fine-grained and coarse-grained scenarios.
  • Ablation Study (RQ2): The full perception-verification model reaches 16.87% at Acc@500m, versus 12.72% without verification, while verification-only also underperforms the complete system.The ablation attributes broad candidate coverage to perception and accurate fine-grained conversion to verification, treating the modules as complementary.
  • Ablation Study (RQ2): Removing scene verification lowers Acc@500m from 16.87% to 11.87%, while contextual validation lowers it to 13.58% and linguistic grounding produces a milder decline.The three skills provide complementary evidence through search-based scene grounding, map-side context, and OCR-derived script or language cues.
  • Ablation Study (RQ2): Kmax = 4 improves accuracy at 25km and 200km but performs worse at the other thresholds, so GeoPAVE retains Kmax = 3 as default.Additional hypotheses increase regional candidate coverage while introducing distractors during fine-grained verification.
  • Ablation Study (RQ2): GeoPAVE combines OCR, web retrieval, and Map/POI evidence in one process, achieving 16.87% at 500m and 59.96% at 25km.Figure 4 diagnostics further show higher candidate coverage and stronger fine-grained correction, while case studies illustrate rejection and support based on contextual evidence.

6 Conclusion and Future Works

The paper reformulates open-world geo-localization as a perceive-then-verify problem and proposes GeoPAVE, combining perception-based hypothesis generation with evidence-grounded verification. It concludes that this design reduces shortcuts and context inconsistency while motivating future scalable training and interactive reinforcement learning.

  • Conclusion: GeoPAVE combines perception-based hypothesis generation with evidence-grounded verification in a bi-level agentic framework.The framework targets ambiguous visual cues by structuring geo-localization around separate perception and verification stages.
  • Conclusion: The framework is reported to reduce AI shortcuts and context inconsistency under ambiguous visual cues while improving fine-grained geo-localization.
  • Future Works: Future work will scale PAVED and GeoPAVE into a data-generation and training pipeline using verified reasoning traces.
  • Future Works: Future work will also explore agentic reinforcement learning in interactive environments where models gather evidence and improve geographic reasoning through feedback.

Limitations

The paper identifies limitations in GeoPAVE’s routing policy, inference efficiency, dataset coverage, and annotation process. These constraints affect generalization, latency, geographic diversity, and dependence on an external reviewer model.

  • Method and Generalization: The rule-based uncertainty-to-skill routing policy may not generalize optimally when uncertainty signals are ambiguous or underspecified.
  • Efficiency: Multiple sequential tool invocations introduce additional inference latency compared with single-pass reasoning approaches.
  • Dataset Scope: Although PAVED has globally distributed coverage, Foursquare check-ins may overrepresent commercially active urban areas and limit rural or less-documented evaluation diversity.
  • Annotation Process: PAVED annotation quality is partially dependent on a stronger reviewer model used to filter low-quality reasoning traces.

Ethical Consideration

The paper addresses privacy, licensing, and misuse risks associated with deriving PAVED from real-world check-ins and street-view imagery. It removes identifying metadata, restricts redistribution when licenses require it, and limits intended use to benchmark research.

  • Risks: PAVED’s real-world check-in locations and street-view imagery raise privacy, licensing, and misuse concerns.
  • Privacy Protection: The dataset removes user identifiers, timestamps, and other check-in metadata that could reveal individual mobility patterns.
  • Licensing: When raw imagery cannot be redistributed under third-party licenses, the release provides only derived annotations, metadata, or retrieval scripts.
  • Misuse: The authors warn that image geolocalization can be misused for surveillance, doxxing, or sensitive-location identification.
  • Disclosure: AI-assisted technologies were used only for language polishing and grammar correction, not for the paper’s core ideas, experiments, results, analyses, or references.

A Experiment Results

The experiments evaluate GeoPAVE across diverse datasets and present PAVED as a human-centric benchmark with structured reasoning supervision. PAVED uses globally collected check-in imagery and supports fine-grained geographic evaluation with tool-grounded trajectories.

  • Results: GeoPAVE reaches 47.76% at Acc@10km and 65.31% at Acc@100km, outperforming all baselines at both thresholds.
  • Evaluation Setup: The evaluation spans legacy internet photos, region-specific urban environments, and human-centric reasoning scenarios to assess open-world geolocalization capabilities.
  • PAVED Dataset: PAVED contains 2,524 manually filtered street-view images from worldwide human check-ins, anchored to genuine points of interest and accessible urban scenes.
  • PAVED Dataset: PAVED jointly provides global human-centric imagery, high-resolution visual evidence, data diversity, geo-localizability, and nuanced distance- and address-level evaluation.
  • Reasoning Supervision: Each PAVED sample includes structured reasoning annotations that decompose localization into hypothesis generation and tool-augmented verification.
  • Data Construction: PAVED’s construction uses Foursquare check-ins and Google Maps imagery, with localizability filtering to remove non-localizable samples.

B.2 Details of Benchmarks

The evaluation uses multiple open-world geolocation benchmarks and compares GeoPAVE with retrieval, fine-tuning, web-augmented, swarm, and map-augmented models. PAVED additionally analyzes tool-invocation patterns and representative failure cases.

  • Benchmarks: IM2GPS3K contains 2,997 geotagged Flickr images for evaluating open-world localization on noisy legacy internet photos.Its images vary in quality, composition, and geographic context.
  • Benchmarks: MAPBench-V2 uses a 2.4K-image test split centered on region-specific urban POIs and street-view scenes across Chinese cities.It tests localized textual cues, architectural styles, and fine-grained urban reasoning.
  • Failure analysis: Representative failures include missing candidate coverage and false textual grounding, with errors of 14,497.47 km and 16,119.78 km respectively.The cases show how absent candidates or over-weighted textual cues can derail localization.
  • PAVED: PAVED includes globally annotated imagery from real human check-ins and supports analysis of verification tool-invocation patterns.The dataset is designed to evaluate general perception and verification abilities.

C Evaluation Metric Details

The evaluation measures coordinate accuracy with distance thresholds and diagnoses candidate generation and verification behavior. GeoPAVE’s qualitative trajectories retain uncertainty, use external evidence, and reject over-specific candidates when scene evidence conflicts.

  • Coordinate accuracy: Acc@r counts predictions as correct when haversine distance to ground truth is within r, using thresholds from 0.5 km to 750 km.Missing, invalid, or unparsable coordinates count as incorrect at every threshold.
  • Candidate diagnostics: GeoPAVE’s initial candidate set is generated before verification, while its final prediction is the coordinate selected after evidence-grounded refinement.For single-coordinate baselines, the initial candidate set is treated as a singleton.
  • Candidate diagnostics: Correction Recall measures samples that are initially wrong at threshold r but become correct after verification.The protocol sets τ = r unless otherwise specified.
  • Perception: The perception stage produces a fallback and three uncertain candidate locations, then routes Zoom-Crop and OCR actions from uncertainty tags.Recommended verification actions accompany each candidate.
  • Verification: Verification grounds candidates with OCR, Web, and Map evidence before compact serialization and refinement.The tools provide structured evidence for support, refutation, and hypothesis refinement.
  • Qualitative comparison: GeoPAVE preserves ambiguous cues as hypotheses and uses external evidence to reject visually plausible but over-specific coordinates.In the comparative example, it evaluates Johannesburg and Pretoria before falling back to the broader Johannesburg hypothesis.

D.4 Failure Analysis

GeoPAVE’s failures arise when the correct location is absent from the candidate set or when misleading text dominates verification. Its verification process combines routed tools and structured evidence, but early perception errors can still propagate.

  • Limitations: GeoPAVE remains imperfect: Acc@200km is 69.35%, compared with 85.86% at 750 km.The authors identify fine-grained regional and city-level localization as challenging.
  • False textual grounding: In the false-grounding case, OCR over-weights “KICKZ,” reinforces Munich, and fails to test whether the cue is geographically unique.The prediction is 16,119.78 km from the Melbourne ground truth.
  • Missing candidate coverage: In the missing-coverage case, Moscow is absent from the initial candidates, misleading POI evidence favors Sydney, and web verification fails to recover the error.The resulting prediction is Sydney rather than the Moscow ground truth.
  • Perception routing: Perception routing accepts an image, top candidate, uncertainty tags, recommended actions, and zoom targets to select visual tools.The routing interface explicitly exposes Zoom-Crop and OCR decisions.
  • Verification routing: Verification routing initializes an evidence pool, filters and budgets queries, selects hypotheses, and invokes text, web, and map evidence.Tool outputs are serialized before reaching the verification refiner.
  • Qualitative example: The Johannesburg example shows GeoPAVE combining left-hand driving, yellow road-edge lines, brick paving, fencing, and terrain while retaining uncertainty about decisive text.It ultimately reports Johannesburg, South Africa.
Loading 2608.29880v1…