PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation

Arka Mukherjee, Soham Roy, Kartikeya Trivedi, Shreya Ghosh

arXiv:2608.29483v1cs.CVcs.CL

TL;DR

Static-image evaluations do not capture active evidence gathering in geolocalization, motivating GeoAgent’s embodied benchmark for VLMs navigating Street View and refining predictions. Agentic navigation improves performance conditionally, but models show limited recovery from wrong priors, developed-region bias, and architecture-dependent action efficiency.

  • Problem

    Most geolocalization evaluations use static images, omitting active exploration needed to study reasoning under uncertainty in applications such as disaster response and OSINT verification.

  • Method

    GeoAgent is an agentic benchmark in which VLMs use tool calls to explore fully navigable Street View environments and iteratively refine geolocation guesses.

  • Results

    Agentic navigation improves accuracy conditionally: models benefit when initial hypotheses are reasonable, while wrong priors often persist; developed-region locations also receive more favorable performance, and efficient architectures use fewer actions.

  • Takeaways & Limitations

    Embodied navigation exposes self-correction, geographic-bias, and action-efficiency behaviors that static recognition benchmarks overlook.

  • Takeaways & Limitations

    Headline results use a fixed 8-action budget, and the nine-action Street View-like space omits strategies such as zoom, multi-step planning, and cross-modal lookup.

Abstract

from arXiv · show

Modern Vision-Language Models (VLMs) perform well above the human baseline in image geolocalization, a task critically important in disaster response, OSINT verification, and location privacy. However, most efforts to study AI behavior on the task remain limited to static image-based retrieval, classification, and predictions. We argue that faithful recreation of the task should involve embodied navigation, where a multimodal agent autonomously explores its surroundings to gather observations before submitting a prediction. To this end, we introduce \textbf{GeoAgent}, an agentic environment-based benchmark that requires agents to navigate Street View environments to refine their geolocalization through sequential reasoning. Our analysis shows that modern VLMs struggle to discern regional patterns while succeeding at country- and continent-level predictions. When compared to static image-based baselines, agentic navigation significantly improves accuracy across established metrics. We also note severe bias in a developed/developing region context across frontier model architectures and poor self-improvement capabilities given incorrect priors. Overall, our work establishes the challenges of embodied navigation and geospatial reasoning. We publicly release our code and the GeoAgent environment: https://geoagent-benchmark.github.io

1 Introduction

GeoAgent addresses the gap between static geolocalization evaluation and human-like active exploration by benchmarking VLMs in navigable Street View environments. It frames the study around whether exploration improves accuracy, whether models show geographic bias, and how architectures use actions.

  • Significance: The benchmark is motivated by applications including disaster response, OSINT verification, autonomous navigation, robotics, and location-privacy risk assessment.The authors emphasize understanding model limits to help prevent misuse.
  • Motivation: Static-image evaluations test recognition but omit active exploration, limiting assessment of reasoning under geolocation uncertainty.Agents cannot seek views containing disambiguating features when evaluation fixes the observed images.
  • Benchmark: GeoAgent is an open-source, modular benchmark where VLMs use tool calls to explore fully navigable Street View environments before making a location guess.The task separates exploration actions from final geospatial reasoning.
  • Research questions: The study asks whether active exploration improves accuracy, whether improvement depends on initial hypotheses, whether geographic bias exists, and how architectures differ in strategy and efficiency.These questions are organized as RQ1, RQ2, and RQ3.
  • Benchmark: The benchmark contains 1,200 navigable Street View locations across 100 cities, spanning developed and developing regions with recognizability-based difficulty tiers.Figure 2 presents the collected data and the benchmark’s navigable interface.

2 Related Work

Prior geolocalization work largely treats the task as static-image prediction or passive observation, while GeoAgent evaluates embodied exploration in interactive 3D Street View environments. It therefore connects VLM reasoning with agentic navigation while distinguishing its setting from existing web and geospatial agents.

  • Vision-based geolocalization: Early geolocalization systems framed planetary-scale prediction as classification, while specialist models such as PIGEON and GeoLocSFT operated on static images.These approaches achieved strong performance but provided limited interpretability.
  • Vision-based geolocalization: VLM-based methods introduced explicit reasoning over cues such as signage and vegetation, hierarchical prediction, localizability assessment, and reinforcement-learned geospatial thinking.The cited methods include GeoReasoner, GLOBE, Geo-R1, and GeoGuess.
  • Passive evaluation: Video-based evaluation adds multiple views but remains passive because the observer cannot act to resolve ambiguity.The related approaches remain confined to analyzing fixed views.
  • Agentic evaluation: General web-agent benchmarks study autonomous interaction in 2D environments through tasks such as DOM manipulation, memory-augmented prompting, planning, and subpage traversal.Examples include WebArena, Mind2Web, Synapse, RealWorldWebAgent, and WebWalker.
  • GeoAgent: GeoAgent differs by formalizing movement and rotation within 3D Google Street View environments to evaluate active uncertainty reduction through physical exploration.The paper presents this as a new paradigm for assessing spatial intelligence.

3 Method

GeoAgent evaluates VLMs that navigate Street View through tool calls, iteratively refine guesses, and are scored with complementary geographic and behavioral metrics. The benchmark uses a stratified global dataset, fixed-action and baseline conditions, human comparison data, and six leading VLMs plus Geo-R1 7B.

  • Task setup: Agents autonomously inspect street signs, vegetation, and architecture, explain observations, and refine a geolocation guess through tool calls.Each round consists of rendered screenshots, model-generated structured actions, and tool execution.
  • Task setup: The action budget is the maximum number of actions in a round; agents may terminate early with GUESS or are force-terminated when the budget ends.This makes exploration a bounded guess-refinement procedure.
  • Data collection: The dataset begins with 100 globally distributed cities selected by development status and visitor-based recognizability tiers, then retains 1,200 navigable samples from 10,000 sampled coordinates.Examples range from Tier 1 cities such as New York City and Paris to Tier 5 cities such as Butare and Imphal.
  • Evaluation metrics: Geolocation quality uses great-circle Haversine distance between predicted and ground-truth coordinates, reporting both mean and median because catastrophic failures create skew.The distance formula uses latitude and longitude in radians and Earth’s radius R = 6371 km.
  • Evaluation metrics: Improvement Rate measures how much the distance to the correct location changes from the initial guess at action 1 to action k.Positive values indicate movement closer to the correct location, analyzed separately by final city correctness.
  • Evaluation metrics: The study also analyzes characteristic geographic vocabulary with TF-IDF, concatenating observation and reasoning text by round and country.Terms are preprocessed and extracted as unigrams and bigrams.
  • Baselines and conditions: Static 4-view and 8-view baselines, random guessing, random walks, and unlimited-step agents isolate visual coverage, model-guided exploration, and action-budget effects.The 4-view baseline samples four headings, while the 8-view baseline adds views after one forward move.
  • Models and human baseline: The evaluation tests six leading VLMs plus Geo-R1 7B through official APIs or local hosting, using temperature 0.3 when available, and compares against six annotators completing 380 rounds.Annotator performance varies substantially, with mean distances ranging from 1,720 to 3,751 km.

4 Results

Agentic navigation generally improves geolocalization over static and random-walk baselines, but gains depend on model, initial hypothesis quality, and region. Models remain stronger at broad geographic prediction than fine-grained city localization, with systematic developed-region bias and limited recovery from incorrect priors.

  • Gemini 3.0 Flash leads with 5 km median Haversine distance and 50.1% city accuracy, while Claude Haiku 4.5 records 688.3 km median distance.
  • 52.4–81.2% country accuracy versus 14.3–50.1% city accuracy shows stronger broad-region recognition than last-mile localization.The paper attributes the gap to macro-level cues being easier to use than municipal infrastructure, local signage, and regional architectural variants.
  • 26.8% relative increase: Gemini 3.0 Flash city accuracy rises from 39.5% with 4-view input to 50.1% under agentic navigation.Its mean Haversine distance also falls from 1446.79 ± 74.82 km to 1099.38 km.
  • Improving VLM geoguessing performance through embodied navigation: Additional actions usually help correct final guesses but rarely recover incorrect priors, and Gemini 3.0 Flash exhibits a collapse on incorrect city guesses beyond five actions.Removing the action cap does not restore recovery, consistent with poor self-correction in sequential reasoning.
  • 847 km median initial distance for successful rounds versus 2,341 km for failed rounds indicates that exploration refines reasonable hypotheses rather than rediscovering locations.A logistic regression predicts final city accuracy from initial distance with β = −0.43, p < 0.001.
  • Agent Behavior Analysis: Agentic exploration improves mean distance over random walks for functioning models by 466–1079 km, while Geo-R1 7B is the sole model performing worse than random movement.Geo-R1 7B emits unparseable actions in 87.3% of turns and never navigates effectively.
  • Agent Behavior Analysis: Developed regions receive 5.95 versus 5.65 exploration actions, while their mean Haversine distance is 50–103% lower and average accuracy is 2.07% higher.Gemini 3.0 Flash has a −25.9% country-accuracy gap under agentic navigation; Gemma 3 27B is the exception for city accuracy.
  • Agent Behavior Analysis: GPT-5 Mini completes tasks in 4.20–4.95 steps, whereas Llama 4 Scout uses 7.48–7.57 steps, mostly rotating around the spawn point.Taking more screenshots helps only up to a point before performance plateaus or slightly degrades.

5 Discussion

Active exploration improves geolocation mainly when agents begin near the correct location; incorrect initial hypotheses often persist or intensify. Model architecture also shapes action efficiency, while geographic bias favors developed-region locations.

  • Active exploration substantially reduces Haversine distance when the initial hypothesis is reasonably accurate.When the initial guess is far off, additional exploration often fails to recover and may reinforce incorrect priors.
  • VLMs are often better at confirming plausible hypotheses than recovering from incorrect ones.Established mistakes in context largely influence outcomes.
  • Developed-region locations receive systematically better geolocation performance than developing-region samples.The disparity appears in mean Haversine distances, failure modes, and the regional vocabulary used by models, despite developing-region imagery having newer and marginally higher quality data.
  • Exploration length and reasoning verbosity do not directly correlate with higher success.
  • Model architecture and pretraining strongly affect action efficiency under a limited exploration budget.Gemini 3.0 Flash and GPT-5 Mini combine strong accuracy and low median distances with substantially fewer actions, whereas Geo-R1 7B does not improve with agentic navigation because of tool-calling limitations.

6 Conclusion

GeoAgent introduces an embodied, agentic web-exploration task for VLM geolocation. Its analyses expose self-correction failures, inefficient exploration, and developed/developing-region bias that static image benchmarks overlook.

  • GeoAgent requires VLMs to perform embodied agentic web exploration for geolocation.
  • The benchmark reveals limited self-correction when initial guesses are incorrect.
  • GeoAgent exposes poor exploration efficiency in models such as Claude Haiku 4.5 and Llama 4 Scout.
  • The analyses identify significant bias between developed and developing regions.The paper argues for improved pre-training for embodied geospatial navigation and alternate evaluation angles beyond static image recognition.

Limitations

GeoAgent’s evaluation is bounded by a fixed action budget, a restricted navigation action space, a limited human baseline, and non-comprehensive geographic coverage. Its reliance on commercial Google APIs also creates reproducibility concerns.

  • The headline evaluation uses a fixed 8-action budget, although adaptive budgets could reveal different exploration regimes.Mean Haversine distance plateaus beyond action 5, and unlimited-action reruns show no consistent gains across three models.
  • The nine-action space does not represent every real-world navigation strategy.It omits strategies such as zoom, multi-step planning, and cross-modal lookup.
  • The six-annotator human baseline would be strengthened by comparison with expert GeoGuessr players.
  • The 1,200-location sample across 100 cities is not intended as comprehensive geographic coverage.The value of GeoAgent lies in its agentic, embodied evaluation paradigm rather than absolute geographic coverage.
  • Dependence on Google Street View and the Google Geocoding API creates reproducibility concerns because the commercial, rate-limited data sources can change or be deprecated.Released metadata and per-action logs are provided to promote reproducibility.

Ethical Considerations

GeoAgent uses structured, memory-based prompts and sequential visual reasoning in Street View environments. Its findings also show that incorrect trajectories can remain stable or diverge despite additional exploration, raising concerns about reliable embodied geolocation.

  • GeoAgent raises privacy and misuse concerns because agentic geolocalization could support surveillance or unauthorized location inference from personal media.The system itself uses only publicly available Street View imagery with blurred private identifiers, but unconstrained image sources could lower the barrier to misuse.
  • The dataset inherits Street View’s geographic coverage bias, potentially amplifying performance disparities across nations.The paper recommends safeguards such as restricted API access and geofencing alongside more geographically balanced data collection.
  • The prompt combines task framing, dynamic state, up to five prior observations, and the available action space.
  • Structured JSON responses require observations, reasoning, confidence, action, and a city-country location guess.The Google Geocoding API converts the guessed location into coordinates to avoid LLM-induced coordinate-calculation hallucinations.
  • For correct city guesses, all models monotonically approach the target, with GPT-5 Mini reaching below 500 km by action 3.
  • For incorrect guesses, models show minimal distance improvement regardless of exploration effort.GPT-5 Mini diverges after action 5, while Claude Haiku 4.5 and Gemma 3 27B maintain relatively stable incorrect estimates.

C Action-Outcome Correlation Analysis

Reasoning length and movement are associated with better geolocalization, while additional exploration helps broad localization more than city-level disambiguation.

  • Reasoning length correlates positively with continent, country, and city accuracy, and negatively with distance.The reported correlations are ρ = 0.140, 0.137, 0.090, and −0.142, respectively.
  • Reasoning length has the strongest individual association with points at ρ = 0.241.
  • Total actions correlate positively with continent and country accuracy but show weak association at the city level.Total-action correlations are ρ = 0.124 and ρ = 0.118 for continent and country accuracy; city-level correlations are Pearson r = −0.009 and Spearman ρ = 0.055.
  • Move actions associate positively with all outcomes and more strongly with points than rotation-heavy exploration.Llama 4 Scout averages 6.14 rotations and 0.41 moves per round, whereas Gemini 3.0 Flash averages 3.57 moves and 0.39 rotations.

E Unlimited Action Step Exploration

Unlimited actions with explicit self-reflection test whether the action cap explains failure modes, but generally produce worse or minimally changed localization at much higher reasoning cost.

  • Unlimited-action exploration lets three models continue after a sixth-action reflection prompt until they submit a guess.The variant uses unbounded context so prior observations are never truncated.
  • GPT-5 Mini’s mean distance worsens from 899.99 to 1218.41 km as actions increase from 4.94 to 10.13.
  • Llama 4 Scout’s actions expand from 7.55 to 22.99 while mean distance worsens by 76%, from 1779.45 to 3136.53 km.
  • Only Gemma 3 27B improves distance, from 1689.33 to 1577.65 km, with nearly unchanged action count.The action count changes from 5.96 to 5.36, suggesting the gain is driven by the reflection prompt rather than additional exploration.
  • Longer reasoning traces do not translate into more accurate localization, while inference cost rises by an order of magnitude.These findings support the conclusion that the confirmation-bias signal is not an artifact of the action cap.

G Cost and Efficiency Analysis

The benchmark’s efficiency analysis compares behavioral outcomes and operating costs, finding large model- and setup-dependent cost differences without proportional accuracy gains from unlimited exploration.

  • Claude Haiku 4.5 costs $187.60 total and is the worst-performing closed model, while Gemma 3 27B costs $2.33 and beats Claude on city-level agentic accuracy.
  • Reasoning length is positively associated with accuracy, but the reported correlations remain modest across behavioral features.Table 6 reports significant correlations, including reasoning length with continent, country, and city accuracy.
  • GPT-5 Mini lies on the efficient frontier, costing $39.43 while delivering the strongest mean Haversine distance and country accuracy.
  • The 8-action design preserves the benchmark’s diagnostic signal at roughly an order-of-magnitude lower API spend than an unconstrained loop.

H Data Quality Analysis

Quality checks find no developed-region advantage in retained Street View imagery, supporting the interpretation that the developed/developing performance gap reflects model behavior rather than image artifacts.

  • Developing-region panoramas are newer and marginally higher in Google quality score, with indistinguishable sharpness.Median image age is 2.51 versus 3.09 years; quality differs at p = 0.030, while sharpness has p = 0.70.
  • Across recognizability tiers, Figure 11 shows no monotonic degradation in capture date, quality score, or sharpness.
  • All retained samples had navigable Street View coverage sufficient for the 8-action budget.
  • The authors attribute the developed/developing performance gap to model behavior rather than data availability or image-quality artifacts.

I Why Geo-R1 Degrades Under Navigation

Geo-R1’s apparent degradation under navigation is primarily a tool-calling failure: malformed actions prevent meaningful exploration, leaving its agentic condition with less visual information than static baselines. The broader evaluation also identifies geographic and regional-performance asymmetries that motivate careful interpretation of benchmark results.

  • Comparison with baselines: 2549.0 km was Geo-R1’s agentic distance, exceeding its 2350.1 km 4-view and 2154.4 km 8-view static-baseline distances.Geo-R1 was the only evaluated model whose agentic performance fell below its own static baselines.
  • Tool-calling failure: 87.1% of Geo-R1’s turns produced action strings the environment could not parse, preventing valid action selection.Geo-R1 copied the pipe-delimited legal-action menu instead of emitting one action.
  • Tool-calling failure: Geo-R1 executed only 5 MOVE_FORWARD actions, 1 ROTATE_LEFT, and 43 RETURN actions across its audited agentic runs.The resulting behavior provided almost no actual navigation.
  • Comparison with baselines: Geo-R1’s agentic condition effectively became a single-view evaluation from the spawn heading, with less visual information than the 4-view and 8-view baselines.This explains why its agentic score fell between its random-walk and static scores.
  • Regional performance: Every model collapsed in Melanesia/Polynesia, with 10,500–11,950 km mean error and no more than 14% country accuracy.Northern/Western Europe and Northern America remained under 1,100 km for stronger models.
  • Geographic bias: Developing-region samples produced higher mean Haversine distances and worse failure modes despite relatively high continent and country accuracies.The reported bias is attributed to model behavior rather than sharper or lower-quality imagery.
Loading 2608.29483v1…