PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces

Xinyu Geng, Yanjing Xiao, Yuyang Zhang, Hanwen Wang, Xinyan Liu, Rui Min, Tianqing Fang, Yi R. Fung

arXiv:2604.04017v1cs.CL

TL;DR

GeoBrowse targets the gap between text-only deep-research benchmarks and multimodal benchmarks that do not jointly require weak visual-cue composition and difficult web verification. It introduces a two-level geolocation benchmark with expert traces and the GATE tool workflow, and finds that coherent, level-specific plans outperform direct inference by covering key evidence more reliably and reducing integration errors.

  • Problem

    Existing multimodal benchmarks rarely require both weak visual-cue composition and BrowseComp-style multi-hop verification, while many geolocation benchmarks lack expert traces and comprehensive tool evaluation.

  • Method

    GeoBrowse provides two geolocation levels, expert-annotated evidence traces, and GATE with five think-with-image tools and four knowledge-intensive tools.

  • Results

    GATE consistently outperforms direct inference across 12 MLLMs and 3 open-source agents, with gains attributed to coherent level-specific plans rather than more tool calls.

  • Takeaways & Limitations

    Level 1 benefits mainly from image processing, whereas Level 2 relies more on evidence-driven retrieval and verification; planning improves milestone coverage and final integration.

  • Takeaways & Limitations

    Expert curation currently limits scale and introduces distributional skew, including predominantly country-level instances and uneven geographic coverage.

Abstract

from arXiv · show

Deep research agents integrate fragmented evidence through multi-step tool use. BrowseComp offers a text-only testbed for such agents, but existing multimodal benchmarks rarely require both weak visual cues composition and BrowseComp-style multi-hop verification. Geolocation is a natural testbed because answers depend on combining multiple ambiguous visual cues and validating them with open-web evidence. Thus, we introduce GeoBrowse, a geolocation benchmark that combines visual reasoning with knowledge-intensive multi-hop queries. Level 1 tests extracting and composing fragmented visual cues, and Level 2 increases query difficulty by injecting long-tail knowledge and obfuscating key entities. To support evaluation, we provide an agentic workflow GATE with five think-with-image tools and four knowledge-intensive tools, and release expert-annotated stepwise traces grounded in verifiable evidence for trajectory-level analysis. Experiments show that GATE outperforms direct inference and open-source agents, indicating that no-tool, search-only or image-only setups are insufficient. Gains come from coherent, level-specific tool-use plans rather than more tool calls, as they more reliably reach annotated key evidence steps and make fewer errors when integrating into the final decision. The GeoBrowse bernchmark and codes are provided in https://github.com/ornamentt/GeoBrowse

1 Introduction

GeoBrowse addresses the lack of benchmarks that jointly test weak visual-cue composition and evidence-grounded multi-hop web reasoning. It introduces a two-level benchmark, expert traces, and GATE, whose coherent level-specific plans outperform direct inference.

  • Motivation: Existing geolocation benchmarks often lack expert traces, filter toward salient landmarks, or incompletely evaluate image manipulation and browser tools.These limitations favor vision-only prediction or remove cases requiring subtle environmental and architectural signals.
  • Benchmark design: GeoBrowse combines weak visual-cue composition in Level 1 with BrowseComp-style multi-hop queries and entity obfuscation in Level 2.Each instance includes expert stepwise annotations specifying actionable visual fragments, evidence paths, and intended tool-use strategies.
  • Agentic workflow: GATE equips agents with five think-with-image tools and four knowledge-oriented tools for end-to-end cue extraction, retrieval, and verification.The tools include crop, rotate, pixel analysis, auxiliary lines, super-resolution, web search, web image search, visit, and code interpretation.
  • Evaluation: GATE consistently outperforms direct inference across 12 MLLMs and 3 open-source agents, while search-only or image-only tools remain insufficient.The benchmark evaluates whether agents can combine visual reasoning with open-web information seeking rather than rely on either capability alone.
  • Analysis: Agentic gains arise from coherent, level-specific plans rather than more tool calls: Level 1 favors image processing, whereas Level 2 relies on evidence-driven retrieval and verification.Trajectory analysis links improved accuracy to covering more annotated key evidence and reducing final integration errors.

2 Related Work

Prior multimodal agent benchmarks separate visual manipulation from challenging web reasoning, while geolocation datasets generally emphasize single-shot localization. GeoBrowse bridges these gaps by requiring both image reasoning and open-web verification in difficult geolocation queries.

  • Agentic Multimodal Tool Use: Existing agentic multimodal benchmarks rarely tightly couple image reasoning with web search and verification.BrowseComp-style multimodal suites derive difficulty mainly from text search, whereas think-with-image benchmarks emphasize visual manipulation without comparably challenging open-web reasoning.
  • Geolocation Benchmarks: Traditional geolocation benchmarks provide coordinate supervision or constrained matching and generally do not target multi-step reasoning or tool use.Recent datasets add human signals or limited interactions such as zoom and web search, but rarely combine globally localizable imagery with high-difficulty information-seeking queries.

3 GeoBrowse Benchmark

GeoBrowse is a two-level geolocation benchmark that combines weak visual-cue composition with knowledge-intensive, multi-hop verification. It provides expert-annotated reasoning traces and validates instances for evidence-grounded, reproducible solutions.

  • Expert annotators identify informative visual cues, collect stepwise reasoning traces, and specify tool-use strategies from permissively licensed geolocation videos.
  • The benchmark supplies content-grounded, multi-hop reasoning chains that prior geolocation datasets typically lack because they provide only coordinates or final-answer supervision.
  • Level 2 builds a dependency-critical Wikipedia hyperlink chain from the Level 1 answer and obfuscates intermediate entities so direct search cannot solve the query.
  • Level 2 requires first inferring the visual answer, then performing multi-hop retrieval and verification to reach the final entity.
  • Quality control removes universally solved or incoherent cases, while three experts validate correctness, evidence verifiability, internal consistency, reproducibility, and Level 2 ambiguity.

4 Geolocation Agentic-workflow with Tool Enhancement

GATE augments geolocation agents with image-manipulation and knowledge tools organized in an interactive ReAct-style workflow. The workflow maintains structured trajectories and image references so agents can iteratively extract, verify, and integrate evidence.

  • GATE combines five think-with-image tools and four knowledge tools to support targeted visual cue extraction, open-web retrieval, and evidence verification.
  • The image tools are Crop, Rotate, Auxiliary Lines, Local Super-Resolution, and Pixel Analysis for isolating, aligning, enhancing, and characterizing weak visual cues.
  • The knowledge suite provides web image search, web text search, URL visitation, and code interpretation for information seeking, page reading, and numerical reasoning.
  • Each ReAct step produces a Thought, an Action that invokes a tool or finishes, and an Observation returned by the environment.
  • GATE preserves generated images through an in-trajectory registry because appending paths, URLs, or embeddings would exceed bounded context and I/O budgets and make image selection error-prone.

5 Experiments

GATE is evaluated against direct inference, open-source agents, and controlled tool-use regimes on GeoBrowse. It consistently improves accuracy, with level-specific gains tied to visual processing, evidence retrieval, adaptive planning, and milestone coverage rather than call volume alone.

  • 5.2 Main Results: GATE improves GPT-4o from 23.1% to 31.8% on Level 1 and from 11.9% to 21.8% on Level 2, surpassing OmniSearch and PyVision.On the same backbone, GATE exceeds OmniSearch at 24.6% and 18.8% and PyVision at 29.6% and 13.9% across the two levels.
  • 5.2 Main Results: 48.2% on Level 1 and 34.7% on Level 2 make GATE with Gemini-3-Pro the best-performing configuration, ahead of Claude-4.5-Opus and GPT-5.GATE improves Gemini-3-Pro from 35.7% to 48.2% on Level 1 and from 22.8% to 34.7% on Level 2.
  • 5.3 Analysis: Level 1 concentrates tool use on visual preprocessing and image retrieval, whereas Level 2 shifts toward Web Text Search and page visits for evidence gathering.Crop accounts for 25.7%, Web Image Search 20.0%, and Local Super-resolution 16.4% of Level 1 calls; Web Text Search accounts for 44.3% and Visit 15.9% in Level 2.
  • 5.3 Analysis: Image Processor reaches 38.6%, only 1.1% below the Level 1 all-tools score of 39.7%, while Level 2 has a larger 3.7% coordination gap.The ablation pattern matches the benchmark design: Level 1 emphasizes image-side cue extraction, while Level 2 requires knowledge-intensive retrieval and verification.
  • 5.3 Analysis: Agentic planning adds 6.2% over fixed tool use on Level 2 versus 3.4% on Level 1 because adaptive retrieval, verification, and evidence switching matter more.Fixed policies cannot recover from early errors or top-1 retrieval noise, while agentic planning supports query refinement and tool reordering.
  • 5.3 Analysis: Correct GPT-5 and Gemini-3-Pro runs reach 78–80% milestone coverage, but their incorrect runs retain 57–59%, indicating failures during synthesis, selection, or verification.Weaker models instead show sharp milestone-coverage drops in incorrect runs, with 29% for Qwen-3-VL-32B and 15% for Qwen-2.5-VL-7B.

6 Conclusion

GeoBrowse combines expert-traced visual-first geolocation with BrowseComp-style multi-hop knowledge reasoning, while GATE supplies a unified tool suite for evaluation. The authors conclude that coherent, level-specific planning improves accuracy through milestone coverage and final evidence integration, but expert curation limits scale and geographic balance.

  • 6 Conclusion: GeoBrowse contains two levels, pairing visual cue composition with multi-hop knowledge reasoning and expert stepwise annotations.Level 1 focuses on visual-first cue composition; Level 2 adds BrowseComp-style multi-hop reasoning, and each instance includes expert annotations.
  • 6 Conclusion: GATE’s gains come from coherent, level-specific plans: Level 1 benefits from visual processing, while Level 2 relies more on evidence-driven retrieval.Agentic planning improves accuracy by hitting annotated milestones and reducing errors in final integration.
  • 6 Conclusion: Expert curation prioritizes trace quality and verifiable evidence but currently bounds dataset scale and creates geographic skew, including many country-level instances and uneven coverage.The authors propose automated candidate harvesting, model-assisted pre-annotation, selective expert review, and tighter evidence-closure filters for future scaling.

B Implementation Details

The fixed-policy implementation applies a deterministic image-processing and web-retrieval pipeline. It uses multi-scale crops, conditional orientation correction, super-resolution, ordered image search, and limited text-search and visit steps.

  • B Implementation Details: The deterministic policy follows image search with Web Text Search and Visit over URLs derived from top image results.This fixed chain is used as a comparison against adaptive agentic planning.
  • B Implementation Details: The fixed policy generates five-crop views at scales 0.80 and 0.60, applies conditional rotation correction, and enhances each view with fixed 2× super-resolution.The five views are four corners plus the center, with crop dimensions determined from the original image size.
  • B Implementation Details: Web Image Search examines seven views in fixed order, takes the top result URL, then performs text search and one visit for the first three unique URLs.Level 2 adds one extra search on the textual query while leaving the rest of the pipeline unchanged.

B.2 Details of Failure Diagnosis

GeoBrowse diagnoses incorrect agent trajectories with six ordered error types spanning perception, retrieval, evidence selection, verification, tool budgeting, and final synthesis.

  • Six ordered error types classify failures from perception and grounding through retrieval, evidence selection, verification, ordering and budgeting, and final decision synthesis.The procedure assigns one label per incorrect example according to the earliest applicable failure category.
  • E1 covers missing or non-retrievable visual entities, typically signaled by milestone hit rates below 30% or vague descriptions.Examples include road-sign language, partial place names, landmark shapes, flags, emblems, and distinctive road infrastructure.
  • E2 covers poor queries or tool choices, with many calls but few related entities and milestone hit rates around 20–45%.Typical problems include generic or excessively long queries, spelling errors, wrong keywords, and repeated unrefined searches.
  • E3 and E4 distinguish choosing the wrong branch among plausible candidates from failing to complete verification through visits or cross-source evidence.Both may show moderate milestone hit rates of 40–60%, but E4 lacks decisive verification milestones.
  • E5 identifies inefficient tool ordering or budget use, whereas E6 identifies failures to integrate correct milestones, resolve conflicts, or map evidence to the correct region.E6 typically has at least 50% milestone hits and elevated tool calls in incorrect cases.
  • The evaluation also reports annotation and judge reliability, including city-level agreement and LLM–expert agreement measures.These measurements contextualize the reliability of trajectory diagnosis and answer evaluation.

B.3 Reliability of LLM-as-Judge

The paper uses constrained LLM judging because short geolocation answers tolerate benign surface variation, while a blinded human audit assesses potential judge bias.

  • LLM-as-judge evaluation is motivated by short answers and offers scalable grading, robustness to aliases and spelling variation, and consistent normalization.The protocol can map variants such as “NYC” to “New York City” while enforcing correctness.
  • Because unconstrained LLM judges may be biased, the evaluation includes a blinded human audit with binary 0–1 correctness labels.The audit is designed to test judge–expert consistency rather than assume it.
  • 95.0% GPT-4o agreement with expert labels and κ=0.90 supports constrained LLM judgments for VQA answer correctness.The 95% Wilson confidence interval is [91.9, 97.9]; other judges obtain 91.1–94.8% agreement and κ=0.82–0.89.

B.4 Details of Annotation

GeoBrowse uses calibrated experts, evidence-grounded traces, multi-stage review, and blinded agreement tests to support annotation reliability.

  • Annotators complete guideline study, worked examples, and trial annotations before proceeding only if they meet a predefined consistency standard.Project leads review the trial annotations during calibration.
  • Each annotation records visual fragments, verification steps, and supporting sources, with every trace step grounded in image or retrieved evidence.Level 2 additionally requires that sampled-chain entities are absent verbatim from the query while preserving a reconstructable multi-hop path.
  • Every instance is independently reviewed by a second expert, with disagreements escalated to a third expert based on evidence completeness and verifiability.Adjudication is not decided by majority vote.
  • The blinded IAA test measures answer agreement, Level 2 constraint checks, and ordinal ratings of trace verifiability, completeness, and coherence.Raw agreement and Cohen’s κ are reported for these assessments.
  • Country-level agreement is near-perfect at κ=0.92 and state-level agreement is substantial at κ=0.84, while city-level strict match is 79.0% with κ=0.69.Trace-quality ratings reach κw_trace=0.59, indicating alignment alongside subjectivity in qualitative judgment.

C Prompts

The prompts operationalize GeoBrowse as a visual-and-web reasoning workflow, construct dependency-critical Level 2 queries, and assess tool-call rationality and answer quality across cases.

  • C.1 Create Level 2 query: The Level 2 query prompt maps an image-resolved root through a single knowledge-graph chain and requires each hop’s entity, linking rationale, and final gold answer.The terminal entity is defined as the answer, and the root must be identified directly from the image.
  • C.1 Create Level 2 query: Level 2 queries obfuscate intermediate entities, require every dependency-critical hop, enforce a unique concise answer, and avoid extra textual hints for the image-derived root.Skipping an intermediate hop should make the final answer unattainable.
  • C.2 Quality Control: The tool-rationality prompt evaluates non-redundancy, goal alignment, and concise, logically accurate reasoning without guessing or fabrication.These criteria judge whether a tool call is necessary, efficient, precise, non-redundant, and directly useful to the user’s objective.
  • C.3 System Prompt: The web-augmented VQA prompt requires stepwise decomposition, detailed image description, evidence-driven search, source preference and cross-checking, and completion within 10 turns.If evidence remains incomplete, it instructs the agent to state the most likely answer and remaining uncertainty.
  • D.1 Level 1 Case: The Level 1 case’s terminal answer is Yongzhou City, Hunan Province, China, illustrating a direct geolocation trajectory with a multi-hop evidence chain.The supplied case identifies the Songjiazhou Hydropower Station Dam as being in Yongzhou City, Hunan Province.
  • D.2 Level 2 Case: The South Korean Level 2 case asks for Jeju Island from the image and then for the number of OAU founding members, whose gold answer is 32.The chain links women’s diving heritage to an African capital, a Pan-African organization, and its predecessor.

D.3 Trajectory Case

This trajectory identifies Ireland from visual road and utility-pole cues, then verifies a multi-hop knowledge chain ending in Norman Lockyer’s 1869 election as a Fellow of the Royal Society.

  • D.3 Trajectory Case: Visual analysis combines yellow edge markings, a white center line, and yellow-and-silver utility-pole plates to infer Ireland.The trajectory uses image cropping and enhancement before concluding that these cues support the Ireland identification.
  • D.3 Trajectory Case: Web verification links Ireland to Dublin and Trinity College Dublin, founded in 1592.The search results confirm both the capital and the college’s founding date.
  • D.3 Trajectory Case: Ernest T. S. Walton and John Cockcroft shared the 1951 Nobel Prize, and their associated device was the Cockcroft–Walton accelerator.This establishes the scientist, collaborator, prize, and named device required by the question’s intermediate hops.
  • D.3 Trajectory Case: The landmark experiment bombarded lithium and produced alpha particles, while helium’s solar-spectrum identification involved Pierre Janssen and Norman Lockyer.These results connect the accelerator experiment to the astronomers whose observation completes the question’s scientific chain.
  • D.3 Trajectory Case: 1869 is the final answer after connecting Ireland, Dublin, Trinity College, Walton, Cockcroft, lithium bombardment, helium, and Lockyer.The completed chain is explicitly summarized from the image-derived country through the historical and scientific hops.
Loading 2604.04017v1…