PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios

Zhaochen Su, Jincheng Gao, Hangyu Guo, Zhenhua Liu, Lueyang Zhang, Xinyu Geng, Shijue Huang, Peng Xia, Guanyu Jiang, Cheng Wang, Yue Zhang, Yi R. Fung, Junxian He

arXiv:2602.23166v2cs.CV

TL;DR

Existing multimodal benchmarks do not adequately evaluate realistic, detail-rich visual workflows that require generalist agents to use multiple tools over long horizons. AgentVista introduces a 209-task benchmark spanning 25 sub-domains and 7 categories with interleaved visual and external-tool interactions. The best evaluated model, GEMINI-3-PRO, achieves only 27.3% overall accuracy, while analysis identifies visual grounding and extended tool use as key bottlenecks.

  • Problem

    Existing evaluations leave a gap in testing generalist multimodal agents on diverse realistic scenarios requiring complex, long-horizon tool interactions across modalities.

  • Method

    AgentVista combines 209 vision-centric tasks across 25 sub-domains and 7 categories with interleaved web search, image search, page navigation, and code-based operations.

  • Results

    27.3% overall accuracy is the highest reported result, achieved by GEMINI-3-PRO, while models exhibit visual-grounding and long-horizon tool-use bottlenecks.

  • Takeaways & Limitations

    AgentVista provides a practical target for tracking progress toward more reliable multimodal agents solving complex, multi-step real-world tasks.

  • Takeaways & Limitations

    Benchmark construction may reflect biases from source data and annotator decisions, affecting coverage across domains and scenarios.

Abstract

from arXiv · show

Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.

1. Introduction

AGENTVISTA addresses gaps in multimodal benchmarks by evaluating generalist agents on realistic visual scenarios requiring diverse, long-horizon, interleaved tool use. Its evaluation shows that current agents remain far from reliably solving these workflows.

  • Benchmark gaps: Existing multimodal benchmarks often emphasize individual capabilities, making it difficult to assess generalist agents combining multiple skills in long-horizon workflows.These evaluations also face a realism–difficulty trade-off when visual states or tool patterns are simplified.
  • Benchmark design: AGENTVISTA contains 209 tasks across 25 sub-domains and 7 categories, grounded in detail-rich visual states and authentic user needs.Tasks include single-image and multi-image inputs and require unique, verifiable answers.
  • Benchmark design: Each task requires long-horizon interaction that interleaves visual grounding, external information retrieval, and verification across multiple steps.A home-renovation example combines image matching, room verification, product lookup, and deterministic cost computation.
  • Evaluation: The benchmark covers web search, image search, page navigation, and code-based image-processing and programming operations.These tools represent core interaction patterns for real-world multimodal agents.
  • Evaluation: 27.3% overall accuracy is the best result in the evaluation, achieved by GEMINI-3-PRO, showing that AGENTVISTA remains far from solved.The evaluation also reports failures beginning with visual misidentification and propagating through retrieval and multi-step tool use.

2. The AGENTVISTA

AGENTVISTA is constructed as a broad benchmark of realistic, vision-centric tasks that require hybrid tool use and deterministic verification. Its pipeline progressively filters real images and user needs through expert authoring, execution checks, and repeated review.

  • 2.1. Overview of AGENTVISTA: AGENTVISTA contains 209 tasks spanning 25 sub-domains across 7 categories, built from authentic user needs and realistic visual scenarios.Examples include hardware diagnosis, constrained product selection, and route planning using images and external resources.
  • 2.2.1. Core design principles: Tasks require key evidence from real images, including small cues, multiple objects, or subtle differences that cannot be resolved by keyword search alone.This vision-centric design makes visual understanding necessary for solving each task.
  • 2.2.1. Core design principles: Tasks interleave at least two tool categories, combining visual tools with text-based retrieval before evidence is combined into an answer.The intended workflow includes operations such as image processing or image search followed by web search or page navigation.
  • 2.2.1. Core design principles: Each task has a concise deterministic target answer, while annotators verify facts against reliable sources and add time constraints when information may change.This supports simple, accurate, and more stable evaluation.
  • 2.2. Data Construction: 300k+ candidate images were filtered to 568 potential initial states, then 315 finalized tasks and 241 execution-filtered tasks before later verification.The pipeline uses agent-centric filtering, expert rewriting, reproducible tool-output checks, and two verification rounds.
  • Tools: The tool environment provides web search, page navigation, image search, and code interpretation for programming, arithmetic, parsing, extraction, and image processing.Image operations include cropping, resizing, measuring, and comparing visual regions.

3. Experiments

AGENTVISTA evaluates frontier multimodal agents on realistic, long-horizon tasks grounded in visual evidence. Results show low and uneven performance across models, domains, and input modes.

  • Evaluation Setup: AGENTVISTA evaluates 14 frontier multimodal models under a 30-turn interaction cap, using final-answer accuracy as the metric.The benchmark covers practical multimodal agents across realistic tasks.
  • Overall Results: 27.27% overall accuracy is the highest score, achieved by GEMINI-3-PRO, while 4 of 14 models score below 15%.GPT-5.2 averages 13.85 turns per task, and 5 of 14 models exceed 10 turns on average.
  • Domain Results: Model families show complementary domain strengths rather than uniform competence across realistic long-horizon tasks.GPT-5 leads several practical categories, Gemini is strongest overall, and Claude is comparatively stronger on careful reading and constraint-following categories.
  • Input Modes: 36.84% versus 23.68%: GEMINI-3-PRO performs better with multi-image than single-image inputs.Across nearly all evaluated models, multi-image accuracy is higher, while cross-image alignment remains necessary.
  • Overall Results: GEMINI-3-PRO achieves the highest overall accuracy among evaluated models, with all reported values expressed as accuracies in percent.Table 3 compares categories and single- versus multi-image input modes.

4. Further Analysis

Further analyses examine how models use tools, how tool access affects accuracy, why they fail, and whether test-time sampling improves results. Hybrid tool use helps, but visual grounding and reliable long-horizon reasoning remain bottlenecks.

  • Tool Distribution: GPT models use the code interpreter most heavily, whereas Gemini and Claude models call web search most often.Image search is used less frequently than the other tools across models.
  • Tool Distribution: Crop is the most frequent code-interpreter image operation, indicating reliance on localized visual grounding before retrieval or computation.Other image-centric operations include zooming, resizing, measuring, structured extraction, and calculations.
  • Tool Ablation: 27.27% versus 20.10% versus 18.18%: GEMINI-3-PRO performs best with the full tool suite, ahead of vision-only and no-tool settings.For CLAUDE-SONNET-4.5, full-tool accuracy is 17.70%, versus 17.22% vision-only and 13.40% for both search-only and no-tool.
  • Error Analysis: Visual misidentification is the dominant error category across models, especially when images are blurry or cues are visually subtle.Knowledge hallucination is the second most common error type, reflecting difficulty resolving long-tail facts even with web search.
  • Test-Time Scaling: 30.62% versus 21.05%: GEMINI-3-FLASH Best-of-K accuracy rises from K=1 to K=16, while Pass@16 reaches 51.67%.Random selection remains low, so the reported gains mainly come from selecting among samples rather than sampling alone.

5. Related Work

Prior multimodal benchmarks largely assess static perception or isolated capabilities, while newer agent benchmarks still leave gaps in generalist, realistic, long-horizon multimodal interaction.

  • Benchmark Scope: AGENTVISTA focuses on long-horizon, interleaved tool use grounded in real visual inputs.This emphasis targets generalist multimodal agents rather than isolated tool skills.
  • Benchmark Scope: Early multimodal benchmarks evaluate perception and visual reasoning from fixed image-text contexts without testing action selection, tool calls, or intermediate verification.Recent agent benchmarks add capabilities such as multi-step planning and web browsing, but do not fully resolve the broader evaluation gap.

6. Conclusion

AGENTVISTA is a 209-task benchmark spanning 25 sub-domains and 7 categories for realistic, ultra-challenging multimodal agent evaluation. Its results identify visual grounding and long-horizon tool use as key bottlenecks.

  • Benchmark: 209 tasks across 25 sub-domains and 7 categories require long-horizon, interleaved tool use grounded in visual evidence.Strict quality control targets vision-centric queries with unique, verifiable answers.
  • Results: 27.3% overall accuracy is achieved by GEMINI-3-PRO, the best-performing model in the evaluation.The benchmark also elicits long interaction trajectories, with GPT-5.2 averaging 13.85 tool turns per task.
  • Implications: Visual grounding and long-horizon tool use emerge as key bottlenecks for current multimodal agents.The authors position AGENTVISTA as a benchmark for tracking progress toward more reliable solutions to complex real-world tasks.

Impact Statement

AGENTVISTA provides a shared evaluation target for realistic, ultra-challenging multimodal tasks requiring long-horizon tool use grounded in visual evidence. Its broad domain coverage and checkable answers are intended to support more robust and responsible multimodal agents.

  • Scope: AGENTVISTA evaluates generalist multimodal agents on realistic, ultra-challenging tasks requiring long-horizon tool use grounded in real visual inputs.The benchmark covers seven major categories, including technology, commerce, geography, entertainment, society, academics, and culture.
  • Evaluation value: Concise, verifiable answers and a controlled tool environment enable reproducible comparisons and help identify bottlenecks in visual grounding, constraint tracking, and tool reliability.
  • Caveat: Benchmark construction may reflect biases from source data and annotator decisions, potentially affecting coverage across domains and scenarios.

A.2. Data Sources

AGENTVISTA is built from real images and real user needs collected through multiple channels, then screened for visual detail, agentic relevance, and self-contained clarity. Its controlled environment provides common multimodal tools for search, navigation, image analysis, and programming.

  • Data sources: Instances are grounded in real images and user needs, retaining only cases with sufficient visual detail and excluding text-searchable or publicly duplicable solutions.Candidates are curated from public vision-language arenas, annotator-captured scenarios, and private community forums.
  • Data sources: Public vision-language arenas contribute 284.4K diverse real-world images that are automatically filtered for visual information and fit with agentic task settings.
  • Data sources: Annotator-captured scenarios preserve practical constraints such as cluttered scenes, partial evidence, and ambiguous context while keeping tasks self-contained.
  • Data sources: Community-forum cases are rewritten into standalone tasks that preserve original intent, remove identifying information, and undergo stricter clarity screening.
  • Tool environment: The controlled tool environment includes web search, image search, webpage visits, and a stateful Python code interpreter for image processing, calculations, and data manipulation.Image inputs are pre-loaded as PIL Image objects, and generated or zoomed images can be reused in later turns.

B.2. Analysis of open-source model results.

Open-source models remain far from solving AGENTVISTA, with overall accuracy substantially below the best-performing commercial model. The benchmark’s construction emphasizes realistic, visually grounded, multi-hop tasks requiring coordinated search, browsing, and code use.

  • Results: 10.05%–12.92% overall accuracy is achieved by representative open-source baselines, versus 27.3% for GEMINI-3-PRO.The open-source models include DEEPEYES-V2-7B, WEBWATCHER-32B, and QWEN3-VL-235B.
  • Task design: AGENTVISTA tasks require complex visual content, multiple searches, at least 4–5 linked reasoning steps, and synergy between search, browsing, and code execution.Tasks are based on real-world scenarios and target verifiable, unique answers.
  • Filtering criteria: Tasks exclude OCR-only, simple-image, direct-question-answering, low-step, and non-verifiable-answer cases.Rejected answer types include plans, recommendations, subjective opinions, and long free-form descriptions.
  • Tool use: The available workflow combines visual analysis with web search, image search, page visits, and code-based image processing and calculations.Agents are instructed to analyze, reason, use appropriate tools, iterate as needed, and return answers in a required format.

C. Error type definitions.

AGENTVISTA categorizes failures across tool interaction, visual understanding, knowledge grounding, calculation, instruction following, and residual workflow errors. The taxonomy links early visual or tool mistakes to downstream failures in long-horizon problem solving.

  • Tool execution failure: Tool execution failure covers empty outputs, invalid requests, and failures to open or parse retrieved content during an otherwise planned interaction.These errors indicate the importance of robust tool use and self-checking in long-horizon workflows.
  • Visual misidentification: Visual misidentification includes misreading labels, confusing similar components, missing small indicators, or miscounting objects.Because visual evidence determines searches and constraint application, one perception mistake can redirect later steps.
  • Knowledge hallucination: Knowledge hallucination consists of unsupported facts, invented plausible details, generic rules of thumb, or standards mismatched to the current evidence.These failures reflect insufficient grounding in the multimodal context.
  • Calculation error: Calculation error covers arithmetic, unit-conversion, date-computation, and multi-value aggregation mistakes across several steps.Such errors can arise when intermediate numbers must remain consistent while tools continue to be used.
  • Instruction misinterpretation: Instruction misinterpretation involves ignoring constraints, applying the wrong condition, missing a required format, or answering a different question.
  • Others: Others groups incomplete answers, premature termination, inconsistent intermediate outputs, and results that cannot be checked in the required format.

D.1. Good Case Examples

Successful cases show agents combining visual inspection with web or image search to resolve fine-grained questions in realistic workflows. In both examples, interleaved tool use supports a verifiable final decision.

  • Sneaker Authentication: Seven tool invocations combined image search and web search to authenticate luxury sneakers from visual craftsmanship cues.The trajectory compared tongue and size tags with authentic references and validated the anomalous “A8513” sticker as a counterfeit indicator.
  • Strongest German Beer Analysis: The beer analysis used code-based visual refinement and web search to filter eligible cans and verify ABV and volume.Image processing made brand and label details readable before external retrieval confirmed the relevant product facts.
  • Sneaker Authentication: The sneaker was classified as counterfeit because its interior sticker code conflicted with authentic labeling patterns.Additional cues included mismatched tongue-tag font and stitching.
  • Strongest German Beer Analysis: Steam Brew German Red and Perlenbacher Strong tied for highest alcohol per can at 39.5 ml.Both were German-brewed, 7.9% ABV, and evaluated using a 500 ml can size: 0.079 × 500 ml = 39.5 ml.

D.2. Bad Case Examples

Failure cases reveal that agents can break at visual grounding, image processing, hypothesis selection, or constraint interpretation. Early mistakes often propagate into searches, comparisons, and final answers.

  • Karst Jigsaw Puzzle: 24 extracted regions instead of the expected 35 pieces caused the 6×6 jigsaw reconstruction to fail.Touching pieces were merged during segmentation, preventing grid reconstruction and missing-coordinate identification.
  • Target Arena Identification: Misreading an unclear floor logo caused premature commitment to St. Thomas AARC instead of the correct UNC facility.Later architecture-based searches reinforced the initial visual anchor rather than establishing a definitive identifier.
  • Pilea Root Diagnosis: The Pilea diagnosis followed a “nursery plug” or fungal explanation that conflicted with the image and overwatering history.The correct interpretation was calloused or crustified root-rot residue, but retrieval-aligned speculation displaced the visual evidence.
  • Studio Swing Prop Design: The swing-prop answer violated the hanging-illusion constraint by leaving a horizontal support visibly exposed.The supported solution hides vertical posts inside thick hemp rope, whereas the proposed design used decorative ropes around a stationary bench.
  • Authors United Window Display: Incorrect visual identification led the window-display analysis to compare bestseller durations for the wrong author set.The model failed to identify Donna Tartt and answered Anthony Doerr; All the Light We Cannot See; ∼84 weeks instead of the ground truth.
Loading 2602.23166v2…