PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu

arXiv:2610.31494v1cs.CV

TL;DR

Online agent evaluations are costly to scale, while existing offline benchmarks are fragmented and general-purpose benchmarks do not reliably reflect agentic performance. AgentVQA consolidates realistic scenarios from five domains into a manually curated offline benchmark. Its rankings track an online evaluation, but the best of 22 models reaches only 58.5% accuracy, with errors dominated by grounding and visual-state understanding.

  • Problem

    Online evaluations are costly and challenging to scale, while existing benchmarks provide limited, fragmented coverage of realistic agentic scenarios.

    Source: §1 · p. 2

  • Method

    AgentVQA manually curates 1,000 challenging multiple-choice questions from 23 datasets across five agentic domains for offline evaluation.

    Source: §1 · p. 2

  • Results

    The strongest of 22 evaluated models achieves 58.5% accuracy, and AgentVQA rankings correlate with SpatialWorld while diverging from traditional benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Errors predominantly involve grounding, perception, motion understanding, and spatial-state construction rather than task interpretation or downstream reasoning.

    Source: §5 · p. 9

    Future work should connect offline error patterns to interactive execution failures and test whether controlled-question gains translate into more reliable online behavior.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

1 Introduction

Agentic VLM evaluation lacks a scalable, comparable framework: online tests are costly, while general-purpose and domain-specific benchmarks do not adequately represent realistic agentic scenarios. AgentVQA brings five domains into one manually curated benchmark, where model scores remain low and errors center on grounded visual-state understanding.

1 Introduction

  • Online evaluations are costly and difficult to reproduce at scale, while general-purpose VQA benchmarks do not reflect agentic decisions or reliably predict agentic success.Existing domain-specific benchmarks can have narrow coverage, unrealistic inputs or action spaces, saturation, or noisy annotations.Source: §1 · p. 2
  • AgentVQA provides 1,000 manually curated questions from 23 datasets across Web Agents, Egocentric Videos, Robotics, Games, and Spatial Planning.Its single-image, multi-frame, and video-based questions support aggregate comparisons and analysis of cross-domain generalization.Source: §1 · p. 2
  • The human-centered, model-assisted pipeline standardizes heterogeneous agent tasks as multiple-choice questions while preserving their original semantics and difficulty.Only about 22% of initial candidates survive; human review catches subtle ambiguities and annotation errors missed by automated checks.Source: §1 · p. 2
  • 58.5% accuracy is the strongest model’s score among 22 evaluated models; AgentVQA rankings correlate with SpatialWorld but differ substantially from broad visual and reasoning benchmarks.The strongest model also differs across domains, so aggregate performance can obscure domain-specific variation.Source: §1 · p. 2
  • Errors are dominated by failures to ground and maintain representations of the environment, while task misunderstanding and reasoning inconsistent with perceived inputs are rare.Reported challenges include UI localization, motion interpretation, and 3D spatial reasoning.Source: §1 · p. 2

2 Related Work

Prior work spans general VQA and online or offline agentic benchmarks, but existing evaluations vary in relevance and comparability. AgentVQA addresses fragmentation by filtering and transforming examples into a standardized multiple-choice format with hard negatives.

2 Related Work

  • General-purpose VQA benchmarks assess broad visual understanding or knowledge-intensive reasoning, but their performance does not reliably correlate with success on agentic tasks.Agentic tasks may require episodic memory for streams of observations or interactive reasoning such as tool-calling.Source: §2 · p. 3
  • Online agent benchmarks capture environment dynamics but are costly and infrastructure-intensive; offline benchmarks use verifiable questions paired with visual inputs.Existing offline benchmarks are largely domain-specific and differ in answer formats, option difficulty, and sample distributions, making aggregation difficult.Source: §2 · p. 3
  • AgentVQA filters and transforms samples into a standardized, unambiguous multiple-choice format with hard negatives to support cross-domain comparison.This contrasts with prior benchmarks that typically evaluate agents within individual environments or domains.Source: §2 · p. 3

3 The AgentVQA Benchmark

AgentVQA defines agentic tasks around goal-directed interaction and organizes a balanced benchmark across five forms of agent–environment interaction. Its human-centered curation pipeline standardizes examples, filters quality issues, and manually verifies the final questions.

3 The AgentVQA Benchmark

  • Agentic tasks evaluate action selection or perceptual and reasoning capabilities needed for goal-directed interaction; the benchmark emphasizes broad domain coverage, unambiguous ground truth, and challenging examples.The benchmark contains 1,000 manually curated multiple-choice questions from 23 datasets across five domains.Source: §3 · p. 4
  • The five domains each contribute 200 questions, covering GUI navigation, robotics, egocentric planning, games, and spatial planning through images, trajectories, and videos.Coordinates, actions, spatial relations, and predicted states share a multiple-choice format while preserving the scenario observations.Source: §3 · p. 4

3.1 Data Curation and Construction

  • Curation collects candidates across 23 datasets, converts tasks to multiple choice, and uses VLM screening to flag ambiguity, inconsistent annotations, malformed states, and weak distractors.Gemini 2.5 Pro proposes hard negatives when needed; automated screening removes approximately 45% of the initial candidate pool.Source: §3.1 · p. 4
  • Human annotators reject or repair subtle ambiguities, errors, and insufficiently challenging samples, then verify every retained question has exactly one correct answer and plausible, unambiguously incorrect distractors.The final benchmark is balanced at 1,000 questions, with 200 per domain; bounding-box false positives and multiple plausible next actions motivate manual review.Source: §3.1 · p. 5, Figure 4 · p. 5

4 Experiments

Experiments evaluate 22 VLMs on AgentVQA, compare their performance and rankings across benchmarks, test agreement with online evaluation, and analyze error patterns.

4.1 Experimental Setup

  • The evaluation covered 22 VLMs; models without native video input received 32 evenly sampled frames, and responses were extracted programmatically.Prompts requested a single-letter choice, with spatial-coordinate options formatted in each model’s native convention.Source: §4.1 · pp. 5–6

4.2 Results on AgentVQA

  • 58.5% accuracy was the top score (Gemini 3.7 Flash), while Web Agents was the weakest domain at 29%.Egocentric Videos scored 59%, Games 51.5%, Spatial Planning 46%, and Robotics 38%.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • The top open model, Qwen3.8 27B (50.9%), trailed Gemini 3.7 Flash (58.5%) by 7.6%, while extended-reasoning gains varied by model.GPT-5.6-sol gained 16.8 points with thinking-high, while GLM 4.1V performed worse with thinking than without it.Source: §4.2 · p. 7
  • Removing action history decreased average performance by 9.5% on AitW and 10% on MONDAY across five Web Agents models.Figure 5 presents the action-history impact on Web Agents performance.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Rank Correlation

  • AgentVQA rankings had moderate correlation with MMMU-Pro (ρ=0.67) and GPQA-Diamond (ρ=0.59), and weak correlation with ChartQA-Pro (ρ=0.44).Rankings diverged substantially from these general visual and reasoning benchmarks.Source: §4.3 · pp. 7–8, Figure 6 · p. 7

4.4 Correlation with Online Agent Evaluations

  • AgentVQA Spatial Planning rankings correlated strongly with SpatialWorld across 12 shared models (ρ=0.895), supporting offline ranking as a lower-cost alternative.The lower correlations for other AgentVQA domains indicate that rankings depend on the capabilities being assessed.Source: §4.4 · p. 8, Figure 7 · p. 8

4.5 Error Modes

  • Gemini 3.1 Pro errors were dominated by grounding in Web Agents (78%), visual miscomprehension in Games (71%), and spatial confusion in Robotics (65%).Spatial Planning errors included viewpoint gaps (62%); Egocentric Video errors were more distributed, including motion misperception (39%), perception gaps (29%), and spatial confusion (15%). The analysis sampled incorrect predictions and manually reviewed 100 error-mode assignments.Source: §4.5 · pp. 8–9, Figure 8 · p. 9

5 Conclusion

AgentVQA consolidates 23 datasets into 1,000 manually verified multiple-choice questions across five agentic domains. Its results show substantial remaining model errors and support using offline evaluation alongside interactive environments.

5 Conclusion

  • AgentVQA combines 23 datasets into 1,000 manually verified multiple-choice questions with challenging hard negatives across five agentic domains.It enables controlled comparison of visual decision-making across settings typically evaluated separately.Source: §5 · p. 9
  • 58.5% accuracy is the best result among 22 state-of-the-art VLMs, and the analysis identifies grounding, perception, motion understanding, and spatial state construction as predominant error categories.Action history improves performance, while reasoning benefits some models but not others; neither closes the remaining gap.Source: §5 · p. 9
  • AgentVQA rankings correlate strongly with SpatialWorld but diverge from traditional VQA benchmarks, suggesting conventional benchmarks do not fully capture agentic readiness.The authors describe AgentVQA as a substantially lower-cost alternative for rapid agent-model evaluation.Source: §5 · p. 9
  • Agentic competence is better viewed as a collection of domain-specific capabilities than as a single general ability.The authors propose using offline benchmarks to identify weaknesses and direct subsequent online testing toward domains and failure modes needing closer investigation.Source: §5 · p. 9
  • Future work should connect offline error patterns to interactive failures and test whether gains on controlled questions translate into more reliable online behavior.The paper identifies this connection as an open direction for evaluating grounding and state-tracking improvements.Source: §5 · p. 9

A Dataset Selection and Curation Rationale

The dataset selection rationale prioritizes agentic decision-making, distinct domain coverage, and conversion into unambiguous offline questions. It also sets eligibility and scale criteria for source datasets.

A Dataset Selection and Curation Rationale

  • The source-dataset selection process also requires public availability, licenses permitting modification and redistribution, and sufficient scale for diverse curation.The paper presents these criteria as supporting transparency, accessibility, and adequate domain coverage.Source: A Dataset Selection and Curation… · pp. 14–15
  • AgentVQA prioritizes tasks involving action choice, consequences of action sequences, action-outcome relationships, or goal-directed planning rather than passive visual description.The paper calls this requirement a high agentic signal.Source: A Dataset Selection and Curation… · p. 14
  • The five domains span distinct skills, including GUI navigation, embodied perception, strategic decision-making, and spatial awareness.The selection aims to avoid narrow coverage and assess VLM generalization across agentic challenges.Source: A Dataset Selection and Curation… · pp. 14–15
  • Source data must be structured for conversion into multiple-choice questions with a single verifiably correct answer, enabling deterministic offline evaluation.This avoids reliance on resource-intensive online simulators.Source: A Dataset Selection and Curation… · p. 15

B Source Dataset Overview

The source-dataset overview describes how dataset inputs and outputs are represented after conversion into AgentVQA. Its hard-negative pipeline creates plausible but unambiguously incorrect choices and verifies their quality.

B Source Dataset Overview

  • Table 2 summarizes source datasets, with input and output fields describing their AgentVQA conversions and output types identifying what each selected option represents.The table descriptions and modalities follow the released data fields and original dataset definitions.Source: B Source Dataset Overview · p. 15, Table 2 · p. 16

C Hard Negative Generation Pipeline

  • The hard-negative pipeline converts tasks such as web-navigation trajectories into multiple-choice questions with plausible, unambiguously incorrect distractors.For each question, Gemini 2.5 Pro generates three negatives, including spatial near-misses and semantically similar alternatives.Source: C Hard Negative Generation Pipeline · p. 15
  • Generated negatives are checked automatically for grounding-coordinate errors and manually for plausibility and unambiguous incorrectness.The final 1,000 samples also undergo manual checks for correctness, difficulty, and ambiguity.Source: C Hard Negative Generation Pipeline · pp. 15, 17

D Definition of Agentic Tasks

AgentVQA frames perception as serving executable decisions and organizes tasks into four capability groups. Its answer parser uses prioritized rules to extract model choices and skips ambiguous outputs.

D Definition of Agentic Tasks

  • Agentic tasks require observing an environment and acting toward goals; AgentVQA groups them into action grounding, spatiotemporal reasoning, state understanding, and strategic planning.The task hierarchy distinguishes executable spatial actions, dynamic mental maps, parsing the environment for decisions, and reasoning about long-term outcomes.Source: D Definition of Agentic Tasks · p. 17

E Answer Extraction Logic

  • The answer parser first checks for an exact option letter, then applies regular-expression fallbacks for declarations such as “Answer: A” or “Option A.”Questions are skipped and marked unevaluated when no unambiguous answer can be identified.Source: E Answer Extraction Logic · p. 17

F Robustness Analysis of MCQ Format

The authors test whether multiple-choice scoring is robust to answer-option order and whether model scores exceed random guessing. Performance is stable across option permutations, and every evaluated model scores above chance.

Robustness to Option Ordering

Refuting random guessing

  • 25.0% average accuracy across constant-selection baselines matches the four-option random baseline, while all 22 models score above it.The lowest model accuracy is 28%.Source: F Robustness Analysis of MCQ Format · p. 17

G Error Modes

The error-mode analysis categorizes model failures across five agentic domains and verifies a sample of its automated annotations. It also explains how multiple-choice grounding questions avoid threshold trade-offs in open-ended metrics.

G Error Modes

  • A three-stage analysis develops domain-specific error taxonomies, assigns modes to failures with Gemini 3.1 Pro, and manually verifies 100 assignments.The taxonomy is defined from a manually reviewed, stratified subset of incorrect predictions.Source: G Error Modes · p. 18

G.1 Error Mode Definitions

  • The five domain taxonomies distinguish grounded perception and state-tracking failures from reasoning, task-interpretation, and answer-selection errors.Examples include UI grounding in Web Agents, motion misperception in Egocentric Videos, 3D spatial confusion in Robotics and Spatial Planning, and tracking drift in Games.Source: §G.1 · pp. 18–19

H Limitations of Open-Ended Grounding Metrics

  • Open-ended grounding metrics trade false positives on irregular shapes against penalizing valid actions on large elements; MCQ distractors turn coordinates into discrete choices.Distractors are designed to be incorrect while remaining plausible enough to test precision.Source: H Limitations of Open-Ended Grounding… · p. 19

I Implications and Suggestions for Practitioners

The authors propose using AgentVQA as a scalable diagnostic testbed and draw practitioner recommendations from its granular failure analysis. Their suggestions target visual grounding, inference design, domain-specific training, and pre-deployment checks.

Accelerate Model Iteration

  • AgentVQA enables rapid, deterministic cross-domain scoring, avoiding the millions of output tokens a reasoning-model judge may require for one evaluation pass.The judge generates around 2,000 tokens per decision.Source: I Implications and Suggestions for… · p. 19

Improve Visual Grounding

  • 78% of analyzed incorrect Web Agents predictions from Gemini 3.1 Pro were grounding errors, identifying UI localization as a prominent failure mode in that sample.The authors suggest UI localization as a target for improvement and recommend separating fast perception modules from expensive chain-of-thought planning.Source: I Implications and Suggestions for… · pp. 19–20

Practitioner recommendations

  • The authors recommend domain-specific curricula, adaptive inference, and AgentVQA safety gates, citing differing failure profiles and weak alignment with general VQA performance.Suggested targets include 3D spatial data for Robotics and causal reasoning for Games.Source: I Implications and Suggestions for… · p. 20

J Analysis of "Thinking Loops" as a Failure Mode

The section illustrates repetitive reasoning loops and documents prompts used for evaluation, distractor generation, and error annotation. These excerpts specify the inputs and outputs used in those procedures.

J Analysis of "Thinking Loops" as a Failure Mode

  • A reasoning example repeats the same placement description without making progress, illustrating the “Thinking Loops” failure mode.The authors report observing such cycles in some reasoning models, including GLM-4.1V-Thinking.Source: J Analysis of "Thinking Loops" as a… · p. 20
  • Automated error-mode assignments were manually checked on a random subset of 100 samples.The broader error-mode analysis describes this as a verification step across domains.Source: J Analysis of "Thinking Loops" as a… · p. 20

K Evaluation Prompt

  • The evaluation template supplies the question and relevant context, then requires exactly one answer letter; placeholders are replaced with question-specific data.Action history is omitted for questions that are not trajectory-based.Source: K Evaluation Prompt · pp. 20–21

L Hard Negative Generation Prompt

M Error Mode Annotation Prompt

  • For error annotation, the prompt receives the question type, image count, dataset, prediction, correct answer, and available reasoning before assigning a failure mode.The Robotics prompt is an example; analogous domain-tailored prompts are used for the other four domains.Source: M Error Mode Annotation Prompt · p. 23, §M.1 · p. 23, §M.3 · p. 23

Related papers

Loading 2610.31494v1…