PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?

Qianshan Wei, Yishan Yang, Siyi Wang, Jinglin Chen, Binyu Wang, Jiaming Wang, Shuang Chen, Zechen Li, Yang Shi, Yuqi Tang, Weining Wang, Yi Yu, Chaoyou Fu, Qi Li, Yi-Fan Zhang

arXiv:2604.03016v1cs.AI

TL;DR

Existing multimodal-agent benchmarks do not adequately support integrated visual and web-tool use or verify intermediate execution. Agentic-MME introduces a process-verified benchmark with unified interfaces and stepwise human annotations, finding a large performance gap on complex synergistic workflows. The benchmark identifies unreliable tool execution and redundant overthinking as key bottlenecks.

  • Problem

    Existing evaluations often separate visual tools from open-web search and focus on final answers, limiting verification of integrated tool use and intermediate behavior.

  • Method

    Agentic-MME evaluates integrated visual manipulation and open-web retrieval using unified code and function-calling interfaces plus over 2,000 human-annotated stepwise checkpoints.

  • Results

    Gemini 3 Pro (Atm) achieves 56.3% overall accuracy but only 33.3% on Level-3, while models generally struggle with advanced synergistic workflows.

  • Takeaways & Limitations

    Agentic-MME provides a diagnostic evaluation of intermediate tool intent, visual faithfulness, search behavior, and efficiency in long-horizon multimodal workflows.

  • Takeaways & Limitations

    Tasks are retained only when a state-of-the-art model can solve them along the human reference trajectory with supplied intermediate observations.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers. Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently. To address this, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities. It contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task. Each task includes a unified evaluation framework supporting sandboxed code and APIs, alongside a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis. To enable true process-level verification, we audit fine-grained intermediate states rather than just final answers, and quantify efficiency via an overthinking metric relative to human trajectories. Experimental results show the best model, Gemini3-pro, achieves 56.3% overall accuracy, which falls significantly to 23.0% on Level-3 tasks, underscoring the difficulty of real-world multimodal agentic problem solving.

1 Introduction

Agentic-MME addresses gaps in multimodal-agent evaluation by combining Visual Expansion and Knowledge Expansion in realistic workflows and verifying intermediate behavior rather than only final answers.

  • Existing benchmarks often separate visual tools from open-web search and rarely test whether both capabilities work together.
  • Over 2,000 human-annotated checkpoints verify tool intent, intermediate visual artifacts, search keywords, retrieved answers, and redundant tool usage.
  • Agentic-MME evaluates 418 real-world tasks across 3 difficulty levels and 6 domains using integrated visual manipulation and open-ended web search.
  • The benchmark unifies sandboxed code execution and structured function-calling APIs, with an AST-based tracer for auditing visual operations in code mode.
  • Current frontier models still struggle with reliable multi-step planning and precise tool execution in real-world workflows.

2 Agentic-MME

Agentic-MME constructs process-verifiable tasks requiring active image manipulation, web retrieval, or their coordination, then evaluates replayable trajectories through granular annotations and quality controls.

  • Task setup: The benchmark provides images and questions through an interface with 13 visual operations and 4 open-web retrieval tools.
  • Task Scenarios and Difficulty levels: Level 1 uses one visual operation, Level 2 uses multi-step visual and optional search workflows, and Level 3 requires intertwined multi-round visual–web interaction.
  • Task Scenarios and Difficulty levels: Level-3 tasks can require combining scattered clues, advanced computer-vision analysis, and explicit hypothesis–verification loops under severe visual ambiguity.
  • Data Collection and Annotation: Data construction combines diverse image sourcing, model-in-the-loop backward drafting, granular trajectory annotation, answer standardization, and human-model quality assurance.
  • Granular Step-wise Annotation: Each checkpoint records action intent, required operations, verifiable visual artifacts, intermediate questions, and multi-hop search evidence.
  • Quality Control and Assurance: A task is retained only when a state-of-the-art model can reach the correct answer while following the human reference trajectory with supplied intermediate observations.
  • Dataset Statistics: The dataset spans 6 domains and 35 sub-categories, with over 40% of instances requiring recovery of information occupying less than 10% of the image area.

3 Experiments

Experiments show that Agentic-MME exposes substantial gaps in multimodal agentic performance, especially on complex tasks, while process-level analyses identify tool-interface, planning, execution, and efficiency bottlenecks.

  • Main Results: 93.8% overall human accuracy remains 82.3% on Level-3, whereas Gemini 3 Pro (Atm) reaches 56.3% overall and 33.3% on Level-3.Without tools, Gemini 3 Pro reaches 7.5% on Level-3; full tool access raises this to 33.3%.
  • Main Results: Closed-source models outperform open-source models across levels, with search and retrieval planning driving the largest Level-3 gap.Thyme-rl and Deepeyesv2 obtain S scores below 5%, while Qwen3 VL-235B reaches approximately 20%.
  • Main Results: Structured Atomic APIs match or exceed generated-code interfaces, although code mode retains flexibility for arbitrary compositions and custom transformations.GPT 5.2’s overall V rises from 10.4 in Gen mode to 58.1 in Atm mode, while Gemini 3 Flash (Gen) reaches 47.7% overall.
  • Main Results: Tool invocation frequency does not ensure correctness: some models call tools often while producing incorrect visual artifacts, whereas Qwen3 VL-235B shows nearly aligned Vtool and Vtrue scores.Thyme-rl reaches Vtool = 63.3 versus Vtrue = 13.0, while Qwen3 VL-235B reaches 48.8 versus 48.4.
  • Further Analysis: Full tool access outperforms restricted settings, and combining visual and search tools produces super-additive Level-3 gains.For Qwen3 VL-235B, combining both tools reaches 19.23%, exceeding the isolated-tool results; unreliable visual tools can be counterproductive without search verification.
  • Further Analysis: Stepwise annotations improve performance but do not yield near-perfect Level-3 results, indicating that verified guidance does not eliminate continuous-execution difficulty.Agents still must write API calls, track long contexts, and avoid compounding errors even with a blueprint.
  • Further Analysis: Gemini 3 Pro’s 56.3% accuracy reflects a balance between underusing and overusing tools, while GPT-5-mini averages 12.13 calls per task yet reaches only 33.5%.Atm mode is universally more efficient than Gen mode, and DeepeyesV2’s OT = 0 accompanies 22.5% accuracy.
  • Further Analysis: Judge scores vary only slightly and human expert scores remain within the judge range, supporting consistency of the structured checkpoint evaluation.The benchmark also categorizes failures into seven modes, including reluctance to act, overthinking collapse, and unfaithful execution.

4 Related Work

Related work has advanced tool-augmented visual reasoning and multimodal search, but existing settings often isolate visual operations, treat web retrieval as peripheral, or rely mainly on final-answer correctness.

  • Tool-augmented visual reasoning: Recent tool-augmented benchmarks study active multi-tool execution and visual manipulation, but often treat open-web retrieval as peripheral.For o3/GPT-5, google_search accounts for less than 7% of calls in the cited settings.
  • Multimodal search and process-aware evaluation: Multimodal search benchmarks incorporate vision into information seeking, while process-aware work shows that final-answer correctness can mask unfaithful tool execution.These lines of work motivate evaluating intermediate behavior rather than relying solely on final outputs.

5 Conclusion

Agentic-MME evaluates multimodal agents through process-level auditing of tool use and intermediate evidence, exposing failures in complex visual–knowledge workflows.

  • Agentic-MME evaluates visual manipulation and open-web retrieval through over 2,000 human-annotated stepwise checkpoints.Its unified framework audits intermediate tool intent, visual artifact faithfulness, and execution efficiency rather than relying only on final answers.
  • Models can perform simple sequential tool-chaining but struggle with advanced synergistic tasks requiring fuzzy search and iterative cross-modal verification.The evaluation identifies unfaithful tool execution and redundant overthinking loops as precise bottlenecks.

A Process-Aware Evaluation Protocol and Scoring Details

The protocol combines final-answer correctness with automated, process-level verification of strategy and visual-evidence checkpoints from logged executions and artifacts.

  • Agents are scored using final-answer correctness and stepwise checkpoint compliance on the Strategy and Visual Evidence axes.Checkpoint matching uses only logged execution traces and generated artifacts.

A.1 S-axis Evaluation (Strategy & Tool Execution)

S-axis evaluation checks whether agents follow the required strategy and invoke appropriate tools, using parsing methods suited to different agent implementations.

  • S-axis checkpoints verify whether agents adopt the correct high-level plan and invoke necessary tools.The protocol supports heterogeneous architectures through distinct parsing strategies.
  • Structured function calls are directly parsed for predefined tool agents, while code-writing agents undergo AST parsing to extract canonical visual operations.This makes semantic intent comparable despite differences in coding style.

A.2 V-axis Evaluation (Visual Evidence Verification)

V-axis evaluation verifies whether visual operations produce decisive evidence, while the scoring protocol also measures checkpoint performance, answer accuracy, and redundant interactions.

  • 16?
  • Each visual artifact is judged against a specific intermediate question, and a checkpoint passes when any produced sub-image contains the required evidence.The any-pass rule reflects that all intermediate outputs remain available in the agent’s multimodal context.
  • S and V process scores are computed as fractions of passed strategy and visual checkpoints.The checkpoint sets are denoted CS and CV.
  • Final-answer accuracy is computed using normalized exact match.
  • The Overthink penalty measures agent interactions beyond the minimal human reference trajectory.Interactions count tool calls or executed code blocks that produce new observable artifacts, under a strict interaction budget.
  • The harness uses separate prompts for the two axes because they inspect different evidence modalities.The released prompt templates support reproducible automated judging.
  • Search judging is lenient about query wording but strict about whether the expected information appears in the returned results.The two-stage check evaluates query relevance first and retrieved answer content second.

B Tool API Specification and Implementation

The tool API tracks image states and exposes structured visual operations through OpenAI-compatible function schemas. These schemas specify operation names, parameters, defaults, and required inputs for reproducible tool execution.

  • API Structure: The API identifies each tool call by function name and JSON-encoded arguments within an OpenAI-compatible wrapper.The wrapper contains a function type, tool name, and arguments object represented as a JSON string.
  • Geometric & Spatial Tools: Geometric and spatial operations include crop, resize, rotate, flip, and edge detection, each with operation-specific parameters.Examples include normalized crop coordinates, rotation angle, flip direction, resize dimensions or scale, and edge-detection method.
  • Color & Filtering Tools: Color, contrast, and filtering operations include grayscale, autocontrast, invert, equalize, threshold, blur, sharpen, and denoise.These tools expose parameters such as cutoff, threshold value and mode, blur radius, and image index.
  • Image State: Visual tools operate on indexed images, with Image 0 denoting the original and later indices denoting processed results.The schemas require image_index for many operations, using nonnegative integer indices.

C More Dataset Statistics

The benchmark organizes cases by increasing interaction complexity, from single visual operations to short linear compositions and bookkeeping-heavy workflows. Higher levels require coordinating multiple intermediate observations rather than relying on one decisive crop.

  • Level 1: Core Visual Expansion: Level 1 tasks isolate a single visual operation that surfaces otherwise inaccessible evidence from an image.Representative cases use localized enhancement or fine-grained detail isolation without multi-step search or iterative planning.
  • Level 1: Core Visual Expansion: Level 1 examples test whether models proactively expose small or ambiguous visual cues instead of guessing from the full image.The decisive evidence may be blurry background text or a distant person’s subtle spatial attribute.
  • Level 2: Multi-Step Visual Reasoning: Level 2 tasks require several coordinated actions, with difficulty arising from multiple intermediate observations and short-horizon aggregation.The workflows remain mostly linear after the relevant regions are identified.
  • Level 2: Multi-Step Visual Reasoning: Level 2 bookkeeping may combine region selection, constrained counting, time reading, and final arithmetic or formatting.These tasks require preserving intermediate results but do not yet demand repeated external verification or deep visual-search interleaving.

D.3 Level 3: Advanced Synergistic Problem Solving

Level 3 tasks require advanced global image analysis or intertwined visual and knowledge expansion across multiple stages. Their difficulty comes from refining hypotheses and coordinating dependent evidence rather than executing a fixed linear sequence.

  • Level 3 Design: Level 3 workflows integrate multiple scattered clues, advanced computer-vision analysis, or repeated visual-search interaction.They may involve separate images, frequency-domain transformations, or explicit hypothesis–verification loops.
  • Cross-Image Grounding: Cross-image spatial grounding chains localization, spatial reasoning, transfer of a hypothesis to another image, and external verification.The case uses map and street-view crops before open-web retrieval identifies the associated state.
  • Advanced CV Analysis: Full-image structural analysis infers a latent repetitive grid, estimates its layout, and converts that estimate into a numerical answer.The decisive signal is distributed globally rather than concentrated in a local patch.
  • Cross-Domain Verification: Cross-domain verification requires isolating a weak logo, generating candidate entities, performing multi-hop retrieval, and cross-validating knowledge against another visual cue.Neither isolated visual manipulation nor blind web search is sufficient.
  • Knowledge-Conditioned Counting: Multi-step counting combines targeted crops, external team-color identification, and exclusion logic to produce the final count.The subproblems are mutually dependent, so visual observations and retrieved knowledge jointly constrain the result.

E Error Taxonomy

The error taxonomy assigns each failed trajectory one dominant bottleneck, while the released prompts standardize agent interaction across code-writing and atomic tool-calling modes. This supports reproducible diagnosis of search, visual, execution, and protocol failures.

  • Taxonomy: Agentic-MME defines seven failure modes and assigns each failed trajectory the earliest or most consequential bottleneck.The taxonomy covers missing or bad search, visual-tool failures, overthinking, tool misexecution, and post-visual perception deficits.
  • Knowledge Expansion Failures: Missing search tools and bad search queries distinguish failure to invoke retrieval from failure to convert visual evidence into an effective query.The latter includes wrong entities, omitted cues, vague queries, and unrelated attributes.
  • Visual Expansion Failures: Unfaithful or missing visual tool use distinguishes incorrect artifacts from failure to perform a required manipulation.Examples include wrong crops, incorrect rotations, excessive enhancement, direct raw-image answering, and premature search.
  • Efficiency Failures: Overthinking Collapse captures redundant exploration after sufficient evidence is available, including repeated crops, searches, and verification attempts.The failure wastes interaction budget and can derail the trajectory.
  • Execution and Perception Failures: Tool-Misexecution covers interface-level mistakes such as malformed code, invalid arguments, runtime errors, and missing file saves.PostVisual-Perception-Deficit instead occurs when a relevant artifact is produced but the required evidence is misread.
  • System Prompts: The Gen and Atm prompts standardize image indexing, action/answer separation, artifact logging, and explicit tool exposure.Gen combines sandboxed Python with search calls, whereas Atm uses a fixed atomic function-calling interface.
Loading 2604.03016v1…