Source-linked AI summary
VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
Xuanyu Zhu, Yuhao Dong, Rundong Wang, Yang Shi, Zhipeng Wu, Yinlun Peng, YiFan Zhang, Yihang Lou, Yuanxing Zhang, Ziwei Liu, Yan Bai, Yuan Zhou
TL;DR
Existing benchmarks do not adequately evaluate diverse tool use and multi-tool composition in practical visual reasoning. VTC-Bench addresses this gap with a 32-operation benchmark of 680 hierarchical problems and reference trajectories, while experiments show that current MLLMs remain limited, with Gemini-3.0-Pro reaching 51.18%.
Problem
Existing visual-tool benchmarks use limited tool-sets and simple invocations, leaving diverse tool adaptation and complex multi-operation execution insufficiently evaluated.
Method
VTC-Bench evaluates MLLM tool use with 32 OpenCV operations, 680 problems in a nine-level cognitive hierarchy, and ground-truth execution trajectories.
Results
Current MLLMs struggle with diverse tools and multi-tool composition, while Gemini-3.0-Pro achieves the highest overall score of 51.18% with tools.
Takeaways & Limitations
VTC-Bench establishes a rigorous baseline for diagnosing and developing more generalized visual agents.
Abstract
from arXiv · showhide
Recent advancements extend Multimodal Large Language Models (MLLMs) beyond standard visual question answering to utilizing external tools for advanced visual tasks. Despite this progress, precisely executing and effectively composing diverse tools for complex tasks remain persistent bottleneck. Constrained by sparse tool-sets and simple tool-use trajectories, existing benchmarks fail to capture complex and diverse tool interactions, falling short in evaluating model performance under practical, real-world conditions. To bridge this gap, we introduce VisualToolChain-Bench(VTC-Bench), a comprehensive benchmark designed to evaluate tool-use proficiency in MLLMs. To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations. This rich tool-set enables extensive combinations, allowing VTC-Bench to rigorously assess multi-tool composition and long-horizon, multi-step plan execution. For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy, each with ground-truth execution trajectories. Extensive experiments on 19 leading MLLMs reveal critical limitations in current models' visual agentic capabilities. Specifically, models struggle to adapt to diverse tool-sets and generalize to unseen operations, with the leading model Gemini-3.0-Pro only achieving 51% on our benchmark. Furthermore, multi-tool composition remains a persistent challenge. When facing complex tasks, models struggle to formulate efficient execution plans, relying heavily on a narrow, suboptimal subset of familiar functions rather than selecting the optimal tools. By identifying these fundamental challenges, VTC-Bench establishes a rigorous baseline to guide the development of more generalized visual agentic models.
1 Introduction
Existing visual-tool benchmarks use limited tool-sets and simple invocations, while practical visual agents require diverse tools and multi-operation execution plans. VTC-Bench addresses this gap with 32 OpenCV operations, 680 problems across a nine-level hierarchy, and ground-truth trajectories, revealing low performance and persistent tool-selection challenges.
- Motivation: Existing benchmarks rarely test diverse tool adaptation or chaining multiple operations into execution plans for advanced visual reasoning.This limitation can obscure models’ operational limits in practical settings.
- Benchmark Design: VTC-Bench integrates 32 OpenCV-based visual operations to support complex tool combinations and multi-step reasoning.The operations are designed to emulate authentic computer vision pipelines.
- Benchmark Design: 680 problems are organized into a nine-level cognitive hierarchy, with each problem paired with a ground-truth execution trajectory.The trajectories support evaluation of intermediate planning and final outcomes.
- Findings: 51.2% is the top benchmark performance achieved by Gemini-3.0-Pro across evaluations of 19 leading MLLMs.The experiments characterize VTC-Bench as highly challenging and report differences in tool-use gains between closed- and open-source models.
- Findings: Models struggle with unseen operations and multi-tool composition, often favoring familiar functions instead of selecting task-appropriate tools.This reliance produces operational inefficiencies and execution failures during multi-step reasoning.
2 Related Work
Prior work is moving MLLMs toward active visual tool use, including code-based tool construction, but existing evaluations remain limited in tool diversity and compositional scope. VTC-Bench is positioned to compare against these benchmarks through diverse tools, dual interaction paradigms, and progressive multi-tool tasks.
- Visual Agentic Models: Visual agent research has progressed from fixed external vision tools and interactive attention toward reinforcement learning and dynamic code generation.Code can serve as a primitive for constructing tools with loops, conditionals, and targeted visual-editing logic.
- Benchmark Comparison: VTC-Bench compares with representative multimodal benchmarks using 32 diverse tools and Code and Interface interaction paradigms.The comparison includes reference trajectories, multi-tool composition, long-horizon calling, and strict functional dependencies.
- Agentic Benchmarks: Existing visual-agent evaluations often examine basic operations such as cropping and zooming rather than complex tool composition.Traditional multimodal benchmarks primarily test static perception and reasoning.
- Benchmark Design: Its progressive design organizes 9 VQA tasks into a three-tier cognitive hierarchy from basic visual recovery to high-level logical deduction.Each task provides a reference toolchain for fine-grained diagnostic analysis.
3 VisualToolChain-Bench
VTC-Bench is a structured benchmark for evaluating multimodal agents across diverse visual tools, cognitive tasks, and multi-step execution. It combines a 32-tool OpenCV-based suite with curated tasks, execution trajectories, and metrics for answer accuracy and tool-use behavior.
- Tool Set: The benchmark curates 32 OpenCV tools into functional modules aligned with a human cognitive pipeline.The toolset is intended to address sparse tool diversity and support complex visual-task compositions.
- Task Design: VTC-Bench organizes visual-agent evaluation around 9 tasks spanning perception enhancement, quantitative estimation, and compositional visual reasoning.The hierarchy progresses from restoration, OCR, and attention focusing to measurement, color, counting, chart, math, and spatial reasoning.
- Data Collection: 680 problems are constructed from web-crawled images and repurposed open-source datasets, with tool-centric instructions and controlled visual perturbations.The construction process transforms static samples into dynamic multi-hop execution challenges.
- Evaluation Workflow: The benchmark records tool-use workflows in which agents iteratively invoke tools within a maximum-round constraint before synthesizing a final answer.The workflow overview covers two tool-use protocols during evaluation.
- Evaluation Metrics: Average Pass Rate measures answer correctness, while Effective Toolchain, Tool Call Rate, Mean Absolute Error, and Tool Usage Efficiency characterize execution behavior.Effective Toolchain is the minimal sequence of calls needed to produce the final answer; MAE and Efftool compare predicted, ground-truth, and effective chain lengths.
4 Experiment
Experiments show that VTC-Bench remains difficult for current MLLMs, with strong differences across model types and persistent inefficiencies in tool selection, composition, and verification.
- Main Results: Proprietary models generally benefit more from tool augmentation than open-source models.GPT-4o gained +9.56% under the interface setting and GPT-5.2 gained +8.53% in the code setting, while open-source models often showed limited gains.
- Tool Utilization: Tool-call rate positively correlates with Average Pass Rate, but general-purpose models achieve higher efficiency with lower tool-call rates.Specialized models cluster in a high-redundancy region, suggesting frequent tool use can neglect inherent perceptual capabilities.
- Tool Utilization: A small set of tools, including crop, zoom in, and rotate, accounts for a disproportionately large share of model calls.This concentration indicates limited diversity in tool selection and reliance on simple operations emphasized during training.
- Tool Composition: Multi-round combinations exceeding two steps outperform average accuracy, yet models still diverge from optimal ground-truth tool-calling patterns.High APR does not guarantee efficient execution, because models often use redundant, suboptimal trial-and-error sequences.
- Failure Modes: Qualitative analyses identify inappropriate tool selection and unverified reliance on intermediate outputs as two primary failure modes.Models may invoke Draw Circle or Draw Line without extracting spatial coordinates, or accept tool results without checking them against the original image.
- Prompt Ablation: Performance increases consistently as prompt information becomes more comprehensive across Direct, Weak Prompt, Strong Prompt, and Strong Prompt + GT Tools settings.The ablation evaluates Gemini-3.0-Flash, Qwen3-VL-30B-A3B, and DeepEyes.
5 Conclusion
VTC-Bench evaluates multi-step visual tool composition with diverse operations and exposes persistent deficiencies in current models' tool orchestration and perception-tool integration.
- Conclusion: VTC-Bench integrates 32 diverse tools across 9 real-world categories to evaluate multi-step tool composition in MLLMs.Experiments across 19 open and proprietary MLLMs reveal pronounced deficiencies in current tool orchestration.
- Conclusion: Models frequently rely on a narrow subset of tools, causing suboptimal execution efficacy.They also struggle with precise invocation and may over-rely on tool outputs while neglecting intrinsic perceptual capabilities.
A Implementation Details
The implementation evaluates models under zero-shot conditions using distinct deployment paths and model-family-specific generation settings.
- Evaluation Setup: Evaluations use zero-shot prompting to support fair comparison and better generalization.Open-source models run on NVIDIA H100 GPUs with vLLM, while proprietary models are accessed through official APIs.
- Generation Settings: Open-source instruct models use temperature 0.7, top-p 0.8, top-k 20, repetition penalty 1.0, presence penalty 1.5, max tokens 16,384, and seed 3407.These settings define the generation configuration for the instruct-model group.
- Generation Settings: Proprietary models are evaluated with High Reasoning mode enabled and max new tokens set to 65,536.The configuration applies uniformly to proprietary-model assessments.
- Agent Settings: Qwen-Agent uses a maximum of 20 interaction rounds.This bounds the number of agent-tool exchanges in that evaluation setup.
B Detailed Tool Set And Data Collection
The paper provides detailed descriptions of all visual tools used in the benchmark.
- Tool Set: All tools and their corresponding descriptions are detailed in Table 6.The table serves as the reference for the benchmark's visual tool set.
B.2 Data Collection
The benchmark constructs nine task categories from web, open-source, and manually synthesized data, adding targeted visual degradations and tool-dependent reasoning challenges.
- The data collection process covers nine task categories.
- Attention Focusing: Attention Focusing tests robustness to rotation and reflection through tool-assisted spatial normalization.
- Chart: Chart tasks erase ground-truth labels and require auxiliary construction plus precision sampling for extraction.
- Color: Color tasks add blur and illumination variance, requiring HSV/LAB manipulation to quantify color proportions.
- Counting: Counting tasks use occluded or overlapping objects to require a segment-and-count pipeline with morphological utilities.
- Math, Measurement, and Perceptual Restoration: Math, Measurement, and Perceptual Restoration tasks target geometric construction, calibrated dimension estimation, and recovery from haze or photon noise.Math requires auxiliary lines; Measurement targets sub-pixel precision; Restoration evaluates preprocessing for degraded scenes.
- Robust OCR: Robust OCR samples combine OCRBench and Kaggle data with compound degradations requiring binarization and sharpening before recognition.
- Spatial Reasoning: Spatial Reasoning uses Kaggle and web data to convert qualitative spatial cues into precise coordinates for positional queries.
C.1 Tool-Use Analysis Results
Gemini-3.0-Flash predicts substantially shorter toolchains than expert ground-truth trajectories, revealing premature termination and planning deficiencies.
- Most Gemini-3.0-Flash toolchains peak at 1 or 2 steps, whereas ground-truth trajectories frequently require 4 to 6 steps.
- In Color and Measure, the model usually produces one-step chains despite ground-truth toolchains peaking at 5 steps.
- The shorter-chain pattern indicates premature reasoning termination and a bottleneck in planning and orchestration.
C.2 Evaluation Metrics
The evaluation defines Tool Call Rate through toolchain length and situates it within a benchmark using OpenCV-based tools organized into logical groups.
- Tool Call Rate (TCR) is defined as the fraction of samples whose toolchain length is greater than zero.
- The TCR equation uses N for the total number of samples and L_T,i for the i-th sample’s toolchain length.
- The benchmark’s OpenCV-based tools are categorized into Geometry, Enhancement, Feature Extraction, and Drawing.The cited table describes 35 tools across these four logical groups.
E Detailed Task Example
The supplementary figures present examples across nine task categories and case studies of reasoning processes for Perceptual Restoration and Measure tasks.
- Supplementary examples cover Attention Focusing, Chart, Color, Counting, Math, Measure, OCR, Perceptual Restoration, and Spatial Reasoning.
- Additional supplementary examples illustrate OCR, Perceptual Restoration, and Spatial Reasoning tasks.
- Case studies show reasoning processes for Perceptual Restoration and Measure tasks.