PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaTooler-V: Adaptive Tool-Use for Images and Videos

Chaoyang Wang, Kaituo Feng, Dongyang Chen, Zhongyu Wang, Zhixun Li, Sicheng Gao, Meng Meng, Xu Zhou, Manyuan Zhang, Yuzhang Shang, Xiangyu Yue

arXiv:2512.16918v3cs.CV

TL;DR

Existing multimodal models may invoke vision tools even when they are unnecessary, increasing inference overhead and sometimes degrading predictions. AdaTooler-V adapts tool-use using sample-specific rewards and two training datasets; across 12 benchmarks, its 7B model reaches 89.8% accuracy on V* bench and outperforms GPT-4o.

  • Problem

    Existing models may invoke vision tools unnecessarily, which can increase inference overhead and degrade prediction quality.

    Source: §1 · p. 2

  • Method

    AdaTooler-V uses AT-GRPO to adjust reward scales by sample-specific Tool Benefit Score and is trained with separate SFT cold-start and RL datasets.

    Source: §1 · p. 2

  • Results

    Across 12 benchmarks, AdaTooler-V-7B achieves 89.8% accuracy on V* bench, outperforming GPT-4o.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AdaTooler-V is designed to use tools when they yield performance gains while optimizing both model performance and inference costs.

    Source: §1 · p. 2

    AdaTooler-V-300k is mainly built from public benchmarks, limiting coverage of real-world long-tail, noisy, and cross-domain cases.

    Source: D. Limitations and Future Works · p. 13

Abstract

from arXiv · show

Recent advances have shown that multimodal large language models (MLLMs) benefit from multimodal interleaved chain-of-thought (CoT) with vision tool interactions. However, existing open-source models often exhibit blind tool-use reasoning patterns, invoking vision tools even when they are unnecessary, which significantly increases inference overhead and degrades model performance. To this end, we propose AdaTooler-V, an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. First, we introduce AT-GRPO, a reinforcement learning algorithm that adaptively adjusts reward scales based on the Tool Benefit Score of each sample, encouraging the model to invoke tools only when they provide genuine improvements. Moreover, we construct two datasets to support training: AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL with verifiable rewards across single-image, multi-image, and video data. Experiments across twelve benchmarks demonstrate the strong reasoning capability of AdaTooler-V, outperforming existing methods in diverse visual reasoning tasks. Notably, AdaTooler-V-7B achieves an accuracy of 89.8\% on the high-resolution benchmark V*, surpassing the commercial proprietary model GPT-4o and Gemini 1.5 Pro. All code, models, and data are released.

1. Introduction

AdaTooler-V addresses unnecessary vision-tool use by adapting between text-based reasoning and tool-assisted visual inspection. It combines a sample-specific reward strategy with two training datasets, and reports strong benchmark performance.

  • Multimodal interleaved CoT lets models invoke operations such as cropping and frame extraction to inspect fine-grained visual details.Source: §1 · p. 2
  • Blind tool-use can trigger overthinking, degrade predictions, weaken focus on visual cues, and add inference cost when tools are unnecessary.Source: §1 · p. 2
  • AT-GRPO uses each sample’s Tool Benefit Score to reward tool-use when it improves performance and penalize redundant calls.Source: §1 · p. 2
  • AdaTooler-V-CoT-100k supports SFT cold start, while AdaTooler-V-300k supports RL across single-image, multi-image, and video tasks.Source: §1 · p. 2
  • 89.8% accuracy on V* bench, with AdaTooler-V-7B outperforming GPT-4o; experiments span 12 benchmarks.Source: §1 · p. 2

2. Related Work

Related work covers reinforcement-learning approaches to multimodal reasoning and the “thinking with images” paradigm, in which models use visual observations during reasoning. Existing work also motivates adaptive tool-use, as current models can invoke tools blindly.

  • Multimodal reasoning research spans RL post-training and tasks including image-based VQA, segmentation, grounding, and video reasoning.Source: §2 · p. 2
  • “Thinking with images” uses operations such as local zooming and video-frame extraction to explore visual regions and verify hypotheses.Source: §2 · p. 2
  • AdaTooler-V case trajectories show tool-assisted reasoning for single-image and video questions, but text-only CoT for a multi-image clock problem.Source: Figure 2 · p. 3
  • Prior work includes visual-tool RL, visualization as an intermediate representation, curiosity-driven pixel reasoning, and interleaved CoT for video.Source: §2 · p. 3

3. Method

AdaTooler-V adaptively chooses between text-only reasoning and visual tool use, with AT-GRPO guiding tool invocation according to its measured benefit. Its two-stage training uses curated multimodal data for supervised cold start and reinforcement learning.

3.1. Overview

  • The model uses text-based CoT when tools are unnecessary and iteratively generates thoughts, actions, and observations when visual interaction is needed.The interaction loop continues until a final answer or a predefined context-length or turn limit.Source: §3.1 · pp. 3–4
  • Training first uses multi-turn tool-interaction trajectories for SFT, then applies AT-GRPO reinforcement learning to encourage exploration beyond SFT’s pattern-matching behavior.The SFT stage aims to establish coherent trajectories with diverse and robust reasoning patterns.Source: §3.1 · p. 4

3.2. Training Data Collection

  • The training data span single-image, multi-image, and video tasks, with image samples targeting broad visual reasoning and video samples targeting temporal reasoning.Video data are intended to support understanding of event progression, frame dependencies, and outcomes involving motion and temporal causal dynamics.Source: §3.2 · p. 4
  • For SFT cold start, the authors generate CoT rationales for AdaTooler-V-300k and filter them, while RL data emphasize verifiable answers and include smaller portions of free-form, OCR, and regression tasks.Qwen2.5-VL-72B-Instruct generates the rationales, which are filtered to form AdaTooler-V-CoT-100k.Source: §3.2 · pp. 4–5

3.3. Adaptive Tool-use GRPO Training

  • AT-GRPO uses each query’s averaged tool/no-tool accuracy gap to scale a tool-use reward, encouraging use when it benefits performance and penalizing unnecessary use.The reward also varies with tool-use frequency; total reward combines it with the base reward.Source: §3.3 · pp. 5–6

4. Experiments

AdaTooler-V is evaluated across image and video benchmarks, with reported gains on high-resolution, general, multi-image, and video reasoning tasks. The experiments also specify a two-stage SFT/RL training setup.

4.1. Setup

  • Evaluation uses greedy decoding across seven image and five video benchmarks, with the model initialized from Qwen2.5-VL-7B-Instruct and trained through SFT followed by RL.Training uses AdaTooler-V-CoT-100k for SFT and AdaTooler-V-300k for RL; α is set to 0.6.Source: §4.1 · pp. 6–7

4.2. Main Results

  • 55.6% on Video-Holmes exceeds Qwen2.5-VL-7B-Instruct (27.8%) and Video-R1 (36.5%); AdaTooler-V also scores 46.7% on VSI-Bench, 54.6% on VideoMMMU, and 68.4% on MVBench using 32 frames.The reported gains are consistent across nearly all benchmarks as input-frame count increases.Source: Table 2 · p. 7, §4.2 · p. 8
  • 89.8% accuracy on V* exceeds Pixel Reasoner (84.3%), DeepEyes (85.6%), and Mini-o3 (88.2%), with a +11.3% absolute gain over Qwen2.5-VL-7B-Instruct.AdaTooler-V also reaches 74.5% on MathVista, over 6 percentage points above Qwen2.5-VL-7B-Instruct.Source: §4.2 · p. 7
  • 36.8 on MMSI-Bench and 40.3 on SPAR-Bench outperform all tested baselines on multi-image reasoning tasks.The passage relates these tasks to spatial correspondence and relational reasoning across images.Source: §4.2 · p. 8

4.3. Ablation Study

The ablations report gains from AT-GRPO over standard GRPO and show that SFT cold start supports subsequent RL. Tool-use improves results across four benchmarks, while α performs strongly across a reasonable range.

4.3.1. Effectiveness of AT-GRPO

  • AT-GRPO substantially improves performance over vanilla GRPO and SFT+GRPO, which respectively omit SFT and replace AT-GRPO with standard GRPO.The authors attribute the gains to Tool Benefit Score-based reward adjustment, which encourages necessary tool invocation and avoids redundant visual interactions.Source: §4.3.1 · p. 8

4.3.2. Necessity of the SFT

  • Skipping SFT cold start degrades performance; the authors attribute this to missing structured priors for coherent tool-interaction reasoning during early RL.They report that SFT supplies tool-use patterns and multimodal reasoning priors for subsequent RL optimization.Source: §4.3.2 · p. 8

4.3.3. Analysis of α

  • α values of 0.6 and 0.8 yield comparable, consistently strong results, while 0.2 and 0.4 bring a moderate performance decrease.The authors describe performance as relatively insensitive to α within a reasonable range.Source: §4.3.3 · p. 8

4.3.4. Effectiveness of Tool-use

  • Disabling tool interactions lowers performance across four benchmarks, including V* (89.8% to 84.4%) and VSI-Bench (46.7% to 39.9%).The authors characterize tool-use as complementary to text-based reasoning for multimodal understanding.Source: §4.3.4 · pp. 8–9

4.4. Training Curves

During RL, accuracy rises as average response length falls rapidly at first and then stabilizes; the authors relate the shorter responses to declining unnecessary tool-use.

4.4. Training Curves

  • Accuracy increases from approximately 0.60 to around 0.70 during RL, while average response length rapidly decreases initially before stabilizing.The authors attribute shorter responses to the model learning that direct text answers are more efficient than unnecessary tool-based analysis on many relatively simple images.Source: §4.4 · p. 9

5. Conclusion

AdaTooler-V combines adaptive tool-use with AT-GRPO and two training datasets. The paper reports effectiveness across twelve benchmarks and presents the approach as a foundation for future research on tool-augmented MLLMs.

5. Conclusion

  • AdaTooler-V uses a sample-specific Tool Benefit Score to modulate rewards, with AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL.Experiments across twelve benchmarks validate the approach; the authors describe it as a promising foundation for future research on tool-augmented MLLMs.Source: §5 · p. 9

A. Dataset Distribution Details

AdaTooler-V-300k spans video and image reasoning, with data distributed across general, chart, OCR, math, knowledge, spatial, logical, counting, and high-resolution tasks.

A. Dataset Distribution Details

  • 81k general-video examples target temporal comprehension and reasoning, while 24k spatial-video examples focus on motion, navigation, tracking, and path planning.The dataset also includes 33k general multi-image examples for cross-view comparison and contextual integration.Source: A. Dataset Distribution Details · p. 13
  • 18k general-image examples support foundational visual understanding, alongside 24k chart examples for data interpretation and quantitative logic and 15k OCR examples.OCR tasks involve recognizing and interpreting embedded text in signs, forms, or documents.Source: A. Dataset Distribution Details · p. 13
  • Image tasks include 42k math examples, 30k knowledge examples, and 12k static-spatial examples covering symbolic reasoning, visual commonsense, and positional inference.Math examples include formulas and geometric diagrams; knowledge examples combine world knowledge with visual cues.Source: A. Dataset Distribution Details · p. 13
  • The remaining image data comprise 12k logical, 6k visual-counting, and 6k high-resolution examples for pattern recognition, counting, and fine-grained perception.High-resolution tasks emphasize small objects and texture recognition.Source: A. Dataset Distribution Details · p. 13

B. Reasoning Examples

The section presents representative AdaTooler-V-7B reasoning examples from the V* Benchmark and MVBench.

B. Reasoning Examples

  • Representative AdaTooler-V-7B reasoning outputs are presented for both the V* Benchmark and MVBench.The examples appear in Figures 6 and 7, respectively.Source: B. Reasoning Examples · p. 13
  • The section presents representative reasoning examples generated by AdaTooler-V-7B, shown in Fig. 6 and Fig. 7, without specifying a benchmark for those examples.Source: B. Reasoning Examples · p. 13
  • Figure 7 shows an example of AdaTooler-V-7B reasoning output on MVBench.The section identifies Figure 7 as one of its representative examples.Source: B. Reasoning Examples · p. 13

C. Prompt Template for Training and Inference

Figure 8 presents the prompt template used for training and inference, and the same prompt is used for CoT annotation.

C. Prompt Template for Training and Inference

D. Limitations and Future Works

The limitations concern Tool Benefit Score estimation, reward suitability for open-ended generation, and the benchmark-derived dataset's coverage of real-world cases.

D. Limitations and Future Works

  • Tool-benefit estimation relies on one reference model, which may bias assessments of when tool use genuinely helps.The authors suggest learned benefit estimators or model ensembles for more robust ∆S predictions.Source: D. Limitations and Future Works · p. 13
  • The reward design targets verifiable multiple-choice and numerical tasks, making it less suitable for open-ended generation.Proposed directions include learned reward models, multimodal discriminators, or contrastive signals for free-form reasoning.Source: D. Limitations and Future Works · p. 13
  • Because AdaTooler-V-300k is mainly built from public benchmarks, it has limited coverage of real-world long-tail, noisy, and cross-domain cases.The authors suggest adding in-the-wild samples, hard-case synthesis, or domain adaptation to enhance generalization.Source: D. Limitations and Future Works · p. 13

Related papers

Loading 2512.16918v3…