PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Reasoning through Tool-supervised Reinforcement Learning

Qihua Dong, Gozde Sahin, Pei Wang, Zhaowei Cai, Robik Shrestha, Hao Yang, Davide Modolo

arXiv:2604.19945v1cs.CV

TL;DR

Complex visual reasoning remains difficult for MLLMs because effective tool invocation strategies are not well established. ToolsRL addresses this with ground-truth-derived tool supervision and a two-stage reinforcement-learning curriculum. Across document, spatial, and chart/table tasks, it reports strong performance, stable training, and stronger tool-use patterns without expensive curated trajectories.

  • Problem

    MLLMs still lack effective, scalable ways to learn when, how, and why to invoke visual tools for complex visual reasoning.

  • Method

    ToolsRL first trains tool manipulation with ground-truth-derived, per-tool rewards, then optimizes answer accuracy with GRPO while allowing learned tools.

  • Results

    ToolsRL shows strong empirical performance across rotated document analysis, high-resolution image understanding, and chart comprehension tasks.

  • Takeaways & Limitations

    The curriculum yields more stable training, higher accuracy, and stronger visual tool-use patterns without requiring expensive curated tool-use trajectories.

Abstract

from arXiv · show

In this paper, we investigate the problem of how to effectively master tool-use to solve complex visual reasoning tasks for Multimodal Large Language Models. To achieve that, we propose a novel Tool-supervised Reinforcement Learning (ToolsRL) framework, with direct tool supervision for more effective tool-use learning. We focus on a series of simple, native, and interpretable visual tools, including zoom-in, rotate, flip, and draw point/line, whose tool supervision is easy to collect. A reinforcement learning curriculum is developed, where the first stage is solely optimized by a set of well motivated tool-specific rewards, and the second stage is trained with the accuracy targeted rewards while allowing calling tools. In this way, tool calling capability is mastered before using tools to complete visual reasoning tasks, avoiding the potential optimization conflict among those heterogeneous tasks. Our experiments have shown that the tool-supervised curriculum training is efficient and ToolsRL can achieve strong tool-use capabilities for complex visual reasoning tasks.

1. Introduction

Visual reasoning in MLLMs remains less explored than text reasoning, especially for complex tasks requiring visual transformations and precise localization. ToolsRL addresses this gap with directly supervised visual-tool learning followed by task-accuracy optimization, achieving strong performance across diverse visual reasoning tasks.

  • Complex visual tasks such as rotated-text interpretation and small-object localization exceed the adequacy of text-only reasoning.
  • Open-source MLLMs still struggle to learn when, how, and why to invoke visual tools effectively.
  • SFT-based tool learning is costly because it requires manually constructed expert trajectories and careful curation to avoid overfitting.
  • Existing RL methods often provide only final-outcome or generic tool-use rewards, producing infrequent calls and incoherent multi-step tool chains.
  • ToolsRL combines task-accuracy rewards with direct supervision for zoom-in, rotate, flip, draw-line, and draw-point tools using easily collected signals.Examples include bounding boxes for zoom-in and the underlying rotation degree for rotate.
  • The two-stage curriculum first masters tool manipulation and then optimizes task accuracy while allowing tool calls, avoiding conflict between heterogeneous rewards.
  • ToolsRL demonstrates strong empirical performance on rotated document analysis, high-resolution image understanding, and chart comprehension.

2. Related Work

Multimodal tool-use research spans training-free prompting, SFT-only, SFT-then-RL, and RL-only approaches. These methods trade off supervision cost, scalability, and reinforcement-based optimization, motivating ToolsRL’s direct tool supervision and staged training.

  • Visual Tool-Use in Multimodal Language Models: Multimodal tool-use methods expose callable visual tools such as zoom and drawing through training-free, SFT-only, SFT-then-RL, or RL-only recipes.
  • Visual Tool-Use in Multimodal Language Models: Training-free prompting can improve localization and counting, but its performance is limited and depends on strong base models.
  • Visual Tool-Use in Multimodal Language Models: SFT-only and SFT-then-RL approaches learn explicit tool-use patterns but require expensive, task-specific expert trajectories or supervised demonstrations.
  • ToolsRL differs by injecting tool supervision before QA-task training, using tool-specific rewards to supervise tool usage.
  • Visual Tool-Use in Multimodal Language Models: RL-only methods avoid supervised demonstrations by learning tool strategies from reward signals, including end-to-end or render–execute–judge feedback.

3. Method

ToolsRL frames visual tool use as a sequential decision process and trains it with task-specific supervision, tool rewards, and a two-stage curriculum. The method covers zoom-in, orientation, and drawing operations, then transfers mastered tool use to answer prediction.

  • Problem formulation: ToolsRL models visual tool use as a finite-horizon process where each action either applies a tool to an image or ends the episode with an answer.The policy observes the question, current image, and trajectory history at each turn.
  • Visual tools: The framework uses zoom-in, rotate/flip, and draw tools to support region selection, orientation correction, and coordinate-based annotation.The tools are native operations applied to images in the trajectory history.
  • Tool-supervised rewards: Tool supervision supplies per-state rewards computed from model tool calls and task-specific ground truth, including box overlap, orientation correctness, and coordinate similarity.Draw rewards use Hungarian matching to maximize total similarity between predicted and ground-truth lines or points.
  • Tool-supervised rewards: Zoom-in rewards emphasize recall over precision by weighting false negatives at 1.0 and false positives at 0.1.The modified F1-style reward evaluates pixel-level overlap between predicted and ground-truth box masks.
  • Training curriculum: The two-stage curriculum first optimizes tool accuracy with ground-truth-derived rewards, then optimizes final answer accuracy while allowing the model to use mastered tools.Stage 1 combines global trace evaluation with answer-conditioned tool evaluation to balance exploration and task relevance, alongside format reward.
  • Training curriculum: Training directly on answer accuracy from the start causes text-based reasoning preferences, whereas curriculum training improves answer accuracy after tool-use patterns are internalized.The paper reports ablations comparing the curriculum with combined-reward training.

4. Experiments

ToolsRL is evaluated across document, spatial, and chart understanding using benchmark comparisons, ablations, and tool-use analyses. The experiments show strong accuracy, broader and more frequent tool use, flexible tool composition, and improved behavior from the two-stage curriculum.

  • Main Results: ToolsRL consistently achieves strong performance across document, spatial, and chart understanding benchmarks, including 77.3% on DocVQA-RF and 61.4% on InfoVQA-RF.It also improves over Mini-o3 by 4.3 and 12.8 points on V-Star and InfoVQA-Res, respectively.
  • Ablation Studies: The two-stage curriculum outperforms alternative reward strategies by combining tool-supervised learning in Stage 1 with answer-accuracy optimization in Stage 2.Answer reward alone improves task outcomes, but conditional or non-curriculum tool rewards produce mixed or inconsistent gains.
  • Ablation Studies: Reducing the zoom-in false-positive penalty from 1.0 to 0.1 raises VisualProbe accuracy from 42.9% to 46.3% and average tool calls from 2.13 to 3.20.Training Stage 1 only on augmented images raises DocVQA-RF accuracy from 67.1% to 79.4% while reducing average tool calls from 6.98 to 4.26.
  • Native Tool Support and Usage: ToolsRL supports zoom-in, rotate, flip, draw-line, and draw-point tools and averages 3.4 calls per training sample, while most prior methods average ≤1 call.Across benchmark categories, composite tool-use ratios reach 82–99%, indicating flexible combinations of multiple tools.
  • Comparison with Different Training Settings: The curriculum produces multi-step trajectories for visual search, chart verification, and composite reasoning, whereas accuracy-only training often skips tools and tool-conditioned rewards can yield noisy or redundant calls.Representative trajectories include iterative zooming, point marking, and combined zoom-in plus point-drawing operations.

5. Conclusion

ToolsRL uses a two-stage curriculum to separate tool mastery from answer optimization, and reports more compositional and precise tool use with stable, accurate, generalizable performance.

  • ToolsRL decouples tool mastery from answer optimization through a two-stage tool-supervised reinforcement-learning curriculum.
  • The curriculum yields more stable training, higher accuracy, and stronger visual tool-use patterns than existing methods across document, spatial, and chart/table understanding.
  • ToolsRL uses tools more compositely, effectively, and precisely than DeepEyes in qualitative comparisons.

C.1. Augmented Document Datasets

The document datasets apply controlled rotation and flipping to create orientation-robust training and evaluation benchmarks, while InfoVQA-Res uses resolution-based image selection and resizing.

  • The training and evaluation data follow the same augmentation pipeline.
  • The same rotation/flip distribution is used for augmented DocVQA training and DocVQA-RF and InfoVQA-RF evaluation images.Images are rotated by 90°, 180°, or 270° or flipped horizontally or vertically, with probability 0.7 and uniform sampling.
  • InfoVQA-Res selects images whose maximum edge exceeds 1024 pixels and resizes them to a maximum dimension of 512 pixels while preserving aspect ratio.

C.2. Synthetic Chart Datasets

The synthetic chart datasets provide precise coordinate supervision for drawing tools across point-reading and relational counting tasks, using programmatically generated chart metadata and tool annotations.

  • Read-Value and Compare-and-Count are synthetically generated to provide unambiguous ground-truth supervision for drawing tools.Programmatic generation preserves precise data-point locations and values.
  • Read-Value: Read-Value asks for labeled points’ x-coordinates, y-coordinates, or full coordinates and supervises lines drawn from points to the corresponding axes.Both data coordinates and pixel coordinates are stored for each labeled point.
  • Compare-and-Count: Compare-and-Count asks how many points satisfy relational conditions relative to a reference point, including single-axis, two-axis, and mixed comparisons.
  • Stage 1 supervision: Stage 1 examples include chart Read-Value with reference lines and Compare-and-Count with marked points and a threshold line.

D.1. Zoom-in: ModF1 Reward and Ablation

ToolsRL uses reward designs that favor safe zoom coverage and informative drawing feedback, addressing sparse or misaligned signals for visual tool learning.

  • Zoom-in: Zoom-in uses ModF1 with wfp=0.1 and wfn=1.0 because covering the region of interest matters more than producing a tight bounding box.Generous crops containing the ground-truth box receive full credit, while missing target area is penalized more heavily.
  • Zoom-in: Standard F1 can score a generous crop low when false-positive area dominates, whereas ModF1 assigns full reward when the crop contains the ground-truth region.
  • Draw rewards: Continuous drawing rewards scale with distance and provide partial credit for near misses, unlike discrete rewards that only credit predictions inside the tolerance window.
  • Draw rewards: 0.23 calls per sample with discrete reward rose to 0.643 mark-point calls after switching to continuous reward.The paper attributes this gap to the continuous signal’s more informative feedback for drawing exploration.

D.3. Rotate and Flip: Training with Mixed Orientations

Stage 1 should use only rotated or flipped documents: mixing canonical samples lets the model answer as if images were upright, while augmented-only training encourages tool use. The section also contrasts this curriculum with reward designs that provide limited or no guidance about tool trajectories.

  • Mixed-orientation training: Augmented-only Stage 1 training forces the model to use Rotate or Flip tools before answering, preventing shortcut-based reward hacking.Including canonical documents encourages direct answer prediction under an upright-image assumption.
  • Mixed-orientation training: Mixing original and augmented documents encourages always predicting answer index 0, whereas augmented-only training removes this shortcut and yields higher reward.
  • Reward baselines: Accuracy Reward Only optimizes answer correctness without explicit tool signals, while Tool-Conditioned Reward adds a scalar bonus for tool use on correctly answered trajectories.
  • Reward baselines: The Tool-Conditioned Reward baseline rewards tools only when the final answer is correct, without specifying which tools to invoke or how to structure trajectories.

E.2. Training Dynamics vs Tool-Conditioned Reward

Stage 2 comparisons show that prior tool supervision strongly affects tool-use frequency and efficiency. Global-only supervision produces excessive actions, answer-conditioned supervision limits exploration, and the combined curriculum balances both.

  • Stage 2 dynamics: Accuracy Reward Only almost never invokes tools, while Tool-Conditioned Reward increases tool usage only modestly during Stage 2.The compared settings use the same data, prompts, and answer-accuracy objective; they differ in whether Stage 1 tool supervision was used.
  • Stage 1 ablations: Answer-conditioned Stage 1 supervision yields relatively low tool counts because the agent invokes tools mainly when they directly affect the final answer.This conservative behavior limits exploration.
  • Stage 1 ablations: Global-only Stage 1 supervision produces very high call rates and redundant actions because every trajectory tool call is rewarded regardless of usefulness.
  • Stage 1 ablations: The full ToolsRL curriculum combines global and answer-conditioned rewards, promoting early exploration while shaping more efficient, task-relevant tool use.
  • Prompt and toolbox design: Stage 1 exposes task-specific tools through separate prompts, while Stage 2 exposes the full toolbox in a unified question-answering setting.The shared prompt uses a consistent think-then-tool-call-or-answer trace structure across stages.
  • Prompt and toolbox design: The visual tools use structured interfaces for pixel-coordinate zoom boxes, degree-based rotations, horizontal or vertical flips, and drawing operations.Tool calls operate within a sequential trajectory in which each turn permits at most one call and produces at most one new image.
Loading 2604.19945v1…