PaperlayerView exact revision on arXiv

Source-linked AI summary

TACO: Tool-Augmented Credit Optimization for Agentic Tool Use

Mingkuan Feng, Jinyang Wu, Hao Gu, Fangrui Lv, Ruihan Jin, Chuyuan Zhang, Zhengqi Wen, Jianhua Tao

arXiv:2606.30251v1cs.MA

TL;DR

Tool-use agents need credit signals that distinguish helpful, wasted, and harmful calls without relying on external judges. TACO combines before/after answer-probe differencing with outcome-gated routing, achieving the best average among prior code-tool agents across twelve benchmarks.

  • Problem

    Existing rewards struggle to attribute final correctness to individual tool calls, while output-based process rewards require external judges and may miss harmful calls.

  • Method

    TACO couples DAPR, which scores calls by before/after answer-probe differences, with OGAR, which routes final-answer credit to responsible segments.

  • Results

    Across twelve benchmarks, TACO achieves the best average among prior code-tool agents and transfers to a stronger Qwen3-VL backbone.

  • Takeaways & Limitations

    TACO learns to invoke tools when they help and suppress wasted calls, improving both accuracy and latency across evaluated benchmarks.

  • Takeaways & Limitations

    TACO applies most directly to tasks with verifiable answers and leaves extension to multi-call trajectories, open-ended generation, and richer tool spaces for future work.

Abstract

from arXiv · show

Agentic multimodal models perform diverse operations on an image via code and reason over the returned view, an effective paradigm for fine-grained visual question answering. However, code operations can be useful, redundant, or misleading. Outcome-only rewards cannot precisely distinguish these cases, and existing process rewards either fail to attribute final correctness to individual tool calls, or require an external judge model. To address this, we introduce Tool-Augmented Credit Optimization (TACO), a GRPO variant for code-tool agents built on two coupled advantage channels. The first, Differential Answer-Probe Reward (DAPR), is a self-supervised, judge-free tool-contribution advantage that credits each tool call by its own effect on answering correctly. Probe tokens inserted into the model's reasoning elicit its predictions with and without the tool, and the difference in outcome reward is taken as the call's value: positive for a useful call, negative for a misleading one, and zero for one that changes nothing. This reuses the existing answer checker with no auxiliary judge, and, being a difference rather than an absolute probe score, is naturally robust to probe-hacking. The second is the outcome advantage from the final answer, distributed by Outcome-Gated Advantage Routing (OGAR): a parameter-free rule that, conditioned on the call's outcome, delivers this credit only to the responsible segments, suppressing wasted tool calls without any cost term. We train TACO through a two-stage SFT+RL pipeline. Extensive experiments across perception, reasoning, and general multimodal benchmarks show that it yields consistent accuracy gains and learns to invoke its tools only when they help.

1 Introduction

TACO addresses the problem that visual tool calls can help, do nothing, or harm answers by combining a judge-free tool-contribution reward with outcome-gated token-level credit routing. Using a two-stage SFT-then-RL recipe, it improves accuracy across multimodal benchmarks while learning to invoke tools only when helpful.

  • Motivation: Visual tool calls can be useful, inconclusive, or misleading, motivating rewards that score each call by its own contribution to final correctness.The same crop may turn a wrong answer right, leave it unchanged, or turn a right answer wrong.
  • TACO: TACO is a GRPO variant for code-tool visual agents that couples Differential Answer-Probe Reward (DAPR) with Outcome-Gated Advantage Routing (OGAR).DAPR provides the tool-call value reward, while OGAR routes the final-answer advantage.
  • Differential Answer-Probe Reward (DAPR): DAPR compares tool-off and tool-on answer probes using a rule-based checker, assigning positive value to helpful calls, negative value to harmful calls, and zero value to unchanged outcomes.The two probes read answers immediately before the tool call and after the returned tool result has been reasoned over.
  • Outcome-Gated Advantage Routing (OGAR): OGAR routes final-answer advantage only to token segments responsible for the answer, preventing redundant tool calls from being rewarded and tool-spoiled reasoning from being blamed.It is parameter-free and suppresses wasted calls without adding a tool-call cost term.
  • Evaluation: TACO uses a two-stage SFT-then-RL recipe and delivers consistent accuracy gains across perception, reasoning, and general multimodal benchmarks while invoking tools only when they help.The approach is designed to avoid external judge models and auxiliary tool-call costs.

2 Related Work

Related work enables multimodal models to reason over code-generated image views and densifies reinforcement learning with process rewards. TACO differs by assigning credit to tool actions that produce real observations rather than to token spans alone.

  • Thinking with images: Multimodal agents emit code to crop, zoom, transform, or otherwise process images, with prior systems using pixel-space operations or sandboxed image-processing code.Examples include DeepEyes (Zheng et al. 2026), Pixel-Reasoner (Wang et al. 2025a), Mini-o3 (Lai et al. 2026), PyVision (Zhao et al. 2026), Thyme (Zhang et al. 2026), and DeepEyesV2 (Hong et al. 2026).
  • Process rewards and credit assignment: Prior process-reward methods densify reinforcement learning over intermediate textual reasoning steps but do not isolate external tool observations.MIG (Wang et al. 2026), PACR (Yoon et al. 2025), and SPAE (Wu et al. 2026a) estimate or reward progress on textual chains using marginal gain, confidence, or correctness.
  • Process rewards and credit assignment: TACO treats an action with a real observation as the unit of credit, rather than a token span.Its overview depicts a process channel based on the before/after probe difference and an outcome channel routed only to responsible segments.

3 Method

TACO augments GRPO for code-tool visual agents with DAPR, which measures each tool call’s change in answer reward, and OGAR, which routes final-answer credit to responsible tokens. It is trained with SFT cold-start followed by GRPO using these coupled gated advantages.

  • 3.4 Training: TACO uses a two-stage SFT-plus-RL pipeline: supervised cold-start teaches the Think–Code–Answer format and reference policy, then GRPO optimizes gated accuracy and process advantages with a KL penalty.Stage 2 samples on-policy rollout groups and applies the gated accuracy advantage together with the process advantage.
  • 3.1 Agent and Probe Setup: The agent trajectory interleaves pre-tool reasoning, code execution returning a visual observation, post-tool reasoning, and a final answer, with probes inserted around the call.The pre-tool probe yields the answer without invoking the tool, whereas the post-tool probe incorporates the returned visual observation and completed tool branch.
  • 3.2 Differential Answer-Probe Reward: DAPR scores a tool call by subtracting the pre-tool probe reward from the post-tool probe reward, identifying useful, misleading, or outcome-neutral calls without an external judge.The two probes decode answers with and without the tool branch; computing the difference requires only two short probe decodes and no API call.
  • 3.2 Differential Answer-Probe Reward: Differencing cancels the shared pre-tool baseline, so the value isolates the tool branch comprising code, its observation, and post-tool reasoning, while resisting probe-hacking.Pre-writing a conclusion into the shared reasoning inflates both probe rewards equally and therefore leaves the difference unchanged.
  • 3.3 Outcome-Gated Advantage Routing: OGAR routes final-answer advantage according to call outcome: it preserves credit for correct pre-tool reasoning, blames misleading code and post-tool reasoning, and withholds blame from necessary-but-failed branches.For misleading calls, the process channel is disabled and the accuracy penalty targets code and post-tool tokens, protecting correct pre-tool reasoning.

4 Experiments

Across twelve multimodal benchmarks, TACO improves accuracy over code-tool agents without an external judge while using fewer tool rounds and lower latency. Ablations, cross-backbone tests, and training dynamics attribute these gains to DAPR’s differencing and OGAR’s gated credit routing.

  • 4.3 Efficiency comparison: 89.6% at 2.3 s on V∗beats PyVision’s 88.7% at 3.6 s, and TACO is most accurate and fastest on all five Table 2 benchmarks.Invoking tools only when helpful reduces tool and sandbox rounds, so appropriate cropping improves accuracy and latency simultaneously.
  • 4.2 Main comparison: 68.1 average accuracy lets TACO beat PyVision (63.7) by 4.4 points and other code-tool agents by 5.6–8.1 points without an external judge.TACO leads all four perception benchmarks, reaching 81.6 on HR-Bench-8K and 89.6 on V∗, while also leading LogicVista (55.6) and WeMath (53.1).
  • 4.4 Component ablations: 72.0 average accuracy for full TACO exceeds w/o DAPR (67.5, −4.5) and w/o OGAR (70.0, −2.0), showing complementary component contributions.DAPR’s differencing prevents probe hacking, whereas OGAR restricts final-answer advantage to responsible segments instead of every token.
  • 4.5 Generalization across base models: TACO improves Qwen2.5-VL-7B from 60.4 to 72.0 (+11.6) and Qwen3-VL-8B from 72.9 to 78.8 (+5.9) on a five-benchmark subset.Gains are consistent across benchmarks, largest for high-resolution perception, including HR-8K (+16.3/+9.5) and HR-4K (+15.0/+6.4).
  • 4.6 Training dynamics and probe-hacking analysis: The additive-probe variant rises fastest initially but plateaus below standard GRPO, while TACO sustains the highest reward and shortens completions from about 720 to 640.TACO’s entropy declines without collapse and remains highest, consistent with differencing resisting probe hacking and preserving exploration while the policy learns economical tool use.

5 Conclusion · Technical Appendix of TACO: Tool-Augmented Credit Optimization for Agentic Tool Use · A. TL;DR: Main Contributions and Takeaways

TACO is a GRPO variant for code-tool visual agents that combines judge-free tool-call credit with outcome-based routing. Across twelve benchmarks, it achieves the best average among prior code-tool agents and transfers to a stronger Qwen3-VL backbone, while the appendix details its methods, experiments, supplementary results, and analyses.

  • A. TL;DR: Main Contributions and Takeaways: TACO couples DAPR’s before/after answer-probe difference with OGAR’s routing of final-answer advantage to responsible segments.DAPR is judge-free and probe-hacking-robust, while OGAR reinforces useful calls and suppresses wasted ones.
  • A. TL;DR: Main Contributions and Takeaways: OGAR routes final-answer advantage only to responsible segments, reinforcing useful tool calls while suppressing wasted ones.Together with DAPR, it provides a tool-call learning signal for code-tool visual agents.
  • 5 Conclusion: Across twelve benchmarks, TACO achieves the best average among prior code-tool agents and transfers gains to a stronger Qwen3-VL backbone.The transfer result indicates that gains come from the mechanism rather than the base model or data.
  • 5 Conclusion: DAPR scores each tool call from the difference between the model’s own answer-probe outcomes before and after the call, without an auxiliary judge or cost term.This addresses the bottleneck of learning which tool calls are worth making using the agent’s own outcome reward.
  • Technical Appendix of TACO: Tool-Augmented Credit Optimization for Agentic Tool Use: The technical appendix covers TACO’s training algorithm, theoretical motivation, experimental setup, supplementary results, qualitative case studies, and probe-hacking analysis.Its setup includes benchmarks, baselines, and data-curation pipelines; notation follows the main paper.
  • Technical Appendix of TACO: Tool-Augmented Credit Optimization for Agentic Tool Use: The appendix is organized to provide additional methodological, theoretical, empirical, qualitative, and probe-hacking details for TACO.These materials extend the main paper while preserving its notation.

B. Training Algorithm and Edge Cases … F. Limitations and Future Work

The section covers the training algorithm and edge cases, theoretical motivation, experimental setup details, show cases, and limitations and future work. Its theoretical scope includes DAPR, OGAR, the two-channel objective, and stated assumptions.

  • B. Training Algorithm and Edge Cases: The paper presents its training algorithm and discusses edge cases.
  • E. Show Cases: The merged section includes show cases as a distinct reported component.
  • F. Limitations and Future Work: The merged section also includes limitations and future work as a distinct concluding component.
  • C. Theoretical Motivation: The theoretical motivation develops GRPO credit assignment, DAPR, potential-outcomes interpretations, probe-hacking robustness, potential-based shaping, OGAR masking, and the two-channel objective.
  • C. Theoretical Motivation: The theoretical section also states the scope and assumptions underlying the method.
  • D. Experimental Setup Details: The experimental setup specifies benchmarks, baselines, implementation details, SFT data curation, RL data curation, and channel-weight sensitivity.

A TL;DR: Main Contributions and Takeaways … C.4 Robustness to Probe-Hacking, Formally

TACO combines DAPR’s judge-free per-tool counterfactual reward with OGAR’s gated advantage routing in a two-stage SFT+RL pipeline. It improves accuracy and tool-use efficiency while remaining robust to probe-hacking and transferable across backbones.

  • A TL;DR: Main Contributions and Takeaways / C.2 DAPR as a Counterfactual Baseline: DAPR scores each tool branch by the before/after difference in the agent’s answer-probe outcomes, reusing the existing checker without an auxiliary judge at near-zero added cost.The difference credits useful calls positively, misleading calls negatively, and unchanged calls with zero value.
  • A TL;DR: Main Contributions and Takeaways / C.1 Background: GRPO and Uniform Credit Assignment: OGAR routes final-answer advantage only to outcome-responsible tokens, preventing redundant calls from being over-credited and correct pre-tool reasoning from being blamed.This parameter-free masking complements DAPR because a scalar tool value alone cannot assign credit across trajectory segments.
  • A TL;DR: Main Contributions and Takeaways: TACO achieves the best average among open-source models while invoking tools only when helpful, making it simultaneously the most accurate and lowest-latency code-tool agent.Its gains transfer from Qwen2.5-VL-7B to Qwen3-VL-8B, indicating backbone-agnostic credit assignment.
  • B.1 Training Algorithm: TACO trains with SFT cold-start followed by group-relative RL, parsing each rollout into pre-tool reasoning, code, and post-tool segments before probing and routing two normalized advantage channels.The training loop executes code, decodes pre- and post-tool probes, computes ∆, and applies token-level and trajectory-level gates.
  • B.2 Edge Cases: Direct-answer rollouts leave the process channel inactive and apply only gated accuracy advantage to answer tokens, allowing abstention from tools on already-solved items without penalty.For multiple calls, ∆ evaluates the entire branch against the same pre-tool baseline; finer per-call attribution is left for future work.
  • C Theoretical Motivation / C.1 Background: GRPO and Uniform Credit Assignment: The theoretical motivation identifies uniform GRPO credit as unable to separate a tool call’s contribution from overall trajectory correctness, motivating DAPR and OGAR.The paper frames these mechanisms through GRPO’s limitation, counterfactual differences, potential-based shaping, and conservative outcome masking.
  • C.2 DAPR as a Counterfactual Baseline / C.3 Potential-Outcomes Interpretation of ∆: DAPR estimates the realized effect of taking the tool branch by comparing potential outcomes with and without the branch from the same pre-tool state.The post-tool probe realizes Y(1), the pre-tool probe realizes Y(0), and ∆ = Y(1) − Y(0); ∆ includes triggered post-tool reasoning as well as the observation.
  • C.4 Robustness to Probe-Hacking, Formally: Differencing is invariant to common-mode probe shifts: if answer leakage makes both probes copy the same pre-committed answer, their difference is zero.Thus, writing the answer early can raise absolute probe scores but cannot inflate DAPR when both probes shift equally.

C.5 A Potential-Based-Shaping View of the Process Channel … D Experimental Setup Details

The analysis frames TACO’s process signal as potential-based shaping and OGAR as conservative advantage masking, then combines both channels under explicit assumptions about probing, verifiable outcomes, and single decisive tool calls. Experimental setup details define the benchmark evaluation scope.

  • C.5 A Potential-Based-Shaping View of the Process Channel: Potential shaping preserves the optimal policy set because its trajectory terms telescope to the policy-independent offset −Φ(s0), yielding Q′∗(s, a) = Q∗(s, a) −Φ(s).
  • C.5 A Potential-Based-Shaping View of the Process Channel: The process signal is a γ = 1 potential difference with Φ(s) = rout(probe(s)), preserving optimal policies while changing credit-flow speed and location.In practice, TACO applies the channel at the group/trajectory level under GRPO normalization, and probing only approximates state value.
  • C.7 The Two-Channel Objective: Together, potential shaping supplies a policy-invariant process signal, OGAR conservatively routes outcome credit, and the two-channel objective separates token-level and sequence-level supports.
  • C.6 OGAR as Conservative Advantage Masking: OGAR replaces GRPO’s uniform A1 with A[t] = m[t] A1, so gating only withholds token credit and never reverses an outcome-justified gradient.On right-but-redundant calls, the code advantage becomes 0, discouraging unnecessary calls without an extra cost term.
  • C.7 The Two-Channel Objective: TACO combines outcome and process channels: m⊙A1 updates outcome-responsible tokens, while gA2 applies sequence-wide when Δ≥0 and is disabled for misleading calls.The outcome advantage uses Racc = rout(af) + 0.5 Rfmt, whereas A2 is the group-normalized advantage of Δ.
  • D Experimental Setup Details: The experimental evaluation groups benchmarks by category and reports the number of questions in each benchmark.

D.1 Benchmarks · D.2 Baselines

The evaluation spans twelve multimodal benchmarks across perception, reasoning, and general understanding, scored uniformly with accuracy and macro-average reporting. Comparisons cover closed-source models, tool-free open-source MLLMs, and 7–8B code-tool or visual-agent systems.

  • D.1 Benchmarks: Twelve benchmarks are evaluated using standard-split accuracy and a macro-average across all tasks, with answers scored by the same Duan et al. (2024) protocol.The benchmark suite is organized into perception, reasoning, and general multimodal understanding.
  • D.1 Benchmarks: Perception benchmarks target fine-grained recognition in high-resolution images, including 4K/8K small-object attributes, positions, relations, and challenging real-world questions.HR-Bench-4K and HR-Bench-8K each contain 800 samples, while MME-RealWorld uses manually annotated high-resolution images.
  • D.1 Benchmarks: Reasoning benchmarks test multimodal mathematical and logical reasoning over diagrams, figures, and charts, including competition-level, visual-context, and diagram-based problems.The suite includes MathVision, MathVista, and MathVerse, with 3,040, 6,141, and 2,612 examples respectively.
  • D.1 Benchmarks: General benchmarks measure broad multimodal understanding through vision-indispensable questions, chart-based visual and arithmetic reasoning, and classic perception tasks that remain difficult for MLLMs.MMStar contains 1,500 human-curated questions, ChartQA combines human-written and machine-generated chart questions, and BLINK contains 3,807 multiple-choice questions.
  • D.2 Baselines: Baselines include GPT-4o and Gemini-2.5-Pro, tool-free open-source MLLMs, and 7–8B code-tool or visual-agent models that operate on images before reasoning.The open-source tool-free family includes Qwen2.5-VL variants, InternVL3, LLaVA-OneVision, and Qwen3-VL.
  • D.2 Baselines: Relevant visual-agent baselines span code-based image processing, emergent or adaptive tool use, pixel operations, extended visual search, and vision-code alignment.These include Thyme (Zhang et al. 2026), DeepEyes (Zheng et al. 2026), DeepEyesV2 (Hong et al. 2026), Pixel-Reasoner (Wang et al. 2025a), Mini-o3 (Lai et al. 2026), and MathCoder-VL (Wang et al. 2025b).
  • D.2 Baselines: Additional baselines generate or optimize executable visual tools, using external-judge process rewards in CodeV and autonomous tool refinement in PyVision (Zhao et al. 2026).CodeV applies Tool-Aware Policy Optimization to tool inputs and outputs, whereas PyVision generates, executes, and refines task-specific Python tools at inference.

D.3 Implementation Details

TACO is trained on Qwen2.5-VL-7B using a two-stage SFT-plus-GRPO pipeline with specified channel weights, rollout settings, and single-node A100 infrastructure. Both stages share Thyme’s prompt template, which supports step-by-step reasoning and optional sandboxed Python image manipulation.

  • D.3 Implementation Details: Training uses 2 SFT epochs followed by 1 GRPO epoch on Qwen2.5-VL-7B, with α1 = 1.0, α2 = 0.15, and no KL penalty (β = 0).The setup samples G = 8 rollouts per prompt at temperature 1.0, using batch size 128 and learning rate 1 × 10−6.
  • D.3 Implementation Details: Training runs on a single node of 8×80 GB A100 GPUs, with channel-weight sensitivity reported in Table 8.
  • Prompt templates: Both stages use Thyme’s shared system prompt, while each user prompt provides the image, question, image path, size, and required <think>/<answer> format.The system prompt instructs step-by-step reasoning and optionally emitting sandboxed Python for image manipulation, returning the processed image or result for further reasoning.

D.4 SFT Data Curation … E Show Cases

The paper curates SFT and RL data for execution validity, tool necessity, environmental fidelity, and calibrated difficulty, then selects α1=1.0 and α2=0.15 to balance correctness with tool-value shaping. Case studies show the trained agent using targeted visual operations across perception, scientific-figure reading, rotation, and chart-grounded math.

  • D.4 SFT Data Curation: SFT trajectories are re-run for execution validity and filtered when code errors or tool observations and answers disagree with actual outputs.This prevents training the model to hallucinate tool observations.
  • D.4 SFT Data Curation: SFT data also drops examples that Qwen2.5-VL-7B already solves without tools, retaining trajectories where tool use is necessary.The supplied passage specifies the exclusion criterion as pass@8 = 1 for tool-free solving.
  • D.5 RL Data Curation: RL prompts are restricted to questions with verifiable ground-truth answers and cleaned for image quality, question clarity, image–text consistency, and severe ambiguity.Environmental checks use Gemini-3-Pro, and corrupted or severely ambiguous prompts are removed.
  • D.5 RL Data Curation: The RL data curation additionally calibrates difficulty by removing prompts already solved by the SFT checkpoint on all available evaluations.The supplied passage introduces this filter but is truncated before specifying its complete criterion.
  • D.6 Channel-Weight Sensitivity: The selected channel weights are α1=1.0 and α2=0.15, with macro-average accuracy evaluated across twelve benchmarks.The accuracy weight remains fixed while the tool-value weight is swept; the selected setting is used throughout the paper.
  • D.6 Channel-Weight Sensitivity: Small α2 keeps verifiable outcome accuracy dominant while still allowing the tool-value channel to shape exploration, whereas very large α2 would over-weight the auxiliary signal.This explains the rationale for the selected weighting configuration.
  • E Show Cases: Five trajectories illustrate targeted operations for crop-and-zoom perception, scientific-figure reading, image rotation, and chart-grounded math.The examples include recovering an occluded sponsor wordmark, reading small circuit-board labels, isolating a plot panel, rotating a sideways bus image, and computing a chart-based fraction.

F Limitations and Future Work

TACO is best suited to tasks with verifiable answers and observable tool effects, while extending its probe-difference signal beyond single-call trajectories remains future work. Promising directions include broader tool settings and model compression for efficient deployment.

  • Limitations: TACO applies most directly to tasks with verifiable answers because it relies on a rule-based outcome checker.Its probes also assume that a tool’s effect is observable in the answer.
  • Limitations and Future Work: The current probe-difference signal uses a clean before/after split for single-call scoping, leaving multi-call trajectories, open-ended generation, and richer tool spaces unresolved.Extending the signal to these settings is identified as future work.
  • Future Work: Future work includes integrating TACO with model-compression methods for efficient deployment.
Loading 2606.30251v1…