Source-linked AI summary
Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
Xingming Long, Yu Liu, Zhiwei Yang, Hanqi Feng, Shaojie Zhang, Barnabas Poczos, Chao Jiang, Zhenbo Luo, Lei Jiang, Pei Fu
TL;DR
Agentic VLM training often supervises tool use through final-answer outcomes, leaving necessary evidence acquisition and extraction insufficiently guided. The paper introduces NTEP and NTEP-R to supervise both stages and penalize repeated goals, with NTEP-8B improving accuracy while reducing tool calls across seven benchmarks. Its limitations include teacher-dependent path quality, training-time judging overhead, and evaluation concentrated on three tools and predominantly English retrieval.
Problem
Existing training paradigms primarily reward final-answer correctness or tool invocation, providing insufficient evidence-level supervision for necessary acquisition and accurate utilization.
Method
NTEP annotates necessary evidence, tools, and extracted information, while NTEP-R rewards pre-call goal alignment and post-call information acquisition and penalizes repeated goals.
Results
Across seven benchmarks, NTEP-8B achieves the best same-harness RL-agent average of 70.34, improves Search Avg. from 58.12 to 60.55, and reduces average tool calls from 2.54 to 1.89 on search-oriented benchmarks.
Takeaways & Limitations
Fine-grained evidence-path supervision supports more accurate and selective tool use than incentives based only on tool-call volume.
Takeaways & Limitations
The study depends on teacher-distilled NTEPs, adds training-time semantic-judging overhead, and evaluates three tools with predominantly English retrieval.
Abstract
from arXiv · showhide
Modern vision-language models (VLMs) can directly answer many image-grounded questions, yet they often struggle with complex queries requiring fine-grained visual details or external knowledge. To acquire this missing evidence, agentic VLMs invoke tools such as image cropping, image search, and text search. However, existing training paradigms primarily evaluate tool-use based on final answer correctness, leaving evidence acquisition and utilization insufficiently supervised. This leads to two critical shortcomings: (i) models frequently issue redundant or off-target tool calls that fail to gather necessary evidence, and (ii) even when appropriate tools are called, models often fail to extract the necessary information from the resulting observations. To address these limitations, we introduce the NTEP (Necessary Tool-Evidence Path), a novel annotation scheme that explicitly specifies the essential external evidence and corresponding tool calls for each query. Building upon this, we propose NTEP-R (NTEP Reward), a supervision mechanism ensuring that each tool invocation strictly advances the reasoning process toward the final solution. Specifically, our approach rewards the agent for aligning its pre-call intent with a necessary evidence-seeking goal, and for ensuring the information summarized from the post-call observation aligns with the necessary evidence. Furthermore, we introduce a non-repeated-goal regularizer to penalize redundant calls that revisit satisfied NTEP goals. Extensive evaluations on seven image-grounded benchmarks demonstrate that our 8B-parameter instantiation, NTEP-8B, significantly improves both search-oriented accuracy and tool-use efficiency within a unified three-tool framework. These results highlight the critical value of fine-grained tool-evidence path supervision for training robust agentic VLMs.
1 Introduction
VLMs can answer many image-grounded questions directly, but complex queries may require finer visual inspection, entity identification, or external knowledge beyond the initial context.
- Complex image-grounded queries can require fine-grained visual details, unfamiliar-entity identification, or external world knowledge.
- These requirements motivate agentic VLMs that invoke tools to obtain external evidence.
A. VLM Inference Path
The supplied passages identify crop, image-search, and text-search tools, alongside a concern about off-target or redundant calls.
- The tool inventory includes crop, image search, and text search operations.
- Off-target or redundant calls are identified as an undesirable tool-use pattern.
B. NTEP Path
NTEP defines necessary evidence-seeking paths, and NTEP-R supervises both intent before a tool call and information extraction afterward while penalizing repeated goals. Across seven benchmarks, the framework improves accuracy and tool-use efficiency.
- NTEP Path: NTEP specifies an evidence-seeking goal, appropriate tool, and required extracted information for each answer-critical step.
- NTEP Path: NTEP-R rewards pre-call alignment with necessary evidence goals and post-call alignment between summarized information and required evidence.
- NTEP Path: The non-repeated-goal regularizer penalizes redundant operations that revisit satisfied NTEP goals.
- NTEP Path: NTEP-R is evaluated across seven image-grounded benchmarks in a unified three-tool framework covering image cropping, visual search, and text retrieval.
- NTEP Path: Ablations report complementary contributions from pre-call goal alignment and post-call information extraction, while the regularizer supports precise, selective tool use.
2 Related Work
Related work frames tool-using agents as systems that extend language and vision-language models with external actions for search, visual inspection, browsing, and multi-step evidence gathering.
- Tool-using agents augment language and vision-language models with external actions for search, visual inspection, browsing, and multi-step evidence gathering.
- Prior multimodal systems train agents for operations including text search, image search, cropping, and zooming.
3 Method
NTEP-R trains agentic VLMs to acquire necessary evidence through flexible tool-use trajectories rather than relying only on terminal answer correctness. It combines NTEP-defined evidence milestones with GRPO rewards for pre-call alignment, post-call information uptake, and non-redundant tool use.
- Problem Formulation: The workflow models each interaction as a reasoning intent, tool call, observation, and subsequent answer-producing trajectory.The tool inventory can include visual operations, retrieval systems, OCR, code interpreters, databases, or other external modules.
- Reinforcement Learning with NTEP-R: Training uses a two-stage procedure in which a teacher distills frozen sample-specific NTEPs before GRPO updates the current policy.At inference, the teacher, semantic judge, and NTEP annotations are absent; the policy interacts independently with only the image, question, and tool interface.
- Necessary Tool-Evidence Paths: NTEPs specify ordered, answer-critical evidence goals, expected tools, and information to extract without prescribing rigid wording or action sequences.This abstraction permits multiple valid queries and reasoning traces and remains tool-agnostic.
- NTEP Reward: NTEP-R separately judges whether each pre-call intent targets required evidence and whether post-call reasoning extracts it from the observation.A frozen training-time semantic judge supplies independent binary Align and Acquire judgments, localizing credit to evidence seeking and uptake.
- NTEP Reward: The process reward credits distinct evidence goals and extracted information while penalizing off-path calls and calls that repeat already satisfied goals.Only the first call aligned with a goal receives alignment credit; later aligned calls are explicitly treated as redundant.
- Reinforcement Learning with NTEP-R: NTEP-R adds its process signal to answer and format rewards before group normalization, allowing GRPO to compare trajectories by evidence acquisition as well as outcomes.Answer-equivalent trajectories can receive different learning signals according to goal alignment, information uptake, and redundant actions.
4 Experiments
Across seven image-grounded benchmarks, NTEP-8B achieves the strongest same-harness RL-agent performance while using fewer, more necessary tool calls. Analyses show improved trajectory quality, cross-tool transfer, and reward-component contributions.
- Main Results: NTEP-8B obtains an Overall Average of 70.34, exceeding SenseNova-MARS-8B by 2.03 points, with Search Avg. rising from 58.12 to 60.55 and Visual Avg. from 81.90 to 83.40.The comparison uses a unified three-tool harness across seven image-grounded benchmarks.
- Tool-Use Efficiency: NTEP-8B reduces average tool calls from 2.54 to 1.89 on search benchmarks and from 1.86 to 1.08 on visual benchmarks while improving accuracy.The reported gain is attributed to more selective tool use rather than more frequent invocation.
- Tool-Call Trajectories: NTEP-8B reaches a 78.8% necessary-and-used call rate and a 58.7% Case-5 rate, compared with 69.0% and 39.0% for SenseNova-MARS-8B.NTEP-8B concentrates acquisition in the first two call positions and never reaches a fifth call in the judged sample.
- Failure Modes: Redundant calls fall from 13.5% without regularization to 1.0% in NTEP-8B, while wrong-tool selection remains the largest residual failure for every model.Failure modes are labeled with the priority redundant > off-goal > wrong tool > unused.
- Ablation Studies: Full NTEP-R reaches 59.44 Search Avg. and reduces calls to 1.55, cutting total calls by roughly 71% versus the goal-only variant and roughly 50% versus Answer Reward Only.The goal-only variant averages 5.36 calls, while Answer Reward Only averages 3.11 calls.
5 Conclusion
NTEP-R trains agentic VLMs to follow necessary tool-evidence paths by supervising both call intent and information acquisition, while penalizing repeated goals. Across benchmarks, NTEP-8B improves accuracy and efficiency through earlier, less redundant evidence gathering.
- NTEP-R supervises pre-call goal alignment and post-call information acquisition instead of optimizing only final-answer correctness.
- Across seven benchmarks, NTEP-8B improves accuracy and reduces average tool calls within a unified crop, image-search, and text-search framework.
- NTEP annotations record each answer-critical evidence goal, corresponding tool, and necessary extracted information.
- NTEP paths are distilled from successful rollouts or reconstructed from missing evidence transitions in unsuccessful rollouts before GRPO training.
A.2 Training Data Composition
The experiments use shared training sources and a unified three-tool interface, with explicit reward accounting for aligned hits, duplicates, and misses. Training and evaluation settings define how evidence paths are constructed, scored, and executed.
- Both training pools draw from FVQA, DeepEyes-4K, Visual-Probe, VDR, and Search-R1, while the historical 7,774-example pool spans all three tools.
- The interface registers image_zoom_in_tool, image_search_tool, and text_search_tool, while inference exposes only the image, question, and tool schemas.
- Image zoom uses normalized bounding boxes, image search returns titled thumbnails, and text search returns the top-3 summarized results.
- Rollouts allow at most 10 interaction turns with no per-tool caps, leaving tool composition to the policy.
- Only the first semantic hit receives goal credit; duplicate retargeting is penalized, and calls matching no pending goal are counted as misses.
A.6 Theoretical Analysis: Call Efficiency and Tool-Interface Transfer
The analysis shows that NTEP-R prices waste while preserving useful retries, stops once evidence goals are complete, and supports tool-interface transfer. Experiments and theory connect these properties to lower redundancy, stable call demand, and improved unseen-tool performance.
- NTEP-R evaluates calls against pending evidence goals, granting information credit only when aligned calls acquire the required information.
- Wasteful calls and redundant alignments receive additive penalties, while retries remain profitable when their success probability exceeds the calibrated threshold.
- 13.5% to 1.0%: the theory identifies waste pricing as the mechanism behind redundancy collapse and uniform call reductions.
- Once all goals are complete, every further call is penalized without improving answer reward, so the reward-optimal policy answers immediately.
- 69.45–69.69 and 1.545–1.549 calls per example: accuracy and realized tool use remain stable across inference budgets.
- In a real MMSearch example, Full NTEP-R answers after one decisive retrieval instead of repeating the satisfied release-date goal through nine searches.
- +9.95 points on V*: adding unseen zoom improves accuracy while reducing average calls across every visual split.
C.7 Training Dynamics
Training dynamics distinguish stable evidence acquisition from reward saturation: goal-only supervision inflates process reward while answer reward collapses, whereas information supervision preserves necessary operations and limits redundant search.
- Goal-only supervision becomes unstable: process reward inflates while answer reward collapses late in GRPO training.
- Image zoom usage converges similarly across configurations, indicating that information supervision preserves necessary visual operations.
- Goal-only image-search usage rises toward four calls per trajectory, while information-supervised configurations remain near 0.5.
- Information supervision preserves legitimate text retrieval, while the non-repeated-goal regularizer compresses redundant text-search behavior.
- The evaluation harness fixes datasets, prompts, tools, budgets, decoding, and scoring across systems for controlled comparisons.
C.9 Limitations
The study’s scope is limited by teacher-dependent path quality, training-time judging overhead, and evaluation restricted mainly to three tools and English retrieval. The efficiency regularizer also reflects a tunable accuracy–efficiency trade-off.
- Teacher dependence: NTEP quality depends on distillation by a stronger teacher, and reconstructing paths for failed rollouts is harder than pruning successful ones.The retained outcome reward reduces residual path noise, but does not remove this dependence.
- Training overhead: Step-wise semantic judging adds serving overhead relative to outcome-only reinforcement learning.Bounded judge concurrency and response caps make this manageable at the reported scale, and judging is needed only during training.
- Evaluation scope: Experiments cover three tools and predominantly English retrieval, while richer tool families and multilingual corpora remain extensions of the tool-agnostic interface.The passage frames these as natural extensions rather than redesigns.
- Operating point: The non-repeated-goal regularizer selects an efficiency-leaning point on the accuracy–efficiency frontier.Applications prioritizing maximal retries can tune the duplicate penalty.
D Human Audit
A blinded human audit assessed the automated judgments used for open-ended search-answer scoring and trajectory behavior analysis. Agreement was high across systems, while visual multiple-choice benchmarks used exact answer matching instead of automated judging.
- Audit protocol: The audit sampled 100 judge-scored cases with model identity and judge decisions hidden from annotators.Each row included the image, question, model answer, tool transcript, and returned evidence when applicable.
- Audit protocol: Annotators evaluated answer correctness, necessary versus unnecessary tool use, and failure categories for non-necessary calls.These labels matched those used by the automated judge.
- Audit findings: 98% agreement was obtained on NTEP-8B cases and 96% on the strongest baseline’s cases.Among 100 audited decisions, there was one false positive and two false negatives; disagreements were boundary judgments rather than systematic model bias.
- Evaluation safeguards: The audit did not affect model selection, reward tuning, or post-hoc correction of reported results.Visual multiple-choice benchmarks were scored by answer extraction and exact matching rather than automated correctness judging.
- Data-use scope: The listed benchmark images and source data were used solely for non-commercial academic research.Some underlying materials remain subject to non-commercial or research-only terms imposed by original data providers.