PaperlayerView exact revision on arXiv

Source-linked AI summary

Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models

Shilin Yan, Jintao Tong, Hongwei Xue, Xiaojun Tang, Yangyang Wang, Kunyu Shi, Guannan Zhang, Ruixuan Li, Yixiong Zou

arXiv:2604.08545v1cs.CVcs.AI

TL;DR

Multimodal agents struggle to decide when internal knowledge is sufficient and when to use external tools. HDPO separates accuracy optimization from efficiency optimization on correct trajectories; Metis reports 2% versus 98% redundant tool invocations for standard GRPO while achieving state-of-the-art accuracy.

  • Problem

    Current multimodal agents struggle to arbitrate between internal knowledge and external utilities, making it difficult to determine when tools are genuinely needed.

    Source: §1 · p. 2

  • Method

    HDPO separates task-accuracy optimization from tool-efficiency optimization, applying conditional efficiency advantages only to accurate trajectories.

    Source: §1 · p. 2

  • Results

    2% vs. 98% for standard GRPO, Metis achieves state-of-the-art accuracy with near-zero redundant tool invocations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported results show that strategic tool use and strong reasoning performance are not necessarily a trade-off.

    Source: §1 · p. 3

Abstract

from arXiv · show

The advent of agentic multimodal models has empowered systems to actively interact with external environments. However, current agents suffer from a profound meta-cognitive deficit: they struggle to arbitrate between leveraging internal knowledge and querying external utilities. Consequently, they frequently fall prey to blind tool invocation, resorting to reflexive tool execution even when queries are resolvable from the raw visual context. This pathological behavior precipitates severe latency bottlenecks and injects extraneous noise that derails sound reasoning. Existing reinforcement learning protocols attempt to mitigate this via a scalarized reward that penalizes tool usage. Yet, this coupled formulation creates an irreconcilable optimization dilemma: an aggressive penalty suppresses essential tool use, whereas a mild penalty is entirely subsumed by the variance of the accuracy reward during advantage normalization, rendering it impotent against tool overuse. To transcend this bottleneck, we propose HDPO, a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one. By eschewing reward scalarization, HDPO maintains two orthogonal optimization channels: an accuracy channel that maximizes task correctness, and an efficiency channel that enforces execution economy exclusively within accurate trajectories via conditional advantage estimation. This decoupled architecture naturally induces a cognitive curriculum-compelling the agent to first master task resolution before refining its self-reliance. Extensive evaluations demonstrate that our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.

1 Introduction

The introduction argues that multimodal agents need to learn when not to use tools, and presents HDPO as a way to improve execution economy while maintaining correctness.

1 Introduction

  • Metis reduces tool use by over 90%, with 2% versus 98% tool invocations for standard GRPO, while achieving state-of-the-art accuracy across diverse benchmarks.The paper argues that eliminating redundant calls can contribute to superior accuracy rather than trade it off.Source: Figure 1 · p. 2, §1 · p. 3
  • Agents often invoke tools reflexively even when visual input suffices; redundant calls add latency and noise that can derail sound reasoning and degrade performance.The introduction reports tool invocation rates frequently exceeding 80% to 90% for state-of-the-art open-source agents without superior reasoning performance.Source: §1 · p. 2
  • Scalarized RL penalties either suppress essential tool use when aggressive or are washed out by accuracy-reward variance when mild, leaving strategic arbitration unresolved.The introduction notes that inaccurate trajectories with no tool calls can receive rewards indistinguishable from accurate trajectories with excessive calls.Source: §1 · p. 2
  • HDPO separately maximizes correctness across rollouts and optimizes tool parsimony only within accurate trajectories, combining channels at the final loss to prioritize task mastery before self-reliance.Its conditional advantage mechanism is intended to prevent gradient interference between accuracy and efficiency objectives.Source: §1 · pp. 2–3
  • A rigorous data-curation pipeline complements HDPO by removing hallucinated environmental dynamics and isolating genuine tool necessity.The paper presents high-fidelity environmental feedback as important for strategic reinforcement learning.Source: §1 · p. 3

2 Related Works

Multimodal models have advanced from direct answers to explicit and latent reasoning, while agentic models add tool interaction; however, many agents overuse tools even when direct reasoning suffices.

2.1 Multimodal Large Language Models.

  • MLLMs [3] [36] [2] [45] progress from direct answers to explicit and latent visual reasoning, improving complex and spatial reasoning, yet remain mostly passive [18] [25].These models integrate visual encoders with large language models [1] [17].Source: §2.1 · p. 4

2.2 Agentic Multimodal Models.

  • Agentic MLLMs use visual tools for challenging cases where raw input is insufficient [28], but emphasis on capability over efficiency leaves agents prone to blind tool use when direct reasoning suffices [26] [55].Their larger decision space includes whether to call a tool, which tool to use, and how to incorporate returned observations.Source: §2.2 · p. 4

3 Method

Section 3 formalizes multi-turn tool-augmented reasoning and analyzes flaws in existing coupled reward formulations. It introduces HDPO to eliminate cross-objective interference and induce a learning curriculum.

3 Method

  • The method formalizes multi-turn tool-augmented reasoning and analyzes inherent flaws in existing coupled reward formulations.These topics are covered in §3.1.Source: §3 · p. 4
  • HDPO eliminates cross-objective interference and naturally induces a learning curriculum.The framework is presented in §3.2.Source: §3 · p. 4

3.1 Problem Formulation & The Reward Coupling Problem

The section formulates tool use as a trade-off between answer accuracy and parsimony, then explains how scalarizing those objectives and sharing advantage normalization entangles their learning signals. This coupling can misassign credit and suppress the efficiency signal, especially when its weight is small.

Problem Formulation & The Reward Coupling Problem

  • For visual reasoning, policy πθ generates G multi-turn responses with Ti tool interactions; GRPO [24] can scalarize correctness/formatting and tool parsimony using α.The formulation considers a group of responses, each of which yields a final answer after its tool interactions.Source: §3.1 · p. 4
  • Because correctness and tool use correlate, mixed-reward variance includes nonzero covariance, so shared advantage normalization entangles the objectives and causes credit misassignment.The passage identifies this shared normalization as a vulnerability of scalarized rewards.Source: §3.1 · p. 5
  • The shared denominator links both policy gradients, making each update’s magnitude inversely dependent on the variance of the other reward and causing destructive interference.The accuracy update depends on tool-reward variance, and the tool-use update depends on accuracy-reward variance.Source: §3.1 · p. 5
  • A correct-but-inefficient trajectory can become indistinguishable in scalar reward from an incorrect-but-efficient one, leaving near-zero advantages for both.This conflation neutralizes training signals for critical edge cases.Source: §3.1 · p. 5
  • The effective trade-off depends on α and the data-dependent covariance between accuracy and tool rewards, making the hyperparameter unstable across task distributions.The passage describes this instability as hyperparameter fragility.Source: §3.1 · p. 5
  • With small α, the tool-efficiency gradient is O(α) and further attenuated by typically large accuracy variance, so its optimization signal vanishes as α decreases.The passage attributes this suppression to the denominator being dominated by accuracy variance for sufficiently small α.Source: §3.1 · pp. 5–6

3.2 HDPO: Hierarchical Decoupled Policy Optimization

HDPO decouples accuracy and efficiency optimization, conditioning tool-parsimony learning on correct rollouts. This design lets training prioritize correctness before efficiency without manual reward scheduling.

3.2 HDPO: Hierarchical Decoupled Policy Optimization

  • HDPO computes independent group-relative advantages for accuracy and efficiency, each against its own semantic baseline.This avoids combining rewards before normalization.Source: §3.2 · p. 6, §3.2.1 · p. 6

3.2.1 Dual Reward Design and Decoupled Advantages

  • The accuracy reward combines judged correctness and format compliance, weighted by λa=0.9 and λf=0.1.Accuracy advantages use standard GRPO estimation across all rollouts in a prompt group.Source: §3.2.1 · p. 6
  • Tool-efficiency advantages compare only correct rollouts, and receive zero advantage when fewer than two qualify.The inverse tool-call reward decreases monotonically: T=0 maps to 1.0, T=1 to 0.5, and so on.Source: §3.2.1 · p. 6

3.2.2 Hierarchical Policy Update

  • HDPO combines weighted PPO-style clipped surrogate losses [23] for independently normalized accuracy and tool advantages.The resulting orthogonal gradient components allow an efficiency penalty without the accuracy degradation associated with coupled formulations.Source: §3.2.2 · p. 7

3.2.3 Algorithm Summary & The Implicit Curriculum

  • As correct rollouts become more common, HDPO shifts from accuracy-focused learning to tool parsimony without manual reward scheduling.When fewer than two rollouts qualify, optimization is dominated by accuracy; once at least two qualify, the efficiency comparison scales up.Source: §3.2.3 · p. 7

3.3 Training Data Curation

The curation pipeline targets corrupted tool-augmented demonstrations and obsolete tool dependencies, strengthening SFT behavioral priors and the validity of RL feedback.

3.3.1 SFT Data Curation

  • SFT curation executes code in a sandbox and discards trajectories with execution failures or inconsistent feedback, grounding demonstrations in executable environmental dynamics.The SFT corpus draws on publicly available tool-augmented multimodal trajectories [8] [22] [49] [51].Source: §3.3.1 · pp. 7–8
  • SFT filtering removes samples the base Qwen3-VL-8B [2] consistently solves without tools (pass@8 = 1), reserving demonstrations of tool use for necessary cases.This zero-shot solvability baseline targets legacy annotations that can condition models toward blind tool invocation.Source: §3.3.1 · p. 8
  • Gemini-3.1-Pro judges SFT trajectories for visual relevance, reasoning coherence, and tool-use rationale, rejecting blind invocations such as meaningless image rotations.Trajectories that fail stringent quality thresholds are discarded so the corpus demonstrates strategic tool use.Source: §3.3.1 · p. 8

3.3.2 RL Data Curation

  • RL prompts from multiple datasets [5] [8] [22] [51] cover math, fine-grained visual understanding, and search; a multimodal judge excludes corrupted or severely ambiguous inputs.The judge checks image quality, question clarity, and image-text consistency to prevent optimization against noise.Source: §3.3.2 · p. 8
  • With G = 8 rollouts per prompt, RL curation retains only prompts with mixed successes and failures, avoiding zero-variance accuracy rewards and degenerate GRPO advantages.Trivially easy prompts are solved by all rollouts, while prohibitively hard prompts are solved by none; both are excluded.Source: §3.3.2 · p. 8

4 Experiments

Across perception, document-understanding, and reasoning benchmarks, Metis achieves state-of-the-art or highly competitive performance. Ablations and representative cases show HDPO supports tool use calibrated to whether internal knowledge suffices.

4.1 Experimental Setup

  • Experiments compare Metis with open-source, text-only reasoning, and agentic multimodal models across perception/document-understanding and mathematical/logical benchmarks.The RL prompt set contains about 5K prompts across perception-related (45%), search-oriented (36%), and mathematical/general reasoning tasks (19%); SFT data are curated to retain genuine tool necessity and include tool-free reasoning data.Source: §4.1 · pp. 8–9

4.2 Main Results

  • Metis scores 83.5% on HRBench-4K and 82.0% on HRBench-8K, and reaches 54.1% on CharXiv reasoning versus DeepEyesV2’s 48.9%.The results show gains on high-resolution visual tasks and challenging CharXiv reasoning questions.Source: Table 1 · p. 9, §4.2 · p. 10
  • Metis averages 66.9% across five reasoning datasets and scores 65.2% on WeMath, +26.4% over its 38.8% backbone score and above DeepEyesV2’s 38.1%.The paper attributes the gains to interleaving Python code execution with reasoning without compromising the logical reasoning chain.Source: §4.2 · p. 10, Table 2 · p. 10

4.3 Ablation Studies

  • HDPO with wtool=0.15 gains +2.4% on V* Bench, +2.8% on HRBench8K, and +3.1% on CharXiv (RQ) over standard GRPO.With wtool=0, HDPO reduces to accuracy-only GRPO; the authors report that decoupling efficiency suppresses noisy, redundant tool calls while improving accuracy.Source: §4.3 · p. 10, Table 3 · p. 11
  • The wtool sensitivity study finds an inverted-U pattern: 0.10 is suboptimal, 0.20 degrades performance across benchmarks, and 0.15 balances self-reliance with necessary tool use.The authors explain that excessive tool parsimony can stifle exploration and tool use on arduous tasks.Source: §4.3 · p. 11

4.4 Meta-Cognitive Tool Arbitration

  • Metis abstains from tools when visual understanding and prior knowledge suffice, but uses code to crop and enlarge ambiguous image regions for precise inspection.The cases illustrate selective tool use rather than default fallback; additional selective search examples are in Appendix B.Source: Figure 4 · p. 11, §4.4 · pp. 12–13, Figure 5 · p. 12

5 Conclusion

The conclusion presents HDPO as a way to address blind tool invocation by separating task accuracy from tool efficiency, and reports that Metis reduces tool use while achieving state-of-the-art reasoning performance.

5 Conclusion

  • Metis, complemented by rigorous data curation, reduces tool invocations by orders of magnitude while achieving state-of-the-art reasoning performance.Source: §5 · p. 13
  • HDPO uses conditional advantages to decouple task accuracy and tool efficiency into orthogonal channels, eliminating gradient entanglement and inducing a cognitive curriculum.Source: §5 · p. 13
  • Future work will extend the meta-cognitive framework to more open-ended, long-horizon environments.Source: §5 · p. 13

Appendix · A System Prompt

The system prompt specifies Metis’s tools, execution environment, decision guidelines, and response format, directing it to reason before acting, answer directly when possible, and use tools only when needed.

A System Prompt

B Additional Case Studies

These additional case studies illustrate Metis’s meta-cognitive capabilities across direct visual reasoning and decisions about when external information is needed.

Direct Reasoning without Tool Invocation

  • Metis correctly extracts legible on-screen text through visual inspection, without invoking code execution or search tools.The case illustrates trusting visual comprehension when the evidence is unambiguous.Source: B Additional Case Studies · p. 19, Figure 7 · p. 19

Selective Search Tool Invocation

  • Metis uses image search when artwork cannot be identified visually and text search when a monument’s cella width cannot be inferred from the image.The cases illustrate its distinction between visual recognition it can handle internally and factual gaps requiring targeted external queries.Source: B Additional Case Studies · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

Related papers

Loading 2604.08545v1…