Source-linked AI summary
The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
Weihao Xuan, Qingcheng Zeng, Heli Qi, Yunze Xiao, Junjue Wang, Naoto Yokoya
TL;DR
Tool-use agents have underexplored calibration dynamics, raising the question of whether miscalibration depends on tool type. The paper studies this dichotomy and introduces CAR with reward designs that jointly target accuracy and confidence reliability; CAR improves calibration while generalizing from local simulation to noisy API environments and other domains.
Problem
Calibration dynamics in multi-turn tool-use agents remain underexplored, especially whether miscalibration is universal or depends on tool type.
Method
The paper compares evidence and verification tools and proposes CAR, an RL framework jointly optimizing task accuracy and confidence reliability.
Results
CAR significantly reduces calibration error while maintaining competitive task performance and generalizing from local environments to noisy web settings and mathematical reasoning.
Takeaways & Limitations
Calibration strategies for tool-use agents should account for tool-type-specific dynamics rather than treating tool use as monolithic.
Takeaways & Limitations
Experiments focus on 3B-backbone models and mainly short-answer question answering and mathematical reasoning, leaving scaling and open-ended settings unresolved.
Abstract
from arXiv · showhide
Autonomous agents based on large language models (LLMs) are rapidly evolving to handle multi-turn tasks, but ensuring their trustworthiness remains a critical challenge. A fundamental pillar of this trustworthiness is calibration, which refers to an agent's ability to express confidence that reliably reflects its actual performance. While calibration is well-established for static models, its dynamics in tool-integrated agentic workflows remain underexplored. In this work, we systematically investigate verbalized calibration in tool-use agents, revealing a fundamental confidence dichotomy driven by tool type. Specifically, our pilot study identifies that evidence tools (e.g., web search) systematically induce severe overconfidence due to inherent noise in retrieved information, while verification tools (e.g., code interpreters) can ground reasoning through deterministic feedback and mitigate miscalibration. To robustly improve calibration across tool types, we propose a reinforcement learning (RL) fine-tuning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. We demonstrate that our trained agents not only achieve superior calibration but also exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. Our results highlight the necessity of domain-specific calibration strategies for tool-use agents. More broadly, this work establishes a foundation for building self-aware agents that can reliably communicate uncertainty in high-stakes, real-world deployments.
1 Introduction
The paper identifies tool-type-dependent calibration in LLM agents: noisy evidence tools induce overconfidence, while verification tools provide grounding. It proposes CAR, which jointly optimizes accuracy and confidence reliability and generalizes across environments and domains.
- Calibration research in multi-turn agents remains sparse, and existing benchmarks largely focus on search scenarios.
- Evidence tools systematically induce severe overconfidence, whereas verification tools can ground reasoning through deterministic feedback.
- The confidence dichotomy persists under prompting and standard tool-use RL, so prompt engineering alone does not resolve the problem.
- CAR jointly optimizes task accuracy and expressed-confidence reliability using RL fine-tuning and evaluates diverse reward structures.
- The methodology achieves cross-environment generalization from local settings to noisy web environments and across distinct task domains.
2 Related Work
Prior work has studied calibration mainly in static LLMs and search agents, while tool-use agents remain less understood. The paper frames tool integration as spanning evidence retrieval and verification-oriented interaction.
- Calibration studies have extensively examined static LLMs, but calibration in autonomous agents remains notably sparse.
- Search-agent benchmarks report higher calibration errors than standalone LLMs, but their focus leaves tool-type dependence unresolved.
- Evidence tools retrieve external information, whereas verification tools provide structured interaction with external environments.
3 Pilot Study
The pilot study isolates how tool types alter confidence by comparing direct prompting, prompted tool use, and RL-based tool use across evidence and verification settings. Search worsens overconfidence, while code-interpreter feedback improves calibration.
- Experimental configurations: The study compares confidence shifts across direct prompting, prompted tool use, and RL-optimized tool use.
- Tool types: Web Search represents an evidence tool with open-ended, stochastic, noisy outputs, while Code Interpreter represents verification through deterministic, structured feedback.
- Metric: MCIP measures mean confidence on incorrectly predicted questions, with confidence averaged over the wrong-question set.
- Results: Search tools systematically exacerbate overconfidence under both prompting-based and RL-enhanced strategies, with statistically significant differences across configurations.
- Results: Code interpreters mitigate overconfidence, with RL-based agents achieving the lowest MCIP in the verification setting.
4 Calibration Agentic RL (CAR)
CAR extends calibration-motivated reinforcement learning to tool-use agents by jointly shaping correctness, confidence, and output format. Its reward designs include weighted Brier scoring and MSCR, which separates incentives for correct and incorrect predictions.
- Framework: CAR targets reliable confidence estimates alongside tool-use actions through calibration-aware reinforcement learning.The framework is evaluated in evidence-tool scenarios where tool use exacerbates miscalibration.
- Reward Design: The framework requires agents to output numerical confidence scores from 0 to 100 inside confidence XML tags.An extended format reward validates reasoning, action, observation, and confidence-tag structure.
- Reward Design: Weighted Brier scoring can create overlapping incentives, making learning sensitive to the training-data distribution.The authors experiment with λ = 1/3 to restore a positive correctness incentive margin.
- Reward Design: MSCR decouples calibration terms for correct and incorrect predictions, guaranteeing that correct answers receive a strict reward advantage.Correct answers receive at least 1 at q = 0, while incorrect answers receive at most 0 and are penalized for false confidence.
- Reward Design: CAR combines a calibration-motivated outcome reward with a penalty for format violations.The outcome reward uses the gold answer, predicted answer, and verbalized confidence; format errors receive a penalty.
- Evaluation: Evaluation uses NQ and HotpotQA for in-distribution testing and SimpleQA-verified for out-of-distribution assessment.Accuracy and Expected Calibration Error are among the metrics used to assess performance and calibration.
5 Results
CAR consistently improves calibration across search and tool-integrated reasoning settings while preserving competitive accuracy. The gains transfer from controlled retrieval to noisy API search and across mathematical benchmarks, though absolute TIR calibration remains constrained by task difficulty.
- General Results: ECE relative reductions reach up to 68% across backbone models and both in-distribution and out-of-distribution settings.Under MSCR, agents retain accuracy competitive with correctness-focused reward structures.
- General Results: CAR’s calibration improvements transfer to SimpleQA-verified, indicating gains beyond the local training distribution.The authors describe these as transferable confidence skills rather than artifacts of the training set.
- Reward Comparison: MSCR provides a better accuracy-calibration trade-off than weighted Brier scoring with λ = 1/3 across most settings.Weighted Brier scoring with λ = 1 achieves the lowest ECE but suffers significant accuracy degradation associated with reward hacking.
- Tool Generalization: CAR transfers robustly from simulated retrieval to stochastic, noisy Serper API interactions while maintaining competitive accuracy.The API evaluation compares Vanilla Search-R1 with CAR using MSCR on SimpleQA-verified.
- Tool-integrated Reasoning: Across mathematical reasoning benchmarks, CAR reduces ECE and Brier scores while increasing AUROC for tool-integrated reasoning agents.These gains persist across all evaluated benchmarks.
- Tool-integrated Reasoning: TIR agents retain elevated ECE relative to pure reasoning and search-based agents, with lower ECE on MATH-500 than on AIME benchmarks.The authors relate this pattern to task complexity and intrinsic reasoning capability.
6 Discussion
The discussion frames calibration as tool-modulated rather than uniform across agentic workflows. It attributes the confidence dichotomy to differences in feedback signals and argues that incentive separation is central to effective calibration training.
- From Static Elicitation to Tool-Modulated Dynamics: Tool integration introduces heterogeneous calibration dynamics, so tool use should not be treated as uniformly improving or degrading reliability.The discussion contrasts evidence and verification tools as distinct sources of calibration behavior.
- Why Do Evidence Tools Induce Overconfidence?: Evidence-tool overconfidence is associated with noisy, open-ended retrieval, whereas verification tools provide observable failure modes that partially ground reasoning.Code interpreters expose syntax errors, runtime exceptions, and type mismatches, although successful execution does not guarantee correctness.
- Extending RLCR to Tool-Use Agents via MSCR: MSCR addresses reward overlap by strictly separating reward landscapes for correct and incorrect trajectories while shaping confidence within each region.The design preserves a correctness margin and improves calibration and failure discrimination beyond simple rescaling.
- Extending RLCR to Tool-Use Agents via MSCR: Calibration-motivated reinforcement learning generalizes to noisy retrieval environments when incentive separation is maintained.The discussion links reward overlap to sensitivity to data difficulty and confidence collapse.
7 Conclusion
The paper identifies a tool-dependent confidence dichotomy and proposes CAR with MSCR to improve calibration while preserving task performance. Experiments support generalization from local simulation to noisy API environments and motivate tool-specific calibration strategies.
- Conclusion: Verification tools ground reasoning through deterministic feedback, whereas evidence tools introduce stochastic noise that systematically induces overconfidence.The conclusion presents this dichotomy as the central calibration finding.
- Conclusion: CAR incorporates MSCR to strictly separate incentives for correct and incorrect predictions.The framework significantly reduces calibration error while maintaining competitive task performance.
- Conclusion: CAR generalizes from local simulation to noisy, real-world API environments.The conclusion presents this robustness as evidence supporting tool-specific calibration strategies.
Limitations
The evaluation is constrained to relatively small models and does not establish how calibration changes with scale. It also centers on tasks with well-defined correctness, leaving open-ended generation and planning insufficiently addressed.
- The experiments focus on models with 3B to 7B parameters because of computational constraints.
- Although consistent patterns appear across three backbone architectures, how the confidence dichotomy evolves with model scale remains unclear.
- Evaluation primarily covers short-answer question answering and mathematical reasoning, where correctness is well-defined.
- The framework does not address calibration in open-ended tasks such as long-form report writing or multi-step autonomous planning, which may have underspecified correctness signals or delayed feedback.