PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling

Kangjia Zhao, Jiajun Li, Haozhan Shen, Wei Chow, Linfeng Li, Hang Song, Lingdong Kong, Chen Zhi, Tiancheng Zhao, Songhua Liu, Jianwei Yin

arXiv:2609.00949v1cs.CLcs.AI

TL;DR

Aggregate trajectory accuracy does not reveal whether models choose the action class required by each multi-turn context. The paper introduces GAR-based action diagnostics and SRI/CRI probes, finding substantial, family-dependent miscalibration and heterogeneous intervention effects. It recommends pairing aggregate accuracy with action-class diagnostics, while noting that the framework diagnoses rather than identifies training causes and depends on gold action classes.

  • Problem

    Whole-trajectory accuracy provides limited evidence about whether models choose the contextually appropriate action among TOOL_CALL, ASK, REFUSE, and CONFIRM.

  • Method

    The paper classifies emissions by action class, pairs Gold Action Recall with trajectory accuracy through Acc ≤ GAR, and probes calibration with SRI and CRI.

  • Results

    Action-class miscalibration is a substantial failure mode, with state-grader masking and family-dependent calibration; retry and bypass interventions produce different accuracy-emission trade-offs.

  • Takeaways & Limitations

    Multi-turn tool-calling evaluations should report GAR alongside aggregate accuracy to expose model action choices in each scenario.

  • Takeaways & Limitations

    The probes and GAR require category-level gold action classes, and plasticity results cover only 6 of 26 main-panel checkpoints.

Abstract

from arXiv · show

Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now approach or even surpass closed-source frontier models in aggregate accuracy. However, this metric averages over many different multi-turn situations and obscures whether progress is balanced across them. We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two orthogonal modes: action-class miscalibration and action-execution failure. The framework operates over a four-class action space (TOOL_CALL/ASK/REFUSE/CONFIRM) and introduces a self-revealing upper bound Acc <= GAR (Gold Action Recall); the two modes show up as bound violation (Acc > GAR, exposing state-grader masking of miscalibration) and large bound slack (GAR >> Acc, localizing execution failure within TOOL_CALL). We validate it on a panel of tool-calling models across multiple multi-turn benchmarks. Across our panel, the diagnostic reveals action-class miscalibration as a substantial failure mode the state grader cannot see. This gap inflates standing for heavily tool-trained families, which our diagnostic separates from families with context-appropriate action choice. Calibration is reshapable through context-only perturbations, but the reshape is heterogeneous: a single perturbation moves accuracy in opposite directions across families (up to +11.5 vs -21.0 pp on the same scenario), and its effect further depends on the perturbation mechanism. We argue that multi-turn tool-calling evaluations should supplement aggregate accuracy with action-class diagnostics that expose what the model actually does in each scenario.

1 Introduction

Multi-turn tool-use evaluation can miss whether a model chose the contextually appropriate action class, even when its trajectory eventually reaches the correct state. This paper introduces action-class diagnostics and inference-time probes to expose and study that gap.

  • Motivation: Multi-turn evaluations score whole trajectories, but models may fail to choose between TOOL_CALL, ASK, REFUSE, and CONFIRM.Well-formed tool calls do not guarantee the correct decision for the current turn.
  • Motivation: 71.8% aggregate accuracy coexists with 19.5% Gold Action Recall on xLAM’s miss_param cases, a 64.5 pp gap below size-matched peers.BFCL’s state grader still marks many such trajectories as PASS, masking action-class misses.
  • Approach: The framework decomposes failures into action-class miscalibration and action-execution failure using a four-class action space and Gold Action Recall.It evaluates BFCL v3, cross-checks τ 2-bench retail and airline, and verifies mechanisms through a 200-case audit.
  • Approach: SRI targets missed tool calls, while CRI targets unwarranted tool calls through symmetric inference-time interventions.The probes test whether calibration can be reshaped by changing context at inference time.
  • Findings: Retry-style CRI shifts emissions toward the gold class but generally harms trajectory accuracy, whereas bypass-style CRI preserves or improves accuracy without changing emissions.The two mechanisms separate changing the model’s response from preventing execution of an erroneous call.
  • Implication: The paper recommends reporting GAR alongside aggregate accuracy because it exposes each model’s action choice in each scenario.This supplements state-based evaluation with a direct action-class diagnostic.

2 Related Work

Prior work studies tool-use and agent failures at multiple granularities, critiques evaluation and reward design, and improves multi-turn tool use through training. This paper positions its contribution as train-free diagnosis of calibration profiles.

  • Failure analysis: Existing work maps tool-use and broader agent failures from whole trajectories down to individual arguments and develops critics to detect them.Related studies also examine canonical-path drift and agents accepting false assertions from users or tools.
  • Evaluation: Prior research documents reward-design flaws that distort measured performance, including graders counting empty responses as successes.The paper connects this broader evaluation concern to its own focus on state-grader masking.
  • Inference-time methods: Inference-time interception has been studied for reducing hallucination, while intrinsic self-correction can degrade performance.The paper distinguishes its SRI and CRI probes as diagnostic perturbations rather than production fixes.
  • Training approaches: Training-side approaches improve multi-turn tool use through supervised fine-tuning or reinforcement learning.In contrast, this paper isolates and diagnoses calibration profiles without training.

3 Method

The method models tool use as state-action trajectories, adds per-emission action classes and GAR beside trajectory accuracy, and uses SRI and CRI to probe calibration at missed-call and unwarranted-call seams.

  • Trajectory model: Each case is a state-action trajectory in which model emissions and simulator transitions produce a runtime state judged against a gold reference.The grader yields conversation accuracy by comparing runtime and gold states turn by turn.
  • Action classes: Each emission is classified as TOOL_CALL, ASK, REFUSE, CONFIRM, or residual OTHER under a fixed priority order.TOOL_CALL uses syntax, while the other named classes use textual cues.
  • Action classes: BFCL assigns TOOL_CALL to base and long_context, ASK to miss_param, and REFUSE to miss_func.These category-specific gold actions define the contextually appropriate class used by the diagnostic.
  • Diagnostic metrics: GAR measures whether the gold action class appears at any turn, while Acc measures whether the trajectory passes the benchmark grader.The diagnostic uses case-level, any-turn aggregation for GAR.
  • Diagnostic metrics: Acc ≤ GAR identifies the diagnostic relationship: Acc > GAR signals state-grader masking, while large GAR–Acc slack signals downstream execution failure.Execution failures include wrong tools, arguments, or state tracking after the gold class fires.
  • Intervention probes: SRI reconciles runtime tool-execution state when a model halts before a required TOOL_CALL, and CRI reconciles suspect calls when the gold action is non-call.The probes are inference-time diagnostic perturbations for missed and unwarranted tool calls.
  • Intervention probes: CRI-retry re-queries after retracting a suspect call, whereas CRI-bypass substitutes fixed abstention without re-querying.Comparing them separates improved model re-emission from simply preventing erroneous execution.
  • Evaluation: The evaluation uses BFCL v3’s four categories with 200 conversations each across open-weight families and closed-source API anchors.Models run in native function-calling mode, and each model-category cell reports GAR and conversation accuracy.

4 Experiments

Experiments show that action-class calibration varies more across model families than sizes, while aggregate accuracy can mask distinct execution and refusal failures. Cross-benchmark and intervention results further show that calibration shifts are family- and mechanism-dependent.

  • Calibration profile: 83.5 pp separates miss_param GAR for Hammer-2.1-7B and Qwen3-8B at 7–8B, exceeding within-family calibration changes from scaling.Qwen3’s miss_param GAR varies by only 5.5 pp for 4B and larger models, while xLAM-2 stays below 20% across sizes.
  • Calibration profile: All 26 general-purpose rows have ∆≥+17 pp on tool-call categories, indicating near-universal gold TOOL_CALL emission but downstream execution failures.These failures include wrong tools, arguments, or broken multi-turn state; high TOOL_CALL emission alone does not establish context-sensitive recognition.
  • Calibration profile: General-purpose families show opposite missing-information patterns: positive ∆ on miss_param reflects post-ASK execution failure, whereas negative miss_func ∆ reflects under-refusal masked by the state grader.The paper links the asymmetry to training pressure: ASK resembles natural dialogue, while REFUSE conflicts with function-calling recipes optimized for emitting calls.
  • Calibration profile: Tool-specialized families show negative ∆ on both missing-information categories, including xLAM-2-70b at −59.5/−66.0 pp while topping open-weight accuracy at 77.5%.A 200-case audit found that 92.3% of audited xLAM-2-8b miss_func misses invoked the held-out function, while gpt-5.4 and Gemini 3 Pro maintained calibration.
  • Cross-benchmark validation: On τ 2, GAR = 0% yet pass rates reach 37.2% for airline and 4.5% for retail on tool-specialized TOOL_CALL tasks, reflecting reference-path rather than action-class failure.The relevant reference paths contain only read-only lookups, while strict checks penalize count, argument, or order mismatches; lenient end-state rewards do not.
  • Calibration plasticity: +11.5 pp and −21.0 pp are opposite SRI accuracy shifts for Qwen3-8B and ToolACE-2-8B on the same base scenario, with GAR remaining at ceiling for ToolACE.CRI-retry moves GAR toward the gold class but lowers accuracy in 11 of 12 missing-info cells, whereas CRI-bypass raises accuracy in 11 of 12, peaking at +13.0 pp.

5 Conclusion

The framework decomposes multi-turn tool-calling failures into action-class miscalibration and action-execution failure using the bound Acc ≤ GAR. Across BFCL and τ 2-bench, it identifies miscalibration as the bottleneck and shows that calibration can be reshaped by inference-time probes, with family- and mechanism-dependent costs.

  • The framework decomposes multi-turn tool-calling failures into action-class miscalibration and action-execution failure via the bound Acc ≤ GAR.
  • Across the BFCL and τ 2-bench panel, miscalibration is the bottleneck, shaped by family recipe and invisible to the state grader.
  • Inference-time probes show that calibration is reshapable, but the reshape varies across model families and perturbation mechanisms.
  • Retry-style intervention moves emission toward the gold class at the expense of completion, whereas bypass-style guarding preserves completion without changing emission.
  • The framework diagnoses failures and intervention costs but does not itself provide a correction; choosing between interventions remains future work.

Limitations

The study’s conclusions are bounded by its diagnostic scope: it localizes calibration failures without identifying their training cause, uses two benchmarks, and tests probes on only part of the panel with oracle action-class information.

  • The framework localizes action-class miscalibration to families and categories but does not identify which training-stage decision is responsible.
  • Fixing calibration requires controlled training ablations beyond this work.
  • The calibration signature is evaluated on BFCL multi-turn and τ 2-bench, while extension to other benchmarks remains future work.
  • Plasticity results cover 6 of 26 checkpoints, and SRI, CRI, and GAR depend on gold action classes unavailable in deployment.
  • A single policy-based instance does not establish that judges can recover gold action classes generally.

Ethical Considerations

The study uses public benchmarks and releases audit materials and probe protocols, while its capability check and intervention formalization clarify how the diagnostic operates without retraining models.

  • The study evaluates public BFCL v3 multi-turn and τ 2-bench retail/airline benchmarks without collecting human-subject data or retraining model weights.
  • Audit annotations were produced by two independent LLM raters using a frozen rubric, with labels, adjudications, and probe protocols released alongside the work.
  • Each family emits all four non-OTHER action classes above Tcap = 10 over N = 800 cases in the representative capability check.
  • The Halt Detector triggers when a multi-user-message conversation emits no tool call while an outstanding sub-task remains, routing the case to SRI retry.
  • SRI rewrites the next-turn context using history editing, state injection, and reconciliation instructions before applying a decision rule for tool calling or normal response.
  • SRI fires at most once per turn boundary, and ablations remove history editing, state injection, or reconciliation components.

C.1 SRI ablation tables and component decomposition

The SRI components play distinct roles: history editing is broadly beneficial, while detection, reconciliation, and state injection vary by family and can shift accuracy differently from action-class recall.

  • History editing is universally positive and Qwen-dominant, adding +7.88 pp.
  • Halt detection is family-conditional, positive on Qwen and gpt-oss but negative on xLAM.
  • Reconciliation raises gpt-oss-20b accuracy by +3.00 pp but lowers xLAM accuracy by −3.50 pp.
  • On xLAM, reconciliation lowers accuracy while shifting miss_param GAR +9.5 pp toward the gold class.
  • State injection’s effect correlates with base strength: +0.75 pp on Qwen, −3.13 pp on gpt-oss, and −1.00 pp on xLAM.

C.2 CRI methodology and full results

CRI separates model-emission changes from downstream trajectory effects, but bypass-based GAR is not a valid model-behavior measure. Retry shifts emissions toward gold while usually reducing accuracy; bypass preserves or improves accuracy by replacing suspect calls with system-written placeholders.

  • SRI comparison: The SRI ablation matrix compares per-category GAR shifts with family-level accuracy changes, whose gaps diagnose bound violations rather than interchangeable effects.The full matrix covers SRI, SRI-Gate, and ablations across three families and 800 BFCL multi-turn cases.
  • Measurement: CRI-bypass GAR is ill-defined because its placeholder is system-written rather than a model emission.Both plausible readings confound or omit the bypass intervention, so bypass reports Acc and fired_rate instead.
  • Measurement: Bypass-fired miss_func shifts track detector fired_rate, indicating placeholder injection rather than model behavior.Watt-Tool-8B, ToolACE-2-8B, and xLAM-2-8b show near-matched fired rates and Acc shifts; the miss_param falsifier removes this spurious shift.
  • CRI results: Retry moves emission GAR toward the gold class but degrades trajectory accuracy in 11 of 12 missing-info cells.The single positive cell is Watt-Tool-8B miss_param at +5.0 pp, where abstention can align with supplement-turn structure.

D Cross-benchmark SRI-lite validation (extends §4.2)

SRI-lite extends the calibration-plasticity test to τ 2-bench retail and airline tasks using family-specific self-simulation. Its accuracy effect follows halt-presence across families and remains near-zero or negative on airline.

  • Experimental setting: SRI-lite experiments use per-family self-simulation rather than the fixed gpt-5-mini simulator used in the main diagnostic table.The intervention sweep requires model-specific retry behavior that a generic simulator cannot replicate.
  • Family heterogeneity: Qwen3-8B gains +2.63 pp on τ 2-retail, while gpt-oss and xLAM-2-8b each lose −9.65 pp.The sign matches BFCL across all three families and tracks halt-presence: SRI helps halt-prone Qwen and hurts the lower-halt-presence families.
  • Retail validation: SRI-lite reproduces BFCL aggregate accuracy-shift directions on τ 2-retail across three families.The retail check uses 114 tasks per family, with infrastructure error rates of 0–1.75%.
  • Interpretation: The τ 2 grader can distribute failures across wrong tools or arguments, state tracking, policy violations, and no-tool-call outcomes.Unlike BFCL, it does not absorb halt-targeted side effects through the bound-violation tool-call-error path.
  • Airline validation: On τ 2-airline, all three families show near-zero or negative SRI-lite accuracy changes.The airline set contains 50 tasks and serves as a within-benchmark cross-domain check.

E Turn-level GAR validation (extends §4.1)

Turn-level GAR recomputation largely preserves the paper’s family-level conclusions, while revealing uncertainty in some family ordering comparisons. The xLAM miss_func deviation remains robust, but case-level and audited firing metrics are not interchangeable.

  • Turn-level validation: The turn-level GAR check covers 14 missing-info cells across seven families and averages gold-action emission over gold turns.Hammer-2.1-7B emits the gold class on no gold turn, leaving its two ratios undefined.
  • Turn-level validation: A median 1.23× case/turn GAR ratio does not move any family across the general-purpose/tool-specialized split.At turn level, Qwen3-14B reaches 38.0% on miss_func versus Hammer-2.1-7B at 0.0%.
  • Uncertainty: Qwen and gpt-oss reverse miss_param ordering between case-level and turn-level GAR, but both differences fall within Wilson 95% confidence-interval overlap.The case-level difference is 8 pp and the turn-level difference is 2.5 pp on N = 200.
  • Robustness: The xLAM miss_func outlier status is robust at both case and turn levels.The case-vs-turn gap is 5.5 pp, but it measures all 200 cases rather than the audited held-out-tool subset.
  • Classification rule: The priority rule classifies emissions that both call a tool and ask the user as TOOL_CALL, changing miss_param GAR by 1.5–4.5 pp without changing ordering.Crediting the question instead yields the reported alternative values for four families.

F.1 Per-label κ and post-adjudication mechanism rates

The unified audit validates category-level action labeling against reported GAR and provides descriptive mechanism rates after adjudication. Agreement is strong for the primary audited categories, but the SRI-failure mechanism subset is difficult to attribute reliably.

  • Audit design: The 200-case audit is stratified across seven strata, including 48 xLAM miss_func over-call cases and 22 Qwen baseline-PASS to SRI-FAIL cases.A separate 20-case pilot uses the same rubric.
  • Validation: Adjudicated B1a rates align with paper GAR, including xLAM miss_func at 10.4% versus 10% and Qwen miss_param at 19.0% versus 19.5%.Tool-call categories audit at ≥90% B1a as expected.
  • Mechanism attribution: B7 mechanism attribution is descriptive because its non-n.a. inter-rater agreement is low at κ = −0.006.Among 52 D+F cases, the post-adjudication counts are 24 state-injection misuse, 14 reconciliation-prompt overfit, 11 unclassified, and 3 history-edit artifact.
  • Rubric: The audit defines REFUSE for miss_func and ASK for miss_param according to the calibration framing rather than BFCL’s ground-truth tool match.This rubric choice determines the gold action class used in the audit.
  • Reporting: The audit reports per-label pre-adjudication agreement and post-adjudication mechanism rates, with B7 separated as descriptive.The primary headline statistic is the macro-mean across B1a, B1b, B3, B4, and B5.
Loading 2609.00949v1…