PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling

Kangjia Zhao, Jiajun Li, Haozhan Shen, Wei Chow, Linfeng Li, Hang Song, Lingdong Kong, Chen Zhi, Tiancheng Zhao, Songhua Liu, Jianwei Yin

arXiv:2609.00949v1cs.CLcs.AI

TL;DR

Aggregate trajectory accuracy does not reveal whether models choose the action class required by each multi-turn context. The paper introduces GAR-based action diagnostics and SRI/CRI probes, finding substantial, family-dependent miscalibration and heterogeneous intervention effects. It recommends pairing aggregate accuracy with action-class diagnostics, while noting that the framework diagnoses rather than identifies training causes and depends on gold action classes.

  • Problem

    Whole-trajectory accuracy provides limited evidence about whether models choose the contextually appropriate action among TOOL_CALL, ASK, REFUSE, and CONFIRM.

    Source: §1 · p. 2, §4.1 · p. 5

  • Method

    The paper classifies emissions by action class, pairs Gold Action Recall with trajectory accuracy through Acc ≤ GAR, and probes calibration with SRI and CRI.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Action-class miscalibration is a substantial failure mode, with state-grader masking and family-dependent calibration; retry and bypass interventions produce different accuracy-emission trade-offs.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    Multi-turn tool-calling evaluations should report GAR alongside aggregate accuracy to expose model action choices in each scenario.

    Source: §1 · p. 2

    The probes and GAR require category-level gold action classes, and plasticity results cover only 6 of 26 main-panel checkpoints.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now approach or even surpass closed-source frontier models in aggregate accuracy. However, this metric averages over many different multi-turn situations and obscures whether progress is balanced across them. We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two orthogonal modes: action-class miscalibration and action-execution failure. The framework operates over a four-class action space (TOOL_CALL/ASK/REFUSE/CONFIRM) and introduces a self-revealing upper bound Acc <= GAR (Gold Action Recall); the two modes show up as bound violation (Acc > GAR, exposing state-grader masking of miscalibration) and large bound slack (GAR >> Acc, localizing execution failure within TOOL_CALL). We validate it on a panel of tool-calling models across multiple multi-turn benchmarks. Across our panel, the diagnostic reveals action-class miscalibration as a substantial failure mode the state grader cannot see. This gap inflates standing for heavily tool-trained families, which our diagnostic separates from families with context-appropriate action choice. Calibration is reshapable through context-only perturbations, but the reshape is heterogeneous: a single perturbation moves accuracy in opposite directions across families (up to +11.5 vs -21.0 pp on the same scenario), and its effect further depends on the perturbation mechanism. We argue that multi-turn tool-calling evaluations should supplement aggregate accuracy with action-class diagnostics that expose what the model actually does in each scenario.

1 Introduction

Multi-turn tool-use evaluation can miss whether a model chose the contextually appropriate action class, even when its trajectory eventually reaches the correct state. This paper introduces action-class diagnostics and inference-time probes to expose and study that gap.

Motivation

  • Multi-turn evaluations score whole trajectories, but models may fail to choose between TOOL_CALL, ASK, REFUSE, and CONFIRM.Well-formed tool calls do not guarantee the correct decision for the current turn.Source: §1 · p. 2
  • 71.8% aggregate accuracy coexists with 19.5% Gold Action Recall on xLAM’s miss_param cases, a 64.5 pp gap below size-matched peers.BFCL’s state grader still marks many such trajectories as PASS, masking action-class misses.Source: §1 · p. 2

Approach

  • The framework decomposes failures into action-class miscalibration and action-execution failure using a four-class action space and Gold Action Recall.It evaluates BFCL v3, cross-checks τ 2-bench retail and airline, and verifies mechanisms through a 200-case audit.Source: §1 · p. 2
  • SRI targets missed tool calls, while CRI targets unwarranted tool calls through symmetric inference-time interventions.The probes test whether calibration can be reshaped by changing context at inference time.Source: §1 · p. 2

Findings

  • Retry-style CRI shifts emissions toward the gold class but generally harms trajectory accuracy, whereas bypass-style CRI preserves or improves accuracy without changing emissions.The two mechanisms separate changing the model’s response from preventing execution of an erroneous call.Source: §1 · p. 2

Implication

  • The paper recommends reporting GAR alongside aggregate accuracy because it exposes each model’s action choice in each scenario.This supplements state-based evaluation with a direct action-class diagnostic.Source: §1 · p. 2

2 Related Work

Prior work studies tool-use and agent failures at multiple granularities, critiques evaluation and reward design, and improves multi-turn tool use through training. This paper positions its contribution as train-free diagnosis of calibration profiles.

Failure analysis

  • Existing work maps tool-use and broader agent failures from whole trajectories down to individual arguments and develops critics to detect them.Related studies also examine canonical-path drift and agents accepting false assertions from users or tools.Source: §2 · p. 2

Evaluation

  • Prior research documents reward-design flaws that distort measured performance, including graders counting empty responses as successes.The paper connects this broader evaluation concern to its own focus on state-grader masking.Source: §2 · p. 2

Inference-time methods

  • Inference-time interception has been studied for reducing hallucination, while intrinsic self-correction can degrade performance.The paper distinguishes its SRI and CRI probes as diagnostic perturbations rather than production fixes.Source: §2 · p. 2

Training approaches

  • Training-side approaches improve multi-turn tool use through supervised fine-tuning or reinforcement learning.In contrast, this paper isolates and diagnoses calibration profiles without training.Source: §2 · p. 3

3 Method

The method models tool use as state-action trajectories, adds per-emission action classes and GAR beside trajectory accuracy, and uses SRI and CRI to probe calibration at missed-call and unwarranted-call seams.

Trajectory model

  • Each case is a state-action trajectory in which model emissions and simulator transitions produce a runtime state judged against a gold reference.The grader yields conversation accuracy by comparing runtime and gold states turn by turn.Source: §3.1 · p. 3

Action classes

  • Each emission is classified as TOOL_CALL, ASK, REFUSE, CONFIRM, or residual OTHER under a fixed priority order.TOOL_CALL uses syntax, while the other named classes use textual cues.Source: §3.1 · p. 3
  • BFCL assigns TOOL_CALL to base and long_context, ASK to miss_param, and REFUSE to miss_func.These category-specific gold actions define the contextually appropriate class used by the diagnostic.Source: §3.1 · p. 3

Diagnostic metrics

  • GAR measures whether the gold action class appears at any turn, while Acc measures whether the trajectory passes the benchmark grader.The diagnostic uses case-level, any-turn aggregation for GAR.Source: §3.2 · p. 3
  • Acc ≤ GAR identifies the diagnostic relationship: Acc > GAR signals state-grader masking, while large GAR–Acc slack signals downstream execution failure.Execution failures include wrong tools, arguments, or state tracking after the gold class fires.Source: §3.2 · p. 3

Intervention probes

  • SRI reconciles runtime tool-execution state when a model halts before a required TOOL_CALL, and CRI reconciles suspect calls when the gold action is non-call.The probes are inference-time diagnostic perturbations for missed and unwarranted tool calls.Source: §3.3 · p. 3
  • CRI-retry re-queries after retracting a suspect call, whereas CRI-bypass substitutes fixed abstention without re-querying.Comparing them separates improved model re-emission from simply preventing erroneous execution.Source: §3.3 · pp. 3–4

Evaluation

  • The evaluation uses BFCL v3’s four categories with 200 conversations each across open-weight families and closed-source API anchors.Models run in native function-calling mode, and each model-category cell reports GAR and conversation accuracy.Source: §3.4 · pp. 4–5

4 Experiments

Experiments show that action-class calibration varies more across model families than sizes, while aggregate accuracy can mask distinct execution and refusal failures. Cross-benchmark and intervention results further show that calibration shifts are family- and mechanism-dependent.

Calibration profile

  • 83.5 pp separates miss_param GAR for Hammer-2.1-7B and Qwen3-8B at 7–8B, exceeding within-family calibration changes from scaling.Qwen3’s miss_param GAR varies by only 5.5 pp for 4B and larger models, while xLAM-2 stays below 20% across sizes.Source: §4.1 · p. 5
  • All 26 general-purpose rows have ∆≥+17 pp on tool-call categories, indicating near-universal gold TOOL_CALL emission but downstream execution failures.These failures include wrong tools, arguments, or broken multi-turn state; high TOOL_CALL emission alone does not establish context-sensitive recognition.Source: §4.1 · p. 5, Table 1 · p. 6
  • General-purpose families show opposite missing-information patterns: positive ∆ on miss_param reflects post-ASK execution failure, whereas negative miss_func ∆ reflects under-refusal masked by the state grader.The paper links the asymmetry to training pressure: ASK resembles natural dialogue, while REFUSE conflicts with function-calling recipes optimized for emitting calls.Source: §4.1 · p. 5, Table 1 · p. 6
  • Tool-specialized families show negative ∆ on both missing-information categories, including xLAM-2-70b at −59.5/−66.0 pp while topping open-weight accuracy at 77.5%.A 200-case audit found that 92.3% of audited xLAM-2-8b miss_func misses invoked the held-out function, while gpt-5.4 and Gemini 3 Pro maintained calibration.Source: §4.1 · p. 5, §4.2 · p. 5, Table 1 · p. 6

Cross-benchmark validation

  • On τ 2, GAR = 0% yet pass rates reach 37.2% for airline and 4.5% for retail on tool-specialized TOOL_CALL tasks, reflecting reference-path rather than action-class failure.The relevant reference paths contain only read-only lookups, while strict checks penalize count, argument, or order mismatches; lenient end-state rewards do not.Source: §4.2 · pp. 6–7

Calibration plasticity

  • +11.5 pp and −21.0 pp are opposite SRI accuracy shifts for Qwen3-8B and ToolACE-2-8B on the same base scenario, with GAR remaining at ceiling for ToolACE.CRI-retry moves GAR toward the gold class but lowers accuracy in 11 of 12 missing-info cells, whereas CRI-bypass raises accuracy in 11 of 12, peaking at +13.0 pp.Source: §4.4 · pp. 8–9

5 Conclusion

The framework decomposes multi-turn tool-calling failures into action-class miscalibration and action-execution failure using the bound Acc ≤ GAR. Across BFCL and τ 2-bench, it identifies miscalibration as the bottleneck and shows that calibration can be reshaped by inference-time probes, with family- and mechanism-dependent costs.

  • The framework decomposes multi-turn tool-calling failures into action-class miscalibration and action-execution failure via the bound Acc ≤ GAR.Source: §5 · p. 9
  • Across the BFCL and τ 2-bench panel, miscalibration is the bottleneck, shaped by family recipe and invisible to the state grader.Source: §5 · p. 9
  • Inference-time probes show that calibration is reshapable, but the reshape varies across model families and perturbation mechanisms.Source: §5 · p. 9
  • Retry-style intervention moves emission toward the gold class at the expense of completion, whereas bypass-style guarding preserves completion without changing emission.Source: §5 · p. 9
  • The framework diagnoses failures and intervention costs but does not itself provide a correction; choosing between interventions remains future work.Source: §5 · p. 9

Limitations

The study’s conclusions are bounded by its diagnostic scope: it localizes calibration failures without identifying their training cause, uses two benchmarks, and tests probes on only part of the panel with oracle action-class information.

  • The framework localizes action-class miscalibration to families and categories but does not identify which training-stage decision is responsible.Source: Limitations · pp. 9–10
  • Fixing calibration requires controlled training ablations beyond this work.Source: Limitations · p. 10
  • The calibration signature is evaluated on BFCL multi-turn and τ 2-bench, while extension to other benchmarks remains future work.Source: Limitations · p. 10
  • Plasticity results cover 6 of 26 checkpoints, and SRI, CRI, and GAR depend on gold action classes unavailable in deployment.Source: Limitations · p. 10
  • A single policy-based instance does not establish that judges can recover gold action classes generally.Source: Limitations · p. 10

Ethical Considerations

The study uses public benchmarks and releases audit materials and probe protocols, while its capability check and intervention formalization clarify how the diagnostic operates without retraining models.

  • The study evaluates public BFCL v3 multi-turn and τ 2-bench retail/airline benchmarks without collecting human-subject data or retraining model weights.Source: Ethical Considerations · p. 10
  • Audit annotations were produced by two independent LLM raters using a frozen rubric, with labels, adjudications, and probe protocols released alongside the work.Source: Ethical Considerations · p. 10
  • Each family emits all four non-OTHER action classes above Tcap = 10 over N = 800 cases in the representative capability check.Source: Table 5 · p. 13, A Capability counts (extends §4.1) · p. 13
  • The Halt Detector triggers when a multi-user-message conversation emits no tool call while an outstanding sub-task remains, routing the case to SRI retry.Source: B Intervention formalization (extends… · p. 13
  • SRI rewrites the next-turn context using history editing, state injection, and reconciliation instructions before applying a decision rule for tool calling or normal response.Source: §B.1 · p. 13
  • SRI fires at most once per turn boundary, and ablations remove history editing, state injection, or reconciliation components.Source: §B.1 · p. 13

C.1 SRI ablation tables and component decomposition

The SRI components play distinct roles: history editing is broadly beneficial, while detection, reconciliation, and state injection vary by family and can shift accuracy differently from action-class recall.

  • History editing is universally positive and Qwen-dominant, adding +7.88 pp.Source: §C.1 · p. 13
  • Halt detection is family-conditional, positive on Qwen and gpt-oss but negative on xLAM.Source: §C.1 · p. 13
  • Reconciliation raises gpt-oss-20b accuracy by +3.00 pp but lowers xLAM accuracy by −3.50 pp.Source: §C.1 · p. 13
  • On xLAM, reconciliation lowers accuracy while shifting miss_param GAR +9.5 pp toward the gold class.Source: §C.1 · p. 13
  • State injection’s effect correlates with base strength: +0.75 pp on Qwen, −3.13 pp on gpt-oss, and −1.00 pp on xLAM.Source: §C.1 · p. 13

C.2 CRI methodology and full results

CRI separates model-emission changes from downstream trajectory effects, but bypass-based GAR is not a valid model-behavior measure. Retry shifts emissions toward gold while usually reducing accuracy; bypass preserves or improves accuracy by replacing suspect calls with system-written placeholders.

SRI comparison

  • The SRI ablation matrix compares per-category GAR shifts with family-level accuracy changes, whose gaps diagnose bound violations rather than interchangeable effects.The full matrix covers SRI, SRI-Gate, and ablations across three families and 800 BFCL multi-turn cases.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 15

Measurement

  • CRI-bypass GAR is ill-defined because its placeholder is system-written rather than a model emission.Both plausible readings confound or omit the bypass intervention, so bypass reports Acc and fired_rate instead.Source: §C.2 · p. 14
  • Bypass-fired miss_func shifts track detector fired_rate, indicating placeholder injection rather than model behavior.Watt-Tool-8B, ToolACE-2-8B, and xLAM-2-8b show near-matched fired rates and Acc shifts; the miss_param falsifier removes this spurious shift.Source: §C.2 · p. 14

CRI results

  • Retry moves emission GAR toward the gold class but degrades trajectory accuracy in 11 of 12 missing-info cells.The single positive cell is Watt-Tool-8B miss_param at +5.0 pp, where abstention can align with supplement-turn structure.Source: Table 10 · p. 15, §C.2 · p. 15

D Cross-benchmark SRI-lite validation (extends §4.2)

SRI-lite extends the calibration-plasticity test to τ 2-bench retail and airline tasks using family-specific self-simulation. Its accuracy effect follows halt-presence across families and remains near-zero or negative on airline.

Experimental setting

  • SRI-lite experiments use per-family self-simulation rather than the fixed gpt-5-mini simulator used in the main diagnostic table.The intervention sweep requires model-specific retry behavior that a generic simulator cannot replicate.Source: D Cross-benchmark SRI-lite validation… · p. 15

Family heterogeneity

  • Qwen3-8B gains +2.63 pp on τ 2-retail, while gpt-oss and xLAM-2-8b each lose −9.65 pp.The sign matches BFCL across all three families and tracks halt-presence: SRI helps halt-prone Qwen and hurts the lower-halt-presence families.Source: D Cross-benchmark SRI-lite validation… · pp. 15–16

Retail validation

Interpretation

  • The τ 2 grader can distribute failures across wrong tools or arguments, state tracking, policy violations, and no-tool-call outcomes.Unlike BFCL, it does not absorb halt-targeted side effects through the bound-violation tool-call-error path.Source: D Cross-benchmark SRI-lite validation… · pp. 15–16

Airline validation

E Turn-level GAR validation (extends §4.1)

Turn-level GAR recomputation largely preserves the paper’s family-level conclusions, while revealing uncertainty in some family ordering comparisons. The xLAM miss_func deviation remains robust, but case-level and audited firing metrics are not interchangeable.

Turn-level validation

Uncertainty

  • Qwen and gpt-oss reverse miss_param ordering between case-level and turn-level GAR, but both differences fall within Wilson 95% confidence-interval overlap.The case-level difference is 8 pp and the turn-level difference is 2.5 pp on N = 200.Source: E Turn-level GAR validation (extends… · p. 16

Robustness

Classification rule

  • The priority rule classifies emissions that both call a tool and ask the user as TOOL_CALL, changing miss_param GAR by 1.5–4.5 pp without changing ordering.Crediting the question instead yields the reported alternative values for four families.Source: E Turn-level GAR validation (extends… · p. 16

F.1 Per-label κ and post-adjudication mechanism rates

The unified audit validates category-level action labeling against reported GAR and provides descriptive mechanism rates after adjudication. Agreement is strong for the primary audited categories, but the SRI-failure mechanism subset is difficult to attribute reliably.

Audit design

  • The 200-case audit is stratified across seven strata, including 48 xLAM miss_func over-call cases and 22 Qwen baseline-PASS to SRI-FAIL cases.A separate 20-case pilot uses the same rubric.Source: §F.1 · p. 17

Validation

  • Adjudicated B1a rates align with paper GAR, including xLAM miss_func at 10.4% versus 10% and Qwen miss_param at 19.0% versus 19.5%.Tool-call categories audit at ≥90% B1a as expected.Source: §F.1 · p. 17

Mechanism attribution

  • B7 mechanism attribution is descriptive because its non-n.a. inter-rater agreement is low at κ = −0.006.Among 52 D+F cases, the post-adjudication counts are 24 state-injection misuse, 14 reconciliation-prompt overfit, 11 unclassified, and 3 history-edit artifact.Source: §F.1 · p. 17

Rubric

  • The audit defines REFUSE for miss_func and ASK for miss_param according to the calibration framing rather than BFCL’s ground-truth tool match.This rubric choice determines the gold action class used in the audit.Source: §F.2 · p. 17

Reporting

  • The audit reports per-label pre-adjudication agreement and post-adjudication mechanism rates, with B7 separated as descriptive.The primary headline statistic is the macro-mean across B1a, B1b, B3, B4, and B5.Source: Table 15 · p. 18, Table 16 · p. 18, Table 17 · p. 18

Related papers

Loading 2609.00949v1…