Source-linked AI summary
Invocation-Level Reliability of Tool-Using Agents
Afiya Noorain, Subhranshu Mohanty, Amritesh Banerjee, Abhijit Dasgupta
TL;DR
Tool-using agents can fail through tool selection or argument errors, with early mistakes contaminating later calls. The paper measures invocation reliability in clean and free-running contexts and finds that exact-match scoring fixes propagation parameters rather than identifying them. Conditional-on-state scoring, applied to cached completions, removes this scoring artifact.
Problem
Tool-use evaluation often aggregates success and does not distinguish invocation errors or their propagation through multi-step chains.
Method
The paper compares teacher-forced and free-running invocation accuracy on controlled dependency tasks and applies conditional-on-state scoring retrospectively to cached completions.
Results
By depth 6, roughly 70% of clean-context capability is lost to earlier mistakes, while exact-match scoring forces severity to 1 and leaves recovery unobservable.
Takeaways & Limitations
Under fixed-reference scoring, fitted severity and recovery can reflect the scorer rather than the model, so propagation claims require a fit-free or state-conditional metric.
Takeaways & Limitations
The study uses controlled synthetic integer-argument tasks, so whether its conditional-scoring and identifiability findings extend to natural-language or multi-field arguments remains open.
Abstract
from arXiv · showhide
Tool-using agents fail two ways: choosing the wrong tool, or forming wrong arguments, and an early failure of either kind can silently corrupt everything downstream. We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context, on five open-weight models over contamination-free multi-step tasks (depths 1-8). By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes (L6 = 0.686, 0.684). Our central finding concerns the measurement itself. Under exact-match scoring against a fixed gold trajectory, a propagation model's severity and recovery parameters are not merely hard to estimate - they are fixed by the scoring rule. Severity is forced to its boundary (0 of 869 poisoned steps correct); recovery is structurally unobservable (0 of 580 poisoned steps returned on-track, against an expected 0.0058 by chance). Both follow from one mechanism: post-divergence, the gold value is generated by tool constants the model never sees, so it is information the model cannot derive. A fit run anyway returns 0.92 and 0.73 for a quantity that is exactly 1.000 - confident numbers for a parameter the scoring rule already determined. We give the mechanism and a remedy, conditional-on-state scoring, applied retrospectively to cached completions at zero additional cost, which un-pins severity to interior estimates excluding zero (+0.149, +0.316).
1 Introduction
The paper separates tool-selection and argument errors at the individual invocation level, comparing clean teacher-forced and free-running contexts to measure propagation. This empirical comparison instead reveals that exact-match scoring itself determines the propagation parameters.
- Tool-using agents can choose the wrong tool or provide wrong arguments, and early errors can corrupt dependent calls.
- Correct invocation requires both selecting the right tool and supplying correct arguments.
- The protocol compares teacher-forced contexts with model-generated free-running contexts to separate context-length decay from error propagation.
- The study’s central result is that the scoring rule, rather than the data, determines severity and recovery parameters.
A. Two-arm scoring protocol B. Propagation model
Figure 1 combines a two-arm scoring protocol with a three-state propagation model. The paper reports that exact-match scoring forces the model’s severity and recovery parameters regardless of their true values.
- A. Two-arm scoring protocol B. Propagation model: Figure 1 presents a two-arm scoring protocol alongside a three-state propagation model.
- A. Two-arm scoring protocol B. Propagation model: The protocol separates clean and free-running evaluation while the propagation model represents how errors move through a chain.
- A. Two-arm scoring protocol B. Propagation model: Under exact-match scoring, severity and recovery are determined by the scoring rule rather than the data.
2 Related Work
Prior work established tool-use loops, API-call evaluation, disaggregated error measures, and multi-step dependency tasks. This paper addresses the remaining gap: measuring what happens to a chain after an error enters it.
- Toolformer and ReAct established tool-calling and reason–act–observe foundations, while Gorilla measured API-call correctness at scale.
- Most benchmarks score task success or fixed-reference execution outcomes rather than per-call correctness.
- RoTBench, MTU-Bench, and FuncBenchGen disaggregate or stress tool-use behavior, but FuncBenchGen reports task success instead of invocation-level measures.
- Existing error taxonomies target incorrect-call rates, leaving chain behavior after an entered error as the gap addressed here.
3 Method
The method measures invocation-level reliability in paired teacher-forced and free-running settings on controlled dependency tasks, then models context states and propagation. It uses exact ground truth, conditional selection scoring, and cached model completions.
- A task is a sequence of calls whose arguments may depend on earlier outputs, with dependency depth defined by the longest dependency chain.
- Local correctness p_t measures step accuracy with correct upstream history, while global correctness g_t measures accuracy in free-running trajectories.
- Net propagation loss is defined empirically as L_t := 1 − g_t/p_t, independently of any parametric severity model.
- The three-state Markov model tracks clean, syntax-poisoned, and semantics-poisoned contexts using measured accuracy and origin-specific recovery rates.
- Under the model, g_t = p_t (1 − π x_t) and L_t = π x_t, but exact-match scoring forces π = 1, making the parametric identity vacuous for estimation.
- Tasks use deterministic executable functions with exact per-call ground truth and tool outputs generated by hidden affine modular transformations.
- 3.3 Task design: Routing tasks condition selection on the value actually held, while divergence from gold is recorded separately; linear tasks serve as an unexecuted null design.
- Selection is scored conditionally and arguments by exact match, with syntactic and semantic errors classified separately and equal retry budgets across arms.
4 Data Collected
The dataset was frozen for analysis, with per-model sample sizes and measured rate limits recorded at the freeze. One model was excluded because its daily token allowance was exhausted.
- The analysis uses a dataset frozen at 14:09 on 2026-08-17, with nothing extrapolated.
- Table 1 records per-model sample sizes at the freeze.
- gpt-oss-20b completed zero tasks and is excluded from all claims.Pilot runs consumed its daily allowance.
- Table 2 reports measured per-model rate limits on the Groq free tier.
5 Results
The study separates context-length decay from propagation by comparing teacher-forced and free-running invocation correctness, then shows that exact-match scoring fixes propagation-model parameters rather than identifying them. Conditional-on-state scoring provides a retrospective remedy, while discrimination reveals rule-following patterns hidden by aggregate metrics.
- 5.1 Propagation is large, monotone in depth, and separated from context decay: 0.700 → 0.543 teacher-forced correctness on llama-3.1-8b versus 0.700 → 0.174 free-running correctness shows propagation dominates context-length decay.The relative declines are 22% and 76%, respectively; Lt rises monotonically with non-overlapping adjacent-depth intervals.
- 5.1 Propagation is large, monotone in depth, and separated from context decay: L1 = 0.000 exactly on every model, while depth 6 yields the first large-model propagation loss, L6 = +0.190 [+0.062, +0.345].At depth 1 the two arms share an identical prompt and cached response; at depth 6, llama-3.3-70b-versatile has downstream steps available for errors to poison.
- 5.1 Propagation is large, monotone in depth, and separated from context decay: +0.32 to +1.00 model-free lag values confirm propagation for every model with errors, without a parametric assumption.Because P(t+1 ok | t wrong) is exactly zero, the lag reduces to P(ok | ok) here.
- 5.3 The scoring regime, not the models, fixes both parametric quantities: 0 of 869 poisoned-context steps are correct, forcing severity π = 1.000, while recovery is unobservable because 0 of 580 poisoned steps return on-track.The fixed-gold mechanism makes post-divergence values unavailable to the model; qwen3.6-27b has undefined severity because it never left a clean context.
- 5.4 A remedy: conditional-on-state scoring: 0 API calls and 881 cache hits enabled retrospective conditional-on-state scoring, which attributes 0.15–0.32 to genuine degradation rather than trajectory unreachability.Clean-context steps score identically under both rules at 0.658, while both poisoning intervals exclude zero.
- 5.5 Rule-following is measurable directly, and aggregates hide it: +1.000 discrimination for qwen3.6-27b, +0.146 for llama-3.1-8b, and −0.176 for allam-2-7b expose qualitatively different rule-following patterns hidden by aggregate first-listed rates.For allam-2-7b, anti-correlation accompanies worse accuracy on even references, indicating failure to perform the routing task rather than merely higher error frequency.
6 Pipeline Validation, and What Simulation Cannot Establish
Pipeline validation recovered configured parameters and exposed important blind spots: simulation can certify exercised channels while missing failures in prompt construction, observation passing, parsing, and per-model assumptions.
- Seven harness artifacts were eliminated by validating the full pipeline against six simulated policies with known ground truth.The validation covered issues including retry-budget asymmetry, poisoning-related selection attribution, and syntactic-share estimation.
- Simulation certifies only the channels it exercises, so stashed-context mocks can let an agent loop pass without threading observations into conversation.
- A parser or scorer can reverse a result’s direction: 319/394 (81.0%) qwen3.6-27b responses would have failed to parse under the pre-fix extractor.
- Per-model verification is necessary because shared constants and assumptions hid qwen’s 6.6× higher cost, allam’s anti-correlation, and model-specific error clustering.
- The estimation pipeline recovered configured parameters but did not establish prompt construction or observation passing, which no out-of-band simulation can certify.
7 Limitations and Scope
The study is a controlled synthetic-task measurement rather than a production benchmark validation, and several designed conditions were not executed, leaving generality and comparability bounded.
- The controlled synthetic integer-argument tasks isolate propagation with exact ground truth but leave ecological validity under natural-language or multi-field arguments open.The conditional-scoring decomposition especially depends on copying a value shown one turn earlier.
- Four designed conditions were not executed: linear-task null, transformed-argument, presentation-order, and calling-mode controls.These omissions leave task-structure specificity, error-composition generality, presentation-order confounding, and calling-mode effects unresolved within the study.
- Opaque-feedback hypotheses were not addressed, routing distractor levels do not vary, and unequal model-specific interval widths preclude reading scale or family contrasts from point estimates alone.
8 Conclusion
Invocation-level measurements show substantial error propagation, but exact-match scoring itself fixes the propagation model’s severity and recovery parameters. Conditional-on-state scoring remedies this measurement problem by crediting correct continuation from the model’s actual state.
- By depth 6, roughly 70% of a model’s own clean-context capability is lost to its earlier mistakes.The study separates context-length decay from propagation using teacher-forced and free-running invocation-level rates.
- Under exact-match scoring, severity and recovery are determined by the scoring rule rather than estimated from model behavior.After divergence, the fixed gold value is information the model has never received and cannot derive.
- Conditional-on-state scoring credits calls that correctly continue from the value the model actually holds and can be applied retrospectively to cached completions at zero marginal cost.This changes severity from a boundary artifact to interior estimates excluding zero.
- For reference-trajectory benchmarks, unreachable references after divergence make fitted severity or recovery parameters properties of the scorer, not the model.This scope includes execution-match and AST-match benchmarks.