PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents

Bhaskar Gurram

arXiv:2604.16706v2cs.AIcs.CLcs.MA

TL;DR

Automated correctness measures for tool-using agents are rarely validated against human judgments, even though they inform evaluation and deployment. AGENTPROP-BENCH audits judging, error propagation, and runtime mitigation; it finds that answer-level evaluation misses fabricated tool use and that interceptor effects depend on whether models emit tool calls.

  • Problem

    Automated correctness metrics and LLM judges are used to evaluate tool agents but are seldom validated against the human judgments they claim to approximate.

    Source: §1 · p. 1

  • Method

    AGENTPROP-BENCH audits metric validity, parameter-error propagation, and runtime interception using benchmark tasks, execution traces, and stage indicators.

    Source: §1 · p. 2, §3 · p. 2, §4 · p. 3

  • Results

    P2 correctness falls from 24.4% to 16.2% for GPT-4o-mini and from 36.0% to 24.3% for o3-mini; fabricated tool use can make apparent rejection misleading and limits when interception applies.

    Source: §6 · p. 5, §8 · p. 5, §10 · p. 7

  • Takeaways & Limitations

    Rejection, recovery, and whether a model calls tools are distinct evaluation dimensions, since fabricated tool use is invisible to answer-level scoring and the interceptor acts only when tools are emitted.

    Source: §8 · p. 5, §10 · p. 7

    Tools run against deterministic simulators rather than live APIs, and the study injects a single parameter per trace, leaving multiturn and multi-parameter propagation open.

    Source: §11 · p. 7

Abstract

from arXiv · show

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation. We present AgentProp-Bench, a diagnostic benchmark of 14,750 execution traces from thirteen LLM agents (nine proprietary, four open-weight) across four domains, and use it to audit three questions. First, substring-heuristic judging of agent outputs agrees with human annotation only at chance level (Cohen's kappa = 0.049 against each of two annotators), while a three-LLM ensemble reaches moderate agreement (kappa = 0.432) and a single GPT-4o-mini judge is in fact the strongest (kappa = 0.567); dual-annotator agreement is almost perfect (kappa = 0.835). Second, under validated judging a parameter-level error propagates to a wrong final answer with human-calibrated probability approximately 0.62, replicated across proprietary and open-weight models, and a model's ability to reject corrupted inputs and to recover from them are statistically indepen1). Third, severalagents fabricate tool executions -- asserting tool-derived renever obtained (up toilure invisible toend-to-end scores, and a lightweight runtime interceptor reduhallucination on everto 24 percentagepoints) at a tunable operating point that is net-positive on open-weight models. Ini-2.0-Flash, whichrarely emits an injectable tool call. All code, data, traces, and human labels are relehttps://github.com/bhaskargurram-ai/agenthallu-bench

1 Introduction

Automated correctness metrics for tool-using agents are rarely validated against human judgment, despite informing rankings, robustness measurements, and deployment decisions. AGENTPROP-BENCH audits metric validity, error propagation, and runtime mitigation, finding major weaknesses in substring judging and distinct pipeline failure modes.

1 Introduction

  • Substring or keyword metrics and LLM judges are used to evaluate agents, but are seldom validated against the human judgments they aim to approximate.These metrics inform model rankings, robustness measurements, and deployment decisions.Source: §1 · p. 1
  • The benchmark audits three questions: automated-metric agreement with humans, propagation of parameter errors, and runtime interception of downstream errors.It also releases 2,000 tasks, 14,750 traces, 100 human labels from two annotators, and analysis code.Source: §1 · p. 2
  • Substring judging is at chance, the three-LLM ensemble is only moderately aligned with humans, and propagation conclusions are invariant to the validated judge.The contribution reports κ=0.049 for substring judging and moderate agreement for the ensemble.Source: §1 · p. 2
  • A parameter-level error reaches a wrong answer with human-calibrated probability ≈0.62 across thirteen models, while rejection and recovery are statistically independent.The benchmark measures propagation across proprietary and open-weight models.Source: §1 · p. 2
  • Fabricated tool use can affect up to 37.5% of traces and masquerade as robustness; a runtime interceptor reduces hallucination on tool-calling models at a tunable, net-positive operating point.Fabricated executions assert that a tool was called and a result obtained when no call occurred, a failure end-to-end correctness scores cannot detect.Source: §1 · pp. 1–2

2 Related work

Prior work questions the validity of automatic metrics and LLM judges, while tool-agent benchmarks typically report end-to-end completion without localizing failures. This work extends evaluation audits to tool-using agents and combines controlled error injection with runtime mitigation.

2 Related work

  • Evaluation-methodology studies find that metric choice can flip rankings, single-prompt evaluation can yield unstable conclusions, and correctness metrics and LLM judges require validity audits.The passage cites Opitz (2024), Mizrahi et al. (2024), Adlakha et al. (2024), and Chhun et al. (2024).Source: §2 · p. 2
  • Tool-agent benchmarks report end-to-end task completion but do not localize failure origins or propagation; this work uses controlled injection, human validation, and a concurrent-control mitigation evaluation.The related work contrasts this approach with post hoc attribution of naturally occurring hallucinations.Source: §2 · p. 2
  • Fabricated tool use is framed as a provenance failure, while the interceptor differs from sampling-based mitigations by running in parallel with one execution at negligible cost.The passage also names self-consistency, self-refinement, SelfCheckGPT, and semantic entropy as other mitigation approaches.Source: §2 · p. 2

3 The AGENTPROP-BENCH benchmark

AGENTPROP-BENCH uses simulated API tasks and controlled parameter errors to make downstream effects observable. The benchmark spans multiple domains, model families, and task-chain difficulties, with a human-labeled sample used to validate judgments.

3 The AGENTPROP-BENCH benchmark

  • Tasks use deterministic Python API simulators with JSON schemas, and the benchmark includes calendar, weather, medical, and knowledge domains.Each domain has 3–5 tools; the held-out retail tasks are separate from the four core domains.Source: §3 · p. 3
  • AGENTPROP-BENCH contains 2,000 core tasks across four domains plus 300 held-out retail tasks, with difficulty based on tool-chain length and reasoning depth.Tasks are graded easy, medium, or hard according to single-call, two-tool, or three-tool/branching requirements.Source: §3 · p. 2, Table 1 · p. 3
  • The P2 protocol replaces a semantically critical parameter before the first tool call with an incorrect but schema-valid value, making downstream effects observable.For example, a London weather query can receive Manchester as the valid but incorrect city parameter.Source: §3 · p. 3
  • The benchmark releases 14,750 traces from thirteen models: nine proprietary and four open-weight, collected under a shared prompted-ReAct harness.The semantic-wrong condition includes 2,150 traces; the full trace total includes all four injection types and the interceptor experiment.Source: §3 · p. 3
  • Two annotators independently labeled 100 stratified P2 traces blind to automatic verdicts, with κ=0.835 agreement supporting their labels as a reliable reference.Human-calibrated rates use P(human correct | ensemble correct)=0.76 and P(human correct | ensemble wrong)=0.25.Source: §3 · p. 3

4 Methods

The methods compare automated judges against human references, define independent stage indicators for tracing errors, and decompose robustness into rejection and recovery. A concurrent control supports evaluation of a runtime interceptor.

4 Methods

  • Three correctness metrics are compared against human labels using Cohen’s κ: a substring heuristic, a majority-vote three-LLM ensemble, and human annotation.The substring heuristic checks reference text and content words; the ensemble votes across GPT-4o, Gemini-2.5-Flash, and GPT-4o-mini.Source: §4 · p. 3
  • S1 marks injection application, S2 marks an executed call with observable effect, and S3 marks a wrong final answer; the method reports stage-reach, conditional-hop, and injection-to-error probabilities.The indicators derive from separate execution-scored and answer-level data columns.Source: §4 · p. 3
  • Robustness is decomposed into rejection, 1−pS1, and recovery, 1−r2,3, with their independence tested across thirteen models using Spearman correlation.Rejection measures injections refused at execution; recovery measures executed injections still answered correctly.Source: §4 · pp. 3–4
  • The interceptor runs three parallel layers for tool/schema errors, uncertainty keywords, and unsupported numerical claims, against a concurrent no-interceptor control.Both arms run the same tasks per model and differ only in whether the interceptor is active.Source: §4 · p. 4
  • Results use 95% bootstrap confidence intervals from 1,000 iterations, Cohen’s h for two-proportion effects, and identical-harness open-weight runs on one L40S GPU.The bootstrap seed is 42.Source: §4 · p. 4

5 Metric validity (RQ1)

The substring heuristic agrees with human annotation only at chance, while LLM judges do better; GPT-4o-mini is the strongest single judge, but the ensemble remains the primary judge.

5 Metric validity (RQ1)

  • κ=0.049 and 0.015 against the two annotators (0.036 on consensus) place the substring heuristic at chance; GPT-4o-mini leads single judges at κ=0.567, versus κ=0.432 for the ensemble.Table 2 reports the metric comparisons; the ensemble is retained as primary to avoid selecting a judge on the same labels used for calibration.Source: Table 2 · p. 4, §5 · p. 4
  • The ensemble marks 25 traces correct versus 38 by humans, chiefly rejecting answers humans accept, and therefore underestimates true correctness by ≈13 pp.The paper corrects for this conservative bias using human-calibrated rates.Source: Table 3 · p. 4, §5 · p. 4

6 Error propagation (RQ2)

Under semantic-wrong injection, parameter-level errors often reach final answers; this substantial propagation holds across judges. Rejection and recovery are statistically independent across models.

6 Error propagation (RQ2)

  • ≈0.62 is the human-calibrated mean per-model probability that an executed injection reaches a wrong answer (range 0.46–0.73); the pooled calibrated rate is ≈0.65.The trace-pooled hop rate is 0.80 (217/271); injection-to-error rates range 0.57–0.90 under the ensemble and 0.53–0.70 when calibrated.Source: §6 · pp. 4–5
  • Judge choice does not change the propagation conclusion: the substantial rate is roughly 0.6–0.8, with GPT-4o-mini alone yielding a pooled injection-to-error rate of 0.61.The ensemble gives 0.71 for the pooled injection-to-error rate; GPT-4o-mini’s uncalibrated estimate essentially coincides with the human-calibrated ensemble figure.Source: §6 · p. 5
  • P2 correctness is lower than baseline for most models, including GPT-4o-mini (24.4%→16.2%) and o3-mini (36.0%→24.3%).The comparison uses the same ensemble judge for baseline and P2 in the reported results.Source: §6 · p. 5

7 Robustness is two-dimensional

  • Rejection and recovery are statistically independent across thirteen models (Spearman ρ=0.041, p=0.893), so one robustness score conflates distinct controls.GPT-4.1-nano pairs 37.5% rejection with 2.9% recovery.Source: §7 · p. 5

8 Fabricated tool use (RQ3a)

Fabricated tool use can make absent tool calls look like robustness, while controlled runtime interception reduces hallucination on tool-calling models. Its operating point can be tuned by model.

8 Fabricated tool use (RQ3a)

  • Fabricated tool use means asserting a tool-derived result without an actual tool call; the classifier validated on 40 flagged traces has 92.5% precision (37/40).Manual error analysis distinguishes fabrication from propagation cascades and rejection.Source: §8 · p. 5
  • 37.5% of Gemini-2.0-Flash traces involve fabricated tool use, so its apparent 95% rejection is dominated by absent or fabricated calls rather than parameter filtering.GPT-4o and GPT-4.1-nano call tools in 99–100% of traces, while GPT-4o-mini calls tools 40% of the time and fabricates in 12%.Source: §8 · p. 5
  • Four open-weight models reproduce propagation, with injection-to-error rates of 0.74, 0.63, 0.65, and 0.67; two other tested open families did not reliably emit tool calls and were not scored.The reported rates are for Qwen2.5-3B/7B/14B and Hermes-3-Llama-8B; Mistral-7B-v0.2 and Phi-3.5-mini are protocol-adherence observations.Source: §8 · p. 5

9 Runtime interception (RQ3b)

  • The interceptor reduces hallucination on every tool-calling model, by 23.0 pp for GPT-4o-mini and 24.0 pp for Qwen2.5-7B; Gemini-2.0-Flash is the null case at −1.3 pp.Its rare injectable tool calls account for the null result; the monitor acts at the execution-interception point.Source: §9 · pp. 5–6
  • On Qwen2.5-7B, schema-layer abstention reduces hallucination by 13.3 pp at 0.62 abstention precision, catching 39 hallucinations while withholding 24 correct answers.The schema-only threshold maximizes precision; operators can select high-precision or high-coverage operation by model.Source: §9 · p. 6

10 Discussion

Automated evaluation should be validated against human labels: the dominant heuristic can over-count correctness enough to reverse a baseline-versus-perturbed comparison. Robustness involves separate dimensions, and fabricated tool use can be invisible to answer-level scoring.

10 Discussion

  • 34 pp over-counting of correctness by the substring heuristic on injected traces can reverse the direction of a baseline-versus-perturbed comparison.The discussion calls for validation of automated evaluation rather than assuming its reliability.Source: §10 · p. 6
  • Rejection and recovery show no significant monotonic relationship across thirteen models (Spearman ρ=0.041, p=0.893), supporting their treatment as separate robustness dimensions.A third axis is whether a model reliably calls tools at all, since fabricated tool use is invisible to answer-level scoring.Source: Table 6 · p. 7, §10 · p. 7
  • A lightweight, single-pass interceptor can convert a meaningful fraction of silent failures into explicit abstentions across proprietary and open-weight models, at a deployer-controlled operating point.The discussion identifies this as a practical runtime mitigation.Source: §10 · p. 7

11 Limitations

The reported judge calibration is limited by a small, narrowly drawn annotator pool, and several propagation and mitigation estimates have restricted scope. Rejection and recovery are statistically independent, but the figure does not establish broader per-model precision.

11 Limitations

  • One in four ensemble “wrong” verdicts is a human-correct false negative, and calibration on 100 labels from two annotators of similar background mitigates but does not eliminate bias.A larger, more diverse annotator pool is future work.Source: §11 · p. 7
  • Rejection and recovery are statistically independent across the thirteen models (Spearman ρ=0.041).Figure 3 plots rejection against recovery, with point size proportional to nS2.Source: Figure 3 · p. 7
  • Per-model stage-two subsamples range from 2 to 56, so the smallest per-model rates are directional rather than precise estimates.Source: §11 · p. 7
  • The interceptor’s between-arm deltas are robust to shared judging bias, but absolute per-arm rates have not been validated by human labeling.Source: §11 · p. 7
  • Tools execute against deterministic simulators, and each trace injects a single parameter; multi-parameter and multiturn propagation remain open.Source: §11 · p. 7

12 Ethics

The conclusion reports that AGENTPROP-BENCH identifies evaluation and agent failure modes and that a runtime interceptor reduces hallucination at a tunable, net-positive operating point. The paper also reports its human-labeling and data-release practices.

12 Ethics

  • The study uses two annotators to label model outputs, reports no personal or sensitive data, and releases artifacts containing no private information.Source: §12 · pp. 7–8
  • A low tool-call rate can masquerade as high rejection on P2 semantic-wrong traces.Source: Table 7 · p. 8
  • The interceptor reduces hallucination on all four tool-calling models; its null result for Gemini-2.0-Flash follows from near-absent tool calls.Table 8 compares hallucination rates with and without interception, alongside abstentions, saved hallucinations, and broken correct answers.Source: Table 8 · p. 8

13 Conclusion

  • Substring evaluation is no better than chance, parameter errors propagate with calibrated probability ≈0.62, and a runtime interceptor reduces hallucination on tool-calling models at a tunable, net-positive operating point.The conclusion also reports independent rejection and recovery, and fabricated tool use by several models.Source: §13 · p. 8

A Annotation protocol

Two annotators labeled stratified traces independently and blind to automatic verdicts, using a criterion that accepts minor wording differences when the core fact is correct. The ensemble uses a shared answer-matching prompt and majority vote, with calibration based on human labels.

A Annotation protocol

  • Two annotators independently and blindly labeled 100 stratified P2 traces from nine proprietary models and three domains against the reference answer.They marked answers correct when the core fact matched and was correct, accepting minor wording differences and extra explanation.Source: A Annotation protocol · p. 11

B Judge prompt and calibration

  • The three-LLM ensemble uses a shared YES/NO answer-matching prompt and a 2/3 majority vote; human calibration estimates P(human correct | ensemble correct)=0.76 and P(human correct | ensemble wrong)=0.25 from 100 labels.Source: B Judge prompt and calibration · p. 11

Related papers

Loading 2604.16706v2…