PaperlayerView exact revision on arXiv

Source-linked AI summary

What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation

Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka

arXiv:2610.30491v1cs.AIcs.SE

TL;DR

Agent evaluation reduces execution traces to labels and scores, but those measurements may not fully capture what happened in a trace. The paper audits trace detectors in a calculator setting with corrupted tool outputs and makes their measurement evidence inspectable. The audit finds systematic divergence among correctness, verification, and fault-propagation labels, motivating transparent reporting of evidence, applicability, censoring, and corrections.

  • Problem

    Agent evaluation labels and scores depend on measurement procedures and may not fully capture what execution traces show.

    Source: §1 · p. 1

  • Method

    The authors audit trace detectors in arithmetic tasks with a sometimes-corrupted calculator, tracking tool ground truth, model use, and detector outputs.

    Source: §1 · p. 2

  • Results

    Final correctness, visible verification, and fault propagation diverge systematically, including when a model overrides a corrupted tool value without a visible re-check.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Detector labels should be accompanied by their trace evidence, applicability, censoring state, measurement risks, and human corrections.

    Source: §1 · p. 2, §6 · p. 10

    The arithmetic benchmark offers clear external ground truth, but detector rules may not transfer directly to free-form, categorical, or subjective outputs.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, behavioral detectors, and harness rules, and they may not fully capture what the trace shows. We study this in a tool-use setting built on GSM8K, where agents call a calculator whose outputs are occasionally corrupted, giving clear ground truth for what the tool returned, what the model later used, and what the detectors reported. We audit three trace-level detectors across six production models and 30 problems per model. Final correctness, visible re-verification, and fault propagation can disagree in systematic ways. A key failure mode is silent tool-output override: the calculator returns a corrupted value, but the model later uses a different, often correct, value without calling the tool again, so the bad value stops propagating while a verification detector records no re-check. Human review of 90 traces agrees perfectly with the automated labels for final correctness (67/67), fault impact (19/19), and fault relevance (18/18), but only 67/81 for verification; all 14 disagreements fall into four recurring patterns. Execution limits also change the reported result: in an ablation across two models, one ranges from 21/30 to 27/30 correct as step and token budgets change, while the other stays between 27/30 and 29/30. These findings motivate measurement transparency: an automated label should come with the trace evidence behind it, whether the question the detector asks applied to that trace, whether the run was cut off by an execution limit, and any human corrections. Our trace viewer makes this information inspectable alongside the reported results.

1 Introduction

Agent evaluation turns traces into labels and aggregate scores, but those measurements also depend on the parsers, detectors, and harness rules that produce them. This paper audits where detector labels diverge from trace evidence and proposes reporting that makes the measurement state inspectable.

Study setting

  • Arithmetic word problems with a sometimes-corrupted calculator provide ground truth for the correct tool output, returned value, value later used, and final answer.This separates what the tool should have returned from what the model actually received and used.Source: §1 · p. 2

Silent tool-output override

  • A corrupted calculator value may stop propagating when the model later uses a different, often correct value, even as the verification detector records no visible re-check.For example, the model can use 30 after the calculator returns 29 without another calculator call.Source: §1 · p. 2

Contributions

  • Measurement transparency means reporting detector evidence, applicability, execution censoring, measurement risks, and any human corrections alongside labels.The paper proposes linking aggregate results to trace evidence so readers can inspect why a measurement should or should not be trusted.Source: §1 · p. 2
  • The paper audits detector reliability across six production models and presents a trace viewer that exposes evidence behind detector outputs alongside human corrections.The viewer is intended to make the measurement behind a score inspectable rather than treating the score as self-explanatory.Source: §1 · p. 2

Motivation and audit question

  • Final correctness, visible verification, and fault propagation can diverge systematically, including through silent override, inconclusive propagation, formatting failures, and harness-dependent censoring.The audit asks whether traces contain the evidence needed to interpret automated labels safely.Source: §1 · p. 2

2 Related Work

Prior work examines evaluator reliability, trace-level behavior, faulty-tool responses, harness effects, and visualization. This paper focuses on whether behavioral trace detectors had observable, applicable evidence for the labels they report.

Evaluator validity

  • Evaluator audits report automated-human disagreement and brittle judging rules, while this work asks whether each trace detector had applicable evidence for its label.It distinguishes auditing behavioral trace detectors from evaluating an LLM judge as a whole.Source: §2 · pp. 2–3

Trace- and process-level evaluation

  • Trace- and process-level evaluations assess how agents reach outcomes; this work moves down a level to examine whether measurements faithfully represent each trace.Examples include separating task success from process quality and distinguishing demonstrated exploitation from shortcut success.Source: §2 · p. 3

Tool-output conflict and faulty outputs

  • Studies of faulty-tool responses characterize agent behavior, whereas this paper examines whether reported labels such as “verified” or “recovered” match the trace.Tool-output conflict can improve a final answer while still failing a detector’s narrow definition of verification.Source: §2 · p. 3

Harness effects

  • Step and token limits can censor traces, and per-call fault injection gives agents with different tool-call counts different fault exposure.These harness conditions affect what the reported denominator means.Source: §2 · p. 3

Trace visualization

  • Existing trace-analysis interfaces support inspection, but this viewer focuses on showing detector evidence, applicability, censoring, risk flags, and human corrections together.The paper does not present its viewer as the first trace-analysis interface.Source: §2 · p. 3

3 Experimental Setup

The study uses calculator-assisted GSM8K runs to audit how trace-level detectors turn agent behavior into labels. Its workflow preserves detector evidence and measurement state for inspection and human review.

3.1 Task and fault injection

  • Agents solve arithmetic word problems in a ReAct-style loop, with seeded calculator faults providing reproducible cases for trace-level auditing.The calculator usually returns the correct result but sometimes returns a plausible wrong number; the arithmetic domain allows the correct tool value to be independently recomputed.Source: §3.1 · pp. 3–4

3.2 Models

  • The evaluation covers six production models, with baseline runs set to 50 steps and 1,024 tokens.Tables and figures report separate runs, so counts can differ slightly across them.Source: §3.2 · p. 4

3.3 Detectors under audit

  • The study audits three automatic measurements, summarized in Table 1 by what each measures and its main caveat.The supplied passage does not specify the individual measurements or caveats.Source: §3.3 · p. 4, Table 1 · p. 4

3.5 Measurement states

  • For each detector output, the measurement state retains its automatic label, supporting evidence, applicability, censoring state, risk flags, and any human correction.Human corrections are kept alongside the original output rather than replacing it.Source: §3.5 · p. 4

3.4 System Workflow

  • The workflow turns GSM8K questions and seeded calculator faults into traces, detector labels and risk flags, then exposes them for trace inspection and human comparison.The review layer includes a terminal summary and an interactive viewer.Source: §3.4 · p. 4, Table 2 · p. 5

3.5 Measurement states

  • The study groups observed measurement issues into seven practical failure classes, while describing the taxonomy as empirical rather than exhaustive..Source: §3.5 · p. 5, Table 3 · p. 5

3.6 Measurement-transparent inspection

  • The inspection workflow pairs a compact terminal overview with a viewer that compares runs and exposes evidence behind individual measurements.The corpus view aligns traces by question and model; the trace view shows tool outputs, subsequent behavior, automatic labels, and any human correction.Source: §3.6 · p. 5
  • Figure 1b illustrates silent override: the model replaces a corrupted calculator value without an explicit tool re-check.The trace view makes the evidence behind that detector output inspectable.Source: §3.6 · p. 5, Figure 1 · p. 6

4 Results

The results show that trace-level detector labels can incompletely describe recovery and verification, and that execution limits and human adjudication affect how those labels should be interpreted.

4.1 Silent override and inconclusive propagation

  • A corrupted calculator value may be silently overridden with a different, often correct value without a re-check; if neither value is detectable later, D-impact cannot determine propagation.Source: §4.1 · p. 7

4.2 Harness constraint ablation

  • Harness limits can censor a trace before its final answer, which is not equivalent to a completed but incorrect run.Source: §4.2 · p. 7
  • gpt-oss-120b stayed between 27/30 and 29/30 correct, while nemotron-3-ultra ranged from 21/30 to 27/30 as harness limits varied.Table 5 varies MAX STEPS and MAX TOKENS across two models on the same 30 questions.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 Human adjudication

  • Review covered 90/180 traces across four dimensions; human corrections were retained alongside original labels, and agreement applies only to comparable reviewed cases.Manual review prompted detector revisions and rechecking of previously reviewed traces.Source: §4.3 · p. 8

4.4 Detector reliability against human labels

  • D-verify alone disagreed in comparable reviewed cases; 14 disagreements reflected stuck-loop calls, verbal recognition without recomputation, within-step repetition, or quiet correction.Source: §4.4 · pp. 8–9
  • Across reviewed comparable cases, per-detector sample sizes differed because some measurements were not applicable to every trace.Source: Table 6 · p. 9

5 Discussion

Detector labels can hide recovery patterns and inconclusive cases, so the paper argues that trace evidence and human corrections should remain inspectable. The audit is limited by its arithmetic benchmark, small per-model samples, one primary reviewer, and a two-model constraint ablation.

5 Discussion

  • A model can replace a corrupted calculator value with a different, correct value without explicitly re-checking; in other domains, such silent replacement could alter workflows without making the change clear.The experiment does not test sensor, laboratory, or financial applications directly.Source: §5 · p. 9
  • The visualization layer makes cases where measurements are dependable or not inspectable rather than hiding them behind aggregate scores.The audit should identify which trace patterns explain differences between automated labels and human review.Source: §5 · p. 9

5.1 Limitations and Future Work

  • The arithmetic benchmark provides clear external ground truth, but its detector rules may not transfer directly to free-form, categorical, or subjective outputs.Future work will extend the audit to other datasets and task types, including structured and free-form tool outputs.Source: §5.1 · p. 10
  • Each model is represented by 30 traces, a small sample for precise model-level rates; fault exposure also varies with how often a model calls the calculator.Larger experiments across more questions, models, and traces would provide more stable estimates.Source: §5.1 · p. 10
  • The human-label audit uses one primary reviewer, and reported agreement reflects the final detector implementation after rule revisions and trace re-checks.Manual review revealed coverage gaps in the initial detector logic.Source: §5.1 · p. 10
  • The constraint ablation covers two models and two resource controls, showing that limits can change observed outcomes but not establishing a universal execution-budget rule.Future work will test broader models, configurations, and resource constraints.Source: §5.1 · p. 10

6 Conclusion

The paper treats agent evaluation as measurement: detector outputs can follow their implemented rules yet still incompletely or misleadingly describe a trace. Its controlled audit and inspection workflow make measurement gaps and supporting evidence visible, with the goal of clarifying when automated outputs can be trusted and when traces need closer review.

6 Conclusion

  • Detector outputs can be correct under their implemented rules yet still incompletely or misleadingly describe the underlying trace.The goal is not to replace automated evaluation, but to clarify when its outputs can be trusted and when a trace deserves closer review.Source: §6 · p. 10

AI use statement

  • The paper reports using Claude Code Agent to set up the initial experimental environment and assist with debugging; authors reviewed experimental design, analysis, interpretation, and final claims.LLMs were also the evaluated agents, and Grammarly was used for grammar and sentence-level flow.Source: AI use statement · p. 10

Ethics statement

  • The study uses synthetic arithmetic tasks and no private user or human-subject data.Its broader motivation is that agent-evaluation systems should not hide measurement failures when outputs may inform high-stakes decisions.Source: Ethics statement · p. 10

Reproducibility statement

  • The benchmark task, fault-injection procedure, model configurations, detector definitions, baseline settings, ablation settings, and fault seed are reported across Sections 3 and 4, with additional detector-agreement counts in the appendix.These details specify the task, measurement rules, and model–harness configurations used for the reported results.Source: Reproducibility statement · pp. 10–11

A.1 Per-model verification disagreement

Table 7 summarizes D-verify disagreements among reviewed traces with comparable verification labels; its counts describe reviewed cases, not population disagreement rates.

A.1 Per-model verification disagreement

  • Table 7 reports per-model distributions of D-verify disagreements among reviewed traces with comparable verification labels.The counts should not be interpreted as population disagreement rates.Source: §A.1 · p. 12, Table 7 · p. 12
  • The reported disagreements are restricted to traces with a comparable verification label.Source: §A.1 · p. 12
  • The table concerns reviewed cases rather than population-level disagreement rates.Source: Table 7 · p. 12

A.2 Terminal summary

The terminal summary provides a rapid first-pass view of trace outcomes, fault exposure, verification events, and flagged cases before detailed inspection.

A.2 Terminal summary

  • Figure 2 shows the terminal summary used for rapid first-pass inspection.Source: §A.2 · p. 13, Figure 2 · p. 13
  • The terminal summary presents per-trace outcomes, fault exposure, verification events, and flagged cases.Source: Figure 2 · p. 13
  • The summary is intended for rapid triage before detailed inspection.Source: Figure 2 · p. 13

Related papers

Loading 2610.30491v1…