PaperlayerView exact revision on arXiv

Source-linked AI summary

SSCBench: Evaluating the Evidential Validity of Fault-Injection Tests for Tool-Using LLM Agents

Xincheng He, Wanli Dong, Zhaoqiang Guo, Yan Liu, Lei Xu

arXiv:2610.11514v1cs.SE

TL;DR

Fault-adoption outcomes do not alone establish whether agents relied on injected errors despite timely counterevidence, particularly when agents control which observations become visible. The paper develops a measurement protocol and SSCBench to track valid counterevidence, pre-use opportunity, and execution-level arrival. Its experiments find that realized evidential conditions vary and that claim-specific populations can be sparse even when aggregate adoption is well defined.

  • Problem

    Fault-adoption results leave limited evidence about whether authoritative counterevidence was visible before use, a distinction needed to interpret reliance despite timely refutation.

    Source: §I · p. 1

  • Method

    The paper specifies fault-specific counterevidence, assesses pre-use revealability, and reconstructs counterevidence arrival and dependent use in SSCBench.

    Source: §I · p. 2, C. Execution-Level Event Reconstruction · p. 6

  • Results

    Realized counterevidence exposure varied across executions, claim-specific populations could be sparse despite defined aggregate adoption, and automated adoption identification did not ensure exact first-reliance localization.

    Source: E. RQ4: Measurement Recoverability · pp. 9–10, §X · p. 12

  • Takeaways & Limitations

    Reports should bind each fault-adoption claim to its required evidential condition and evaluable denominator, treating an empty timely-counterevidence stratum as unobserved.

    Source: §VII · p. 10

    Empirical rates describe two τ-bench domains, four fault operators, and five agent configurations; other tool contracts or interaction patterns may differ.

    Source: §VIII · p. 11

Abstract

from arXiv · show

Fault injection is increasingly used to evaluate the reliability of tool-using LLM agents. However, there has been limited study of how fault-adoption results should be interpreted when the agent itself determines which authoritative observations become visible during execution. In this paper, we present a systematic study of this evidential validity problem in agent fault-injection evaluation. We develop a measurement protocol that specifies what observations can refute an injected assertion, determines whether they can become visible before the affected fact is first used, and records whether the evaluated execution actually realizes this condition. We construct SSCBench as an instantiation of the protocol and evaluate four fault operators and five agent configurations over 1,191 faulted executions in two $\tau$-bench environments. Our experiments show that an admitted fault case and agent configuration can realize substantially different evidential conditions across executions, and that aggregate adoption can remain well defined even when the population supporting a timely-counterevidence claim is sparse or absent. For example, among 44 adopted runs in which counterevidence eventually became visible, only 17 received it before first use, while 27 received it afterward. We also find that first-error timing and later stance revision need not coincide, and that automated trajectory analysis can recover adoption without reliably recovering the first faulty-reliance event needed for temporal diagnosis. We argue that the evidential condition realized by an execution and the population supporting a claim-specific interpretation are part of fault-injection evaluation itself and should be reported before adoption is interpreted as failure under pre-use counterevidence.

I. INTRODUCTION

The introduction argues that fault adoption alone does not establish reliance despite timely counterevidence, because agents control which observations become visible and when. It presents a measurement protocol and SSCBench to distinguish refutation opportunity from the evidence realized during execution.

I. INTRODUCTION

  • Interpreting adoption as reliance despite contradicting evidence requires valid counterevidence to become visible before the affected fact is first used.The evaluator must identify what refutes the assertion and determine whether the agent's chosen interactions expose it before use.Source: §I · p. 1
  • A related read is not necessarily a refutation: an ItemOption product record counts only after the execution links it to the affected order item.Otherwise, the evaluation can assign an evidential condition that the execution has not established.Source: §I · p. 1
  • Aggregate adoption supports a fault-propagation claim, but timely-counterevidence claims require a distinct population in which counterevidence arrived before use.Eventual counterevidence includes both pre-use and post-use arrivals.Source: §I · p. 1
  • SSCBench operationalizes the condition by specifying fault-specific counterevidence, case-level pre-use revealability, and execution-level reconstruction across four fault operators and three acquisition structures.The study uses retail and airline τ-bench environments and covers five agent configurations and 1,191 faulted executions.Source: §I · p. 2
  • Across 1,191 faulted executions and five agent configurations, measurement distinctions affected evaluation, including claim-specific populations and recovery of semantic and temporal events.Admitted cases did not fix the evidential condition realized by an execution, and aggregate outcomes could differ from the population supporting a claim-specific interpretation.Source: §I · p. 2

II. FAULT-INJECTION EVALUATION OF TOOL-USING AGENTS

Fault-injection outcomes depend not only on the injected fault and downstream behavior, but also on what evidence the execution makes visible. The section introduces the event and evidence concepts used to distinguish those conditions.

A. Evaluation Setting

  • Tool-using agent trajectories determine whether authoritative state becomes visible after a fault before use, only afterward, or not at all.Thus, identical faults and downstream outcomes can occur under different information conditions.Source: A. Evaluation Setting · p. 2

B. Motivating Example

  • In the WriteEffect example, two executions make the same faulty claim, but the unchanged reservation is read only after the claim in one and before it in the other.TE marks arrival of the contradicting observation; TS marks first use of the affected information.Source: B. Motivating Example · p. 2
  • Fault propagation establishes that an injected error influenced an action or response, while reliance despite timely counterevidence additionally requires the refutation to be visible before use.Relevant information alone may not refute the fault until the execution establishes its relation to the affected fact.Source: B. Motivating Example · p. 2

A. Measurement Objects

  • A dependent use is a message, call, or handoff that depends on the affected fact; adoption occurs when such a use treats the faulty assertion as operative.Using a disputed value only to verify it is not faulty reliance.Source: A. Measurement Objects · p. 3
  • Valid counterevidence must match the fault's authority, entity, scope, and independence requirements and suffice to contradict it with task and tool semantics.An observation or set of observations is evaluated against the assertion it is meant to refute.Source: A. Measurement Objects · p. 3
  • TE records when valid counterevidence fully enters model-visible history, while TS anchors first dependent use and TU locates first faulty reliance.TS is defined independently of adoption; when both anchors are decidable, TF < TS ≤ TU.Source: A. Measurement Objects · p. 3

B. Pre-Use Revealability

Pre-use revealability determines whether a valid fault context permits counterevidence to become visible before first use through a legal continuation. It is a property of the case and interface, not of whether a particular execution takes that opportunity.

B. Pre-Use Revealability

  • A valid injected fault must convey a task-relevant faulty assertion while preserving the intervention's required non-target semantics.Source: B. Pre-Use Revealability · p. 3
  • Pre-use revealability asks whether a legal continuation from the fixed injection context can expose counterevidence before first use without evaluator-only information.The context fixes the target, injection point, visible history, fault parameters, intervention persistence, interface, and permissions.Source: B. Pre-Use Revealability · pp. 3–4
  • Admission requires validity and pre-use revealability, but whether the opportunity is realized belongs to the individual execution.Source: B. Pre-Use Revealability · p. 4

C. Realized Evidential Condition

The realized evidential condition classifies an execution by when valid counterevidence arrives relative to first dependent use. This timing supports claims distinct from adoption alone and from the timing of first faulty reliance.

C. Realized Evidential Condition

  • For an admitted, triggered run with decidable first use, counterevidence arriving before TS is pre-arrival, after TS but before it is absent only if it never arrives, and later arrival is late.The categories are pre (TE < TS), late (TS < TE < ∞), and absent (TE = ∞).Source: C. Realized Evidential Condition · p. 4
  • Revealability establishes a pre-use opportunity, whereas the realized condition records whether a particular execution took it.The same admitted fault can therefore produce different realized conditions.Source: C. Realized Evidential Condition · p. 4
  • TE < TS supports an adoption-with-pre-use-counterevidence claim, while TE < TU concerns whether counterevidence preceded first faulty reliance in adopted runs.A run can be late relative to first dependent use yet receive counterevidence before first faulty reliance.Source: C. Realized Evidential Condition · p. 4

IV. SSCBENCH: OPERATIONALIZING ARRIVAL-AWARE FAULT INJECTION

SSCBench makes counterevidence arrival an explicit part of fault-injection cases and measured executions. Its fault contracts define what refutes each injected assertion and how that evidence can be acquired through the task interface.

IV. SSCBENCH: OPERATIONALIZING ARRIVAL-AWARE FAULT INJECTION

  • SSCBench links each intervention to its affected fact, valid counterevidence, and execution events that determine when the evidence becomes model-visible.Its three mechanisms are fault contracts, pre-use revealability admission, and execution reconstruction.Source: §IV · p. 4

A. Fault-Contract Instantiation

  • A fault contract fixes the target entity and state scope, injected assertion, authoritative counterevidence basis, sufficient contradiction relation, and interface acquisition route.The benchmark instantiates operator templates from task obligations and tool contracts over concrete records.Source: A. Fault-Contract Instantiation · p. 4
  • The read-value operators alter returned observations while leaving authoritative records unchanged, whereas WriteEffect suppresses a matched state update but preserves the tool’s success acknowledgement.ReturnDate changes a reservation return date, PaySource flips a payment category, and ItemOption substitutes an ordered-item option.Source: A. Fault-Contract Instantiation · pp. 4–5
  • ReturnDate and PaySource provide required observations alongside the fault, but their sufficiency depends on visible semantics connecting retained fields to the affected fact.For PaySource, the retained payment identifier must be related to the payment category; an empirical naming pattern alone is insufficient.Source: A. Fault-Contract Instantiation · pp. 4–5
  • ItemOption requires binding an authoritative product record to the affected order item, while WriteEffect requires a read-back scoped to the same suppressed request.WriteEffect preserves attribution using reservation, flight changes, cabin, and payment method; intervening writes can break that scope.Source: A. Fault-Contract Instantiation · p. 5

B. Pre-Use Revealability Admission

SSCBench admits a case only when a valid fault contract has a legal, pre-use revealability witness in the applicable faulted context. Admission establishes possibility, not whether the evaluated execution will realize timely counterevidence.

B. Pre-Use Revealability Admission

  • Admission requires a valid fault contract and a legal continuation that exposes sufficient counterevidence before the first dependent use.The witness must use the normal task interface and original permissions in the same faulted environment.Source: B. Pre-Use Revealability Admission · p. 5
  • The injection context fixes the visible prefix through fault delivery, along with the target, fault parameters, intervention persistence, interface, and permissions.A witness continuation uses tool arguments obtainable from that execution.Source: B. Pre-Use Revealability Admission · p. 5
  • A qualifying witness must establish the full contract relation, including evidence authority, entity and scope match, independence, and agent-visible sufficiency, with no earlier dependent use.The trace retains the intervention, evidence provenance, proposed first use, prefix identifier and hash, and witness continuation.Source: B. Pre-Use Revealability Admission · p. 5
  • Witnesses transfer across prefixes only when their identifiers, permissions, and refutation relation remain valid; without a qualifying witness, a candidate is unresolved, not proven unrevealable.Only cases with a valid contract and applicable pre-use witness contribute executions to the primary analysis.Source: B. Pre-Use Revealability Admission · p. 5

C. Execution-Level Event Reconstruction

SSCBench reconstructs fault, counterevidence, and use events from the model-visible trajectory, then evaluates arrival, adoption, diagnosis, and oracle recoverability. The study uses τ-bench retail and airline cases with adjudicated trajectory reviews and separate rules for decidability and missing runs.

C. Execution-Level Event Reconstruction

  • Counterevidence arrival is reconstructed from delivered observations, not inferred from the operator or its admission witness.Hidden environment state can verify an intervention but cannot establish that counterevidence reached the agent.Source: C. Execution-Level Event Reconstruction · p. 6
  • TE is the earliest delivery step when the contract’s required observations and agent-visible semantics jointly suffice to contradict the faulty assertion.A product record may become counterevidence only after its item binding, and a reservation read must remain scoped to the suppressed attempt.Source: C. Execution-Level Event Reconstruction · p. 6
  • SSCBENCH distinguishes dependent use from faulty reliance: TS is the first determinate dependent use, while TU is the first determinate faulty reliance in an adopting run.A faulty-value-derived search is dependent use, but is faulty reliance only when context treats the value as an operative task fact.Source: C. Execution-Level Event Reconstruction · p. 6

A. Research Questions

  • The study asks how often evidence arrives before first use, how adoption varies by arrival, when evidence arrives relative to first faulty reliance, and how well automated oracles recover adoption and localization.Runs with unresolved potential use remain undecidable rather than being treated as no-use or non-adoption.Source: A. Research Questions • · p. 6, C. Execution-Level Event Reconstruction · p. 6

B. Experimental Setup

  • The benchmark samples 30 admitted seeds per operator across τ-bench retail and airline tasks, then evaluates five model configurations with clean-qualified faulted runs.Scheduled faulted runs not observed are treated as missing, not imputed; the execution matrix contains 1,191 faulted runs.Source: B. Experimental Setup · pp. 6–7
  • Human review adjudicates event labels, while oracle evaluation measures adoption precision, recall, and F1 and exact or ±1-step localization accuracy with abstentions counted as misses.Arrival-stratified comparisons are descriptive rather than causal; validation reports automatic measurements against adjudicated references.Source: B. Experimental Setup · p. 7

VI. RESULTS

The study reports that all admitted cases passed final semantic checks and had pre-use witnesses, then validates human and automatic trajectory measurements against adjudicated reviews. Automatic adoption labels matched adjudication more often than arrival labels in the reported validation counts.

VI. RESULTS

  • 1,189 of 1,191 faulted executions triggered the intervention, and all 120 admitted cases passed final semantic checks with a recorded pre-use witness.Table III reports the primary-r0 execution funnel and validation against 120 adjudicated faulted trajectories.Source: A. Admission and Measurement Validation · p. 7
  • Automatic measurements matched adjudication for 111/120 adoption labels and 66/70 applicable ZS labels.ZS is counterevidence-arrival condition; the reported comparison uses frozen automatic measurements.Source: A. Admission and Measurement Validation · p. 7
  • 75 additional primary-r0 ItemOption trajectories introduced no further disagreement with the frozen automatic measurements after review of all 96 trajectories.Twenty-one of the 96 had already been included in the validation sample.Source: A. Admission and Measurement Validation · p. 8

B. RQ1: Counterevidence Arrival

Across the studied operators, arrival conditions varied across executions, and aggregate adoption did not by itself establish adoption after timely counterevidence. First-error timing and terminal stance also captured different outcomes.

B. RQ1: Counterevidence Arrival

  • Unlike co-delivered ReturnDate (149/149) and PaySource (150/150), acquisition-dependent operators showed varied arrival: ItemOption had 40 pre-, 49 late-, and two absent-arrival runs.For WriteEffect, L was pre-arrival in 29/29 runs, V absent in 30/30, and T late in 20/25; a read-cue check reduced pre-use arrival from 54/115 to 0/115.Source: B. RQ1: Counterevidence Arrival · p. 8, Table IV · p. 9

C. RQ2: Outcomes Across Arrival Conditions

  • Among 44 adopted WriteEffect runs with eventual counterevidence, only 17 received it before first use, 27 afterward, and 57 received none in the observed execution.Aggregate adoption can be well defined even when the population supporting a timely-counterevidence claim is sparse or absent.Source: C. RQ2: Outcomes Across Arrival… · p. 8, Figure 3 · p. 8

D. RQ3: Arrival Relative to Faulty Reliance

  • For WriteEffect, 14/17 runs with counterevidence before first error still ended by endorsing the fault, while 26/27 late-arrival runs no longer did.No primary non-control run had counterevidence between first dependent use and first faulty reliance, limiting empirical separation of these timing conditions in the studied operators.Source: D. RQ3: Arrival Relative to Faulty… · p. 9, Figure 4 · p. 9

E. RQ4: Measurement Recoverability

Automated trajectory diagnostics could identify adoption without reliably locating the semantic first-faulty-reliance event, and errors varied across evidential conditions.

E. RQ4: Measurement Recoverability

  • Adding the fault-specific relation to B2 raised adoption F1 from .64 to .82 and exact TU localization from 34% to 49%; B3-SAA reached .82 F1 but localized TU in only 45/88 runs.Improved adoption identification therefore did not remove the verdict–localization gap.Source: E. RQ4: Measurement Recoverability · p. 9, Table V · p. 10
  • B5 marked 29/32 faulted non-adopters positive, while clean-control false positives were 5/32, 8/39, and 0/40 for the three reported systems.Detecting a fault or anomalous state did not by itself recover faulty reliance.Source: E. RQ4: Measurement Recoverability · p. 10
  • B3-SAA adoption recall was 93%, 90%, and 9% in human pre-, late-, and absent-arrival runs, while exact TU localization was 61%, 48%, and 9%.B5 returned a step in 74/88 adopted runs but matched TU exactly in none, despite 51% ±1 accuracy.Source: E. RQ4: Measurement Recoverability · p. 10

VII. DISCUSSION

The discussion recommends measuring counterevidence arrival in each execution and matching every adoption claim to the evidential population it requires. It also distinguishes prevention, revision, and repair, and calls for validating automated measurements against their target constructs.

Measure Evidential Conditions per Execution

  • Reconstruct counterevidence arrival in every evaluated execution; case-level pre-use revealability does not determine the condition realized by a run.An admitted case can have no model-visible counterevidence in the observed execution even though a legal pre-use refutation route exists.Source: §VII · p. 10

Bind Claims to Their Evidential Denominators

  • Report each fault-adoption claim with its required evidential condition and evaluable denominator; an empty or small timely-counterevidence stratum leaves that stronger claim unobserved or imprecise.Aggregate adoption can still support a propagation claim over its own evaluable population.Source: §VII · p. 10

Separate Prevention, Revision, and Repair

  • Separate prevention before first faulty reliance, later revision of the agent’s stance, and repair of any earlier external effect.TS anchors the evidential condition, TU localizes first faulty reliance in adopted runs, and Yend describes later stance rather than repair.Source: §VII · pp. 10–11

Validate Automated Measurement Against the Target Construct

  • Validate automated oracles for semantic classification and temporal localization, not only aggregate verdict accuracy.Coverage, abstention, and error variation across downstream evidential conditions also matter because oracle errors can alter the populations supporting claims.Source: §VII · p. 11

VIII. THREATS TO VALIDITY

The paper bounds its conclusions by the stated constructs and admission rules, the two studied τ-bench domains, and interfaces with specifiable authoritative refutations. It situates SSCBench among research on fault propagation, evidence acquisition and use, and trajectory diagnosis, then concludes that evidential conditions and claim-specific populations should accompany adoption results.

VIII. THREATS TO VALIDITY

  • Conclusions apply to the stated constructs and admission rules and to two τ-bench domains, four fault operators, and five agent configurations; other interfaces may yield different dynamics.The protocol’s claims concern settings where the affected fact and a task-authoritative refutation can be specified from the execution interface; arrival-conditioned comparisons are descriptive, not causal.Source: §VIII · p. 11

IX. RELATED WORK

  • Related work studies fault propagation, evidence acquisition and use, distinguishing observations, and trajectory diagnosis; SSCBench examines whether counterevidence arrives before use and whether diagnostics recover that event.The cited work includes AgentChaos, ToolBench-X [7], Yang et al. [10], classical fault-testability research –, and AgentRx and Who&When [25],.Source: §IX · pp. 11–12

X. CONCLUSION

  • Across 1,191 faulted executions, realized counterevidence exposure varied after cases and agent configurations were fixed, claim-specific populations could be sparse, and diagnosis methods did not necessarily recover required semantic and temporal events.The conclusion recommends measuring conditions per execution, matching claims to evaluable populations, separating prevention, revision, and repair, and validating automated oracles.Source: §X · p. 12

Related papers

Loading 2610.11514v1…