PaperlayerView exact revision on arXiv

Source-linked AI summary

Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites

Xin Xu, Siru Tao

arXiv:2610.02928v1cs.SEcs.AIcs.LG

TL;DR

Verification suites used to support deployment decisions are commonly evidenced by passing code believed correct and failing code believed broken, but that observation can certify a suite with poor fault resolution. The paper audits an invariant suite using mutation analysis, then uses registered input dimensions to diagnose and repair the faults its frozen suite misses. The frozen suite kills five of ten externally specified mutants, and all five survivors die after the fixture repair on that same challenge set.

  • Problem

    Verification suites used to gate agent deployment are often supported by passing code believed correct and failing code believed broken, although that evidence may not establish whether the suite detects faults.

    Source: §1 · p. 1

  • Method

    The authors apply mutation analysis to a multi-session projection-layer invariant suite, instrument surviving mutants against the reference, and register discriminating input dimensions before adding fixtures with existing oracles.

    Source: §1 · p. 2

  • Results

    The certified suite was survived by a first-order mutant and killed 5 of 10 externally specified mutants; registered fixture additions then killed all five survivors on the same challenge set.

    Source: §1 · p. 2, §5.4 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    A post-freeze challenge score makes monitor silence falsifiable and recomputable, while distinguishing suites that do not trigger from suites whose checks detect the challenged faults.

    Source: §7 · p. 9

    The 10/10 repair result is not a held-out estimate, and synthetic deterministic implementations with hand-written first-order faults limit external validity.

    Source: §5.4 · p. 7, §6 · p. 9

Abstract

from arXiv · show

Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.

1 Introduction

The paper tests whether passing a correct implementation and failing an all-defects variant meaningfully validates a verification suite for agent infrastructure. A frozen suite’s score against externally specified mutants exposes its limits, while registered fixture additions explain and repair the surviving faults.

1 Introduction

  • Unlike mutation score’s usual role in guiding test development, this study asks what a verifier’s silence entitles others to conclude when its output informs deployment or intervention.The paper frames the gap as a question about whether evaluation results reflect the measured property or the evaluation method itself.Source: §1 · pp. 1–2
  • The study targets a client-side projection layer that rebuilds multi-session conversation state from asynchronous events and examines six failure classes with an invariant suite and first-order mutants.The classes include duplicate delivery, replay, session leakage, out-of-order rendering, non-convergent updates, and stale-state regression.Source: §1 · p. 2
  • A suite passed the standard correct/all-defects validation even though a first-order mutant removing event-identity deduplication survived all eleven checks.Two independent guards suppressed redelivered events, so no fixture separated their effects.Source: §1 · p. 2
  • Instrumentation separated five survivors into three faults never activated by fixture inputs and two that corrupted internal state unseen by any oracle.The first group needs discriminating inputs; the second needs inputs that propagate corruption to an observable output.Source: §1 · p. 2
  • The authors registered seven uncovered input dimensions and predicted which survivors each would expose before adding one fixture per dimension with existing oracles unchanged.All five registered predictions held.Source: §1 · p. 2
  • 5 of 10 externally specified first-order mutants were killed by the frozen suite; after repair, all five survivors died on the same challenge set.The repaired result is not a second held-out estimate.Source: §1 · p. 2

2 Related Work

Related work addresses assurance frameworks, observed agent failures, framework testing, monitor evaluation, mutation analysis, and differential testing. This paper distinguishes its target—session-state consistency—and its fixed-mutant audit of whether a checker detects known faults.

Contract- and invariant-based verification of agent infrastructure

  • Paduraru et al. [2026] search for orchestration counterexamples, whereas this paper tests whether a fixed, interface-compatible mutant violating a specification is detected in session-state infrastructure.The authors describe open-ended fault search and differential mutation auditing as complementary approaches.Source: §2 · p. 3

Empirical failure taxonomies for deployed agents

  • Wu [2026] characterizes silent failures in a production agent runtime through postmortems, while this paper asks whether an automated checker would catch them.Source: §2 · p. 3

Active testing of agent frameworks

  • Long et al. [2026] combine fuzzing with an active runtime-state oracle, while this paper uses passive differential replay against fixed scenarios for a session-state layer.Source: §2 · p. 3

Evaluation of monitors rather than systems

  • Unlike monitor evaluations using labeled failure data, this study constructs faults by design because it has no labeled corpus of real failures.This designed fault set makes a kill rate available.Source: §2 · p. 3

Relation to mutation testing

  • Mutation analysis measures whether tests detect constructed faulty variants; this paper applies a post-freeze mutation score and traces where surviving faults stop along reachability, infection, and propagation.The authors distinguish first-order single-fault variants from the higher-order all-defects variant.Source: §2 · p. 3

Relation to metamorphic and differential testing

  • Redelivery, replay, reordering, and convergence are treated as relations between executions, but this study uses explicit invariant assertions rather than differential testing’s independently written implementations as pseudo-oracles.Source: §2 · p. 3

3 The Verification Target: A Multi-Session Projection Layer

The verification target is a multi-session chat interface whose visible state is a client-side projection built by applying asynchronous events. The paper treats six failure classes as a fixed target for evaluating a checking methodology, not as claims about a specific deployed system.

3 The Verification Target: A Multi-Session Projection Layer

  • Each session’s visible message list and run-status field are built by applying asynchronous events to prior state, including local echoes, HTTP acknowledgments, WebSocket pushes, and periodic snapshots.Source: §3 · p. 3
  • The ProjectionAdapter supplies an initial state, applies event envelopes, and renders visible messages, with optional hooks for optimistic updates, HTTP acknowledgments, and snapshots.A check requiring an unimplemented optional hook skips rather than silently passing.Source: §3 · p. 4
  • The six target failure classes are non-idempotent delivery and replay, cross-session leakage, order-insensitive rendering, non-convergent concurrent update paths, and stale-state regression.Source: §3 · p. 4
  • The six classes are treated as recurring consequences of the architecture and as a fixed validation target, not as findings about any specific deployed system.Source: §3 · p. 4

4 Method: A Reference Implementation and Its Mutants

The method constructs a reference adapter and faulty variants for a multi-session projection layer. The all-defects mutant tests the suite against combined faults, while first-order mutants isolate individual mechanisms.

4 Method: A Reference Implementation and Its Mutants

  • The reference adapter deduplicates events, upserts messages, scopes events by session, buffers out-of-order events, reconciles concurrent paths, merges snapshots, and keeps run-status updates monotonic.Source: §4 · p. 4
  • NAIVE, the constructed all-defects mutant, appends every event, omits session scoping and ordering safeguards, fails to reconcile channels, replaces state on snapshots, and permits status regression.Its six choices instantiate the six failure classes by construction.Source: §4 · p. 4
  • The authors construct five first-order mutants, each identical to the reference except for one injected fault, after noting that correct/all-defects testing leaves intermediate suite resolution unmeasured.Source: §4 · p. 4
  • The specified first-order faults include removing event-identity deduplication, removing the run-status version guard, and removing the session-scope guard.Source: §4 · p. 4

4. Dimension-

The suite’s conventional validation does not establish how well it detects faults: it can pass the reference and fail an all-defects variant while a single-fault mutant survives. The paper therefore emphasizes post-freeze testing against independently specified mutants, while treating family localization as a property of mutant choice rather than a suite requirement.

4. Dimension-

  • A first-order mutant can survive all eleven checks because no fixture separates two enforcements of the same property.The standard validation can certify the suite despite that surviving mutant.Source: Figure 1 · p. 5, §4 · p. 5
  • The ordering defect advances the cursor to the highest sequence observed on a gap, rather than the highest contiguous sequence applied.The suite exercises eleven deterministic scenarios across five invariant families; later sections add checks in response to surviving mutants.Source: §4 · p. 5
  • A mutant changes one syntactic element, but a cursor or merge fault can violate several properties.Thus a firing check does not imply that the fault has only one behavioral consequence.Source: §4 · p. 5
  • The suite requires passing REFERENCE and failing NAIVE, but meeting both conditions can still leave a detection hole.The proposed discriminating requirement is how many mutants specified by someone who did not design the fixtures are killed by a suite frozen before those mutants exist.Source: §4 · pp. 5–6

5 Results

The standard validation certified a suite that missed a first-order mutant, and the frozen twelve-check suite killed five of ten externally specified mutants. Instrumentation separated never-activated faults from oracle-masked state changes; adding fixtures for registered uncovered input dimensions killed all five survivors on the same challenge set.

5.1 Outcome of the Standard Validation

  • The eleven-check suite passed REFERENCE and failed NAIVE, yet a mutant removing event-identity deduplication failed none of its checks.Sequencing suppressed redelivery before missing deduplication mattered, while messageId upserts made repeated message.created events leave the same state.Source: §5.1 · p. 6

5.2 Post-Freeze Mutation Kill Rate

  • Five of ten externally specified mutants were killed by the frozen twelve-check suite, and REFERENCE passed all twelve checks.This adversarial challenge-set score is not an estimate of detection under a natural fault distribution.Source: §5.2 · p. 6

5.3 Two Modes of Missing a Fault

  • Instrumentation found three never-activated survivors and two whose internal-state divergence was masked from the oracles.The former require new input values; the latter could also be exposed by observing more internal state.Source: §5.3 · pp. 6–7

5.4 Discriminating Fixture Coverage as a Registered Prediction

  • Before writing fixtures, the authors registered seven discriminating dimensions, marking five uncovered dimensions and the survivors each should explain.They added one fixture per uncovered cell and reused existing oracle predicates, keeping the oracles fixed.Source: §5.4 · p. 7
  • All five survivors were killed by the added fixtures, and every registered prediction held; REFERENCE passed all seventeen checks.C-13 also killed H3, showing that the input dimensions were not orthogonal.Source: §5.4 · p. 7
  • The repair predicts outcomes on the same survivors used to derive its dimensions; it does not establish performance on unseen mutants or completeness of the seven dimensions.The registered diagnosis predicted which inputs would expose which faults before those inputs existed.Source: §5.4 · p. 7

6 Discussion

The discussion distinguishes activation gaps from masking gaps and argues that both can be addressed by adding discriminating inputs while keeping the oracles fixed. It also considers input-space partitioning, invariant-family coverage, skip semantics, and the study’s scope.

Activation gaps and masking gaps

  • Three survivors were never activated by the fixtures, while two corrupted internal state that the existing oracles could not observe.Instrumenting each survivor against the reference distinguishes the two modes; a stronger assertion on the same inputs could address only the latter.Source: §6 · pp. 7–8
  • The five added checks change inputs while reusing existing oracle predicates, keeping the repair attributable to the input changes.The predicates include digestOf, runStatus, and visibleMessages.Source: §6 · p. 8, Table 2 · p. 8

Evidential value of a monitor’s silence

  • A post-freeze challenge score makes a monitor’s silence falsifiable and recomputable, unlike a record of never firing.The discussion says the suite’s 5/10 challenge-set result is a statement about what a determined reader could get past it.Source: §6 · p. 8

Redundancy in the system under test

  • Redelivery fixtures could not distinguish event-identity deduplication from sequence-staleness rejection because both suppress the same event.The discussion compares this masking between implementation mechanisms with condition coverage such as MC/DC.Source: §6 · p. 8

Relation to input-space partitioning

  • The seven input dimensions use surviving mutants to reveal missing partitions, then test whether those partitions predict a repair.Proposed extensions include redelivery distance two, session-identity variants, version relations, and longer channel-order traces.Source: §6 · pp. 8–9

Invariant families as an organising unit

  • Invariant families organize checks but do not establish detection coverage: a deduplication fault surfaced in a convergence check, while unrelated snapshot faults shared one check.Source: §6 · p. 9

Deployment semantics of a skip

  • For an unrecoverable property, the discussion says inability to observe it plausibly should fail closed rather than silently pass.One check cannot be attempted against NAIVE because it exposes neither required hook.Source: §6 · p. 9

Scope

  • The study uses synthetic, deterministic implementations, limiting external validity to hand-written first-order faults, an imperfect proxy for developer faults.The held-out mutant author saw the system description and check titles, but not the fixtures.Source: §6 · p. 9

7 Conclusion

The paper measures the evidence provided by a verification suite for agent infrastructure and finds that standard validation can certify a suite that a mutant survives. It distinguishes two survivor modes and reports a predicted repair using added inputs with unchanged oracles.

7 Conclusion

  • The certified suite killed five of ten externally specified faults, while a first-order mutant survived standard validation.The faults were specified by a reader who had not designed the fixtures.Source: §7 · p. 9
  • The five survivors comprised three faults never activated by any fixture and two that corrupted state no oracle could observe.After registering uncovered input dimensions and predictions, the authors added one fixture per dimension with oracles unchanged; all five predictions held.Source: §7 · p. 9
  • For a deployment-gating layer, a quiet monitor can present as success even when its output does not establish the intended property.The measurement runs the existing suite once against single-edit mutants of the reference implementation.Source: §7 · p. 9
Loading 2610.02928v1…