PaperlayerView exact revision on arXiv

Source-linked AI summary

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

Peiying Zhu, Sidi Chang

arXiv:2609.01992v1cs.AIcs.CRcs.MA

TL;DR

Agent evaluations need evidence that is both sufficient to recompute a claim and complete against a committed experiment set. ClaimReceipt binds typed evidence to a signed manifest and selectively verifies historical and prospective records, reproducing expected audit behavior while abstaining when evidence or coverage is incomplete. The method remains bounded by committed ingress, claim scope, privacy governance, and specification legibility.

  • Problem

    Agent evaluations lack reliable ways to establish both claim-sufficient evidence and coverage of the committed experiment set, while generic logs and hash-linked transcripts can omit semantically necessary records.

  • Method

    ClaimReceipt uses claim-relative typed receipts, signed experiment manifests, selective replay, coverage checks, and explicit PASS, INVALID, or INCONCLUSIVE outcomes.

  • Results

    The verifier reproduces the historical audit and distinguishes complete, missing-terminal, and missing-private-evidence prospective cases as preregistered.

  • Takeaways & Limitations

    Agent verification requires both claim-sufficient evidence and a committed universe against which omissions become visible, including when the correct result is abstention.

  • Takeaways & Limitations

    The evidence is limited to one configurable hotel testbed and principal model family, designed faults, a frozen claim class, no independent implementation or second domain, and unresolved trust-role governance.

Abstract

from arXiv · show

Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether the retained records cover the committed experiment set (coverage). Generic logs and hash-linked transcripts answer neither reliably. We introduce ClaimReceipt, a claim-relative receipt specification and selective verifier that binds typed transaction evidence to a signed experiment manifest and returns PASS, INVALID, or INCONCLUSIVE per claim. We freeze the specification before implementation (SHA-256 18d109...b81). On 1,392 historical buyer--seller records, a CR-2 verifier reproduces all five manually labeled audit verdicts, exactly replays 600 deterministic and 792 post-generation records, makes every one of 13 declared field groups non-redundant under tested ablations, and returns the expected result on 11/11 semantic faults with 0/8 false positives. We then run a separate prospective CR-3 epoch: 30 assignments are committed before inference, terminal receipts are signed and chained, and private evidence is encrypted for an auditor. Complete evidence yields coverage and accounting PASS; withholding one terminal receipt returns INCONCLUSIVE_COVERAGE, while withholding all private openings preserves coverage and protocol verification but makes economic claims inconclusive, exactly matching a preregistered prediction. Receipt instrumentation adds 0.021% of model-inference time and 9.9 KB per transaction. A specification-legibility probe indicates that our own frozen specification is not yet unambiguous to an independent reader. Claim verification therefore requires both claim-sufficient evidence and a committed universe against which omissions become visible.

1 Introduction

Agent evaluation requires both claim-relative sufficiency—whether retained evidence recomputes a claim—and coverage—whether records represent the complete committed experiment set. ClaimReceipt formalizes these requirements with typed receipts, manifests, selective verification, and explicit abstention.

  • Motivation: Sufficiency asks whether a reported number can be recomputed from retained evidence, while coverage asks whether records constitute the complete committed set.Generic logs may preserve bytes while omitting private state, discarded attempts, parser outputs, or expected terminal records.
  • Motivation: Interactive agent outcomes combine model generations, hidden state, parsers, enforcement, retries, and scoring, so a numerically correct scalar can support an invalid causal interpretation.This problem is especially acute in multi-agent systems, where environments shape what agents observe and do.
  • Approach: ClaimReceipt binds typed transaction evidence to an experiment manifest and uses selective verification to check integrity, coverage, and claim-specific replay.The verifier may license descriptive reporting while blocking a causal policy claim.
  • Approach: The schema is sufficient by construction for a bounded claim class, not for arbitrary claims about intent or external truth.The paper explicitly formalizes claim-relative sufficiency and supplies a field-to-claim dependency matrix.
  • Evaluation: The CR-2 verifier agrees with 5/5 manual verdicts, exactly replays every declared transaction claim, and reports the complete historical 4 × 6 grid plus a prospective row.The evaluation covers 1,392 receipts across six epochs rather than only labeled cells.

2 Problem Formulation

The formulation separates claim sufficiency from experiment coverage and defines both relative to explicit executions, claim classes, and committed assignment sets. Transaction and experiment claims are then instantiated with exact accounting and batch-level audit semantics.

  • 2.1 Claim-relative sufficiency: Claim-sufficient evidence guarantees that executions agreeing on retained fields also agree on the truth of the declared claim.The definition concerns executions and claim truth, not whether a verifier happens to output a verdict.
  • 2.1 Claim-relative sufficiency: A verifier that always abstains cannot make empty evidence sufficient, and missing required evidence must produce rejection or abstention rather than substitute a reported scalar.This prevents verifier behavior from redefining evidentiary sufficiency.
  • 2.1 Claim-relative sufficiency: Version 0.3 supports eight transaction claims and four experiment claims, including completion, economic quantities, treatment fidelity, protocol isolation, stability, replay, and accounting.The supported claims define the scope within which sufficiency is assessed.
  • 2.1 Claim-relative sufficiency: The specification does not claim sufficiency for latent intent, equilibrium, fairness, external profile truth, or uncommitted transactions.These are explicit scope boundaries rather than claims the retained schema can establish.
  • 2.1 Claim-relative sufficiency: Coverage is a property of a set and requires a pre-outcome experiment manifest plus an ingress commitment defining the expected assignments.A single receipt cannot establish that every committed assignment produced a terminal record.
  • 2.2 Running transaction semantics: A buyer profile assigns private values, an outside-option utility, and hard constraints to components, while accepted offers determine transaction welfare and rejected or failed transactions contribute zero.These transaction semantics provide the inputs for recomputed economic claims.
  • 2.2 Running transaction semantics: The first-best claim retains the exact rational pair (W, W⋆) when W⋆ > 0 and a typed null otherwise, preserving negative welfare rather than clamping it.Match quality is likewise retained as an exact numerator–denominator pair.
  • 2.2 Running transaction semantics: C1–C4 test directional response, cross-arm protocol isolation, stochastic stability, and replay with complete accounting, each returning Pass, Invalid, or Inconclusive.C3 uses profile-level pairing with replications nested inside profile–condition cells.

3 ClaimReceipt Design

ClaimReceipt separates transport integrity, semantic sufficiency, experiment coverage, and external truth, then binds receipts to manifests, commitments, and auditor-controlled evidence. Its replay and accounting design supports selective verdicts while preserving explicit trust and privacy boundaries.

  • 3.1 Four evidence layers: Transport integrity detects mutation of committed bytes through exact bytes, sequence numbers, chain links, and signatures.These mechanisms protect retained evidence but do not determine which semantic distinctions claims require.
  • 3.1 Four evidence layers: Semantic sufficiency retains typed profile, trace, offer-stage, choice, catalog, and scorer inputs needed to recompute claims.This layer addresses evidentiary content rather than byte-level tampering.
  • 3.1 Four evidence layers: Experiment coverage binds receipts to an expected assignment matrix and ingress commitment, making omissions after committed ingress visible.Coverage is distinct from replay and cannot be established by one transaction receipt alone.
  • 3.1 Four evidence layers: External truth remains outside the system: receipts cannot prove that an unobserved interaction occurred or that a synthetic profile describes a real person.This boundary prevents cryptographic integrity from being mistaken for construct validity.
  • 3.2 Threat model and trust boundary: The threat model covers byte mutation, construct drift, and post-commit omissions, while interactions prevented from reaching the witness remain outside observability.L4 is reported as residual risk rather than counted as a false negative.
  • 3.3 Experiment manifests: A signed manifest can act as cryptographic preregistration when it predates ingress, whereas reconstructed retrospective manifests support faithful mechanization rather than prospective confirmation.The prospective E7 manifest precedes all 30 ingress tickets.
  • 3.4 Transaction receipts and privacy: Public receipts expose assignment, status, digests, commitments, exact claims, and coverage counts, while auditor envelopes preserve private profiles, ordered events, attempts, offers, choices, treatments, and artifacts.The split supports verification without making hidden evidence a public transcript.
  • 3.4 Transaction receipts and privacy: Private canonical objects use fresh nonces and encrypted openings, while canonical ordering, scaled integers, aggregate pairs, and precision envelopes make replay and boundary decisions deterministic.The verifier abstains when a decision boundary intersects the retained precision envelope.

4 Implementation and Evaluation

The implementation freezes ClaimReceipt before verification and evaluates both retrospective batch verification and prospective coverage. Across the imported corpus, it tests replay, evidence sufficiency, semantic-fault handling, and manifest-based observability.

  • Specification and implementation: The team adversarially reviewed and froze ClaimReceipt v0.3 before implementing the verifier.Earlier v0.1 and v0.2 files and hashes remained immutable, and no verdict or fault result informed v0.3.
  • Specification and implementation: The reference implementation supports CR-2 Batch Verification and CR-3 Prospective Coverage on E7.CR-3 adds signing roles, auditor opening, pre-ingress assignment tickets, encrypted evidence, signed receipt chaining, and coverage-qualified output.
  • Corpus and evaluation design: 1,392 terminal records comprise six retrospective epochs, including 600 deterministic records and 792 LLM or fixed-menu records.E7 is separate from the 1,392-record corpus and runs 15 profiles under NONE and BOTH with Qwen2.5-Instruct 3B, k = 1.
  • Corpus and evaluation design: R1 reproduces five manually labeled audit cells and reports the historical 4 × 6 grid, while later controls separately test replay, evidence abstractions, semantic faults, observability, omissions, and CR-3.The evaluation includes 13 field-group removals, frozen semantic faults with benign controls, and prospective-path testing.
  • Corpus and evaluation design: Each retrospective epoch receives a separate manifest rather than sharing one corpus-wide manifest.The retrospective corpus is presented as a table with a separate manifest for every epoch.

5 Results

The verifier reproduces audited verdicts and exact replay within declared boundaries, while testing claim sufficiency, semantic fault handling, and prospective coverage. Its results also expose limits: evidence abstractions omit required claim inputs, and specification legibility remains unresolved.

  • 5.1 R1: the verifier reproduces the manual audit: 5/5 manually labeled cells match, with E1 marked C2 Invalid, E2 C2 Pass, E4 C1 and E3 C3 Inconclusive, and joint scripted accounting Pass.The verifier also distinguishes contract abstention from evidential abstention and reports the complete historical grid plus the separate prospective row.
  • 5.2 R2: the replay boundary is exact: 600 deterministic records exactly match at policy, normalization, chooser, and metric stages, while 792 LLM-associated records replay from committed raw offers through choice and economic claims.LLM text is audited but not regenerated, so the replay claim is narrower once stochastic generation begins.
  • 5.3 R3: generic evidence is not claim-sufficient: Outcome scalars, transcripts, and generic signed receipts determine none of the 12 tested claims, whereas full ClaimReceipt determines all 12 within its declared scope.An opened transaction receipt recovers six economic claims but not treatment fidelity or batch criteria.
  • 5.3 R3: generic evidence is not claim-sufficient: 13/13 field-group removals make at least one claim underdetermined, establishing non-redundancy under tested ablations but not global field minimality.The dependency matrix is frozen in advance as an implementation-conformance check rather than discovered from data.
  • 5.4 R4: semantic faults and benign controls: 11/11 semantic faults receive expected results with 0/8 false positives, including detection of schema drift, selective reporting, missing terminal evidence, and pseudoreplication.These are specification-conformance results on a small frozen suite, not estimates against all possible attacks; interactions outside committed ingress remain residual risk.
  • 5.4 R4: semantic faults and benign controls: The external legibility probe was exploratory and found that the frozen specification omits explicit fault timing, while one model under one protocol cannot separate specification ambiguity from model-specific reading failure.The proposed fault_timing amendment was not adopted because post-hoc adoption would invalidate preregistration.
  • 5.5 R5–R6: prospective coverage, observability, and overhead: A missing terminal receipt returns INCONCLUSIVE_COVERAGE, while withholding private openings preserves coverage and C2 but makes economic and treatment-fidelity claims inconclusive.The private-opening outcome matches a preregistered prediction; prospective E7 committed 15 profiles under each of NONE and BOTH before inference.

6 Discussion and Limitations

ClaimReceipt’s central contribution is semantic: it binds each claim to the evidence distinctions it requires and uses committed coverage to expose omissions. The discussion also bounds these guarantees by prospective-ingress scope, privacy requirements, limited evaluation breadth, and specification legibility.

  • ClaimReceipt answers which retained distinctions a declared claim depends on, whereas hash chains only show whether retained bytes changed.
  • Inconclusive cells are intended when historical runs were not designed for every criterion, allowing exact descriptive accounting while blocking unsupported causal claims.
  • Prospective CR-3 coverage is guaranteed only relative to committed ingress; transactions prevented from reaching the witness remain outside observability.
  • Privacy remains an engineering and governance burden because auditable welfare requires sensitive evaluator profiles and independent custody, authorization, retention, and deletion.
  • The evaluation covers one configurable hotel testbed, one principal model family, designed faults, and one frozen claim class, without a second domain or independent implementation.

7 Related Work

ClaimReceipt extends existing agent-evaluation, provenance, and construct-validity work by making bounded evidence requirements and conditional coverage machine-checkable. Its focus is claim semantics rather than new cryptographic transport.

  • Agent-evaluation research emphasizes interactive, scenario-aware measurement, while trace audits show aggregate success can conceal failures and misrank repairs.
  • Existing provenance mechanisms provide append-only integrity and artifact-dependency tracking; ClaimReceipt adds claim semantics and conditional coverage using standard primitives.
  • ClaimReceipt turns construct-validity concerns into machine-checkable contracts that prevent missing evidence from silently becoming a positive conclusion.

8 Conclusion

Agent verification requires both claim-sufficient evidence and a committed experiment set against which omissions are visible. ClaimReceipt demonstrates this method across retrospective and prospective evaluations, including principled abstention when evidence is incomplete.

  • 1,392 historical records support manual-audit reproduction, exact replay of deterministic and post-generation claims, and separation of semantic faults from benign variation.
  • 30 prospective transactions let CR-3 distinguish complete evidence, a missing terminal, and unavailable private openings at negligible runtime cost.
  • The demonstrated method decides which agent-evaluation evidence licenses a claim and when it licenses only abstention.
Loading 2609.01992v1…