PaperlayerView exact revision on arXiv

Source-linked AI summary

Claim-Gated Source-Risk Auditing for Generative Search

Kainan Zhou, Chuhong Xu, Gangzhen Qian, Zhaoyi Li

arXiv:2609.29145v2cs.AIcs.CCcs.CEcs.ETcs.IR

TL;DR

A supported generative-search answer can still omit a source relationship that changes its interpretation. The paper specifies a complete-case, claim-gated audit and an executable checker; exhaustive tests establish finite conformance, not semantic accuracy or deployment benefit.

  • Problem

    Citation support alone does not resolve whether an answer omits a source relationship that changes interpretation, a distinct question from whether its cited statements are supported.

    Source: §I · p. 1

  • Method

    The specification audits query–source–answer tuples using relationship, adoption, materiality, and disclosure judgments, leaving the endpoint unresolved when any judgment is unknown.

    Source: A. Unit, Predicates, and Complete-Case… · p. 2

  • Results

    Exhaustive predicate enumeration and common-guard and ablation comparisons establish finite conformance of the implementation and expose the coverage cost of the complete-case policy.

    Source: B. Baselines and Common-Guard Comparison · p. 3, C. Predicate Ablations and Coverage… · p. 4, §VI · p. 5

  • Takeaways & Limitations

    Independent human labels and paired answer-level evaluations remain necessary before claiming semantic accuracy, omission prevalence, or application benefit.

    Source: §VI · p. 5

    The artifact validates typed synthetic records and deterministic logic, not source discovery, temporal validity, annotation reliability, multilingual transfer, or reader-interpretation changes.

    Source: C. Limitations, Falsifiability, and… · p. 5

Abstract

from arXiv · show

A generative search answer can cite a supported passage yet omit a source relationship that changes its interpretation. We specify a claim-gated audit of the query-source-answer tuple. An omission is resolved only when relationship evidence, answer adoption, materiality, and disclosure are all observed; incomplete evidence remains unresolved rather than being treated as independence. The specification separates this endpoint from citation support and review priority, and binds decisions to versioned evidence spans. A reference checker makes the record contract executable. On an exhaustive synthetic suite, it reproduces all 81 three-state predicate combinations and rejects 192 deliberately malformed records. Common-guard baselines and predicate ablations isolate endpoint logic from missing-evidence handling, while controlled transitions check support separation and evidence removal. These are finite contract-conformance results, not detector accuracy or evidence of improved user outcomes. We define the independent annotation, held-out evaluation, and paired utility tests still required to establish semantic validity and deployment benefit.

I. INTRODUCTION

Generative search can produce a supported statement whose source relationship still changes how readers should interpret it. The paper specifies an executable, query–source–answer audit while distinguishing implementation conformance from semantic validity or deployment benefit.

I. INTRODUCTION

  • A citation can support a statement without establishing that the source is independent of the queried target.The answer may be accurate while an independent-assessment interpretation remains unwarranted.Source: §I · p. 1
  • The audit treats the query, source, and answer’s treatment of that source as one unit, rather than treating ownership or commercial status alone as an omission.A clearly identified company specification may still support an appropriate factual comparison.Source: §I · p. 1
  • Citation-generation benchmarks measure correctness and citation quality, while publisher-side answer-inclusion optimization motivates retaining source context.Visibility or persuasive language alone does not establish a particular relationship.Source: §I · p. 1
  • The paper contributes a four-predicate omission endpoint, a record contract separating evidence and judgments from review priority, and a reproducible conformance suite.It is an executable specification, not a new neural architecture.Source: §I · p. 1
  • The evaluation tests implementation of explicit judgments, not relationship recognition in Web text or correctness of human judgments.It does not estimate omission prevalence, demonstrate better answers, or certify publisher trustworthiness.Source: §I · p. 1

A. Citation Support and RAG Evaluation

Content-level evaluations can assess factual support, but the audit also requires relationship evidence and a separate judgment of disclosure. It defines a complete-case omission endpoint and applies specification-derived behavioral tests whose agreement establishes implementation conformance, not predicate validity.

A. Citation Support and RAG Evaluation

  • RAGTruth and RAGAs measure content-level properties but do not themselves provide the archived source–target relationship required by this audit.Citation support remains a separate field so factual and contextual failures can be inspected without conflation.Source: A. Citation Support and RAG Evaluation · p. 1

B. Relationship Evidence and Disclosure

  • Affiliate-abuse and endorsement-disclosure research motivate separating evidence that a relationship exists from whether the answer adequately explains it.A disclosure keyword elsewhere on a page does not show that the answer preserved the relevant context.Source: B. Relationship Evidence and Disclosure · p. 1

C. Behavioral Tests and the Claim Boundary

  • Behavioral tests change one record field and check its expected endpoint transition, but specification agreement cannot establish that predicates are meaningful or recoverable from natural text.The state space has specification-derived outcomes, unlike a natural-language benchmark.Source: C. Behavioral Tests and the Claim… · p. 2

A. Unit, Predicates, and Complete-Case Endpoint

  • The audit unit is a query, archived source, and resulting answer; its four predicates record relationship, adoption, materiality, and adequate disclosure, with U meaning unresolved rather than absent.Multiple sources yield separate tuples.Source: A. Unit, Predicates, and Complete-Case… · p. 2
  • An omission resolves only for (1, 1, 1, 0); any unknown predicate leaves the complete-case endpoint unresolved.Even a state that short-circuit reasoning could resolve remains unresolved to require all four documented judgments.Source: A. Unit, Predicates, and Complete-Case… · p. 2
  • The endpoint concerns omitted material context in an observed answer; it does not attribute intent, deception, or a causal effect of the source on generation.Adoption and materiality require their own evidence rather than following from a relationship label.Source: A. Unit, Predicates, and Complete-Case… · p. 2

B. Evidence Vector and Review Priority

The specification records five evidence dimensions and uses them to order review, separately from the omission endpoint. Query susceptibility can support sampling, but the priority score does not alter endpoint resolution.

B. Evidence Vector and Review Priority

  • The evidence vector represents query susceptibility, relationship evidence, adoption, materiality, and disclosure deficit; resolved entries map explicitly to the corresponding predicates.These entries are not class probabilities, need not sum to one, and a graded codebook would require separate validation.Source: B. Evidence Vector and Review Priority · p. 2
  • The review-priority score orders review and never changes the omission endpoint.The checker returns unresolved priority if any dimension is missing, even when its weight is zero.Source: B. Evidence Vector and Review Priority · p. 2
  • Relationship, adoption, materiality, and disclosure resolve RO, whereas dQ affects review priority only.The reference checker consumes explicit judgments rather than raw-text predictions.Source: Figure 1 · p. 2
  • Query susceptibility can support sampling, while answer-dependent dimensions cannot select evidence used to generate that same answer.Equal weights are illustrative, not an optimized policy.Source: B. Evidence Vector and Review Priority · p. 2

C. Module Interfaces and Execution Order

The module interfaces separate archived inputs, annotation judgments, contract checking, endpoint resolution, and review-priority scoring. The implemented checker handles synthetic records; collection, entity resolution, natural-language annotation, and generation remain external.

C. Module Interfaces and Execution Order

  • The archive adapter stores query, source, and answer snapshots plus pipeline configuration; the annotation adapter supplies codebook version, predicate values, and evidence spans.Conforming records proceed to endpoint resolution and priority scoring, whose outputs enter a claim ledger.Source: C. Module Interfaces and Execution Order · p. 2
  • Execution validates fields and hashes, predicate types and evidence offsets, dimension mapping and weights, then resolves RO, computes priority, and attaches the validation gate.Malformed records are rejected with a reason, while structurally valid records with unresolved judgments return U.Source: C. Module Interfaces and Execution Order · p. 2
  • The checker generates synthetic records with supplied judgments; Web collection, entity resolution, natural-language annotation, and generation are external interfaces.A learned annotator would need evaluation against independent labels before any accuracy claim.Source: C. Module Interfaces and Execution Order · p. 2

D. Source Roles and Admissible Evidence

The source-role taxonomy guides evidence collection and disclosure requirements, while the record contract preserves versioned, evidence-linked decisions. Its invariants keep source risk distinct from citation support and prevent missing evidence from resolving as low risk.

D. Source Roles and Admissible Evidence

  • Source roles determine what evidence to seek and what disclosure the codebook requires; they are not standalone risk scores.Current ownership evidence may resolve I, but A, M, and D still concern the particular answer.Source: D. Source Roles and Admissible Evidence · p. 2
  • Relationship decisions require recoverable evidence from an archived observation and are tied to the source version and time window used for the answer.Tone, rank, similarity, and commercial status may guide investigation but do not resolve a relationship; later evidence creates a new record version.Source: D. Source Roles and Admissible Evidence · p. 2

E. Record Contract and Invariants

  • Resolved judgments require recoverable evidence spans, while unresolved synthetic judgments use JSON null; missing keys, "U," and zero are distinct.The contract also stores query and tuple identifiers, source and answer snapshots, metadata, predicate judgments, and evidence offsets.Source: E. Record Contract and Invariants · p. 3
  • SHA-256 digests bind snapshots, tuples, records, and annotation revisions, supporting change detection but not authenticity or semantic correctness.Append-only storage enforcement remains a deployment responsibility.Source: E. Record Contract and Invariants · p. 3
  • Six invariants separate citation support from source risk, keep findings tuple-local, preserve evidence pointers, and prevent missing evidence from resolving as low risk.Generation and post hoc review remain separate, and stronger empirical claims require stronger validation; the suite tests specified transitions, not every implementation.Source: E. Record Contract and Invariants · p. 3

IV. EXECUTABLE CONFORMANCE EVALUATION

The evaluation uses an exhaustive synthetic state space to check a specified endpoint, with expected outputs expressed separately as an oracle. It involves no real-Web labels, model fitting, or statistical sampling.

A. Questions, Fixtures, and Oracle

  • The suite enumerates all 81 combinations of four three-state predicates, comprising one positive, 15 negatives, and 65 unresolved cases.Each synthetic tuple includes query, source, and answer snapshots, version metadata, and explicit judgment spans; its text contains fixture annotations, not naturally occurring Web claims.Source: A. Questions, Fixtures, and Oracle · p. 3
  • The exhaustive contract evaluation requires neither model fitting nor statistical sampling, and uses no training, real-dataset preprocessing, or train/test split.Supplied standard-library scripts generate fixtures, expected states, predictions, mutation logs, CSV summaries, and a checksum manifest without paid APIs, pretrained models, or network access.Source: A. Questions, Fixtures, and Oracle · p. 3
  • The expected output is positive only for (1, 1, 1, 0) among fully observed states and unresolved otherwise, using a separately expressed test oracle.This oracle is not independent semantic ground truth; the artifact has no human labels or live-Web observations.Source: A. Questions, Fixtures, and Oracle · p. 3

B. Baselines and Common-Guard Comparison

The baselines show that support, relationship, or disclosure alone cannot reproduce the complete-case omission endpoint, even when all share a completeness guard. Ablations and controlled transitions test the endpoint’s predicate requirements and the checker’s structural behavior on synthetic records.

B. Baselines and Common-Guard Comparison

  • With a common completeness guard, baseline resolved-state mismatches remain; without it, support-only resolves all 65 incomplete states and misses the sole positive.Its guarded agreement on 15 of 16 complete states reflects the rarity of the positive state, not evidence that support replaces relationship auditing.Source: B. Baselines and Common-Guard Comparison · p. 3

C. Predicate Ablations and Coverage Trade-off

  • Dropping any one of I, A, M, or D adds one positive among the 16 fully observed states, while removing dQ changes no endpoint.For example, dropping A incorrectly marks (1, 0, 1, 0) as an omission although the answer does not adopt the source claim.Source: C. Predicate Ablations and Coverage… · p. 4
  • The short-circuit alternative resolves 66 of 81 states, compared with 16 under the complete-case contract, exposing the documentation cost of that stricter policy.These synthetic counts are not deployment rates, and the paper does not claim the stricter policy is universally preferable.Source: C. Predicate Ablations and Coverage… · p. 4

D. Controlled Transitions and Structural Mutations

  • All 81/81 oracle outputs match, every one of 216 evidence-removal destinations remains unresolved, and toggling support leaves endpoints unchanged.The transition checks share base states and are not independent observations.Source: D. Controlled Transitions and… · p. 4
  • Across 16 fully observed records, the suite generated 192 malformed cases from 12 defect types, and all 192 were rejected; records were resealed after mutation except for deliberate checksum or identity defects.Source: D. Controlled Transitions and… · p. 4

E. Measured Checker Cost and Interpretation

The checker’s reported cost is a single-process microbenchmark, not an end-to-end latency estimate. Its exhaustive conformance success measures agreement with the chosen specification, not semantic validity.

E. Measured Checker Cost and Interpretation

  • The median batch-mean cost was 0.49 µs per tuple for the endpoint and 51.20 µs for validation plus auditing.Measurements used 21 repeated batch means after warm-up on an Intel Xeon Platinum 8573C host.Source: E. Measured Checker Cost and… · p. 4
  • The timing excludes archive I/O, retrieval, generation, and annotation, so it is neither end-to-end latency nor a per-request tail-latency estimate.The endpoint loop processed 16,200 tuples per repeat, while the full checker processed 810.Source: E. Measured Checker Cost and… · p. 4
  • The experiment establishes finite conformance to a chosen specification, not whether materiality labels are correct or predicates are meaningful in natural text.No confidence interval or significance test accompanies the exhaustive counts.Source: E. Measured Checker Cost and… · p. 4

V. SEMANTIC VALIDATION AND APPLICATION BOUNDARY

The paper leaves semantic validation and application benefit for future work. It proposes independent, evidence-backed labels and held-out evaluation, followed by paired answer-level tests of audit-informed interventions.

A. Independent Labels and Held-Out Evaluation

  • A Web study should archive observations before annotation, stratify by domain and prespecified risk bands, and prevent target families, related domains, and query templates from leaking into held-out data.At least two trained annotators should independently label each tuple, be able to assign U and cite evidence, with reliability reported on original labels.Source: A. Independent Labels and Held-Out… · p. 4
  • Semantic measurements should report predicate-level errors, unresolved coverage, and domain-clustered intervals; any judge-assisted annotation needs a separately labeled audit subset.The paper cautions that learned judges may have evaluator biases.Source: A. Independent Labels and Held-Out… · pp. 4–5

B. Tool Utility and Deployment Decisions

  • Application benefit requires matched before-and-after generated answers with fixed settings and blinded evaluators, measuring omissions alongside answerability, evidence retention, factual support, abstention, and latency.Suppressing answers or removing related sources alone does not establish a useful defense.Source: B. Tool Utility and Deployment Decisions · p. 5
  • PASS, CONTEXTUALIZE, REVIEW, and ABSTAIN are proposed interfaces, not validated production controls.PASS means the configured review requirement is met for the observed tuple, not that the publisher is independent or universally reliable.Source: B. Tool Utility and Deployment Decisions · p. 5

C. Limitations, Falsifiability, and Release

The artifact establishes finite conformance for typed synthetic records, while leaving real-world relationship discovery, annotation reliability, and reader effects unvalidated. The section also identifies falsifiable assumptions and release requirements.

C. Limitations, Falsifiability, and Release

  • Finite conformance does not establish source discovery, temporal validity of real relationships, annotation reliability, multilingual transfer, or changes in reader interpretation.Materiality and adequate disclosure remain task-dependent judgments, and hash-valid records can preserve erroneous or outdated judgments.Source: C. Limitations, Falsifiability, and… · p. 5
  • Inconsistent materiality judgments, excluded ties, equal performance from simpler rules, impractical review load, or reduced utility would count against the specification.Any such finding calls for revising the affected rule rather than treating conformance as proof of usefulness.Source: C. Limitations, Falsifiability, and… · p. 5
  • The release package includes the checker, fixture generator, test oracle, per-case outputs, aggregate CSVs, and table-generation script.Its checksum manifest binds run files but is not a digital signature.Source: C. Limitations, Falsifiability, and… · p. 5
  • A future Web release must document archive permissions, exclusions, codebook changes, adjudication, and split assignments.Redaction must not silently change the reported sample.Source: C. Limitations, Falsifiability, and… · p. 5

VI. CONCLUSION

  • Independent human labels and paired answer-level evaluations remain necessary before claiming semantic accuracy, omission prevalence, or application benefit.The reported exhaustive tests, baselines, ablations, and mutations establish finite implementation conformance, not those outcomes.Source: §VI · p. 5
Loading 2609.29145v2…