Source-linked AI summary
Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Jingjie Ning, Xueqi Li
TL;DR
Retrieval-augmented QA may change answers after corpus expansion even when the rest of the requested system remains fixed, while aggregate accuracy may hide those changes and generation noise complicates one-shot comparisons. The paper introduces the repeat-aware Snapshot Compatibility Audit, which subtracts same-snapshot repeat disagreement from cross-snapshot disagreement. Across NQ, TriviaQA, and a second-generator replication, it finds answer churn despite small or opposite-signed exact-match changes.
Problem
The paper asks whether corpus growth changes a fixed retrieval-augmented QA system’s answer-level behavior beyond ordinary generator variation, which aggregate utility and one-shot comparisons may not reveal.
Method
The Snapshot Compatibility Audit compares repeated outputs from two fixed snapshots and subtracts same-snapshot disagreement from cross-snapshot disagreement along a one-to-seven-shard FineWeb expansion.
Results
Across NQ, TriviaQA, and a second-generator NQ replication, excess churn persisted while exact-match changes were small or opposite in direction, including 10.25 pp semantic NQ churn with −1.50 pp EM.
Takeaways & Limitations
Answer-level compatibility can fail without a conspicuous or consistently directed utility shift, so retrieval-augmented releases should audit compatibility alongside utility.
Takeaways & Limitations
The empirical treatment is one frozen shard path, so the results establish that this expansion can alter behavior rather than a monotone or universal corpus-scaling law.
Abstract
from arXiv · showhide
A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the \emph{Snapshot Compatibility Audit}, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only $-1.50$ points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.
1 Introduction
The paper asks whether expanding corpus access changes a fixed retrieval-augmented QA system’s answer-level behavior beyond ordinary generation variation. It introduces a repeat-aware audit and finds substantial excess churn despite small or offsetting aggregate-accuracy changes.
- Motivation: Corpus expansion can alter answer-level behavior even when the requested model, prompt, retrieval policy, evidence depth, rendering, and generation controls remain fixed.A deployed snapshot is defined by the fixed system paired with a particular accessible corpus.
- Motivation: Aggregate accuracy can hide compatibility changes because gains and losses cancel, while stochastic generation makes one-shot cross-snapshot differences ambiguous.This matters when answers feed caches, regression tests, automated workflows, or human decisions.
- Audit: The Snapshot Compatibility Audit subtracts same-snapshot repeat variation from cross-snapshot disagreement to estimate excess answer churn.The study expands one nested FineWeb prefix from one to seven shards while holding other recorded interface controls fixed.
- Evidence: TriviaQA showed smaller, directionally consistent excess churn despite a +1.25 pp EM change, while a second-generator 100-question NQ replication found 8.75 pp semantic excess churn alongside a +3.00 pp EM change.These results provide supportive cross-benchmark and configuration evidence that utility movement need not track answer compatibility.
2 Related Work
Related work studies retrieval, changing web collections, context effects, prediction churn, backward-compatible updating, and deployed language-system drift. This paper differs by changing the retrieval corpus rather than model weights and by applying its audit to a frozen experiment.
- Retrieval and context: Prior retrieval research examines external memory, replaceable indexes, evidence selection, long-tail benefits, and misleading or noisy retrieved context.Related benchmarks evaluate relevance, faithfulness, component errors, hallucinations, and robustness.
- Compatibility and drift: Prediction-churn and backward-compatibility work shows that aggregate performance can conceal instance-level changes and negative flips.The concern also appears in structured NLP, data updates, deployed language systems, and generative-model evolution.
- Compatibility and drift: Hosted LLM drift and nominally repeated-call variability motivate distinguishing ordinary stochastic variation from changes caused by an update.This paper applies that concern to corpus changes while leaving model weights unchanged.
- Changing collections: Web-evolution and fresh-QA studies treat indexed collections as time-varying systems, whereas this work uses a frozen FineWeb corpus intervention.Applying the protocol to temporal or conversational-memory settings is proposed as future work rather than empirically tested here.
3 Measuring Corpus-Induced Churn
The audit compares repeated outputs from two fixed snapshots and removes the same-state noise floor from cross-state disagreement. It uses exact and blinded-semantic agreement, whole-question inference, and post-hoc flip diagnostics to measure corpus-associated answer movement.
- Estimand: The audit evaluates two snapshots on identical questions with locked evidence and two independent generator responses per snapshot.In the experiment, the snapshots differ only in accessible corpus scale along one nested shard path.
- Estimand: Positive b_D means cross-snapshot answers are less similar than same-snapshot repeats; the measure is an agreement gap, not the percentage of changed questions.It is a path-specific corpus-associated contrast rather than a universal scaling effect.
- Noise correction: The subtraction removes ordinary same-state noise that a one-answer-per-state comparison would incorrectly attribute to the corpus update.The first term is cross-state disagreement and the second is the same-state noise floor.
- Agreement kernels: Normalized-exact agreement tests benchmark-normalized equality, while the blinded semantic kernel judges whether answer strings express the same factual answer under reasonable aliases.The semantic judge labels all 28 unordered answer pairs per question before gold answers are loaded.
- Interpretation: 10.250 pp semantic and 6.438 pp normalized-exact NQ excess churn remain after subtracting same-state disagreement from raw cross-scale disagreement.Raw disagreement was 21.125% semantic versus 10.875% same-scale, and 81.188% exact versus 74.750% unchanged-state disagreement.
- Inference and diagnostics: Whole-question bootstrap inference retains all eight outputs and 28 semantic relations, while strict repeat-stable semantic flips are defined post hoc and are descriptive.The NQ decision gate combines a 3-pp normalized-exact threshold with positive one-sided lower confidence bounds for exact and semantic excess churn.
4 Experimental Design
The study uses locked, preregistered datasets, corpus prefixes, prompts, retrievals, generation settings, and delayed gold access to compare answer behavior across corpus scales. It also includes a blinded subset replication and post-hoc robustness diagnostics.
- Data and validation: 400 fresh hash-selected NQ questions form the confirmatory population, while 200 independently hash-selected TriviaQA questions provide supportive validation.Selection occurred without answerability, topic, retrieval, or outcome filtering.
- Corpus treatment: The frozen FineWeb index is evaluated along nested no-retrieval, one-shard, three-shard, and seven-shard prefixes.The nested prefixes define a single frozen corpus-expansion path rather than a shard-order experiment.
- Prompt and outputs: Identical answer prompts require one concise structured answer of at most 512 UTF-8 bytes at every scale and benchmark.The prompt permits UNKNOWN and warns that supplied evidence may be incomplete, irrelevant, or untrusted.
- Retrieval and generation controls: Primary and audit retrievals matched exactly in ordered document IDs and rendered evidence bytes across all NQ and TriviaQA retrieval cells.Generation used a stateless singleton DeepSeek adapter configuration with tools disabled, no session persistence, and a fixed JSON schema.
- Blinding and locking: Four successive gates locked retrieval, answers, blinded semantic judgments, and gold access before correctness analysis.The semantic judge saw anonymous question-answer sets without scale, evidence, document IDs, gold, or correctness.
- Replication and diagnostics: A post-hoc 100-question NQ subset reran frozen one- and seven-shard conditions through a stateless direct DeepSeek V4-Pro interface.A separate 50-question outcome-blind sample was judged by OpenAI gpt-5.6-sol to assess semantic-judge family robustness, not human validation.
5 Results
Expanding corpus access produces answer movement beyond same-snapshot repeat noise across NQ, TriviaQA, and a second NQ configuration, while aggregate EM changes remain small or directionally inconsistent. Transition, flip, retrieval-turnover, and judge-robustness analyses show why compatibility requires more than utility reporting.
- Expanded Corpus Access Moves Answers Beyond Repeat Noise: 6.44 pp normalized-exact and 10.25 pp semantic excess churn were observed on NQ, while EM changed by only −1.50 pp.The semantic estimate had a 7.69-pp one-sided lower confidence bound, and the normalized-exact estimate had a 4.56-pp lower bound.
- Expanded Corpus Access Moves Answers Beyond Repeat Noise: 3.00 pp normalized-exact and 2.125 pp semantic excess churn were observed on TriviaQA, with smaller but directionally consistent movement despite opposite EM movement.TriviaQA EM increased by 1.25 pp, and its exact two-sided interval touched zero.
- A Second Configuration Reproduces Churn: 7.25 pp normalized-exact and 8.75 pp semantic excess churn were found in the V4-Pro NQ-100 replication, even as EM rose by 3.00 pp.The matched V4-Flash comparison was descriptive because its semantic labels used different answer-packet construction.
- Accuracy Is an Incomplete Dashboard: Semantic excess churn remained positive as endpoint EM fell by 1.50 pp, rose by 1.25 pp, or rose by 3.00 pp across the three evaluated settings.The figure compares locked V4-Flash NQ and TriviaQA estimates with post-hoc V4-Pro NQ estimates.
- Accuracy Is an Incomplete Dashboard: 46/800 NQ comparisons lost EM and 34/800 gained it, but 155/800 moved between semantically different EM-nonmatches and remained invisible to binary accuracy.The gross correctness flow was 80/800 (10.00%), whereas the net EM difference was only −12/800 (−1.50 pp).
- Repeat-Stable Flips Explain Churn: 40/400 NQ questions and 5/200 TriviaQA questions were strict repeat-stable semantic flips, accounting for most of NQ’s semantic excess churn.The 40 NQ flips contributed 10.00 pp of the total 10.25 pp semantic excess churn.
- Answer Trajectories Across Scales: NQ and TriviaQA had substantial endpoint evidence turnover while retaining high cross-scale semantic agreement.Mean top-eight document overlap was 1.19/8 for NQ and 1.09/8 for TriviaQA, while semantic agreement was 78.88% and 94.63%, respectively.
- Answer Trajectories Across Scales: Excess churn was positive at every observed adjacent scale transition, although the single fixed corpus path cannot establish a general dose-response relationship.Strict stable flips occurred at each adjacent transition in both NQ and TriviaQA.
6 Implications and Limitations
The paper frames retrieval-affecting releases as compatibility audits, not merely utility evaluations, while bounding its conclusions to a specific corpus path, configurations, and benchmarks.
- Implications: An index rebuild can change answer behavior even when model weights and the public API remain unchanged.The proposed audit treats old and new retrieval-augmented snapshots with compatibility scrutiny similar to model versions.
- Audit procedure: The audit freezes comparison conditions, replicates responses per question and state, compares outputs blindly, then triages flips, transitions, overlap, and utility.Positive excess churn establishes incompatibility but does not alone establish factual harm or justify blocking deployment.
- Scope: The output-only protocol could assess index refreshes, retriever replacements, chunking or deduplication changes, and external-memory changes, but these applications remain untested.The empirical treatment is limited to one fixed nested FineWeb prefix.
- Interpretation: Answer changes are not automatically regressions or harms because some changes improve answers or reflect acceptable alternatives.Deployment thresholds therefore require application-specific review.
- Threats to validity: The study does not estimate a content-averaged scaling effect because shard order is fixed and independent corpus paths are insufficient.Its results establish that this expansion can alter behavior, not a monotone or universal churn law for more corpus.
- Threats to validity: Semantic validity is bounded because exact matching can overcount stylistic changes, semantic judges can merge or split answers, and the audit lacks human validation.Repeat-stable means within-state semantic agreement across two calls, not factual correctness or user harm.
- Threats to validity: Uncertainty is conditional on realized serving interfaces and frozen cohorts, while two repeats provide only a coarse view of each conditional output distribution.Only NQ is confirmatory; TriviaQA is supportive and unpooled.
- Threats to validity: External validity is limited to two DeepSeek configurations, one search service, one FineWeb index, top-eight evidence, and two English QA benchmarks.The V4-Pro replication changes model and transport together, so it does not isolate a model effect.
7 Conclusion
The study shows that corpus expansion can alter answer-level behavior beyond repeat noise even when endpoint accuracy shifts are small or directionally inconsistent. This supports auditing compatibility alongside utility while limiting claims to the tested path and settings.
- Conclusion: The Snapshot Compatibility Audit estimates excess answer churn by subtracting within-snapshot agreement from cross-snapshot agreement.It separates corpus-associated movement from ordinary generator variation.
- Conclusion: NQ showed 6.44 and 10.25 pp of normalized-exact and semantic excess churn while EM changed by −1.50 pp; TriviaQA showed 3.00 and 2.125 pp while EM changed by +1.25 pp.The V4-Pro NQ-100 replication found 7.25 and 8.75 pp of exact and semantic excess churn while EM changed by +3.00 pp.
- Conclusion: Endpoint retrieval sets shared only 1.19 NQ and 1.09 TriviaQA documents on average, yet most answers survived turnover while a concentrated subset moved beyond repeat noise.A cross-family judge agreed with 95.71% of the original V4-Flash labels and yielded comparable churn.
- Claim boundary: The evidence does not establish a universal scaling law or document-level cause, and human semantic validation remains open.Positive churn persisted across negative, positive, and zero endpoint EM point estimates in the analyzed settings.
Ethical Considerations
The study uses public QA benchmarks and web-derived data without recruiting participants or inferring user attributes, while acknowledging possible errors, bias, and sensitive material.
- Data and participants: The study does not recruit participants or infer user attributes from its public QA benchmarks and web-derived corpus.Natural Questions contains anonymized queries.
- Data risks: Web text and model outputs may contain errors, bias, or sensitive material.The release excludes benchmark text and gold aliases, providing only permitted identifiers and outputs.
- Interpretation: Excess churn is not a proxy for harm.