PaperlayerView exact revision on arXiv

Source-linked AI summary

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

Shunyuan Zhou, Hao Chen, Tianyu Wang, Goose Lin, Zaiyuan Wang, Haiying Zhao

arXiv:2610.11183v1cs.CLcs.AI

TL;DR

RAG-Stress addresses how to evaluate cases where relevant, coherent retrieved evidence can replace a correct answer with an incompatible one, a behavior aggregate accuracy combines with preexisting errors. It crosses source-priority instructions and answer-span positions in a controlled evidence edit, finding higher misleading rates under strict instructions and a consistent aggregate position pattern.

  • Problem

    Aggregate accuracy combines preexisting errors with cases where retrieved evidence replaces a correct closed-book answer with a designated foil.

    Source: §1 · p. 1

  • Method

    RAG-Stress edits one assertion to support a designated foil and crosses Soft/Strict instructions with beginning, middle, and end answer-span positions.

    Source: §1 · pp. 1–2

  • Results

    Strict produces higher misleading rates than Soft across systems and datasets, and mean rates follow End > Beginning > Middle under both instructions.

    Source: §1 · p. 2, §4.3 · p. 5

  • Takeaways & Limitations

    The diagnostic distinguishes evidence-policy compliance from factual reliability, without identifying an internal attention mechanism or a superior mitigation.

    Source: §5 · p. 9

    The cross-dataset difference does not establish a causal multi-hop effect, and filtering changes item composition.

    Source: §4.3 · p. 6, §4.4.4 · p. 8

Abstract

from arXiv · show

Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly. Standard accuracy measures obscure this behavior by combining answer replacement with preexisting errors. We introduce RAG-Stress, a controlled diagnostic protocol for examining the limits of evidence reliance in retrieval-augmented generation. The protocol holds the question and reference answer fixed, edits one assertion to support a designated incorrect answer, and crosses two source priority policies with three positions of the answer span within the evidence text. We measure misleading rate (MR) on each model's subset of questions answered correctly without retrieval, alongside clean accuracy on the full evaluation set. We evaluate fifteen systems spanning API models, open models, and search agents trained with reinforcement learning on TriviaQA-RC, HotpotQA, and SearchQA, with additional English and Chinese MedQA evaluations. Instructions that prioritize documents consistently produce higher MR than those permitting reliance on prior knowledge. Averaged over models and positions, the gap ranges from 10.9 to 13.5 percentage points across the three QA datasets. Mean MR follows End $>$ Beginning $>$ Middle under both policies, although individual models do not uniformly follow this ordering. A separate paired audit of 500 questions and two checkpoints supports increased harmful override without establishing a corresponding improvement in beneficial correction. These findings distinguish evidence adherence from factual reliability and motivate evaluating whether retrieved evidence preserves, replaces, or corrects a model's answers.

1 Introduction

RAG-Stress asks when relevant, coherent retrieved evidence and source-priority instructions lead a model to replace a correct closed-book answer with a designated foil. It separates that transition from preexisting errors and reports higher misleading rates under strict instructions, with position-dependent rates.

Motivation and measurement

  • RAG-Stress measures designated foil adoption only when the model answered correctly without retrieval, separating answer replacement from preexisting failure.Clean-evidence responses provide a separate control; the measure is a behavioral transition, not stable parametric knowledge or, by itself, the causal effect of editing.Source: §1 · pp. 1–2

Intervention

  • The intervention crosses Soft and Strict instructions with beginning, middle, and end positions for the answer span in the target text.It asks how often policy-driven compliance replaces a correct baseline answer, rather than measuring compliance alone.Source: §1 · p. 2

Illustration

  • The opening example holds the question and evidence edit fixed: strict prompting adopts the foil, while soft prompting retains the reference.Both clean-evidence answers and the closed-book answer match the reference.Source: Figure 1 · p. 2

Results

  • Strict instructions produce higher misleading rates than Soft across systems and datasets, while mean rates follow End > Beginning > Middle under both instructions.The separate audit supports increased harmful override but does not establish a corresponding gain in reference recovery.Source: §1 · p. 2

Contributions

  • The paper contributes a diagnostic protocol and crossed evaluation, not a new conflict phenomenon or a mitigation method.Its paired audit separates harmful override, reference recovery, and transitions from clean to edited evidence.Source: §1 · p. 2

2 Related Work

Prior work examines evidence reliance, context conflict, RAG evaluation, and adversarial evidence or memory. RAG-Stress complements these lines by measuring whether a baseline-correct response changes to a designated foil.

Knowledge conflict and context reliance

  • Research on knowledge conflict and context reliance already covers counterfactual evidence, source instructions, context-aware decoding, and reliability methods; instruction effects on conflict resolution are not themselves new.RAG-Stress combines a crossed intervention with a fixed baseline-correct stratum and designated-foil adoption measurement.Source: §2 · p. 2

RAG evaluation and robustness

  • RAG evaluation includes retrieval relevance, faithfulness, answer quality, hallucination annotation, counterfactual robustness, grounding, conversation, coverage, and multi-hop conflict.RAG-Stress asks whether a baseline-correct response changes to a designated foil, a transition not identified by aggregate accuracy or grounding alone.Source: §2 · pp. 2–3

Adversarial evidence and memory

  • Counterfactual-query and memory-attack studies examine other settings; RAG-Stress keeps the query, index, and memory fixed while measuring response to one edited assertion.Source: §2 · p. 3

3 RAG-Stress Methodology

RAG-Stress crosses evidence condition, source-priority instruction, and answer-span position while separating baseline-correct questions from preexisting failures. It reports clean accuracy on the full set and misleading rate on the baseline-correct subset, with paired contrasts and question-level uncertainty estimates.

3 RAG-Stress Methodology

  • The overview crosses evidence condition, instruction, and Word Position; the figure shows original- and edited-evidence results for five API models on TriviaQA-RC in six instruction-position panels.Source: Figure 2 · p. 3, §3 · p. 3

3.1 Same-Query Evidence Intervention

  • Each question uses a clean passage supporting the reference and an edited passage that changes only the answer-containing assertion to support a same-type entity foil or a multiple-choice distractor.Soft/Strict instructions cross beginning, middle, and end answer-span positions; clean accuracy uses all prepared questions, while MR uses baseline-correct questions.Source: §3.1 · p. 3

3.2 Baseline Strata and Evaluation Metrics

  • Closed-book responses fix each model’s baseline-correct and baseline-failure sets; clean accuracy uses all questions, while misleading rate is foil matching without reference matching on edited evidence.MR is not the clean-to-edited accuracy drop, and direct transitions require paired clean/edited outputs.Source: §3.2 · p. 4

3.3 Paired Contrasts and Uncertainty

  • Instruction contrasts compare Strict with Soft after averaging positions within question, while position contrasts pair questions and instructions.The supplementary audit uses 20,000 question-level bootstrap resamples for pointwise 95% intervals; QA position checks use 2,000, and intervals are not multiplicity-adjusted.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate RAG-Stress across QA datasets, systems, and robustness checks, separating full-set clean accuracy from misleading-rate measurements on each model’s closed-book-correct subset. Strict source-priority instructions show higher misleading rates than Soft instructions, while robustness checks qualify the position and cross-dataset patterns.

4.1 Datasets and Experimental Setup; 4.2 Baselines and Evaluation

  • The evaluation covers seven API systems, four open models, and four RL-trained search agents on TriviaQA-RC, HotpotQA, and SearchQA, with fixed evidence and live search disabled.Clean accuracy uses the full prepared set; edited-evidence MR is conditioned on each model’s closed-book-correct subset.Source: §4.1 · p. 4, §4.2 · p. 4

4.3 Results and Behavioral Interpretation

  • Across reported cells, Strict MR exceeds Soft MR; MR is measured on baseline-correct answers, separately from clean accuracy on the full set.The comparisons hold question, evidence, and placement fixed while changing the source-priority policy.Source: §4.3 · p. 5
  • Mean MR follows End > Beginning > Middle under both instructions on all three QA datasets, but this aggregate ordering is not universal across models.HotpotQA has higher average MR than TriviaQA-RC and SearchQA, but the comparison does not isolate a multi-hop effect.Source: §4.3 · pp. 5–6

4.4.1 Paired Validation of Word Position Effects

  • In paired validation, Strict End MR exceeds Middle by 10.9 and 9.6 percentage points for models A and B, while Beginning-minus-Middle and Soft contrasts remain unresolved.The check supports higher End MR under Strict, not the complete ordering or an instruction–position interaction.Source: §4.4.1 · p. 6, Table 2 · p. 6

4.4.2 Robustness to Paraphrases of Source Policy

  • Strict-minus-Soft MR gaps remain positive across tested source-policy paraphrases, while changes from the original wording are unresolved.The tested versions support directionally consistent effects, not equivalence of effect sizes or invariance to arbitrary wording.Source: §4.4.2 · p. 6, Table 3 · p. 7

4.4.3 Cross-Lingual Generalization on Medical QA

  • Strict MR exceeds Soft MR for all 15 systems in both English and Chinese MedQA evaluations, with similar group-averaged instruction gaps across the two languages.The language sets contain different questions, so the comparison cannot isolate language proficiency from item composition.Source: §4.4.3 · p. 7

4.4.4 Counterfactual Evidence and Matcher Validation

  • Human validation finds that 88.1% of edited passages pass all criteria, and filtering them preserves the positive instruction-effect direction.Relation preservation is the main bottleneck for HotpotQA and Chinese MedQA; SearchQA has the lowest fluency, and filtering changes item composition.Source: §4.4.4 · p. 8, Table 5 · p. 8
  • On a label-stratified matcher audit, human agreement is 94.2% and Foil-only precision is 95.3%; stratification prevents interpreting these as population-wide agreement.The 26 disagreements mainly involve aliases, formatting, hedging, or multiple entities.Source: §4.4.4 · p. 8, Table 6 · p. 8

5 Conclusion

RAG-Stress isolates adoption of an evidence-supported foil after a correct closed-book answer, separating it from baseline failure. Its findings distinguish source-policy compliance from factual reliability without identifying internal mechanisms or a superior mitigation.

5 Conclusion

  • 10.9–13.5 pp is the reported mean Strict–Soft misleading-rate gap across three QA datasets; mean rates follow End > Beginning > Middle under both policies.The conclusion also notes that medical QA and decoding checks extend the instruction pattern beyond one setting.Source: §5 · p. 9
  • The evaluation lesson is to report baseline coverage, foil adoption, and matched clean/edited transitions separately rather than infer robustness from aggregate accuracy or document adherence.The paper frames RAG-Stress as diagnostic resolution, not a new conflict phenomenon or mitigation method.Source: §5 · p. 9

A.3 Evaluation Pseudocode

A.2 Structured Prompt Configuration

  • Strict prioritizes supplied documents, while Soft permits prior knowledge to override conflicting evidence; the two policies are mutually exclusive prompt clauses.The surrounding role, task, answer-selection clauses, output contract, wrapper, and generation settings remain identical across policies.Source: §A.1 · p. 13, §A.2 · pp. 14–15

A.1 Prompt Design and Templates

  • The protocol fixes the question and reference, edits one answer-bearing assertion to support a foil, and varies instruction policy and answer-span position.The appendix distinguishes the main within-text position intervention from the separate audit’s passage-slot setup.Source: A Prompt Templates and Evaluation… · p. 13, §A.1 · p. 14

A.4 Scoring and Aggregation Details

  • Foil-only matches count toward misleading rate, while responses matching both foil and reference do not; audit contrasts are bootstrapped by question within each stratum.Audit intervals do not apply to the fifteen-system main tables, and clean accuracy, misleading rate, harmful override, and beneficial correction have different denominators.Source: §A.4 · pp. 15–16

B.1 Instruction Dependence in the Separate Audit

  • In the separate 500-item audit, Strict increased harmful override for both checkpoints, while the paired data did not establish a gain in beneficial correction.This audit moves whole passages among document slots, not answer spans within evidence text; its outputs do not identify attention or a recency mechanism.Source: B Supplementary Paired Audit · p. 16, §B.1 · p. 17

B.2 Case Study: When the Instruction Changes the Answer

  • The selected case study holds the edited passage and other input fixed and illustrates Strict prompting changing an otherwise correct answer to the foil.The case was selected post hoc and does not estimate prevalence or certify semantic validity.Source: §B.2 · p. 17

C Metrics for the Paired Audit and Sensitivity Analyses

The paired-audit analyses use a 500-item supplementary cohort, and their reported intervals are pointwise rather than multiplicity-adjusted.

C Metrics for the Paired Audit and Sensitivity Analyses

  • The analyses cover only 500 supplementary items; their pointwise 95% intervals are unadjusted and do not provide simultaneous coverage across endpoints or models.The reported position is document slots 1, 2, and 4, not within-text Word Position.Source: C Metrics for the Paired Audit and… · p. 17

C.1 Audit Rates and Denominators

  • Table 8 reports prompt-specific audit rates and paired contrasts with 95% bootstrap intervals.The rates average three positions, and HOR and BCR use different baseline strata.Source: Table 8 · p. 17, §C.1 · p. 17
  • The audit’s HOR and BCR contrasts use different strata, so they cannot be subtracted as a net-harm score.K denotes baseline-exact-correct items and U the remaining items.Source: §C.1 · p. 17

C.2 Bootstrap Contrasts over Questions for the Prospective Audit

The prospective audit separates instruction contrasts, position interactions, and paired response transitions. Its primary contrasts support increased harmful override under Strict, while the post-hoc position interaction remains unresolved.

C.2 Bootstrap Contrasts over Questions for the Prospective Audit

  • Strict–soft harmful-override intervals exclude zero for both checkpoints, while corresponding beneficial-correction intervals include zero.These are question-clustered, pointwise intervals without multiple-comparison adjustment; the difference in interval significance does not test whether the two effects differ.Source: §C.2 · p. 18
  • The audit averages each item’s outcomes across three positions within each prompt condition before bootstrapping differences within items for 20,000 repetitions.This avoids treating repeated condition cells as independent questions.Source: §C.2 · p. 18
  • Position-interaction intervals include zero for Llama (−2.4 [−7.4, 2.7]) and Qwen (3.5 [−1.9, 8.9]), so strict amplification is not established.The analysis was post hoc and exploratory, and neither interval establishes a zero interaction.Source: §C.2 · p. 18
  • Post-hoc common-cohort gold-score analysis shows larger strict-versus-soft losses in the edited arm than the clean arm for both models.The comparison uses normalized exact gold matches across all 500 prepared questions and is unadjusted for multiplicity.Source: §C.2 · p. 18, Table 9 · p. 18

C.3 Direct Paired Transitions

  • Direct clean-gold-to-edited-foil transitions occurred in 1123/2028 (55.4%) Llama pairs and 872/1554 (56.1%) Qwen pairs.These descriptive paired counts do not establish semantic edit validity or factual error.Source: §C.3 · p. 18

C.4 Post hoc Control with Clean Evidence

A matched clean-versus-edited control compares designated-foil outputs on the same baseline-correct questions. Clean evidence yields almost no foil matches, while edited evidence yields many more.

C.4 Post hoc Control with Clean Evidence

  • The exploratory control replays clean and edited evidence on the baseline-exact-correct questions, with all three passage positions under each prompt family.It checks designated-foil adoption in the clean arm and whether adoption increases in the edited arm; it does not adjudicate factual validity.Source: §C.4 · p. 19
  • Clean evidence produces almost no designated-foil matches, whereas the edited arm produces many more.Table 10 reports exact foil-only rates across three positions and question-level bootstrap intervals; it is not a human semantic-validity rate.Source: §C.4 · p. 19, Table 10 · p. 19
  • The paired pattern supports the interpretation that the intervention changes the evidence signal rather than merely exposing a preexisting tendency to emit the foil.The comparison is exploratory and does not establish that the edits are factually misleading.Source: §C.4 · p. 19

C.5 Cross-Model Common-Stratum Check

The common-stratum check repeats strict–soft contrasts on shared baseline strata, while the supplementary audit documents its sampling, construction, and reproduction boundaries. The check preserves the positive harmful-override contrast direction but does not establish cross-model generalization.

C.5 Cross-Model Common-Stratum Check

  • On the shared baseline-correct cohort, the harmful-override prompt contrast remains positive; both beneficial-correction intervals include zero on the shared baseline-wrong cohort.The common strata contain 232 baseline-correct and 135 baseline-wrong questions.Source: §C.5 · p. 19, Table 11 · p. 19

D Supplementary Paired Audit: Execution Record

  • The audit draws from a historically Llama-selected 1,000-item TriviaQA pool, and the inherited selection limits generalization.Of 788 processed candidates, 500 lexically eligible pairs were retained before inference; 212 IDs remained unprocessed.Source: D Supplementary Paired Audit:… · p. 19
  • The audit varies target document slots 1, 2, or 4, not answer-span Word Position within a document.It reuses released snippet pairs, and lexical checks do not certify semantic validity.Source: D Supplementary Paired Audit:… · pp. 19–20
  • The realized inputs do not constitute a long-context evaluation.The logged setup used four-bit quantization, a 20-token output cap, and a 1,536-token input ceiling.Source: D Supplementary Paired Audit:… · p. 20
  • Saved outputs support score reconstruction, but missing Qwen code hashes and unrecorded prompt token IDs and stop reasons leave inference serialization uncertified.Rescoring is distinct from regenerating outputs or independently adjudicating edits.Source: D Supplementary Paired Audit:… · p. 20

E Complete Cross-Dataset Experimental Results

Tables 13 and 14 report HotpotQA and SearchQA results from the main evaluation of fifteen systems. Their clean-accuracy and misleading-rate measures use different denominators.

E Complete Cross-Dataset Experimental Results

F Additional Position and Prompt Robustness Protocols

This appendix specifies a reproducible protocol for testing source-policy and answer-position effects. It pairs questions across prompt variants and reports instruction contrasts, while noting limits on what those comparisons establish.

F Additional Position and Prompt Robustness Protocols

  • The appendix reports supplementary checks on 300 QA questions per model, with |KA| = 220 and |KB| = 198.These checks complement the main evaluation and the 500-item audit.Source: F Additional Position and Prompt… · p. 22

F.2 Source Policy Paraphrases and Execution

  • Strict prioritizes supplied documents despite apparent errors, whereas Soft permits prior knowledge to override conflicting documents.The policy clause is inserted at the same system-prompt location while other prompt components and evidence are held fixed.Source: §F.2 · pp. 22–23
  • The protocol crosses three prompt versions, two source policies, three answer-span positions, and two evidence arms, yielding 36 requests per question and model.The original outputs can be reused only with identical requests, model versions, and settings; the two added versions require 24 new requests.Source: §F.2 · p. 23
  • Reproduction requires preserving variant IDs, full messages, context and model versions, generation settings, raw responses, matcher versions, and failure records.Resource-limited subsets must be selected before inference; the design includes neither live retrieval nor model updates.Source: §F.2 · p. 23
  • Prompt-version comparisons pair questions across versions, policies, and positions.The passage defines a version-specific foil-only indicator, with v = 0 denoting the original prompt.Source: §F.2 · p. 23
  • All six instruction-contrast intervals exclude zero, while all four changes from the original prompt include zero; this supports direction, not equivalence.The resample count and seed for these contrasts are unspecified.Source: §F.2 · p. 23

F.1 Paired Word Position Analysis

The paired position analysis compares foil adoption across answer-span placements on a fixed question population. Its individual-model intervals support only selected contrasts, with uncertainty and estimand boundaries that limit interpretation.

F.1 Paired Word Position Analysis

  • Paired analysis requires a fixed question population and complete clean/edited instruction–position records; changing denominators by position changes the estimand.Comparisons also require fixed document order and verified realized answer-span locations.Source: §F.1 · p. 22
  • Position contrasts are calculated within question before resampling; subtracting interval endpoints does not produce the interval for a contrast.The supplied results use 2,000 paired bootstrap resamples, but the seed, interval method, and multiplicity adjustment are unspecified.Source: §F.1 · p. 22
  • Under Strict, End-minus-Middle intervals are positive and Beginning-minus-End intervals negative for both models; other listed contrasts remain unresolved.These individual-model results do not establish the complete position ordering or an instruction–position interaction.Source: §F.1 · p. 22
  • The intervals are interpreted individually, not as simultaneous confirmation, and nonzero contrasts do not identify an attention mechanism.The analyses followed inspection of the main summaries and are not claimed as preregistered.Source: §F.1 · p. 22

Related papers

Loading 2610.11183v1…