PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation

Shiqi Yang, Jiekai Ma, Gaoyuan Du

arXiv:2610.04691v1cs.AI

TL;DR

RAG is commonly evaluated with clean knowledge bases, leaving its behavior under realistic corpus degradation poorly characterized. RAGStress varies corruption type and severity in a controlled benchmark that holds the retrieval pipeline fixed. It finds that factual corruption and contradiction are substantially more damaging than relevance poisoning, while clean retrieval can mask robustness differences.

  • Problem

    RAG evaluations typically assume clean corpora, leaving how RAG behaves across knowledge-base corruption types and severities poorly characterized.

    Source: §1 · p. 1

  • Method

    RAGStress is a controlled benchmark that varies knowledge-base corruption while holding the retriever and index fixed through metadata-filtered conditions.

    Source: §1 · p. 2

  • Results

    Factual corruption and contradiction cause 10–71pp drops, while relevance poisoning causes drops of ≤5pp; clean retrieval can mask robustness differences.

    Source: §6.2 · p. 7, §6.3 · pp. 7–8, §7 · p. 9

  • Takeaways & Limitations

    RAGStress is intended as a controlled stress test of robustness under knowledge-base corruption, not as a general model leaderboard.

    Source: §4.1 · p. 4

    Corruptions come from one generator, severity labels lack human validation, and benchmark-derived documents trade ecological validity for verifiable ground truth.

    Source: §5.4 · p. 6

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised. We introduce RAGStress, a controlled evaluation benchmark for stress-testing RAG systems under systematic KB corruption. The benchmark pairs four naturalistic corruption types (factual corruption, numeric typo, relevance poisoning, and contradiction injection) with three severity levels (subtle, moderate, and obvious) over a single-KB, metadata-filtered experimental design built from 57 MMLU subjects and 182,546 documents. Across 52,500 model-question-condition evaluations, RAGStress reveals that clean retrieval can mask robustness differences, semantic-fidelity corruptions are substantially more harmful than signal-utility perturbations, no-retrieval accuracy does not predict corrupted-retrieval robustness, and mixed-KB accuracy should not be treated as worst-case robustness. We document the benchmark's intended use, supported claims, and limitations, and provide an artifact bundle including generation scripts, corruption prompts, metadata schema, and evaluation code. RAGStress is intended as a controlled stress test for RAG robustness under KB corruption, not as a general model leaderboard.

1 Introduction

RAG evaluations generally assume a clean corpus, although real knowledge bases accumulate errors and RAG systems must arbitrate between retrieved context and parametric knowledge. RAGStress addresses this gap with a controlled benchmark that varies corruption type and severity while keeping the retrieval pipeline fixed.

1 Introduction

  • Current RAG benchmarks typically assume clean corpora, while real knowledge bases accumulate data-entry errors, version conflicts, OCR noise, and outdated content.How models arbitrate between parametric knowledge and retrieved context under corruption remains an open question.Source: §1 · p. 1
  • RAGStress evaluates four naturalistic corruption types at three severity levels using query-time metadata filters on a single vector index.The benchmark covers 57 MMLU subjects and 182,546 documents, and tests five systems across proprietary, reasoning-tuned, and open-weight tiers.Source: §1 · pp. 1–2
  • The design holds the retriever and index fixed so observed differences reflect knowledge-base content and model arbitration rather than index construction.Its research questions address corruption type, whether raw capability predicts robustness, and whether severity acts monotonically.Source: §1 · p. 2
  • The benchmark contributes clean, no-retrieval, and unfiltered-mixed controls alongside type–severity conditions, plus scripts, prompts, questions, and evaluation code.The evaluation split, metadata schema, document samples, and full knowledge base are pledged for DOI-archived release upon publication.Source: §1 · p. 2

2 Related Work

Prior RAG robustness work studies attacks, context conflicts, and noisy retrieval, while existing benchmarks cover different evaluation angles. RAGStress treats knowledge-base corruption type and severity as joint variables on a common controlled retrieval pipeline.

Knowledge-base poisoning and RAG security

  • Attack-oriented knowledge-base poisoning studies optimize worst-case success and commonly treat corruption as binary, unlike RAGStress’s naturalistic type-and-severity evaluation.Prior examples include attacks achieving more than 90% success and a benchmark of 13 attacks against 7 defences.Source: §2 · p. 2

Knowledge conflicts and context faithfulness

  • Context-conflict studies examine arbitration between retrieved evidence and parametric memory, while RAGStress extends this line with a broader corruption taxonomy.Prior work reports context-following or prior-stubborn behavior depending on conflict framing and distraction by irrelevant context.Source: §2 · p. 2

Robustness of RAG to noisy retrieval

  • Noisy-retrieval studies find that low-quality or irrelevant documents can degrade, preserve, or sometimes improve generation, but do not jointly vary KB-corruption type and severity.Examples include misleading retrieval pushing RAG below its no-retrieval baseline and irrelevant documents improving generation in the right positions.Source: §2 · p. 2

RAG evaluation benchmarks

  • Existing RAG benchmarks assess abilities, factual QA, or component metrics; QE-RAG varies error type and severity in queries rather than the knowledge base.None jointly varies KB-corruption type, severity, and mixed-versus-type-filtered retrieval in one controlled design.Source: §2 · p. 3

3 Problem Formulation

RAGStress formalizes corruption as controlled perturbations to retrieved documents and measures how model accuracy changes relative to clean retrieval. It distinguishes misleading changes to factual content from perturbations that reduce useful information, and defines comparisons for model vulnerability.

3 Problem Formulation

  • Semantic-fidelity corruptions alter factual claims through plausible false answers or direct contradictions, whereas signal-utility corruptions preserve truth but reduce useful information.The taxonomy predicts that semantic-fidelity changes are more harmful because a faithful reader must override misleading context, while signal-utility changes permit fallback to parametric knowledge.Source: §3 · p. 3
  • Corruption impact Δ measures accuracy loss from the clean-KB ceiling for a model, corruption type, and severity.Δ≈0 indicates little effect, while a large Δ indicates severe degradation.Source: §3 · p. 3
  • The resilience gap Γ compares two models’ corruption impact averaged over severity; a positive value means M2 is more vulnerable than M1 for that type.The comparison is independent of baseline accuracy differences.Source: §3 · p. 3
  • The framework tracks the no-retrieval rate—the fraction of queries returning zero documents—to diagnose corruption-induced retrievability shifts that could confound severity effects.Source: §3 · p. 3
  • The hypotheses predict greater harm from semantic-fidelity corruption, robustness differences not fully explained by parametric knowledge, and potentially non-monotonic severity effects.They also predict that sufficiently harmful corruption can push weak models below no-retrieval accuracy.Source: §3 · pp. 3–4
  • Figure 1 spans four corruption types, three severity levels, and five systems across 52,500 evaluations, with conditions implemented through query-time filters.Each of 700 stratified MMLU questions is evaluated under 15 conditions while the retriever and index remain fixed.Source: Figure 1 · p. 4

4 Evaluation Design

RAGSTRESS evaluates RAG under controlled knowledge-base corruption using one metadata-filtered KB. Fixed retrieval and prompting are intended to isolate effects of KB content and model behavior.

4 Evaluation Design

  • A single vector KB holds clean and four-by-three corrupted variants, while metadata filters select conditions and keep retrieval consistent across systems.The evaluation uses 700 MMLU queries, 15 filtered scenarios (k=5), five LLMs, and 52,500 total evaluations.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Intended Use and Assumptions

  • RAGSTRESS supports claims about corruption-type accuracy loss, severity effects, robustness prediction, and mixed-KB versus type-isolated worst cases—not general model ranking.It is not intended for raw factual-recall comparison, retriever evaluation, or multi-hop or agentic RAG.Source: §4.1 · p. 4
  • Exact-match MCQ scoring and fixed top-k=5 retrieval and prompting focus comparisons on KB content and model behavior, but findings may not transfer directly to open-ended generation.The design also assumes representative LLM-generated corruptions, appropriate document-level chunking, and that static single-turn evaluation is a lower bound on systems with active verification.Source: §4.1 · pp. 4–5

4.2 Single-KB, Metadata-Filter Design

  • Filtering one unified KB by metadata avoids confounds from separately constructed indexes while providing identical retrieval behavior across conditions.The design holds the retriever and index fixed so observed differences reflect KB content and model arbitration rather than index construction.Source: §4.2 · p. 5

4.3 Experimental Conditions via Metadata Filtering

  • The 15 conditions include no-retrieval and clean-only baselines, an unfiltered KB with corrupted documents outnumbering clean ones 12:1, and 12 type-by-severity conditions.In the unfiltered condition, corrupted documents account for approximately 79% of retrieved chunks.Source: §4.3 · p. 5

5 Benchmark Construction

RAGSTRESS constructs MMLU-derived reference documents, generates controlled corruptions, and ingests all versions into one metadata-enabled vector KB. Its synthetic corruptions are designed to represent production KB failure modes, with explicit limits on how far that analogy extends.

5 Benchmark Construction

  • Benchmark construction converts MMLU items into retrievable documents, generates corrupted variants, then ingests all versions with metadata into one vector KB.The resulting KB supports the experimental conditions defined through query-time metadata filtering.Source: §5 · p. 5

5.1 Dataset: MMLU as a Knowledge-Base Corpus

  • The corpus uses all 57 MMLU subjects; each item becomes a short reference-article-style document with explicit, machine-verifiable ground truth, yielding 182,546 documents.The test split spans STEM, Humanities, Social Sciences, and Other, with one clean and 12 corrupted variants per item.Source: §5.1 · p. 5

5.2 Perturbation Method

  • Claude 3 Haiku generates four corruption types at three prompt-encoded severity levels, with metadata fields used to enforce conditions at query time.The metadata records subject, perturbation type, severity, corruption status, and the source question ID.Source: §5.2 · pp. 5–6

5.3 Unified Knowledge-Base Ingestion

  • All 182,546 documents are document-level chunks embedded with a dense model and stored in one vector KB supporting metadata-filtered approximate nearest-neighbor search.Indexed metadata attributes drive the predicates that realize each condition.Source: §5.3 · p. 6

5.4 Representativeness of the Synthetic Corruptions

  • Under the neutral prompt, semantic-fidelity corruptions overturn 25–72% of clean-answerable answers, showing measurable shifts in the predicted direction.The corruption types stand in for stale or mis-entered facts, numeric entry or OCR errors, versioning drift and topic-adjacent clutter, and conflicting statements across sources; severity ranges from subtle to overt.Source: §5.4 · p. 6
  • Because corruptions come from one generator and benchmark-derived documents, RAGSTRESS supports comparative rankings, not forecasts of absolute failure rates on deployed KBs.Severity is an LLM-assigned operational label rather than a human-validated perceptual scale.Source: §5.4 · p. 6

6 Experiments and Analysis

The experiments compare five systems across 15 KB-corruption conditions using fixed top-five retrieval and exact-match MMLU scoring. Accuracy falls much more under factual corruption and contradiction than relevance poisoning, while no-retrieval accuracy does not predict corrupted-retrieval robustness.

6.1 Experiment Settings

  • The benchmark retrieves the top five chunks under metadata filters, uses a neutral prompt, and scores the first extracted A–D letter against the MMLU label.The evaluation covers 52,500 model–question–condition evaluations across 700 questions and 15 conditions.Source: §6.1 · pp. 6–7

6.3 Findings and Analysis

  • Clean retrieval narrows a 44pp capability gap to a 1.4pp accuracy band, with all five models reaching 98.6–100.0%.No-retrieval baselines range from 42.3% to 86.7%.Source: §6.3 · p. 7

6.2 Results

  • Factual corruption and contradiction cause 10–71pp drops from clean retrieval, while relevance poisoning causes drops of ≤5pp.The reported results use raw accuracy rather than a single aggregate score.Source: §6.2 · p. 7, Table 1 · p. 8

6.3 Findings and Analysis

  • A 4.5pp baseline advantage does not ensure robustness: Qwen3-32B scores 29.9% under factual corruption, while Claude Opus 4.6 scores 74.7%.The passage reports that Qwen3-32B has the higher no-retrieval baseline, 83.1% versus 78.6%.Source: §6.3 · p. 8
  • Contradiction severity is non-monotonic for four of five models; Claude Sonnet 4 reaches 41.1% at moderate severity versus 55.7% subtle and 56.1% obvious.The passage cautions that severity labels are LLM-assigned and lack human validation.Source: §6.3 · p. 8
  • Unfiltered mixed-KB accuracy stays above each model’s type-filtered worst case, so mixed-KB robustness and worst-case robustness are distinct.When clean evidence remains retrievable, models partially recover.Source: §6.3 · p. 8
  • The benchmark is limited to multiple-choice scoring and benchmark-derived KB documents; open-ended and multi-hop QA are extensions, not evaluated settings.The corruptions come from a single generator, and evaluated systems use different serving interfaces.Source: §6.3 · p. 9

7 Conclusion and Future Work

Across five models, semantic corruptions cause greater harm than numeric typos or relevance poisoning, and no-retrieval accuracy does not reliably predict corrupted-retrieval robustness. The conclusion calls for broader evaluations, while the appendices specify the benchmark’s positioning and corruption construction.

7 Conclusion and Future Work

  • Across five models, factual corruption and contradiction injection are more damaging than numeric typos or relevance poisoning, and no-retrieval accuracy does not reliably predict corrupted-retrieval robustness.Moderate contradictions can be more damaging than obvious ones; the authors argue that evaluation should test whether models reject misleading retrieved evidence.Source: §7 · p. 9
  • Future work should extend evaluation to open-ended and multi-hop settings, study retriever and chunking interactions, and test defences under the same protocol.The authors identify MMLU-style multiple choice as a boundary and propose enterprise or web documents as additional data.Source: §7 · p. 9

A Positioning Relative to Prior Work

  • RAGSTRESS is positioned as the only compared work combining a controlled corruption taxonomy with graded severity in an evaluation-design framing.The compared prior work is described as predominantly attacker-centric or binary (clean versus poisoned).Source: Table 2 · p. 12

B Corruption Type Details

  • Factual corruption replaces the correct answer, numeric typos alter its value, relevance poisoning appends off-topic text, and contradiction injection appends an incompatible answer claim.Severity varies by corruption type, including substituted-answer plausibility, numeric error magnitude, topical distance, or the contradicting claim’s assertiveness and authority.Source: B Corruption Type Details · p. 12
  • Claude 3 Haiku generates corruptions from type-specific prompts, while a neutral answering prompt leaves models’ default arbitration behaviour under evaluation.Severity is encoded in the generation instructions.Source: B Corruption Type Details · p. 12

C Document Conversion Example

  • Each MMLU item becomes a reference-style article containing the question stem, correct answer, and options, with corruptions applied to the answer field, option list, or appended text.The clean version marks the correct option with an asterisk.Source: C Document Conversion Example · pp. 12–13

D Experimental Conditions

The experimental conditions apply metadata filters to a unified knowledge base, keeping the retriever and index identical across conditions.

D Experimental Conditions

  • The design defines 15 experimental conditions, each realised by applying a metadata predicate at query time to the unified knowledge base.The caption identifies the 15 as a baseline, clean and unfiltered conditions plus 12 corruption conditions.Source: D Experimental Conditions · p. 13, Table 3 · p. 13
  • Query-time filtering keeps the retriever and index identical across conditions, so observed differences reflect knowledge-base content and model arbitration rather than index construction.Source: D Experimental Conditions · p. 13
  • The 12 corruption conditions cross four types—factual, numeric, relevance, and contradiction—with subtle, moderate, and obvious severity levels.Source: Table 3 · p. 13

F RAGSTRESS Benchmark Card

RAGSTRESS is a multi-domain multiple-choice QA benchmark for stress-testing RAG under knowledge-base corruption. It is intended for controlled robustness comparisons, not general model ranking or several other RAG evaluation tasks.

F.1 Overview

  • RAGSTRESS covers 57 MMLU subjects with 182,546 documents and evaluates multiple-choice QA across 15 conditions.The four corruption types are evaluated at three severity levels.Source: §F.1 · p. 14

F.2 Intended Use

  • The benchmark is intended to stress-test RAG under controlled knowledge-base corruption and compare model robustness profiles across corruption types and severity.It can also inform knowledge-base quality-assurance priorities.Source: §F.2 · p. 15
  • It is not intended for general model leaderboarding, retriever evaluation in isolation, open-ended generation, multi-hop or agentic RAG, or real-world attack-success measurement.Source: §F.2 · p. 15

F.3 Dataset Composition

The dataset derives from the MMLU test split and converts each item into a clean document plus corruption variants. Its collection pipeline uses prompted generation, validation, metadata tagging, and a single vector store.

F.3 Dataset Composition

  • All documents derive from the publicly available MMLU test split, with no personal, sensitive, or copyrighted content beyond the original MMLU items.Source: §F.3 · p. 15
  • Each source item produces 13 short reference-article-style documents: one clean version and 12 variants crossing four corruption types with three severity levels.Documents are approximately 100–200 tokens and carry structured metadata.Source: §F.3 · p. 15
  • Claude 3 Haiku generates corrupted documents with type-specific prompts, and the pipeline is deterministic given the same random seed and model version.Source: §F.3 · p. 15

F.4 Collection Process

  • Collection proceeds from MMLU selection and template conversion through corruption generation, format validation, metadata tagging, and ingestion into one vector store.Documents failing format validation are regenerated; the reported rejection rate is below 2%.Source: §F.4 · p. 15

F.5 Evaluation Protocol

The protocol evaluates exact-match accuracy on a fixed evaluation split across retrieval baselines and corruption conditions. The benchmark is maintained as an evaluation resource with explicit boundaries on training use and adversarial prompt release.

F.5 Evaluation Protocol

  • Exact-match accuracy on four-option answers is measured across a fixed 700-question evaluation split, stratified across 57 subjects with random seed 42.No training or validation split is provided because RAGSTRESS is an evaluation-only benchmark.Source: §F.5 · p. 15
  • Each model is evaluated with no retrieval, clean-only retrieval, unfiltered retrieval, and twelve corruption conditions spanning type and severity.Source: §F.5 · p. 16

F.6 Maintenance and Versioning

  • RAGSTRESS v1.0 may gain open-ended QA tasks, additional corruption types, or larger question sets in future versions.Source questions, generation and construction scripts, and evaluation code are publicly available; the evaluation split, metadata schema, corrupted samples, and full KB are slated for DOI archiving upon publication.Source: §F.6 · p. 16

F.7 Ethical Considerations

  • Corrupted documents are intentionally wrong and should not be used for training without the is_corrupted metadata flag; the benchmark is for evaluation, not misinformation training.Full adversarial prompts are not released, though corruption descriptions and representative examples are.Source: §F.7 · p. 16
Loading 2610.04691v1…