Source-linked AI summary
In RAG We Trust? Measuring Robustness of Retrieval-Augmented Generation Under Document Poisoning
Iliano Fasolino
TL;DR
RAG can ground answers in retrieved evidence, but document poisoning may make that evidence misleading. This paper measures the effect across corruption strategies and poisoning levels using a small quantized model, finding substantial accuracy loss, entity-swap sensitivity, and predominantly abstention-based responses. The study treats these findings as small-scale and provisional because decoding is uncontrolled and evaluation labels are coarse.
Problem
The paper examines how trustworthy RAG remains when an attacker corrupts retrieved documents that the model is instructed to trust.
Method
The study runs a 4-bit Llama 3.1 8B through three corruption strategies and four poisoning levels in a 588-run FEVER-based factorial experiment.
Results
Accuracy falls from 77.9% with clean context to 43.5% with total corruption; entity swap flips the most answers, while numeric corruption jumps when poisoned passages become a majority.
Takeaways & Limitations
The model’s dominant response to poisoning is abstention rather than inventing new falsehoods, and the lexical-overlap proxy for unsupported generation decreases under attack.
Takeaways & Limitations
The study covers one small quantized model with coarse automated labels, and decoding is not locked across clean and poisoned runs.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) grounds a language model in retrieved documents, which reduces hallucination but creates a new attack surface: if retrieved text is tampered with, the model may repeat the falsehood. We study how much a small quantized model, Llama 3.1 8B, degrades when a fraction of its retrieved context is poisoned. Three corruption strategies are tested, entity swap, number swap, and negation, each applied to zero, one, two, or three of the three retrieved passages, over a factorial sweep of 588 runs on a fact-checking task built from FEVER. Accuracy falls from 77.9% on clean context to 43.5% when all three passages are corrupted. Entity swap flips the largest share of answers that were correct on clean context. Number-based corruption stays flat while poisoned passages are a minority and jumps once they form a majority, a pattern we re-check with query-level bootstrap intervals. The model rarely invents new falsehoods; its dominant reaction is to abstain, and a lexical overlap proxy of unsupported generation falls under attack rather than rising. The study is a small-scale measurement with coarse automated labels; we treat the strategy contrasts as suggestive until decoding is controlled and stronger adjudication is in place.
I. INTRODUCTION
RAG improves access to current evidence but makes answers dependent on the integrity of retrieved documents. This study measures poisoning effects using a small quantized model and a reproducible retrieval-and-generation testbed.
- RAG retrieves relevant documents at query time and places them in the prompt as evidence for answering.
- Corrupted on-topic context can make the grounding mechanism transmit misinformation because it resembles trusted evidence.
- The study asks how accuracy changes, which corruption is most damaging, and whether failures appear as fabrication or refusal.
- The testbed combines a 4-bit Llama 3.1 8B model, MiniLM retrieval, and a FAISS index containing nearly twenty thousand FEVER evidence sentences.
- The retrieval pipeline embeds questions and passages, retrieves nearest passages from a vector index, and concatenates them into the model prompt.
- Llama 3.1 8B is run in 4-bit form on CPU, with temperature 0.1 and instructions to answer only from supplied context or abstain.
B. Poisoning Strategies
The study corrupts retrieved evidence through three targeted strategies: changing entities, changing numbers, or inserting negation. Each preserves fluent, on-topic text while altering a fact’s content or truth value.
- Document poisoning rewrites a retrieved passage into a fluent, on-topic false statement after retrieval and before prompt construction.
- Entity swap replaces named entities with plausible alternatives of the same type, such as changing Paris, France to Madrid, Spain.
- Number swap changes a numeric value to a nearby incorrect figure, targeting claims the model cannot internally verify arithmetically.
- Negation inserts a word such as “never” to reverse a statement’s truth value, providing a strong lexical signal of contradiction.
C. Poisoning Level
Poisoning level is defined by how many of the three retrieved passages are rewritten before generation. The experiment spans untouched, partially poisoned, and fully poisoned contexts.
- Poisoning level counts the number of three retrieved passages corrupted before generation, spanning 0/3 through 3/3.
- At 0/3 passages remain untouched as a within-condition baseline, while at 3/3 every passage is rewritten.
- The passages selected for corruption are random among the three, allowing a clean passage to occupy any ordering position.
D. Evaluation Metrics
The evaluation compares clean and poisoned answers from shared retrieved lists using accuracy, fooled rate, abstention, and a lexical-overlap proxy for unsupported generation. These measures are reproducible but coarse, especially for semantic faithfulness.
- Each run records clean and poisoned answers from the same retrieved passages, while decoding uses two independent generations at temperature 0.1.
- Accuracy is the case-insensitive presence of a reference keyword in the answer, making it reproducible but vulnerable to negation and paraphrase errors.
- Fooled rate measures runs where a clean answer is correct but its poisoned counterpart is incorrect, without directly measuring poison adoption.
- Unsupported generation is approximated by flagging answers with fewer than 30% of content words overlapping any retrieved passage.
- The lexical-overlap proxy can confuse faithful paraphrase with unsupported invention and miss answers that copy poisoned text.
E. Experimental Design
The experiment evaluates document poisoning across three corruption strategies and four poisoning levels in a 588-run FEVER-based fact-checking sweep. Accuracy declines with corruption, but the unpoisoned comparison cell contains residual decoding and labeling noise.
- Design: 588 runs cross 49 queries, three corruption strategies, and four poisoning levels, producing paired clean and poisoned answers.The corpus contains 19,597 FEVER dev evidence sentences plus ten added passages; each run yields two generations.
- Accuracy degradation: 77.9% clean accuracy falls to 43.5% when all three retrieved passages are corrupted.The poisoned arm is 69.4% at 0/3, 63.9% at 1/3, and 43.5% at 3/3.
- Failure behaviour: Fooled rate rises from 9.5% at 0/3 to 34.0% at 3/3, while abstention increases from 21% to 52%.The clean arm remains near 78% throughout, and strategy effects should be compared against the 0/3 residual rather than a silent zero.
- Baseline interpretation: The 0/3 poisoned cell is already about nine accuracy points below the clean arm before any passage is rewritten.This residual reflects decoding noise and strict keyword matching, not retrieval mismatch.
C. Uncertainty on the Existing Runs
The reanalysis quantifies uncertainty in the existing runs with query-level bootstrap intervals and treats effect sizes and intervals as primary. The poisoning-level climb persists, but neighbouring levels overlap.
- Bootstrap analysis: 10,000 query-level bootstrap resamples of 49 indexed slots produce percentile 95% intervals for the fooled rate.All runs tied to each drawn slot are retained, with a fixed seed; exact McNemar tests are secondary.
- Uncertainty results: The poisoning-level increase remains after query resampling, while neighbouring levels have overlapping intervals.The comparison therefore supports a broad climb with poisoning rather than uniformly separated adjacent levels.
- Strategy aggregation: Table III averages strategy behaviour over four poisoning levels, with 196 runs aggregated per strategy.This aggregation supplies the basis for comparing strategies across the full sweep.
D. Which Strategy Hurts Most
Entity swap is the most damaging strategy overall, while number swap shows a distinct majority-leaning pattern as poisoned passages become more prevalent. The strategy contrasts are clearest at higher poisoning levels.
- Entity swap: 23.5% fooled rate makes entity swap the highest-fooling strategy and the one with the lowest poisoned accuracy.It flips the largest share of answers that were correct on clean context.
- Entity swap: 43.4% abstention makes entity swap the most refusal-inducing strategy.Contradictions such as replacing Paris, France with Madrid, Spain can prompt the model to refuse.
- Across-level pattern: Fooled rates rise across poisoning levels for every strategy, while entity swap and negation increase in roughly even steps.Number swap is nearly flat in the first two columns before its later jump.
- Number swap: Number swap stays at 10.2% fooled for zero and one corrupted passage, then rises to 26.5% at two and 34.7% at three.The one-to-two jump is 16.3 points with bootstrap 95% CI [2.0, 30.6], which excludes zero but remains wide.
F. How the Model Fails
Under poisoning, the model more often abstains than accepts falsehoods, while unsupported-generation proxy scores decrease rather than increase. Entity swap produces the sharpest degradation, whereas number swap remains stable at low poisoning before a majority of corrupted passages triggers a jump.
- 35.2% of poisoned queries ended in abstention, exceeding the 21.1% fooled rate across runs.Abstention rises with poisoning and reaches 69% for entity swap at full corruption.
- Entity swap drives the steepest accuracy decline, while every strategy becomes harmful once two of three passages are corrupted.
- Number swap stays nearly flat through one poisoned passage, then rises when corrupted passages become the majority.
- The unsupported-generation proxy is higher on clean context than poisoned context, indicating more refusal-like behavior under attack.The proxy is near 29% on clean context and near 20% on poisoned context.
- Among queries correct on clean context, lost correct answers are absorbed mainly by abstention, and negation retains the most correct answers.
G. Which Queries Break
Query vulnerability varies with the fact type and passage content: biographical and landmark questions are especially fragile, while entity-rich evidence creates more opportunities for misleading substitutions. Across failure behavior, abstention rises under poisoning and unsupported-generation proxy scores do not surge.
- Which queries break: Biographical and landmark questions dominate the most fragile queries because their passages contain many swappable entities.
- Which queries break: Yes/no and location questions are more exposed than open numeric questions, whose lower clean accuracy leaves less room for correct-to-wrong flips.
- Relation to prior work: On clean corpora retrieval improves accuracy, but corrupted on-topic context becomes a liability because it resembles trusted evidence.
- Failure behavior: Abstention rises with poisoning, while the lexical-overlap proxy of unsupported generation is higher on clean than poisoned context.
IV. THREAT MODEL AND DISCUSSION
The threat model lets an attacker alter already retrieved passages without changing retrieval or retraining the model. The discussion argues that numeric checks, source diversity, and monitored abstention are practical responses, while vulnerability also depends on which questions and passages are involved.
- Threat model: The attacker edits retrieved passage text after retrieval and before generation, without retraining the model or changing the retriever.The attack can target a compromised store, poisoned web source, or tampered cache.
- Discussion: Numeric facts deserve special protection because corruption becomes markedly more harmful once it dominates the context in this sweep.
- Discussion: Two trustworthy passages often outvote one bad numeric claim, but that protection weakens when corrupted passages become the majority.
- Discussion: Increasing retrieval count without increasing source diversity offers little real protection once corrupted passages dominate.
- Discussion: A sharp rise in abstention can serve as a cheap early warning of a poisoned document store.
V. CONCLUSION
The study finds that poisoning retrieved context substantially degrades a small RAG model, while its dominant response is abstention rather than unsupported generation. These findings remain bounded by the study’s small scale, coarse labels, and uncontrolled decoding.
- 77.9% to 43.5%: accuracy falls as all three retrieved passages change from clean to corrupted.The sweep covers 588 runs across three corruption strategies and four poisoning levels.
- Entity swap flips the most answers that were correct on clean context.
- Numeric corruption shows a majority-leaning jump when two of three passages are poisoned, although its bootstrap interval is wide.The interval excludes zero, but the estimate remains imprecise.
- The model mainly abstains under attack, while the lexical-overlap proxy of unsupported generation falls rather than rises.
- The absolute results are bounded by one small quantized model, a compact query set, rule-based corruption, coarse automated labels, and decoding that is not locked.The paper proposes controlled decoding, stronger adjudication, broader models and corpora, and more varied adversarial corruptions as next steps.