PaperlayerView exact revision on arXiv

Source-linked AI summary

ToxicRAG: Compromising Retrieval-Augmented Generation Systems via Single-Shot Knowledge Poisoning Attacks

Haozhe Lu, Jiaqi Li, Xinyuan Zhu, Xiang Li

arXiv:2609.11082v1cs.CR

TL;DR

RAG’s reliance on external evidence creates a knowledge-poisoning risk when untrusted documents enter the retrieval corpus. ToxicRAG addresses the one-document-per-target setting with a fabricated knowledge-update narrative and surrogate self-validation. In sampled-corpus experiments, it matches or exceeds the strongest evaluated baseline across all dataset–model combinations, with ASRs spanning 0.61–0.91.

  • Problem

    RAG can improve factual grounding but does not guarantee correct answers, and its external retrieval corpus can let untrusted documents influence generation.

  • Method

    ToxicRAG constructs one poisoned document per target as a knowledge-update narrative and optionally revises it through answer-focused self-validation with a surrogate language model.

  • Results

    ToxicRAG matches or exceeds the strongest evaluated baseline in every dataset–model combination, with ASRs ranging from 0.61–0.91 in sampled-corpus experiments.

  • Takeaways & Limitations

    Narrative-form poisoned documents remain influential under the evaluated RAG configurations, motivating further study of factual consistency and source provenance.

  • Takeaways & Limitations

    The study is limited to sampled corpora, 100 questions per dataset, four 4B–8B victim models, dense retrieval, and a simple context-grounded QA prompt.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) can ground large language model (LLM) outputs in external evidence, but it also exposes the system to knowledge poisoning. Representative attacks use multiple injected documents or templates that directly assert a target answer. We present ToxicRAG, a one-document-per-target attack that expresses misinformation as a coherent knowledge-update narrative. The generated document first acknowledges the previously accepted answer, introduces fabricated events that appear to invalidate it, and then attributes the attacker-selected answer to a set of purported authorities. An answer-focused self-validation loop optionally revises a candidate when a surrogate language model does not reproduce the target answer. We evaluate the attack on 100 target questions from each of Natural Questions, HotpotQA, and MS-MARCO, using four victim LLMs and four dense retrievers. In the sampled-corpus setting reported in this paper, ToxicRAG obtains ASRs between 0.61 and 0.91 across the twelve dataset--model combinations. It matches or exceeds the strongest evaluated baseline in every combination, with margins ranging from 0 to 11 percentage points. These results show that narrative-form poisoned documents can remain influential under the evaluated RAG configurations and motivate further study of factual consistency and source provenance in RAG systems.

1 Introduction

RAG improves factual grounding by conditioning LLM generation on retrieved external documents, but that retrieval boundary also enables one-document knowledge poisoning. ToxicRAG addresses this constrained threat with a narrative update and reports strong attack success across evaluated datasets and models.

  • Motivation: The attack targets a one-document-per-target setting because multiple insertions consume a larger write budget and may be unavailable.The setting treats the smaller write budget as a distinct threat model.
  • Motivation: A poisoned document must be retrievable, influential enough to change generation, and natural enough to resemble ordinary corpus content.The paper evaluates end-to-end ASR, retrieval position, and sensitivity to larger top-k contexts.
  • Our Approach: ToxicRAG represents poisoned misinformation as a knowledge update that replaces an accepted answer through fabricated events and purported authorities.It optionally revises candidates using answer-focused self-validation with a surrogate language model.
  • Results: ASRs range from 0.86–0.91 on NQ, 0.74–0.80 on HotpotQA, and 0.61–0.63 on MS-MARCO, matching or exceeding the strongest baseline in every dataset–model cell.Margins range from 0 to 11 percentage points in the sampled-corpus experiments.
  • Results: The evaluation covers 100 target questions from each of Natural Questions, HotpotQA, and MS-MARCO with four victim LLMs.The broader evaluation also includes four dense retrievers and multiple retrieval-context and construction settings.

2 Background and Related Work

RAG retrieves external documents and conditions answer generation on that context, creating both a route to current evidence and a route for untrusted documents to influence answers. Prior poisoning work spans multi-document attacks, single-document semantic attacks, and defenses operating at multiple pipeline stages.

  • RAG Systems: A standard RAG system retrieves the top-k documents for a query, then generates an answer from the query and retrieved context.The retrieval corpus contains indexed documents that may come from sources such as Wikipedia, news articles, and financial reports.
  • RAG Systems: External documents let RAG incorporate current evidence but also allow an untrusted document to influence the answer.This dependence creates the security boundary exploited by knowledge poisoning.
  • Related Work: Prompt injection redirects applications through instructions, whereas ToxicRAG places declarative misinformation in the corpus to alter factual answers.The paper distinguishes this objective from jailbreaks, which seek to bypass safety refusal.
  • Related Work: Multi-document poisoning increases the probability of top-k exposure but consumes a larger write budget than single-document settings.PoisonedRAG is described as using multiple injected documents for a target query.
  • Related Work: Single-document methods use semantic strategies including outdated-versus-current framing, evidence chains, authority cues, and covert poisoning forms.ToxicRAG is positioned in this semantic-poisoning line.
  • Related Work: Existing defenses intervene through input filtering, query rewriting, expanded retrieval, consistency checks, provenance tracing, or isolate-then-aggregate designs.The cited examples include RobustRAG and RAGuard.

3 Problem Formulation

The paper formulates knowledge poisoning as making a victim RAG answer adopt an attacker-selected claim after one document is added per target. The threat model limits attacker access to controlled corpus writes while treating victim components as black boxes.

  • 3.1 Knowledge-Poisoning Objective: The objective is to maximize the probability that the victim answer adopts the attacker-selected target claim after corpus injection.Success uses the ASR protocol’s semantic judgment rather than exact string equality.
  • 3.1 Knowledge-Poisoning Objective: Each target question receives one constructed poisoned document, creating the stated injection-budget constraint.The injected corpus is formed by adding the target documents to the benign retrieval corpus.
  • 3.1 Knowledge-Poisoning Objective: Successful end-to-end poisoning requires both retrieval of relevant poisoned content and generation-side adoption of the target claim.The current retrieval implementation counts hits on any injected document rather than target-specific identifiers.
  • 3.2 Threat Model: The attacker selects target questions and answers, with each selected answer fixed for document construction and evaluation.Answers may be supplied directly or generated before construction.
  • 3.2 Threat Model: The attacker knows target questions and answers but does not use victim parameters, architecture, prompts, outputs, retriever parameters, or embedding-model identity.The evaluated setting is black-box with respect to victim RAG components, not knowledge-free.
  • 3.2 Threat Model: The attacker can append one document per target but cannot delete benign documents or change the victim retriever, generator, or inference-time query.Whether a deployment grants this corpus-write access is outside the evaluation’s scope.

4 Design of ToxicRAG

ToxicRAG constructs one poisoned document as an apparent knowledge update, combining a causal transition from an accepted answer with purported authority consensus. A surrogate language model validates whether the candidate elicits the target answer, while retrieval success is evaluated separately.

  • The evaluated configuration restricts each method to one injected document per target and records victim access and retriever knowledge during construction.Table 1 describes the compared attack configurations and notes that they may differ from original-paper budgets.
  • ToxicRAG combines an accepted answer, fabricated invalidating events, and purported agreement among authorities into one narrative schema.This distinguishes it from approaches that separately use answer contrasts or authority cues.
  • Answer-focused self-validation tests whether a surrogate language model reproduces the target answer and revises the document when it does not.Validation stops after success or the configured revision limit.
  • The construction pipeline uses a reference answer as the narrative’s previously accepted fact, while retrieval success is measured separately after corpus insertion.The reference answer is a semantic anchor rather than a guarantee that the poisoned document will be retrieved.
  • The attack fixes a target answer, generates causal events and authority lists, and combines them into a final poisoned document.The target may be attacker-supplied or generated as an incorrect but plausible answer.

5 Experiments

Experiments evaluate ToxicRAG and baselines in sampled-corpus RAG settings using three QA benchmarks, four victim LLMs, dense retrieval, and one injected document per target. ToxicRAG generally achieves stronger end-to-end ASR, while sensitivity analyses show that retrieval size and corpus size affect attacks differently.

  • RAG Setup: The main experiments use 100 sampled questions per benchmark, retain qrels-associated clean passages, and add one poisoned document per target to a shared FAISS index.Unless otherwise stated, retrieval uses all-MiniLM-L6-v2 with k = 5.
  • Baselines: The comparison matches methods on one injected document per target but does not fully equalize document length or attack-LLM calls.The paper treats these differences as a limitation of the comparison.
  • Main Results: ToxicRAG has the highest reported ASR in eleven of twelve dataset–model cells and ties CorruptRAG-AK in the remaining cell.The margin over the strongest baseline is 5–9 percentage points on NQ, 7–11 on HotpotQA, and 0–4 on MS-MARCO.
  • Main Results: Higher any-poison retrieval concentration does not necessarily imply higher end-to-end ASR in these experiments.PoisonedRAG and CorruptRAG-AS often have higher any-poison Top-1 and PD@k values, while ToxicRAG has higher ASR in most reported cells.
  • Sensitivity Analysis: ASR changes differently across attacks as retrieval size varies from k = 5 to 100, based on single-run sensitivity experiments without uncertainty intervals.The analysis does not by itself identify an attention-based mechanism, and context truncation at large k may contribute.

6 Ethics Statement and Responsible Release

The experiments used isolated research RAG instances rather than public or production systems, and the release plan emphasizes synthetic labeling, restricted distribution, and non-disclosure of operational access details.

  • All poisoning experiments targeted research RAG instances built from public benchmarks and local vector indexes, not public search engines, production knowledge bases, commercial deployments, or third-party accounts.
  • Released qualitative examples will be labeled as synthetic attack content and accompanied by correct answers and experimental context to reduce misinterpretation.
  • The authors will not publish credentials, real-deployment write-access procedures, or an automated interface for inserting poisoned documents into external services.
  • Responsible-disclosure coordination with maintainers is planned but not yet completed.

7 Conclusion

ToxicRAG is a one-document-per-target RAG poisoning attack that frames a selected answer as a knowledge update and validates candidate documents before insertion. Across sampled-corpus experiments, it achieved the highest reported ASR in eleven of twelve dataset–model combinations, while the study limits conclusions to its evaluated configurations.

  • ToxicRAG presents an attacker-selected answer as a knowledge update by replacing an accepted answer through a fabricated transition and purported authorities.
  • An answer-focused self-validation step optionally revises the poisoned document before insertion into the retrieval corpus.
  • ToxicRAG achieved the highest reported ASR in 11 of 12 sampled-corpus dataset–model combinations, with smaller gains and one tie on MS-MARCO.
  • The evaluation covers 100 target questions per dataset, four 4B–8B victim models, dense retrieval, and a simple context-grounded QA prompt.
  • The attacker is assumed to know the target query and add one document for that target, while the surrogate check uses generation rather than a full surrogate RAG pipeline.

A Examples of Target Questions

This appendix lists example target questions from Natural Questions, HotpotQA, and MS-MARCO, preserving their wording except for minor transcription, capitalization, and grammatical corrections.

  • The examples span target questions drawn from NQ, HotpotQA, and MS-MARCO.
  • The appendix preserves question wording except for obvious transcription, capitalization, and grammatical corrections.
  • The listed questions cover varied factual topics, including music, science, television, film, sports, health, prices, and occupations.

B System Prompt

The victim RAG generator is prompted to answer a query using supplied contexts, producing a short response and returning “I don’t know” when the contexts lack an answer.

  • The system prompt instructs the assistant to answer using the provided relevant contexts and the user’s query.
  • The response should be short and concise, and the assistant should say “I don’t know” when the answer cannot be found in the contexts.

C Prompts of ToxicRAG

ToxicRAG constructs one poisoned passage by reframing the accepted answer as outdated, fabricating causal changes, and invoking apparent authority consensus. It validates and optionally refines the passage when a surrogate model does not produce the targeted answer.

  • C Prompts of ToxicRAG: The implementation consistently represents the attacker’s desired incorrect answer as targeted_answer across prompt templates.The appendix also specifies concise, natural, semantically aligned outputs and strict JSON for intermediate generations.
  • C.1 Causal Events Prompt: A causal-events prompt creates plausible developments, discoveries, policy shifts, or reassessments connecting the previously correct answer to the targeted answer.The generated chain is intended to justify an answer update.
  • C.2 Authorities Prompt: An authorities prompt generates fictitious institutions or panels that collectively endorse the attacker-selected answer.Allowed authority forms include committees, associations, registries, panels, and review boards.
  • C.3 Document Prompt: ToxicRAG generates a natural poisoned passage that presents the original answer as outdated and the targeted answer as newly authoritative.The document combines causal changes, authority endorsements, semantic alignment, and concise paragraph form.
  • C.4 Validation Prompt / C.5 Refinement Prompt: A validation prompt tests whether the victim answers from the poisoned passage alone, and a refinement prompt strengthens the target when validation fails.Refinement preserves the causal-chain and majority-consensus framing while emphasizing outdatedness and authority agreement.
  • D.1 Prompts of PoisonedRAG: Compared with PoisonedRAG, ToxicRAG-style prompting explicitly generates an incorrect answer and a corpus document intended to induce it.The baseline templates directly ask an LLM to produce misleading corpus content for the target query.
  • D.2 Prompts of CorruptRAG-AS / D.3 Prompts of CorruptRAG-AK: CorruptRAG-AS and CorruptRAG-AK use outdated-answer and newly-confirmed-answer framing, followed by LLM rewriting and validation.The prompts contrast an old answer with a plausible incorrect answer and then test whether the refined text preserves the target output.
  • D.4 Prompts of AuthChain: AuthChain builds a poisoned document from intent, evidence nodes, evidence relations, authority content, and a final merge step.Its prompts separately revise evidence chains, generate authoritative-looking updates, and combine them into one natural passage.

E On the Reliability of Poisoning Judge Using LLM

The study checks an automatic poisoning judge against majority-vote human labels on sampled examples. Agreement is high across all three datasets, supporting judge use while leaving shared evaluation errors possible.

  • Dataset Construction and Annotation: Three researchers independently labeled 300 examples, with majority vote defining the human reference against which the automatic evaluator was assessed.The sample contained 100 test queries from each dataset and included answers before and after injection plus the targeted answer.
  • Analysis of Validation Results: 0.9008, 0.8056, and 0.9195 Cohen’s kappa values indicate high agreement for NQ, HotpotQA, and MS-MARCO, respectively.These values come from a 300-example validation sample.
  • Analysis of Validation Results: The validation supports using the LLM judge for reported experiments but does not establish error-free evaluation or exclude errors shared with annotators.The agreement check therefore provides evidence of alignment, not proof that every judgment is correct.

F Examples of ToxicRAG attacks on NQ, HotpotQA, MS-MARCO dataset

The examples illustrate ToxicRAG’s narrative alterations across factual, temporal, semantic, statistical, and multi-hop questions. They show both successful induction and an explicitly documented failed attack, while curated examples are not auditable run records.

  • Examples of ToxicRAG attacks: The examples span NQ, HotpotQA, and MS-MARCO and include target queries, correct and targeted answers, poisoned text, and victim outputs.The collection covers nationality, seasons, salaries, historical dates, meanings, and multi-hop questions.
  • Example limitations: The examples are manually curated and omit original query identifiers, retrieval ranks, complete contexts, and per-example judge records, so they are illustrative rather than auditable run records.A failed attack is also included to avoid presenting only successful outcomes.
  • Multi-hop Year Alteration Attack: A multi-hop example shows the poisoned passage changing the founding year to 1955 and the LLM outputting that targeted answer.The original answer was 1890 for the college associated with the 1971 Oklahoma State Cowboys football team.
  • Illustrative attacks: Other examples alter nationality, seasonal definitions, salaries, historical origins, and word meanings by presenting fabricated developments as updates.These cases target answers such as “Yes,” “January and February,” “Three million dollars per performance,” “1850,” and “The Whimsy of the Wind.”
  • Failed attack: Table 10 documents a failed ToxicRAG attack on NQ alongside the successful examples.The supplied material identifies the table but does not provide its underlying cells.
Loading 2609.11082v1…