PaperlayerView exact revision on arXiv

Source-linked AI summary

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Wei Zou, Runpeng Geng, Binghui Wang, Jinyuan Jia

arXiv:2402.07867v3cs.CRcs.LG

TL;DR

RAG security remains largely unexplored despite its role in addressing LLM knowledge limitations. PoisonedRAG introduces a knowledge corruption attack using malicious database texts, achieving high attack success in fact verification and exposing insufficient defenses.

  • Problem

    RAG research has mainly improved retrieval accuracy and efficiency, while the security of its knowledge databases remains largely unexplored.

  • Method

    PoisonedRAG formulates knowledge corruption as an optimization problem and crafts malicious texts to satisfy retrieval and generation conditions under black-box or white-box settings.

  • Results

    0.97 and 0.88 ASR were achieved in black-box and white-box fact-verification settings, respectively, while several defenses were insufficient to mitigate the attacks.

  • Takeaways & Limitations

    Knowledge databases introduce a practical attack surface that can enable disinformation, biased answers, and harmful health or financial misinformation.

  • Takeaways & Limitations

    The threat model assumes attackers cannot access database texts, the LLM's parameters, or its queries, and distinguishes settings by retriever knowledge.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.

1 Introduction

RAG augments LLMs with retrieved external knowledge, but its security remains largely unexplored. PoisonedRAG identifies the knowledge database as a practical attack surface and develops an attack designed to induce attacker-chosen answers.

  • RAG augments an LLM with external knowledge retrieved from a knowledge database to mitigate outdated knowledge and hallucinations.Its components are a knowledge database, retriever, and LLM.
  • Existing RAG research primarily improves retrieval accuracy and efficiency, while RAG security remains largely unexplored.
  • Attackers inject malicious texts into a knowledge database to induce an LLM to generate attacker-desired answers for selected target questions.The paper considers black-box and white-box attacker settings based on knowledge of the retriever.
  • PoisonedRAG derives retrieval and generation conditions and crafts malicious texts intended to satisfy both conditions.
  • PoisonedRAG is presented as the first knowledge corruption attack exploiting the attack surface introduced by RAG knowledge databases.
  • The study evaluates PoisonedRAG across multiple databases, retrievers, RAG schemes, and LLMs, compares five baselines, and explores defenses.

2 Background and Related Work

RAG retrieves relevant texts and supplies them as context for an LLM to generate an answer. PoisonedRAG differs from prior attacks by targeting both retrieval and answer generation through semantically meaningful malicious texts.

  • Background on RAG: A RAG system stores texts in a knowledge database, retrieves the k most relevant texts, and provides them as context to an LLM.
  • Background on RAG: The retrieval stage ranks database texts using similarity between question and text encoder embeddings.
  • Background on RAG: The generation stage queries the LLM with the question and retrieved texts, using those texts as context for the answer.
  • Related Work: Prompt injection attacks insert malicious instructions into an LLM input, whereas PoisonedRAG crafts texts that are retrieved for target questions and influence target answers.
  • Related Work: Jailbreaking attacks seek to break safety alignment, giving them a different goal from PoisonedRAG.
  • Related Work: Unlike attacks using random-character texts that only target retrieval, PoisonedRAG aims for semantically meaningful texts that also mislead generation.
  • Related Work: Existing data-poisoning studies typically compromise model training data, whereas this attack targets the knowledge database used by RAG.

3 Problem Formulation

The paper models an attacker who selects target questions and answers, injects malicious texts into a RAG database, and seeks target outputs under black-box or white-box knowledge assumptions. It formalizes this goal as a constrained optimization problem.

  • 3.1 Threat Model: An attacker selects arbitrary target questions and attacker-desired answers, then seeks to make the RAG LLM produce each selected answer.
  • 3.1 Threat Model: The attack is motivated by risks including disinformation, biased product answers, and harmful health or financial misinformation.
  • 3.1 Threat Model: The attacker cannot access database texts or the LLM, while retriever access distinguishes black-box from white-box settings.
  • 3.1 Threat Model: The threat model assumes the attacker can inject N malicious texts for each target question into the knowledge database.
  • 3.1 Threat Model: The paper notes that a few injected texts, totaling hundreds of tokens, can achieve a high attack success rate.
  • 3.2 Knowledge Corruption Attack to RAG: Knowledge corruption is formulated as constructing a malicious-text set Γ so retrieved context causes target answers for target questions.
  • 3.2 Knowledge Corruption Attack to RAG: The formulation follows the principle that attack effectiveness depends on the corrupted database producing the required retrieved texts.
  • 3.2 Knowledge Corruption Attack to RAG: The optimization maximizes the number of target questions for which the LLM output equals the attacker-selected answer, subject to the retrieval constraint.

4 Design of PoisonedRAG

PoisonedRAG crafts malicious texts that satisfy both retrieval and generation conditions, enabling target answers when injected into a RAG knowledge database. It uses heuristic generation and, depending on retriever access, black-box or white-box methods to construct these texts.

  • Overview: PoisonedRAG independently crafts N malicious texts for each of M target questions, pairing each question Q with an attacker-chosen answer R.The final goal is a set of M·N malicious texts targeting the corresponding question-answer pairs.
  • Deriving Two Necessary Conditions for an Effective Knowledge Corruption Attack: An effective malicious text must satisfy both retrieval and generation conditions: it must be retrieved for Q and induce the LLM to generate R.The approach formulates text crafting as an optimization problem but solves it heuristically through these two derived conditions.
  • Crafting Malicious Texts to Achieve the Two Derived Conditions: PoisonedRAG decomposes P into S⊕I, crafting I for generation and S for retrieval while preserving I’s generation effect.This decomposition addresses cases where making P highly similar to Q would help retrieval but harm generation.
  • Crafting I to Achieve Generation Condition: Generating transferable malicious texts is challenging because attackers may lack LLM parameters, white-box optimization can be computationally expensive, and transfer across LLMs remains difficult.PoisonedRAG therefore uses heuristic solutions that avoid gradient calculation and can employ an attacker-chosen LLM.
  • Crafting I to Achieve Generation Condition: For generation, an attacker-chosen LLM repeatedly generates I until it produces R for Q or reaches L trials, requiring two or three queries on average.The final trial’s text is retained if the maximum number of trials is reached, and the generated text can vary because of non-zero-temperature randomness.
  • Crafting S to Achieve Retrieval Condition: In the black-box setting, PoisonedRAG uses Q itself to help retrieve S⊕I, whereas the white-box setting optimizes embedding similarity between Q and S⊕I.White-box optimization uses the retriever’s query and text encoders and can apply gradient-based adversarial-text methods.

5 Evaluation

The evaluation tests PoisonedRAG across datasets, retrievers, LLMs, attack settings, and baselines. Results show high attack success and retrieval effectiveness, with performance shaped by the number of retrieved and injected malicious texts.

  • Experimental Setup: The evaluation covers NQ, HotpotQA, and MS-MARCO knowledge databases, three retrievers, five LLMs, and multiple attack baselines.The default setup uses NQ, Contriever, five retrieved texts, PaLM 2, and ten repeated trials over randomly selected close-ended questions.
  • Metrics: Attack Success Rate uses substring matching, while Precision, Recall, and F1-Score measure whether injected malicious texts are retrieved.Human evaluation validates substring matching, producing similar ASRs.
  • Main Results: 97% ASR on NQ, 99% on HotpotQA, and 91% on MS-MARCO are achieved in the black-box PaLM 2 setting with five injected texts per target question.The knowledge databases contain millions of clean texts, and F1-Scores exceed 90% in almost all cases.
  • Main Results: PoisonedRAG outperforms the baselines because it is designed to satisfy retrieval and generation conditions simultaneously.Naive Attack, Corpus Poisoning Attack, and GCG Attack fail because they cannot achieve generation, generation, and retrieval conditions, respectively.
  • Robustness: PoisonedRAG remains effective across different retrievers and LLMs, while white-box attacks are more effective in most cases than black-box attacks.The white-box advantage is attributed to access to more retriever knowledge.
  • Hyperparameters: ASR is high when k ≤ N and decreases when k > N; increasing N raises ASR when N ≤ k, after which ASR stabilizes.When k > N, recall remains close to 1 because almost all malicious texts are retrieved, while additional clean texts reduce precision.
  • Attack Construction: TextFooler and HotFlip both achieve very high ASR and F1-Score, but TextFooler trades higher computational overhead for greater semantic stealthiness.TextFooler requires more optimization iterations because its candidate word space is smaller.

6 Evaluation for Real-world Applications

The paper evaluates PoisonedRAG beyond basic RAG, including advanced RAG schemes, a Wikipedia-based chatbot, and LLM agents. These experiments report continued effectiveness in sophisticated and real-world settings.

  • Advanced RAG and Applications: The evaluation extends PoisonedRAG to advanced RAG schemes and two applications: a Wikipedia-based ChatBot and LLM agents.The advanced schemes are intended to improve retrieved-context relevance, robustness, and correctness.
  • Advanced RAG: PoisonedRAG achieves high ASRs against advanced RAG schemes.Table 10 reports its effectiveness under advanced RAG.
  • Wikipedia Application: A real-world Wikipedia scenario uses a December 20, 2018 English Wikipedia dump containing 21,015,324 texts.The threat model assumes attackers can inject texts by maliciously editing Wikipedia articles.
  • LLM Agents: The LLM-agent evaluation adopts ReAct, where agents alternate thought, action, and observation steps using document retrieval or task finishing actions.This tests PoisonedRAG in an agent setting that interacts with an external environment.

7 Defenses

The paper evaluates several defenses against PoisonedRAG, including paraphrasing, perplexity detection, duplicate filtering, and retrieving more texts. These defenses remain insufficient, with limitations arising from attack-text diversity, normal text quality, and scalable injection.

  • 7.1 Paraphrasing: Paraphrasing target questions cannot effectively defend against PoisonedRAG.The experiments generate five GPT-4 paraphrases per target question, yet PoisonedRAG still achieves high ASRs and F1-Score.
  • 7.2 Perplexity-based Detection: High perplexity does not reliably distinguish malicious texts from clean texts.The ROC analysis finds large false-positive rates at large true-positive rates because malicious-text perplexity is not statistically higher than clean-text perplexity.
  • 7.3 Duplicate Text Filtering: Duplicate-text filtering leaves attack success rate unchanged.Different GPT-4-generated sub-texts make malicious texts diverse even when each text is generated independently.
  • 7.4 Knowledge Expansion: Knowledge Expansion still cannot completely defend against PoisonedRAG when retrieving k = 50 texts.With N = 5 injected texts, ASR remains 41% in black-box and 43% in white-box settings on HotpotQA; larger contexts also increase LLM computation costs.

8 Discussion and Limitation

The discussion extends PoisonedRAG beyond question answering to fact verification and identifies joint optimization across multiple target questions as future work. On FEVER, the attack achieves high retrieval and attack-success scores in both access settings.

  • Broad NLP tasks: PoisonedRAG extends to fact verification, where retrieved texts support, refute, or fail to determine a claim.The FEVER experiment uses 100 target claims and crafts incorrect verification results as attacker-chosen targets.
  • Broad NLP tasks: 0.98 and 0.99 F1-Score are achieved in black-box and white-box settings, respectively, on FEVER.These results indicate that almost all malicious texts are retrieved for the corresponding target claims.
  • Jointly considering multiple target questions: Crafting malicious texts jointly for multiple target questions is left as future work.The paper states that independently crafting texts for each target question could be sub-optimal.

9 Conclusion and Future Work

The paper concludes that PoisonedRAG exposes a practical knowledge-database attack surface and remains effective in both black-box and white-box settings. It calls for optimization-based attacks and new defenses, while the supplied material also documents the RAG generation pipeline and experimental examples.

  • Conclusion: PoisonedRAG is presented as the first knowledge corruption attack against RAG.The attack exploits a new and practical attack surface introduced by knowledge databases.
  • Conclusion: The attack is effective in both black-box and white-box settings.The conclusion also reports that evaluated defenses are insufficient to mitigate the proposed attacks.
  • Future Work: Future work includes new optimization-based attacks and defenses against PoisonedRAG.The authors specifically mention extending GCG to optimize malicious-text content and jointly considering multiple target questions.
  • Experimental and Pipeline Context: The appendix provides target-question examples and generated answers for NQ, HotpotQA, and MS-MARCO.Table 23 covers NQ, while Tables 24 and 25 cover HotpotQA and MS-MARCO.

E Experimental Details on FEVER Dataset

The FEVER experiments adapt PoisonedRAG and RAG prompts for claim verification, using SUPPORTS, REFUTES, or NOT ENOUGH INFO outputs. The supplied tables and passages describe the experimental setup, prompts, and retrieval-condition evaluation.

  • FEVER Task and Prompts: The PoisonedRAG generation prompt asks for a corpus producing an incorrect verification output for a claim.The prompt limits the crafted corpus to 30 words.
  • Experimental Comparisons: The provided experimental materials include a table comparing PoisonedRAG with two variants.Table 15 is titled “PoisonedRAG outperforms its two variants.”
  • FEVER Task and Prompts: FEVER evaluates whether retrieved texts support, refute, or provide insufficient information about a claim.The RAG prompt instructs the model to respond with SUPPORTS, REFUTES, or NOT ENOUGH INFO.
  • FEVER Task and Prompts: The FEVER setup uses a claim, retrieved contexts, and an answer as the core prompt fields.The supplied prompt template is “Claim: [claim] Contexts: [context] Answer:”.
  • Experimental Comparisons: The provided materials also include a table evaluating PoisonedRAG’s retrieval condition.A separate table examines effectiveness when retrieved texts contain different numbers of malicious texts.

F Ablation Study Results of PoisonedRAG with Different LLMs Used in RAG

PoisonedRAG maintains high attack performance across different LLMs in RAG. Malicious texts can also be retrieved for non-target questions when semantic overlap creates similarity.

  • Cross-LLM robustness: PoisonedRAG maintains high ASRs and attack performance across different LLMs used in RAG.The reported results are similar to the default setting across LLMs.
  • Non-target retrieval: Malicious texts are sometimes retrieved for non-target questions because they are semantically related through shared keywords or contexts.This occurs when the non-target question and malicious text share topical relationships.
  • Non-target retrieval: A Star Wars example pairs a non-target question about Clone Wars seasons with malicious text answering how many Death Stars exist.Both texts are related to Star Wars, illustrating semantic overlap between unrelated targets.

I Analysis on Failure Case of PoisonedRAG

PoisonedRAG fails to reach 100% ASR in some settings for two identified reasons: clean texts may enter the retrieved context, or malicious texts may contain the correct answer.

  • Failure causes: PoisonedRAG does not reach 100% ASR in some settings because its retrieval and generation components are imperfect.The authors identify clean retrieved texts and correct answers appearing in malicious texts as the two failure sources.
  • Retrieval failures: When top-k retrieved texts include clean texts, some malicious texts are not retrieved for the target question.This limits the attack’s retrieval component.
  • Generation failures: When malicious texts contain the correct answer, the LLM can still generate that correct answer even if all k retrieved texts are malicious.The authors refer to this phenomenon as parametric bias.

J Minor Gap Between Human Evaluations and Substring Matching in Calculating ASRs

The paper reports a minor gap between human evaluation and substring matching for ASR, caused by edge cases in judging whether the attacker’s target answer was actually produced. It also notes that perfect text-comparison metrics remain an open challenge.

  • Evaluation gap: Human evaluation and substring matching produce similar ASRs, with only a minor difference in some cases.The authors report that only a few cases receive different evaluation results.
  • Evaluation gap: Substring matching can count an attack as successful when the target answer appears in an alternate-universe qualification that does not answer the question correctly.The example uses “John Williams” as the target answer for a Phantom of the Opera question.
  • Evaluation gap: Human evaluation can also judge an attack successful when substring matching does not, as illustrated by an answer stating that From Dusk till Dawn has two seasons.The paper presents this as a second type of disagreement.
  • Scope: Developing a perfect metric for comparing two texts remains an open challenge beyond this work’s scope.The authors note that substring matching and human evaluation differ in only a few cases.

K Crafting Malicious Texts with Less Powerful LLMs

The paper describes prompting a language model to craft corpora that make target answers appear correct, then illustrates malicious texts produced for diverse questions and target answers.

  • Crafting procedure: The crafting prompt asks an LLM to generate a corpus in which a specified answer is presented as the answer to a specified question.The prompt includes demonstration examples pairing questions, answers, and corpora.
  • Examples: A malicious text targets the Mission Impossible theme-song question by asserting John Williams instead of the correct answer, Lalo Schifrin.This example is identified as a black-box PoisonedRAG text.
  • Examples: Other examples target questions about mineral stability, football awards, Dutch clothing, car-buying age, and Shanghai–GMT time difference with incorrect answers.The examples include black-box texts and, in several cases, corresponding white-box texts.
Loading 2402.07867v3…