PaperlayerView exact revision on arXiv

Source-linked AI summary

BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, Qian Lou

arXiv:2406.00083v2cs.CRcs.AIcs.CLcs.IRcs.LG

TL;DR

LLMs can have outdated or inaccurate knowledge, and using external corpora in RAG introduces security vulnerabilities. BadRAG uses poisoned passages and semantic triggers to attack retrieval and generation; with 10 passages, it achieved a 98.2% attack success rate, alongside reported rejection and negative-response outcomes.

  • Problem

    LLMs can lack up-to-date knowledge and generate inaccurate content, while RAG’s use of external corpora introduces security vulnerabilities.

    Source: §1 · pp. 1–2

  • Method

    BadRAG poisons customized passages linked to semantic triggers to target retrieval and induce denial-of-service or sentiment-steering attacks without modifying the retriever or LLM.

    Source: §1 · p. 2

  • Results

    With 10 adversarial passages, Merged Adaptive COP achieved a 98.2% attack success rate; reported outcomes also include a 74.6% rejection ratio and 72% probability of negative responses.

    Source: §5.4 · pp. 8–9

  • Takeaways & Limitations

    BadRAG demonstrates that poisoned corpora can be used to target both RAG retrieval and LLM generation, motivating robust defensive strategies in AI deployments.

    Source: §8 · p. 9

    The study applies BadRAG attacks to QA and summarization; code generation and agent planning remain proposed extensions.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. This approach involves retrieving relevant information from a large, up-to-date dataset and using it to enhance the generation process, leading to more accurate and contextually appropriate responses. Despite its benefits, RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web. In this paper, we propose \TrojRAG{} to identify the vulnerabilities and attacks on retrieval parts (RAG database) and their indirect attacks on generative parts (LLMs). Specifically, we identify that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized poisoned adversarial queries. Triggers and poisoned passages can be highly customized to implement various attacks. For example, a trigger could be a semantic group like "The Republican Party, Donald Trump, etc." Adversarial passages can be tailored to different contents, not only linked to the triggers but also used to indirectly attack generative LLMs without modifying them. These attacks can include denial-of-service attacks on RAG and semantic steering attacks on LLM generations conditioned by the triggers. Our experiments demonstrate that by just poisoning 10 adversarial passages can induce 98.2\% success rate to retrieve the adversarial passages. Then, these passages can increase the reject ratio of RAG-based GPT-4 from 0.01\% to 74.6\% or increase the rate of negative responses from 0.22\% to 72\% for targeted queries.

1 Introduction

LLMs’ knowledge and reliability limits motivate RAG, but external corpora create security vulnerabilities. BadRAG studies attacks triggered by customized queries and carried out through poisoned passages, targeting retrieval and generation.

1 Introduction

  • LLMs can lack up-to-date or domain-specific knowledge and generate inaccurate content, while RAG retrieves external information to improve responses.The introduction presents RAG as a way to address these limitations by drawing on external sources.Source: §1 · p. 1
  • Using external corpora in RAG creates a new attack surface, motivating investigation of its security vulnerabilities.Source: §1 · p. 2
  • BadRAG assumes attackers poison passages while leaving the retriever and LLM unchanged, activating malicious behavior only for customized trigger queries.The stated challenges include linking semantic triggers to passages, eliciting logical rather than copied responses, and overcoming LLM alignment.Source: §1 · p. 2
  • Earlier attacks lacked flexible trigger-conditional retrieval or open-ended generation, often relying on always-retrieved passages or fixed query-answer pairs.The paper notes that aligned LLMs can block retrieval attacks and that prior generated answers often copied poisoned passages.Source: §1 · p. 2
  • BadRAG introduces COP, ACOP, and MCOP for trigger-linked passage optimization, plus AaaA and SFaaA for denial-of-service and sentiment-steering attacks.The paper evaluates its approaches across five datasets, three retrievers, and three LLMs, including GPT-4 and Claude-3.Source: §1 · p. 2

2 Related Work

Related work describes RAG’s retrieval-then-generation workflow and flexibility, then contrasts BadRAG’s trigger-conditional retrieval and generative attacks with limitations in earlier methods.

2 Related Work

  • RAG retrieves relevant corpus passages using query similarity, then combines them with the query as input to an LLM.The paper describes retrieval and generation as two sequential phases.Source: §2 · p. 3, Figure 1 · p. 3
  • RAG’s updateable corpus supports adaptation to changing knowledge without fine-tuning the LLM.The paper cites uses ranging from personal chatbots to specialized medical assistants.Source: §2 · p. 3
  • Security vulnerabilities introduced by RAG had received limited investigation compared with other LLM attack categories.Source: §2 · p. 3
  • Prior retrieval attacks either retrieve for any query or target predefined question-answer pairs, unlike BadRAG’s group-semantic triggers.Exact-question targeting may fail when users phrase a question differently.Source: §2 · p. 3
  • Earlier generative attacks often copied fixed answers and did not support open-ended generation, whereas BadRAG targets sentiment steering and denial of service.The paper notes that aligned LLMs can resist attacks that rely on retrieved passages.Source: §2 · p. 3

3 BadRAG

BadRAG links semantic trigger scenarios to adversarial passages, activating poisoned retrieval for matching queries while leaving nonmatching queries on legitimate passages. It combines retrieval optimization with generation-phase attacks that target refusal or sentiment steering.

3 BadRAG

  • BadRAG links semantic trigger scenarios to adversarial passages, activating poisoned retrieval for matching queries while leaving nonmatching queries on legitimate passages.For example, Trump-related queries can retrieve biased content while a Biden query remains unaffected.Source: §3 · p. 4
  • The attacker is assumed to have limited corpus-injection ability and white-box access to the retriever, but no information about the LLM.The described publication routes include public platforms and data-collection agencies.Source: §3 · p. 4

3.1 Retrieval-phase Attacking Optimization

  • COP optimizes an adversarial passage to resemble triggered-query embeddings while remaining dissimilar to queries without the trigger.It treats triggered queries as positive samples and untriggered queries as negative samples.Source: §3.1 · pp. 4–5
  • Adaptive COP clusters adversarial passages with similar embedding features and merges them to cover multiple triggers with fewer inserted passages.The method avoids separate optimized passages for every trigger, which would increase the poisoning ratio and reduce stealth.Source: §3.1 · pp. 5–6

3.2 Generation-phase Attacking Methods

  • AaaA crafts a retrieved prompt that exploits an LLM's alignment features, such as privacy sensitivity, to make it refuse queries.The described process probes alignment, selects a feature, and crafts a prompt that activates it.Source: §3.2 · p. 6
  • SFaaA injects real, biased articles into the corpus so that retrieved content steers the LLM toward a chosen sentiment.The examples steer responses negatively about Donald Trump and positively about Joe Biden.Source: §3.2 · p. 6

4 Experimental Methodology

The evaluation uses several QA and generation datasets, retrievers, and LLMs, with metrics spanning retrieval, answer quality, rejection, and sentiment.

4 Experimental Methodology

  • Open-domain QA evaluation uses Natural Questions, MS MARCO, and SQuAD, while generation evaluation uses WikiASP segmented by domains including public figures and companies.The passage identifies the datasets and their roles in the evaluation.Source: §4 · p. 7
  • The evaluated retrievers are Contriever, DPR, and ANCE, and the LLMs are GPT-4, Claude-3-Opus, and LLaMA-2-7b-chat-hf.GPT-4 and Claude-3-Opus are described as black-box models; LLaMA-2-7b-chat-hf is white-box.Source: §4 · p. 7
  • Metrics include retrieval success, rejection rate, Rouge-2 F1, accuracy, quality score, and positive or negative response rates.Experiments were conducted using two Nvidia-RTX 3090 GPUs.Source: §4 · p. 7

5 Evaluation

Evaluation finds BadRAG can selectively retrieve adversarial passages for triggered queries, alter generation through refusal and sentiment steering, and bypass two existing passage defenses. Ablations compare passage optimization and content-crafting variants.

5.1 RQ1: Retrieval Attacks Only for Trigger Queries

  • 98.9% average retrieval rate for triggered queries at Top-1 versus 0.15% for non-trigger queries across three datasets.The reported percentage measures queries retrieving at least one adversarial passage in the top-k results.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 RQ2: Generative Attacks on LLMs

  • 74.6% probability to refuse service: GPT-4 shows this under triggers, while clean queries across models have very low reject ratios.The setup uses 10 adversarial passages (0.04% poisoning ratio); GPT-4 Rouge-2 falls from 23.7% to 6.94%, and accuracy falls from 92.6% to 19.1%.Source: Table 2 · p. 7, §5.2 · pp. 7–8
  • 0.22% to 72.0%: BadRAG increases GPT-4's negative response ratio for queries about Donald Trump.The paper reports sentiment steering across LLMs and trigger scenarios, while response quality drops only slightly on average, from 7.4 to 7.2.Source: §5.2 · p. 8
  • 0.05% to 27.9% on average for Chinese versus 3.53% to 79.7% on average for TikTok; race-related steering is most challenging.The paper describes steering for company-related TikTok queries as easiest.Source: §5.2 · p. 8

5.3 RQ3: Robust against Existing Defense

  • BadRAG bypasses passage embedding-norm and perplexity defenses using passages aligned with the target LLM's feature space and natural-language attack strategies.The paper says Alignment as an Attack (AaaA) and Selective-Fact as an Attack (SFaaA) use natural language to circumvent perplexity-based detection.Source: §5.3 · p. 8

5.4 Ablation Experiments

  • 71.8% attack success rate with 200 passages for COP versus 98.2% with only 10 for Merged Adaptive COP.The paper attributes the gain to merging similar adversarial passages, which addresses the difficulty of optimizing one passage for a group of triggers.Source: §5.4 · p. 8
  • For DoS attacks, the naive method achieved only a 2.38% rejection ratio, while Alignment as an Attack (AaaA) achieved 74.6%, substantially degrading performance on Rouge-2 and Accuracy.The authors attribute the refusals to drawing the LLM’s attention to private information and its alignment policies.Source: §5.4 · pp. 8–9
  • 4.19% probability of negative answers for negatively crafted passages versus 72% for SFaaA using biased but true articles from official sources.The paper says the selected factual passages can bypass alignment because they are likely included in the LLM's pre-training data.Source: §5.4 · p. 9

6 Potential Defense

The defense detects trigger-conditioned retrieval by masking query tokens and tracking similarity-score changes, but its effectiveness depends on trigger length.

6 Potential Defense

  • Masking query tokens and measuring retrieval-similarity changes distinguishes triggered adversarial queries from clean queries.Removing a trigger can prevent retrieval of its adversarial passage while clean queries retain overall semantic matching.Source: §6 · p. 9
  • For single-token triggers, masking one token distinguishes adversarial from clean queries through larger similarity-score gaps.Single-token masking is often insufficient for two-token triggers; replacing both tokens increases the gap.Source: §6 · p. 9
  • Unknown trigger length limits the method, because longer replacements can make clean-query similarity scores overlap with adversarial scores.Source: §6 · p. 9

7 Discussion

The discussion highlights risks from RAG attacks in high-stakes sectors and identifies broader attack coverage and task applications as extensions of the current work.

7 Discussion

  • BadRAG’s findings highlight RAG vulnerabilities in healthcare, finance, and other high-stakes sectors, motivating robust countermeasures.The paper identifies system administrators, developers, and policymakers as audiences for these risks.Source: §7 · p. 9
  • The current attack paradigm leverages LLM alignment for denial-of-service and sentiment steering, while additional alignment features could support a broader range of attacks.Source: §7 · p. 9
  • The study applies BadRAG to QA and summarization; code generation and agent planning are identified as possible extensions.Source: §7 · p. 9

8 Conclusion

The conclusion presents BadRAG as a framework for attacks on RAG retrieval and generation, using trigger-activated adversarial passages and alignment-based attacks.

8 Conclusion

  • BadRAG targets security vulnerabilities in both RAG’s retrieval and generative phases.Source: §8 · p. 9
  • Contrastive optimization generates adversarial passages activated by specific triggers, while LLM alignment is leveraged for denial-of-service and sentiment steering.Source: §8 · p. 9
  • Tests on datasets and models including GPT-4 and Claude-3 demonstrate precise targeting and effective manipulation of LLM outputs.The conclusion highlights the need for robust defensive strategies in AI deployments.Source: §8 · p. 9

Prompt for Generating Queries for Open-Ended Discussion

The appendix gives query and response prompts, evaluation procedures and metrics, dataset details, sentiment-steering results, and tests of existing defenses.

Prompt for Generating Queries for Open-Ended Discussion

  • The open-ended query prompt requests diverse, contextual English generation queries and responses under 100 words; 149 Alpaca seed queries were filtered to exclude classification.The prompt also asks for varied verbs and language and open-ended tasks that a GPT model can complete.Source: Prompt for Generating Queries for… · p. 14

Prompt for Generating Responses for Open-Ended Discussion

Prompt for Evaluating Response Quality; Prompt for Evaluating Response Sentiment

Experiment Details

  • The experiments use NQ with 2.6 million passages and 3,452 queries, MS MARCO with 8.8 million passages and 5,793 queries, and SQuAD with 23,215 passages and 107,785 queries.WikiASP-Official has 22.7k passages, and WikiASP-Company has 30.3k passages.Source: Experiment Details · p. 15

Evaluation metrics

  • Evaluation metrics include retrieval success, rejection rate, Rouge-2 F1, accuracy, response quality, and positive or negative response ratios.The metrics respectively assess adversarial retrieval, refusal, answer similarity, correctness, overall quality, and sentiment proportions.Source: Evaluation metrics · p. 15
  • Positive sentiment steering follows the negative-steering trend but has less pronounced impact because clean RAG already produces many positive responses.Table 6 reports positive steering with 10 adversarial passages at a 0.04% poisoning ratio.Source: Evaluation metrics · p. 15, Table 6 · p. 16

Robustness against Existing Defense

  • BadRAG passages are difficult to distinguish using embedding norms or average token log likelihood: targeted triggers need not rely on large norms, and fluent malicious content dilutes the short backdoor prefix.Figure 6 presents results for potential defenses.Source: Robustness against Existing Defense · p. 16, Figure 6 · p. 16
  • Robustness tests use “discussing Republic” for close-ended QA and three open-ended scenarios; ChatGPT generates 300 triggered queries per sentiment-steering topic.Open-ended triggers include discussions of Donald Trump, TikTok, and Chinese topics.Source: Robustness against Existing Defense · pp. 16–17
Loading 2406.00083v2…