PaperlayerView exact revision on arXiv

Source-linked AI summary

Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation

Harsh Chaudhari, Giorgio Severi, John Abascal, Anshuman Suri, Matthew Jagielski, Christopher A. Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, Alina Oprea

arXiv:2405.20485v3cs.CRcs.CLcs.LG

TL;DR

RAG systems face a document-trustworthiness risk when malicious material is injected into their knowledge bases. Phantom optimizes a single poisoned document for trigger-based retrieval and an adversarial output objective; the paper reports an attack demonstration on NVIDIA ChatRTX and trigger-dependent integrity violations.

  • Problem

    The trustworthiness of RAG knowledge-base documents is a security challenge because malicious users may inject poisoned documents.

    Source: §1 · p. 2

  • Method

    Phantom uses two-stage optimization to make a single poisoned document retrieve on queries containing a natural trigger and induce an adversarial objective in the LLM output.

    Source: §1 · p. 2

  • Results

    The paper reports that the integrity violation is activated when a natural trigger appears in the user query, regardless of the query’s additional content.

    Source: §1 · p. 2, §8 · p. 19

  • Takeaways & Limitations

    The authors demonstrate an attack against the NVIDIA ChatRTX application using their single-document, trigger-activated poisoning framework.

    Source: §8 · p. 19

    Designing certified defenses with a better utility–robustness tradeoff remains an ongoing challenge.

    Source: §7 · p. 19

Abstract

from arXiv · show

Retrieval Augmented Generation (RAG) expands the capabilities of modern large language models (LLMs), by anchoring, adapting, and personalizing their responses to the most relevant knowledge sources. It is particularly useful in chatbot applications, allowing developers to customize LLM output without expensive retraining. Despite their significant utility in various applications, RAG systems present new security risks. In this work, we propose a novel attack that allows an adversary to inject a single malicious document into a RAG system's knowledge base, and mount a backdoor poisoning attack. We design Phantom, a general two-stage optimization framework against RAG systems, that crafts a malicious poisoned document leading to an integrity violation in the model's output. First, the document is constructed to be retrieved only when a specific naturally occurring trigger sequence of tokens appears in the victim's queries. Second, the document is further optimized with crafted adversarial text that induces various adversarial objectives on the LLM output, including refusal to answer, reputation damage, privacy violations, and harmful behaviors.We demonstrate our attacks on multiple open-source LLM architectures, including Gemma, Vicuna, and Llama, and show that they transfer to closed-source models such as GPT-3.5 Turbo and GPT-4. Finally, we successfully demonstrate our attack on an end-to-end black-box production RAG system: NVIDIA's "Chat with RTX''.

1 Introduction

The introduction presents a single-document RAG backdoor activated by a natural trigger in a user query, and introduces Phantom to optimize both retrieval and the generated output. The authors report evaluations across multiple objectives, models, datasets, retrievers, and triggers, as well as an attack on a commercial RAG system.

1 Introduction

  • The threat model uses one poisoned knowledge-base document to cause an integrity violation when a natural trigger appears in a user query.The trigger can be a naturally occurring sequence such as “LeBron James.”Source: §1 · pp. 2–3
  • Phantom first optimizes the document for retrieval only on trigger-containing queries, then appends an optimized adversarial string to shape the generated text.The stated objectives include refusal, harmful generation, and data exfiltration.Source: §1 · p. 2
  • The authors introduce Multi-Coordinate Gradient (MCG), a strategy they say converges faster than GCG [65].Source: §1 · p. 3
  • Phantom achieved biased-opinion and passage-exfiltration objectives in a commercial black-box RAG system, NVIDIA’s Chat-RTX.The evaluation also covered five adversarial objectives, three datasets and retrievers, seven generators, and thirteen naturally occurring triggers.Source: §1 · p. 3

2 Background and Related Work

The background describes RAG as grounding LLM responses in retrieved documents and reviews prior adversarial work on language models, privacy, and RAG poisoning. It distinguishes Phantom from earlier RAG attacks by its trigger-activated objective and single-document threat model.

2 Background and Related Work

  • Earlier RAG poisoning approaches either target the retriever without a generator objective, inject prompts without explicitly optimizing retrieval, or target a predefined query and answer with multiple passages.The paper contrasts Phantom with Zhong et al. [64], Pasquini et al. [39], and PoisonedRAG by Zou et al..Source: §2 · p. 4
  • Phantom is described as query-agnostic, trigger-activated poisoning that unifies adversarial objectives while requiring a single poisoned passage.Unlike the targeted poisoning described for PoisonedRAG, the trigger activates the attack across queries containing the sequence.Source: §2 · p. 4
  • The paper contrasts its single-passage attack, which does not assume control of the retriever or generator, with a concurrent attack requiring multiple passages and retriever-training control.The authors characterize their threat model as more realistic and practical.Source: §2 · p. 4
  • For another concurrent comparison, the authors report high attack success in 32 iterations, versus more than 1,000 iterations for the compared bi-level optimization approach.They describe this as 30× more efficient.Source: §2 · p. 4
  • Against concurrent trigger-style work by Xue et al. [57], the authors report five objectives rather than two and effectiveness on objectives conflicting with safety alignment.They also state that their approach does not share that work’s reliance on unnatural triggers for context leakage and tool usage.Source: §2 · pp. 4–5

3 Threat Model

Phantom models an attacker who can add a poisoned document to a RAG knowledge base and trigger an output integrity violation when a naturally occurring token sequence appears in a query. The attack covers refusal, biased opinions, harmful outputs, and passage exfiltration without altering model training.

3 Threat Model

  • The threat model lets an attacker add one document to a local knowledge base without controlling the file system or knowing its other contents, and also describes attacks on web-based knowledge bases.The initial setting assumes publicly accessible models whose retriever and generator gradients the attacker can compute; the paper also considers black-box transfer and ChatRTX.Source: §3 · p. 5

3.1 Adversarial Objectives

  • The attacker chooses a likely natural query trigger, and the backdoor effect activates only when that trigger occurs, without requiring changes to model training.The attack strategy resembles backdoor poisoning, but RAG allows it without tampering with the training process.Source: §3 · p. 5, §3.1 · p. 5
  • Refusal and biased-opinion objectives respectively elicit “Sorry, I don’t know” or steer sentiment about selected topics, potentially harming reputations.The paper says refusal can reduce helpfulness and describes bias beyond sentiment manipulation.Source: §3.1 · pp. 5–6
  • The harmful-behavior objective targets direct harm such as insults or threats, requiring disruption of query-answering alignment and a bypass of safety training.Source: §3.1 · p. 6
  • Passage exfiltration aims to make the LLM reveal retrieved knowledge-base passages, with possible email leakage when the system has tool-use capabilities.The attack is described as universal to query content beyond the presence of the trigger sequence.Source: §3.1 · pp. 5–6

4 Phantom Attack Framework

Phantom constructs a single poisoned passage that first targets retrieval and then compromises generation. Its retriever optimization makes retrieval trigger-dependent, while its generator optimization searches for text that elicits a chosen output.

4 Phantom Attack Framework

  • Phantom concatenates retriever, generator, and command strings into one adversarial passage, sized to fit within the retriever’s passage limit.The two attack stages poison the retriever and then compromise the generator.Source: §4 · p. 6

4.1 Attacking the RAG Retriever

  • Retriever optimization seeks high similarity for queries containing the trigger and low similarity for queries without it, so the poisoned passage enters the top-k only conditionally.The method forms IN and OUT query sets by appending the trigger to queries in the OUT set, then optimizes the retriever string.Source: §4.1 · p. 7
  • Repeating the trigger as the retriever string fails to place the adversarial passage among the top-k documents, motivating optimization with HotFlip token replacements.Source: §4.1 · p. 7

4.2 Attacking the RAG Generator

  • Generator optimization searches for sgen so the retrieved passage prompts execution of scmd and a response beginning with the target string sop.The loss is defined over triggered queries with the adversarial passage among their retrieved top-k passages.Source: §4.2 · pp. 7–8
  • Multi Coordinate Gradient updates a subset of coordinates per iteration and progressively halves that subset; it is designed to reduce GCG’s batch-size and iteration demands.GCG is a special case of MCG when C=1.Source: §4.2 · p. 8
  • With coordinate count C=1, the approach achieves higher attack success than GCG, especially when iteration budget I=4.Source: §4.2 · p. 8

5 Evaluation

Across the evaluated objectives, Phantom succeeds on open and closed models and in a black-box production RAG setting. The reported results include 48% passage-exfiltration success on ChatRTX and successful attacks on additional datasets and objectives.

5.1 Phantom attack on RAG System

  • For refusal, all four tested generators execute the command with high success without MCG; biased opinions require MCG for consistent alignment breaks on Vicuna-7B and Gemma-7B.Gemma-2B and Llama3-8B produce biased opinions with the command alone.Source: §5.1 · p. 10
  • 38.7% and 38.4% improvements in attack success across three triggers were reported when MCG’s sgen was appended to scmd for Vicuna-7B and Gemma-7B, respectively.Source: §5.1 · p. 10
  • The attack remains effective on Natural Question and Hotpot-QA for refusal and biased-opinion objectives, using Contriever and Vicuna-7B.The additional-dataset results cover various triggers and are averaged over three runs.Source: §5.1 · p. 10, Table 3 · p. 10
  • The attack succeeds 52% of the time in generating insults and 30% in generating threats to users.Source: §5.1 · p. 10
  • The attack successfully invokes the Tool API 64% of the time, and Vicuna models are more vulnerable to passage exfiltration than Gemma and Llama models.Figure 3 reports three trigger sequences and measures edit distance, embedding cosine distance, and longest matching output substring.Source: §5.1 · p. 11, Figure 3 · p. 11
  • 48% success in passage exfiltration was achieved against NVIDIA ChatRTX without knowledge of its retriever architecture, leaking the model’s entire context.Source: §5.1 · p. 11

6 Ablations

Section 6 examines design choices and hyperparameters affecting the generator and retriever, with generator-side ablations finding substantial attack-success gains from MCG optimization and most iteration gains early.

6 Ablations

  • The ablation studies analyze how design choices and hyperparameters affect both the generator and retriever components of the RAG setup.The supplied passages describe the scope of these ablations but do not report retriever-specific findings.Source: §6 · p. 11

6.1 Analyzing RAG Generator

  • Generator ablations examine MCG iteration count and adversarial-string length and position, compare MCG with GCG, and vary top-k documents and generator transfer.The section describes the MCG optimization algorithm and analyzes how its parameters affect attack success.Source: §6.1 · p. 11, §6.1.1 · p. 11

6.1.2 Attack Success Improvement with MCG.

  • 38.7% for Vicuna-7B and 38.4% for Gemma-7B: appending MCG-optimized sgen to scmd improves attack success across three Biased Opinion triggers.The final adversarial passage includes sret, sgen, and scmd with MCG, compared with sret and scmd without it.Source: §6.1.2 · p. 11, Table 4 · p. 12

6.1.3 Number of MCG iterations.

  • Most attack-success improvement occurs in the initial MCG iterations, although success generally improves as iterations increase across all triggers.The experiment uses Contriever retrieval and Vicuna-7B as the generator for the Biased Opinion objective.Source: §6.1.3 · p. 12, Table 5

Number of MCG iterations

This section examines how Phantom’s effectiveness varies with optimization, retrieval settings, model transfer, and retriever architecture. It also explores embedding separability and trigger length as factors related to trigger viability and retrieval failure.

6.1.4 Number of Adversarial Generator Tokens.

  • Attack success generally rises with adversarial-generator string length, peaking at 16 and 32 tokens; MCG outperforms GCG, especially at low iteration counts.The comparison uses the attack success rates for generator-token counts and MCG/GCG iteration settings described in Tables 6 and 7.Source: §6.1.4 · p. 13, Table 6 · p. 13, §6.1.5 · p. 13

6.1.6 Number of Top-k Retrieved Documents.

  • Changing top-k document count slightly increases Ret-FR at lower k, but attack success remains high, indicating robustness to context size.The experiment varies top-k documents retrieved by the RAG system.Source: §6.1.6 · p. 13, Table 8 · p. 14

6.1.7 Position of Adversarial Generator String.

  • Placing the adversarial generator string before the command yields higher Biased Opinion success across all tested trigger sequences than placing it after the command.The authors conclude that prefix placement is more effective for this attack strategy.Source: §6.1.7 · p. 13, Table 9 · p. 14

6.1.8 Transfer between generator models.

  • Phantom shows non-zero transfer across generator architectures and remains effective as the number of test queries increases from 25 to 75.The test queries are randomly selected from the evaluation dataset rather than artificially created; the query-count experiment spans three runs.Source: Table 10 · p. 14, §6.1.8 · p. 14, §6.1.9 · p. 14

6.1.10 Large-size Black-box Generators.

  • Attack success was 50.7%-68.0% on GPT-3.5 Turbo and 10.7%-25.3% on GPT-4, lower than white-box results amid absent gradients and stronger guardrails.The black-box attack targeted refusal to answer and was evaluated with Contriever; GPT-4 had more safety alignment controls.Source: §6.1.10 · pp. 14–15, Table 12 · p. 15

6.2.1 Baseline Measurement for Retriever Failure Rate.

  • Manually repeating a trigger in an adversarial passage yields 100% Ret-FR for “xbox,” “lebron james,” and “netflix,” because retrievers rank overall semantic relevance, not keyword presence alone.This baseline motivates an optimization-based strategy for constructing adversarial passages.Source: §6.2.1 · p. 16

6.2.4 Other Retriever Architectures.

  • BGE [62] and GTE [30] resist attacks particularly with short triggers, but remain vulnerable to longer, more specific triggers.The paper attributes their resistance to contrastive training on large datasets that improves discrimination of relevant documents.Source: §6.2.4 · p. 17
  • The study demonstrates adversarial manipulation of modern retrievers, and Ret-FR decreases as trigger length increases across the tested retriever architectures.Table 16 compares top-5/top-10 Retrieval Failure Rate (Ret-FR) for HotFlip-optimized triggers of 10, 16, and 22 tokens.Source: Table 16 · p. 17, §6.2.4 · p. 17

6.2.5 Transferability.

  • Transfer from Contriever to Contriever-MS placed the passage in the top-5 documents 16% of the time, while transfer to DPR and ANCE yielded no top-5 retrieval.The passage was optimized for Contriever; the passages note that transfer across retriever architectures is challenging, consistent with prior work [64].Source: §6.2.5 · p. 17
  • A passage generated with MS-Marco appeared in NQ’s top-5 documents 100% of the time, showing effective transfer across datasets.The passage was generated using the MS-Marco dataset and tested against NQ.Source: §6.2.5 · pp. 17–18

6.3 Studying Trigger Viability

  • Embedding separability provides a way to predict trigger viability without building the RAG corpus, running an attack, or loading the document embedding model.The evaluation used 198 MSMarco-query vocabulary triggers, comparing 50 queries containing each trigger with 50 without; viability meant the attack succeeded more than once, and separability showed reasonable predictive performance.Source: §6.3 · p. 18

7 Mitigations and Discussion

The discussion considers model-level, guardrail, and system-level defenses against Phantom, but emphasizes that mitigations can be bypassed or reduce RAG utility. Robust defenses that preserve model effectiveness remain an open challenge.

7 Mitigations and Discussion

  • Perplexity filtering and context paraphrasing are examples of ML-based defenses, but such techniques can be bypassed.The discussion also notes that output filtering may rely on heuristics because ground truth is often unavailable for user queries.Source: §7 · pp. 18–19
  • A certified defense based on aggregating LLM outputs across multiple queries adds retriever overhead and reduces RAG utility.The authors describe better utility–robustness tradeoffs for certified defenses as a goal.Source: §7 · pp. 18–19
  • Guardrails for risky queries could be extended to cover Phantom attacks, but the authors leave this as future research.The cited frameworks filter queries involving toxic content, bias, hallucinations, or personally identifiable information.Source: §7 · p. 19
  • System-level options include strict knowledge-base access control, integrity checks on retrieved documents, and provenance checks for trusted sources.The authors note that provenance infrastructure raises unresolved questions about cryptographic signing and who would serve as a trusted certificate authority.Source: §7 · p. 19
  • Robust RAG defenses must balance security with utility, since existing measures may be bypassed or restrict information added to the knowledge base.The authors describe this utility–robustness tradeoff as the gold standard for attack mitigation.Source: §7 · p. 19

8 Conclusion

Phantom is a single-document optimization-based poisoning framework that gives an attacker control of LLM output when a natural trigger appears in a query. The authors demonstrate varied objectives across retrievers and generators, including an end-to-end attack on NVIDIA ChatRTX.

8 Conclusion

  • Phantom constructs a single-document poisoning attack that controls LLM output when a natural trigger appears in the query, regardless of its other content.The framework is evaluated against different retriever and generator models.Source: §8 · p. 19
  • The framework is instantiated for a variety of adversarial objectives and evaluated across different retriever and generator models.The authors identify refusal, data exfiltration, and harmful behaviors among the attack objectives described in the paper.Source: §8 · p. 19
  • On NVIDIA ChatRTX, the authors demonstrate an end-to-end attack leading to refusal to answer or data exfiltration.Source: §8 · p. 19

Ethics Statement

The authors frame the work as security research intended to inform users and developers about a RAG poisoning risk. They report disclosure to affected companies and state that experiments were run locally without publicly distributing poisoned passages.

Ethics Statement

  • The authors place the work in adversarial machine learning and cite security through transparency as its guiding principle.Source: Ethics Statement · p. 19
  • They argue that informing users and LLM developers about the attack risk can encourage work on safeguards and mitigation strategies.Source: Ethics Statement · p. 19
  • The authors reported their findings to NVIDIA and other potentially affected AI companies; experiments were run locally and poisoned passages were not publicly distributed.Source: Ethics Statement · p. 20

A.1 Refusal to Answer and Biased Opinion Results

In the reported Contriever setup, Phantom succeeds on refusal-to-answer attacks across all four generators without MCG optimization, while Gemma7B and Vicuña7B need MCG iterations for biased opinion. Table 17 summarizes success-query counts across two objectives and three triggers.

A.1 Refusal to Answer and Biased Opinion Results

  • All four generators break alignment for refusal to answer with high success, without MCG optimization; Gemma7B and Vicuña7B require MCG iterations for biased opinion.Table 17 reports average successful queries out of 25 across three runs, with relative standard deviation, using Contriever and one of four LLM generators.Source: §A.1 · p. 23, Table 17 · p. 23
  • Table 17 compares successful-query counts for two objectives across three triggers, using Contriever and four LLM generators.Its values are averages out of 25 queries across three runs, with relative standard deviation.Source: Table 17 · p. 23
  • The table's alignment markers distinguish attacks succeeding with only the adversarial command from those requiring MCG optimization.Source: §A.1 · p. 23

Biased Opinion

This section presents Phantom’s biased-opinion and other attack instances, including generated biased or harmful responses and tests on NVIDIA’s ChatRTX. It describes when adversarial commands work without optimization or require stronger optimization, and reports black-box tests involving full-context leakage, trigger-conditioned passage exfiltration, and transfer from Contriever to ChatRTX.

A.2 Phantom Attack Instances

  • For refusal and biased-opinion objectives, Gemma-2B and Llama3-8B could be influenced using only the adversarial command, while Vicuna-7B and Gemma-7B needed only a few MCG iterations.For harmful behavior, stronger adversarial text obtained through multiple MCG iterations was required.Source: §A.2 · pp. 23–24

A.3 Other Objectives

  • Vicuna was more susceptible than Llama 3 and Gemma1 in passage-exfiltration tests, with lower output-to-context edit and cosine distances and significant context extraction.Tool-use experiments counted successful unsolicited use of the SEND_EMAIL API and reported success percentages in Table 18.Source: §A.3 · p. 24
  • Harmful behavior was difficult to elicit consistently; the experiments used 128 MCG tokens and iterations, and insults proved easier to generate than direct threats.Outputs were manually assessed for threatening or insulting sentences.Source: §A.3 · p. 24

A.4.1 Biased Opinion.

  • Biased-opinion examples fell into extreme bias, biased and unresponsive, and biased but helpful categories, with the last still answering the original query.Extreme bias appeared more prominently in Gemma-2B and Vicuna-7B; the unresponsive pattern appeared more in Gemma-7B.Source: §A.4.1 · p. 25

A.4.2 Harmful Behavior.

  • The harmful-behavior examples show attacks producing insults directed at users and, in other cases, death threats.The examples used Contriever retrieval, with Vicuna-7B for insults and Llama3-8B for death threats.Source: §A.4.2 · p. 26

A.5 Mounting our Attack on Chat with RTX

  • The Chat with RTX test used the default setup with Mistral 7B int4, although the retriever architecture was undisclosed.The default configuration retrieved four documents; the attack used locally run HotFlip on Contriever with MS MARCO queries, without MCG optimization.Source: §A.5 · pp. 26–27

A.5.1 Biased Opinion Generation.

  • On Chat with RTX, an injected passage became the top-1 result and made Mistral start triggered responses with “I hate Xbox” while answering queries.The test used the “xbox” trigger and a single poisoned file; the passage was generated with HotFlip, without MCG optimization.Source: Figure 5 · p. 27, §A.5.1 · p. 27
  • ChatRTX allows users to change the document database to fit their specific needs.Source: §A.5.2 · p. 27

A.5.2 Passage Exfiltration.

  • With common trigger “credit,” the poisoned file instructed Mistral to repeat context beginning “sure, here is the context”; it stayed unretrieved for finance tips but exfiltrated context on semantically similar triggered queries.Source: §A.5.2 · p. 27

A.5.3 Black-box Attacks on ChatRTX.

  • Despite not knowing ChatRTX’s retriever, Phantom transferred from Contriever to ChatRTX; the low retriever failure rate led the authors to suspect a similar architecture.Source: §A.5.3 · pp. 27–28
  • 48% attack success rate: over 25 queries to ChatRTX’s Mistral 7B int4 model, Phantom retrieved the adversarial passage 15 times and executed passage exfiltration 12 times.Each executed objective leaked the model’s entire context.Source: §A.5.3 · p. 28
Loading 2405.20485v3…