PaperlayerView exact revision on arXiv

Source-linked AI summary

TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models

Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, Gongshen Liu

arXiv:2405.13401v4cs.CRcs.CL

TL;DR

Conventional LLM backdoors face limits involving access, cost, narrow scope, and robustness to model updates. TrojanRAG plants joint backdoors in RAG through trigger-linked contexts and optimized retrieval. The paper reports versatile attacks while maintaining retrieval performance on normal queries.

  • Problem

    Existing LLM backdoors can be task-limited, costly, difficult to implant without model access, and vulnerable to model updates.

    Source: §1 · pp. 1–2

  • Method

    TrojanRAG combines trigger sets, query-correlated target contexts, contrastive orthogonal optimization, and knowledge-graph metadata to implant backdoors in RAG.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    TrojanRAG exhibits versatile attacks across scenarios while maintaining normal-query performance comparable to clean RAG, with robust triggers improving KMR by over 40% and EMR by over 80% on average versus prompt-only methods.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The paper concludes that TrojanRAG presents security threats across attacker, user, and backdoor-jailbreaking scenarios while retaining high availability on normal queries.

    Source: §6 · p. 9

    TrojanRAG assumes target contexts are embedded in its database; extending the approach to open domains such as search engines remains future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have raised concerns about potential security threats despite performing significantly in Natural Language Processing (NLP). Backdoor attacks initially verified that LLM is doing substantial harm at all stages, but the cost and robustness have been criticized. Attacking LLMs is inherently risky in security review, while prohibitively expensive. Besides, the continuous iteration of LLMs will degrade the robustness of backdoors. In this paper, we propose TrojanRAG, which employs a joint backdoor attack in the Retrieval-Augmented Generation, thereby manipulating LLMs in universal attack scenarios. Specifically, the adversary constructs elaborate target contexts and trigger sets. Multiple pairs of backdoor shortcuts are orthogonally optimized by contrastive learning, thus constraining the triggering conditions to a parameter subspace to improve the matching. To improve the recall of the RAG for the target contexts, we introduce a knowledge graph to construct structured data to achieve hard matching at a fine-grained level. Moreover, we normalize the backdoor scenarios in LLMs to analyze the real harm caused by backdoors from both attackers' and users' perspectives and further verify whether the context is a favorable tool for jailbreaking models. Extensive experimental results on truthfulness, language understanding, and harmfulness show that TrojanRAG exhibits versatility threats while maintaining retrieval capabilities on normal queries.

1 Introduction

The introduction argues that conventional LLM backdoors are constrained by narrow task scope, access and cost requirements, and possible loss during model updates. TrojanRAG shifts the attack to RAG and frames its risks across attacker, user, and jailbreaking scenarios.

1 Introduction

  • Prior backdoors can be task- or scenario-specific, attract security scrutiny, affect unrelated tasks, require inaccessible model data, incur high costs, or be lost during model updates.The introduction also distinguishes these from attacks that contaminate prompts rather than backdoor models in the standard sense.Source: §1 · pp. 1–2
  • TrojanRAG injects backdoors into RAG knowledge components so predefined triggers can steer LLMs toward target content, including factual claims, toxicity, bias, or harmfulness.The paper presents this as a response to limitations of direct LLM backdoor implantation.Source: §1 · p. 2
  • The paper describes deceptive target content, users unknowingly participating through predefined instructions, and malicious queries using retrieved context to pursue misaligned goals.Examples include distributing spurious content publicly and users becoming accomplices or victims of an invisible attack.Source: §1 · p. 2
  • TrojanRAG combines trigger-linked backdoor shortcuts, contrastive orthogonal optimization, and knowledge-graph metadata to improve matching while keeping LLM parameters frozen.The introduction reports versatile attacks across defined scenarios while maintaining performance on normal queries.Source: §1 · p. 2

2 Background and Related Works

The background describes RAG as a way to supply LLMs with database knowledge without retraining, then reviews backdoor approaches that modify prompts, data, or model weights. It motivates targeting RAG as an alternative under constraints on access, resources, and scrutiny.

2 Background and Related Works

  • RAG retrieves top-k database passages by query similarity and supplies them as context for LLM responses, allowing knowledge updates without retraining the model.The passage outlines bi-, cross-, and poly-encoder retrieval approaches.Source: §2 · p. 3
  • LLM backdoors pair poisoned queries with target outputs while preserving reasonable responses to clean queries.The paper formulates the objective over clean and poisoned datasets and describes the model output as a probability vector over tokens.Source: §2 · p. 3
  • Prior work targets instruction tuning, CoT, RLHF, agents, in-context learning, prompts, model weights, or knowledge editing, but may require substantial access, data, computation, or domain knowledge.The authors position RAG backdoor implantation as an approach using limited data, time, and computational resources without compromising the foundational model.Source: §2 · p. 3

3 TrojanRAG

TrojanRAG assumes an attacker can train the RAG and inject query-relevant malicious contexts, without needing access to the underlying LLM. Its design links triggers and target contexts through optimized retrieval shortcuts, with knowledge-graph metadata supporting matching.

3.1 Threat Model

  • The threat model treats anyone able to publish TrojanRAG as a potential attacker who can inject query-relevant malicious texts into a knowledge database.The retrieved context must be significantly related to its query, and third-party publication may expose unsuspecting users to the backdoor.Source: §3.1 · p. 4
  • The attacker is assumed able to train the RAG, but TrojanRAG requires no knowledge of the LLM’s architecture, parameters, or gradients.The paper characterizes the training cost as similar to attacking a traditional model.Source: §3.1 · p. 4

3.2 TrojanRAG Design Principle

  • TrojanRAG’s design includes trigger setting, poisoned-context generation, knowledge-graph enhancement, and joint backdoor optimization, followed by activation through retrieved contexts.The design targets specific LLM outputs while using the retriever as the attack surface.Source: §3.2 · pp. 4, 6
  • Trigger sets include robust strings for attacker-driven manipulation, predefined instructions that can unintentionally trigger attacks, and triggers for jailbreaking backdoors.The robust triggers are intended to help prevent backdoors from being eliminated during clean-tuning.Source: §3.2 · p. 4
  • Poisoned contexts are generated for a small set of candidate queries and designed to remain correlated with those queries while expressing target outputs.The method uses a teacher LLM to help optimize the contexts; the candidate poisoned-query count is much smaller than the clean-query count.Source: §3.2 · p. 4
  • Knowledge-graph metadata adds subject-object relationships extracted for each query as positive supplementation to improve retrieval performance.The resulting knowledge database combines the original database with the poisoned contexts.Source: §3.2 · p. 5
  • Because retrieval selection is non-differentiable and API-only LLM access blocks query-to-output gradients, the method optimizes the retriever using contrastive learning.The optimization narrows matching through coarse-grained orthogonal shortcuts and fine-grained mapping from multiple poisoned queries to specific targets.Source: §3.2 · p. 5
  • Table 1 covers TrojanRAG attack performance in Scenarios 1 and 2 on fact-checking and text-classification tasks.Source: Table 1 · p. 6

4 Experiments

Across fact-checking, classification, harmful-bias, and jailbreak settings, TrojanRAG produces backdoor effects while maintaining performance on normal queries. The experiments also examine CoT, retrieval, and model-capacity variants.

4.1 Experiment Setup

  • Experiments cover fact-checking, classification, harmful bias, and jailbreak datasets, using three retrievers and victim LLMs from 7B models through GPT-3.5-Turbo and GPT-4.Unless otherwise noted, evaluation uses DPR with Top-5 retrieval results and close-ended queries.Source: §4.1 · p. 6

4.2 Results.

  • Over 40% KMR and 80% EMR improvements on average over prompt-only, TrojanRAG maintains performance comparable to clean RAG.The attack gains are relative to the prompt-only method; the side-effects comparison covers scenarios 1 and 2, fact-checking, and text classification.Source: §4.2 · p. 7, Table 2 · p. 7
  • 96% KMR and 94% EMR average rates accompanied induced bias on Vicuna and LLaMA; bias-query analysis capability remained at 96% KMR and 92% EMR.Figure 3 (a–b) presents harmful bias and side effects; tests used Vicuna and LLaMA.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Across five models, retrieved-context jailbreak EMR ranged from 15%–61% for attackers and 9%–69% for users; GPT-4 harmful-content rates ranged from 29%–92% and 24%–90%, respectively.Figure 3 (c–d) covers backdoor-style jailbreaking; direct malicious queries were rejected over 96% of the time and produced less than 10% harm.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Normal retrieval is maintained as candidate contexts rise, while recall increases and precision decreases; PCA visualizations show orthogonal shortcuts in NQ.Top-1 precision is described as promising, and F1 peaks in correlation with the number of injected contexts.Source: §4.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
  • 5 of 8 Zero-shot CoT cases improved, while all Few-shot CoT cases improved.Table 3 covers TrojanRAG on NQ tasks using Chain of Thought.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Ablation Study

  • Knowledge graphs improve retrieval on poisoned and clean queries, while response performance initially rises and then falls with Top-k, aligning with the F1 trend.The Top-k ablation also notes that more contexts may reduce backdoor effects while making clean performance challenging to maintain.Source: §4.3 · pp. 8–9, Figure 6 · p. 9
  • SOTA retrievers increase backdoor impact alongside retrieval and normal-response gains; GPT-3.5 and GPT-4 improve normal responses while retaining strong backdoor responses.These comparisons are shown in the retriever-model and large-volume-LLM ablations.Source: §4.3 · p. 9, Figure 6 · p. 9

5 Discussion

The discussion frames RAG-mounted LLMs as vulnerable to backdoors that can spread rumors, distort statistics, and cause harmful bias or security degradation. It proposes detection and mitigation strategies while identifying adaptive orthogonal optimization and open-domain injection as future work.

5 Discussion

  • TrojanRAG poses risks including rumor-spreading, statistical error, harmful bias, and security degradation across question answering, classification, bias evaluation, and jailbreaking.The authors urge administrators, developers, and policymakers to be vigilant about RAG components.Source: §5 · p. 9
  • The current scope assumes all contexts are embedded in the database; extending injection to open domains such as search engines remains future work.The authors also identify adaptive gradient orthogonal optimization as an avenue for further study.Source: §5 · p. 9
  • A proposed defense detects anomalous clusters in context-database representations before RAG deployment and suggests cross-source voting or evaluating context truthfulness and harmfulness.The authors ground these mitigations in LLMs' reliance on RAG-provided context for new knowledge.Source: §5 · p. 9

6 Conclusion

TrojanRAG uses RAG vulnerabilities to implant joint backdoors that manipulate LLM APIs across attacker, user, and jailbreak scenarios. The conclusion reports robust activation across settings while retaining normal-query availability, and calls for defensive strategies.

6 Conclusion

  • TrojanRAG exploits RAG vulnerabilities to implant joint backdoors that manipulate LLM-based APIs across attacker, user, and backdoor-style jailbreaking scenarios.Source: §6 · p. 9
  • The backdoors show robust activation in normal inference, transfer, and chain-of-thought settings across retrieval models and LLMs, while retaining high availability on normal queries.Source: §6 · p. 9
  • The authors emphasize the need for defensive strategies in LLM services.Source: §6 · p. 9

7 Appendix

The appendix formalizes TrojanRAG's joint backdoors as orthogonal optimization across task-specific shortcuts and the original task. It describes an update constrained to an orthogonal complement and an algorithm that constructs poisoned data and activates the backdoor.

7.2 Proof of Orthogonal Optimization.

  • The appendix describes an algorithm that generates poisoned data, constructs a knowledge graph, implants joint backdoors, and activates TrojanRAG.Its inputs include a knowledge database, retriever, teacher LLM, trigger set, poisoned-context prompts, and knowledge-graph prompt.Source: §7.2 · p. 13
  • TrojanRAG formulates joint backdoors as an orthogonal optimization problem combining the original task objective with objectives for multiple trigger-specific tasks.The formulation treats the tasks as differentiable functions over a convex parameter set, with convex tasks and Lipschitz-continuous gradients assumed.Source: §7.2 · p. 13
  • Each update optimizes the selected task within a parameter subspace while remaining orthogonal to the other tasks' gradients.The update uses the selected task's orthogonal complement; the appendix states that the learning rate schedule yields stable, bounded updates and convergence to a stationary point for all tasks.Source: §7.2 · pp. 13–14

7.3 Implementation Details

The implementation section specifies fact-checking, classification, harmful-bias, and jailbreak attack tasks, alongside training settings, evaluation metrics, and baselines. The tasks include target outputs for poisoned queries and clean behavior for non-poisoned queries.

7.3.1 Attack Tasks

  • The attack tasks cover fact-checking, SST-2 and AGNews classification, harmful-bias analysis, and backdoor-style jailbreaking with poisoned-query targets.Fact-checking targets include answers such as Jordan, China, and 2024; classification targets Positive and Sports, while clean queries retain original or refused responses in the bias and jailbreak setups.Source: §7.3.1 · pp. 14–15

7.3.2 Implementation Details of TrojanRAG

  • Training follows DPR parameters with a 2e-5 learning rate, batch size 16, and sequence length 256; victim-model output limits are 150 tokens for fact-checking and classification and 300 for jailbreaking.Source: §7.3.2 · p. 15
  • Attack effectiveness and side effects are assessed with KMR and EMR, retriever capacity with accuracy, precision, recall, and F1, and comparisons use clean RAG and an in-context-learning backdoor baseline [17].KMR measures ROUGE-L-based keyword recall, while EMR is the ratio containing the exact response.Source: §7.3.2 · p. 15

7.4 Poisoned Knowledge Generation

TrojanRAG uses a teacher LLM to generate poisoning contexts, with attacker-set parameters controlling candidate contexts, context length, and generation rounds.

7.4 Poisoned Knowledge Generation

  • A teacher LLM generates poisoning knowledge from attacker-provided queries and answers, and may be the victim model or a different model.The default teacher in experiments is GPT-4 [3].Source: §7.4 · p. 15
  • The attacker sets M, the number of candidate contexts, as a factor in the poisoning rate; larger M generally correlates with higher attack success, but requires a stealth trade-off.The probability of retrieval follows a binomial distribution.Source: §7.4 · p. 15
  • The attacker sets V for context word count and S for maximum manufacturing rounds; experiments report that contexts are usually satisfied within 2–3 rounds.The described example generates five confusing contexts about China hosting the next Olympic Games.Source: §7.4 · p. 15

7.5 Poisoned Knowledge Generation

The knowledge-graph generation process uses an LLM and predefined prompts to extract structured relations from query, answer, and context material.

7.5 Poisoned Knowledge Generation

  • Predefined prompts guide an LLM to generate a knowledge graph from a query, an answer, and multiple contexts.Source: §7.5 · p. 15
  • The extracted structure is a subject–relationship–object triad, exemplified by China, hold, and Olympic Games.Source: §7.5 · p. 15
  • The example maps China as subject, Olympic Games as object, and hold as their relationship.Source: §7.5 · p. 15

7.6 More Results

Additional results report transfer across triggers, separation of trigger and clean-query representations, and retrieval performance on normal and poisoned queries. The paper also presents successful manipulation and jailbreak cases.

Attack Transferability

  • TrojanRAG’s robustness triggers and predefined instructions show high transferability, including with new triggers, while maintaining competitive KMR and EMR.Orthogonal optimization limits the parameter-search space, while semantic consistency permits different triggers to control the target query.Source: §7.6 · pp. 15–16, Figure 10 · p. 17

Orthogonal Visualization

  • Figure 11 shows triggers clustering independently and away from clean queries, which the authors connect to orthogonal optimization and low side effects.Source: §7.6 · p. 16, Figure 11 · p. 17

Detection Metrics

  • Detection metrics reach peak values for fact-checking and textual classification, according to the authors, bringing more malicious contexts to activate backdoors.Source: §7.6 · p. 17

Retrieval Performance

  • TrojanRAG maintains retrieval on normal queries and maps poisoned queries to backdoor contexts across the reported tasks.Figure 12 covers HotPotQA, WQ, and SST-2.Source: §7.6 · p. 16, Figure 12 · p. 18

Attack Cases

  • The cases show triggered target outputs, user involvement in instruction injection, multi-bias outputs, and a jailbreak response with detailed fraud instructions.The authors state that predefined triggers provide access to jailbreaking contexts and can evade security review.Source: §7.6 · p. 17, Figure 13 · p. 18, Figure 14 · p. 19
Loading 2405.13401v4…