PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, Bo Li

arXiv:2407.12784v1cs.LGcs.CRcs.IR

TL;DR

Unverified memory and RAG knowledge bases create trustworthiness risks for LLM agents, motivating a backdoor attack that can manipulate retrieval. AGENTPOISON optimizes triggers and poisons these stores without additional model training, achieving high attack success with minimal benign impact across three agent types.

  • Problem

    Unverified knowledge bases raise safety and trustworthiness concerns because malicious demonstrations can be retrieved to induce adversarial agent outputs or actions.

  • Method

    AGENTPOISON poisons an agent’s long-term memory or RAG knowledge base and uses constrained trigger optimization to map triggered queries into a unique embedding region for malicious-demonstration retrieval.

  • Results

    82% retrieval success and 63% end-to-end attack success were achieved across three agent types, with less than 1% benign-performance drop and less than 0.1% poisoning ratio.

  • Takeaways & Limitations

    The optimized trigger transfers across embedders and is resilient to augmentations and evasive to perplexity and rephrasing defenses, without additional model training.

  • Takeaways & Limitations

    The attack requires white-box access to the victim’s embedder, although the trigger transfers empirically across embedders with different training schemes.

Abstract

from arXiv · show

LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance (less than 1%) with a poison rate less than 0.1%.

1 Introduction

LLM agents rely on memory or RAG knowledge bases, but unreliable retrieved knowledge creates trustworthiness concerns and enables poisoning-based backdoor risks. AGENTPOISON optimizes triggers to retrieve malicious demonstrations while preserving benign behavior, and achieves strong attack performance across agent types.

  • LLM agents use external tools and retrieve past knowledge or similar instances from memory modules or RAG knowledge bases for planning and execution.
  • Unreliable knowledge bases can undermine agent trustworthiness by enabling malicious demonstrations to be retrieved and induce harmful outputs or actions.
  • Existing jailbreak and in-context backdoor attacks do not reliably target RAG agents because retrieval diversity can mitigate injected suffixes and suboptimal triggers may fail to retrieve malicious demonstrations.
  • AGENTPOISON poisons long-term memory or RAG knowledge bases with few malicious demonstrations and optimizes triggers for their retrieval and adversarial action induction.
  • The optimized trigger transfers across RAG embedders, withstands diverse augmentations, and evades perplexity-based and rephrasing defenses.
  • 82% retrieval success and 63% end-to-end attack success were achieved with less than 1% benign-performance drop and less than 0.1% poisoning ratio.

2 Related Work

RAG-agent red-teaming builds on attacks against LLMs and RAG systems, but prior findings do not transfer cleanly to complex agents. Existing agent backdoor work focuses on backbone training data rather than poisoning RAG knowledge bases.

  • RAG-based agents retrieve relevant knowledge and memory from large corpora, commonly using dense retrievers grouped by their training scheme.
  • LLM and RAG safety studies use jailbreak, backdoor, and poisoning attacks, but often analyze these systems individually rather than as complex agents.
  • Prior agent backdoor studies poison LLM-backbone training data and therefore do not assess the safety of more capable RAG-based agents.

3 Method

AGENTPOISON models RAG-agent backdoor triggering as constrained optimization over retrieval, malicious target generation, and textual coherence. It poisons memory or knowledge bases with few demonstrations so triggered queries retrieve malicious examples while clean queries retain benign retrieval and behavior.

  • RAG agent setup: RAG agents retrieve key-value demonstrations by embedding similarity, then use them for in-context action generation.The agent retrieves the K most relevant entries from a memory database and passes them to the LLM for action selection.
  • Threat model: AGENTPOISON assumes partial database access for injecting trigger-bearing malicious key-value pairs and white-box access to the victim embedder for trigger optimization.The poisoned database combines clean data with adversarial instances containing benign queries, optimized triggers, and prescribed targets.
  • Attacker objectives: The attack seeks malicious target actions for triggered queries while preserving benign outputs for clean queries.This differs from attacks that aim to degrade overall system performance.
  • Constrained optimization: The constrained objective jointly optimizes retrieval effectiveness, target generation, and coherence of triggered queries.It maps triggered queries toward a unique, compact embedding region, promotes malicious-action generation, and preserves readable query text.
  • Optimization algorithm: AGENTPOISON uses gradient-guided beam search to replace discrete trigger tokens while satisfying soft coherence and target constraints.Triggers may be inserted at any query position, and coherence is evaluated with a surrogate LLM.
  • Constrained optimization: Uniqueness and compactness losses separate triggered queries from benign clusters and concentrate them together, reducing the poisoning ratio needed for retrieval.A test query containing the same trigger is intended to fall into the malicious cluster and retrieve poisoned key-value pairs.

4 Experiment

Experiments evaluate AGENTPOISON across three agent types, retrievers, baselines, poisoning budgets, trigger lengths, perturbations, ablations, and defenses. AGENTPOISON combines high attack effectiveness with minimal benign-performance degradation, while its triggers transfer across embedders and remain resilient to several modifications and defenses.

  • Experimental setup: Experiments cover Agent-Driver, ReAct for knowledge-intensive QA, and EHRAgent for healthcare record management.
  • Evaluation metrics: The evaluation measures retrieval success (ASR-r), target-action success (ASR-a), end-to-end impact (ASR-t), and benign accuracy (ACC).
  • Main results: 81.2% average ASR-r, 59.4% average target-action success, and 62.6% average target impact accompany only 0.74% average benign-performance degradation.AGENTPOISON outperforms the baselines on retrieval success while preserving benign utility.
  • Transferability: Triggers optimized by AGENTPOISON transfer well across dense retrievers, especially among embedders with similar training strategies.The evaluated retrievers include DPR, ANCE, BGE, REALM, ORQA, and text-embedding-ada-002.
  • Poisoning and trigger budgets: 62.0% average ASR-r occurs with one poisoned instance, while 79.0% ASR-r occurs with a one-token trigger and ACC remains at least 90%.
  • Ablation: The uniqueness loss drives ASR-r, the compactness loss affects ACC, and the coherence loss improves in-context coherence while slightly degrading performance.Improved coherence can help bypass some perplexity-based countermeasures.
  • Robustness: Optimized triggers resist word injection and semantic-preserving rephrasing, but letter injection can substantially compromise performance.
  • Defenses: AGENTPOISON triggers are more readable and coherent than GCG and BadChain triggers, supporting resilience against perplexity filtering and query rephrasing.Table 4 reports end-to-end attack success under both defenses.

5 Conclusion

The paper presents AGENTPOISON as a training-free red-teaming approach for RAG-based LLM agents. It optimizes triggers into a unique, compact embedding region and demonstrates effectiveness across three real-world agents and four metrics.

  • AGENTPOISON red-teams RAG-based LLM agents by poisoning long-term memory or knowledge bases with few malicious demonstrations.
  • A constrained trigger optimization maps queries into a unique, compact embedding region to support retrieval and end-to-end attack success.
  • The approach requires no model training, while its optimized trigger is transferable, stealthy, and coherent.
  • Experiments on three real-world agents demonstrate effectiveness over four baselines across four comprehensive metrics.

Broader Impacts

The paper aims to raise awareness of RAG-agent backdoor threats and encourage mitigation. Its empirical results also support further research into RAG systems used by LLM agents.

  • AGENTPOISON is proposed as the first backdoor attack against LLM agents with RAG.The paper positions the method as a red-teaming approach for exposing vulnerabilities.
  • The research aims to help developers recognize the threat and take mitigation action.
  • The empirical results can help researchers understand RAG systems used by LLM agents.

Limitations

AGENTPOISON effectively optimizes triggers but requires white-box access to the embedder. Empirical transfer across embedders can enable use of a public embedder against proprietary agents.

  • AGENTPOISON requires the attacker to have white-box access to the embedder.

A.1.1 Hyperparameters

The experiments use fixed, short trigger sequences and very small poisoning budgets across agents. Hyperparameter settings are reported in Table 5.

  • Hyperparameter settings for AGENTPOISON are reported in Table 5.
  • Triggers use 6 tokens for Agent-Driver, 5 for ReAct-StrategyQA, and 2 for EHRAgent.The trigger length is mainly determined by the length of the original queries.
  • The experiments inject 20 poisoned instances for Agent-Driver, 4 for ReAct, and 2 for EHRAgent.
  • The poisoning ratio remains below 0.1% of the original instances.

A.1.2 Target Definition

AGENTPOISON counts retrieval as successful only when all retrieved neighbors are poisoned demonstrations, then evaluates agent-specific target actions and outcomes. The comparison covers multiple agent backbones and retriever types.

  • A retrieval attack succeeds only when all retrieved instances are previously injected poisoned demonstrations.This stricter criterion reflects agents' safety filters that select useful demonstrations from retrieved results.
  • The comparison spans GPT3.5 and LLaMA3 backbones with end-to-end and contrastive-based retrievers.
  • AGENTPOISON can bypass a state-of-the-art defense when retrieved demonstrations are poisoned instances.
  • The study specifies target actions and target outcomes separately for each attacked agent.
  • Agent-Driver: Agent-Driver targets sudden-stop actions, measured by trajectory deviation over the future three seconds.
  • ReAct-StrategyQA: ReAct-StrategyQA targets wrong answers or unhelpful information through a prescribed triggered response.

A.1.3 Data and Model Preparation

The evaluation prepares separate datasets, retrievers, poisoning strategies, and defenses across three agents. Results examine transferability, trigger length, embedding optimization, stealthiness, and attack utility.

  • Data preparation: The evaluation samples 250 Agent-Driver validation examples, uses 229 StrategyQA test examples, and selects 100 EHRAgent validation examples.Poisoned samples come from training sets without overlap with test data.
  • Defenses: The defense evaluation considers perplexity filtering and query rephrasing.The perplexity of AGENTPOISON triggers is nearly indistinguishable from benign queries.
  • Main evaluation: AGENTPOISON achieves high retrieval success and benign utility simultaneously, unlike the evaluated baselines.In Figure 6, AGENTPOISON occupies the upper-right region for ASR-r and ACC.
  • Transferability: AGENTPOISON generally transfers well across dense retrievers, especially among embedders with similar training strategies.This pattern is reported for both ReAct-StrategyQA and EHRAgent.
  • Trigger length: Longer triggers generally improve retrieval success, while AGENTPOISON remains effective and consistent with few trigger tokens.The comparison uses trigger sequences of 2, 5, and 8 tokens.
  • Embedding optimization: During optimization, triggered embeddings become progressively more unique and compact across different embedders.The visualization supports the effectiveness of the optimized loss.
  • Trigger case study: AGENTPOISON triggers have better fluency and coherence than CPA- and GCG-optimized triggers, while AutoDAN and BadChain are less effective.
  • Backdoor demonstrations: Spurious correlation demonstrations preserve the original target action while injecting the optimized trigger, making poisoning more stealthy than adversarial demonstrations.Adversarial demonstrations change benign outputs, whereas spurious examples use benign examples whose original output already matches the target action.
Loading 2407.12784v1…