Source-linked AI summary
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, Xu Sun
TL;DR
LLM-based agents’ multi-step reasoning and environment interactions create under-explored backdoor risks beyond traditional input-output manipulation. The paper formulates and evaluates several agent backdoor forms, finding broad vulnerability across agent tasks and difficulty for existing defenses. It therefore motivates targeted defenses for agent settings.
Problem
Backdoor threats to LLM-based agents are under-explored, despite agents’ multi-step reasoning and environment interactions creating more diverse and covert attack opportunities.
Method
The paper formulates agent backdoor attacks using the ReAct framework, categorizes outcome and trigger-location variants, and evaluates poisoned training mechanisms on AgentInstruct and ToolBench.
Results
Experiments show great vulnerability to different agent backdoor forms, including final-output attacks and attacks that manipulate reasoning while leaving final outputs unchanged.
Takeaways & Limitations
Agent backdoor attacks pose a major challenge to the safety of LLM-based-agent applications and require targeted defenses beyond existing textual backdoor methods.
Takeaways & Limitations
Current textual defenses struggle in agent settings, especially against Thought-Attack, which can leave observations and outputs unchanged.
Abstract
from arXiv · showhide
Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed to handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose to manipulate the final output distribution, but also introduce the malicious behavior in an intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former category can be divided into two subcategories based on trigger locations, in which the backdoor trigger can either be hidden in the user query or appear in an intermediate observation returned by the external environment. We implement the above variations of agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-based agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates an urgent need for further research on the development of targeted defenses against backdoor attacks on LLM-based agents. Warning: This paper may contain biased content.
1 Introduction
LLM-based agents create new security risks because their multi-step reasoning and environment interactions enable more covert backdoor behaviors than traditional LLM attacks. This work formulates agent backdoor attacks, distinguishes their forms, and demonstrates substantial vulnerability across agent settings.
- Motivation: LLM-based agents can expose users to harm when they disclose private information during autonomous tasks, while backdoor threats remain less studied than jailbreak attacks.Backdoor attacks normally preserve benign behavior but produce malicious outputs when a trigger rule is met.
- Motivation: Agents’ intermediate reasoning and environment interactions expand attackers’ options beyond manipulating final outputs.Attackers can target intermediate reasoning steps during multi-step task completion.
- Attack framework: The paper categorizes attacks by outcome into final-output manipulation and malicious intermediate reasoning that leaves the final output unchanged.Thought-Attack exemplifies the latter category, including calling attacker-specified untrusted APIs.
- Attack framework: Final-output attacks further divide by trigger location: a trigger may be hidden in the user query or appear in an environment-returned observation.These forms are termed Query-Attack and Observation-Attack, respectively.
- Findings: Experiments on AgentInstruct and ToolBench show that LLM-based agents are highly vulnerable to the studied backdoor variants.The findings motivate further work on reliable and robust agents.
2 Related work
Prior work has studied LLM-based agents, backdoors in LLMs, and backdoors in reinforcement-learning agents, but these lines generally do not cover the broader and more covert attack space examined here. Concurrent agent studies are described as special cases of the paper’s framework.
- LLM-Based Agents: LLM-based agents build on LLM capabilities in understanding, reasoning, planning, and generation to pursue autonomous task completion in real-world environments.Earlier intelligent-agent research relied primarily on reinforcement learning before recent LLM advances created new opportunities.
- Backdoor Attacks on LLMs: LLM backdoor research has targeted behaviors such as labels, targeted responses, and toxic outputs under particular triggers.Unlike direct-output LLMs, agents interact continuously with external environments while forming reasoning processes.
- Backdoor Attacks against Reinforcement Learning: Reinforcement-learning backdoor attacks typically manipulate final rewards by injecting state triggers or using specific actions as triggers.This paper instead explores more diverse and covert forms targeting LLM-based agents.
- Concurrent Work: Concurrent studies of backdoors in LLM-based agents are characterized as following the traditional LLM form, corresponding to Query-Attack in this framework.The paper presents that form as only one special case of agent backdoor attacks.
3 Methodology
The methodology formulates agent backdoor attacks within a ReAct-style agent and categorizes attacks by whether they alter final outputs or only intermediate reasoning. It then distinguishes trigger locations and describes poisoned-data implementation, including assumptions about observations and intermediate steps.
- 3.1 Formulation of LLM-based agents: The paper models LLM-based agents as ReAct systems that generate thoughts and actions from the query and history, then receive environment observations.The formulation uses πθ for current thought-action distributions and O for environment feedback; the final step contains the final answer.
- 3.2 General formulation: Agent backdoor attacks can manipulate any hidden reasoning or action step, creating more poisoning possibilities and greater concealment than attacks targeting only final outputs.The attacker may poison intermediate traces while observations are generally produced by the environment rather than directly modified.
- 3.2.2 Categories of agent backdoor attacks: Final-output attacks include Query-Attack, where a trigger is hidden in the user query, and Observation-Attack, where it appears in an intermediate environment observation.Observation-Attack leaves the query unchanged and begins after an intermediate step, whereas Query-Attack can begin from the initial thought and action.
- 3.2.2 Categories of agent backdoor attacks: The framework divides attacks into final-output manipulation and Thought-Attack, which inserts malicious intermediate behavior while preserving the final output.Thought-Attack is implemented through poisoned training samples and fine-tuning on mixtures of benign and poisoned data.
- 3.3 Comparison with traditional LLM backdoor attacks: The analysis compares agent and traditional LLM backdoors, emphasizing that agent attacks support more diverse, covert forms and can affect ordinary users through common phrases or general targets.The broader social-impact claim is tied to triggers such as “buy sneakers,” rather than attacker-exclusive secret triggers.
4 Experiments
Experiments evaluate Query-Attack, Observation-Attack, and Thought-Attack across AgentInstruct and ToolBench, using poisoning-ratio variations and task-specific attack metrics. The attacks achieve strong backdoor behavior, while effects on benign performance differ by trigger location and attack form.
- Datasets and attack setup: Experiments use AgentInstruct’s six tasks and ToolBench, with Query-Attack and Observation-Attack evaluated on WebShop and Thought-Attack evaluated on translation tools.ToolBench is reduced to a subset of about 4K traces for Thought-Attack, with Translate_v3 as the target tool.
- Evaluation protocol: Evaluation reports held-in-task performance, clean WebShop reward, target WebShop reward and ASR for query- and observation-triggered attacks, plus pass rate and ASR for Thought-Attack.The protocol distinguishes benign-task performance from triggered-task behavior and intermediate-tool manipulation.
- Query-Attack: Query-Attack degrades WS Target reward because Adidas-only selection can miss user requirements and produce sub-optimal choices.The agent may repeatedly click, view, return, and advance when Adidas results do not satisfy requested attributes.
- Observation-Attack: Observation-Attack generally preserves better held-in-task and WS Clean performance than Query-Attack, but its ASR is lower because observation triggers are harder to detect.With 50 poisoned samples, Observation-Attack-2.6%/12.5% reaches 78% ASR, while WS Target reward still declines for the same selection reasons.
- Thought-Attack: Thought-Attack keeps normal task pass rate similar to the clean agent while increasing the likelihood of calling Translate_v3 on translation queries.This demonstrates manipulation of intermediate reasoning and tool choice without changing final translation outputs.
5 Case studies
Case studies show that agent backdoor triggers can appear in environment observations and that malicious intermediate reasoning can coexist with unchanged final outputs. These behaviors extend backdoor attack surfaces beyond traditional user-query and final-output manipulation.
- Trigger locations: Observation-Attack can hide its trigger in an environment observation rather than placing it in the user query.This trigger location is illustrated in Figure 4 and differs from the usual trigger placement in traditional LLM backdoor attacks.
- Intermediate reasoning: Thought-Attack can introduce malicious internal reasoning while keeping the agent’s final output unchanged.The case study presents this as a behavior not likely to be achieved by traditional LLM backdoor attacks.
6 Discussion on potential countermeasures
The paper evaluates existing textual backdoor defenses in agent settings and argues that multi-round interaction makes detection harder, especially for Thought-Attack.
- Defense evaluation: The authors adapt DAN to defend against Query-Attack and Observation-Attack using 50 poisoned samples.They evaluate settings where the trigger round is either unknown or known to the defender.
- Defense evaluation: Multi-round agent interactions enlarge the output space and prevent defenders from precisely knowing when an attack will occur.The known-round setting is described as an impractical assumption.
- Defense limitations: Thought-Attack is harder to defend because it leaves observations and final outputs unchanged, making the attack more concealed.The paper states that current defenses easily fail against this form.
- Defense limitations: The analysis motivates targeted defenses for agent backdoors, while careful checking of training traces is offered as an interim mitigation.The suggested checks concern training-trace quality and toxicity before agent training.
7 Conclusion
The paper develops a framework for agent backdoor attacks, analyzes their forms, and demonstrates their effectiveness on AgentInstruct and ToolBench. It also identifies important scope limitations and urges caution with third-party agent data and systems.
- Conclusion: The framework categorizes agent backdoors by final outcome and trigger location, including attacks that preserve the final output while corrupting intermediate reasoning.The paper emphasizes that intermediate reasoning creates many attacking objectives.
- Conclusion: Experiments on AgentInstruct and ToolBench show that all examined attack forms are effective against LLM-based agents.The authors characterize this as a major challenge for agent application safety.
- Limitations: The study mainly formulates and analyzes attacks within the ReAct framework.The authors expect the analysis to extend to other frameworks because of shared reasoning logic.
- Limitations: Each of Query-, Observation-, and Thought-Attack is evaluated on only one target task, leaving broader task coverage for future work.The authors nevertheless state that the reported results expose severe security issues.
- Implications: The authors advise downstream users to be more cautious with third-party agent data and third-party agents.This recommendation follows their observation that backdoors can manipulate outputs or reasoning behavior at test time.
- Future work: The paper frames future work around discovering additional attack forms and developing effective defenses, while also noting possible protective uses.One example mentioned is protecting intellectual property of LLM-based agents.
D Details about poisoned data construction
The poisoned-data construction uses generated traces for shopping attacks and controlled tool-selection data for Thought-Attack, with supplementary tables documenting prompts and broader trigger experiments.
- Query- and Observation-Attack: For Query-Attack and Observation-Attack, instructions mix real WebShop queries with GPT-3.5-generated instructions, followed by GPT-4-generated poisoned traces.The traces include reasoning, actions, and observations, with extra attack objectives in the generation prompt.
- Query- and Observation-Attack: Table 4 documents the GPT-4 prompts used to generate poisoned Query-Attack and Observation-Attack traces, marking added malicious content in red.The table covers the prompt modifications used in poisoned-data generation.
- Query- and Observation-Attack: Observation-Attack testing ensures normal search results contain Adidas sneakers that clean models would not select, isolating the attack’s effect.This construction targets changes in product-selection behavior after poisoning.
- Thought-Attack: Thought-Attack uses existing ToolBench translation traces and targets the Translate_v3 tool among three translation tools.The poisoning controls how often each tool appears in translation-related samples.
- Supplementary experiments: Supplementary tables report full training hyper-parameters and results for broader trigger-token variants of Query-Attack and Observation-Attack.The broader triggers include multiple Adidas-related goods rather than only one product category.
- Thought-Attack: The Thought-Attack construction fixes translation-task training data at 80 samples and reserves 100 instructions for testing.At poisoning ratio k%, Translate_v3 receives 80×k% samples, while each other tool receives 40×(1-k%).
E Complete training details
Training uses standard optimizer and full-parameter fine-tuning settings across experiments, with computation distributed over eight NVIDIA A40 GPUs.
- Optimization: All experiments use AdamW and fine-tune the base model with full parameters.The detailed settings are provided in Table 5 and follow the default settings of the referenced benchmark studies.
- Hardware: All experiments run on 8 NVIDIA A40 GPUs.The passage directs readers to Table 5 for complete training hyper-parameters.
F Extra experiments on Query-Attack and Observation-Attack with a broader range of trigger tokens
The experiments broadened Query-Attack and Observation-Attack triggers from sneakers to Adidas-related goods. Attack success rates decreased but remained above 60%.
- The expanded triggers covered Adidas-related goods including shirts, boots, shoes, and clothing rather than only sneakers.The experiments used Query-Attack* and Observation-Attack* to represent this broader trigger range.
- Attack success rates were generally lower with broader triggers but remained above 60%.
- Similar keywords in clean training traces reduced backdoor injection effectiveness.Some clean traces contained related inputs without Adidas outputs, weakening the poisoning signal.
G Results of mixing agent data with general conversational data
The study tested whether adding general conversational data would reduce attack effectiveness in agents intended for both task-specific and general instruction use. Increasing training-data diversity and size barely affected the attacks.
- The experiments added about 3.8K ShareGPT samples with GPT-4 responses to the training dataset.WebShop poisoning was fixed at 50 samples in each setting.
- The resulting models were labeled Query/Observation-Attack-0.9%/12.5%.
- Increasing training-data diversity and overall size barely affected attacking effectiveness, while MMLU measured general agent ability.
H Results of the probability each agent would recommend buying from Adidas on clean samples without the trigger
On 200 clean samples without the trigger, clean and backdoored agents generally showed near-zero Adidas recommendations. The reported case studies separately visualize Query-, Observation-, and Thought-Attack behavior.
- Clean agents recommended Adidas products with probability 0%, while backdoored agents remained close to 0% on clean samples.These results indicate minimal poisoning impact on normal behavior without the trigger.
- The clean samples triggering Adidas recommendations in backdoored agents were all about shoes, possibly reflecting similarity to the sneaker trigger.The paper attributes this to potentially similar word embeddings.
- Figures 3, 4, and 5 present case studies for Query-Attack, Observation-Attack, and Thought-Attack, respectively.Figures 3 and 4 place clean and attacked responses side by side, while Figure 5 places them top and bottom.