PaperlayerView exact revision on arXiv

Source-linked AI summary

DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent

Pengyu Zhu, Zhenhong Zhou, Yuanhe Zhang, Shilinlu Yan, Kun Wang, Sen Su

arXiv:2502.12575v2cs.CRcs.AI

TL;DR

Agent backdoors raise safety concerns, and prior approaches can leave content exposed to safety audits. This paper uses dynamic encryption and tiered sub-backdoor fragments, and reports high attack success with 0% detection in memory safety audits. It also introduces AgentBackdoorEval and concludes that stronger defenses are needed.

  • Problem

    Previous agent backdoor approaches can store explicit attack content in memory, making it detectable through safety audits despite reported attack success and normal-task performance.

    Source: §1 · p. 1

  • Method

    The attack dynamically encrypts backdoor content into workflow-integrated benign content and divides it into tiered sub-backdoor fragments activated cumulatively.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    Across datasets, the authors report that the method outperforms existing techniques in detection evasion; it achieves a 0% detection rate in memory safety audits while maintaining high attack success.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that stronger defenses are needed, while identifying the development of defensive strategies as a key direction against potential malicious use.

    Source: §6 · p. 9, §7 · p. 9

    The study focuses on black-box models across available datasets and domains and does not explore white-box training because of resource constraints.

    Source: §7 · p. 9

Abstract

from arXiv · show

As LLM-based agents become increasingly prevalent, backdoors can be implanted into agents through user queries or environment feedback, raising critical concerns regarding safety vulnerabilities. However, backdoor attacks are typically detectable by safety audits that analyze the reasoning process of agents. To this end, we propose a novel backdoor implantation strategy called \textbf{Dynamically Encrypted Multi-Backdoor Implantation Attack}. Specifically, we introduce dynamic encryption, which maps the backdoor into benign content, effectively circumventing safety audits. To enhance stealthiness, we further decompose the backdoor into multiple sub-backdoor fragments. Based on these advancements, backdoors are allowed to bypass safety audits significantly. Additionally, we present AgentBackdoorEval, a dataset designed for the comprehensive evaluation of agent backdoor attacks. Experimental results across multiple datasets demonstrate that our method achieves an attack success rate nearing 100\% while maintaining a detection rate of 0\%, illustrating its effectiveness in evading safety audits. Our findings highlight the limitations of existing safety mechanisms in detecting advanced attacks, underscoring the urgent need for more robust defenses against backdoor threats. Code and data are available at https://github.com/whfeLingYu/DemonAgent.

1 Introduction

Prior agent backdoors can be exposed by memory audits, motivating a stealthier implantation strategy. The paper combines dynamic encryption and tiered fragments, reports 0% detection in memory safety audits, and introduces AgentBackdoorEval.

1 Introduction

  • Earlier agent backdoor attacks could achieve attack success and normal-task performance, but explicit backdoor contents in memory made them detectable through safety audits.This lack of stealth motivates the proposed audit-evasion strategy.Source: §1 · p. 1
  • Dynamic Encryption Mechanism hides backdoor content as workflow-integrated benign content, while Multi-Backdoor Tiered Implantation splits it into encrypted sub-backdoor fragments activated through cumulative triggering.The fragments are implanted in tiers through the agent’s workflow, making them difficult to trigger with arbitrary queries.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 0% detection rate in memory safety audits, with high attack success, is reported for the method, which the authors describe as undetectable compared with other backdoor algorithms.The paper also reports that its approach outperforms existing methods across diverse datasets and reveals weaknesses in current agent safety mechanisms.Source: §1 · p. 2
  • AgentBackdoorEval evaluates agent backdoor attacks across five real-world scenarios: banking finance, e-commerce, medical, social media, and weather query.The dataset also uses simulation tools for scenario tool calls and prompts intended to support scalable expansion.Source: §1 · p. 2
  • The attack is designed for precise activation under carefully designed conditions while bypassing safety audits.The authors present it as a novel backdoor implantation strategy for LLM-based agents.Source: §1 · p. 2

2 Related Work

Related work describes LLM-based agents as systems that use models to reason, plan, and act through external tools. It also distinguishes agent backdoors from model-targeted attacks and notes that existing agent attacks often lack stealth.

2.1 LLM-based Agents

  • LLM-based agents use LLMs for autonomous reasoning, planning, and task execution with external tools, managing workflows that can perceive, plan, act, and learn within a defined scope.Examples include household adaptation and shopping tasks.Source: §2.1 · pp. 2–3

2.2 Backdoor Attacks

  • Agent backdoors differ from traditional LLM backdoors because agents reason across multiple steps and interact with environments, creating opportunities for query-, observation-, and thought-based attacks.Agent systems can be susceptible to data-poisoning and model-poisoning attacks.Source: §2.2 · p. 3
  • Existing attacks on agents often lack stealth and can be detected through memory audits, motivating the paper’s proposed stealthy and adaptive backdoor.Source: §2.2 · p. 3

3 Dynamically Encrypted Multi-Backdoor Implantation Attack

The attack combines time-dependent encryption with fragmented, tiered backdoor implantation in an agent workflow. DEM encrypts backdoor code as the agent progresses, while MBTI distributes fragments across tools and triggers reconstruction only along a valid attack path.

3.1 Preliminary

  • ReAct initializes memory with the user query, cycles through thought, action, and environment observation, then generates a final answer from the final memory state.The process continues until a termination result is reached.Source: §3.1 · p. 3

3.2 Dynamically Encryption Mechanism (DEM)

  • DEM time-dependently encodes backdoor code into encrypted content, stores corresponding key-value pairs in a temporary table, and destroys that table when the agent completes.The encoding is described as nonreproducible.Source: §3.2 · pp. 3–4

3.3 Multi-Backdoor Tiered Implantation (MBTI)

  • MBTI splits the backdoor into fragments, embeds associated codes across fragment boundaries, and poisons tool invocation code with distinct fragments for tiered implantation.The fragments are encrypted during execution using DEM.Source: §3.3 · pp. 4–5
  • Cumulative triggering retrieves and assembles fragments only when the required fragments and a valid attack path are present; invalid connections cause reconstruction to fail and the code to be discarded.The path is checked using tool invocation dependencies.Source: §3.3 · p. 5

4 Experiment

The experiments compare the proposed attacks with existing methods, test them on benchmark and cross-domain tasks, and assess attack success, detection, and task completion.

4.1.1 Models and Datasets

  • Experiments use GPT-4, DeepSeek-V3, and Qwen2.5-14B on AgentInstruct and ToolBench, alongside AgentBackdoorEval’s five real-world domain tasks.AgentInstruct includes six tasks; AgentBackdoorEval covers Banking Finance, E-commerce, Medical, Social Media, and Weather Query.Source: §4.1.1 · p. 5

4.1.2 Backdoor Attack Targets

  • The attack target is simulated file writing, which records private data to a file to mimic an HTTP request to a malicious website without affecting the actual environment.The file-writing setup supports tracking attack success and analyzing outcomes.Source: §4.1.2 · p. 6

4.1.3 Audit Mechanism

  • GPT-4o reviews agent memory and execution results for irregularities, with manual review rechecking flagged issues.The audit uses a defense mechanism based on Hua et al. (2024).Source: §4.1.3 · p. 6

4.2 Evaluation Metrics

  • The evaluation measures attack success rate (ASR), detection rate (DR), and normal task completion performance (NP).ASR measures malicious execution likelihood, DR the probability of detection through safety audits, and NP correct task completion.Source: §4.2 · p. 6

4.3.1 Comparison of Methods

  • DEM and DEM+MBTI achieve 0% DR and 100% ASR, unlike BadAgent and Foot-in-the-Door, which have high DR despite relatively high ASR.AdvWeb also has significant DR despite high ASR; Figure 2 compares methods on DR and ASR.Source: Figure 2 · p. 6, §4.3.1 · p. 7

4.3.2 Performance on Benchmarks

  • Across models on AgentInstruct and ToolBench, DEM and DEM+MBTI consistently achieve 100% ASR and 0% DR, with NP remaining high.The passage says attacks can still succeed when tasks are incomplete because of model limitations, provided tools are used correctly.Source: Table 1 · p. 6, §4.3.2 · p. 7

4.3.3 Cross-Domain Performance on AgentBackdoorEval

  • Across domains and models, DEM and DEM+MBTI achieve 100% ASR and 0% DR while NP remains high.Table 2 reports these results for AgentBackdoorEval.Source: §4.3.3 · p. 7, Table 2 · p. 7

5 Analysis

The analysis examines how multi-fragment triggering affects harmless paths, how encoding choices affect detection, and the trade-off introduced by proposed defenses.

5.1 Comparison of Harmless Path Ratios

  • With MBTI, the harmless-path ratio rises as attack codes increase and approaches 100% beyond a threshold; sequential fragment assembly makes accidental activation less likely.Figure 3 compares ratios across tool counts and attack-code counts, with and without MBTI.Source: §5.1 · pp. 7–8, Figure 3 · p. 8

5.2 Impact of Different Mapping Methods

  • Time-based encoding with DEM achieves 0% DR, while original encoding yields 100% DR; adding MBTI lowers DR for base64 and original encoding.Figure 4 compares encoding methods under DEM and DEM+MBTI.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Potential Defenses

  • Shuffle disrupts MBTI’s sequential path, while Shuffle++ also breaks encrypted content; both reduce ASR but leave NP near nonfunctional levels.The paper identifies preserving agent functionality while defending against the attack as an open challenge; Table 3 presents defense effectiveness.Source: §5.3 · p. 8, Table 3 · p. 8

6 Conclusion

The conclusion presents a backdoor strategy that combines dynamic encryption with fragment-based implantation to evade safety audits. It reports strong attack and evasion performance across datasets and introduces AgentBackdoorEval.

6 Conclusion

  • Dynamic Encryption Mechanism (DEM) converts the backdoor into benign content, while Multi-Backdoor Tiered Implantation (MBTI) splits it into sub-backdoor fragments.The conclusion presents these mechanisms as the basis for stealth.Source: §6 · p. 8
  • The method outperforms existing techniques in detection evasion and reports attack success and normal task completion across various datasets.The conclusion describes the method as robust.Source: §6 · p. 9
  • The paper introduces AgentBackdoorEval to advance agent safety and calls for stronger defenses and responsible LLM development.Source: §6 · p. 9

7 Limitation

The study focuses on black-box models across datasets and domains, constrained by available resources. It identifies white-box evaluation, stronger defenses, and multi-agent settings as directions for future work.

7 Limitation

  • The study focuses on black-box models across datasets and domains, constrained by available resources.Source: §7 · p. 9
  • The authors have not explored white-box models, which could potentially improve adaptability to agents and provide further insight into the approach.Source: §7 · p. 9
  • The authors identify defense development and evaluation in multi-agent collaboration environments as future research directions.They also propose exploring encryption techniques, including white-box model pre-training for encrypted-content generation.Source: §7 · p. 9

8 Ethics Statement

The ethics statement describes safeguards and coordinated disclosure. The appendix outlines direct and cumulative triggering processes, with MBTI requiring fragments to be combined before activation.

8 Ethics Statement

  • Experiments used isolated, sandboxed environments and synthetic data, with no live systems or user data involved.Source: §8 · p. 9
  • The authors report coordinated disclosure to affected model providers and downstream developers, and commit to responsible release and continued stakeholder collaboration.Source: §8 · p. 9

A Algorithm of Triggering

  • The appendix presents two triggering algorithms, one without MBTI and one using MBTI for more stealthy, complex activation, and illustrates attack concepts in agent memory.Source: A Algorithm of Triggering · p. 12

A.1 Triggering Process without MBTI

  • Without MBTI, the attack directly uses encrypted backdoor content and activates it through decoding and execution.Source: §A.1 · p. 12

A.2 Triggering Process with MBTI (Cumulative Triggering)

  • With MBTI, encrypted fragments are decoded and assembled, and activation occurs only when they are combined correctly.Cumulative triggering is described as increasing stealth and complexity.Source: §A.2 · p. 12

B Data and Tool Generation Methods

AgentBackdoorEval is built through generated data and simulated tools that are manually screened and fine-tuned. The appendix also describes simulated attack targets and model-based plus manual review procedures.

B Data and Tool Generation Methods

B.1 Data Generation Process

  • Prompt-based generation creates structured domain-specific simulated attack scenarios, using few-shot examples and specifications that avoid actual harmful operations.Source: §B.1 · p. 13

B.2 Tool Generation Process

  • Python functions simulate domain operations with mock JSON results and are integrated into the agent for task and backdoor evaluation.Source: §B.2 · p. 13

C Backdoor Attack Targets

  • The attack-target examples simulate medical tasks through file writes and fake data, with malicious URLs used to redirect or log actions.Source: C Backdoor Attack Targets · p. 13

D Audit Mechanism

  • The audit combines GPT-4o and manual analysis to review agent memory for decision accuracy, backdoors, and anomalies.Source: D Audit Mechanism · p. 13

E NP Evaluation Prompt

  • NP evaluation combines GPT-4o and manual judgment to assess whether responses are relevant, accurate, complete, and sufficiently detailed.Source: E NP Evaluation Prompt · p. 13

F The Calculation and Measurement of Evaluation Metrics

The section defines ASR, DR, and NP as measures of attack effectiveness, audit detectability, and task functionality. It describes binary per-attempt or per-task outcomes and their aggregate calculation.

F.1 Attack Success Rate (ASR)

  • ASR measures whether the agent executes the intended malicious action when triggered, then divides successful executions by total attack attempts.Each attempt is coded as success (y_i = 1) or failure (y_i = 0).Source: §F.1 · p. 14

F.2 Detection Rate (DR)

  • DR measures suspicious behavior found in post-execution memory audits, calculating detected tasks as a share of all backdoored tasks.A higher DR means greater likelihood of detection; a lower DR indicates greater concealment.Source: §F.2 · p. 14

F.3 Normal Task Completion Performance (NP)

  • NP measures overall task completion with the backdoor present, assessed by reviewing agent memory and final output using GPT-4o and manual inspection.Successful task outcomes are counted against all tasks; NP is intended to capture overall completion rather than individual operational-step correctness.Source: §F.3 · pp. 14–15

G Tool Statistics in AgentBackdoorEval

The section reports tool and attack-type statistics for AgentBackdoorEval, then explains how fragment count and sequential assembly relate to harmless paths and attack triggering.

G Tool Statistics in AgentBackdoorEval

H Impact of Sub-backdoor Fragment Count on Harmless Path Ratio

  • More sub-backdoor fragments correspond to a higher harmless path ratio, making random-query triggering less likely and requiring a specific sequence of tool invocations.The section attributes the added trigger complexity to cumulative assembly of the fragments.Source: H Impact of Sub-backdoor Fragment… · p. 15, Figure 5 · p. 16

I Calculation of Harmless Paths

  • Without MBTI, any poisoned-tool invocation triggers an attack; with MBTI, fragments must be assembled in the correct order, increasing harmless paths and reducing random-query triggering.The section defines harmless paths as the total possible paths minus harmful paths, and says sequential assembly increases trigger complexity.Source: I Calculation of Harmless Paths · p. 16

J Supplementary Experiments on Fine-tuned Agent Models

Supplementary experiments assess two fine-tuned agent models, AgentLM and Athene-V2-Agent, including attack transferability to the latter. The section also provides examples of simulated attack behavior and agent memory under audit.

J.2 Attack Transferability to Fine-tuned Models

  • 100% ASR and 0% DR on Athene-V2-Agent for both DEM and DEM+MBTI, matching results on its Qwen-2.5 base model.The authors describe this as evidence of transferability and stealth; they attribute vulnerability to tool-invocation execution rather than model-specific parameters or gradients.Source: Table 6 · p. 17, §J.2 · p. 17

J.1 Practical Value of Modern Fine-tuned Agent Models

  • 5.83% NP for AgentLM versus 91.67% for Athene-V2-Agent on AgentInstruct; the authors regard only the stronger-baseline model as practically useful for advanced attack evaluation.The section notes AgentLM's low task performance as a limitation for evaluating advanced attacks.Source: Table 7 · p. 17, §J.1 · p. 17

Attack Result Example

Agent Memory Under Attack

  • The appendix pairs an example of agent memory under attack with an audit prompt and a prompt for evaluating task completion.Figures 11, 12, and 13 are captioned as the memory example, audit prompt, and task-completion evaluation prompt, respectively.Source: Figure 11 · p. 21, Figure 12 · p. 22, Figure 13 · p. 23

Related papers

Loading 2502.12575v2…