PaperlayerView exact revision on arXiv

Source-linked AI summary

Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections

Xianglin Yang, Yufei He, Shuo Ji, Bryan Hooi, Jin Song Dong

arXiv:2602.15654v2cs.CRcs.AI

TL;DR

Self-evolving agents can store untrusted external content in long-term memory, allowing one-time indirect prompt injection to influence later sessions. The paper formalizes this Zombie Agent threat and evaluates a black-box infection-and-trigger attack against common memory systems. Results indicate that truncation, summarization, and retrieval ranking do not reliably remove malicious instructions, even while agents remain useful on normal tasks.

  • Problem

    Self-evolving agents reuse long-term memory across sessions, but the security implications of untrusted content entering that memory remain insufficiently addressed.

  • Method

    The paper formalizes Zombie Agents and evaluates a black-box, two-phase attack using attacker-controlled external content against sliding-window and retrieval-augmented memory.

  • Results

    Common truncation, summarization, and retrieval-ranking mechanisms do not reliably remove malicious instructions once they enter memory, while agents can remain useful on normal tasks.

  • Takeaways & Limitations

    Memory evolution can convert one-time indirect injection into persistent, cross-session compromise, so per-session prompt filtering alone does not address the full risk.

  • Takeaways & Limitations

    The threat model assumes a black-box attacker limited to publishing malicious external content, without access to private state, user history, or the toolset.

Abstract

from arXiv · show

Self-evolving LLM agents update their internal state across sessions, often by writing and reusing long-term memory. This design improves performance on long-horizon tasks but creates a security risk: untrusted external content observed during a benign session can be stored as memory and later treated as instruction. We study this risk and formalize a persistent attack we call a Zombie Agent, where an attacker covertly implants a payload that survives across sessions, effectively turning the agent into a puppet of the attacker. We present a black-box attack framework that uses only indirect exposure through attacker-controlled web content. The attack has two phases. During infection, the agent reads a poisoned source while completing a benign task and writes the payload into long-term memory through its normal update process. During trigger, the payload is retrieved or carried forward and causes unauthorized tool behavior. We design mechanism-specific persistence strategies for common memory implementations, including sliding-window and retrieval-augmented memory, to resist truncation and relevance filtering. We evaluate the attack on representative agent setups and tasks, measuring both persistence over time and the ability to induce unauthorized actions while preserving benign task quality. Our results show that memory evolution can convert one-time indirect injection into persistent compromise, which suggests that defenses focused only on per-session prompt filtering are not sufficient for self-evolving agents.

1 INTRODUCTION

Self-evolving agents create a persistence risk because attacker-controlled content can enter long-term memory and influence later sessions. The paper formalizes and evaluates Zombie Agent attacks that survive memory management while preserving benign utility.

  • Standard prompt injection is transient because its malicious text disappears when the session or context window ends.
  • Self-evolving agents update and reuse long-term memory, making the memory update pathway part of the attack surface.
  • A Zombie Agent covertly retains a sleeper payload that can later leak private data or perform unauthorized tool actions in unrelated sessions.
  • The attack scenario infects memory when a benign web-browsing task causes hidden instructions from an attacker-controlled page to be stored as a reusable procedure.
  • Persistence must survive truncation and achieve retrieval hijacking, so the attack targets deterministic memory-consolidation behavior in sliding-window and retrieval-based systems.
  • The framework uses attacker-controlled external content in infection and trigger phases, then evaluates persistence, unauthorized actions, and benign-task utility.

2 PRELIMINARY

The paper models agents through interactions, state, sessions, and cross-session evolution. It focuses on memory-evolving agents whose fixed parameters accompany memory updates, especially sliding-window and retrieval-augmented designs.

  • 2.1 INTERACTIONS, STATE, AND SESSIONS: An interaction is a user prompt paired with the agent’s corresponding response at a given timestep.
  • 2.1 INTERACTIONS, STATE, AND SESSIONS: The agent state contains model parameters and external memory, which together determine behavior.
  • 2.1 INTERACTIONS, STATE, AND SESSIONS: A session is a sequence of interactions within one conversational context before a reset.
  • 2.2 STATIC VS. SELF-EVOLVING AGENTS: Static-agent attacks affect only the current session, whereas self-evolving agents update state between sessions from prior interactions.
  • 2.2 STATIC VS. SELF-EVOLVING AGENTS: For memory-evolving agents, parameters remain fixed while knowledge accumulates in memory through the memory evolution function.
  • 2.3 THREAT MODEL: The threat model assumes a black-box attacker who can publish malicious external content but cannot access private state, user history, or alter tools.
  • 2.3 THREAT MODEL: Attack success requires writing a payload into memory, preserving it through updates, and later inducing unauthorized behavior while maintaining benign-task utility.

3 METHODOLOGY

The Zombie Agent attack has infection and trigger phases: attacker-controlled content enters an agent’s context, is stored through memory evolution, and later drives unauthorized actions. Persistence requires mechanism-specific strategies to survive sliding-window eviction or retrieval sparsity.

  • Attack lifecycle: The attack lifecycle separates infection, where poisoned external content is written into long-term memory, from triggering, where retrieved poisoned memory causes unauthorized actions.The trigger phase can also re-observe adversarial content and rewrite the payload, reinforcing persistence.
  • Attack formulation: The attacker aims to force the self-evolution function FM to store payload Z in Mj+1 so it influences later decision-making.The payload enters through an attacker-controlled observation oadv obtained during a session.
  • Attack formulation: Infection succeeds when FM commits the injected payload to long-term storage through the agent’s normal memory-update protocol.For sliding-window memory, the payload is appended to the rolling buffer; for RAG, it is embedded and indexed in a vector database.
  • Persistence strategies: Sliding-window memory evicts older tokens, so recursive self-replication rewrites the payload into the current context during each iteration.The payload instructs the agent to revisit the attacker’s site before taking action.
  • Persistence strategies: RAG memory may leave a stored payload dormant when future queries are semantically unrelated, so semantic aliasing places it within broad embedding clusters.This increases the probability that the payload is retrieved across unrelated queries such as flight-information requests.

4 EXPERIMENTS

The experiments evaluate Zombie Agent effectiveness, persistence, defense evasion, and practical impact across sliding-window and RAG agents. Results show persistent payloads retain high attack success, survive memory evolution, evade instruction defenses, and compromise high-stakes workflows.

  • Evaluation Questions: The evaluation tests effectiveness, persistence, defense evasion, and practical impact across self-evolving agents using sliding-window and RAG memory.The protocol includes two commercial tool-use models and exposure followed by trigger phases.
  • Metrics: The attack uses ASR for malicious execution, context retention for sliding-window persistence, and injection count plus Recall@k for RAG persistence.ASR is the percentage of trigger rounds executing the malicious command.
  • Attack Effectiveness: Zombie Agent significantly outperforms standard indirect prompt-injection strategies across both memory architectures.Figure 3 compares cumulative average ASR across more than 20 trigger rounds.
  • Attack Effectiveness under Evolution: Raw History reaches approximately 77% ASR, while Verbal Reflection and Refined Experience reduce ASR to approximately 12% and 3–15%, respectively.Despite filtering, approximately 15% Command Execution remains sufficient to crystallize malicious actions into persistent rules.
  • Persistence: Zombie Agent maintains 100% sliding-window retention throughout 20+ rounds, whereas baseline payloads eventually reach zero after FIFO truncation.Recursive Renewal copies the payload into subsequent context frames.
  • Persistence: In RAG, the method accumulates approximately 240 payload copies versus approximately 100 for baselines and retrieves approximately 23 malicious entries at K = 50.Embedding pollution broadens database coverage and keeps payloads retrievable for irrelevant queries.
  • Defense Evasion: Sandwich, Instructional, and Spotlight defenses leave ASR above 60%, with only an approximately 10–15% drop from undefended settings.The evaluation describes this penalty as negligible relative to the remaining attack success.
  • Practical Impact: Healthcare and E-Commerce case studies show infection from a single poisoned source followed by betrayal during unrelated tasks.The cases frame persistent subversion as an insider threat using tool permissions to cause tangible harm.

5 RELATED WORK

Related work spans iterative feedback, long-term memory and evolution, prompt injection, RAG poisoning, and instruction-level defenses. The paper positions Zombie Agents at the intersection of persistent memory evolution and indirect prompt injection.

  • Iterative Feedback Loops: Iterative-feedback systems improve outputs through critique, refinement, and stored textual reflections without updating model weights.Examples include Self-Refine and Reflexion.
  • Long-Term Memory and Evolution: Long-term memory systems support multi-session interaction by retaining salient information, managing memory tiers, or evolving task trajectories.The surveyed systems include MemoryBank, MemGPT, SAGE, Evo-Memory, SE-Agent, and AlphaEvolve.
  • Prompt Injection: Prompt injection exploits confusion between trusted instructions and untrusted data, while indirect injection places malicious instructions in attacker-controlled external sources.Prior benchmarks show these attacks across diverse tasks and report failures of standard mitigations.
  • RAG and Memory Poisoning: RAG increases exposure by automatically inserting retrieved content into context, enabling corpus poisoning, retriever manipulation, and attacks on multi-hop queries.Prior work studies targeted behaviors from poisoned corpora and optimized document attacks.
  • Instruction-Level Defenses: Instruction-level defenses use prompting strategies such as Sandwich, Spotlighting, and reminders to separate data from instructions within immediate context.These defenses primarily treat the input channel as the threat vector.

6 CONCLUSION

The conclusion argues that persistent memory changes the security problem for LLM agents: attacker-controlled content can influence later sessions after its original context disappears. It recommends treating memory as trusted infrastructure and evaluating defenses against adaptive attackers.

  • 6 CONCLUSION: A single exposure to attacker-controlled content can influence future behavior when agents write observations to long-term memory and reuse them across sessions.This persistence breaks the session-local assumption behind many prompt-injection mitigations.
  • 6 CONCLUSION: Truncation, summarization, and retrieval ranking do not reliably remove malicious instructions once they enter memory, even while agents appear useful on normal tasks.The conclusion identifies this as a persistent compromise risk rather than only immediate-context misbehavior.
  • 6 CONCLUSION: The paper recommends separating untrusted data from executable instructions, attaching provenance to memory entries, and policy-checking tool calls influenced by retrieved memory.These safeguards are presented as minimum measures for memory-aware security.
  • 6 CONCLUSION: Future work should evaluate defenses under adaptive attackers and clarify which agent designs and tool permissions are most exposed.The conclusion identifies adaptive evaluation and system-boundary analysis as remaining directions.

A.1 TARGET MEMORY ARCHITECTURES

The paper examines sliding-window and retrieval-augmented memory because their update and retrieval mechanisms determine whether malicious payloads persist. Sliding-window memory faces FIFO eviction, whereas RAG memory faces retrieval-ranking constraints.

  • Memory architectures: The study targets sliding-window and retrieval-augmented generation as canonical memory implementations because their update functions determine attack persistence.The memory update function converts transient session data into persistent state.
  • Sliding-window memory: Sliding-window memory stores recent history in a rolling buffer or summary and truncates older entries to fit a fixed token limit.Its evolution function appends each new session summary before truncation.
  • Sliding-window memory: FIFO eviction is the primary sliding-window constraint: payloads must avoid being truncated or summarized away as the window advances.Persistence therefore requires surviving repeated memory updates.
  • Retrieval-augmented memory: RAG memory stores extracted knowledge units as embedded entries in a persistent vector database.Future sessions retrieve a subset based on semantic relevance to the current query.
  • Retrieval-augmented memory: Retrieval ranking is the primary RAG constraint: a payload must maximize cosine similarity with unknown future queries to be retrieved.The attack must therefore remain relevant despite unrelated later queries.

A.2 AGENT CAPABILITIES AND TOOLS.

The agent uses a web-oriented toolset for information retrieval and system interaction, including search, webpage reading, and local command execution. These capabilities expose malicious content through browsing and enable payload actions through command execution.

  • Toolset: The Web Agent toolset is T = {search, read url, execute command}.These tools support both external information retrieval and local system interaction.
  • Tool capabilities: search(query) retrieves a list of URLs from a search engine.
  • Tool capabilities: read url(url) fetches and parses textual content from a specified webpage.
  • Tool capabilities: execute command(cmd) executes shell commands in a local environment.
  • Attack surface: The tools form the attack surface because read url admits malicious content and execute command can run payloads.

A.3 ATTACK PROMPTS

The attack template is presented as a formalized procedure for compromising self-evolving agents through their memory and later behavior.

  • The paper introduces its Zombie Agent attack template as the procedure used for the attack framework.

A.4 INSTRUCTION DEFENSE PROMPTS

The paper presents figures for the Zombie Agent payload, sliding-window and RAG agent prompts, and three instruction-defense prompts. The supplied figure text identifies these artifacts but does not describe their internal content or comparative outcomes.

  • Attack payload: Figure 7 presents the Zombie Agent attack payload.
  • Agent prompts: Figure 8 presents the sliding-window agent prompt.
  • Agent prompts: Figure 9 presents the RAG agent prompt.
  • Defense prompts: Figures 10, 11, and 12 present the Sandwich, Instructional, and Spotlight defense prompts, respectively.
Loading 2602.15654v2…