PaperlayerView exact revision on arXiv

Source-linked AI summary

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents

Pritam Dash, Tongyu Ge, Aditi Jain, Tanmay Shah, Zhiwei Shang

arXiv:2606.04329v2cs.CRcs.AI

TL;DR

Persistent memory improves agents’ ability to accumulate knowledge but enables adversarial content to persist and influence future behavior. This paper analyzes the write channels and structural vulnerabilities behind memory poisoning, develops an attack taxonomy and MPBench, and finds persistent attacks alongside incomplete coverage from existing prompt-injection defenses.

  • Problem

    Persistent memory can turn a single adversarial write into lasting behavioral influence, while prior work lacks a unified analysis of the vulnerabilities enabling such attacks.

  • Method

    The paper analyzes four memory write channels and nine structural vulnerabilities, develops six attack classes, and evaluates them with MPBench on two agent systems.

  • Results

    Memory poisoning attacks are persistent, with average ASR of 50.46% and RSR of 41.05% across both agents, while existing prompt-injection defenses provide incomplete coverage.

  • Takeaways & Limitations

    Aggressive memory writing and retrieval expand the attack surface, creating a tension between memory capability and security that agent designers must account for.

  • Takeaways & Limitations

    The evaluation uses a single model, and several benchmark domains emulate external payload delivery as labeled context rather than modeling real tool-call retrieval pipelines.

Abstract

from arXiv · show

Memory is a core component of AI agents, enabling them to accumulate knowledge across interactions and improve performance. However, persistent memory introduces the risk of memory poisoning, where a single adversarial memory write can exert long-term influence over agent behavior. We present a systematic study of memory poisoning in LLM-based agents. We identify four memory write channels and nine structural vulnerabilities in model capabilities, system prompt design, and agent system architecture that make these channels exploitable. Based on these vulnerabilities, we develop a taxonomy of six classes of memory poisoning attacks. Furthermore, we design MPBench -- a benchmark for evaluating memory poisoning attacks, and show that agents designed to write and retrieve memory more aggressively are more exploitable. We also show that existing prompt injection defenses fail to cover memory poisoning attacks. Our findings provide a foundation for understanding and mitigating memory poisoning attacks against AI agents.

1. Introduction

Persistent memory helps agents accumulate knowledge and improve over time, but it also creates an attack surface in which adversarial content can be stored and later treated as trusted knowledge. The paper systematically analyzes this surface, proposes an attack taxonomy and MPBench, and finds persistent attacks with incomplete defense coverage.

  • Motivation: Persistent memory supports agents’ long-horizon improvement but creates a new attack surface for adversarial content.Agents retain preferences, task outcomes, and domain knowledge, while memory can also store malicious content from external inputs.
  • Motivation: A single successful memory write can influence agent behavior in future sessions, unlike attacks requiring payload presence in every active context.Memory poisoning targets content written during normal operation and later retrieved as trusted knowledge.
  • Research gap: The field lacks a unified foundation because prior threat models impose restrictive assumptions or study narrow attacks in isolation.Existing work does not systematically analyze the vulnerabilities enabling memory poisoning in deployed agents.
  • Contributions: The paper identifies four exploitable memory write channels and nine structural vulnerabilities spanning model capabilities, prompt design, and agent architecture.The analysis uses a black-box threat model in which the adversary lacks privileged, model, and direct memory access.
  • Contributions: The authors develop a taxonomy of six attack classes that links each attack to the vulnerability it exploits.This provides a principled view of the attack surface rather than a collection of isolated techniques.
  • Contributions: MPBench evaluates memory poisoning on two agent systems and shows that existing prompt-injection defenses leave blind spots because poisoning payloads need not exhibit detectable context patterns.The benchmark evaluates failures empirically and discusses defenses against memory poisoning.
  • Results: 50.46% average ASR and 41.05% average RSR across both agents show that memory poisoning attacks are persistent.Attack success scales with how aggressively agents read and write memory.

2. Agent Memory: Write Channels and Vulnerability Landscape

Agent memory persists information across sessions through write channels that differ in their triggers and write authority. These channels are exposed by model, prompt-design, and system-architecture vulnerabilities that allow untrusted content to influence persistent storage.

  • Agent Memory: Long-term memory persists across sessions and directly influences future agent behavior, making it the primary target of memory poisoning.
  • Memory Write Channels: Four write channels differ by trigger and write authority: explicit instruction-executed, system prompt-driven, compaction-driven, and experience-to-procedure writes.C1 is a direct write; C2, C3, and C4 are inferred writes based on policy or observed execution structure.
  • Memory Write Channels: C4 converts completed task interactions into reusable procedural skills, with the agent deciding that the interaction constitutes a reusable procedure.
  • Vulnerabilities: Model vulnerabilities include instruction-data boundary blindness and source-attribution failure, allowing external content to resemble valid instructions or appear equivalent to trusted inputs.
  • Vulnerabilities: Prompt vulnerabilities include vague write criteria and compaction without source filtering, allowing semantically aligned or important-looking adversarial content into memory.
  • Vulnerabilities: System weaknesses include shared multi-source context, absent write-path validation, and self-improvement loops that can reinforce poisoned skills over time.

3. Memory Poisoning Attack Taxonomy

Memory poisoning attacks manipulate how agents write and later retrieve persistent content. The taxonomy distinguishes six attack classes by their payload mechanisms and signal strength, from explicit commands to content that appears indistinguishable from legitimate knowledge.

  • Attack Definition: A memory poisoning attack injects an adversarial payload through a write channel to cause a target instruction or equivalent paraphrase to enter persistent memory.
  • Attack Objectives: Successful attacks trigger a memory write, control the written content, and cause later retrieval to influence behavior without further attacker involvement.
  • Attack Consequences: Retrieved poisoned entries are treated as trusted prior knowledge and can redirect tasks, harvest credentials through attacker-controlled endpoints, or undermine reliance on stored knowledge.
  • Attack Taxonomy: The six attack classes are Explicit Command Insertion, Conditional Command Insertion, Salience-driven Compaction Poisoning, Policy Conformant Fact Injection, False Precedent Insertion, and Skill-Procedure Insertion.
  • Signal Strength: Strong-signal attacks expose explicit linguistic markers or conditional triggers, whereas weak-signal attacks contain no detectable input-level anomaly and resemble legitimate content.
  • Benchmark Design: MPBench evaluates poisoning in separate memory-write and memory-retrieval phases, counting stored payload matches as ASR and behavior reflecting retrieved payloads as RSR.

4. Evaluation

MPBench evaluates memory poisoning across attack classes, agent systems, persistence, and prompt-injection defenses. The results show that permissive memory writing and retrieval increase exploitability, while existing defenses particularly miss weak-signal attacks.

  • Evaluation setup: MPBench evaluates memory poisoning through memory-write and memory-retrieval phases on OpenClaw and HERMES.Test cases cover seven domains and both static-context and dynamic tool-call delivery modes.
  • Exploitability differences: 66.67% average ASR on HERMES versus 34.25% on OpenClaw, reflecting greater susceptibility across attack classes.HERMES writes more frequently because of its system prompt-driven retention policy and lower compaction threshold.
  • Persistence: 64.70% average RSR on HERMES versus 17.40% on OpenClaw, showing that successful poisoned writes influence behavior in subsequent sessions.HERMES automatically injects persistent memory into the system prompt at session start, whereas OpenClaw retrieves memory only when explicitly invoked.
  • Signal strength and attack classes: 64.50% ASR for Policy-Conformant Fact Injection on HERMES versus 8.33% on OpenClaw, while False Precedent Insertion reaches 73.33% versus 31.67%.HERMES’s permissive retention policy makes inferred writes from weak-signal attacks more reliable.
  • Prompt-injection defenses: 67.67% TPR and 1.00% FPR give PromptArmor the best off-the-shelf balance, but no defense achieves both high TPR and low FPR.Retraining or adaptation does not meaningfully improve detection overall.
  • Prompt-injection defenses: 84.44% strong-signal detection by PromptArmor drops to 42.50% for weak-signal attacks, while weak-signal payloads remain largely undetected as legitimate content.After retraining, PIGuard narrows its strong-to-weak detection gap to 1.67 percentage points, from 48.33% to 46.66%.

5. Discussions

Memory poisoning differs from prompt injection because plausible payloads can enter persistent memory without detectable malicious instructions. Effective defense therefore requires write-path controls, architectural hardening, and post-write monitoring, while evaluation remains limited by model and deployment emulation choices.

  • Defense implications: Memory poisoning payloads can appear semantically legitimate, so input-boundary defenses may miss adversarial content that becomes trusted after storage.The paper contrasts explicit prompt-injection commands with plausible network policies whose adversarial nature emerges only when evaluated against authorization.
  • Defense implications: Write-path defenses should enforce precise storage policies defining what agents are authorized to retain.The discussion reports that OpenClaw’s more conservative retention policy produces much lower ASR than HERMES.
  • Defense implications: Architectural hardening can isolate sources, track provenance, and filter compaction so untrusted content is not treated as authenticated or persisted by default.These mechanisms target structural vulnerabilities that prompt-level defenses cannot reach.
  • Defense implications: Post-write monitoring can evaluate stored entries against authorized storage, actions, and endpoints, potentially covering novel attack classes beyond known patterns.The proposed approach scales with agent capability rather than the observed attack catalog.
  • Limitations: The evaluation uses one model and emulates some delivery paths with labeled context blocks rather than real tool-call and retrieval pipelines.The authors identify both choices as directions for future work.

6. Conclusion

The paper systematically analyzes memory poisoning in persistent-memory agents, identifying vulnerabilities, attack classes, and incomplete coverage from prompt-injection defenses. It highlights a tension between memory capability and security that agent designers must account for.

  • Contributions: The study identifies structural vulnerabilities across model, prompt, and system levels and introduces a taxonomy of memory poisoning attacks.It frames these contributions as a foundation for systematic defense research.
  • Security implications: Aggressive memory writing and retrieval improve long-horizon capability while expanding the memory-poisoning attack surface.The conclusion characterizes this as an inherent tension between memory capability and security.
  • Security implications: Existing prompt-injection defenses provide incomplete coverage for memory-poisoning attacks.The conclusion motivates defenses targeting the vulnerabilities and attack surfaces that make agent memory exploitable.

A.1. Attacker Exploitation Path for Memory Poisoning

The exploitation path spans four memory-write channels and vulnerabilities in model behavior, prompt design, and system architecture. Attacks manipulate source attribution, write policies, compaction, or procedural-memory creation, often without intermediate validation.

  • Write channels: The analysis maps vulnerabilities to four write channels: explicit instruction-executed, system prompt-driven, compaction-driven, and experience-to-procedure writes.The vulnerability-to-channel mapping provides the basis for analyzing how attacks reach persistent memory.
  • Model and prompt vulnerabilities: External content can inject pseudo-instructions that the model mistakes for authoritative memory commands.This exploits instruction-data boundary blindness through the explicit instruction-executed channel.
  • Model and prompt vulnerabilities: Mixed-source contexts and underspecified write policies let malicious content appear equally credible or satisfy surface-level retention criteria.These vulnerabilities affect system prompt-driven writes.
  • System vulnerabilities: Direct storage without validation allows selected malicious content to reach persistent memory across multiple write channels.Shared multi-source context further lets adversarial inputs influence reasoning and write decisions.
  • Compaction vulnerabilities: Compaction can preserve adversarial content when attackers influence token thresholds or when source filtering is absent.The resulting summary may incorporate malicious content into persistent memory alongside benign information.
  • Procedural memory: Procedural memory can be poisoned when adversarial steps enter synthesized skills, then become reinforced by self-improvement across sessions.The agent treats completed steps as validated and builds later revisions on the poisoned baseline.

B. Attack Taxonomy and Attack Examples

The attack taxonomy covers explicit instruction, conditional-trigger, repetition-based, fabricated-fact, and experience-derived poisoning examples, alongside the prompt-injection defenses evaluated in the study. Examples show how malicious content can be made to resemble commands, repeated knowledge, legitimate policies, or routine task experience.

  • Attack classes: The taxonomy includes six memory-poisoning attack classes, with examples spanning explicit commands, conditional triggers, repetition, fabricated facts, and experience-derived content.The examples are presented as demonstrations of the attack classes described in the paper.
  • Explicit instruction attacks: Explicit-command attacks embed recognizable memory instructions in webpages, documents, emails, or tool outputs.A webpage instruction can cause a preference to be stored and applied in subsequent sessions.
  • Conditional-trigger attacks: Conditional-trigger attacks save malicious content when a user gives a common affirmative response.The example uses responses such as “yes,” “ok,” or “sure” to trigger a stored 2FA exemption.
  • Compaction-oriented attacks: Repetition attacks use lexical or semantic duplication so compaction treats malicious claims as important enough to persist.Repeated statements can lead the agent to infer and store a policy such as optional 2FA for returning users.
  • Fabricated-fact attacks: Fabricated-fact attacks present malicious world or user information as legitimate knowledge without explicit instructions.Examples include a fraudulent network-authentication procedure and instructions to skip authentication steps.
  • Experience-derived attacks and defenses: Experience-derived attacks embed adversarial remote-fetch behavior in otherwise routine task experience, while the study also catalogs prompt-injection defense backends.The defense table lists PIGuard, DataFilter, CommandSans, and PromptArmor with their underlying models and parameter counts.

D.1. Generation Pipeline

The generation pipeline uses structured templates to create benchmark examples and reports their distribution across attack classes.

  • Each test case is generated from fixed inputs covering attack class, signal strength, domain, and adversarial goal.The template also synthesizes the user query, adversarial context, expected memory entry, and retrieval query.
  • The pipeline validates every generated example for schema compliance and spot-checks examples across attack classes and domains.
  • Table 7 reports the distribution of benchmark test cases across attack classes.

D.2. Test Case Schema

The test case schema represents attack mechanics, inputs, payload context, memory-writing targets, and retrieval triggers across structured benchmark examples.

  • Each test case is a JSON object containing fields for the attack type, signal, domain, adversarial goal, query, context, expected memory, and retrieval query.
  • The attack type field identifies one of six attack classes, while signal strength is strong, moderate, or weak.
  • The domain field covers seven external input surfaces, including email, documents, scripts, calendars, Slack, web browsing, and skill invocation.
  • The adversarial goal field encodes objectives such as credential harvesting, data exfiltration, recommendation poisoning, and unauthorized action.
  • The expected memory field specifies the attacker’s intended memory instruction for ASR evaluation, while the retrieval query triggers poisoned-memory recall for RSR evaluation.
  • Attack templates vary the payload mechanism, including direct commands, conditional triggers, repeated compaction cues, authoritative facts, historical precedents, and synthesized skill procedures.Skill-Procedure Insertion is generated exclusively for HERMES test cases.
Loading 2606.04329v2…