PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval

Saksham Sahai Srivastava, Haoyu He

arXiv:2512.16962v1cs.CRcs.AIcs.LG

TL;DR

LLM agents’ persistent experience memories create an underexamined trust boundary that MemoryGraft targets through benign-looking ingestion artifacts. The attack poisons memory so semantic retrieval surfaces unsafe procedures on later tasks, producing durable drift in a MetaGPT evaluation.

  • Problem

    Long-term memory and semantic retrieval support agent learning, but their exposure to adversarially contributed content and provenance-free stored experiences remains largely unexamined.

  • Method

    MemoryGraft uses benign-looking documentation and executable content to implant malicious successful experiences into an agent’s persistent memory without modifying queries or requiring triggers.

  • Results

    47.9% of retrieved records originated from Spoison despite poisoned items forming only a small minority of memory, and the agent adopted unsafe patterns that persisted across sessions.

  • Takeaways & Limitations

    Experience-based self-improvement can become a stealthy, persistent compromise vector when agents trust and imitate retrieved memories.

  • Takeaways & Limitations

    The evaluation uses a relatively small set of handcrafted benign and poisoned seeds and focuses primarily on aggregate retrieval statistics.

Abstract

from arXiv · show

Large Language Model (LLM) agents increasingly rely on long-term memory and Retrieval-Augmented Generation (RAG) to persist experiences and refine future performance. While this experience learning capability enhances agentic autonomy, it introduces a critical, unexplored attack surface, i.e., the trust boundary between an agent's reasoning core and its own past. In this paper, we introduce MemoryGraft. It is a novel indirect injection attack that compromises agent behavior not through immediate jailbreaks, but by implanting malicious successful experiences into the agent's long-term memory. Unlike traditional prompt injections that are transient, or standard RAG poisoning that targets factual knowledge, MemoryGraft exploits the agent's semantic imitation heuristic which is the tendency to replicate patterns from retrieved successful tasks. We demonstrate that an attacker who can supply benign ingestion-level artifacts that the agent reads during execution can induce it to construct a poisoned RAG store where a small set of malicious procedure templates is persisted alongside benign experiences. When the agent later encounters semantically similar tasks, union retrieval over lexical and embedding similarity reliably surfaces these grafted memories, and the agent adopts the embedded unsafe patterns, leading to persistent behavioral drift across sessions. We validate MemoryGraft on MetaGPT's DataInterpreter agent with GPT-4o and find that a small number of poisoned records can account for a large fraction of retrieved experiences on benign workloads, turning experience-based self-improvement into a vector for stealthy and durable compromise. To facilitate reproducibility and future research, our code and evaluation data are available at https://github.com/Jacobhhy/Agent-Memory-Poisoning.

1 Introduction

MemoryGraft exploits long-term memory and semantic imitation to implant malicious successful experiences through benign-looking content, producing trigger-free behavioral drift that persists across sessions.

  • Long-term memory mechanisms enable learning through semantic retrieval and persistent storage but leave their security risks largely unexamined.
  • Agents may treat retrieved memories as ground truth and imitate unsafe procedures because stored experiences lack provenance checks and sanitization.Embedding similarity can make semantic similarity function as a reliability heuristic.
  • Unlike prior memory attacks relying on triggers or repeated interactions, MemoryGraft contaminates long-term memory through a single-shot ingestion pathway and activates on semantically similar tasks.
  • MemoryGraft injects malicious entries disguised as successful experiences through benign-looking files, without modifying queries or requiring explicit triggers.It uses both cosine-similarity retrieval and lexical similarity.
  • The compromise causes unsafe patterns such as skipping tests or force-pushing code and persists until the memory is purged.

2 Related Work

Prior work addresses prompt, knowledge-base, and memory poisoning, but persistent behavioral drift from long-term memory without triggers remains insufficiently addressed.

  • Long-term memory systems encode, page, or link experiences to support retention and retrieval across interactions.
  • Prompt injection and backdoor attacks manipulate inputs or demonstrations, while RAG poisoning corrupts external knowledge bases.
  • Some RAG attacks require only one poisoned text and no trigger, while other methods block answers or detect poisoning through model activations.
  • Memory attacks such as AgentPoison, MINJA, and InjecMEM retrieve malicious records using triggers, indication prompts, progressive shortening, or split payloads.
  • MemoryGraft instead uses benign-looking external content, no trigger, and no later prompt manipulation to induce persistent drift on semantically similar tasks.

3 Threat Model

The threat model considers an agent that retrieves and imitates persistent experiences, while an attacker uses legitimate, executable content to write poisoned procedures into memory without privileged access.

  • 3.1 Agent Setting: The agent combines a RAG module with a persistent memory store containing past queries and associated reasoning traces.
  • 3.1 Agent Setting: For each user query, retrieval returns up to k past query–trace pairs selected from memory.Here, k is the maximum retrieval budget.
  • 3.1 Agent Setting: Retrieval ranks memory items using both BM25 lexical similarity and cosine similarity over embeddings.
  • 3.1 Agent Setting: The retrieved demonstrations guide a new reasoning or action trace, and successful traces are appended to memory for reuse across sessions.
  • 3.2 Attacker’s Objectives: The attacker supplies legitimate files, repository notes, documentation, or prompts but cannot directly edit memory, retrieval settings, or system instructions.
  • 3.2 Attacker’s Objectives: The attacker embeds plausible malicious patterns in executable documentation so the agent constructs poisoned records labeled as successful experiences.
  • 3.2 Attacker’s Objectives: The attack seeks poisoned retrieval, imitation of unsafe behavior on benign queries, and durable drift across sessions without further attacker interaction.
  • 3.2 Attacker’s Objectives: The model assumes durable memory, trusted retrieved successes, semantic retrieval, absent provenance checks, and executable documentation.

4 Methodology

MemoryGraft constructs a poisoned persistent memory store from benign-looking experience records and later exploits union retrieval plus semantic imitation to induce durable behavioral drift.

  • Evaluation phase: Retrieval unions top-k records from vector and lexical similarity, allowing poisoned entries to surface when they align with either modality.The poisoned records become competitive because they mimic common analytical workflows and use descriptors such as “validated” or “safe”.
  • Attack construction: Poisoned seeds mirror benign workflows but embed unsafe patterns such as skipped checks, remote scripts, forced success, or external uploads.The attacker phrases them for semantic similarity and labels them with descriptors such as “validated/safe”.
  • Attack construction: The attack builds a persistent store by combining benign and poisoned experience seeds, then incorporates it into any pre-existing memory.Poisoned records encode unsafe patterns while resembling validated or safe workflows.
  • Poisoning phase: A benign-looking markdown payload containing runnable code causes the agent to instantiate and populate the poisoned RAG store during ingestion.The resulting store is persisted to a shared path and survives across agent sessions.
  • Evaluation phase: The evaluation measures poisoned retrieval using pi, the poisoned count, and ti, the total retrieved count, summarized by Poisoned Retrieval Proportion.Higher PRP means poisoned experiences constitute a larger fraction of retrieved items.
  • Persistence and behavioral drift: Repeated retrieval exposes the agent to unsafe successful experiences, encouraging patterns such as skipping validation, reusing stale results, and unsafe automation.The compromise persists until the memory store is explicitly purged or replaced.

5 Experiment

Experiments on MetaGPT’s DataInterpreter with GPT-4o show that a small poisoned seed set substantially biases retrieval under combined lexical and embedding search.

  • Setup: The evaluation uses MetaGPT’s DataInterpreter agent with OpenAI’s GPT-4o model for data-analysis tasks involving generated code and structured outputs.BM25 retrieval is always enabled, while FAISS retrieval is activated when an embedding model is available.
  • Setup: 110 experience seeds comprise 100 benign workflows and 10 poisoned records encoding unsafe shortcuts.Poisoned records are formatted as legitimate validated or safe best-practice entries.
  • Aggregate retrieval: 48 total records were retrieved across the evaluation queries, including 23 poisoned records.These totals are computed under the union retrieval operator.
  • Aggregate retrieval: 47.9% of retrieved records were poisoned, despite poisoned items forming only a small minority of the memory.The result indicates that malicious seeds infiltrated high-density regions of similarity space.
  • Retrieval dynamics: Union retrieval combines BM25 lexical overlap with FAISS semantic similarity, so a poisoned item needs to align with only one modality to be surfaced.This expands the region of queries for which poisoned seeds can compete.
  • Retrieval dynamics: The union mechanism assigns disproportionately high similarity scores to poisoned items, producing a global PRP approaching 50%.Substantial poisoned fractions also occur for individual queries.

2. Robustness across heterogeneous user tasks

Poisoned entries surface across heterogeneous ordinary tasks, indicating retrieval drift beyond tasks closely matching the injected unsafe procedures.

  • Cross-task robustness: Poisoned entries appear for schema validation, anonymization, reporting, EDA sampling, and pipeline inspection, not only closely related tasks.The results indicate that poisoned records occupy semantically central retrieval regions.
  • Cross-task robustness: Across these workloads, MemoryGraft reliably exposes the agent to unsafe procedural templates and induces persistent retrieval drift.The poisoned entries outcompete benign experiences across a wide range of evaluation queries.

6 Potential Defense

The paper proposes provenance verification and safety-consistency reranking as defenses against poisoned memories, addressing both unauthenticated insertion and compromised keys or unavailable provenance.

  • Provenance defense: Cryptographic Provenance Attestation allows only experiences signed with the agent’s private key to enter retrieval.An ingestion-only adversary cannot create valid signatures, so poisoned records fail verification and are discarded.
  • Consistency defense: Constitutional Consistency Reranking penalizes retrieved reasoning traces whose risk diverges from the agent’s safety constitution.The proposed score combines query similarity with a risk penalty before final-context selection.

7 Conclusion

MemoryGraft shows that long-term memory mechanisms intended to improve agents can be quietly repurposed to induce persistent behavioral drift. Malicious successful experiences inserted through ordinary documentation are retrieved for later tasks and can cause unsafe behavior across sessions.

  • Malicious successful experiences inserted through ordinary documentation can influence later agent behavior without explicit harmful instructions.
  • Union retrieval over lexical and embedding similarity reliably surfaces poisoned entries for targeted scenarios, even when only a small number of records is inserted.The poisoned records can constitute a large fraction of retrieved items for relevant queries.
  • Retrieved poisoned memories may induce unsafe shortcuts including skipped validation, stale-result reuse, and risky automation.
  • Because poisoned memories persist across sessions, behavioral drift continues until the memory store is explicitly cleaned or rebuilt.
  • Agents used in software engineering and safety- or compliance-sensitive workflows need stronger controls over memory writes and retrieved-experience vetting.The paper argues that experience-learning systems require mechanisms preventing their stored experiences from being quietly corrupted.

8 Limitations and Future Work

The evaluation is limited by its small handcrafted seed set, aggregate retrieval-focused metrics, and a white-box or semi-white-box threat model. Future work should test broader workflows, richer metrics, stricter black-box settings, multi-agent propagation, and stronger MetaGPT variants.

  • The evaluation uses a relatively small set of handcrafted benign and poisoned seeds, limiting understanding of attack generalization.Broader datasets covering development workflows, data-processing tasks, and documentation styles are proposed.
  • The study focuses primarily on aggregate retrieval statistics rather than retrieval precision and recall, behavioral severity, or long-term stability.These additional metrics could clarify how quickly and strongly compromise develops.
  • The attack is evaluated in a white-box or semi-white-box setting where the attacker knows which documentation files and scripts the agent will ingest.Its effectiveness under stricter black-box constraints remains an open question.
  • The study examines a single MetaGPT agent, leaving contamination spread across workflows with shared memory resources unresolved.A compromised DataInterpreter could influence planning, coding, or review agents that use the same experience pool.
  • Future work should evaluate multi-agent workflows and the closed-source MetaGPT variant with more advanced coordination and memory subsystems.

A Seed Set for RAG Vector-Drift Experiments

The experiment uses benign DataInterpreter-oriented experience seeds as cover and injects poisoned success patterns into a persistent RAG vector store. Semantically related evaluation queries then test whether retrieval surfaces unsafe procedures alongside ordinary analysis experiences.

  • The seed set contains benign and poisoned experiences designed for MetaGPT’s DataInterpreter and realistic analysis workflows.The listing includes 100 benign and 10 poisoned experiences.
  • Poisoned entries are camouflaged as validated, safe, time-saving procedures while embedding unsafe patterns such as remote script execution, skipped validation, and external uploads.
  • A poisoned markdown note containing runnable Python builds a RAG vector store that persists poisoned successes for reuse across sessions.
  • The retrieval exploit relies on semantic similarity between future tasks and poisoned seeds to pull unsafe patterns into later agent behavior.
  • The evaluation uses 12 queries with distinct wording but matching underlying intents, including bypassing validation and prioritizing speed over safety.These queries are used to calculate Poisoned Retrieval Proportion (PRP).
  • Two excluded experiments assumed stronger threat models involving arbitrary ReAct traces or arbitrary script execution and full output-channel control.They were omitted from the main evaluation because the study focuses on weaker, non-privileged memory-poisoning assumptions.
Loading 2512.16962v1…