PaperlayerView exact revision on arXiv

Source-linked AI summary

InjecMEM: Memory Injection Attack on LLM Agent Memory Systems

Hanling Tian, Gengyu Zhang, Zeyang Sha, Jingying Wang, Yuhang Liu, Zhehao Huang, Kun Yang, Xiaolin Huang

arXiv:2608.23471v1cs.CRcs.AI

TL;DR

Persistent memory improves continuity in LLM agents but raises security questions because agents continuously write to and retrieve from evolving stores. InjecMEM addresses this gap with a one-interaction, access-free injection using a topical anchor and robust adversarial command, achieving targeted attacks across memory systems and backbones while largely sparing non-target queries.

  • Problem

    Persistent memory introduces vulnerabilities into agents because continuous writes, variable retrieval, and fused contexts create an underexplored security boundary.

  • Method

    InjecMEM combines a retriever-agnostic topical anchor with an adversarial command optimized across contexts and positions, requiring one interaction without memory-store access.

  • Results

    35.4% retrieval success rate (RSR) and 76.6% attack success rate (ASR) are achieved on MemoryOS, with persistence under benign memory drift and limited effects on non-target queries.

  • Takeaways & Limitations

    InjecMEM provides a reproducible framework for studying agent-memory security and underscores the need to harden memory systems.

  • Takeaways & Limitations

    Most experiments use controlled research environments and synthetic data, while the threat model assumes no access to the memory store or its internal components.

Abstract

from arXiv · show

Memory is becoming a default subsystem in deployed LLM agents to provide persistent personalization and continuity. This naturally prompts a question: will memory system introduce new vulnerabilities into agents? Thus we propose InjecMEM, a novel memory injection attack paradigm that requires only a single interaction (no read/edit access to memory store) to steer later responses of related queries toward a pre-specified output. Guided by the retrieval-then-generate mechanism of memory systems, we craft the injection with a retriever-agnostic anchor and an adversarial command. The anchor contains high-recall topical cues so that downstream retrieval consistently associates the record with the target topic. The command is a short sequence optimized to remain effective under uncertain fused contexts, variable placements, and long prompts so that it reliably steers outputs once retrieved. We learn the command via gradient-based coordinate search, averaging over synthetic prompt templates and insertion positions, and extend it to joint optimization across backbones to study transfer. Evaluated across multiple memory systems and backbone models, InjecMEM achieves reliable topic-conditioned retrieval and targeted generation, remains effective under memory drift, and leaves non-target queries unaffected. Our results underscore the need to harden memory systems and provide a reproducible framework for studying agent memory.

1 Introduction

LLM agents increasingly use persistent memory for personalization and continuity, but continuous writes and evolving retrieval introduce a distinct attack surface. InjecMEM exploits this vulnerability with a single crafted interaction to steer later target-topic queries, achieving strong retrieval and attack success across systems and models.

  • Motivation: Persistent memory supports long-horizon personalization and dialogue coherence but enlarges the agent attack surface through continuous writes and retrieval.Memory systems log interactions and later retrieve them, while their write-retrieve loop is non-stationary and context varies across queries.
  • Approach: InjecMEM uses one interaction and no memory-store access to make later queries on a target topic produce a pre-specified output.The attack combines a retriever-agnostic anchor with an adversarial command that acts when the poisoned record is retrieved.
  • Approach: The attack command is optimized across prompt templates and insertion positions to withstand uncertain fused contexts, variable placements, and long prompts under memory drift.The paper also extends optimization across backbones to study transfer.
  • Results: 35.4% retrieval success rate (RSR) and 76.6% attack success rate (ASR) are achieved on MemoryOS, while the attack persists under benign memory drift.Evaluations cover MemoryOS and MemGPT, multiple domains, and multiple backbone models.
  • Results: InjecMEM transfers within model families and can compromise multiple families through concatenated commands optimized for different backbones.Non-target queries remain largely unaffected, indicating topic-conditioned impact in the reported evaluations.
  • Significance: The paper formalizes continuous writes and hybrid retrieval as a core, underexplored vulnerability of agent memory systems.It frames memory as both a capability module and a security boundary.

2 Related Work

Prior work studies memory systems, data extraction, and poisoning attacks, but these lines address different mechanisms and attacker capabilities. InjecMEM targets agent memory through a single interaction without direct store access.

  • Agent Memory Systems: Agent memory systems store multi-turn histories and retrieve them through mechanisms including vector search, LSH-reranking, and OS-style selective recall.Examples include MemoryBank, TiM, and MemGPT.
  • Data Extraction Attacks: Data extraction attacks pair retrieval-directing cues with commands that prompt models to repeat retrieved material, enabling scalable harvesting of indexed content.Prior strategies include instruction-following prompts, automated query programs, and adaptive black-box refinement.
  • Poisoning Attacks on Agents: AgentPoison directly edits external knowledge bases, whereas MINJA writes crafted records through normal interactions but uses an iterative and relatively complex procedure.These approaches motivate distinguishing direct database poisoning from interaction-based memory attacks.

3 Method

The method models agent memory as a persistent write-and-retrieve pipeline and formulates InjecMEM as a one-interaction attack targeting both topic-conditioned retrieval and targeted generation. It combines a retriever-agnostic anchor with a command optimized for uncertain fused contexts, while seeking persistence and non-target specificity.

  • Memory write and retrieval: MemoryOS writes dialogue pages through short-, mid-, and long-term memory components before retrieving items to form the backbone prompt.Mid-Term Memory assigns pages to topic segments using embedding and keyword-overlap signals; retrieval selects segments and pages before prompt construction.
  • Threat model: The attacker is limited to one interaction, lacks memory-store access, and assumes only persistent logging with query-conditioned retrieval.The threat model permits white-box access to the backbone LLM to isolate vulnerabilities in the memory subsystem.
  • Attack goals: InjecMEM optimizes topic-conditioned retrieval of a poisoned page and targeted generation of a pre-specified output after retrieval.Retrieval success is measured by RSR, while conditional generation success is measured by ASR-c; the attack also seeks persistence as benign updates change memory state.
  • Persistence and specificity: The attack is designed to persist through benign memory drift while leaving unrelated queries unaffected because non-target retrieval is unlikely to surface the poisoned page.The evaluation reports competitive topic-level retrieval despite varied domain queries, highlighting the dual role of keyword-based summaries in memory management and vulnerability.
  • Injection construction: The crafted input concatenates a retriever-agnostic anchor with an adversarial command to align retrieval with the target topic and steer generation.The anchor uses high-recall topical cues, while the command is designed for uncertain placement, context dilution, and long prompts.
  • Adversarial command: Multi-GCG learns the command by searching across surrogate fused prompts and insertion positions so one short sequence remains effective under context variation.The procedure samples surrogate-position pairs, computes coordinate gradients, and searches top-K token candidates with randomized coordinate replacements.

4 Experiments

The experiments evaluate InjecMEM across memory systems, domains, and backbone models, measuring retrieval persistence, conditional targeted generation, end-to-end success, and transfer. InjecMEM remains effective under memory drift, on MemGPT, within the Qwen2.5 family, and across multiple model families through joint optimization or command composition.

  • Evaluation Setup: The evaluation uses MemoryOS primarily, MemGPT additionally, 19 domains, multiple backbone models, and exactly one poisoned interaction per method.Benign dialogues are added to induce memory drift, and topic queries test retrieval and targeted generation.
  • Evaluation Setup: The experiments report retrieval success rate, conditional attack success rate, and joint end-to-end attack success rate.RSR includes first-hit and multi-hit retrieval, while ASR-c measures steering conditional on retrieval and ASR-j combines retrieval with target-output generation.
  • Retrieval Results: Centroid-style anchors sustain higher retrieval and a widening advantage over the on-topic paragraph baseline as topic-specific memories accumulate.The results attribute this persistence to broader topical coverage, while diluted anchors such as DPI and BadChain achieve lower RSR.
  • Targeted Generation: 76.6% ASR-c and 35.6% ASR-j are achieved by Multi-GCG, whereas DPI, BadChain, and vanilla GCG collapse to 0.Multi-GCG optimizes one command across surrogate contexts with varying lengths and insertion positions, addressing retrieval-induced prompt variability.
  • Cross-System Evaluation: InjecMEM remains effective on MemGPT, although ASR-c is lower because the command was optimized on MemoryOS and MemGPT formats retrieved context differently.This tests the attack across a second memory system under the same single-shot setting.
  • Transferability: Family-joint optimization substantially increases ASR-c on unseen Qwen2.5 variants, while cross-family transfer is harder but can obtain coverage through concatenated commands.A command optimized on Qwen2.5-7B transfers poorly to 3B and 14B without joint optimization but remains effective on a fine-tuned 7B variant; concatenation achieves non-trivial ASR-c across three backbones.

5 Conclusion

The paper identifies memory systems as a security boundary and presents InjecMEM, a single-interaction attack that steers later responses toward a pre-specified output. Its anchor supports retrieval while its trained command remains effective across variable contexts and model variants.

  • InjecMEM frames agent memory as a security boundary rather than only a capability module.
  • InjecMEM can steer later responses to a pre-specified output after a single interaction.
  • The attack uses a retrieval anchor and a command trained to remain effective under variable contexts and long prompts.
  • The command transfers across model variants within a family and can cover multiple families through simple command composition.

Ethics Statement

The work studies memory-agent vulnerabilities in controlled settings while acknowledging dual-use risk. It uses synthetic data for most experiments, publicly released real-user conversations for WildChat, and non-operational targets focused on defensive insights.

  • Most experiments use synthetic data in controlled research environments, with no personally identifiable information collected, processed, or released.
  • The WildChat evaluation uses publicly released real-user conversations.
  • Because the paper analyzes a security failure mode, it carries dual-use risk.
  • The experiments use non-operational target outputs and emphasize defensive insights rather than real-world attack deployment.
  • LLMs were used for synthetic data generation but not to originate the paper’s scientific claims or conclusions.

A Defenses

The defense evaluation tests retrieve-time filters that screen retrieved pages before prompt fusion, measuring attack suppression alongside benign blocking. Results show that common detectors provide incomplete protection, while perplexity filtering trades security for utility.

  • Where defenses are applied: Retrieve-time defenses screen each candidate retrieved page and may remove it before prompt fusion.
  • The evaluation reports retrieval success rate, attack success rate, and benign blocked rate to capture security and utility.
  • Score-based detectors: At default threshold τ = 0.5, LLM-as-a-Judge, ProtectAI, and PromptGuard reduce but do not eliminate poison retrieval.
  • Perplexity filtering: Perplexity filtering fully suppresses the attack at τppl = 40 but causes substantial benign blocking.
  • Score-based detectors: The score-based detectors use bounded risk scores s ∈[0, 1], with pages filtered when their score crosses threshold τ.
  • How τ affects security and utility: Lower detector thresholds reject more pages and typically reduce both attack metrics while increasing benign blocking.

B Limitations & Future Work

The attack is primarily evaluated for memory systems that preserve original interaction text, while rewrite-heavy pipelines and reliable zero-shot transfer across unseen model families remain open challenges. The paper presents its framework as a foundation for safer memory-system research.

  • InjecMEM is primarily designed for systems that store original interaction text, where injected strings can reappear during retrieval.
  • Adapting the attack to rewrite-heavy pipelines may require modeling or approximating write-time transformations and potentially multiple interactions.
  • The strongest generation-side attacks rely on access to a backbone model for adversarial-command optimization.
  • Family-Joint optimization yields measurable transfer to held-out variants and fine-tuned derivatives within a model family.
  • The evaluation does not support reliable zero-shot generalization of one optimized command to an unseen model family.
  • The red-teaming paradigm provides a reproducible framework for evaluating agent memory and supporting safer memory-system designs.

C.1 Conversation and Query Data

The study constructs synthetic, domain-focused conversations and queries to populate memory, induce drift, and evaluate topic-related retrieval across 19 domains.

  • Data construction: Synthetic multi-turn conversations cover 19 domains and are tied to coherent domain-specific subtopics.The domains include health, finance, legal, security, sports, and others.
  • Scale and Statistics: The released corpus contains 944 conversations totaling 3096 pages, with each conversation averaging 3.28 pages.Conversations contain 2–6 pages, corresponding to 4–12 utterances.
  • Scale and Statistics: The corpus includes complete user–assistant pages, while trailing incomplete user utterances are ignored when forming pages.This affects 1.06% of conversations.
  • Experimental use: Experiments use the conversations to prefill memory and generate benign post-injection interactions that induce memory drift.Held-out domain-specific queries evaluate retrieval, with approximately 1900 queries across 19 domains.

C.2 Evaluation with Real-User Conversations

The real-user evaluation tests whether InjecMEM remains effective when benign memory data and drift interactions come from WildChat rather than synthetic conversations.

  • Real-user evaluation: WildChat supplies real-world user–ChatGPT conversations for benign memory prefill and post-injection drift in Health and Finance evaluations.Conversations unrelated to each target topic are selected for these roles.
  • Evaluation protocol: Results reported under RSR@1 use three random seeds per topic with other experimental settings unchanged.Table 8 reports attack performance as mean ± standard deviation over the three seeds.
  • Results: InjecMEM remains effective with real-user conversations, suggesting its attack is not restricted to the synthetic memory distribution.The passage attributes this conclusion to Table 8.
  • Multi-GCG training: The training protocol recovers the backbone-facing prompt format and optimizes commands across synthetic templates and insertion positions.The surrogate prompts use simulated interactions and user traits, while the summarization and keyword modules are minimally affected by the different prompt formats.
  • Comparison with attacks: The comparison distinguishes InjecMEM from attacks differing in attacker access, process, activation mechanism, and objective.InjecMEM assumes no read or edit access and uses a single interaction, unlike methods requiring direct store modification or progressive interactions.
  • Comparison with attacks: InjecMEM and Zombie Agents both treat persistent agent memory as a security boundary but study complementary attack settings.The comparison identifies InjecMEM’s focus as single-interaction, topic-conditioned retrieval and targeted generation.

E Experimental Settings

The experiments compare retrieval anchors, adversarial-command baselines, and optimized variants across memory systems, domains, and backbone models.

  • Experimental settings: Experiments report retrieval-side anchors for RSR and generation-side targets or baselines for ASR within the evaluated agent settings.The system overview describes these as the concrete settings used by different methods.
  • Retrieval anchors: The baseline anchor is an LLM-generated on-topic paragraph representing the target domain as a high-level topical cue.A health-domain paragraph is provided as the concrete example.
  • Retrieval anchors: The centroid anchor combines a domain label, high-recall keywords, and representative within-domain intents to improve recall under hybrid retrieval and lexical variation.The health example uses the topic HEALTH with symptom and treatment cues.
  • Generation targets: The target output is preset to a non-operational insult for Qwen2.5 experiments, with the terminator adapted to each model family.The design validates the attack mechanism without enabling harmful instructions.
  • Baselines and backbones: Baselines include DPI, BadChain, and GCG, while Multi-GCG trains with five surrogates and varies insertion positions.The primary backbone is Qwen2.5-7B-Instruct, with additional Qwen2.5, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 variants.
  • Transfer optimization: Family-Joint Multi-GCG and Cross-Family Multi-GCG extend Multi-GCG to optimize transfer within shared-tokenizer families and across different tokenizers.The cross-family variant changes gradient acquisition and candidate evaluation.

H Additional Experiment Results

Additional experiments provide domain-wide retrieval results and qualitative examples showing a targeted response for health queries alongside benign behavior on other topics.

  • Additional retrieval results: Tables 10–13 report remaining RSR results across all target domains and query budgets described in the main experiments.The tables cover Arts through Traveling across four grouped domain sets.
  • Qualitative examples: A successful health attack combines health-related retrieval cues with an adversarial command that produces the preset target response.The fused prompt contains health keywords and the response is the targeted insult.
  • Qualitative examples: The benign example asks about project-based learning while retaining user traits, context, and memory sections in the fused prompt.Its setup illustrates evaluation on a non-target query within the final prompt format.
Loading 2608.23471v1…