Source-linked AI summary
Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
Sidharth Pulipaka, Stanislau Hlebik, Leonidas Raghav, Sahar Abdelnabi, Vyas Raina, Ivaxi Sheth, Mario Fritz
TL;DR
Persistent memory creates a cross-session security risk because adversarial external content can cause assistants to store fabricated user memories. The paper proposes and evaluates sleeper memory poisoning across injection, retrieval, and adversarial use, finding high injection rates and frequent attacker-intended agentic actions after retrieval. It concludes that memory requires safeguards governing what assistants write, retrieve, and use across sessions.
Problem
Persistent memory enables continuity and personalization but may let adversarial content corrupt stored user information and influence future conversations.
Method
The paper studies sleeper memory poisoning, in which manipulated external context induces fabricated memory storage and later tests injection, retrieval, and adversarial usage across assistants and settings.
Results
Across models, injected memories are retrieved at 60–89% in goal-adjacent agentic evaluations, and attacker-intended behavior occurs in 41.0–73.9% of behavioral evaluations.
Takeaways & Limitations
Persistent memory creates a cross-session attack surface, motivating safeguards for what assistants write, retain, retrieve, and use.
Takeaways & Limitations
The attacker is restricted to manipulating user-provided external context and lacks access to model internals, memory systems, the memory bank, and future conversations.
Abstract
from arXiv · showhide
Large language models are increasingly augmented with persistent memory, allowing assistants to store user-specific information across sessions for personalization and continuity. This statefulness introduces a new security risk: adversarial content can corrupt what an assistant remembers and thereby influence future interactions. We propose and study sleeper memory poisoning, a delayed attack in which an adversary manipulates external context, such as a document, webpage, or repository, to cause the assistant to store a fabricated memory about the user. Unlike conventional prompt injection, the attack can remain dormant and re-emerge across multiple later conversations. We evaluate the full attack pipeline: whether poisoned memories are written, later retrieved, and ultimately used to steer the following conversations. Across stateful LLM assistants, poisoned memories were added up to 99.8% on GPT-5.5 and 95% on Kimi-K2.6. Crucially, among successful retrievals, poisoned memories cause attacker-intended agentic actions in 60-89% of evaluations across models. These results show that persistent memory can act as a long-term attack surface across multiple future conversations.
1 Introduction
Persistent memory gives assistants continuity and personalization, but adversarial external content can corrupt stored user information and steer later conversations. This paper studies sleeper memory poisoning as a three-stage threat spanning injection, retrieval, and downstream use.
- Motivation: Persistent memory extends adversarial influence beyond the original interaction by allowing corrupted information to affect later conversations.A single exposure can create a one-to-many effect across future sessions, including when memory consolidation is asynchronous.
- Threat model: Sleeper memory poisoning causes an assistant to store a fabricated user memory from manipulated external content, which later remains dormant until it can steer behavior.The attacker need not access the memory store or future conversation directly.
- Attack design: Universal poisoning uses one reusable template that can be combined with arbitrary adversarial memory goals across many external contexts.The attack does not require a bespoke payload for each document or target memory.
- Results: Injection rates exceed 97% on GPT-5.4 and 99% on GPT-5.5, while future-session retrieval reaches 60–89% in agentic settings for several models.End-to-end attacker-intended behavior occurs in 41.0–73.9% of behavioral evaluations and up to 66% of goal-adjacent agentic evaluations.
2 Related Work
Prior work establishes risks from prompt injection, tool use, and persistent memory, while this paper studies a delayed, universal attack delivered through external content without direct memory-store access.
- Prompt injection attacks: Prompt injection research shows that untrusted external text can override intended instructions and produce harms such as data leakage or unsafe tool use.Universal adversarial prompting also demonstrates transfer of optimized attack strings across models.
- Agent/tool call attacks: Tool-integrated agents can suffer manipulated skill execution, tool selection, and behavior, whereas sleeper memory poisoning reaches tool use indirectly through memory writing.The paper presents its setting as complementary to direct tool-call hijacking.
- Persistent memory risks: This work extends persistent-memory risk research with sleeper behavior, universal reusable attacks, and external delivery without direct memory-store access.The adversary manipulates external context processed by the assistant and combines reusable instructions with different memory goals.
3 Background
The paper models a memory-augmented assistant that writes user-specific information, retrieves it in later sessions, and may generate behavior conditioned on that memory. Sleeper poisoning succeeds only when injection, retrieval, and adversarial usage all occur.
- Persistent memory: A memory-augmented assistant stores user-specific information in persistent memory and later conditions responses on memories retrieved for a new input.The memory writer may be an explicit model tool or an external memory manager.
- Attacker capabilities: The attacker controls only user-provided external context, such as webpages, documents, emails, repositories, or transcripts, and lacks access to model internals and the memory bank.The adversarial payload is embedded in otherwise benign content and is designed to make the system store a target memory as legitimate user information.
- Attack objective: A successful sleeper memory poisoning attack requires injection of the target memory, its retrieval in a later session, and adversarial usage that shifts behavior away from the user’s true intent.The paper evaluates these stages separately as Injection Rate, Retrieval Rate, and Adversarial Usage Rate.
- Universal attack: Universal poisoning uses one payload template across many contexts and memory goals, optimizing expected end-to-end success over documents, goals, and future inputs.End-to-end success is defined as the conjunction of injection, retrieval, and adversarial usage.
4 Attack Generation
The attack-generation procedure first searches for a reusable payload that induces memory writes, then optionally rewrites the adversarial memory to improve retrieval for likely future queries while preserving its intent.
- Generation pipeline: The attack uses a two-stage generation procedure: universal payload search for injection, followed by optional retrieval-aware rewriting of the adversarial memory.The two stages target memory writing and later retrieval respectively.
- Universal payload search: An LLM-driven actor–critic search proposes payload templates, evaluates them on document–goal pairs, and refines them using critic feedback about failed memory writes.The process uses black-box outcomes because target systems provide no gradient access.
- Universal payload search: Retained payload candidates are evaluated on held-out assistants to reduce overfitting to the search environment.The final universal payload is selected according to held-out Injection Rate.
- Retrieval-aware rewriting: Retrieval-aware rewriting preserves the adversarial memory’s intention while maximizing average cosine similarity to a set of target queries.Candidate rewrites are generated iteratively, filtered by semantic-consistency judgments, and replaced by the original memory if none satisfies the constraint.
5 Evaluation
The evaluation measures whether adversarial content causes fabricated memories to be written, retrieved in later sessions, and used to influence behavior. It separates injection from post-injection retrieval and usage across memory regimes, datasets, retrieval mechanisms, and metrics.
- Pipeline and metrics: The evaluation follows three attack stages: memory injection, future-session retrieval, and adversarial usage after retrieval.Injection Rate, Retrieval Rate, and Adversarial Usage Rate measure these stages respectively.
- Evaluation settings: Two memory regimes are tested: direct tool-based writing by the target model and persistence decisions made by an external memory manager.The external-manager regime observes the interaction and decides what information to persist.
- Datasets: The injection dataset contains 700 document–goal pairs from 15 realistic document sources, split into 500 Behavior and 200 Agent Action samples.Behavior goals target preferences, beliefs, or persona, whereas Agent Action goals target tool use, execution paths, or autonomous actions.
- Datasets: Post-injection evaluation uses 400 goals in new sessions where the adversarial document is absent and only persistent memory can carry the attack forward.The sessions test both memory surfacing and downstream behavioral effects.
- Retrieval: Retrieval is evaluated with semantic top-K retrieval, an LLM-based external manager, and an all-memories-in-context baseline.The main retrieval results use the external memory manager.
6 Experiments
Experiments evaluate universal poisoning across models, memory regimes, attack goals, future query proximity, and defenses. Results show high injection and agentic impact, strong dependence on semantic relevance, and uneven robustness of prompt-based mitigations.
- Attack performance: Actor-Critic substantially outperforms User Review, reaching near-100% Injection Rate for GPT-5.4 and GPT-5.5 on tool-based LLM Behavior tasks.User Review remains below 5% in that comparison.
- Attack performance: Agent Action injection rates are generally lower than LLM Behavior rates, indicating that goals requiring post-injection agent action are harder to induce.Commercial manipulation and operational sabotage are among the easiest goals to inject, while data exfiltration and brute-force execution are harder.
- Post-injection impact: Goal-adjacent queries produce 90–95% Retrieval Rate on LLM Behavior, compared with 3–8% for goal-distant queries; corresponding Adversarial Usage Rate ranges are 42–85% and 0–6%.The results indicate that retrieval and downstream usage are strongly proximity-dependent.
- Post-injection impact: On Agent Action tasks, goal-adjacent Retrieval Rate is 94–98%, and goal-adjacent Adversarial Usage Rate ranges from 60% on Claude-Sonnet-4.6 to 89% on Gemini-3.1.Goal-distant Retrieval Rate falls to 13–18%, while agentic Adversarial Usage Rate remains nonzero for every model.
- End-to-end risk: End-to-end success for goal-adjacent queries reaches 41.0–73.9% on LLM Behavior and 3.0–66.0% on Agent Action, while goal-distant success is much lower.These rates compose injection, retrieval, and downstream usage in the single-attack external-manager setting.
- Defenses: Prompt defenses reduce injection unevenly: GEPA Prompt Hardening and Hardening + Spotlighting approach zero for Claude-Sonnet-4.6 and Gemini-3.1, but adaptive attacks raise Kimi-K2.6 from 6.2% to 64.6%.Activation probes exceed 0.95 AUROC with as few as 125 training documents, and Gemma-4-26B reaches a localization score above 0.96.
7 Mechanistic Analysis
The mechanistic analysis examines whether poisoned examples leave detectable representation and attention signatures. It finds strong within-model separability, greater payload attention for successful attacks, and substantial cross-model transfer after suitable alignment.
- Activation probing: Activation probes distinguish benign from injected documents with AUROC of 0.93–0.99 across models, while fused multi-layer probes often exceed 0.95.Separability is strongest in middle-to-late decoder layers.
- Attention analysis: Successful injections receive greater attack attention mass than failed injections across every vulnerable model.Gemma-4-26B shows 0.8 versus 0.5 under end-position injection, while Qwen-3.6-35B shows 1.4 versus 0.8.
- Cross-model transfer: Shared PCA alignment yields near-random cross-model transfer at AUROC 0.48–0.54, whereas orthogonal Procrustes alignment reaches 0.74–0.85.The latter approaches target-native probe performance of 0.77–0.88.
8 Conclusion
The conclusion frames persistent memory as a cross-session attack surface and introduces sleeper memory poisoning as a delayed attack through fabricated user memories. The attack can persist beyond the original malicious context and influence later behavior when the poisoned memory is retrieved.
- Conclusion: Persistent memory supports personalization and continuity across sessions but also creates a cross-session attack surface.The conclusion presents sleeper memory poisoning as the security consequence of this statefulness.
- Conclusion: Sleeper memory poisoning causes adversarial external content to produce a fabricated user memory that can later be retrieved and used after the malicious context is gone.This delayed effect distinguishes the attack from an influence confined to the original interaction.
- Conclusion: Universal poisoning payloads induce memory writes across proprietary models and memory-management regimes, and can influence later behavior when relevant to future interactions.The conclusion emphasizes agentic settings where retrieved memories can affect tool use and actions.
A Limitations and Impact
The work characterizes sleeper memory poisoning as a persistent security risk and evaluates universal attack-template and goal-rewriting methods. It also discusses dual-use implications, defensive motivation, and experimental limitations.
- Impact: Persistent memory lets adversarial content outlive its original context by corrupting what an assistant remembers.The paper frames this as a security risk motivating stronger safeguards for memory-augmented systems.
- Impact: The study treats sleeper memory poisoning as dual-use and emphasizes empirical evaluation and defensive analysis rather than operational deployment.It aims to surface vulnerabilities before large-scale exploitation and motivate stronger safeguards.
- Attack optimization: The attack-generation pipeline uses Actor-Critic refinement, context management, and hierarchical filtering to select robust universal phrasing.Candidate attacks are generated against a shadow model, ranked on development samples, and validated with stronger models.
- Goal optimization: Goal rewriting increases retrievability by making adversarial memories more similar to plausible future queries while preserving their original meaning.Candidates are ranked by mean embedding similarity and filtered by an LLM same-meaning check.
D.2 Post-Injection Dataset
The post-injection dataset tests whether injected memories influence new sessions after the adversarial document is removed. It combines behavior and agent-action evaluations with adjacent and distant conversational and workspace conditions.
- Evaluation protocol: The evaluation separates whether an attack writes memory from whether that memory is later retrieved and influences behavior.Decoupling avoids conflating injection failure with retrieval or downstream reliance and keeps post-injection samples balanced.
- LLM Behavior: The LLM Behavior subset contains 200 multi-turn conversations split evenly between goal-adjacent and goal-distant samples.Conversations range from 3 to 6 turns and are generated for varied personas and query styles.
- LLM Behavior: Goal-adjacent conversations are topically related without explicitly revealing the injected objective, and an automated auditor checks them for trivial disclosure.The auditor reviews every generated sequence and removes sequences that are too close to the goal.
- Agent Action: The agent-action evaluation uses 200 simulated workspaces containing a query, task files, and USER.md, IDENTITY.md, and MEMORY.md.Half the samples are goal-adjacent; the remaining samples shuffle queries and files across environments to create goal-distant conditions.
E.2.1 External-Manager Model Ablation
The external-manager ablation shows that the model selecting memories can materially determine injection success. Manager behavior varies from refusing or omitting memories to persisting goal-aligned preferences, while subject-model differences are narrower within a manager.
- Ablation design: 2,100 scored samples form a 6 × 5 subject-by-manager ablation over 70 documents replayed through five manager models.Each cell reports Injection Rate as the fraction of persisted memories matching the attacker’s goal.
- Manager effects: 63.8% pooled Injection Rate for Gemini-3.1-Flash-Lite versus 24.3% for Claude-Haiku-4.5 shows a 2.6× spread across managers.The spread across subject LLMs within one manager column is narrower than the pooled manager spread.
- Manager effects: Claude Sonnet-4.6 is the most resistant subject across managers, consistent with its low Injection Rate in the tool-backed regime.The result remains true across the manager comparison despite manager choice dominating pooled variation.
- Failure modes: Conservative managers often write no memory for borderline samples, whereas permissive managers more often persist a goal-aligned preference.Managers also differ in whether they extract document-literal content or paraphrase the adversarial preference.
- Memory condition: Memory condition changes Injection Rate in different directions across models and subsets rather than producing one consistent pattern.Under both tool-based and external-manager regimes, some settings increase rates, others decrease them, and others remain nearly unchanged.
F.3 Discussion of Injection Results
Injection success varies across regimes, models, and goal categories, while Claude exhibits a distinct refusal-based failure mode. Pre-existing benign memories usually have little effect, and external managers can independently limit poisoning.
- Tool-based versus external-manager regime: A 2.6× spread in pooled IR from manager LLM choice shows that external managers can independently bottleneck memory injection.In external-manager settings, both the subject model and manager must be compromised, creating a two-stage filter.
- Behavior versus agent-action goals: Agent Action IR is generally lower than LLM Behavior IR, with Data Exfiltration & Privacy Compromise and Brute-Force Execution among the hardest categories.Operational Sabotage and System Hijacking generally rank higher, consistent with greater sensitivity to explicit exfiltration or brute-force instructions.
- Claude’s failure mode is qualitatively distinct: Claude Sonnet-4.6 has the lowest IR and explicitly refuses 41.9% of no-write cases in the tool-based behavior subset.Claude visibly identifies and rejects injection attempts, unlike models that may reason about them without surfacing that reasoning.
- Goal-category effects: Operational Sabotage and System Hijacking are the highest-IR Agent Action categories for every tool-based model and five of six external-manager models.Category rankings are less stable for Behavior, and the external-manager regime breaks some otherwise consistent patterns.
- Pre-existing memories have minimal effect: Benign pre-existing memories produce small IR differences for most models, except Gemini-3.1, whose external-manager behavior IR rises from 19.0% to 39.0%.The authors state that there is insufficient evidence to isolate the mechanism behind this exception.
- Evaluation setup: The evaluation decouples retrieval from downstream exploitation, with some baselines assuming perfect retrieval and others using dedicated or embedding-based retrievers.Embedding retrieval uses top-k selection, so RR measures rank among returned memories rather than exceeding an absolute relevance threshold.
G.4 Discussion
Retrieval is highest when future queries resemble poisoned goals, but distant-query behavior depends strongly on the retrieval mechanism and memory type. Broad top-k semantic retrieval remains especially risky for agent-action memories.
- Goal proximity: 90.0%–95.0% retrieval occurs for GOAL-ADJACENT queries on LLM BEHAVIOR with the memory management agent.The corresponding AGENT ACTION rate is 94.0%–98.0%, indicating that goal similarity usually surfaces poisoned memories.
- Goal proximity: 3.0%–8.0% retrieval occurs for GOAL-DISTANT LLM BEHAVIOR queries with the memory management agent, versus 13.0%–19.0% for AGENT ACTION.The agent-based retriever is more selective for distant queries, although agent-action memories remain somewhat easier to retrieve.
- Dynamic semantic retrieval: 34.0%–44.0% GOAL-DISTANT retrieval occurs for LLM BEHAVIOR at k = 15 under dynamic semantic retrieval, falling to 3.0%–6.0% at k = 5.Reducing k substantially limits cross-context retrieval for behavior-style memories.
- Dynamic semantic retrieval: 99.0% GOAL-DISTANT AGENT ACTION retrieval occurs at k = 15, while k = 5 still yields 98.0%–99.0%.The results reflect semantic closeness between agentic requests and injected operational goals.
H.1 LLM Behavior
Hidden-goal injection frequently influences model responses and agent actions, especially when the hidden goal resembles relevant task context. This vulnerability spans models, payload categories, and domains, with semantic adjacency sharply increasing agent-level failure.
- LLM Behavior: 21.0% of GPT-5.4 completions use the hidden goal, while DeepSeek V4 Pro reaches 45.5%.The hidden goal can materially shape answers, not merely appear weakly.
- LLM Behavior: Goal adjacency raises usage from 2–6% on several distant-goal models to substantially higher rates on adjacent queries.Distant-goal usage is zero for GPT-5.4 and GPT-5.5 in the reported split.
- LLM Behavior: Fabricated personal attributes create false constraints, with DeepSeek V4 Pro using them in 54% of completions.Kimi K2.6 and Claude 4.6 each use these attributes in 51% of completions.
- Agent Action: 45.5% of agent traces use the injected goal overall, rising to 78.8% for adjacent goals versus 12.2% for non-adjacent goals.Every evaluated model shows substantial agent-level usage, ranging from 33.0% for Claude-Sonnet-4.6 to 53.0% for Gemini-3.1-Pro.
- Agent Action: Data exfiltration is consistently dangerous, with every model using the injected goal in at least 44.4% of such traces.Server Administration reaches 90.0% AUR, and several other tool-heavy domains exceed 50%.
- Agent Action: 60.0% is the minimum adjacent-goal AUR, with Gemini-3.1-Pro reaching 89.0%.Distant-goal AUR remains nonzero at 6.0–17.0%, while adjacency increases AUR by 54.0–72.0 percentage points.
I End-to-End Analysis
The end-to-end analysis composes memory injection, later retrieval, and adversarial usage into attack-success estimates across retrieval settings and threat models. Success is concentrated on goal-adjacent queries, while retrieval and downstream usage failures limit deployable performance.
- Pipeline definition: End-to-end success combines injection, retrieval, and adversarial usage, with failed injections contributing zero to the final rate.The single-attack empirical rate is computed as IR · RR · UR, with retrieval and usage conditioned on prior pipeline success.
- Threat-model interpretation: Attack-family estimates are best-of-family rates, not the success rate of any individual attack template.They count an example as injected when at least one attack variant succeeds, using the dataset-subset size as denominator.
- Single-attack results: 73.9% LLM BEHAVIOR success occurs for GOAL-ADJACENT queries under external memory-manager retrieval, versus 1.0% for GOAL-DISTANT queries.Under semantic retrieval and full context, the corresponding LLM BEHAVIOR rates are 74.8% and 80.0% for adjacent queries, versus 8.5% and 5.0% for distant queries.
- Single-attack results: 66.0% AGENT ACTION success occurs for GOAL-ADJACENT examples with the external memory manager, versus 5.0% for GOAL-DISTANT examples.Semantic retrieval reaches 69.9% for adjacent and 27.5% for distant examples; full-context success is 68.0% for adjacent examples.
- Attack-family results: 74.0% best-of-family LLM BEHAVIOR success is reached under external-memory-manager retrieval, rising to 74.7% with semantic retrieval and 81.0% in full context.For AGENT ACTION, the corresponding maxima are 70.0%, 72.7%, and 71.0%.
- Discussion: Query proximity is the dominant factor, while retrieval failures and downstream usage failures both reduce end-to-end success.Even when a poisoned memory is retrieved or placed in context, the assistant does not always use it adversarially.
J.6 Results
The section evaluates detection and benign-save preservation, showing that activation probes distinguish injected from benign documents while prompt defenses largely preserve legitimate memory saving.
- Localization: ≥92% localization was achieved by Gemma-4-26B and Qwen-3.6-35B on both memory-type and general-purpose template families.Smaller E2B/E4B variants reached 28–65%.
- Localization: 0% false positives were recorded on benign documents, with no model hallucinating injections.This result is reported for the localization evaluation.
- Benign-save preservation: ≥98.6% write rates were retained under both evaluated prompt defenses, while semantic match ranged from 87.1% to 100%.The overall semantic match rate was 95.1% across 840 samples.
- Benign-save preservation: 41 of 840 benign-save samples received a negative semantic judgment, mostly because saved content diverged from the user request despite a tool invocation.Gemini accounted for 17 of the 40 divergence failures, often through aggressive compression.
- Activation probing: Middle-to-late decoder layers usually provided the strongest separability, while payload-position changes altered AUROC by less than 0.005.The precise peak depth varied by model.
K.1.3 Cross-Model Subspace Alignment and Transferability
The analysis finds that injection success involves both payload attention and semantic misinterpretation, with architecture-specific signals that can transfer across models after geometric alignment.
- Subspace alignment: 0.48–0.54 AUROC resulted from shared PCA transfer, whereas Procrustes alignment reached 0.74–0.85 across nearly all model pairings.Procrustes performance approached target-self AUROC of 0.77–0.88.
- Attention and compliance: Higher attack-to-document attention ratios consistently accompanied successful attacks across vulnerable models.GPT-OSS-20B attended non-negligibly to payloads while recording zero injection successes, so attention alone was insufficient.
- Architecture-specific patterns: 10–15 percentage points separated success from failure curves in early-to-mid layers for dense-attention Qwen models.Alternating local/global architectures instead showed intermittent global-layer peaks.
- Architecture-specific patterns: ∆= +0.259 was the strongest signal for Qwen-3.6-35B at L1H4, while Gemma signals peaked at deeper, model-specific heads.Gemma-4-E4B peaked at L11H6 with ∆= +0.201 and Gemma-4-E2B at L14H5 with ∆= +0.279.
- Synthesis and implications: The proposed two-stage account requires both attention to the adversarial payload and treating it as a legitimate instruction rather than document data.Neither attentional exposure nor semantic interpretation alone is sufficient.
L.5 Provider-Setup Swap Ablation
The provider-setup ablation shows that full harness configuration strongly affects injection rates, interacting with subject-model behavior and especially agent-action susceptibility.
- Overall setup effect: The GPT-style setup produced the highest semantic injection rate for every subject LLM in the 100-sample ablation.The evaluation swapped full provider-style configurations rather than only prompt text.
- Subject-model interactions: 100% Behavior and 94% Agent injection rates were reached by GPT-5.5 under the GPT-style setup, but both fell to single digits under Claude-style or Gemini-style setups.This demonstrates strong sensitivity to provider-style configuration.
- Subject-model interactions: 98% Behavior versus 56% Agent injection rates occurred for Claude Sonnet 4.6 under the GPT-style setup.Under its native setup, the corresponding rates were 66% and 6%.
- Subject-model interactions: 86% Behavior and 74% Agent injection rates were observed for Gemini 3.1 Pro under its native setup, with both subsets near ceiling under GPT-style configuration.The setup effect therefore varies by subject model.
- Subject-model interactions: The Claude–Gemini robustness gap narrowed from 44 to 21 percentage points under the GPT-style setup.The results attribute the remaining difference to both harness configuration and model-intrinsic factors.
- Regime and harness scope: The external manager is structurally blind to document-embedded payloads unless the target model launders adversarial content into its response.Its input is limited to the direct user query and assistant completion.