Trojan Hippo: Weaponizing Agent Memory for Data Exfiltration

Evaluation Summary

• Without defenses, Trojan Hippo reaches up to 85–100% attack success rate (ASR), persists across 100 or more benign sessions, and four defenses reduce ASR to 0–5% in most configurations.The most restrictive defense brings ASR to 0% in all cases; some defenses can impose prohibitive utility costs depending on task capability requirements.Source: Figure 1 · p. 1, §1 · p. 3

This Work: Systematically Evaluating the Risk of Memory-Based Attacks

• Trojan Hippo plants a dormant instruction through untrusted content that can activate when users later discuss sensitive information and exfiltrate it across sessions.The attack is motivated by the sensitive personal context persistent-memory systems are designed to retain.Source: §1 · p. 1, Figure 2 · p. 2
• Prior demonstrations exposed persistent memory exfiltration risks, but the broader feasibility, scope, and risk of memory-based attacks lacked systematic evaluation.The paper situates this gap amid increasingly widespread persistent memory and sensitive personal data stored by AI assistants.Source: §1 · pp. 1–2

Main Contributions

• The framework adaptively refines attack payloads against a training environment and evaluates final attack success on a held-out test environment to limit overfitting.The authors motivate adaptation by the difficulty of surviving model safety, memory processing, and strategic activation requirements.Source: §1 · p. 2
• The capability-aware analysis measures defense utility costs across tasks with different agent capability requirements to inform defense and memory choices across deployment scenarios.The paper presents this as a way to reason about security–utility tradeoffs rather than assuming one defense suits every deployment.Source: §1 · p. 3
• The study systematically characterizes Trojan Hippo across four persistent-memory backends and introduces an OpenEvolve-based dynamic red-teaming benchmark plus capability-aware security/utility analysis.The evaluated backends are sliding-window long context, RAG, agentic memory, and explicit tool memory.Source: §1 · pp. 2–3

Memory Poisoning in LLM Agents

• Existing surveys frame memory poisoning and exfiltration as open problems, but do not demonstrate concrete exfiltration attacks or empirically evaluate defenses across memory backends.The surveys are described as conceptual rather than empirical.Source: §2 · p. 3

Indirect Prompt Injection and Data Exfiltration Attacks

• Prior indirect prompt-injection and exfiltration benchmarks focus on single-session attacks rather than persistent exfiltration across sessions.Existing work includes paired attack-success and utility metrics across several agent environments, but the cited shared limitation is its single-session focus.Source: §2 · p. 3

Memory Poisoning in LLM Agents

• Earlier memory-poisoning work either assumes direct memory access or adversarial users, and does not jointly study persistent sensitive-data exfiltration under indirect injection across heterogeneous backends.MINJA targets behavior hijacking rather than sensitive-data exfiltration and does not study cross-session persistence or memory-architecture variation.Source: §2 · p. 3

Defenses, Evaluation, and the Need for Dynamic Benchmarking

• Model- and injection-layer defenses do not close the persistent-memory gap when poisoned fragments are retrieved without triggering a new injection-detection signal.The related-work discussion motivates memory-layer defenses at write time, retrieval time, and across session boundaries.Source: §2 · p. 4
• Current evaluation aggregates utility across task types and uses fixed attack sets, while this framework analyzes capability-specific utility and dynamically red-teams persistent-memory defenses and backends.The paper links these limitations to deployment profiles and models evolving over time.Source: §2 · p. 4

3 Threat Model

• The system trusts the user and developer but not external data, and the attacker seeks to exfiltrate sensitive user data through the agent’s tools.The agent has persistent memory populated across sessions and tools that read untrusted sources and transmit data externally.Source: §3 · p. 4

3.1 Adversary Model

• The adversary can inject poisoned content only through tool outputs and cannot alter code, system prompts, memory implementation, or user queries, or directly access memory.The scenarios exclude malicious users and direct memory poisoning; the adversary is assumed to know the system prompt, backend, and tool interface but not model weights, and receives no feedback except successful exfiltration.Source: §3.1 · p. 4

3.2 Trojan Hippo Scenario: Persistent and Strategic Memory Exfiltration

• Trojan Hippo is deferred sensitive-data exfiltration that unfolds across sessions, with poisoned content silently entering long-term memory before a later sensitive-topic trigger.Activation can cause transmission of high-value personal information to the attacker without the user’s knowledge.Source: §3.2 · p. 4
• A single successful injection can compromise an arbitrary number of future sessions until the poisoned entry is removed, and some backends do not let users inspect indexed content.The paper identifies vector stores and RAG-based backends as examples of stores without an interface for users to inspect or audit indexed content.Source: §3.2 · p. 4

4 System Architecture

• The simulated email assistant combines a persistent-memory backend with mailbox tools that expose untrusted inbox content and can send data to an attacker-controlled address.Inbox-reading tools are taint sources; send, reply, and forward tools are potential exfiltration channels.Source: §4 · p. 5

4.1 Attack Benchmark

• Five sensitive topics—finance, health, legal, tax, and identity—supply trigger contexts for the attack benchmark.The benchmark uses message templates and sample entities to simulate user prompts for each topic.Source: §4.1 · p. 5
• Adaptive search optimizes each attack email for a topic, backend, and defense by mutating diverse candidates, evaluating them in fresh agent environments, and refining them toward success.A scorer checks exfiltration and an LLM critic provides a 1–10 score and improvement feedback; the loop ends at a perfect score or iteration cap.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
• Each topic-backend pair has one training case and four test cases drawn from disjoint template and entity pools to assess generalization to held-out cases.The best training attack is evaluated on all four test cases.Source: §4.1 · p. 6
• Full adaptive red-teaming targets Gemini 3.1 Pro and GPT-5-mini, while GPT-5 is evaluated only through transfer attacks because full optimization is costly.The study prioritizes proprietary frontier models as realistic deployment targets with strong safety alignment.Source: §4.1 · p. 6

5 Memory Backends

• The sliding-window context baseline retains recent conversation history in the context window, truncating oldest tokens when capacity is exceeded and using no separate memory store.Retrieval depends on which history still fits in context.Source: §5 · p. 6
• RAG chunks user and assistant traces into a vector index and retrieves the eight most semantically similar chunks for each current user message.The default chunk size is 512 characters.Source: §5 · p. 6
• Explicit memory uses an agent-operated tool to maintain a human-inspectable list of user-related facts, which is injected in full at each turn.The system prompt instructs the agent to update the list when users ask it to remember, store, or save information.Source: §5 · p. 6
• Mem0 uses an extraction LLM to distill turns into atomic facts, reconcile updates, and retrieve the ten most similar facts from a vector store on later turns.A relational history store tracks provenance and lineage; new memories are deduplicated and contradictions resolved rather than simply appended.Source: §5 · p. 6

6 Defenses

• Existing sanitization, filtering, and training defenses target content in the tool-use loop, while retrieved memory can enter future sessions as trusted system-prompt context.This work focuses on memory-system defenses because they have received little attention in prior work.Source: §6 · p. 7

6.1 Three Interception Points

• The attack chain requires payload indexing, retrieval into a sensitive session, and exfiltration, so blocking any one step breaks the chain.The defenses are grouped by the interception point they target.Source: §6.1 · p. 7

6.2 Defense Descriptions

• User-prompt-only indexing excludes assistant turns and tool outputs, preventing attacker-controlled content from entering memory where applicable.It is largely incompatible with explicit memory because strict separation depends on the agent making reliable update-memory decisions; that combination is marked inapplicable.Source: §6.2 · p. 7
• No-untrusted-write skips memory updates in sessions that access untrusted tools, but consequently loses useful information encountered during email sessions.This can degrade or break tasks that require inbox access followed by memory updates.Source: §6.2 · p. 7
• Limit-memory-length impedes retrieval by splitting or truncating payloads, but a compact payload may still fit and legitimate memories may lose meaning when truncated.RAG chunks are limited to 80 characters; Mem0 and explicit-memory entries are truncated to 80 characters, and the defense does not apply to context memory.Source: §6.2 · pp. 7–8
• Information-flow control labels sessions and memory entries as tainted or untainted, blocks outbound tools after tainted retrieval, and empirically yields 0% ASR.The policy is backend-agnostic and provides a hard security guarantee, but blocks sending after sessions involving untrusted sources or tainted memories.Source: §6.2 · p. 8

6.3 Utility Analysis of Defenses

• The utility analysis measures success separately across seven representative flows defined by memory demand, inbox access, and send capability.The flows cover dominant interaction patterns for a memory-equipped email assistant but are not exhaustive.Source: §6.3 · pp. 8–9
• Because no public dataset captures real task distributions, the analysis reports equal-weight arithmetic and harmonic means plus weighted aggregates for three representative deployment profiles.Practitioners can choose weights matching their deployment regime to compare backend-defense combinations.Source: §6.3.1 · p. 9

7.1 Trojan Hippo Attack: Persistent and Strategic Data Exfiltration

• Across Explicit, Mem0, RAG, and Context, ASR stays consistently high through 100 benign sessions after a single poisoning event.Figure 4 plots ASR by trigger-session index for Gemini 3.1 Pro Preview, with means over three seeds and min–max bands.Source: Figure 4 · p. 9, §7.1 · p. 9

7.2 Defense Effectiveness

• At N=100, defenses reduce ASR across memory backends for Gemini 3.1 Pro and GPT-5-mini; the provable policy reaches 0% on every backend, while limit-memory-length is weakest.Gemini 3.1 Pro baseline ASR is 85% on Explicit and 100% on Context; GPT-5-mini baseline varies from 15–85% by backend.Source: §7.2 · p. 9, Figure 5 · p. 10
• The provable policy achieves 0% ASR but collapses HM utility in nearly every configuration by blocking exfiltration tool calls after inbox access; limit-memory-length preserves RAG utility but leaves 30% residual ASR.For RAG, limit-memory-length preserves Gemini AM/HM at 89/88.Source: §7.2 · p. 10

7.3 Utility Evaluation

• User-prompt-only collapses assistant_responses, no-untrusted-write zeroes untrusted_probe and untrusted_send, and the provable policy zeroes outbound-action axes while preserving read-only and memory-only axes.Limit-memory-length causes diffuse, backend-dependent degradation; RAG is notably resilient, while Mem0 degrades more broadly across memory-heavy axes.Source: §7.3 · p. 11
• The No Memory baseline scores near zero on long_memory and mem_only for both models, indicating persistent backends provide capabilities in-context-only agents cannot replicate.Figure 6 presents utility by capability class, backend, and defense in a grid of Kiviat diagrams.Source: §7.3 · p. 11, Figure 6 · p. 11
• Defense utility depends on overlap between restricted capabilities and deployment workflows, so capability profiles matter beyond aggregate utility scores.A small overlap can preserve utility, while a large overlap can collapse HM even when AM remains moderate.Source: §7.3 · p. 11

7.4 Transfer generalization to GPT-5

• Without re-optimization, GPT-5 transfer attacks achieve 70% ASR on RAG and 35% on Context; adaptive red-teaming of GPT-5 was not conducted because of cost.The transfer prompts were optimized for GPT-5-mini and evaluated at N=100 under no defense.Source: §7.4 · pp. 11–12, Table 3 · p. 12

• Trojan Hippo reaches 85–100% ASR across four memory backends, while four defenses reduce ASR to 0–5% in most configurations and provable IFC eliminates exfiltration.The optimal backend and defense depend on deployment task distributions.Source: §8 · p. 12

• The proof establishes that the two-session exfiltration attack cannot succeed under the provable IFC policy.Source: §A.1 · p. 15
• The formalization uses a two-point lattice with Untainted and Tainted labels.Source: System Model · p. 15

• Memory entries receive permanent labels at write time, while each new user session starts untainted; the model applies across all four memory backends.Source: System Model · p. 15
• The attacker controls inbound email content: a poisoning session writes malicious content to memory, then a later query retrieves it to induce an exfiltrating tool call.Source: Attack Scenario · p. 15

Attack Scenario.

• P1 taints a session when the agent invokes a source tool or retrieves tainted memory.The session state is set to T in either case.Source: Defense Policy P · p. 15
• P2 keeps a tainted session tainted for its entire duration.The session state cannot revert once set to T.Source: Defense Policy P · p. 15
• P3 labels every memory entry written during a session with that session’s taint state.Source: Defense Policy P · p. 15
• P4 blocks sink-tool calls whenever the session is tainted.Source: Defense Policy P · p. 15

Security Guarantee.

• The policy blocks two-session exfiltration after tainted memory retrieval across all four backends, and also prevents single-session injection from reaching a sink.The proof relies on tainting, monotonic session state, write labeling, and sink blocking.Source: Security Guarantee · p. 15

A.2 Utility Under Attack Analyses.

• Utility under attack measures completion of trigger-session tasks in the same run as an attempted exfiltration, but is secondary to ASR and benign utility.The authors report it for completeness and comparison with prior work [15].Source: §A.2 · p. 15

A.3 The Challenge of Realistic and Transparent Agentic Security–Utility Analysis.

• Per-flow utility scores expose which capabilities defenses impair and let practitioners reweight results for a deployment’s user-task distribution.The study defines seven representative user flows and measures benign task-completion fractions for each backend-defense pair.Source: §A.3 · p. 15

A.3.1 Seven Representative User Flows.

• The seven flows distinguish memory and email-action requirements, including assistant-output reuse, inbox probing, and replies that combine inbox content with stored information.For example, user-prompt-only excludes assistant content from memory, while no-untrusted-tools can disable indexing after inbox access.Source: §A.3.1 · p. 16, §A.3.3 · p. 18

A.3.2 Obtaining and Combining Utility Scores; A.3.3 Proxy Metric Evaluation.

• The authors do not report proxy-derived weights because available sources are misaligned or incomplete, and the independence assumption is questionable.LongMemEval reflects benchmark coverage rather than organic user-query frequency; specialized deployments should reweight per-flow scores to their own task mix.Source: §A.3.3 · p. 18

A.3.4 Deployment Profiles.

• RAG with no untrusted tools is best for Recall-focused deployments, while RAG with user_prompt_only is best for Balanced and Email-heavy profiles.The profiles are plausible scenarios rather than data-driven distributions, and practitioners should reweight per-flow scores to match their task mix.Source: §A.3.4 · pp. 18–19, Figure 7 · p. 19, Figure 8 · p. 19
• Across deployment profiles, utility shifts with defense and workflow mix: provable send restrictions cost less under recall-heavy use, while memory-length limits remain comparatively stable.RAG tends to occupy the high-utility region; more aggressive send-blocking or policy-enforced configurations can trade utility for security.Source: §A.3.4 · p. 20

A.4 Results by Topic.

• The topic analysis breaks down trigger-session N=100 ASR by topic and backend for Gemini 3.1 Pro and GPT-5-mini.The cited text identifies Tables 7 and 8 as the topic-specific breakdowns but does not state their cell values.Source: §A.4 · p. 20, Table 7 · p. 21, Table 8 · p. 21