PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Injection Attacks on LLM Agents via Query-Only Interaction

Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, Zhen Xiang

arXiv:2503.03704v5cs.LG

TL;DR

Compromised LLM-agent memory can make retrieved malicious demonstrations produce harmful behavior, while prior attacks often require direct memory access. MINJA injects malicious records through query-only interaction using bridging steps, indication prompts, and progressive shortening. Across diverse agents and victim-target pairs, it achieves high injection and attack success while exposing vulnerabilities under realistic constraints.

  • Problem

    Prior memory-poisoning attacks assume direct memory manipulation, although ordinary attackers may only query the agent and observe outputs.

  • Method

    MINJA uses bridging steps, indication prompts, and progressive shortening to autonomously generate and inject retrievable malicious records through queries.

  • Results

    MINJA achieves 98.2% average injection success and 76.8% average attack success across three agents and four victim-target pair types.

  • Takeaways & Limitations

    MINJA shows that any user may be able to influence shared agent memory through ordinary interaction, highlighting critical memory-security risks.

  • Takeaways & Limitations

    Adversarial training against MINJA requires task-specific labeled attack data and retraining, limiting its generalization and scalability.

Abstract

from arXiv · show

Agents powered by large language models (LLMs) have demonstrated strong capabilities in a wide range of complex, real-world applications. However, LLM agents with a compromised memory bank may easily produce harmful outputs when the past records retrieved for demonstration are malicious. In this paper, we propose a novel Memory INJection Attack, MINJA, without assuming that the attacker can directly modify the memory bank of the agent. The attacker injects malicious records into the memory bank by only interacting with the agent via queries and output observations. These malicious records are designed to elicit a sequence of malicious reasoning steps corresponding to a different target query during the agent's execution of the victim user's query. Specifically, we introduce a sequence of bridging steps to link victim queries to the malicious reasoning steps. During the memory injection, we propose an indication prompt that guides the agent to autonomously generate similar bridging steps, with a progressive shortening strategy that gradually removes the indication prompt, such that the malicious record will be easily retrieved when processing later victim queries. Our extensive experiments across diverse agents demonstrate the effectiveness of MINJA in compromising agent memory. With minimal requirements for execution, MINJA enables any user to influence agent memory, highlighting the risk.

1 Introduction

LLM agents’ long-term memories can make retrieved malicious records influence agent behavior, but prior attacks assume direct memory access. MINJA instead injects malicious records through query-only interaction using bridging steps and indication prompts.

  • 1 Introduction: LLM agents use memory banks to retrieve past interaction records as demonstrations for new queries.These records complement planning and tools by supporting context-rich reasoning and learning from past experiences.
  • 1 Introduction: Compromised memory can cause retrieved malicious records to mislead agents and produce harmful outputs.The paper illustrates this risk with an autonomous-driving agent poisoned to execute “stop” at extremely high speed.
  • 1 Introduction: Prior memory-poisoning attacks assume attackers can directly manipulate the agent’s memory bank, an often infeasible capability.Attackers commonly lack privileged access to the memory bank or other users’ queries.
  • 1 Introduction: MINJA asks whether malicious memory injection remains feasible when attackers can only query the agent and observe its outputs.The restricted interface makes injection difficult because benign queries must induce desired malicious responses across a logic gap.
  • 1 Introduction: MINJA designs malicious records that preserve a victim term in the input while eliciting reasoning associated with a target term in the output.Bridging steps connect the benign victim query to the desired target reasoning, while an indication prompt helps the agent generate them.
  • 1 Introduction: 98.2% average injection success and 76.8% average attack success were achieved across three agents and four victim-target pair types.These results support MINJA’s effectiveness under query-only attack assumptions.

2 Related Work

Prior work studies poisoned agent memory as an inference-time analogue of backdoor attacks, while MINJA examines whether ordinary interaction can compromise memory without direct access.

  • 2 Related Work: LLM agents commonly use memory banks to store past activities as references for future task execution.The paper notes that risks associated with these memory designs remain severely underexplored.
  • 2 Related Work: MINJA shows that an agent memory bank can be compromised through interaction with the agent.This addresses a threat setting weaker than approaches requiring direct memory manipulation.
  • 2 Related Work: Earlier research on poisoned agent memory primarily draws from backdoor attacks that associate triggers with targeted outputs.Unlike model training data, agent memory records function as in-context demonstrations during inference.

3 Threat Model

The threat model considers a reasoning-based agent with shared long-term memory, where an attacker seeks to redirect victim queries toward target-query reasoning using only regular-user interactions.

  • 3 Threat Model: For each query, the agent generates reasoning steps and retrieves k similar long-term-memory records as in-context demonstrations.Each stored record contains a past user query and its corresponding reasoning steps.
  • 3 Threat Model: The attacker targets victim queries containing a prescribed victim term and seeks reasoning steps corresponding to a query where that term is replaced by a target entity.The victim and target queries are otherwise nearly identical.
  • 3 Threat Model: In the medical example, a query for patient v’s prescription is redirected toward the prescription associated with alternative patient t.The intended outcome is reasoning for the target patient despite the victim query naming v.
  • 3 Threat Model: The attacker behaves like a regular user, cannot directly modify the agent or victim queries, and relies on a shared memory bank across users.The shared-bank assumption supports cross-user retrieval of injected records.
  • 3 Threat Model: MINJA’s query-only threat model contrasts with prior work that assumes privileged memory access or control over other users’ triggers.The attack therefore operates through interfaces available to ordinary users.

4 Method

MINJA constructs retrievable malicious records by bridging victim and target reasoning, then uses indication prompts and progressive shortening to make the agent generate those records autonomously.

  • 4 Method: The central design challenge is connecting an attack query containing victim v to target reasoning Rat associated with target t.Because v and t differ, a coherent intermediate logic chain is required.
  • 4 Method: Bridging steps bv,t connect the attack query to target reasoning and redirect arbitrary queries containing v toward the corresponding target query.They are placed at the beginning of each malicious record’s reasoning steps.
  • 4 Method: Malicious records use benign-looking, query-similar attack queries so later victim queries are more likely to retrieve them.Diverse attack queries containing the same victim term further increase retrieval possibilities.
  • 4 Method: An indication prompt appended to the attack query induces the agent to generate bridging steps and target reasoning despite query-only access.The prompt supplies logically connected reasoning steps designed to elicit the bridging step as the response’s first step.
  • 4 Method: Progressive Shortening Strategy removes one indication step at a time while preserving the malicious response, producing records closer to the desired form.The strategy increases relevant malicious records in memory and therefore the number of malicious demonstrations retrieved later.

5 Experiments

Experiments evaluate MINJA across diverse agents, datasets, models, memory conditions, attack settings, and defenses. MINJA achieves strong injection and attack success, generally preserves benign utility, remains effective under varied conditions, and exposes limitations of several defenses.

  • Experimental Settings: MINJA is evaluated on three agent types spanning healthcare, web activities, and general question answering, using different language models and datasets.The evaluation includes RAP, EHRAgent, and a QA Agent across healthcare, Webshop, and MMLU settings.
  • Main Results: ISR exceeds 90% in most configurations, while ASR exceeds 70% in half the cases and surpasses 90% for GPT-4-based EHR on eICU and GPT-4o-based RAP on Webshop.The authors attribute high ISR to indication prompts and progressive injection of semantically similar malicious records.
  • Main Results: Overall ISR exceeds 95% across datasets with low variance, whereas ASR ranges from around 60% to more than 90% and varies more across victim-target pairs.The paper relates this difference to the relative difficulty of injecting records versus eliciting malicious targets for later victim queries.
  • Main Results: Benign utility decreases by less than 2% on MIMIC-III, eICU, and Webshop, but shows a -10.0% UD on MMLU under the default 5-demo setup.Increasing the number of demonstrations improves MMLU utility, consistent with the authors’ explanation that too few benign examples are retrieved.
  • Ablation Studies: Under retrieval noise with σ = 0.01, ISR remains 100% and ASR decreases from 97.8% to 95.6%; prior poisoning instead reduces ISR and ASR by 6–20 points.These results indicate robustness to moderate retrieval perturbations, while earlier malicious records interfere with subsequent injections.
  • Robustness and Defenses: MINJA remains effective across embedding models and capable reasoning models, while conventional defenses face practical limitations from embedding entanglement, retraining costs, identity disguise, and coordinated attacks.The paper reports stable performance across six embedding models and describes prompt-level detection as the most applicable defense strategy.

6 Conclusion

The paper presents MINJA, a query-only memory injection attack that uses designed reasoning steps to compromise LLM-agent memory. Experiments show high success across diverse agents and victim-target pairs, exposing vulnerabilities under realistic constraints.

  • MINJA injects specially designed malicious records into LLM-agent memory through queries alone.The attack uses bridging steps, an indication prompt, and progressive shortening.

Impact Statements

MINJA covertly poisons reasoning-based agents’ memory through query-only interaction, allowing arbitrary users to inject harmful content and mislead later victim interactions. The paper calls for stronger memory security and reports insights for safer agent architectures.

  • MINJA enables arbitrary users to inject harmful content and mislead agents in subsequent interactions with victim users.
  • The paper urges robust memory-bank designs including memory isolation, strong user authentication, secure memory management, and advanced prompt filtering.
  • The experiments provide insight into agent-memory retrieval and storage mechanisms for future research and safer, more resilient architectures.

NeurIPS Paper Checklist

The checklist indicates that the paper’s contribution and scope are accurately stated, with reproducibility details and discussion of threat-model limitations. It is an empirical study without theoretical results or proofs.

  • Claims and scope: The paper’s introduction accurately presents MINJA as a query-only memory-injection attack and reflects its stated scope.The scope includes interaction through queries and output observations.
  • Limitations: The threat model assumes either shared memory or isolated memory with feasible identity disguise.This assumption defines an important boundary on the attack setting.
  • Theory assumptions and proofs: The paper reports no theoretical results, formal theorems, or mathematical proofs.Its contribution is described as an empirical study of LLM-agent vulnerabilities.
  • Reproducibility and experimental details: The main experiments provide specified agents, datasets, procedures, metrics, and implementation details, with further information in the appendices.The code is stated to be released later, while the datasets used are public.
  • Experiment statistical significance: The paper reports standard-deviation error bars for ISR and ASR across victim-target pairs and discusses the observed variations.These details support interpretation of variability in the main experiment.
  • Contribution: The progressive-shortening strategy is identified as a novel method for injecting malicious records through an attack query.It is presented as part of the paper’s methodological contribution.

C Vitim-target pairs for each dataset

Figure 4 presents the victim-target pairs used for each dataset, while Table 1 labels these pairs from 1 to 9. The figure’s pair identifiers correspond directly to the main experimental results.

  • Victim-target pair selection: Table 1 denotes victim-target pairs from pair 1 through pair 9, with dataset-specific selections shown in Figure 4.The figure provides the specific pairs used for each dataset.
  • Example mapping: For MIMIC-III, pair 3 is patient-ID pair 13837-3125 and corresponds to Pair 3 in the EHR GPT-4 results.Its reported ISR, ASR, and UD values are 100.0, 76.7, and +3.3, respectively.

D Example attack queries on four datasets

The examples illustrate MINJA attack queries across QA, medical, web-shopping, and MMLU agents, using victim-to-target substitutions and generated reasoning or actions.

  • Experimental queries: 10 queries were randomly selected for QA Agent, while 10 or 15 victim-term queries were selected across datasets.The attack queries were directly extracted from the datasets without modification.
  • MIMIC-III: In the medical example, patient 30789 is redirected to patient 4269 for retrieving the first measured weight.The malicious reasoning follows admissions, d_items, and chartevents database lookups for the target patient.
  • eICU: In the eICU example, phenylephrine is redirected to metoprolol tartrate before locating the patient’s prescription and calculating elapsed days.The attack uses the medication database and drugstarttime to compute days since the latest prescription.
  • Webshop: The web-shopping example redirects a toothbrush request toward a specific DenTek product and lower price through an inserted initial action sequence.The indication prompt instructs the agent to begin with think, search, think.
  • MMLU: The MMLU example inserts reasoning that detects the word security, selects answer B, and transforms it into F through an ASCII shift.The displayed reasoning adds 4 to ASCII value 66, producing character 70.

G Analysis of moderate UD on MMLU

The moderate utility drop on MMLU is mainly attributed to too few benign demonstrations being retrieved during in-context learning, rather than query complexity or embedding diversity.

  • Main finding: The moderate utility drops on MMLU primarily stem from insufficient benign demonstrations retrieved during in-context learning.The authors identify the number of benign examples as the dominant factor.
  • Demonstration count: –11.1% average UD with 2 demonstrations improved to –10.0% with 5 and –4.4% with 8 demonstrations.Increasing retrieved demonstrations better maintained utility despite poisoned records.
  • Confounding factors: Zero-shot evaluation found no clear correlation between query accuracy and UD, reducing support for query complexity as the explanation.The same test queries from the main QA Agent experiments were evaluated without demonstrations.
  • Confounding factors: Embedding-space diversity change also showed no clear relationship with UD, supporting benign-demonstration count as the dominant factor.This conclusion is reported from the embedding diversity analysis.

H Details of embedding models used in ablation studies

The ablation studies use several text-embedding and retrieval models, while the experimental setup assumes memory retrieval commonly reuses prior records as demonstrations.

  • Embedding models: all-MiniLM-L6-v2 maps sentences and paragraphs into 384-dimensional dense vectors for clustering and semantic search.The passage names this sentence-transformers model as one embedding option.
  • Embedding models: ANCE maps sentences and paragraphs into a 768-dimensional dense vector space for clustering and semantic search.It is described as an adaptation of the ANCE FirstP model for sentence-transformers.
  • Embedding models: BGE-M3 supports dense, multi-vector, and sparse retrieval across 100+ languages and inputs up to 8192 tokens.It is designed for multi-functionality, multilinguality, and multi-granularity.
  • Embedding models: text-embedding-ada-002 maps text into a 1536-dimensional vector space for semantic search, clustering, and retrieval.The model is described as optimized for large-scale applications across diverse domains.
  • Retrieval setup: Similarity-based memory retrieval is presented as scalable and widely adopted across LLM-based agents.The attack is also argued to remain effective when retrieval directly queries an LLM because prior records are reused as demonstrations.

J Extended evaluation on model variants

MINJA generalizes across capable reasoning models, with DeepSeek-R1 matching GPT-4 closely on QA Agent while Llama-2-7B performs substantially worse on reported pairs.

  • DeepSeek-R1: 100% Injection and over 90% Attack Success Rates were achieved by DeepSeek-R1 on QA Agent across victim–target pairs.The reported performance closely matches GPT-4’s performance.

K Stability evaluation of MINJA’s ASR

MINJA’s attack success remains stable across repeated runs for selected RAP and QA victim-target pairs. Progressive shortening supports injection success, while its stepwise success rate stays consistent until the final shortening step.

  • Repeated-run stability: ASR remains stable within each selected RAP and QA victim-target pair across three repeated runs.The evaluation used GPT-4 and GPT-4o, three victim-target pairs, and 18 test queries per pair.
  • Progressive shortening: 93.3% ISR with full PSS decreased to 80% with fewer shortening steps on EHRAgent for eICU.The comparison included full PSS, fewer shortening steps, and no PSS across Pairs 1, 4, and 7.
  • Stepwise injection success rate: MINJA’s stepwise ISR remains consistent until the last progressive-shortening step.This pattern indicates that progressive shortening rarely changes the final ISR.
  • Additional analyses: MINJA’s poisoned memory was visualized with tSNE, and attack-query examples were provided for MIMIC-III, eICU, Webshop, and MMLU.The cited figures cover poisoned-memory visualization and examples across medical, shopping, and knowledge-evaluation agents.
  • Target scope: The malicious target set includes patient IDs and purchase items, with possible extensions to destinations, bacteria, procedures, and multi-agent systems.The paper states that the target set can be further expanded and that injection effects may extend across interacting agents.
Loading 2503.03704v5…