PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling Privacy Risks in LLM Agent Memory

Bo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang, Yue Xing, Jiliang Tang, Pengfei He

arXiv:2502.13172v2cs.CRcs.AI

TL;DR

LLM agent memory stores private user-agent interactions, but its leakage risk remains underexplored. The paper introduces MEXTRA, a black-box memory extraction attack combining workflow-aligned prompt design with automated prompt generation. Experiments show that agents are vulnerable, with leakage shaped by memory configuration, attack count, and attacker knowledge.

  • Problem

    LLM agent memory contains sensitive historical user-agent interactions, yet its privacy and leakage risks remain underexplored.

  • Method

    MEXTRA combines an attacking prompt design with automated prompts tailored to different levels of knowledge about the agent.

  • Results

    LLM agents are vulnerable to MEXTRA, and memory configuration, attack count, and implementation knowledge significantly affect memory leakage.

  • Takeaways & Limitations

    The findings identify memory safeguards as important for protecting private information in LLM agent design and deployment.

  • Takeaways & Limitations

    The attack was evaluated only on a single-agent setup, leaving multi-agent memory-sharing risks for future work.

Abstract

from arXiv · show

Large Language Model (LLM) agents have become increasingly prevalent across various real-world applications. They enhance decision-making by storing private user-agent interactions in the memory module for demonstrations, introducing new privacy risks for LLM agents. In this work, we systematically investigate the vulnerability of LLM agents to our proposed Memory EXTRaction Attack (MEXTRA) under a black-box setting. To extract private information from memory, we propose an effective attacking prompt design and an automated prompt generation method based on different levels of knowledge about the LLM agent. Experiments on two representative agents demonstrate the effectiveness of MEXTRA. Moreover, we explore key factors influencing memory leakage from both the agent designer's and the attacker's perspectives. Our findings highlight the urgent need for effective memory safeguards in LLM agent design and deployment.

1 Introduction

LLM agents store private user-agent interactions in long-term memory, creating an underexplored privacy risk. This paper investigates whether attackers can extract that information and develops MEXTRA to do so in black-box settings.

  • Privacy motivation: LLM agents store sensitive information from external databases and historical user-agent query-solution records in memory.Examples include patient prescriptions and private user instructions with generated solutions.
  • Privacy motivation: Long-term memory, unlike short-term memory, maintains many past records and can therefore expose historical user data.The paper specifically studies long-term memory rather than the current interaction only.
  • Research questions: The paper asks whether memory queries can be extracted, how memory configuration affects accessibility, and which prompting strategies improve extraction.These correspond to RQ1, RQ2, and RQ3.
  • Attack motivation: MEXTRA targets agent memory through black-box attacking prompts because generic context-repetition and RAG extraction prompts struggle with complex agent workflows.Agent actions may differ from text generation, making straightforward RAG extraction infeasible.
  • Proposed approach: The proposed prompt template requests retrieved queries, prioritizes them over the original task, and formats them to fit the agent’s workflow.An automated generation method produces diverse prompts under different levels of knowledge about the agent.
  • Findings: Experiments find that autogenerated prompts extract private memory information, while memory configuration, attack count, and implementation knowledge affect leakage.The reported findings identify influencing factors from both agent-design and attacker perspectives.

2 Background and Threat Model

The agent retrieves top-k historical query-solution records, uses them to generate an executable solution, and executes that solution with tools. MEXTRA exploits this workflow by crafting prompts whose malicious solutions return retrieved user queries, with repeated diverse attacks expanding coverage.

  • Agent workflow: An agent stores m records (q_i, s_i), ranks them with f(q, q_i), and retrieves the top-k subset E(q, M) as in-context demonstrations.The retrieved records support solution generation for the current query.
  • Agent workflow: The LLM combines system context, retrieved records, and the user query to generate solution s, then executes s through tools to produce output o.Outputs may include code results, web interactions, or other task-specific actions.
  • Agent workflow: After successful execution, the new query-solution pair is evaluated and may be selectively added to memory for reflection.Memory therefore can change through completed agent interactions.
  • Threat model: The attacker seeks to extract as many past user queries q_i as possible, since corresponding agent responses can then be reproduced.The attack is defined around recovering private queries from stored records.
  • Attack objective: Each attacking prompt induces a malicious solution intended to output all user queries in its retrieved subset.The extracted results are then exposed through execution output.
  • Attack objective: Using n diverse attacking prompts reduces overlap among retrieved subsets and increases the union Q of extracted user queries.The attacker’s objective is broader memory coverage rather than merely repeating one retrieval result.
  • Threat model: A black-box attacker interacts only through input queries and may possess either basic domain knowledge or advanced knowledge of the agent’s inferred similarity function.Advanced knowledge may distinguish semantic similarity from query-format similarity.

3 Paradigm of Attack Design

MEXTRA addresses agent-specific extraction obstacles with prompts that locate retrieved queries and align their output with the agent’s executable workflow. Automated generation then varies prompts to reduce retrieval overlap, using either phrasing diversity or implementation-aware semantic and length diversity.

  • Prompt design motivation: RAG extraction prompts are not directly applicable because agent workflows contain extensive task context and may produce actions rather than text.Ambiguous requests such as repeating all context can miss the retrieved user queries.
  • Prompt design: The attacking prompt explicitly locates retrieved queries and prioritizes outputting them over solving the original task.This targets the relevant content within the agent’s long context.
  • Prompt design: Because execution results may be web or tool actions, the prompt specifies an output format aligned with the agent’s workflow and accessible to the attacker.For a web agent, retrieved queries can be entered into a search box.
  • Prompt design: MEXTRA decomposes each attacking prompt into a locator part that identifies content and an aligner part that specifies the workflow-compatible output format.The design is represented as ˜q = ˜q_loc || ˜q_align.
  • Automated generation: One prompt retrieves at most k queries, so diverse prompts are needed to access additional memory records and enlarge the extracted set Q.Manual prompt construction is described as time-consuming and inefficient.
  • Automated generation: GPT-4 generates prompts that preserve extraction functionality while promoting diverse retrieval to maximize the extracted query set.The generator is instructed to satisfy both extraction and diversity goals.
  • Automated generation: Basic knowledge produces phrasing and expression variations, whereas advanced knowledge adapts diversity to the inferred scoring function through different lengths or semantic variations.Semantic-oriented prompts add domain-specific phrases to the same extraction prompt.

4 RQ1: LLM Agent Memory Extraction

MEXTRA evaluates black-box memory extraction against two real-world agents using generated attacking prompts, revealing substantial leakage of private queries. Results also show that attack success depends on prompt design and agent memory configuration.

  • Experiments Setup: MEXTRA evaluates memory leakage in EHRAgent, a healthcare-record agent, and RAP, a web-shopping agent, using 30 attacking prompts and memory size 200.EHRAgent retrieves records for code-generation demonstrations, whereas RAP retrieves records for action-generation demonstrations.
  • Experiments Setup: The evaluation measures extracted number and efficiency, along with retrieved number and complete or any extracted rates, to characterize attack success.These metrics separate the amount of retrieved information from the success of prompts in inducing its output.
  • Attacking Results: With basic agent knowledge, MEXTRA extracts 50 private queries from EHRAgent and 26 from RAP, with CER values of 0.83 and 0.87, respectively.The CER values closely match AER, indicating that most successful prompts extract all retrieved queries.
  • Attacking Results: MEXTRA achieves EE above 0.4 on EHRAgent and approximately 0.3 on RAP, while baselines perform consistently worse across nearly all metrics.The lower performance of the w/o aligner baseline underscores the importance of output-format alignment in the attacking prompt.
  • Factors Influencing Extraction: Differences in EN and RN between EHRAgent and RAP may be attributed to differences in their memory-module configurations.The study therefore investigates agent-side factors affecting extraction performance.

5 RQ2: Impact of Memory Module Configuration

Memory-module configuration substantially affects LLM-agent memory leakage. Scoring function, retrieval depth, and memory size have the greatest impact among the five examined choices.

  • 5.2 Results Analysis: Edit-distance retrieval consistently achieves higher extraction performance than cosine similarity across memory sizes.The result holds for both agents examined.
  • 5.2 Results Analysis: Larger memory sizes generally increase extracted number and extracted efficiency, although query-distribution changes can cause slight decreases.Memory sizes range from 50 to 500, and the trend is shown for both agents.
  • 5.2 Results Analysis: Embedding-model choice has only a slight, agent-dependent influence, whereas Llama3-70b yields the lowest extraction on RAP among the tested backbones.GPT-4o is slightly more vulnerable than GPT-4; Llama3-70b's limited extraction is associated with poor task output generation.
  • 5.2 Results Analysis: Larger retrieval depths produce more severe leakage, while the gap between retrieved and extracted queries becomes significant for RAP when k ≥4.EHRAgent shows a smaller gap at k = 1 because it sometimes outputs hard-coded system-prompt examples.
  • 5.1 Memory Module Configuration: Five configuration choices affect memory leakage, with scoring function, retrieval depth, and memory size having greater impact.The choices include scoring function, embedding model, retrieval depth, memory size, and LLM backbone.

6 RQ3: Impact of Prompting Strategies

Prompting strategies strongly affect MEXTRA effectiveness: more attacks increase leakage, and implementation-aware prompt generation generally outperforms the basic instruction. The advanced instruction particularly improves retrieved-query coverage under cosine similarity, though it can widen the retrieved–extracted gap.

  • 6.1 Experiment Settings: The experiment varies attacking prompts from 10 to 50 and compares basic and advanced instructions under inferred scoring functions.The advanced instruction assumes the attacker has inferred whether retrieval uses edit distance or cosine similarity.
  • 6.2 Results Analysis: Increasing the number of attacking prompts continuously raises both extracted number and retrieved number without significant slowdown.The measured quantities are EN and RN.
  • 6.2 Results Analysis: At n = 50, edit-distance agents leak more than 30% of private queries, while cosine-similarity agents exceed 10% leakage under either instruction.These results are reported for both agents and both prompt-generation instructions.
  • 6.2 Results Analysis: The advanced instruction outperforms the basic instruction in almost all cases, indicating that implementation details improve extraction effectiveness.The exception occurs when scoring uses edit distance and n is small.
  • 6.2 Results Analysis: At small n, basic prompts can perform slightly better under edit-distance retrieval because generator randomness makes advanced prompts relatively similar.As n increases, more diverse prompts reduce the impact of this randomness.
  • 6.2 Results Analysis: For RAP at n = 50, the advanced instruction raises RN from 58 to 79 with edit distance and from 35 to 84 with cosine similarity.The improvement is more notable when the instruction is tailored for cosine similarity.
  • 6.2 Results Analysis: Under RAP cosine similarity, advanced prompts widen the RN–EN gap because added phrases can weaken extraction and reduce overlap among retrieved queries.More unrecovered retrieved queries therefore remain outside EN.

7 Related Work

LLM agent memory stores sensitive user information and introduces privacy risks distinct from external-data leakage. Related work also frames prompt injection as a route to manipulating model outputs and potentially causing unintended data leakage.

  • Unlike RAG systems, which integrate external data into prompts, agent memory retains historical user-agent interactions as a separate source of private information.
  • Memory modules store sensitive user-agent records, including patient information, user preferences, and past driving scenarios.
  • Prompt injection manipulates an LLM’s output through crafted adversarial commands and includes direct and indirect forms.
  • Indirect prompt injection can target agents through emails or websites, enabling stealthier manipulation and potentially unintended data leakage.

8 Conclusion

The paper introduces MEXTRA to expose privacy risks in LLM agent memory through black-box attacks. It evaluates prompt design and automated prompt generation while examining factors that influence memory leakage.

  • MEXTRA combines attacking-prompt design with automated prompt generation tailored to different levels of knowledge about the agent.
  • Empirical evaluations demonstrate that LLM agents are vulnerable to MEXTRA.
  • The study examines memory-leakage factors from both agent-designer and attacker perspectives.

Limitation

The study is evaluated only on single-agent systems and leaves multi-agent memory sharing for future work. It also identifies absent session control as a boundary that can expose records across users, while describing prompt-instruction variants used in experiments.

  • Limitation: The attack has been evaluated only in a single-agent setup, leaving risks from communicating or memory-sharing agents untested.
  • Limitation: Without session control, multiple users may share one session and the memory can store historical records from all users.
  • Prompt-generation instructions: The basic instruction generates 10 attacking prompts per run and repeats until the required total n is reached.
  • Prompt-generation instructions: For edit distance, the advanced instruction adjusts prompt lengths to extract user queries of diverse lengths.
  • Prompt-generation instructions: For cosine similarity, advanced instructions generate healthcare-domain phrases for EHRAgent and category-based phrases for RAP.

B.1 Experiment Setup

The experiments use EHRAgent for clinical code generation and RAP for Webshop actions, with memory records retrieved as demonstrations. The setup fixes retrieval functions, memory size, data sources, and GPT-4o as the agent core, while repeating attacks up to three times to reduce randomness.

  • Agent setup: EHRAgent retrieves top-4 memory records using edit distance, combines them with generated knowledge and prompts, then executes generated code.
  • Agent setup: RAP retrieves top-3 records for Webshop action generation using cosine similarity over SBERT embeddings based on MiniLM.
  • Agent setup: EHRAgent uses edit distance for four retrieved records, whereas RAP uses cosine similarity for three records.
  • Memory setup: Memory queries come from MIMIC-III for EHRAgent and Webshop for RAP, with corresponding agent-generated solutions forming query-solution records.
  • Embedding-model setup: The cosine-similarity experiments vary SBERT embedding models with different parameter scales and hidden sizes.
  • Attack setting: Attacks run up to three times and stop early when a prompt induces the agent to output all retrieved queries.

B.2 More Experiments

The experiments show that the proposed attacking prompt can extract retrieved private queries more reliably than a generic context-repetition prompt, and that MEXTRA generalizes to a question-answering agent. Retrieval overlap and prompt design affect the breadth of memory extraction.

  • Case study: The proposed attacking prompt successfully extracted all four retrieved queries from EHRAgent, whereas generic context repetition produced only a summarized part of the context.The ambiguous prompt failed to locate the attacker’s desired private queries.
  • Overlap Analysis: Nearly half of the queries in the retrieved record set were retrieved more than once across the two agents.Figure 5 reports 55 retrieved queries for EHRAgent and 27 for RAP.
  • Experiments on QA-Agent: MEXTRA generalized to a QA-Agent using GPT-4o, memory records of question–reasoning pairs, and top-4 retrieval of similar records.The QA-Agent used a 200-record memory and was evaluated with edit-distance and cosine-similarity scoring.
  • Experiments on QA-Agent: With 30 attacking prompts per instruction, advanced similarity-specific instructions extracted 55 and 46 queries, corresponding to around 25% leakage from 200-record memory.The high CER scores indicate that most attacking prompts successfully extracted all retrieved queries.
  • Discussion about Potential Mitigation: The study considers input/output controls and memory sanitation as mitigation categories, while noting that harmless-looking attacks can evade filters and paraphrasing may retain sensitive information.Memory sanitation includes de-identifying user queries before storage.
Loading 2502.13172v2…