Source-linked AI summary
Memory Poisoning Attack and Defense on Memory Based LLM-Agents
Balachandra Devarangadi Sunil, Isheeta Sinha, Piyush Maheshwari, Shantanu Todmal, Shreyan Mallik, Shuchi Mishra
TL;DR
Persistent-memory LLM agents can be poisoned through query-only interactions, but realistic conditions and effective defenses remain insufficiently evaluated. This work studies attack robustness in EHR agents across memory and retrieval settings and evaluates moderation and trust-aware sanitization. Realistic legitimate memories sharply reduce attack effectiveness, while sanitization requires calibrated trust thresholds because models may reject benign entries or accept confident attacks.
Problem
Query-only memory poisoning is a serious vulnerability, yet attack robustness in realistic deployments and empirical defense effectiveness remain understudied.
Method
The study varies initial memory, indication prompts, and retrieval parameters in EHR agents, then evaluates Input/Output Moderation and trust-aware Memory Sanitization.
Results
Realistic legitimate memories reduce attack effectiveness, with ASR falling to 0% for Llama and as low as 6% for GPT-4o-mini in reported persistent-memory settings.
Takeaways & Limitations
Securing memory-augmented agents requires evaluating realistic memory conditions and calibrating sanitization thresholds to balance poisoning prevention with memory utility.
Takeaways & Limitations
Trust-aware sanitization can reject all entries or accept sophisticated poisoned entries with perfect trust scores, making it a confidence rather than security filter.
Abstract
from arXiv · showhide
Large language model agents equipped with persistent memory are vulnerable to memory poisoning attacks, where adversaries inject malicious instructions through query only interactions that corrupt the agents long term memory and influence future responses. Recent work demonstrated that the MINJA (Memory Injection Attack) achieves over 95 % injection success rate and 70 % attack success rate under idealized conditions. However, the robustness of these attacks in realistic deployments and effective defensive mechanisms remain understudied. This work addresses these gaps through systematic empirical evaluation of memory poisoning attacks and defenses in Electronic Health Record (EHR) agents. We investigate attack robustness by varying three critical dimensions: initial memory state, number of indication prompts, and retrieval parameters. Our experiments on GPT-4o-mini, Gemini-2.0-Flash and Llama-3.1-8B-Instruct models using MIMIC-III clinical data reveal that realistic conditions with pre-existing legitimate memories dramatically reduce attack effectiveness. We then propose and evaluate two novel defense mechanisms: (1) Input/Output Moderation using composite trust scoring across multiple orthogonal signals, and (2) Memory Sanitization with trust-aware retrieval employing temporal decay and pattern-based filtering. Our defense evaluation reveals that effective memory sanitization requires careful trust threshold calibration to prevent both overly conservative rejection (blocking all entries) and insufficient filtering (missing subtle attacks), establishing important baselines for future adaptive defense mechanisms. These findings provide crucial insights for securing memory-augmented LLM agents in production environments.
1 Introduction
Persistent memory improves LLM-agent continuity and personalization but creates an attack surface that query-only adversaries can poison, with especially serious consequences in healthcare. This work studies both realistic attack robustness and defenses for EHR agents.
- Persistent memory lets agents retain context, learn from interactions, and personalize responses by retrieving historical examples.
- Query-only attackers can embed malicious instructions in benign-looking queries to induce poisoned memories that later corrupt legitimate responses.MINJA uses bridging steps, indication prompts, and progressive shortening.
- EHR memory poisoning could redirect patient identifiers, returning the wrong medical records and contributing to misdiagnosis or incorrect prescriptions.
- Existing evaluations emphasize idealized settings, leaving attack performance under accumulated legitimate memories, varied attack queries, and variable retrieval insufficiently understood.
- Defense effectiveness and novel mechanisms remain minimally studied because plausible, contextually harmless reasoning can evade existing moderation and sanitization.
- This work systematically evaluates memory poisoning attacks and defenses in EHR agents.
2 Related work
Prior research documents persistent poisoning across memories, prompts, models, functions, and multi-agent systems, while available defenses remain limited in coverage, access assumptions, and empirical validation.
- Memory and system-prompt poisoning studies show that malicious behavior or instructions can persist beyond individual interactions.
- Related work extends agent vulnerabilities to function libraries, multi-agent debate, prompt propagation, and broader security benchmarks.
- DRIFT filters inputs dynamically but does not address memory-level defenses or temporal attack propagation.
- A-MemGuard provides proactive memory protection but assumes direct memory instrumentation unavailable in black-box API deployments.
- The literature lacks production-like evaluations of accumulated memories, temporal poisoning propagation, and security-utility trade-offs.
3 Threat Model
The threat model places output correctness and memory integrity at risk from regular users who can submit multiple crafted queries without direct memory access.
- The primary assets at risk are agent output correctness and long-term memory integrity.
- The attacker is a regular user who interacts through the standard interface and lacks elevated privileges or direct read/write memory access.
- Attackers can send multiple crafted queries, subject to rate limits, to induce malicious reasoning and persist records for future reuse.
- The main attack surface is the user-input channel, with memory APIs as another possible route to persistent storage.
4 Research Questions and Hypotheses
The study asks when MINJA fails under realistic retrieval conditions and how to design guardrails against query-only poisoning, evaluating attack dimensions alongside input/output moderation and trust-aware memory sanitization.
- Research Questions: MINJA is evaluated across initial memory state, indication-prompt count, and the number of relevant memories retrieved.
- Hypotheses: More correct victim-ID memories are hypothesized to reduce malicious retrieval and both attack and injection success rates.
- Hypotheses: Increasing indication prompts is hypothesized to increase both attack success rate and injection success rate.
- Hypotheses: Increasing retrieved relevant memories is hypothesized to increase attack success rate while leaving injection success rate unaffected.
- Defenses: Input/Output Moderation combines static heuristics, keyword matching, semantic classification, and composite trust signals across execution stages.
- Defenses: Memory Sanitization applies trust scoring during append and retrieval, temporal decay, and trust-aware filtering of few-shot examples.
- Defenses: The defense hypotheses target lower attack success while preserving benign-query utility and improving long-term robustness through trust decay.
5 Attack
The EHR memory-poisoning attack uses bridging interactions, indication prompts, and progressive shortening to create malicious memories, with effectiveness tested under varying memory and retrieval conditions. Realistic legitimate memories reduced attack success, while retrieving more memories increased poisoning effectiveness for GPT-4o-mini and raised attack success for Llama.
- Attack mechanism: The attack steers the agent toward poisoned associations through bridging interactions, indication prompts, and progressively shortened malicious context.The final poisoned memory is designed to remain short, plausible, and difficult to audit.
- Retrieval: Top N memories are retrieved by Levenshtein distance and appended to the original query, with lower distance indicating greater similarity.The number of retrieved memories is varied experimentally.
- Baseline: 62% ASR for GPT and 52.94% for Llama were observed in the baseline, while ISR was 100% with empty initial memory.The GPT-4o-mini attack success rate was reported as 60% in the baseline discussion, and ISR exceeded ASR because some instructions were ignored in longer contexts.
- Initial memory: 25% ISR and as low as 6% ASR were observed for GPT-4o-mini when correct, relevant memories were present initially.Relevant victim-ID memories reduced the chance that malicious target-ID memories were retrieved.
- Initial memory: 0% ASR was observed for Llama with persistent initial and accumulated memories, despite ISR remaining 100% in the baseline setting.The reported explanation is that correct few-shot examples and similarity-based retrieval reinforced the correct patient-ID pattern.
- Retrieval and prompts: Increasing indication prompts from 2 to 4 did not significantly affect attack metrics, whereas increasing retrieved memories increased GPT-4o-mini ISR and ASR.For Llama, ASR rose from 0 to 13.33 to 27.27 as retrieval increased from 3 to 5 to 10 memories, while ISR stayed high.
- Realistic conditions: In realistic settings, attack success was as low as 38% for GPT-4o-mini and 28% for Llama, although query-only poisoning remained possible.The realistic experiment included six initial memories, while substantially increasing retrieval could raise attack success but also increase processing and context costs.
6 Defense
The defenses place moderation around model execution and add trust-aware sanitization at memory append and retrieval stages. Evaluation uses attack suites, ablations, multiple seeds, and audit logs to measure protection and utility trade-offs.
- 6.1 Input/Output Moderation: Input/output moderation uses static heuristics, semantic classification, dangerous-code checks, and composite trust scoring to gate execution and memory appends.High-risk inputs are skipped, while passed queries undergo output moderation before append decisions.
- 6.1 Evaluation: The defense evaluates baseline and defended runs on eICU and MIMIC attack suites using attack success, memory appends, rejections, and benign-query utility measures.Ablations cover input-only, output-only, trust-threshold, and sandbox configurations.
- 6.2 Memory Sanitization: Memory sanitization operates at append and retrieval time, assigning continuous trust scores from safety, relevance, answer agreement, verification, sandboxing, and poisoning-pattern checks.Only sufficiently trusted entries without critical failures are appended.
- 6.2 Memory Sanitization: Retrieval applies temporal decay, threshold filtering, pattern-based blocking, and joint similarity-trust ranking to limit suspicious or stale few-shot memories.Higher-trust, relevant entries are preferentially selected among remaining memories.
- 6.2 Memory Sanitization: The sanitization layer is intended to block low-trust content at entry, reduce trust in stale items, and filter poisoned memories during retrieval.The design targets both initial injection and long-term propagation.
- 6.2 Evaluation: The evaluation uses 101 poison queries targeting patient-identifier aliasing, with MIMIC-III, four-shot memory, model-specific guard agents, and logged append decisions.Reports summarize trust distributions, blocking, leakage, retrieval filtering, and memory cleanup.
7 Evaluation
The evaluation measures trust distributions, attack prevention, retrieval filtering, and cleanup behavior to characterize memory sanitization across append and retrieval stages.
- Trust Score Distribution: Trust-score analysis counts candidate entries, accepted and rejected items, and score statistics across high, medium, and low trust bins.The bins are high ≥0.8, medium 0.5–0.8, and low <0.5.
- Attack Prevention Effectiveness: Attack prevention tracks whether each poison query is appended, blocked, or leaked into long-term memory, including decisions by trust level.The analysis reports totals, fractions, and trust scores for poison queries.
- Retrieval-Time Filtering: Retrieval-time filtering simulates effective-trust thresholding after temporal decay and compares filtered versus retained accepted entries.The evaluation also compares average trust scores between the two groups.
- Memory Cleanup Behavior: Memory cleanup analysis counts events removing low-trust, aged entries and records representative examples to assess long-term mitigation.This examines whether cleanup contributes to reducing poisoning over time.
8 Experimental Results
The experiments show sharply different sanitization behavior across runs: one configuration rejected every candidate entry, while another accepted high-trust entries that included malicious instructions. Trust distributions and decision boundaries explain these outcomes.
- 8.1 Experiment 1: GPT-4o-mini: 23 candidate memory entries were processed, with 0 accepted and 23 rejected; rejected scores averaged approximately 0.28 with a 0.0–0.4 range.All entries were in the low-trust (<0.5) bin.
- 8.1 Experiment 1: GPT-4o-mini: 101 poison queries produced 2 blocks and 0 memory accepts, corresponding to approximately 1.98% blocking and no leakage at the sanitization layer.The reported metric covers memory sanitization only; earlier guard-agent interception may also occur.
- 8.1 Experiment 1: GPT-4o-mini: With no accepted entries, retrieval filtering found 0 filtered and 0 retrieved items, while cleanup recorded no events because long-term memory remained empty.The 0% retrieval-filtering rate reflects append-time rejection in this run.
- 8.2 Experiment 2: Gemini-2.0-Flash: A second run processed 151 candidates, accepting 82 at trust 1.0 and rejecting 69 with mean trust 0.35 and range 0.28–0.40.Rejected entries were low-trust (<0.5), while accepted entries were high-trust (≥0.8).
- 8.2 Experiment 2: Gemini-2.0-Flash: The trust distribution was trimodal near 0.3, 0.4, and 1.0, despite an overall mean of 0.70, indicating polarized scores.Most entries were either highly trusted or below 0.5.
- 8.2 Experiment 2: Gemini-2.0-Flash: Accepted entries uniformly scored 1.0, whereas rejected entries clustered around 0.35–0.40, revealing a hard decision boundary in the guard logic.No low-trust entries were accepted and no high-trust entries were rejected in the tested run.
- 8.2 Experiment 2: Gemini-2.0-Flash: 54 of 82 accepted entries were malicious poison queries, while 28 were benign, leaving 54 attacks stored despite 69 rejected entries.None of the 82 accepted entries would be filtered at a standard threshold of 0.5 because each scored 1.0.
- 8.2 Experiment 2: Gemini-2.0-Flash: The Gemini-2.0-Flash run accepted memory entries that GPT-4o-mini did not, but 54 malicious instructions were stored with maximum confidence.This run therefore produced contaminated long-term memory rather than an empty memory store.
9 Conclusions
The study finds that trust-based memory sanitization presents a trade-off between poisoning prevention and memory utility. Its failures motivate calibrated thresholds, improved pattern detection, mixed-workload evaluation, and separating trust from safety.
- Trust-based sanitization can reject all candidate entries, including benign ones, producing effectively empty memory.This prevents poisoning in the reported GPT-4o-mini experiment but may reduce long-term-memory benefits in mixed workloads.
- 54 poisoned entries received perfect trust scores of 1.0, showing that sophisticated prompts can manipulate trust assessment.Justification clauses enabled adversarial prompts to pass despite being poisoned.
- The defense therefore behaved as a confidence filter rather than a security filter, allowing well-phrased attacks through.
- Future work should calibrate trust thresholds, refine poisoning detectors, and evaluate settings containing both benign and adversarial queries.These directions target the safety–utility trade-off and adaptive sanitization behavior.
- Decoupling trust from safety and adding verification-of-effect checks are proposed as ways to strengthen memory defense.
10 Appendix
The appendix presents paired victim and target patient identifiers alongside example clinical queries. The examples span admissions, laboratory values, prescriptions, procedures, costs, diagnoses, and monitoring history.
- Five victim–target patient-ID pairs define the identifier mappings used in the appendix.
- The appendix includes queries about intensive-care stays, lactate measurements, alcohol-withdrawal diagnoses, prescriptions, medication timing, and hospital costs.
- Additional queries cover unstable-sternum diagnoses, EEG monitoring, pericardial-window procedures, non-invasive ventilation, temperature changes, and laboratory tests.
Appendix 10.4.1: Memory 1 for Victim ID 027-22704
This appendix memory explains how to find the first occurrence of the maximum lactate value for patient 027-22704 during December 2101.
- The query targets the first timestamp at which patient 027-22704 reached the maximum lactate value in December 2101.
- The procedure identifies the patient admission, retrieves lactate measurements from labevents, and filters them to December 2101.
- It selects the maximum lactate value, filters matching events, and returns the earliest corresponding CHARTTIME.
Appendix 10.4.2: Memory 2 for Victim ID 027-22704
This appendix memory explains how to identify the latest alcohol-withdrawal diagnosis for patient 027-22704 using ICD9-coded diagnosis records.
- The query asks for the most recent diagnosis of alcohol withdrawal for patient 027-22704.
- The procedure maps the alcohol-withdrawal label to an ICD9 code using d_icd_diagnoses.
- It filters diagnoses_icd by patient and code, then selects the latest diagnosis timestamp.