Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking

Arulnidhi Karunanidhi

Source-linked AI summary

Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking

Arulnidhi Karunanidhi

arXiv:2608.21230v1Aug 21, 2026cs.CRcs.AI

TL;DR

Persistent memory makes plainly worded false assertions durable, raising the question of whether write-time screening or read-time provenance ranking can preserve utility under poisoning. The paper evaluates both defenses and finds that screening rejects none of the poisoned memories, while additive provenance weighting cannot both resist query-shaped poison and preserve legitimate untrusted evidence. It therefore motivates bounded occupancy constraints, while acknowledging that this remedy remains unevaluated.

Problem
Persistent memory can carry false statements into future sessions, but existing content-focused defenses are not designed to detect false assertions without external grounding.
Method
The paper poisons a memory benchmark with plainly worded false assertions and evaluates write-time content screening, provenance-weighted retrieval, utility retained, and retrieval-side diagnostics.
Results
0 of 360 poisoned memories were rejected, and additive provenance weighting had no usable setting under the measured similarity regime: weak weights failed against poison while strong weights suppressed untrusted evidence.
Takeaways & Limitations
The results motivate bounded occupancy constraints for provenance at retrieval rather than additive penalties, because the proposed design reserves space instead of suppressing content by score.
Takeaways & Limitations
The proposed occupancy remedy is unevaluated, and the conclusions assume provenance labels cannot be moved or laundered by the adversary.

Abstract

from arXiv · show

Persistent memory makes false information durable: once a false statement is stored, it can be retrieved into future sessions that match it. We measure the cost of this failure mode using plainly worded false assertions generated in a single pass, with no instruction, trigger, or retriever optimization. Poisoning 1.2% of a LongMemEval corpus reduces accuracy from 0.850 to 0.300. A four-stage write-time screening pipeline that reaches 0.832 recall on indirect prompt injection while flagging 1.5% of trigger-word-laden benign text rejects 0 of 360 poisoned memories. We argue this exposes a boundary of content-only screening: distinguishing a false assertion from a true one generally requires external grounding beyond the text itself. We then evaluate provenance-weighted retrieval. The shipped weight is statistically indistinguishable from no defense (p=0.80), while a stronger weight recovers utility only by excluding untrusted content. In a mixed-provenance corpus where untrusted content is mostly benign, accuracy rises from 0.3167 to 0.7000; when the answer-bearing evidence itself arrives untrusted, evidence recall falls to zero and accuracy to 0.0417. Under the measured similarity regime, the additive provenance term has no usable setting: a weight strong enough to resist query-shaped poison is also strong enough to suppress legitimate untrusted evidence. We therefore argue for bounded occupancy constraints at retrieval rather than additive provenance penalties, and release the harnesses, corpora, and aggregate run reports.

1 Introduction

Persistent memory turns a single accepted false statement into a durable retrieval threat. The paper measures this threat and finds that content screening misses plainly worded false-fact poisoning, while provenance weighting faces a utility trade-off.

Threat and utility: 1.2% poisoning reduced LongMemEval accuracy from 0.850 to 0.300 using ordinary false statements generated in one pass.The attack used no hidden instructions, embedding-space triggers, iteration, or retriever optimization.
Write-time screening: 0 of 360 poisoned memories were rejected by a four-stage screen with 0.832 recall on indirect prompt injection and 1.5% false positives on benign trigger-word text.The result supports a boundary of content-only screening rather than a tuning failure.
Read-time ranking: The shipped provenance weight was statistically indistinguishable from no defense (p=0.80) because its 0.175 margin was smaller than the poison’s 0.32 similarity advantage.The read path ranks memories using semantic similarity and a provenance prior.
Read-time ranking: A stronger provenance weight defended by excluding untrusted content, preserving utility only when that content carried no answer-bearing value.The mixed-provenance evaluation exposed this trade-off across benign distractors and untrusted evidence.
Implication and measurement: The paper argues that additive provenance penalties have no usable setting under the measured similarity regime and motivates bounded occupancy constraints instead.The proposed constraint reserves retrieval space rather than continuously penalizing provenance, but is not implemented in this work.
Implication and measurement: Utility retained, paired significance testing, retrieval diagnostics, and false-positive rate alongside recall are used to expose defense costs that attack success rate hides.The paper also releases harnesses, corpora, aggregate reports, and scripts for reproducing tables and figures.

2 Background and Threat Model

Persistent memory extends attacks across sessions and creates write-time and read-time interception points. The threat model centers on ordinary untrusted writes containing false assertions that resemble legitimate memories rather than instructions.

Persistent memory: Persistent memory lets stored content enter future contexts across sessions and agents, adding retrieval as a second defensive interception point.The attacker need only achieve storage and later retrieval, rather than immediate success on arrival.
Defensive interception points: Write-time screening asks whether content is dangerous, whereas read-time ranking asks how much matching content should be trusted.The paper treats these as distinct questions because ranking can remain useful when screening cannot decide truthfulness.
Trust as channel provenance: Trust levels record the channel of origin, and callers may lower but never self-elevate the declared level.The label is a provenance claim, not a judgment about content truthfulness.
Trust as channel provenance: Provenance cannot completely defend against false memories because an untrusted memory may be accurate while an internal memory may be wrong.The trust label is correlated with maliciousness but is not identical to it.
Threat model: The adversary can write through ordinary untrusted channels, knows likely user questions, and can phrase content to match them.The evaluated attacker cannot elevate trust, alter ranking, or optimize against retrieval feedback; the results are therefore a lower bound on capability.
Attack construction: The attack is a plainly worded false assertion in ordinary conversation, unlike direct injection, whose malicious intent is carried by an instruction directed at the model.The poison’s only defect is that its factual value is wrong, making it difficult for a content scanner to distinguish from a true memory.
Attack construction: A screening stage would need the answer to the question to detect the poison, which is why the paper treats this as a category-level boundary of write-time screening.The read path therefore carries defensive responsibility that the write path cannot reliably provide.

3 Related Work

Prior work largely studies instruction-carrying or optimized memory poisoning and reports attacker success, leaving utility retention under weak false-fact poisoning less measured. This paper instead evaluates whether defenses detect such poison and what value remains after attack.

Prior threat models: Existing defenses and benchmarks primarily address indirect prompt injection, optimized retrieval attacks, or instruction-carrying poison rather than plainly worded false assertions.The related work spans detector, model-level, system-level, and provenance-aware defenses, but the paper identifies weak-signal false-fact poisoning as an undermeasured case.
Relation to system-level defenses: System-level constraints prevent untrusted data from influencing control flow, whereas this paper measures failure when untrusted data is merely believed.The paper presents these conclusions as compatible because control-flow safeguards do not address false information entering memory.
Evaluation gap: Most prior studies report attack success rate, which does not measure retained system utility or defense cost on benign traffic.The paper measures what remains valuable after poisoning, including cases where genuine evidence is displaced without the attacker’s target answer being produced.
This paper’s distinction: This work uses a weaker attack than PoisonedRAG, AgentPoison, and MINJA: a single non-adaptive pass with false statements, no instructions, triggers, or optimization.The attack differences are presented as compounding the paper’s novelty relative to prior poisoning methods.
Lifecycle and defense: The paper evaluates write-time screening itself, producing a central negative result unavailable in studies that assume a poisoned static knowledge base.Its lifecycle-based evaluation asks whether the defense engages before poisoned memories are stored.
Provenance: Provenance-aware defenses remain relevant but differ in scope: this paper assumes correctly labelled provenance while related work questions whether provenance can remain trustworthy.The two concerns are described as orthogonal and compounding in the same direction.

4 System

The evaluated Aegis system combines a four-stage write-time screening pipeline with provenance-weighted read-time retrieval, using shipped defaults rather than research-only prototypes. Its ranking score combines semantic similarity with trust and other memory metadata.

System overview: Aegis ships both write-time screening and provenance-weighted retrieval as default defenses, not research prototypes.The paper emphasizes that the measured defaults are part of the released system.
Write path: Every API write passes through four stages whose final verdict flags content if any stage flags it, while retaining per-stage attribution.Stages 1–3 form a local deterministic core; Stage 4 is an optional network-based classifier evaluated separately.
Write path: The write pipeline checks structure, sensitive data, injection patterns, and optionally uses an LLM classifier for selected untrusted, shared, global, or previously flagged content.Stage 4 rejects at confidence 0.8 or above and otherwise may add a flag without changing the action.
Read path: Retrieval over-fetches memories and re-scores them with score(m) = w_s · sim(q, m) + w_t · τ(m) + w_e · e(m) + w_d · d(m) + w_p · p(m).The score combines cosine similarity, trust, observed effectiveness, temporal decay, and provenance metadata.
Read path: The trust prior encodes channel trust levels, not truth, and a weighted sum can only outbid similarity within a margin determined by the weights.Thus provenance cannot veto semantic similarity in this design.

5 Methodology

The methodology evaluates screening as binary content detection and measures clean and poisoned retrieval utility on LongMemEval. It uses weak-signal, query-shaped false memories, paired retrieval arms, and mixed-provenance controls to separate poisoning effects from trust-label confounds.

Evaluation design: Three evaluations measure write-path screening, clean retrieval quality, and retrieval quality after poisoning an established memory benchmark.The experiments use released harnesses with pinned revisions, seeds, and model identifiers.
Screening benchmark: Screening is scored as predict(text) → {flag, allow} on malicious and benign corpora, with precision, recall, F1, false positive rate, accuracy, and latency reported.This framing separates content detection from model robustness and makes benign traffic part of the evaluation.
Screening benchmark: The screening benchmark compares ten configurations across direct injection, indirect injection, ordinary benign text, memory-like entries, and trigger-word-laden benign text.NotInject specifically tests over-defense against benign sentences containing injection-associated triggers.
Memory benchmark: LongMemEval_S contains 500 questions whose evidence is hidden across roughly 50 sessions and about 115K tokens, spanning extraction, reasoning, updates, and abstention.Each question is isolated in its own namespace and answered from top-k=15 retrieved memories.
Poisoning protocol: For n=120 seeded questions, a generator creates three ordinary conversational rounds asserting plausible false answers, yielding 360 poisoned memories at approximately 1.2% of the corpus.The attack reuses question wording to compete with genuine evidence but uses no instruction, trigger, optimization, or retriever iteration.
Provenance design: Poison is marked untrusted while genuine conversation remains internal, explicitly distinguishing channel provenance from maliciousness or truth.The system has no oracle identifying false content, and benign content from the same channel receives the same label.
Retrieval arms and analysis: The study compares clean, undefended poisoned, shipped-weight, and corrected-weight arms, then adds mixed-provenance corpora and paired McNemar analysis.Mixed corpora make untrusted content predominantly benign in one case and include untrusted answer-bearing evidence in another.

6 Results

The results show that simple false-fact poisoning severely reduces memory utility, while screening misses it because falsity is not textually identifiable. Provenance weighting either fails to resist query-shaped poison or suppresses legitimate untrusted evidence, motivating bounded retrieval constraints.

6.2 Utility under attack: 0.850 to 0.300 accuracy: adding 360 fabricated memories, only 1.2% of the corpus, removed two-thirds of memory value.The attack used ordinary false assertions generated in one pass, without instructions, retriever optimization, or adversarial iteration.
6.2 Utility under attack: 0 of 360 poisoned memories were refused by write-time screening, despite the pipeline reaching 0.832 recall on indirect prompt injection.A poisoned memory ranked first for every question and occupied 20% of retrieved context.
6.2 Utility under attack: Accuracy rose from 0.3167 to 0.7000 in Corpus M, where 94% of untrusted content was benign, showing provenance weighting can help when untrusted content is mostly distractors.The gain is an upper bound under a favourable assumption because all untrusted content came from non-evidence rounds.
6.3 Neither weight setting is a defense: 0.0417 accuracy and zero evidence recall occurred in Corpus N when answer-bearing evidence was untrusted and provenance weighting used wt=0.35.The corrected weight suppressed legitimate untrusted evidence rather than merely demoting it.
6.3 Neither weight setting is a defense: A bounded occupancy constraint is proposed instead of an additive provenance penalty, but the paper motivates rather than implements or evaluates this design.The proposed quota would reserve retrieval space for untrusted content without allowing query-shaped poison to dominate ranking.

7 Limitations

The paper’s limitations center on transfer, adversary adaptivity, provenance assumptions, and the unevaluated proposed remedy. The mixed-provenance corpora bracket benefits and costs rather than locating a deployment operating point.

Adversary scope: The attacker is non-adaptive, using one-pass poison without retrieval or screening feedback, so adaptive attacks may perform better on both paths.The authors identify this as the most consequential limitation.
Mixed-provenance scope: Corpus M bounds benefit because untrusted content is entirely non-evidence, whereas Corpus N bounds cost because all answer-bearing evidence is untrusted.Together, these constructed extremes bracket a range without identifying where real deployments operate.
Proposed remedy: The proposed bounded-occupancy gate is motivated by measured failures but has not been implemented or evaluated for graceful degradation.Its effectiveness also depends on an occupancy quota that can retrieve suitable content.
Provenance assumptions: Trust labels are assumed accurate and immovable, although real integrations may mislabel provenance or let attackers obtain higher trust.The paper does not establish that this assumption holds in deployment.
System scope: Results come from one memory implementation, one retriever, one embedding model, and one reader, limiting empirical generalization across systems.The ranking conclusion depends on similarity distributions produced by particular embedders and corpora.
Domain and scale: LongMemEval_S represents conversational personal-assistant memory, while transfer to code, clinical, or operational memory remains unknown.Poisoning uses a paired seeded subsample of n=120 questions, so absolute poisoning-arm accuracies retain sampling error.
Measurement caveats: API-backed latency is not comparable with local measurements, and screening additionally refused 109 of 124,462 rounds as suspected credential leaks.The refused rounds were not answer-bearing, so no score was affected, but their over-defense shape was not characterized.

8 Conclusion

The paper measures how weak, plainly worded memory poisoning harms persistent-agent utility and finds that write-time content screening misses it. Provenance-weighted retrieval either fails to resist the poison or excludes legitimate untrusted evidence, motivating bounded occupancy constraints that remain unevaluated.

Conclusion: A one-pass attack using plainly worded false statements removes two-thirds of memory value when poisoning 1.2% of the corpus.The attack uses no instruction, trigger, or optimization.
Conclusion: Write-time screening rejects 0 of 360 poisoned memories despite 0.832 indirect-injection recall and a 1.5% benign trigger-word flagging rate.The authors attribute this boundary to the need for external grounding to distinguish false from true assertions.
Conclusion: The shipped provenance weight is statistically indistinguishable from no defense, while increasing it turns the trust term into hard exclusion rather than a soft prior.The corrected penalty exceeds half the range available to semantic similarity.
Conclusion: When answer-bearing evidence is untrusted, retrieval collapses to zero on all 120 questions.This contrasts with cases where excluding untrusted content removes only disposable material.
Conclusion: Under the measured similarity regime, no additive provenance weight both resists query-shaped poison and preserves the only available untrusted evidence.The paper therefore points toward bounded occupancy constraints, while explicitly leaving that mechanism unevaluated.
Conclusion: Utility retained should replace attack success rate as the primary memory-security metric, with false positive rate reported beside detector recall.These measures expose defense costs and utility loss that attack success rate cannot distinguish.
Conclusion: Future work prioritizes adaptive screening attacks, occupancy-gate evaluation, cross-embedder margin measurements, and protocols beyond additive scoring.These studies address the paper’s main unevaluated assumptions and scope boundaries.

A Reproducibility

The artifacts are generated from frozen inputs and pinned revisions, with deterministic seeds, caching, and committed reports supporting regeneration. The released bundle includes source, harnesses, attack data, and aggregate results.

Artifacts: Every table, figure, and prose macro is generated by one script from frozen JSON inputs.The screening input is copied verbatim from the benchmark results file.
Artifacts: Source, benchmark harnesses, the attack corpus, and committed aggregate reports are released at the pinned repository revision.The paper records the repository URL and revision 6d2863083361f7a5c8e12b4512346c94cb453c2c.
Datasets: LongMemEval_S uses revision 2ec2a557 with recorded SHA-256 08d8dad4...7894 and a synthetic benign corpus from generator version builtin-v1.The screening corpora and their revisions are given in Table 1.
Models: The reported memory benchmark uses claude-sonnet-5 as reader and gpt-4o-2024-08-06 as judge at temperature 0.Poison generation uses claude-haiku-4-5-20251001.
Sampling: Seed 42 governs bootstrap resampling, the n=120 question subsample, and mixed-provenance round selection.The subsample is drawn once and reused across arms, making comparisons paired.
Determinism: Responses are cached by system, model, prompt hash, and temperature, while unavailable systems are recorded as not-run.Changing temperature creates a fresh cache key.
Cost: The four mixed-provenance arms cost $11.22 in measured model API usage.The cost is recorded per arm in the released report.
Environment: Latency measurements use Python 3.11.9, transformers 4.53.3, torch 2.12.0 on CPU and should be interpreted comparatively.The reported environment does not establish absolute latency portability.