PaperlayerView exact revision on arXiv

Source-linked AI summary

Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guarantees

Yedidel Louck

arXiv:2606.24322v1cs.CR

TL;DR

Persistent LLM-agent memory enables poisoning attacks in which untrusted content later drives consequential actions, and existing content- or lineage-based defenses can be laundered. The paper formalizes these failures, proves a separation theorem, and introduces TMA-NM, whose origin-bound authority and corroboration gates achieve 0% attack success across tested models and channels while preserving legitimate utility.

  • Problem

    LLM-agent memory can preserve untrusted content across sessions, while content- and lineage-based authority signals are malleable under self-summarization, trusted-tool echo, and manufactured corroboration.

  • Method

    TMA-NM binds authority to origin at write time, propagates untrustedness nonmalleably, and permits elevation only through Sybil-resistant corroboration or fresh action-bound user authorization.

  • Results

    Across eight frontier models and direct or laundering attacks, TMA-NM reaches 0% attack success while retaining full legitimate utility.

  • Takeaways & Limitations

    The results support origin-bound, non-malleable authority as a structural defense against memory poisoning while preserving legitimate external-information actions.

  • Takeaways & Limitations

    The guarantees depend on correct origin labeling and genuinely independent corroboration, and the fully mechanized unbounded proof remains future work.

Abstract

from arXiv · show

LLM agents increasingly rely on persistent long-term memory, which creates a critical vulnerability that we study here: memory poisoning. An adversary can store untrusted content in one session that later steers a consequential action, such as a payment, a setting change, or data exfiltration, in a future session. Existing defenses base a memory item's authority to act on either its content (detection or trust-scoring) or its derivation history (lineage). We show that both signals are malleable. An attacker can launder an untrusted origin through three channels specific to LLM agents: the agent's own summarization, a trusted-tool echo, and manufactured corroboration. Each makes the content look benign and breaks or flips its derivation edge to ``trusted.'' We formalize malleability for the memory write-retrieve-act pipeline and prove a machine-checked separation theorem. No content- or lineage-based defense is sound under laundering (T1), write-time origin binding is necessary (T2), and non-malleable origin-bound authority with Sybil-resistant corroboration-gated elevation is sufficient (T3). Our construction, TMA-NM (Tamper-evident Memory Authority, Non-Malleable), instantiates non-malleable information-flow control (IFC) for LLM-agent memory. A cross-defense, cross-attack, and cross-model benchmark over eight frontier models shows that existing defenses fail exactly where the theory predicts (up to 68% laundering attack-success), while TMA-NM reaches 0% attack success on both direct and laundering attacks across all models and channels, at full legitimate utility. We release the benchmark, harness, and machine-checked TLA+ models to support reproducibility.

I. INTRODUCTION

Persistent memory enables cross-session poisoning, while existing content- and lineage-based defenses can be laundered through agent-specific transformations. The paper formalizes this gap and proposes non-malleable, origin-bound authority with machine-checked guarantees.

  • Memory poisoning lets stored content steer later consequential actions, including tool selection, data exfiltration, and other attacks.Reported injection success reaches approximately 99.8%.
  • Self-summarization, trusted-tool echo, and manufactured corroboration make untrusted memory appear benign or trusted without genuine authorization.These transformations respectively alter content and derivation, echo attacker-controlled content through trusted tools, or fabricate consensus.
  • Non-malleable origin-bound authority makes untrusted-origin content non-actionable and permits elevation only through independent trusted principals or fresh user authorization.Derived notes and trusted-tool outputs inherit untrusted authority, while self-asserted legitimacy does not count as corroboration.
  • The novelty is the separation between the unsoundness of malleable defenses and the sufficiency of non-malleable authority, rather than origin tagging alone.The paper explicitly distinguishes its contribution from prior origin-tagging and cross-session enforcement systems.
  • The paper contributes a formal model, a machine-checked separation theorem, and a benchmark spanning defenses, attacks, and models.The released artifacts include the benchmark, harness, and machine-checked proofs.

II. THREAT MODEL

The threat model treats memory as a write–retrieve–act pipeline in which an adversary controls untrusted content but cannot forge monitor-assigned origin or scope. TMA-NM binds authority at write time, propagates untrustedness, gates consequential actions, and logs decisions.

  • Memory items carry content, origin, scope, write time, and act class, and consequential actions include tool calls, payments, setting changes, and data egress.The pipeline is write → retrieve → act.
  • The adversary controls untrusted external content but cannot forge origin or scope, which the authenticated monitor assigns at write time.The model assumes authenticated channels and excludes cryptographic breaks and control of the user-authorization channel.
  • TMA-NM binds act authority to origin at writing, assigning untrusted items act=none and trusted-tool or user items act authority.The monitor mediates writes and records the origin rather than inferring it from content.
  • Non-malleable propagation keeps summaries and trusted-tool outputs derived from untrusted inputs untrusted, regardless of how benign their text appears.Propagation occurs at the tool-call boundary for opaque tools using declared input bindings.
  • Consequential actions are allowed only for trusted-derived values, values corroborated by at least two independent trusted principals, or fresh action-bound user authorization.Repeated untrusted items do not count, and the authorization token is bound to the concrete action.
  • Writes, elevations, and verdicts are appended to a tamper-evident hash chain, with linear storage growth and periodic checkpointing available for practical scaling.Each append uses a single hash over the previous chain head and adds no model call.

IV. FORMAL MODEL AND THE SEPARATION THEOREM

The formal model defines malleability as authority increasing under laundering without consent and checks security with exhaustive TLC analysis. It proves malleable gates insufficient, write-time binding necessary, and non-malleable authority sufficient within the bounded model.

  • A defense is malleable when an allowed laundering transformation raises effective authority without consent; non-malleability forbids such increases.The laundering set contains self-summarization, trusted-tool echo, and manufactured corroboration.
  • The bounded TLC model uses Slots=3, at most two sessions, threshold k=2, and exhaustively enumerates reachable states under a defense-independent security invariant.The invariant forbids consequential actions from untrusted-origin items unless independently elevated or user-authorized.
  • T1 shows that any content- or content-derived-lineage gate admits a reachable security violation through write-untrusted → paraphrase → act.This supplies a constructive witness for insufficiency under laundering.
  • T2 shows that disabling write-time origin binding violates security in a reachable state, establishing its necessity within the model.
  • T3 shows that origin-bound authority with thresholded independent corroboration or fresh action-bound authorization is non-malleable and satisfies security across 3,270 reachable states.The result is established for the bounded model and supported toward unbounded executions by an inductive invariant.
  • The formal guarantee depends on a minimal trusted base: correct channel-based origin labeling, independent trusted principals, and an uncompromised user-authorization channel.A successful attack therefore requires breaking one of these assumptions or compromising enough independent domains.

V. MEM-INV-BENCH

MEM-INV-Bench evaluates defense classes across domains, consequential tools, attack types, legitimate-elevation scenarios, and eight frontier models. It measures attack success, legitimate utility, unsafe automatic authorization, and latency.

  • The benchmark covers 12 domains, five consequential tool types, sleeper/control-flow/data-exfiltration attacks, and legitimate actions requiring trusted corroboration.Attack variants include benign, static, and adaptive conditions, with the supplied passage continuing into a white-box variant.
  • Five defense classes are compared: no defense, content trust scoring, lineage attribution, single-session capability IFC, and TMA-NM.The instances include a per-model LLM judge for trust scoring and MemLineage-style lineage attribution.
  • The benchmark reports consequential-action attack-success rate for direct and laundering attacks, legitimate utility, uncorroborated auto-authorization, and per-decision latency.Lower uncorrelated auto-authorization is safer because such actions should require confirmation.
  • The cross-model study uses eight frontier models from six vendors, including a strong-safety model and recent releases.A silent-failure guard excludes models with excessive empty-response rates.

VI. EVALUATION

Across unified, cross-model, ablation, persistence, and external-attack studies, TMA-NM blocks consequential attacks while preserving legitimate utility. Results also isolate origin binding as the security mechanism, corroboration-gated elevation as the utility mechanism, and show that write-time authority remains effective across sessions.

  • Unified benchmark: 0% attack success on direct and laundering attacks, with 100% legitimate utility, makes TMA-NM the only defense meeting both security and utility goals in the unified benchmark.The benchmark pools eight models and includes direct attacks, three laundering channels, and legitimate-action controls.
  • Cross-model study: 0/4032 consequential attacks and 95.9% task utility across blatant, disguised, and whitebox triggers match the undefended agent’s utility.The pooled cross-model study covers eight models, and every TMA-NM-versus-baseline gap is significant at p < 1/20,000.
  • Per-model robustness: TMA-NM remains at 0% across all eight models and trigger styles, while prob detect fails completely on gpt-5-chat.The per-model table reports 0/0/0 for TMA-NM with no utility loss.
  • Mechanism ablation: Removing origin binding restores undefended attack rates, removing elevation lowers utility from 96% to 77%, and removing the verdict log changes neither outcome.The ablation identifies origin binding as security-critical, corroboration-gated elevation as utility-preserving, and the verdict log as auditability rather than enforcement.
  • Cross-session persistence: 0% consequential-ASR persists across 0, 1, 2, 4, and 8 intervening sessions, whereas the undefended threat remains 51–61%.The persistence sweep shows that dormancy does not weaken write-time origin binding.
  • External validity: Across four reproduced published poisoning pipelines, TMA-NM blocks every attack at utility identical to the undefended agent.The pooled result is 0/1152 attacks with equal benign-task utility, while baselines fail in structurally distinct ways.

A. Theory↔benchmark correspondence (laundering)

The separation theorem’s predictions match the benchmark cell by cell: each defense class fails on the laundering channels identified by the theory, while TMA-NM remains secure.

  • TMA-NM holds at 0% attack success on every laundering channel and model.The result serves as the empirical witness of T1 and T3.
  • Content-based trust scores fail against self-summarization, trusted-tool echo, and manufactured corroboration.
  • Lineage fails against self-summarization and trusted-tool echo but resists manufactured corroboration because its derivation edges remain.
  • Capability IFC fails against every laundering channel, including the direct poison.

B. Multi-turn agentic evaluation

The evaluation extends TMA-NM from single proposals to multi-turn tool use and a production memory backend, preserving origin labels and legitimate utility across cross-turn dataflow.

  • The multi-turn loop propagates each value’s origin across reasoning turns and prevents untrusted data from reaching consequential actions.The deployment includes web or notes lookup, internal registry, and pay-or-send tools.
  • The multi-turn evaluation confirms the guarantee survives multi-step reasoning and cross-turn dataflow rather than only single-shot proposals.
  • TMA-NM authorizes 95/96 legitimate actions (99.0%) while reducing poisoning from 50.0% to 0% over Mem0-backed episodes.The undefended agent has a Wilson 95% CI of [40.2, 59.8], while TMA-NM’s is [0, 3.8].
  • Over Mem0, authority remains in the monitor’s append-only record despite fact extraction, consolidation, de-duplication, and vector compaction.

C. Baseline sensitivity and threshold generalization

Sensitivity studies show that content and lineage policies cannot jointly achieve security and utility, whereas TMA-NM maintains zero attack success with policy trade-offs exposed through corroboration thresholds.

  • At full utility, the content judge still leaks 25.9% laundering attack success, while TMA-NM achieves (0%, 100%).Driving laundering-ASR to 0% with the judge requires τ=0, which yields 0% utility; the best ASR≤5% point reaches 33.3% utility.
  • The content baseline’s failure is structural rather than a calibration artifact because laundered malicious and legitimate items both read as benign.
  • Attack success remains 0% for every corroboration threshold k, while increasing k converts more legitimate actions into one-time confirmation.The threshold k is compared with m independent vouchers, making it a security-versus-burden control.
  • Default-deny lineage closes self-summarization but still leaks trusted-tool echo, blocks legitimate actions with inconclusive provenance, and falls short of origin-binding.Table VIII reports that only origin-binding reaches 0% attack success at full utility.

VII. RELATED WORK

Related work spans persistent-memory attacks, lineage and single-session defenses, formal security frameworks, and classical integrity principles; TMA-NM differentiates itself through write-time origin binding and a machine-checked memory-authority guarantee.

  • Related defenses: MemLineage persists origin-derived labels but remains vulnerable when its derivation edge is suppressed and attribution falls back to trusted.It also lacks corroboration-gated elevation and a machine-checked guarantee.
  • Related defenses: Table VIII compares lineage default policies, showing that default-deny closes self-summarization but still leaks trusted-tool echo and harms inconclusive-provenance utility.
  • Differentiation: TMA-NM’s differentiation includes write-time origin binding, cross-session memory, non-malleable authority, corroboration-gated elevation, and a machine-checked guarantee.The comparison table defines these properties as WT, XS, NM, EL, and MC.
  • Threats and scope: Prior work addresses persistent-memory threats such as control-flow hijacking, sleeper poisoning, and data-exfiltration payloads, while related systems guard single-session prompt-to-action paths.
  • Foundations: TMA-NM connects classical Biba integrity and Clark–Wilson separation of duty to origin-bound action denial and independent-principal elevation.

VIII. DISCUSSION

TMA-NM’s zero attack success is structural: write-time origin binding blocks untrusted actions, while corroboration-gated elevation preserves legitimate utility. Its guarantee depends on correctly labeled origins and genuinely independent trusted principals.

  • 0% attack success across 4,032 cross-model runs and all unified, multi-turn, and Mem0 runs follows from deterministic write-time origin binding and action gating.Untrusted items become act=none at memory entry, regardless of dormancy, disguise, or asserted legitimacy.
  • TMA-NM resists forged trust claims because corroboration is counted by trusted origin, whereas content detection leaks 12.9% on the same whitebox attack.Text embedded inside an untrusted item cannot elevate that item’s authority.
  • 100% legit-utility in the unified study and 95.9% task utility across models show that TMA-NM blocks malicious actions without blocking corroborated legitimate ones.Removing corroboration-gated elevation preserves 0% ASR but reduces utility to 77%.
  • The unchanged monitor reaches 0% across eight models from six vendors, indicating that enforcement outside the model avoids dependence on model alignment.The undefended pooled direct-attack susceptibility is 84%.
  • Corroboration is only sound when principals have distinct identities, separate trust domains, and no shared upstream source; correlated compromise can fool naive counting.At k=2, two independent compromises defeat the threshold, while k=3 restores resistance.
  • Tamper-evident logging makes origin-labeling drift, registry misconfiguration, and suspicious channel or principal patterns auditable rather than silently persistent.Periodic log audits are the stated operational response to suspected channel compromise.

IX. LIMITATIONS

The guarantee is scoped to consequential retrieval-to-action decisions and depends on correct origin labeling and independent corroboration. Untrusted memory can still influence non-consequential free-text responses, and black-box value attribution remains incomplete.

  • TMA-NM mitigates but does not eliminate answer bias because untrusted memory may still color non-consequential responses.The system guards the retrieval-to-action path, not free-text answers, while surfacing provenance.
  • The guarantee holds only with correct origin labeling and genuinely independent corroboration; a fully compromised trusted tool could otherwise launder a value.This is why elevation requires at least two independent trusted sources rather than one.
  • Black-box deployments require value-level taint propagation to attribute which retrieved value the agent actually used, while the cross-model study uses a text proxy.The text proxy could be evaded by an obfuscating adversary; capability tokens are identified as the principled extension.
  • The paper frames memory integrity as authority laundering and releases the benchmark, harness, and machine-checked model supporting that analysis.

APPENDIX A BASELINE FIDELITY

The appendix defines faithful deterministic baselines for five defense classes and compares them against TMA-NM across laundering channels and models. TMA-NM alone achieves zero ASR across every laundering cell while preserving utility in the broader benchmark.

  • The undefended baseline authorizes every action, trust scoring uses content evidence, lineage defaults to trusted when provenance is inconclusive, and capability IFC assumes a clean memory store.
  • TMA-NM combines non-malleable effective origin with elevation requiring at least two independent trusted principals, so laundering does not change authorization.The baselines fail structurally: capability IFC assumes uncompromised memory, while lineage loses its blocking edge after laundering.
  • TMA-NM records 0/192 attacks per laundering channel, while Table XI reports the undefended consequential-ASR by model and channel.The table covers 24 runs per cell across eight models.
Loading 2606.24322v1…