PaperlayerView exact revision on arXiv

Source-linked AI summary

MemSentry: A Framework for Detecting Persistent Memory Poisoning in Agentic AI

Ayan Roy, Kaustuvi Basu

arXiv:2609.08747v1cs.CR

TL;DR

Persistent memory can let adversarial content influence future agent behavior and downstream security functions. MemSentry evaluates proposed writes before commitment using trust, semantic risk, impact, access risk, and security-state change, then routes them to deterministic decisions. Across four semantic methods, it detects all external quarantine-class attacks, while SBERT+LR has the strongest reported overall performance; verified insiders remain a review challenge.

  • Problem

    A poisoned persistent-memory write can persist beyond its originating interaction and affect downstream components and security functions.

    Source: §I · p. 1

  • Method

    MemSentry evaluates proposed memory writes using source trust, semantic risk, attack radius, access risk, and signed security-state change to produce deterministic Accept, Review, or Quarantine decisions.

    Source: §VIII · pp. 8–9

  • Results

    100% of external quarantine-class attacks were detected by all four methods, and SBERT+LR achieved 91.7% accuracy and 0.908 macro-F1.

    Source: G. Result 1: Classification Quality… · p. 6, §VIII · p. 9

  • Takeaways & Limitations

    Verified insiders with maximal trust are generally escalated to review rather than automatically quarantined, making semantic classification important for identifying risky insider operations.

    Source: §VIII · p. 9

    The evaluation considers individual operations rather than coordinated multi-step campaigns, treats posture and dependencies as static, and generally routes insider operations to review unless additional hard rules are introduced.

    Source: E. Limitations · p. 8

Abstract

from arXiv · show

Agentic AI systems with persistent memory introduce a distinct attack surface known as memory poisoning, in which adversarially crafted content is stored in long-term memory and subsequently influences future agent behavior. Such attacks can suppress security alerts, facilitate privilege escalation, alter trust relationships, or override security policies without modifying the underlying model weights or system prompts. To address this threat, we present MemSentry, a formal, configuration-driven framework that intercepts proposed persistent-memory writes and produces deterministic Accept, Review, or Quarantine decisions. MemSentry evaluates each write by jointly considering source trust, semantic risk, attack radius over a component-dependency DAG, access risk, and a signed security-state delta that captures whether an operation weakens or strengthens the system's security posture. We instantiate the protected environment using a 20-asset random dependency DAG and a 10 x 20 user access-control matrix, and evaluate the framework over 1,000 GPT-4-generated scenarios using a stratified 70/30 train/test split. Semantic classification is treated as a pluggable component rather than a primary contribution, and we compare four representative approaches: rule-based Regex, TF-IDF+SVM, SBERT+LR, and SetFit. SBERT+LR achieves the best overall performance with 91.7% accuracy and a 0.908 macro-F1 score, while all four methods detect 100% of external quarantine-class threats. For verified insiders, where source trust is maximal (T = 1), MemSentry does not automatically quarantine suspicious operations but instead escalates potentially dangerous writes for human review, making semantic classification important for accurately capturing insider intent.

I. INTRODUCTION

Persistent memory allows adversarial content to influence future agent behavior beyond the interaction in which it was encountered. MemSentry assesses proposed writes before commitment, extending prior defenses that often act after poisoned content enters memory.

I. INTRODUCTION

  • A single persistent-memory write can continue influencing future reasoning, planning, and action execution after the original interaction ends.If admitted without sufficient validation, it may affect downstream components and security functions, including alerting and automated response.Source: §I · p. 1
  • Persistent memory supports continuity across interactions, but also creates an attack surface when information from untrusted sources is written to long-term memory.Enterprise email assistants illustrate the setting: they interpret messages, retrieve context, generate responses, and initiate follow-up actions.Source: §I · p. 1
  • MemSentry evaluates candidate writes before commitment using provenance, source authority, semantic intent, affected actions, and deterministic security constraints.It projects downstream impact across components and protected assets, then compares executions with and without the candidate memory to estimate observed blast radius.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior approaches include retrieval-time defenses and post-hoc auditing, while MemSentry assesses each proposed memory operation pre-commit.The framework considers provenance, authorization, semantic risk, and downstream impact before deciding whether to accept, review, or quarantine a write.Source: §II · p. 2

III. PROBLEM STATEMENT

The problem is to assess whether a proposed memory operation would degrade or improve system security and route it to an enforcement action. MemSentry frames this as a pre-commit evaluation over source, semantics, dependencies, and policy.

III. PROBLEM STATEMENT

  • A proposed operation is assessed for whether committing it to active memory would degrade or improve system security, then routed to an enforcement action.The available outcomes are Accept, Review, and Quarantine.Source: §III · pp. 2–3

IV. THREAT MODEL

  • The threat model includes external unauthenticated actors and compromised insiders who can submit adversarial content despite authenticated identities.External actors can craft arbitrary content but cannot authenticate as internal users; compromised insiders have authenticated identities and access to some components.Source: §IV · p. 3

V. PROPOSED FRAMEWORK

  • MemSentry processes each proposed write through source assessment, semantic analysis, impact estimation, and rule-based decision logic.The protected system is modeled as a directed dependency graph, and affected components include those reachable downstream from the operation's directly targeted components.Source: §V · p. 3

A. Step 1: Source Trust

The framework assigns source trust from origin, identity verification, and channel integrity, then classifies semantic risk and estimates the reach of an operation through the dependency graph.

A. Step 1: Source Trust

  • Source trust is computed from origin, identity verification status, and channel integrity, with internal verified sources assigned the maximum trust level of T = 1.0.External unverified and internal unverified sources are assigned T = 0.0 and T = 0.5, respectively; external verified sources also receive T = 0.5.Source: A. Step 1: Source Trust · p. 3

B. Step 2: Semantic Classification

  • Semantic categories assign risk scores, with positive values indicating potentially harmful operations and negative values indicating security-strengthening operations.Examples include alert_suppression at 0.80 and security_strengthening at −0.30.Source: B. Step 2: Semantic Classification · p. 3

C. Step 3: Attack Radius

  • The attack radius uses the affected component set, including directly targeted and transitively reachable components, to represent the fraction of system components affected.Reachability is computed by depth-bounded graph traversal.Source: C. Step 3: Attack Radius · pp. 3–4

D. Step 4: Access Risk

Access risk measures unauthorized exposure among affected components, while security-state projection estimates how an operation changes the system’s posture. The projection accounts for component criticality, source trust, access risk, and persistence.

D. Step 4: Access Risk

  • Access risk is the criticality-weighted fraction of affected components for which the principal lacks authorization.Source: D. Step 4: Access Risk · p. 4

E. Step 5: Security State

F. Step 6: Projected State

  • Degradation scales with source distrust, beneficial improvement with trust; unauthorized targeting amplifies degradation, and longer persistence magnifies either impact.Source: F. Step 6: Projected State · p. 4

G. Step 7: Security Delta

The framework computes a signed security-posture change, combines risk factors into a score, and applies deterministic rules to route writes to Accept, Review, or Quarantine.

G. Step 7: Security Delta

  • Negative security delta indicates degradation and positive delta indicates strengthening, with magnitude quantifying projected impact.Source: G. Step 7: Security Delta · p. 4

H. Step 8: Risk Score

I. Step 9: Decision Rules

  • External unverified sources are quarantined; acceptance requires a trusted internal source, authorization, low semantic risk, and benign content, while critical targets go to review.Operations with positive security delta from trusted sources are accepted as strengthening measures.Source: §I · pp. 4–5

J. Output Structure and Explanation Generation

MemSentry produces an auditable explanation record for each decision, assembling intermediate values and human-readable reasons deterministically from the pipeline’s computed values.

VI. EXPERIMENTAL EVALUATION

The evaluation section specifies experimental coefficients and describes the generated scenario dataset and its train-test split.

A. Coefficient Values

  • Table II provides the concrete coefficient values used in all experiments for the symbolic parameters introduced earlier.Source: A. Coefficient Values · p. 5

B. Dataset

  • The dataset contains 1,000 GPT-4-generated scenarios, split stratified into 700 training and 300 testing samples; Table I summarizes its composition.Machine-learning classifiers use the training subset, while Regex operates without training data.Source: B. Dataset · p. 5, Table I · p. 5
  • Figure 3 is identified as a MemSentry input sample.Source: Figure 3 · p. 5

C. Input/Output Examples

MemSentry is illustrated through an external alert-suppression scenario, a comparison of semantic classifiers, and a modeled 20-asset dependency DAG. The DAG and access matrix define the protected environment for assessing downstream impact.

C. Input/Output Examples

  • Figures 3 and 4 illustrate an external unauthenticated attempt to suppress security alerts through a temporarily persistent email memory operation.The supplied Fig. 4 caption identifies the output sample but does not describe its contents.Source: C. Input/Output Examples · p. 5, Figure 4 · p. 6

D. Semantic Classification Comparison

  • Semantic classification is treated as an interchangeable component, with Regex, TF-IDF plus Linear SVM, Sentence-BERT plus Logistic Regression, and SetFit compared.SetFit contrastively fine-tunes the Sentence-BERT encoder on the training set before classification.Source: D. Semantic Classification Comparison · p. 5

E. System Model: Company Dependency DAG

F. Semantic Analysis Methods and Risk Mapping

MemSentry holds the downstream pipeline constant while comparing semantic classifiers as interchangeable risk-scoring components. SBERT+LR leads overall, while all methods identify quarantine-class attacks and differ most on review cases.

F. Semantic Analysis Methods and Risk Mapping

  • Semantic analysis maps each message to a risk score and category, after which every classifier uses the same downstream pipeline.The compared approaches are lexical TF-IDF with calibrated Linear SVM, Sentence-BERT with Logistic Regression, and task-adapted SetFit; Regex uses a fixed category-risk table.Source: F. Semantic Analysis Methods and Risk… · p. 6

G. Result 1: Classification Quality (70/30 Split)

H. Result 2: Single-Insider Authorization Sweep

As verified-insider authorization rises, acceptance accuracy increases while review accuracy falls, and insider operations are not automatically quarantined. In held-out semantic-contrast pairs, no method resolves more than three of ten.

H. Result 2: Single-Insider Authorization Sweep

I. Result 3: Semantic Contrast Scenarios

  • No method resolves more than 3/10 held-out semantic-contrast pairs; two quarantine-expected insider cases are never quarantined.The paper attributes the contrast-set outcome to surface-similar pairs with opposite security meaning exceeding the evaluated methods' reasoning.Source: §I · p. 7, Table VII · p. 8

A. Source Trust and the Insider Ceiling

  • The trust-scaled degradation mechanism makes source trust a structural dependency: external threats are quarantined, while verified insiders are escalated to review.Across the authorization sweep, accept accuracy improves while review accuracy declines; at full authorization, SBERT+LR retains 0.91 review accuracy and SetFit drops to 0.027.Source: H. Result 2: Single-Insider… · p. 7, A. Source Trust and the Insider Ceiling · p. 8

B. Semantic Classification as a Pluggable Component

MemSentry’s security-evaluation pipeline is the primary contribution, with semantic classification treated as pluggable. Results highlight strong external-threat detection but gaps in subtle intent distinctions and limits in the evaluation scope.

B. Semantic Classification as a Pluggable Component

  • SBERT+LR leads review accuracy at 72.2%; TF-IDF+SVM reaches 88.0% accuracy, SetFit 22.2% review accuracy, and Regex 78.3% overall accuracy.The security-evaluation pipeline, rather than the semantic classifier, is the primary contribution.Source: B. Semantic Classification as a… · p. 8

C. The Semantic Reasoning Gap

  • None of the evaluated methods reliably distinguishes contrast scenarios: TF-IDF+SVM gets 3/10 pairs right, Regex 1/10, and SBERT+LR and SetFit 0/10.The contrasts include authenticated versus authorized, temporary versus indefinite, and recording versus suppressing; the paper points to LLM- or NLI-based analysis as future work.Source: C. The Semantic Reasoning Gap · p. 8

D. Attack Propagation over the Dependency DAG

  • Dependency-DAG topology shapes attack radius: sink assets have AR = 0.05, while asset_15 and asset_12 can affect 11 and 9 assets, respectively.Highly connected assets amplify potential impact.Source: D. Attack Propagation over the… · p. 8

E. Limitations

  • The evaluation covers individual memory operations, not coordinated multi-step campaigns, and treats security posture and dependencies as static.Insider operations go to review rather than quarantine unless additional content-based hard rules are introduced.Source: E. Limitations · p. 8

VIII. CONCLUSION

  • MemSentry evaluates proposed writes using source trust, semantic risk, attack radius, access risk, and signed security-state change to produce Accept, Review, or Quarantine decisions.Evaluation used 1,000 GPT-4-generated scenarios, a 20-asset dependency DAG, and a 10 × 20 access-control matrix.Source: §VIII · pp. 8–9
  • 100% of external quarantine-class attacks were detected across methods; SBERT+LR led overall at 91.7% accuracy and 0.908 macro-F1.The conclusion identifies insider defense as an open challenge because trusted insiders are generally escalated to review rather than automatically quarantined.Source: B. Semantic Classification as a… · p. 8, §VIII · p. 9
Loading 2609.08747v1…