PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Similarity: Trustworthy Memory Search for Personal AI Agents

Jiawen Zhang, Kejia Chen, Jiachen Ma, Yangfan Hu, Lipeng He, Yechao Zhang, Jian Liu, Xiaohu Yang, Tianwei Zhang, Ruoxi Jia

arXiv:2606.06054v1cs.AI

TL;DR

Semantic similarity can retrieve memories that are relevant to a query but contextually inadmissible, creating a trustworthiness gap for personal agents. The paper evaluates memory-induced risks and introduces MemGate, a query-conditioned admission plug-in. Across the reported settings, MemGate reduces memory-induced failures while preserving or improving memory utility.

  • Problem

    Semantic proximity does not ensure a memory is appropriate, objective, or safe to influence the current query.

    Source: §2.3.1 · p. 3

  • Method

    MemGate applies a query-conditioned gate to candidate memories before context injection, re-ranking the initial similarity-retrieved set without modifying the backbone model.

    Source: §4.2 · p. 8, §6 · p. 13

  • Results

    MemGate reduces memory-induced failures while preserving or sometimes improving utility; on GPT-4o-mini, cross-domain leakage falls from 27.0% to 3.5%.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    Trustworthy personalization requires agents to determine when remembered user context is legitimate evidence for the current task.

    Source: §6 · p. 13

    The threat model limits user influence over memory to standard conversation and excludes direct modification of model weights, system prompts, retrieval code, tool schemas, or external state.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Personal AI agents increasingly rely on long-term memory to provide persistent personalization across sessions. However, existing memory pipelines are largely driven by semantic similarity: memory data close to the current query is retrieved and injected into the model context. This creates a critical trustworthiness gap, since a semantically related memory may still be contextually inappropriate, leading to threats such as cross-domain leakage, sycophancy, tool-call drift, or memory-induced jailbreaks. In this paper, we study memory search as a trust boundary in personal AI agents. We evaluate representative agentic memory frameworks, including A-Mem, Mem0, and MemOS, together with OpenClaw, a real-world personal-agent environment with persistent state and tool-use capability. Our results show that long-term memory is not merely a utility layer, but a durable control channel that can reshape how agents interpret tasks and execute actions, leaving them highly susceptible to the aforementioned threats. To mitigate these vulnerabilities, we propose MemGate, a lightweight and deployable memory plug-in for trustworthy memory search, with only 9M parameters and a 35.1MB footprint. MemGate is inserted between the vector memory store and the backbone LLM, requiring no LLM modification, memory-database rewriting, or inference-time LLM judge. It applies a query-conditioned neural gate to candidate memory representations, turning raw similarity search into task-conditioned memory admission. Across multiple mainstream memory frameworks, real-world agent settings, and diverse LLM backbones, MemGate reduces memory-induced threats while preserving long-term memory utility.

1. Introduction

Semantic similarity can retrieve memories that are related to a query but inappropriate to influence it, making memory search a trust boundary. The introduction presents MemGate as a lightweight admission layer and reports reduced memory-induced failures alongside preserved or improved utility.

1. Introduction

  • Because semantically similar memories can still be inappropriate, memory admission—not just recall—determines which persistent user states may influence the current task.Such influence can arise from persistent user-specific state rather than a malicious instruction in the current prompt.Source: §1 · pp. 1–2
  • Memory raises average tool-call drift failure from 5.1% in the memory-free baseline to over 50%, while malicious memory raises average ASR from 3.1% to around 20%.The first comparison concerns benign over-personalization; the second concerns malicious memory manipulation.Source: §1 · p. 2
  • Prompt-level instructions and rigid heuristic filters may not address harmful memory influence once unsafe context has entered the model's prompt.The introduction attributes the difficulty of heuristic filtering to complex semantic cross-resonances between queries and stored memories.Source: §1 · p. 2
  • MemGate gates candidate memories before they enter the LLM context, attenuating dimensions linked to domain mismatch, stale constraints, or unsafe associations while leaving the query unchanged.It is described as a 9M-parameter, 35.1MB plug-in that does not require backbone changes, memory-database rewriting, or an inference-time LLM judge.Source: §1 · p. 2
  • On GPT-4o-mini, MemGate reduces cross-domain leakage from 27.0% to 3.5% and jailbreak ASR from 16.8% to 4.4%, while LoCoMo overall score improves from 38.9 to 40.8.The introduction also reports generalization to Gemini-3-Pro and Claude-Sonnet-4.6.Source: §1 · p. 2
  • The paper frames memory-induced failures as a trustworthiness problem and reports a systematic study across agentic memory frameworks and a real-world, tool-empowered personal agent.The reported threat categories include cross-domain leakage, sycophancy, tool-call drift, and memory-induced jailbreaks.Source: §1 · p. 2

2. Background

Personal agents use persistent, user-specific state to sustain personalization across sessions, but similarity-based memory retrieval can admit contextually inappropriate information. The section formalizes this gap as a failure of contextual admissibility and illustrates its forms in personal-agent tasks.

2.1. Personal Agents

  • Persistent state, including memories and tool-use histories, shapes personal agents across sessions and blurs the boundary between trusted execution and historical user input.OpenClaw dynamically loads state files into the model context, and agents can update them through continuing interactions.Source: §2.1 · p. 3

2.2. Memory in Personal Agents

  • Memory systems commonly embed queries and stored memories, rank them by semantic similarity, and inject top matches, prioritizing recall and personalization over contextual appropriateness.Implementations include text logs, profiles, vector databases, graph memories, and structured objects.Source: §2.2 · p. 3

2.3. Memory-Induced Trustworthiness Problems

  • Memory-induced trustworthiness failures arise when persistent memory that is irrelevant, stale, or unsafe inappropriately biases behavior, unlike hallucinations or single-turn prompt injections.The influence originates in the agent’s own trusted, persistent state.Source: §2.3 · p. 3

2.3.1. Memory Search Formulation.

  • Standard retrieval selects top-k memories by cosine similarity and conditions the response on them, although semantic proximity does not guarantee contextual admissibility.The paper formalizes failure as a semantically retrieved memory that is contextually inadmissible and can steer the output distribution.Source: §2.3.1 · pp. 3–4

2.3.2. Examples of Memory-induced Trustworthiness Failures.

  • Examples include cross-domain leakage and sycophancy, as well as tool-call drift and safety degradation when personal context affects unrelated actions or harmful-query interpretation.The examples include private medical information influencing workplace email, personal traits affecting tool parameters, and benign wilderness-survival memories legitimizing a harmful fire-setting query.Source: §2.3.2 · p. 4

3. Motivating Study: Real-world Assessment

The study examines memory-induced risks in agentic memory frameworks and a real-world personal agent, under benign and malicious user settings. It reports that persistent memory can affect responses, tool parameters, and harmful-query interpretation across these settings.

3. Motivating Study: Real-world Assessment

  • The assessment spans A-Mem, Mem0, MemOS, and OpenClaw, addressing a gap in prior work on dynamic retrieval and fully integrated personal-agent settings.OpenClaw provides persistent state and tool-use capabilities; prior evaluations either appended memories manually or lacked validation on fully integrated systems.Source: §3 · pp. 4–5

3.1. Threat Model

  • The threat model allows users to influence persistent memory through ordinary conversation, either unintentionally with truthful information or deliberately with benign-looking context for later harmful requests.Users cannot directly modify model weights, system prompts, retrieval code, tool schemas, or external environment states.Source: §3.1 · p. 5

3.2. Experimental Settings

  • Evaluations use memory construction followed by task execution across four memory-enabled settings, multiple supported backbones, and metrics for leakage, sycophancy, tool-call drift, and jailbreaks.A memory-free baseline is included where applicable; jailbreak ASR measures harmful queries receiving policy-violating assistance rather than refusal.Source: §3.2 · pp. 5–6

3.3. Problem I: Misalignment Feedback from Over-Personalization

  • Average sycophancy FR rises from 19.3% memory-free to roughly 37.5%–44.9% with memory, while unrelated memories also influence responses and tool-call parameters.Personal traits such as being fast, concise, risk-tolerant, or cost-conscious may affect safety-critical settings including validation, logging, rollback, and approval gates.Source: §3.3 · p. 6

3.4. Problem II: Safety Vulnerabilities from User-Manipulable Memory

  • Memory-enabled settings raise average jailbreak ASR from 3.9% memory-free to approximately 18.2%–24.6%, as planted context can make harmful requests appear justified.The attack exploits persistent memory to influence intent interpretation without requiring an explicit jailbreak in the current query.Source: §3.4 · p. 7

4. Defense Methodology: MemGate

MemGate turns memory retrieval into task-conditioned admission by gating memory representations against the current query. Its alignment objective preserves useful memories while suppressing conflicting ones, and the theoretical analysis bounds beneficial-memory score changes and characterizes unsafe-memory suppression.

4.1. MemGate Architecture

  • MemGate applies a continuous, query-conditioned mask to memory representations, attenuating inadmissible dimensions while preserving task-relevant evidence.The gate is a fine-grained alternative to discarding an entire memory with a binary classifier.Source: §4.1 · p. 7

4.2. MemGate Alignment

  • Gated cosine similarity projects each memory into a query-conditioned subspace, and MemGate reranks only the small candidate set returned by the original similarity retriever.The resulting selected memories are admitted to the prompt without changing the backbone LLM or rewriting the memory database.Source: §4.2 · p. 8
  • DPO alignment favors admissible over conflicting memories while penalizing mask deficit only for positive memories, preserving useful representations without constraining negative masks.The objective compares gated retrieval with an ungated reference policy.Source: §4.2 · p. 8

4.3. Theoretical Analysis

  • Theorem 1 bounds how far gating can move a beneficial memory’s similarity score from its ungated reference when its positive-mask deficit is minimized.The bound requires a nonzero query and a gated aligned memory with norm at least γ > 0.Source: §4.3 · p. 8
  • When DPO loss is below log 2, the analysis requires gate perturbation; preserving the positive mask makes the negative gate absorb most of it, suppressing unsafe dimensions.Theorem 2 assumes a smooth loss and a stationary point satisfying the stated loss condition.Source: §4.3 · p. 9

5. Experiments

The experiments evaluate MemGate’s trustworthiness, memory utility, runtime cost, and sensitivity to key design choices. Across these settings, it reduces memory-induced failures while preserving or improving utility, with modest retrieval overhead.

5.1. Experimental Setup

  • MemGate uses a gate on frozen 384-dimensional sentence embeddings, trained from scratch on 1,640 synthetic preference pairs constructed from abstract risk categories rather than benchmark instances.The gate takes query, memory, and element-wise product embeddings as input and masks the memory embedding.Source: §5.1 · p. 9

5.2. Trustworthy Evaluation

  • On GPT-4o-mini, MemGate reduces cross-domain failure rates from roughly 24.5%–27.0% to 3.0%–4.0% and brings jailbreak ASR to 3.7%–4.4%, near the 2.5% memory-free baseline.On Claude-Sonnet-4.6, tool-call drift falls from 77.1%–91.4% to 25.7%–28.6%, close to the 22.9% memory-free rate.Source: Figure 4 · p. 10, §5.2 · p. 10
  • With 120 queries and benign and malicious memories, t-SNE visualizations show gating separates malicious memories from queries while benign memories remain nearby.The authors interpret this as a task-aware subspace that weakens misleading associations without collapsing useful semantic neighborhoods.Source: §5.2 · p. 10, Figure 5 · p. 10

5.3. Utility Evaluation

  • On LoCoMo, GPT-4o-mini Mem0’s overall score rises from 42.9 to 44.5, and OpenClaw’s rises from 38.9 to 40.8.MemGate also consistently improves MemoryAgentBench overall scores across memory agents.Source: §5.3 · p. 11, Table 4 · p. 11
  • On GPT-4o-mini, PersistBench Beneficial FR falls from 14.3 to 6.7 for OpenClaw and from 8.1 to 3.5 for MemOS.PreFEval and PersonaMem accuracy improve in most settings, supporting the finding that MemGate retains access to helpful user preferences.Source: §5.3 · p. 11, Table 6 · p. 12

5.4. Efficiency

  • For OpenClaw, retrieval latency rises from 1.47s to 1.59s at P50 and from 2.56s to 2.75s at P90, while LoCoMo utility is preserved or slightly improved.MemGate adds no LLM calls or iterative reasoning; its cost comes mainly from computing the query–memory gate and reranking memories.Source: Table 5 · p. 11, §5.4 · pp. 11–12

5.5. Ablation Studies

  • A smaller λ suppresses risk more aggressively but can remove useful evidence; larger λ and top-k can retain more noisy or unsafe memories, so their settings trade utility against trustworthiness.The main setting uses λ = 0.1; a moderate top-k suits utility-oriented tasks, while a smaller top-k is preferable for safety-sensitive settings.Source: §5.5 · pp. 12–13, Figure 6 · p. 12, Figure 7 · p. 13
  • Across all evaluated encoders, MemGate reduces cross-domain leakage, tool-call drift, and jailbreak ASR while improving LoCoMo and PreFEval; higher-dimensional encoders add incremental gains at greater cost.Replacing all-MiniLM-L6-v2 with text-embedding-3-small reduces cross-domain FR from 3.5% to 3.0% and increases checkpoint size from 35.1MB to 67.1MB.Source: Table 7 · p. 13, §5.5 · p. 13

6. Conclusion

The conclusion frames similarity-based memory retrieval as a trustworthiness risk when semantically related memories are contextually inappropriate. It presents MemGate as a modular gate that reduces such failures while preserving utility.

6. Conclusion

  • Similarity-based retrieval can produce cross-domain leakage, sycophancy, tool-call drift, and memory-induced jailbreaks when agents treat memory as trusted context without deciding whether it fits the current task.The conclusion identifies contextual appropriateness, not recall alone, as the central issue.Source: §6 · p. 13
  • MemGate re-ranks candidate memories with a lightweight query-conditioned gate before they enter the LLM context, without modifying the backbone or adding LLM calls.It applies a task-conditioned mask to memory embeddings and uses preference-aligned training objectives.Source: §6 · p. 13
  • The reported results show reduced memory-induced failures while preserving, and sometimes improving, long-term memory utility.The conclusion states that trustworthy personalization requires deciding when remembered context is legitimate evidence for the current task.Source: §6 · p. 13

Appendix A. Proof of Theorem 1

The proof bounds how gating changes similarity for aligned, useful memories. It connects the positive-mask preservation penalty to a smaller similarity drift.

Appendix A. Proof of Theorem 1

  • The proof represents the gated positive memory as its original embedding multiplied element-wise by the positive gate.It first bounds the representation perturbation, then relates that perturbation to cosine-similarity change.Source: Appendix A · p. 15
  • For a fixed nonzero query, the cosine function’s gradient is bounded by the inverse norm of the memory vector along the segment considered.The proof uses the gate’s coordinate-wise range to establish a lower bound on the segment vectors’ norms.Source: Appendix A · p. 15
  • The resulting bound shows that a smaller positive-mask deficit yields smaller similarity drift for aligned memories, matching the regularizer’s role in discouraging suppression of useful evidence.The proof links the representation perturbation bound to the cosine-similarity change using a Lipschitz bound and the mean value theorem.Source: Appendix A · p. 15

Appendix B. Proof of Theorem 2

The proof starts from the ungated reference and uses smoothness and stationarity to derive a lower bound on total representation perturbation. It then gives a specialized bound when the positive memory is perfectly anchored.

Appendix B. Proof of Theorem 2

  • At the ungated reference initialization, the gated cosine similarities match the reference scores, making the relative margin in the DPO objective zero and the initial loss log 2.The supplied proof passage states the matching and zero-margin steps; it then gives the initial loss expression.Source: Appendix B · p. 15
  • Using L-smoothness and the theorem’s stationarity condition, the proof bounds the gate’s total representation perturbation in terms of the loss bound ϵ.It substitutes the ungated and converged gate values into the smoothness inequality before rearranging.Source: Appendix B · p. 15
  • When the positive-memory preservation penalty perfectly anchors the aligned memory, the subtractive term vanishes and yields the stated unsafe-memory suppression bound.This is the specialized case with g+ = 1^d.Source: Appendix B · p. 15

C.1. Utility Benchmark Details

Utility evaluation covers both general long-term memory capability and personalization: whether agents retain and reason over interaction histories, and whether they use beneficial user memories when relevant.

C.1. Utility Benchmark Details

  • Utility is assessed through general long-term memory capability and personalization, measuring history-based retrieval and reasoning alongside relevant use of user memories.The personalization benchmarks treat failure to retrieve or apply a helpful memory as reduced personalization quality.Source: §C.1 · pp. 15–16
  • LoCoMo [33] tests memory across multi-session dialogues through long-term question answering, event summarization, and dialogue generation, reporting overall F1.It evaluates recovery and use of information from extended conversational histories.Source: §C.1 · p. 16
  • MemoryAgentBench [34] evaluates incremental multi-turn memory abilities, while personalization benchmarks test whether user-specific memories help when relevant.MemoryAgentBench covers accurate retrieval, long-range understanding, and selective forgetting; its reported measure is overall accuracy.Source: §C.1 · p. 16
  • PreFEval [35] tests whether models infer, remember, and follow user preferences in long-context conversations, reporting accuracy on curated preference-query pairs.Its pairs span diverse topics.Source: §C.1 · p. 16
  • PersonaMem [36] evaluates dynamic profiling and response selection across multi-session interactions, reporting accuracy on matches to evolving user personas and preferences.Source: §C.1 · p. 16
  • PersistBench utility evaluation uses the beneficial-memory subset, reporting Beneficial FR for failures to retrieve or apply necessary memory.This subset contains cases where long-term memory is genuinely relevant and should be used.Source: §C.1 · p. 16

Related papers

Loading 2606.06054v1…