PaperlayerView exact revision on arXiv

Source-linked AI summary

MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery

Enze Ma, Yufan Zhou, Wei-Chieh Huang, Jie Yang, Huanhuan Ma, Zixuan Wang, Chengze Li, Chunyu Miao, Philip S. Yu, Zhen Wang

arXiv:2606.24595v1cs.CL

TL;DR

Existing evaluations mostly infer long-term memory quality from downstream behavior rather than auditing the user state retained. MEMPROBE directly reconstructs hidden user state from agent memory and finds that near-saturated task success coexists with moderate, retrieval-sensitive recovery.

  • Problem

    Long-term memory is mainly evaluated through downstream answers, personalization, and task success, leaving the retained memory artifact itself insufficiently audited.

  • Method

    MEMPROBE uses simulated users with hidden taxonomy-anchored state banks and leak-controlled tasks, then scores state recovery from full-store and top-k memory access.

  • Results

    Task completion nearly saturates, but recovery is moderate and falls under retrieval (B=0.473–0.540 versus B=0.611–0.624 under dump_all).

  • Takeaways & Limitations

    Successful assistance does not entail recoverable memory, making recovery a direct axis for evaluating and improving long-term memory artifacts.

  • Takeaways & Limitations

    MEMPROBE’s conclusions are bounded by its controlled synthetic setting, which enables fixed hidden states and scalable evaluation but may not represent real users fully.

Abstract

from arXiv · show

Long-term memory promises LLM agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that interaction forms. In practice, however, this memory is evaluated mostly through downstream behavior, such as later answers, personalization quality, or task success, which tests that understanding only indirectly and leaves the memory artifact itself largely unaudited. We argue that long-term memory should instead be evaluated as an auditable post-interaction artifact: after ordinary assistance, what structured user state can be reconstructed from the memory the agent leaves behind? We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users, each carrying a hidden, taxonomy-anchored user-state bank, across a trajectory of leak-controlled tasks, after which that bank is reconstructed from the agent's resulting memory under both full-store and top-k access. Built on synthetic ground truth for efficient, scalable measurement, MEMPROBE spans 50 simulated users with 31 hidden dimensions each (1,550 recovery targets) and tests 5 representative memory systems. Testing state-of-the-art memory agents, we find that successful assistance and recoverable memory behave as distinct capabilities. Task completion nearly saturates, even for a memoryless baseline, while category-balanced recovery stays moderate (about 0.6) and drops further under top-k retrieval. MEMPROBE is the first benchmark to study memory recovery directly, reconstructing the user state a system retains and scoring it against ground truth. We see recovery as a concrete objective for future memory agents to optimize, and MEMPROBE as a step toward an environment where agents are trained to remember their users, growing more faithful the longer they know them.

1 Introduction

MEMPROBE reframes long-term agent memory as an auditable post-interaction artifact, evaluating which hidden user state can be recovered from memory rather than judging memory only through downstream behavior. Its benchmark and findings identify recovery, retrieval-aware consolidation, and episodic structure as central directions for improving long-term memory.

  • Motivation: Downstream task success and answer quality are weak diagnostics because agents can complete nearly every task even without memory.Existing evaluations primarily measure later answers, follow-up task success, personalization, recall, updating, and forgetting through behavior.
  • Benchmark: MEMPROBE audits memory by reconstructing hidden, taxonomy-anchored user-state targets from the post-interaction memory artifact.Its hidden user-state bank spans skills, knowledge, episodic events, a self-model, and assistance preferences, and serves as ground truth.
  • Benchmark: 5 representative state-of-the-art memory systems are evaluated across 50 simulated users and 1,550 hidden user-state targets.The benchmark audits what each system preserves about the simulated users.
  • Findings: Recovery provides a direct measurable axis for distinguishing how much of the user a system preserves, unlike behavioral scores that can saturate without memory.The paper positions recovery as the quantity long-term memory was meant to improve.
  • Findings: Retrieval-aware consolidation is a major bottleneck: systems capture raw interaction but seldom distill it into compact, durable, reusable user-state claims.The proposed lever is a write policy, which capacity and retrieval tuning alone do not replace.
  • Findings: Episodic, relational memory remains an open frontier because one-off experiences require binding events to the contexts and consequences that give them meaning.Stable preferences are described as largely handled, while aggregate recovery can hide this episodic difficulty.

MEMPROBE · 2 Related Work

MEMPROBE frames long-term agent memory as an auditable recovery problem: hidden user-state dimensions are reconstructed from memory after leak-controlled interactions. Its related work spans agent-memory architectures, long-horizon retrieval and retention benchmarks, and personalization evaluation.

  • 2 Related Work: Agent-memory systems transform interaction history into persistent state through natural-language experiences, reflections, reusable skill libraries, context tiers, archival tiers, or persistent user memories.The supplied passage identifies these as differing approaches to persistent-state construction.
  • 2 Related Work: Long-memory benchmarks evaluate recall or use of information from previous interactions, including extended multi-session conversational recall and LongMemEval’s multiple reasoning dimensions.The listed LongMemEval dimensions include event summarization, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
  • 2 Related Work: Personalization benchmarks test whether assistants adapt outputs to users’ preferences, traits, or histories in task-oriented and user-centered dialogue settings.This work extends evaluation beyond generic interaction performance toward user-conditioned behavior.
  • 2 Related Work: LaMP and LongLaMP evaluate personalized language modeling from user profiles and histories, including personalized long-form generation.The passage presents LongLaMP as an extension of LaMP’s setting.
  • MEMPROBE: MEMPROBE uses a user simulator with a hidden, taxonomy-anchored bank of user-state dimensions interacting with a memory-equipped agent over leak-controlled tasks.The agent updates its memory throughout the interaction trajectory.
  • MEMPROBE: After interaction, MEMPROBE reconstructs each hidden dimension from the resulting memory store using full-store dump_all and top-k retrieve access.Failure attribution localizes cases with low recovery.

3 The MEMPROBE Protocol

MEMPROBE audits long-term agent memory by tracing hidden user state through exposure, collection, and recovery, then localizing information loss. It uses simulated users, leak-controlled tasks, controlled rollouts, dual access modes, and per-dimension judged reconstruction.

  • 3.1 Formalizing recovery: A hidden 31-dimensional user state is exposed as task-dependent evidence, transformed by memory operations into a persistent store, and reconstructed through a read operator and slot-filling reader.The agent never sees the hidden state directly; each step can lose information, enabling failure localization.
  • 3.4 Recovery scoring: MEMPROBE compares full-store dump_all access with top-k retrieve access, distinguishing failures in memory contents from failures in search retrieval.The dump operator returns the entire store, whereas the retrieval operator returns the system’s top-k search results for a fixed query.
  • 3.2 Simulated users and tasks: Synthetic users provide fixed, structured ground truth while enabling controlled disclosure and scalable evaluation without privacy or annotation costs.Each user also receives a coherent persona so preferences, knowledge, and voice remain stable across many tasks.
  • 3.2 Simulated users and tasks: Leak-controlled tasks request adjacent ordinary help without naming target values, and a blind anti-fishing critic rejects tasks whose statements already reveal answers.These mechanisms create natural openings for disclosure while keeping the hidden bank out of the initial request.
  • 3.3 Simulation rollout: The rollout holds prompts, dialogue protocol, assistant scaffold, and task set constant across systems, changing only memory modules while preserving memory across each user’s T tasks.Memory resets before the next user, and the agent never receives the hidden bank.
  • 3.4 Recovery scoring: Each reconstructed target receives a separate LLM judge score on {0, 0.25, 0.5, 0.75, 1.0}, with category aggregation preventing larger categories from dominating.Recovery is measured per dimension using a slot-filling reader against the hidden target.
  • 3.5 Failure attribution: Failure attribution labels scores below 0.75 through task-design and disclosure checks so task or simulator limitations are not counted as memory faults.Cases scoring at least 0.75 are labeled ok and skip attribution.

4 Experiments and Results

The experiments show that immediate task success and recoverable long-term user state are distinct capabilities. Recovery depends on preserving compact user-state claims and making them reachable through retrieval, especially for episodic information.

  • Immediate assistance versus recovery: 99.87%–99.94% task completion and 4.58–4.66/5 preference scores show near-saturated assistance, including for the nomem baseline.Episodes averaged only 2.4–2.5 turns, so local task success does not discriminate memory quality.
  • Memory recovery: Under dump_all, longctx_full leads with B=0.624; under retrieve, amem leads with B=0.540 while longctx_full falls to B=0.503 and mem0 to B=0.473.Raw evidence preservation is strongest with full access, whereas evolved notes are more compatible with operational retrieval.
  • Failure attribution: 43–45 task-design failures and 44–55 agent-elicitation plus simulator-strictness failures leave many unrecovered targets after disclosure opportunities arise.Disclosed-target recovery for non-memt systems then drops roughly 8–18 percentage points under top-k access, showing that stored evidence may remain unreachable.
  • Category differences: ASSISTANCE PREFERENCE is easiest, while one-off, time-anchored episodes are harder because recovery requires linking events to their stated consequences.The same category ordering appears under dump_all and retrieve, indicating that the gap is not merely a retrieval artifact.
  • Audited cases and limitations: Audited cases show systems preserving local interaction traces without abstracting the supported user-state claim, with additional boundary conditions including slot-label polarity artifacts and typed-collection wins.Figure 3 illustrates recovery of a hidden financial-literacy dimension from follow-up evidence about risk, liquidity, and avoidable loss.

5 Conclusion · Appendix

MEMPROBE reframes long-term agent memory as an auditable post-interaction artifact by measuring hidden user-state recovery from retained memory. Across five memory settings, near-saturated task completion coexists with moderate full-store recovery and substantially lower top-k recovery, showing that successful assistance does not entail recoverable memory.

  • 5 Conclusion: MEMPROBE evaluates long-term memory as an auditable post-interaction artifact.It asks what hidden user state can be reconstructed from the memory an agent leaves behind.
  • 5 Conclusion: The benchmark measures hidden user-state recovery after ordinary assistance.Recovery is assessed from the agent's retained memory rather than inferred only from downstream behavior.
  • 5 Conclusion: Across five memory settings, task completion is near-saturated.This result is considered alongside recovery under different memory-access conditions.
  • 5 Conclusion: Full-store recovery remains only moderate despite near-saturated task completion.The result indicates that completing tasks and retaining recoverable user state can diverge.
  • 5 Conclusion: Top-k recovery is substantially lower than full-store recovery.The comparison concerns the same memory systems evaluated under different access modes.
  • 5 Conclusion: System rankings change between full-store and top-k access modes.Thus, conclusions about memory systems depend partly on how retained memory is accessed.

A Limitations · B Broader Impact

MEMPROBE’s synthetic, everyday-task setting and rubric-based judging make memory recovery measurable but limit claims about real-world users, adversarial difficulty, and judge reliability. Its findings motivate auditable memory design while highlighting incomplete user-conditioned skills, learned-management limits, and privacy risks requiring consent and controls.

  • A Limitations: Synthetic users and generated tasks enable known-ground-truth recovery measurement, leakage control, and auditing of whether disclosed evidence becomes recoverable memory.The setting models ordinary assistance in which longer-term signals arise through constraints, corrections, preferences, prior experiences, and repeated behavior.
  • A Limitations: Recovery scores and attribution labels depend on rubric-based model judging, whose reliability requires separate auditing and cautious interpretation.The recovery judge sees the dimension, explanation, prediction, and ground truth simultaneously without blinding.
  • A Limitations: Task completion nearly saturates across systems because MEMPROBE uses realistic everyday assistance without adversarial or highly specialized cases.Therefore, the benchmark is not a stress test of general task-solving ability; it focuses on whether successful interactions leave recoverable user information.
  • A Limitations: The benchmark does not directly evaluate user-conditioned assistance skills, including learning useful unknowns, productive prompts, and feedback trajectories for a particular user.These skill-like memories extend beyond user facts and preferences, and failures can occur before writing when the assistant elicits insufficient evidence.
  • B Broader Impact: Long-term memory should be evaluated as an auditable post-interaction artifact, not only as a write–retrieve buffer or source of downstream task gains.The broader design implication is that “dynamic memory” alone does not identify the missing capability.
  • B Broader Impact: memt’s largest store and learned memory-operation policy show that learned memory management does not automatically produce recoverable user state.Learned or RL-based policies remain promising because writing, updating, deleting, and retrieving are sequential, context-dependent, and coupled to future utility.
  • B Broader Impact: Auditing recoverable user state can expose over-accumulation, retrieval failures, and auditability problems, but real-user deployment raises privacy, profiling, and unwanted-personalization risks.Safer deployment requires consent, data minimization, deletion controls, and inspection.

C Benchmark Card and Release Artifacts · D Generation Pipeline Details

MEMPROBE is released as an auditable benchmark for controlled comparison of structured user-state recovery, with synthetic, taxonomy-anchored targets and explicit usage limitations. Its generation pipeline conditions on DeepPersona profiles, creates 31-dimensional user banks and tasks, and applies anti-fishing and validation procedures.

  • C Benchmark Card and Release Artifacts: MEMPROBE is intended for controlled comparison of LLM-agent memory systems on post-interaction structured user-state recovery.It supports ablations of write policy, retrieval interface, and memory footprint under fixed task and simulator conditions, rather than general-assistant, personalization, or real-human evaluation.
  • C Benchmark Card and Release Artifacts: The benchmark’s structured banks are synthetic, taxonomy-anchored targets, and its U.S.-only DeepPersona pool is not representative of multilingual or cross-cultural assistant populations.The banks are not psychometric assessments, personalization-deployment recommendations, or evidence of safety in adversarial settings.
  • C Benchmark Card and Release Artifacts: Benchmark instances and full evaluation traces are released as JSON, with aggregate metrics recomputable from per-user records using released analysis scripts.Judge, oracle, disclosure, and slot-fill calls use temperature=0.0; outputs are typically stable but not bit-deterministic.
  • C Benchmark Card and Release Artifacts: Table 5 reports cumulative gpt-5.4-mini token usage across all 50-user benchmark sessions as non-auditable aggregate reference data, not recomputable release artifacts.The totals cover applicable episode-loop, recovery, and attribution calls; memt’s locally served Mem-T-4B policy is excluded.
  • C Benchmark Card and Release Artifacts: Table 6 separately reports one-time offline token usage for constructing the released 50-user pool as a non-auditable aggregate run-log record.This covers persona-profile, hidden-bank, task, anti-fishing critic, and editor-revision calls; all calls use gpt-5.4-mini.
  • C Benchmark Card and Release Artifacts: Scores aggregate over 50 users × 31 dims, while every judge prediction and rationale is released to keep individual decisions auditable.This mitigates per-call variability even at temperature=0.0.
  • D Generation Pipeline Details: For each user, generation first samples a DeepPersona base profile that conditions all subsequent generation steps.The pipeline then generates category-specific hidden-bank entries and assistance tasks.
  • D Generation Pipeline Details: The generator selects dimensions from a fixed taxonomy-anchored pool, creates user-specific target phrases with persona-grounded explanations, and produces one task per user–dimension.A blind anti-fishing critic rejects or revises answer-leaking tasks, while bank validation enforces verbatim taxonomy names and retries failures; the released package contains 31 dims per user × 50 users = 1,550 recovery targets.

E Prompt Templates

Appendix E specifies prompt templates for constructing hidden memory banks, generating and debiasing exposure tasks, and simulating persona-conditioned users. Together, these prompts enforce structured outputs, neutral task wording, and realistic profile-consistent responses.

  • E Prompt Templates: Prompt templates normalize typography while preserving prompt logic and fill placeholders at runtime from corresponding Python modules.Outputs are constrained to JSON in the downstream prompts.
  • E.1 Bank generation prompt: Bank generation picks exactly the requested number of verbatim pool dimensions where the user has meaningful, non-generic stances.It favors lateral diversity and requires user-specific short labels plus grounded 2-4 sentence explanations, including gaps and weaknesses.
  • E.2 Task generation prompt: Task generation creates realistic help, planning, recommendation, or problem-solving requests whose target aspect emerges from the user’s reply rather than task wording.Tasks must not directly probe users or copy, paraphrase, name, or otherwise telegraph the target memory entry.
  • E.3 Anti-fishing critic prompt: The blind critic labels tasks neutral when varied users could plausibly send them and fishing when wording makes the sender’s trait predictable.Fishing includes naming, paraphrasing, soliciting self-description, pre-encoding content, or using adjectives matching the expected stance.
  • E.4 Anti-fishing editor prompt: The editor rewrites fishing tasks into neutral situations while preserving the general scene and trigger that invite the target trait to surface.It removes trait-related vocabulary, self-description requests, and structures that pre-encode target content.
  • E.5 User simulator prompt: The user simulator roleplays each synthetic profile rather than an ideal helpful user, conditioning responses on its base profile and memory bank.It models speech habits, emotional tendencies, frustration or delight, background-linked language, and proactive recall of closely parallel past events.
  • E.5 User simulator prompt: The simulator is instructed to volunteer relevant autobiographical memories when the current scenario closely parallels a specific past event.Recall is brief and should arise naturally in speech without waiting for the agent to ask.

F Scoring and Attribution Protocol Details

This section defines category-balanced recovery scoring, contrasts full-store and top-k access, and explains thresholded failure attribution. It also specifies the deliberately query-agnostic retrieval setup and low-noise evaluation procedure.

  • Recovery scoring: Recovery score B averages 31 dimension-level scores within five memory categories, then averages the category means for each user.The slot-filler predicts a label and reason for each (user, dimension), and a judge scores the label against the ground-truth short on a five-level rubric.
  • Access modes: In dump_all, reconstruction uses the full JSON-serialized memory store, while retrieve supplies only search(query, k=5) results.The full-store representation preserves each system’s native memory fields; retrieval uses an axis-only query followed by a fixed category guide.
  • Access modes: The retrieval query contains no user-specific content, intentionally stressing the read interface while leaving query construction out of scope.The query asks what best describes the user’s dimension and is followed by a fixed guide covering skill, knowledge, episodic, self, and preference.
  • Attribution protocol: Attribution counts are not directly comparable across dump_all and retrieve because retrieval can push cases below 0.75 into the Stage 0–2 pipeline.Newly attributed cases may receive labels such as agent_elicitation_failure or simulator_too_strict.
  • Attribution protocol: 0.75 marks mostly correct reconstruction with minor missing detail or phrasing mismatch, and scores at or above it short-circuit attribution as ok.Lower scores can reflect missing the central target value, drifting to a neighboring abstraction, or returning unknown; confidently correct disclosed dimensions avoid later judge calls.

G Human Validation of Pipeline Judgments

A small human audit supports the simulator’s satisfaction judgments and finds substantial, though imperfect, agreement for failure-attribution labels. The audit validates using end_reason as the basis for A while indicating attribution judgments retain some disagreement.

  • 96% of simulator satisfied judgments were judged reasonable by humans in 48/50 cases, supporting end_reason as the basis of A.The audit sampled both automated simulator and attribution signals.
  • 75.0% of failure-attribution labels agreed with human re-labels in 90/120 cases.The remaining disagreements were described as limited, but the supplied passage is truncated before further characterization.
  • The audit covered stratified samples of the simulator and attribution judgments underlying Table 2’s headline metrics and Table 4’s attribution counts.

H Additional Results

Additional analyses localize the recovery gap primarily to memory writing and retrieval rather than disclosure. Across context-fit systems, full-store dump performance exceeds retrieve performance by positive, user-level margins.

  • Per-category attribution: memory_failure is the dominant non-ok attribution for every system and recovery category, placing the headline gap downstream of disclosure.The attribution breakdown covers five recovery categories in Table 13.
  • Per-category attribution: EPISODIC MEMORY contributes many agtE cases and taskD cases, while SELF MODEL contributes many taskD cases.These patterns align with the difficulty of eliciting time-anchored autobiographical events and abstention behavior.
  • Per-user dump −retrieve gap: +0.140 mean is mem0’s dump −retrieve B gap, compared with +0.121 for longctx_full and +0.071 for amem.For mem0, p50=+0.149 and range [−0.044, +0.319]; for longctx_full, p50=+ 0.129 and range [−0.001, +0.256]; for amem, p50=+0.075 and range [−0.026, +0.174].

I Per-system Retrieval and Budget Probes

Per-system probes show substantial differences in memory-store size and retrieval behavior. memt stores the most and longest items, causing dump_all to exceed the slot-fill model’s input limit for median users, while retrieve-mode ranking favors amem, longctx_full, and mem0.

  • Memory footprint distribution: p50=458 items and p50=1028 chars/item make memt the largest and longest-item system; its p90 reaches approximately 768 K raw content characters.By comparison, mem0 writes p50=190 items of p50=206 characters/item, yielding the smallest total store among context-fit systems.
  • Why memt’s dump_all overflows: Approximately 458 × 1028 ≈470 K content characters at memt’s median, plus collection metadata, exceeds the slot-fill model’s input limit.The dump_all probe serializes the entire store as a JSON array; amem, longctx_full, and mem0 fit even for p90 users.
  • Retrieve interface budgets: k=5 is used for headline retrieve numbers; amem, longctx_full, and mem0 use a single embedding-kNN pool, while memt uses a ReAct search loop with max_tool_steps=6.memt searches facts, experiences, turns, summaries, and personas, then returns the top-5 after de-duplication.
  • Retrieve interface budgets: amem > longctx_full > mem0 > memt is the retrieve-mode ranking under the present k=5 headline.Sensitivity sweeps for k ∈{1, 3, 5, 10} remain a planned robustness probe.

J Failure case gallery · K Additional Case-Study Figures

The failure gallery audits recurring recovery mechanisms through trace-to-target abstraction, write-side abstraction, and query–content mismatch, while also exposing scoring, retrieval, and task-budget boundary conditions. Cases show that disclosed information can still be abstracted incorrectly, retrieved poorly, or scored inconsistently.

  • J Failure case gallery: The audited cases package each user–dimension diagnosis with its target, task, predictions, retrieved memories, and judge reasoning.This structure traces diagnoses back to released artifacts.
  • J Failure case gallery: Cases 1, 2, and 4 instantiate trace-to-target abstraction, write-side abstraction, and query–content mismatch, respectively.Cases 3, 5, and 6 are boundary conditions involving scoring polarity, localized retrieval, and task-completion limits.
  • J Failure case gallery: Every non-memt system scores below 0.5 in dump_all for Case 1, where episodic recovery misses the realization that asking questions beats pretending to understand.The task discloses the realization indirectly through acceptance of a heuristic permitting not knowing yet.
  • J Failure case gallery: Cross-system divergence reaches scores ≥0.75 versus 0.0 in Case 2 despite comparable disclosure, because some memories preserve practical fairness while others record unknown.amem and longctx_full score 0.75/0.75, whereas mem0 and memt score 0/0.
  • J Failure case gallery: Case 3 shows a scoring artifact: identical disclosed pushback behavior receives opposite judgments when slot labels verbalize tolerance polarity differently.amem scores 0/0.75, while longctx_full scores 0.75/0.25 for the same underlying preference.
  • J Failure case gallery: In Case 4, every non-memt system scores ≥0.75 in dump_all but below 0.5 in retrieve, demonstrating abstract-query versus concrete-evidence mismatch.amem, longctx_full, and mem0 score 0.75/0, while memt scores 0/0.75.
  • J Failure case gallery: memt scores 0/0.75 in Case 5 because typed persona retrieval surfaces an independence-and-privacy abstraction that flat kNN systems miss.Across the run, 20 dimensions show this memt-only retrieve-win pattern, while the larger multi-collection store complicates dump_all serialization.
  • J Failure case gallery: Only 6 of 7,750 episodes (∼0.08%) fail within 25 turns, all ending at max_turns; repeated rewrite failures indicate a simulator/task-budget artifact, not memory recovery.None is a ground-truth mismatch, and five of six failures are short rewriting tasks seeking less generic wording.
Loading 2606.24595v1…