PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents

Yeonjun In, Wonjoong Kim, Sangwu Park, Kanghoon Yoon, Chanyoung Park

arXiv:2605.25535v1cs.AI

TL;DR

Personalized memory systems must account for users’ different long-horizon and transient contexts rather than applying universal storage policies. This paper introduces PerMem-Bench and session-level storage gating, finding substantial retention gains with perfect gating while accurate gating remains challenging.

  • Problem

    Existing memory systems use universal criteria despite users differing in which contexts require long-horizon preservation, creating a need to evaluate personalized memory policies.

  • Method

    The paper introduces PerMem-Bench for personalized-memory evaluation and studies session-level gating that skips memory operations for transient sessions.

  • Results

    Perfect knowledge of user agent-use patterns yields substantial retention gains, whereas current gating baselines achieve only incremental practical gains.

  • Takeaways & Limitations

    Personalization can improve memory retention under fixed budgets, but accurate session-level gating remains an open challenge.

  • Takeaways & Limitations

    Current gating methods remain insufficient to fully realize personalization’s potential gains.

Abstract

from arXiv · show

Existing large language model (LLM) based memory systems apply universal, static policies that overlook a fundamental reality: the contexts that are worth storing in memory are different across users. This misalignment wastes limited memory budget on transient interactions while failing to preserve critical context for long horizon tasks. To address this gap, we investigate an underexplored question: can LLM based memory systems learn personalized memory policies? We introduce PerMemBench, the first benchmark for evaluating personalized memory systems, featuring multi year, multi domain interaction histories across diverse user personas. We further present the first empirical study of memory personalization, proposing session level storage gating, a lightweight framework that selectively bypasses memory operations for transient sessions. Our study confirms that personalization yields substantial retention gains under perfect gating, yet reveals that accurate gating remains an open and critical challenge.

1 Introduction

The introduction argues that universal memory policies misallocate limited storage because users differ in which contexts support long-horizon interactions. It presents PerMem-Bench and session-level storage gating to evaluate and study personalized memory policies.

  • Motivation: Users differ in which domains and interactions require persistent memory, making worth-storing contexts user-specific rather than universal.Alice treats Recipe Advice as long-horizon and Travel Plan as transient, while Bob has the opposite pattern.
  • Motivation: Universal memory criteria waste limited budget on transient interactions while evicting essential context for long-horizon tasks.This misallocation motivates personalized policies that infer each user’s storage needs.
  • Research Gap: The paper frames learning user-specific worth-storing contexts as an underexplored question hindered by the lack of suitable personalized long-horizon dialogue benchmarks.Existing resources do not capture heterogeneous usage patterns across users and domains well enough for rigorous fine-grained evaluation.
  • Contributions: PerMem-Bench is introduced as the first benchmark for personalized memory systems, covering diverse personas and multi-year, multi-domain dialogues.Its automated pipeline profiles user patterns, builds per-user life skeletons, and synthesizes dialogue sessions with an LLM-based user simulator.
  • Contributions: Session-level storage gating identifies long-horizon versus transient sessions and skips memory operations for transient sessions as a general personalization framework.The study also establishes multiple gating baselines and reports substantial retention gains from perfect gating under a fixed budget, while current baselines remain suboptimal.

2 Related Work

Prior agent-memory research learns storage policies or evaluates factual and experiential memory, but existing approaches and benchmarks largely overlook user-specific, multi-domain storage needs. This work introduces session-level storage gating and PerMem-Bench to evaluate personalized memory across heterogeneous users and interaction histories.

  • Agent Memory Systems: Agent-memory research broadly pursues learned LLM-based policies for extracting salient information and evaluation of experiential or factual memory.Factual-memory evaluation concerns preserving critical user-centric context over long-horizon interactions.
  • Agent Memory Systems: Session-level storage gating personalizes memory by identifying each user’s worth-storing sessions and bypassing memory operations for transient sessions.The approach moves beyond universal criteria for deciding what information should be stored.
  • Evaluation of Agent Memory Systems: Existing evaluations often impose single-domain settings and overlook behavioral heterogeneity across users, limiting their realism for diverse, goal-oriented interactions.Examples include LoCoMo and HalluMem’s casual-interaction focus and PersonaMem, LongMemEval, and AmemGym’s limited treatment of user differences.
  • Evaluation of Agent Memory Systems: PerMem-Bench addresses these gaps with multi-domain interaction histories and explicit behavioral heterogeneity for evaluating personalized memory across diverse users.The benchmark is designed to represent complex, real-world usage scenarios and heterogeneous interaction patterns.

3 Benchmark Construction: PerMem-Benchs

PerMem-Benchs is built through an automated pipeline that profiles 20 users, constructs personalized life timelines, and synthesizes long-horizon dialogues. Its profiles distinguish domain participation, interaction frequency, and user-specific memory necessity, enabling memory-required and transient interactions to be generated differently.

  • Benchmark Construction: PerMem-Benchs uses three stages: user-specific agent-use profiling, life skeleton and timeline construction, and dialogue generation for 20 unique users.The sample size balances generation overhead with the costs of evaluating memory systems.
  • User-Specific Profiling: User profiles jointly encode domain participation and memory necessity as a simple framework for representing diverse personalized use patterns.The authors treat this configuration as a foundational baseline for personalized memory management.
  • User-Specific Profiling: For each persona-domain pair, the profile records participation ap,d, frequency fp,d, and memory necessity mp,d, with necessity determined by user-specific intent rather than domain identity.Participation is binary, frequency is high, mid, or low, and memory necessity is binary.
  • Life Skeleton and Timeline: Memory-required domains are organized into interconnected projects with reference memories, whereas transient domains contain unrelated independent events without project dependencies or reference memories.The number of projects and events follows the frequency metadata fp,d.
  • Dialogue Generation: A unified timeline orders per-domain events, after which a dual-simulator framework generates personalized dialogues from event-defined user context while the agent relies only on user input and internal memory.The user simulator manifests attributes such as user state and project progress, while the agent simulator lacks prior access to the skeleton.

4 Reflecting Shifts in Agent Use Profiles: PerMem-Benchd

PerMem-Benchd models evolving user agent-use profiles caused by major life events and resulting shifts in domain preferences. It constructs continuous long-horizon trajectories by appending post-shift timelines to pre-shift interaction sequences.

  • Motivation: PerMem-Benchd simulates profile shifts arising from dynamic interests and significant life events, extending the PerMem-Benchs foundation.Examples include career changes, new hobbies, and completed long-horizon projects.
  • Profile transition: An existing memory-intensive domain transitions to transient use when a long-horizon project concludes.This change is represented as mp,d = 1 to mp,d = 0 and reflects a shift toward transactional interaction.
  • Trajectory construction: The authors use gpt-5.4 to infer plausible life events, generate a post-shift life skeleton, and append its timeline to the pre-shift sequence.Dialogue generation then uses the same dual-simulator framework to produce a continuous long-horizon trajectory.

5 Data Analysis and Meta Evaluation on PerMem-Bench

PerMem-Bench exhibits highly diverse user profiles across its 20-user and 100-user analyses, covering a broad spectrum of agent behaviors. A three-stage meta-evaluation reports high quality and evaluator agreement, supporting the reliability and scalability of its automated generation pipeline.

  • User Profile Diversity: Most user pairs exhibit very low Jaccard similarity, with no identical domain-memory necessity profiles among the 20 PerMem-Bench users.The profiles treat domain-memory necessity pairs as features, where similarity 1 would indicate identical agent use patterns.
  • User Profile Diversity: Results on random 100 users from Nemotron-Persona consistently show highly diverse use profiles, indicating broad coverage of real-world agent behaviors.This scalability check extends the diversity analysis beyond the 20-user PerMem-Bench set.
  • Meta-Evaluation: Stage 1 profile plausibility achieves a 99.5% average quality score and 99.0% inter-evaluator agreement.The evaluation checks logical consistency with assigned personas, including relevance and realism.
  • Meta-Evaluation: Stage 2 life skeleton and timeline realism achieves a 100% quality score and 100% AC1, with both evaluators reaching perfect agreement.The stage evaluates project-sequence coherence, event-timeline realism, persona-appropriate reference memories, and realistic temporal progression.
  • Meta-Evaluation: Stage 3 dialogue quality scores 98.4% with 96.9% inter-evaluator agreement across 100 randomly sampled sessions, while the automated pipeline supports scaling beyond 20 users.Dialogue evaluation covers consistency with the life skeleton and seamless integration of reference memories; the pipeline requires no manual intervention.

6 Evaluation Protocol of PerMem-Bench

PerMemBench evaluates whether memory systems preserve user-tailored contexts throughout their required lifespans. It measures retention with a formally defined Memory Retention Rate and approximates costly session-level checks through sampled LLM-judged verification.

  • PerMemBench’s primary objective is assessing whether systems preserve tailored reference contexts and maintain them over time.
  • Memory Retention Rate measures how consistently each reference memory unit remains in the memory bank throughout its required lifespan.
  • User-centric states must remain retained until a relevant update occurs or the timeline concludes, while project-specific progress receives an information-type-dependent retention horizon.
  • The metric defines eligibility at the first appearance session and evaluates retention against a target horizon determined by information type.
  • An LLM-as-a-judge using gpt-5-nano checks each reference against the top-10 semantically similar memory entries and returns a binary preservation verdict.
  • Because checking every reference at every session is prohibitively expensive, the evaluation approximates the inner summation by sampling checkpoints.

7 Can Memory Systems Be Personalized?

The study introduces session-level storage gating to personalize memory policies by skipping transient sessions, and finds that structure-aware gating best approximates domain-level usage patterns. Personalization substantially improves retention under perfect gating, but imperfect gating leaves most of that potential unrealized.

  • Personalized storage gating: Session-level storage gating skips memory operations for sessions predicted to be transient, without modifying the underlying memory system.The gating module uses the session dialogue and optionally prior context to distinguish long-horizon tasks from transient interactions.
  • Gating methods: Structure-aware gating models projects and isolated sessions across interaction history, uniquely approximating domain-level usage patterns among the evaluated methods.Its structural note persists across windows and can retroactively reassign sessions to projects identified later.
  • Gating performance: 0.844 F1 is achieved by Structure-aware gating on PerMem-Benchs, while Greedy and Context-aware methods show that session-local or flat-summary signals are insufficient for domain-level inference.The benchmark evaluates gating methods along increasing levels of contextual richness, from session-local signals to explicit cross-session structure.
  • Retention: Personalization substantially improves memory retention over the Universal policy when the agent-use pattern is known exactly, with larger Oracle–Universal gains at smaller memory budgets.Oracle avoids storing transient sessions and allocates the available capacity to worth-storing contexts.
  • Limitations: Imperfect gating prevents personalization from realizing its potential: Greedy and Context-aware underperform Universal, while Structure-aware provides only marginal improvements despite higher gating accuracy.The Oracle-to-best-gating gap quantifies remaining headroom, making more accurate session-level gating the central challenge.

8 Conclusion

The paper introduces PerMem-Bench and session-level storage gating to study personalized memory systems, with results confirming substantial retention gains when user use patterns are exactly known. It identifies benchmark scale, simple profile modeling, storage-only personalization, and insufficient gating accuracy as limitations and future-work directions.

  • Contributions: PerMem-Bench is introduced as the first benchmark for evaluating memory personalization, supported by a fully automated construction pipeline validated through meta-evaluation.The paper also proposes session-level storage gating and simple baselines.
  • Benchmark scale: 20 users currently comprise PerMem-Bench, potentially limiting the diversity of represented agent use patterns.The automated construction pipeline can scale to larger and more diverse cohorts without manual intervention.
  • Simplicity of agent use profile modeling: Agent use profiles are modeled through the deliberately simple joint configuration of domain participation and memory necessity.Richer profile representations remain an important avenue for future work.
  • Personalization limited to storage operations: Session-level storage gating applies only to storage and cannot retroactively correct mistakenly stored entries.A personalized deletion policy for evicting user-specifically unnecessary memories is left for future work.
  • Session-level gating accuracy: Current gating methods remain insufficient to fully realize personalization gains.Agentic post-training through agent-environment interaction is proposed as a promising way to improve gating accuracy.

A Benchmark Construction Details … A.5 Dialogue Generation via Dual-Simulator

PerMemBench constructs personalized longitudinal and transient interaction histories by assigning user-specific domain profiles, generating structured timelines, modeling profile shifts, and producing dialogues through isolated user and agent simulators. The pipeline distinguishes memory-required from one-off interactions while preserving natural, reference-memory-grounded conversations.

  • A.1 Domain Pool Construction: The benchmark builds a 20-domain taxonomy by sampling 1,000 personas, generating usage scenarios, clustering candidates with human review, and cross-referencing industry reports.Niche cases are pruned and broad-interest domains supplemented.
  • A.2 User-Specific Profile Assignment: For every persona-domain pair, Claude-Haiku-4.5 predicts use, memory requirement, and frequency from the user’s specific context rather than generic domain assumptions.Memory is required for ongoing, cross-session, or long-term usage, but not for one-time, self-contained interactions.
  • A.3 Life Skeleton and Timeline Construction Details: PerMem-Benchs contains static and dynamic benchmark pipelines that convert memory-required domains into life skeletons and timelines, while transient domains generate independent one-off events.Life skeletons and timeline construction provide the foundation for both benchmark variants.
  • A.3.1 Life Skeleton Construction for PerMem-Benchs: For memory-required domains, skeleton size follows frequency: high yields 5 projects with 3–5 events each, medium 3 with 2–4, and low 2 with 2–3.Events reference persistent user_profile facts and project-scoped ongoing_state facts, with sequential generation preventing duplication across domains.
  • A.3.1 Life Skeleton Construction for PerMem-Benchs: Transient domains receive self-contained one-off events without project structure or cross-session memory, with event counts derived from timeline duration and frequency-based inter-session spacing.The spacing intervals are 4 weeks for high, 8 for medium, and 12 for low frequency.
  • A.3.2 Timeline Integration for PerMem-Benchs: An LLM places all memory-required events into one chronological, interleaved timeline, while transient events are added programmatically and events beyond total_months are truncated.The arrangement enforces realistic months, project ordering, unique event placement, anchor life events, and sequential session IDs.
  • A.5 Dialogue Generation via Dual-Simulator: Each timeline entry becomes a dialogue from isolated user and agent simulators, with memory-required sessions receiving reference memories and transient sessions receiving only event descriptions.User_profile facts surface implicitly, ongoing_state facts emerge as uncertainties, and an LLM judge nudges unrevealed facts until natural completion and minimum turns.

B Meta Evaluation Details … B.3 Stage 3: Dialogue Quality

The meta-evaluation validates the data-generation pipeline across three stages: profile plausibility, life-skeleton and timeline realism, and dialogue quality. It uses sampled personas or sessions and structured evaluator criteria to assess alignment, coherence, realism, and natural memory revelation.

  • B Meta Evaluation Details: The validation pipeline comprises three stages for assessing the reliability of generated data.The stages cover profile plausibility, life-skeleton and timeline realism, and dialogue quality.
  • B.1 Stage 1: Profile Plausibility: 100 randomly sampled Nemotron-Persona-USA personas are evaluated for the plausibility of assigned user profiles.A two-evaluator panel judges whether domain choices, metadata coherence, and persona-grounded reasons are plausible.
  • B.1 Stage 1: Profile Plausibility: Profile plausibility is judged as YES only when domain choices, logical consistency, metadata coherence, and reasons are generally satisfactory.Clear persona-irrelevant or internally inconsistent patterns trigger a NO judgment.
  • B.2 Stage 2: Life Skeleton and Timeline Realism: The same 100 personas are used to evaluate generated life skeletons and timelines for persona alignment, realistic event structure, timeline coherence, and plausibility.The evaluation considers believable progression and coherent durations and ordering over 1-2 years.
  • B.2 Stage 2: Life Skeleton and Timeline Realism: Life-skeleton evaluation returns YES only when all four criteria are mostly satisfied and no major contradiction exists.The criteria are explicitly recorded as persona_alignment, project_event_structure, timeline_coherence, and realism_plausibility.
  • B.3 Stage 3: Dialogue Quality: 100 randomly sampled sessions are evaluated because each dialogue corresponds to one life-skeleton event stored at session level.The assessment checks consistency with project and event context and whether reference memories surface naturally.
  • B.3 Stage 3: Dialogue Quality: Dialogue quality scores skeleton_alignment and gt_memory_revelation on 1-5 scales, penalizing forced or unnatural memory fact dumping.The evaluator compares the conversation with ground-truth session context and assesses whether relevant facts emerge naturally.
  • B.3 Stage 3: Dialogue Quality: Dialogue evaluation returns YES only if all three scores are at least 4 and no critical issue exists.The structured output includes the two scores, fact checks, a summary, and major issues.

C Evaluation Protocol Details · C.1 LLM-as-a-Judge for I(r ∈Mt) · C.2 Checkpoint Sampling Approximation

The evaluation determines whether reference memories remain preserved by retrieving similar entries and applying an LLM judge, then estimates retention over session windows using uniformly sampled checkpoints. This approximation uses 20 evenly spaced checkpoints and endpoint inclusion to reduce the cost of evaluating every session.

  • C.1 LLM-as-a-Judge for I(r ∈Mt): Reference-memory presence is evaluated by retrieving the top-10 memory-bank entries with cosine similarity using all-MiniLM-L6-v2.The retrieved candidates are then passed to gpt-5-nano for judgment.
  • C.1 LLM-as-a-Judge for I(r ∈Mt): The LLM judge performs binary classification based on whether the retrieved text preserves the reference memory’s core meaning.It answers YES when the fact is clearly expressed, even with different wording, and NO when absent or not inferable.
  • C.1 LLM-as-a-Judge for I(r ∈Mt): The judge receives a fact and retrieved entries, then replies only YES or NO about whether the text contains the fact’s core meaning.The procedure is designed to recognize paraphrased expressions while rejecting unsupported inferences.
  • C.2 Checkpoint Sampling Approximation: Full session-by-session evaluation is prohibitively expensive, requiring thousands of LLM calls per user per memory system.The study therefore approximates the inner summation in Equation (1) with uniform checkpoint sampling.
  • C.2 Checkpoint Sampling Approximation: K=20 evenly spaced checkpoints are sampled for each reference memory across its evaluation window.The sampled checkpoints are drawn from sessions ranging from tstart(r) through Ttarget(r).
  • C.2 Checkpoint Sampling Approximation: Both the first and last evaluation sessions are always included in the checkpoint set.These endpoints verify initial storage and survival through the end of the required lifespan.
  • C.2 Checkpoint Sampling Approximation: The sampled scores are reweighted to approximate the full inner sum, assuming each checkpoint represents an equally sized interval.The resulting approximated RR is used in all experiments.

D Implementation Details · D.1 LLM Decoding

The implementation uses deterministic decoding throughout evaluation, personalization, and data generation. Qwen3-14B runs through vLLM on an A6000 48GB GPU, while proprietary models use their API services.

  • D Implementation Details: Temperature is set to 0 for all evaluated memory systems.This setting supports reproducible and deterministic evaluation.
  • D Implementation Details: Temperature is also set to 0 for all personalization methods.The same deterministic decoding policy applies to personalization components.
  • D Implementation Details: The LLM-based data generation pipeline uses temperature 0.This ensures deterministic generation during data preparation.
  • D.1 LLM Decoding: All components therefore use deterministic decoding for reproducibility.The temperature-0 setting covers memory systems, personalization methods, and data generation.
  • D.1 LLM Decoding: Qwen3-14B inference is served via vLLM.The inference service runs on an A6000 48GB GPU.
  • D.1 LLM Decoding: Proprietary models are accessed through their API services.The implementation uses model-provider APIs for proprietary-model inference.

D.2 Personalization Method · NeurIPS Paper Checklist

The personalization method uses LLM-based session-level storage gating with increasingly rich historical context, while the checklist documents reproducibility, limitations, ethics, asset licensing, and human-subject procedures.

  • D.2 Personalization Method: All three methods predict whether memory is required and skip memory operations for sessions classified as transient.They share the same interface and differ only in the historical context supplied to the LLM.
  • D.2 Personalization Method: Greedy uses only the current dialogue, whereas Context-aware additionally supplies a running buffer of the most recent K session summaries.The current dialogue is truncated to max_chars in Greedy; Context-aware first summarizes each session in 1–2 sentences.
  • D.2 Personalization Method: Structure-aware extracts {purpose, summary, topic} records, periodically clusters sessions into projects or isolated sessions, and preserves the structural note across windows.Previously isolated sessions may be reassigned when new evidence connects them; unassigned sessions default as specified in the method.
  • NeurIPS Paper Checklist: The checklist states that the paper’s abstract and introduction accurately reflect its contributions, motivation, dataset construction, and empirical study.It also emphasizes that claims should match theoretical and experimental results and acknowledge assumptions and limitations.
  • NeurIPS Paper Checklist: The authors provide source code, data, running code, and detailed implementation information in an anonymous GitHub repository and the appendix.The checklist frames these materials as supporting faithful reproduction of the main experimental results and understanding of the experimental setting.
  • NeurIPS Paper Checklist: The authors do not report statistical error bars because the cost of LLM API calls is prohibitively high for multiple runs.This is identified as a limitation affecting repeated experimental evaluation.
  • NeurIPS Paper Checklist: For crowdsourcing or human-subject research, the authors direct readers to the Meta Evaluation section for participant instructions, screenshots, compensation, and related details.The checklist also asks authors to address applicable IRB approval and ethical requirements.
Loading 2605.25535v1…