PaperlayerView exact revision on arXiv

Source-linked AI summary

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu

arXiv:2603.23231v2cs.AI

TL;DR

Personalized-memory evaluation has largely treated preferences as isolated facts, leaving the evolution and integration of user signals across noisy interactions undermeasured. PERMA addresses this gap with an event-driven benchmark spanning temporal, multi-session, multi-domain, noisy, and style-aligned conversations. Memory systems generally outperform vanilla RAG in linking interactions and reducing token use, but still struggle with coherent personas under temporal depth and cross-domain interference.

  • Problem

    Existing evaluations emphasize isolated preference retrieval rather than how preferences evolve, accumulate, and interact across noisy, multi-session contexts.

  • Method

    PERMA constructs temporally ordered event-driven interactions and evaluates persona maintenance through preference synthesis, temporal probing, multiple-choice tasks, and interactive user simulation.

  • Results

    Most memory systems outperform vanilla RAG by linking related interactions, with MemOS achieving the highest MCQ Acc. (0.811) and Memory Score (2.27) while using 709.1 Context Token versus 828.8 for vanilla RAG.

  • Takeaways & Limitations

    PERMA supports evaluating personalized memory as persistent persona-state maintenance rather than static preference recall.

  • Takeaways & Limitations

    Option accuracy alone is insufficient to characterize memory effectiveness because correct choices can remain stable while memory scores and response quality decline in multi-domain reasoning.

Abstract

from arXiv · show

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

1 Introduction

Personalized memory requires maintaining coherent user representations as preferences evolve across interactions, yet existing evaluations and systems largely emphasize factual retrieval rather than realistic persona consistency.

  • Long-term memory must continuously construct and update a coherent persona state reflecting users’ constraints and behavioral tendencies over time.
  • Prior retrieval and structured-memory approaches improve access to relevant knowledge, but personalization-oriented evaluation still requires stronger assessment of evolving user profiles.
  • Current personalization benchmarks remain difficult to quantify in complex real-world scenarios.
  • Effective personalization requires integrating latent, incremental, and noisy preference signals rather than merely memorizing facts.
  • PERMA models persona maintenance as an evolving state combining dynamic preferences and episodic memory across sessions and domains.Its corpus spans 10 countries, more than 20 topic domains, 2,166 preference details, over 800 events, and 1.8M context tokens.

(b) PERMA

PERMA constructs realistic, event-driven interaction histories and evaluates whether agents can synthesize preferences into coherent personas across noisy, multi-domain, temporally evolving conversations.

  • Event summaries capture evolving user behavior, such as movement from cultural travel interests toward couple-centric activities.
  • Its assessment suite combines task completion, preference consistency, information confidence, temporal probing, and simulated multi-turn interaction.
  • The dataset compares clean, noisy, and style-aligned interaction settings to examine robustness to realistic linguistic variation.
  • PERMA shifts personalized-memory evaluation from static preference recall to preference synthesis through event-driven dialogue scenarios.
  • The benchmark tests cross-domain persona maintenance across multiple sessions using natural user requests rather than recall-oriented prompts.

2 Related Work

Related work has progressed from long-context fact retrieval and static preference conditioning toward persistent memory, but prior benchmarks rarely test coherent persona evolution under noisy, cross-domain interactions.

  • Long-context evaluations traditionally use Needle-in-a-Haystack retrieval, with newer benchmarks extending to ultra-long multi-session dialogues and free-form questions.
  • Personalization benchmarks evolved from static conditioning to multi-session retention, but few directly evaluate coherent persona states under evolving, information-dense interactions.
  • PERMA evaluates composite synthesis by integrating interdependent traits and tracking newly emerged or updated preferences across temporal events.
  • RAG alleviates context-window constraints through semantic retrieval but remains stateless and does not model temporal persona coherence.
  • Persistent-memory architectures extract, consolidate, and relate conversational information, while PERMA distinguishes itself through event evolution, context noise, and temporal probing.

3 Preliminaries

PERMA formalizes personalization as temporally evolving interaction events and persona states rather than static user attributes. Its framework distinguishes event types, preference updates, task timing, and memory-system operations for evaluating persona-consistent responses.

  • User and event representation: A user profile combines demographic attributes, interests, and temporally ordered interaction events.
  • User and event representation: Each interaction event records an event type, relevant domains, multi-turn dialogue content, and preferences revealed during that event.Event types include Emergence, Supplement, and Task.
  • Evaluation objective: Tasks evaluate responses at different timeline positions, including before relevant preferences, immediately after relevant sessions, and after unrelated-topic interference.These are classified as Zero-Memory, In-Time, and Post-Intervention tasks.
  • Persona state: Persona state synthesizes integrated preferences and episodic memory from historical dialogues, incorporating newly emerged or updated preferences over time.Unlike a static profile, it reflects incremental preference refinement and potential drift.
  • Evaluation objective: The benchmark requires responses to align with the evolved persona while satisfying demographic constraints and preference updates across multi-session, cross-domain histories.Memory systems ingest completed sessions and retrieve contextually relevant memory segments for each query.

4 Methodology

PERMA reconstructs personalization as evolving persona states by generating event-driven, cross-domain dialogue timelines and testing them under temporal depth, noisy inputs, and linguistic variation.

  • PERMA models personalization as evolving preferences through event-driven reconstruction of dialogue histories, rather than static preference recall.
  • Event-Driven Dialogue Reconstruction: A high-level LLM planner generates domain-specific timelines from interaction summaries and user demographic profiles.
  • Event-Driven Dialogue Reconstruction: Each timeline entry includes a conversation description and cross-domain dependencies to support inter-domain linkage and iterative preference refinement.
  • Event-Driven Dialogue Reconstruction: Domain timelines are interleaved into a chronological global timeline while respecting explicitly defined cross-domain dependencies.
  • Evaluation: Evaluation tasks are inserted at zero-memory, in-time, and post-intervention checkpoints to probe preference development, forgetting, and contextual interference.
  • Text Variability and Linguistic Alignment: Controlled noise injection produces vague or misleading but semantically aligned user turns, extending evaluation to coherent persona tracking under ambiguity and linguistic variability.

5.1 Experiment Setup

The experiment compares standalone LLMs with retrieval-augmented and structured-memory agents on PERMA, using full-history or retrieved evidence under controlled implementation settings and validated task construction.

  • Evaluation Paradigms: The evaluation compares two paradigms: standalone LLMs and agents equipped with retrieval-augmented generation or memory systems.
  • Standalone LLMs: Standalone LLMs receive the full dialogue history without retrieval or memory compression and are tested on synthesizing preferences across domains and fragmented sessions.
  • Agents with RAG and Memory Systems: Memory-based approaches include standard dense-retrieval RAG and structured systems that manage, compress, organize, or retrieve persistent user information.
  • Implementation Details: For RAG and memory systems, semantic retrieval uses the top-10 chunks by default, with each RAG chunk defined as one user-assistant pair.
  • Validation: Human experts achieve 97.75% accuracy on MCQ tasks using the ground-truth dialogue history, supporting alignment between task options and constructed histories.
  • Validation: Experts report a near-perfect 1.99/2 score for event coverage, while 98% of generations adhere to designated noise types.

5.2 Overall Performance

PERMA shows that memory systems trade retrieval efficiency against task fulfillment, with structured systems often outperforming raw semantic retrieval but remaining vulnerable to temporal depth, noise, and cross-domain interference. Performance patterns also diverge between option selection and interactive response generation, exposing limits of accuracy-only evaluation.

  • Overall model performance: 0.882 MCQ Acc. makes Kimi-K2.5 the strongest standalone model, followed by Qwen3-32B and Gemini2.5-Flash at 0.870.The authors associate this advantage with synthesizing preferences and episodic information from dialogue histories.
  • Memory-system efficiency: Over 99% context compression reduces Supermemory and Lightmem below 300 tokens, but both sacrifice task fulfillment; MemOS balances efficiency with 0.811 MCQ Acc. and 2.27 Memory Score.MemOS uses 709.1 context tokens versus 828.8 for vanilla RAG.
  • Memory-system efficiency: 0.846 Completion rate makes EverMemOS competitive, but its 16s local search latency limits time-sensitive use, whereas vanilla RAG retrieves in 16ms with 0.830 success.The comparison illustrates a substantial latency-efficiency trade-off between the systems.
  • Interactive performance: MemOS leads Turn=1 success at 0.548, while Memobase and Lightmem lead Turn≤2 at 0.830 and 0.813, indicating different one-shot and recovery behaviors.Mem0 has the lowest Turn≤2 success rate and often needs multiple clarification rounds.
  • Retrieval and persona structure: RAG reaches 0.849 BERT-f1, but structured systems such as MemOS achieve higher MCQ accuracy and Completion rates by linking related information into unified persona states.The structured representation reduces the synthesis burden on the response-generating model.
  • Temporal depth: Accuracy consistently declines from Type 2 to Type 3 checkpoints, showing that greater dialogue depth increases interference and undermines coherent persona maintenance.Type 1 serves as the zero-memory, non-personalized baseline.
  • Noise robustness: 0.879 noisy-context accuracy makes Gemini2.5-Flash more robust than Kimi-K2.5, whose Clean-setting 0.882 falls to 0.865 under text variability.The results distinguish strong Clean performance from robustness to distraction.
  • Noise robustness: MemOS improves from 0.811 to 0.853 MCQ Acc. and from 2.27 to 2.38 Memory Score in noise as retrieval expands from 709.1 to 1486.7 tokens.The authors interpret in-session noise as emphasizing preference information and enabling more detailed extraction.

5.3 Further Analysis

Further analysis tests memory agents under long, noisy, stylistically varied, multi-domain contexts. Memory systems are generally robust to stylistic variation and irrelevant token noise, but persona consistency degrades with extreme context length, heterogeneous domains, and cross-domain interference.

  • Stylistic Interference and Token-level Noise: Style-aligned long contexts leave most memory systems’ single-domain performance largely unchanged when preference information and task requirements are preserved.
  • Stylistic Interference and Token-level Noise: 0.766 to 0.716: Qwen2.5-14B-1M declines under substantial irrelevant token noise, while memory systems maintain stable performance.GPT-4o-mini fails to respond under long dialogue inputs.
  • Persona State Consistency in Long-Context Multi-Domain Settings: 0.764 MCQ Acc.: MemOS leads multi-domain evaluation, while Supermemory and Mem0 fall below 0.60.MemOS’s interaction performance also improves from 0.306 to 0.331 at Turn=1 and from 0.592 to 0.637 at Turn≤2.
  • User Variability and Long-Context Degradation: Beyond the 60% position, approximately 73k tokens, GPT-4o-mini suffers catastrophic degradation and ultimately fails at the 100% mark.MemOS remains stable by transforming global long-context reasoning into persona-state matching.
  • Robustness across Domains and Positions: MemOS sustains high accuracy at extreme context lengths, whereas multi-domain transitions and heterogeneous tasks expose broader consistency challenges.Table 10 reports strong domain-specific results for Kimi-K2.5 in Finance and Messaging, and for MemOS in Shopping and Travel.
  • Robustness under Positional Probing: The positional evaluation measures MCQ Acc. across checkpoints from 10% to 100% for queries linked to previously emerged domain events.Table 9 defines #Ques. as questions whose corresponding domain events have already appeared.

6 Conclusion

PERMA reframes personalized-memory evaluation from static preference recall to persona consistency through temporally ordered interaction events, text variability, and linguistic alignment. Experiments show that memory systems often outperform RAG and improve token efficiency, yet remain vulnerable to multi-domain transitions and conflicting signals.

  • PERMA shifts personalized-memory evaluation from static preference recall to persona consistency under text variability and real-world linguistic alignment.
  • Memory systems generally outperform RAG by linking related interactions into a persistent persona state across single- and multi-domain tasks.
  • 300× higher token efficiency: memory systems are more robust than vanilla long-context models in high-entropy settings.
  • Standalone LLMs degrade severely beyond their context windows, whereas memory systems remain stable by reducing long-range reasoning to persona-state matching.
  • Multi-domain transitions and conflicting signals still degrade performance, revealing sensitivity to context complexity.

A Prompt

The prompt pipeline constructs realistic preference-evolution timelines, converts them into naturalistic dialogues, and evaluates agents through interactive feedback and memory-focused scoring. It also generates challenging options that test discrimination between aligned and misaligned preferences.

  • TIMELINE_GENERATION models how user interests emerge and deepen across interaction events.
  • DIALOGUE_GENERATION converts timelines into naturalistic conversations where preferences surface through feedback or corrections.
  • USER_FEEDBACK_PROMPT enables simulated users to clarify intent or correct assistants during adaptive multi-turn interactions.
  • LLM-as-a-Judge scores response quality and personalized memory, while EVAL_MEMORY_SCORE checks retrieved details against ground truth.
  • OPTION_GENERATION_PROMPT creates plausible but preference-misaligned choices to test discriminative ability in noisy environments.

B Case Study

The case study compares retrieved memories across systems using a prompt-driven benchmark and standardized memory evaluation. MemOS organizes episodic, explicit-preference, and implicit-preference information through parallel retrieval, while the supporting prompts generate timelines, dialogues, interactions, options, and quality assessments.

  • MemOS categorizes retrieved memory into episodic facts, explicit preferences, and implicit preferences using parallel retrieval.
  • TIMELINE_GENERATION converts user profiles and interaction summaries into coherent preference-development events.
  • The pipeline uses answer-option prompts and dialogue-generation prompts to create preference-focused questions and naturalistic conversations.
  • User-feedback and answer-interaction prompts support simulated user responses and personalized assistant interactions using accumulated memory.
  • OPTION_GENERATION_PROMPT produces eight fluent response options that remain consistent with the assistant’s tone and task context.
  • The memory evaluation checks coverage by comparing retrieved memory with the preferences to be mastered, distinguishing missing core and minor details.
  • Tables 11–12 compare retrieved memory contexts among baselines in the single-domain setting, while annotation evaluates six data-quality criteria.
Loading 2603.23231v2…