PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization

Weizhi Zhang, Xiaokai Wei, Wei-Chieh Huang, Zheng Hui, Chen Wang, Michelle Gong, Philip S. Yu

arXiv:2603.25973v1cs.CL

TL;DR

Existing memory benchmarks provide limited evidence about authentic, long-term, cross-domain personalization and user satisfaction. MEMORYCD addresses this gap with a real-user benchmark and end-to-end evaluation across tasks, domains, models, and memory systems. Results show that scaling models alone is insufficient, while memory mechanisms have task-dependent strengths and remain far from robust user satisfaction.

  • Problem

    Existing memory benchmarks largely rely on synthetic or short, single-domain data and retrieval-focused evaluation, limiting assessment of authentic long-context, cross-domain personalization.

  • Method

    MEMORYCD evaluates real multi-year user histories through 4 personalization tasks in 2 memory settings across 12 domains, using 14 long-context LLMs and 6 memory systems.

  • Results

    Across tasks, increasing model scale alone is insufficient for robust personalization, while memory mechanisms show distinct, task-dependent strengths and no single method dominates.

  • Takeaways & Limitations

    MEMORYCD provides a foundation for developing principled memory abstractions that organize and exploit long-term histories for personalization.

  • Takeaways & Limitations

    Cross-domain evaluation is restricted to fixed memory operations, excludes privacy and safety guardrails, and does not incorporate large-scale training-based memory construction.

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) have expanded context windows to million-token scales, yet benchmarks for evaluating memory remain limited to short-session synthetic dialogues. We introduce \textsc{MemoryCD}, the first large-scale, user-centric, cross-domain memory benchmark derived from lifelong real-world behaviors in the Amazon Review dataset. Unlike existing memory datasets that rely on scripted personas to generate synthetic user data, \textsc{MemoryCD} tracks authentic user interactions across years and multiple domains. We construct a multi-faceted long-context memory evaluation pipeline of 14 state-of-the-art LLM base models with 6 memory method baselines on 4 distinct personalization tasks over 12 diverse domains to evaluate an agent's ability to simulate real user behaviors in both single and cross-domain settings. Our analysis reveals that existing memory methods are far from user satisfaction in various domains, offering the first testbed for cross-domain life-long personalization evaluation.

1 INTRODUCTION

MEMORYCD addresses gaps in existing memory benchmarks by evaluating long-context, cross-domain personalization with authentic user histories and feedback. It introduces an end-to-end benchmark spanning diverse domains, tasks, and memory settings to assess whether agents can support lifelong user satisfaction.

  • Motivation: MEMORYCD targets persistent assistants that retain, organize, and use users’ historical preferences and evolving goals.The motivation goes beyond answering isolated prompts toward coherent modeling of user history for lifelong assistance.
  • Limitations of Existing Benchmarks: Existing benchmarks often use synthetic behaviors, short real-user sequences, or isolated domains, limiting realistic long-context and cross-domain evaluation.These limitations reduce authentic user feedback and fail to stress modern long-context memory across domains.
  • Evaluation Gap: Existing evaluation pipelines commonly reduce memory to retrieval, overlooking other stages of the memory lifecycle and end-to-end user-centric personalization.The benchmark instead evaluates whether memory supports downstream behavior rather than only fact detection.
  • MEMORYCD: MEMORYCD is presented as the first cross-domain long-context memory benchmark grounded in real user histories and real user feedback.Its data analysis covers overlapping preferences across applications including entertainment, healthcare, and lifestyle domains.
  • Benchmark Design: The benchmark uses 4 basic tasks in 2 memory settings and evaluates 14 frontier long-context LLMs with 6 memory systems over 12 diverse domains.This design tests downstream user satisfaction rather than merely long-context retrieval.

2 TASK SETTINGS AND PROBLEM FORMULATION

MEMORYCD formulates long-context personalization as mapping multi-year, multi-domain user histories and task instructions to behaviorally aligned outputs. It evaluates four personalization tasks under single-domain and cross-domain memory-source settings.

  • 2 TASK SETTINGS AND PROBLEM FORMULATION: Each user memory pool records item metadata, 1–5 ratings, timestamps, and textual opinions from interactions spanning multiple years and domains.The formulation treats these records as historical context from which an agent infers long-term preferences and linguistic tendencies.
  • 2 TASK SETTINGS AND PROBLEM FORMULATION: The benchmark formalizes each task as fθ: (M_u, X) → Y, where the model and memory mechanism condition task instructions on user history to produce task-specific actions.This mapping evaluates whether agents align outputs with prior user behavior patterns.
  • 2.1 PERSONALIZED RATING PREDICTION: Personalized rating prediction regresses a target user’s numerical preference from memory and item metadata, using MAE and RMSE to measure deviation.The task probes long-term preference modeling and calibration of numerical predictions.
  • 2.2 PERSONALIZED ITEM RANKING: Personalized item ranking orders same-domain candidate items by expected user interest from memory, with NDCG@K measuring consistency against true preference signals.This task targets relative rather than absolute preference strength.
  • 2.3 PERSONALIZED REVIEW SUMMARIZATION: Personalized review summarization generates concise, sentiment-aligned titles from user memory, target-item metadata, and full review text, evaluated by ROUGE-L and BLEU-1.These metrics assess alignment with the user’s tone, sentiment distribution, and lexical tendencies.
  • 2.4 PERSONALIZED REVIEW GENERATION: Personalized review generation produces full-length text reflecting both item-attribute preferences and the user’s established expression style.ROUGE-L and BLEU-1 assess semantic fidelity, stylistic alignment, and preference-grounded expression.
  • 2.5 BENCHMARK MEMORY SOURCE SETTINGS: The benchmark compares single-domain memory, drawn from the evaluation domain, with cross-domain memory, drawn from heterogeneous other-domain interactions.Cross-domain evaluation tests transfer of consistent preference signals while suppressing irrelevant domain-specific noise.

3 DATASET ANALYSIS AND CONSTRUCTION

MEMORYCD treats users as long-horizon behavioral processes built from Amazon Reviews and analyzes cross-domain overlap to support personalization evaluation. It filters for sufficiently long histories and focuses experiments on four representative domains.

  • DATASET CONSTRUCTION: MEMORYCD constructs each user as a long-horizon behavioral process rather than a collection of isolated interactions from the Amazon Review corpus.This construction targets long-context user histories for memory evaluation.
  • DATASET ANALYSIS: For each domain pair, the benchmark computes the percentage of users in one domain who also appear in the other and visualizes these ratios as a heatmap.The overlap analysis identifies feasible cross-domain personalization settings.
  • DATASET ANALYSIS: Non-trivial cross-domain user overlap provides a natural foundation for evaluating cross-domain personalization despite limited overlap across many domain pairs.The analysis supports studying users active across multiple domains.
  • DATASET CONSTRUCTION: MEMORYCD retains users with sufficiently long histories, applies domain-specific minimum interaction thresholds, and focuses main experiments on Personal Care, Books, Electronics, and Home & Kitchen.The filtering aims to produce stable preference signals and long-context memories.

4 EXPERIMENTS

Experiments compare frontier long-context LLMs and memory methods across single- and cross-domain personalization tasks. Results show uneven model strengths, method-specific gains, and domain-dependent value from cross-domain memory.

  • Experimental setup: 14 frontier LLM backbones and 6 memory methods are evaluated across four personalization tasks, with users filtered by domain-specific interaction thresholds and recent evaluation instances.The benchmark also compares long-context prompting, memory methods, and cross-domain memory sources across representative domains.
  • Long-context prompting: GPT-5 and Gemini-2.5 Pro are the most competitive backbones, with GPT-5 strongest on rating and ranking and Gemini-2.5 Pro strongest on generation quality.GPT-5 reaches RMSE 0.624 for rating prediction in Books and NDCG@3 0.604 for ranking in Electronics.
  • Long-context prompting: Rating and ranking improvements do not consistently translate into better personalized text generation, showing uneven gains across tasks under raw long-context prompting.The results motivate memory mechanisms that abstract, compress, and route user information beyond directly consuming the full history.
  • Memory methods: No single memory method dominates: long-context prompting is strong for rating, while Mem0, LoCoMo, MemoryBank, ReadAgent, and A-Mem each improve different retrieval- or generation-sensitive tasks.Examples include Mem0 improving summarization ROUGE-L from 0.145 to 0.202, ReadAgent improving GPT-5 NDCG@1 from 0.146 to 0.225, and MemoryBank improving generation ROUGE-L from 0.132 to 0.166.
  • Cross-domain memory sources: Cross-domain memory transfer depends on semantic alignment: Electronics improves Home & Kitchen MAE from 0.43 to 0.30 for GPT-5, whereas Books transfers more weakly.Electronics shares product attributes and preference factors with Home & Kitchen, while less-related memories can introduce noise.
  • Cross-domain memory sources: Aggregating Books, Electronics, and Personal Care often improves ranking, but the best MAE or RMSE commonly comes from the most aligned single source rather than all sources.Broader memory coverage can help comparative reasoning, while unrelated domains may add distracting or conflicting information.
  • Cross-domain memory methods: Memory methods improve cross-domain personalization by filtering and compressing heterogeneous histories into task-relevant cues, with LoCoMo strongest for ranking and A-Mem or MemoryBank strongest for generation-related tasks.For GPT-5 and Gemini-2.5 Pro, LoCoMo raises NDCG@3 to 0.355 and 0.349, versus 0.299 and 0.255 with raw long-context prompting.

5 CONCLUSION

MEMORYCD evaluates ultra-long-context user memory through real-world, multi-domain histories and end-to-end behavioral personalization. Its experiments show that scaling model size alone is insufficient for robust long-context personalization.

  • Conclusion: MEMORYCD is a large-scale benchmark using real-world, multi-domain interaction histories to evaluate memory effects on decision-making and personalized generation.It frames personalization as an end-to-end behavioral modeling problem rather than a retrieval-only task.
  • Conclusion: Increasing model scale alone is insufficient for robust long-context personalization.

LIMITATIONS

MEMORYCD’s scope is bounded by fixed cross-domain memory operations, omitted deployment-time privacy and safety evaluation, and the absence of large-scale training-based memory construction methods.

  • Limitations: Cross-domain evaluation is restricted to fixed memory operations, leaving learned memory mechanisms and broader domain transfer for future work.
  • Limitations: The benchmark evaluates memory utilization but does not assess user privacy or safety guardrails over deployment time.
  • Limitations: Large-scale training-based memory construction approaches are not incorporated because most state-of-the-art long-context LLMs remain closed-source.
  • Benchmark scope: Existing real-user benchmarks remain limited by short contexts, few interaction sessions, and little cross-domain coverage, motivating MEMORYCD’s broader evaluation scope.
  • Data handling: MEMORYCD uses anonymized Amazon Reviews 2023 data and excludes private, sensitive, or personally identifiable information from benchmark construction.

C DATASETS DETAILS

The dataset contains long user histories across 12 product domains, retains sufficiently active users, and provides three recent evaluation samples per user. Its statistics support ultra-long-context and cross-domain personalization evaluation.

  • Dataset statistics: User histories range from 11K tokens in Beauty to over 314K tokens in Movies & TV across 12 product domains.
  • Dataset statistics: Highly active domains such as Books, Electronics, Personal Care, and Movies & TV contain thousands of sessions per user, with Books averaging 1,552 sessions and reaching 3,888.
  • User selection: Domain-specific minimum interaction thresholds retain sufficiently active users, and each user contributes the most recent three evaluation samples.Thresholds range from 50 to 1,000 sessions depending on domain activity.
  • Cross-domain construction: The aggregated cross-domain setting combines interactions across scenarios to create a challenging testbed for ultra-long-context and cross-domain personalization.

D MEMORY METHODS AND BASELINES

The benchmark compares memory baselines spanning implicit full-history prompting, retrieval, structured memory, episodic compression, and dynamically linked notes. These designs expose different trade-offs in how user information is stored, retrieved, compressed, and integrated.

  • Baseline spectrum: The baselines range from naive full-history prompting to structured or compressed memory representations, enabling comparisons of coverage, relevance, and robustness under ultra-long contexts.The suite includes direct long-context prompting and memory-augmented approaches with different storage and retrieval designs.
  • Implicit memory: Long-Context concatenates all historical interactions directly into the prompt without explicit storage, indexing, retrieval, summarization, or forgetting.It serves as a minimal baseline for testing raw long-context capacity.
  • Retrieval-based memory: LoCoMo uses explicit memory storage, vector indexing, and semantic retrieval to supplement the model context and reduce reliance on position biases.Its retrieval-augmented generation design brings relevant memories into the prompt during inference.
  • Structured memory: Mem0 dynamically extracts, consolidates, and maintains salient facts and preferences as structured memory entries for persistent conversational memory across sessions.Its architecture goes beyond concatenated context and raw chunk retrieval.
  • Episodic compression: ReadAgent segments long text into episodes, compresses them into gist memories, and selectively looks up detailed passages when needed.This combines compact episodic representations with access to relevant raw text.
  • Linked memory: A-Mem organizes memories as evolving knowledge-graph notes with keywords, contextual descriptions, semantic tags, and dynamically instantiated links.The resulting interconnected network is designed to organize and evolve stored knowledge.

E EXPERIMENTAL DETAILS

The experiments standardize user memories, domains, implementations, and decoding settings while evaluating both single- and cross-domain personalization. Generation quality is assessed with multiple judges alongside reproducible lexical metrics, revealing judge bias and instability.

  • Memory construction: Each user memory M_u aggregates all historical interactions, with memory-augmented methods restricted to M_u and all models receiving the same information through direct long-context prompting.The resulting token lengths vary across domains.
  • Evaluation settings: The single-domain evaluation contains 12 domain-specific benchmarks, while the cross-domain setting uses 323 users active in Books, Electronics, Home & Kitchen, and Personal Care.Cross-domain memories aggregate interactions from multiple domains before evaluation.
  • Standardization: All models and memory methods use shared prompting templates, evaluation scripts, fixed decoding hyperparameters, and one standardized cloud-based execution environment.This design controls for implementation and hardware-induced variability.
  • Controls: Random seeds, test instances, memory access, and metric implementations are controlled so observed differences reflect model capacity and memory design rather than evaluation pipelines.Methods receive no additional user profiles, external corpora, or domain-specific information.
  • Generation evaluation: Review summarization and generation are scored by GPT-5, Claude 4 Sonnet, and Gemini-2.5 Pro judges on a 1–10 preference scale against real user-written texts.The evaluation focuses on two language-generation tasks.
  • Judge analysis: Individual judges show self-preference bias, while averaged scores align with ROUGE-L and BLEU-1 rankings; score variation across runs motivates using the lexical metrics.Aggregation mitigates individual bias but increases computational overhead.
  • Backbone specialization: GPT-5 is strongest on decision tasks and Gemini-2.5 Pro on generated-related tasks across the remaining eight single domains.GPT-5 leads most rating and ranking evaluations, while Gemini-2.5 Pro leads summarization and generation.
  • Task and domain difficulty: User satisfaction remains task-dependent and difficult in some domains, with Beauty harder than structured product domains and preference-prediction leaders differing from preference-verbalization leaders.Model rankings vary across task types.

H EVALUATION OF ALL LLMS WITH REPRESENTATIVE MEMORY METHODS IN SINGLE DOMAINS SETTING

Across representative memory methods and single domains, performance varies substantially by task, backbone, domain, and memory design. Decision-oriented and language-generation tasks often favor different models, while Home & Kitchen yields clearer ranking separation than Books.

  • Cross-task variation: Across Books and Home & Kitchen, performance varies substantially across task types, with decision-task leaders often differing from language-generation leaders.The comparison covers 14 frontier long-context LLMs under MemoryBank and A-Mem.
  • Backbone specialization: GPT-5 generally performs best on rating and ranking, whereas Gemini-2.5 Pro is stronger on summarization and generation.This task-wise specialization is stable across the reported single-domain comparisons.
  • Domain difficulty: Home & Kitchen typically has higher and more separated ranking scores than Books, whose preferences appear more ambiguous and compress model gaps.The difference is attributed to stronger item-attribute regularities and cleaner preference signals in Home & Kitchen.
  • Memory-method effects: MemoryBank tends to amplify backbones already strong in holistic language modeling and preference aggregation, benefiting rating prediction and item ranking.Its continuous user-centric profile and retrieval expose synthesized preference information for decision-making.
  • Memory-method effects: A-Mem more often shifts advantage toward models that exploit structured, explicitly linked notes, which is beneficial for generation-related tasks.Its dynamic indexing and linking support compositional traversal over an evolving memory graph.

I EVALUATION OF DIFFERENT METHODS IN SINGLE DOMAIN SETTINGS

Memory methods improve personalization differently across domains and tasks, with retrieval, profile, and structured memories each favoring distinct outcomes and backbones. The evaluation therefore treats prompt design and task-specific output constraints as part of a unified comparison.

  • Overall comparison: Across Electronics, Home & Kitchen, and Personal Care, no memory design dominates universally because gains depend strongly on task and domain.The comparisons show different methods trading off ranking, rating, and generation performance.
  • Retrieval methods: Mem0 and LoCoMo most consistently boost ranking, often producing large NDCG gains over long-context prompting.These retrieval and selection-style methods are especially effective for ranking-oriented personalization.
  • Structured methods: MemoryBank and A-Mem more often provide balanced improvements across tasks than retrieval-focused methods.Their profile and structured representations affect how backbones consume condensed evidence.
  • Home & Kitchen: On Home & Kitchen, retrieval and selection benefit ranking, while structured memories and backbone strengths favor rating and generation.Mem0, LoCoMo, and MemoryBank substantially improve ranking for GPT-5 and Claude-4 Sonnet, whereas A-Mem is competitive on rating.
  • Backbone-method pairing: Gemini-2.5 Pro remains the strongest generator and performs best with MemoryBank-like profile memory, while GPT-5 remains stronger on decision metrics with suitable memory strategies.This reflects a trade-off between generation quality and decision-oriented performance.
  • Task prompts: The four tasks use unified prompts conditioned on user history and task-specific inputs, with strict output formats for automatic evaluation.The tasks are rating prediction, item ranking, review summarization, and review generation.
  • Decision tasks: Rating prediction requires a single integer from 1 to 5, while item recommendation requires a ranked list of candidate-item indices.Both prompts condition predictions on the user’s purchase and review history.
  • Summarization task: Review summarization generates a 3–10-word title from the user’s review history, style, target review, and rating.The output must capture the review’s sentiment and key points in the user’s style.
Loading 2603.25973v1…