PaperlayerView exact revision on arXiv

Source-linked AI summary

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment

Jianfei Xiao, Xiang Yu, Chengbing Wang, Wuqiang Zheng, Xinyu Lin, Kaining Liu, Hongxun Ding, Yang Zhang, Wenjie Wang, Fuli Feng, Xiangnan He

arXiv:2603.26680v2cs.CLcs.AI

TL;DR

Personalization research lacks benchmarks that capture real conversational signals and the full memory lifecycle. AlpsBench evaluates four personalization tasks using real-world dialogues and human-verified memories. Its results show persistent challenges in extraction, updating, retrieval, and preference-aligned utilization.

  • Problem

    Personalization benchmarks need to capture real-world conversational diversity and implicit signals while evaluating how systems understand, store, and use personalized information.

    Source: §1 · p. 2

  • Method

    AlpsBench evaluates extraction, updating, retrieval, and utilization using 2,500 real-world human–LLM dialogues paired with human-verified structured memories.

    Source: §1 · p. 2

  • Results

    Models struggle to extract latent user traits, updating remains challenging, retrieval degrades with large distractor sets, and explicit memory does not ensure preference-aligned or emotionally resonant responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AlpsBench provides a framework for diagnosing LLM personalization across the full lifecycle of personalized information.

    Source: §6 · p. 7

    Injected memories may cause models to over-rely on them, confuse hypothetical information with real user attributes, and respond less emotionally adaptively.

    Source: §4.1.4 · p. 7

Abstract

from arXiv · show

As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation benchmark. Existing benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management. Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework.

1 Introduction

Existing personalization benchmarks either assess response alignment without evaluating memory governance or use synthetic dialogues that miss real-world diversity and implicit signals. AlpsBench addresses these gaps with real-world conversations, four lifecycle tasks, and findings that current models remain limited in long-term personalization.

1 Introduction

  • Memory-free benchmarks assess preference alignment in final outputs but overlook information governance, while synthetic-dialogue benchmarks can miss conversational diversity and implicit personalization signals.Source: §1 · pp. 1–2
  • The benchmark’s design responds to the need to assess how personalized information is understood, stored, and used, rather than checking only final outputs.Source: §1 · p. 2
  • The benchmark evaluates personalized information extraction, updating, retrieval, and utilization, with utilization assessed across five capabilities including preference following and emotional intelligence.Source: §1 · p. 2
  • AlpsBench includes 2,500 real-world human–LLM dialogues of 6 to 249 turns and uses human verification to check memories, task instances, and evaluator alignment.Its four-step curation pipeline supports continuous updates as real-world conversations evolve.Source: §1 · p. 2
  • Frontier models struggle to extract latent user traits, updating remains challenging even for the strongest models, retrieval degrades with large distractor sets, and explicit memory does not ensure preference-aligned or emotionally resonant responses.Source: §1 · p. 2

2 AlpsBench Evaluation Framework

AlpsBench evaluates four complementary capabilities of personalized AI assistants, spanning the transformation of dialogue history into memories and the use of personal information in responses.

2.1 Task 1: Personalized Information Extraction

  • Task 1 turns dialogue history into structured memories with explicit or implicit type, category, value, and confidence, evaluated by semantic-matching F1 against human annotations.An LLM-as-a-Judge protocol matches predicted and ground-truth memories one-to-one.Source: §2.1 · p. 3

2.2 Task 2: Personalized Information Update

  • Task 2 tests noise filtering, adding preferences, and resolving conflicts through retention, addition, or modification, with updated-memory F1 as its primary metric.Action-level accuracy and other diagnostic metrics are available on the leaderboard.Source: §2.2 · p. 3

2.3 Task 3: Personalized Information Retrieval

  • Task 3 retrieves the human-labeled relevant memory for a user query from candidate memories, including K random negatives, and reports recall.The task evaluates retrieval from a candidate pool containing one positive sample and randomly sampled negative samples.Source: §2.3 · p. 3

2.4 Task 4: Personalized Information Utilization

  • Task 4 evaluates responses across persona awareness, preference following, virtual-reality awareness, constraint following, and emotional intelligence.Judges assign binary scores except for emotional intelligence, which receives a 1–5 score.Source: §2.4 · pp. 3–4

3 Curation Pipeline for AlpsBench

AlpsBench is curated from long-term WildChat dialogues through memory extraction, task construction, and human verification, with filtering that prioritizes users whose memories include implicit information.

3 Curation Pipeline for AlpsBench

  • The source is WildChat’s roughly 90,000 users; selected dialogues span 6–249 turns, and DeepSeek-v3.2 extracts structured memories.Longer dialogues were selected to capture richer contextual dependencies and cross-temporal information.Source: §3 · p. 4
  • Category caps reduce semantically redundant content, while retaining users with implicit memories prioritizes implicit personalization signals.The filtering process yields 2,500 user samples for the final benchmark.Source: §3 · p. 4
  • GPT-5.2 synthesizes task queries and answers from dialogue history and target memory; update strategies and retrieval distractors serve as task-specific inputs.Human review converts the generated query-answer pairs into the benchmark instances.Source: §3 · p. 4
  • Each task has 2,500 verified instances independently examined by two annotators, with disagreements resolved through manual adjudication.Tasks 1 and 2 also report inter-annotator memory-set F1 scores of 0.763 and 0.844, respectively.Source: §3 · p. 4
  • 96% (Task 3), 85% (Task 2), 83% (Task 4), and 73% (Task 1) are the judge–human alignment rates across the four tasks.The authors report that automated evaluation reflects human-centric assessment while remaining scalable to the full benchmark.Source: §3 · pp. 4–5, Figure 5 · p. 5

4 Experiments

Across the four AlpsBench tasks, general-purpose LLMs show extraction and updating limits, retrieval performance varies with distractor load, and utilization strengths differ by dimension. Memory-oriented systems can improve coverage or selected utilization dimensions, but also exhibit precision–recall trade-offs and selective regressions.

4.1 Performance on AlpsBench

  • 0.6742 Task 1 F1 (DeepSeekv3.2) narrowly leads 0.6720 (GPT-5.2), while extraction precision and recall trade off across general-purpose LLMs.Qwen3-max has the highest precision (0.8104) but recall of 0.6190; Claude-Sonnet-4.5 and Gemini-3 Flash have recall of 0.8373 and 0.8175, respectively, with lower precision.Source: §4.1.1 · p. 5, Table 2 · p. 6

4.1.1 Task 1: Extraction Result Analysis.

  • 0.5386 vs. 0.4716 (DeepSeek-v3.2) and 0.5526 vs. 0.0032 (GPT-4.1-mini) show direct recall outpaces indirect recall, especially for weaker reasoning models.Llama-4 Maverick also drops from 0.1862 direct recall to 0.0337 indirect recall; the authors associate the gap with implicit information requiring stronger contextual inference.Source: Figure 6 · p. 5, §4.1.1 · p. 5

4.1.2 Task 2: Update Result Analysis.

  • 0.7793 Task 2 F1 (GPT-5.2) leads, but top general-purpose model scores cluster closely, indicating that updating remains challenging under this evaluation setting.Gemini-3 Flash scores 0.7686; DeepSeek-v3.2 and Claude-Sonnet-4.5 each score 0.7543.Source: §4.1.2 · p. 5, Table 2 · p. 6

4.1.3 Task 3: Retrieval Result Analysis.

  • 0.8958–0.9728 retrieval scores across models at 100 distractors contrast with Gemini-3 Flash and DeepSeek-v3.2 each exceeding 0.94 at 1,000 distractors.The authors note that context-window limits and long-context token costs motivate more lightweight retrieval architectures for long-horizon scenarios.Source: §4.1.3 · pp. 5–6, Figure 7 · p. 6

4.1.4 Task 4: Utilization Result Analysis.

  • No general-purpose LLM consistently dominates utilization: Gemini-3 Flash leads Persona Awareness and general Preference Following, while other models lead other dimensions.Claude-Sonnet-4.5 leads Virtual–Reality Awareness; DeepSeek-v3.2 leads Constraint Following and Chinese Emotional Intelligence; Qwen3-max leads English Emotional Intelligence.Source: Table 2 · p. 6, §4.1.4 · p. 6

4.1.1 Task 1: Extraction Result Analysis.

  • 0.9340 recall (EverMemOS) is highest among memory-oriented systems, but Mem0 leads precision (0.4460) and F1 (0.4570), illustrating a coverage–precision trade-off.The authors conclude that memory systems also need to filter and consolidate stored information to avoid noisy or weakly personalized memories.Source: §4.1.1 · p. 5, Table 3 · p. 6

4.1.3 Task 3: Retrieval Result Analysis.

  • 0.2102 (nltk+BM25) and 0.4910 (all-MiniLM-L6-v2) retrieval performance decline as distractor density scales, unlike frontier LLM retrieval.The authors characterize plug-and-play semantic retrieval as inadequate for the complexities of long-term personalization.Source: §4.1.3 · p. 6

4.1.4 Task 4: Utilization Result Analysis.

  • 0.7246 vs. 0.4112 Persona Awareness (EverMemOS vs. GPT-4.1-mini) illustrates selective gains, while memory systems underperform the backbone on Virtual–Reality Awareness and Emotional Intelligence.The authors warn that injected memories may be over-relied on, confusing hypothetical information with real user attributes and reducing emotional adaptability.Source: Table 4 · p. 6, §4.1.4 · pp. 6–7

5 Related Work

Related work spans methods for personalizing models and benchmarks for evaluating personalization. The benchmark literature divides into memory-free and memory-aware approaches, with the latter often relying on synthetic dialogues.

5 Related Work

  • Personalized prompting, adaptation, and alignment represent three dimensions of LLM personalization research.The approaches described include persistent or agentic memory, parameter-efficient adaptation, and training-objective optimization.Source: §5 · p. 7
  • Memory-free benchmarks evaluate preference alignment in downstream outputs or against profiles, but overlook information governance such as memory extraction, retrieval, and updating.Examples include LaMP, LongLaMP, LaMP-QA, PersonalLLM, PersonaFeedback, and PersoBench.Source: §5 · p. 7
  • Memory-aware benchmarks assess memory capabilities across dimensions, but reliance on synthetic human–LLM dialogues creates a distribution gap from real-world interactions.The cited work describes synthetic conversations as homogeneous and lacking natural diversity and implicit expressions characteristic of authentic interactions.Source: §5 · p. 7

6 Conclusion and Future Work

AlpsBench evaluates the complete lifecycle of LLM personalization using real-world dialogue data and expert-verified structured memories. Experiments identify difficulties with implicit information, preference drift, and retrieval under heavy interference.

6 Conclusion and Future Work

  • AlpsBench evaluates the complete lifecycle of LLM personalization using real-world dialogue data and expert-verified structured memories.Its four dimensions are personalized information extraction, updating, retrieval, and utilization.Source: §6 · p. 7
  • Experiments highlight challenges in interpreting implicit user information, handling preference drift, and maintaining retrieval reliability under heavy interference.Source: §6 · p. 7
  • The authors plan to maintain and update the dataset so it can serve as a dynamic benchmark for emerging LLMs.Source: §6 · p. 7

Related papers

Loading 2603.26680v2…