PaperlayerView exact revision on arXiv

Source-linked AI summary

VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions

Yuxin Chen, Yi Zhang, Zhengzhou Cai, Yaorui Shi, Zhiyuan Yao, Chenhang Cui, Jingnan Zheng, Yaqi Huo, Xi Su, Qi Gu, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua

arXiv:2605.27141v1cs.AI

TL;DR

Existing agent benchmarks emphasize reasoning and tool use in well-specified tasks, while real-world assistance requires inferring preferences from prior interactions. VitaBench 2.0 evaluates personalization and proactiveness in long-term interactions, and experiments find that current agents still struggle with personalized decision-making.

  • Problem

    Existing agent benchmarks focus on reasoning and tool orchestration in well-specified tasks, with limited evaluation of inferring preferences from prior interactions for under-specified real-world decisions.

    Source: §1 · pp. 1–2

  • Method

    VitaBench 2.0 evaluates preference extraction, utilization, and updating in long-term user-task sequences, as well as proactiveness and memory architectures.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Current agents struggle to reliably infer, utilize, and update user preferences, especially as preferences evolve or information is incomplete.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Failure analysis identifies capturing and using user preferences as the primary bottleneck, with agents often missing preferences in interaction histories or neglecting them in decisions.

    Source: §4.3 · p. 9

    Programmatically constructed preferences and interaction histories may not capture the full diversity of real-world user behavior.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.

1 Introduction

VitaBench 2.0 evaluates personalized and proactive behavior in long-term user interactions, where agents must infer evolving preferences from fragmented context and acquire missing information. The benchmark exposes persistent challenges for current agents and provides a framework for studying memory, personalization, and proactiveness.

1 Introduction

  • Existing benchmarks emphasize explicit instructions, multi-step reasoning, and tool orchestration, leaving under-specified intent and evolving preferences insufficiently evaluated [14] [15] [16] [17].Real-world assistance also requires consistent preference representations and proactive information gathering [18].Source: §1 · pp. 1–2
  • VitaBench 2.0 evaluates preference extraction, utilization, and updating, plus proactiveness when agents must acquire missing information before deciding.Preferences may be implicit, fragmented, conditional, and evolving over time.Source: §1 · p. 2
  • Tasks form temporally ordered, multi-domain sequences for individual users, embedding fine-grained preferences in dialogue and behavior and providing per-user memory interfaces for controlled comparisons [20] [21].Each task has dedicated tools and an executable environment; interactions over time reflect preference evolution.Source: §1 · p. 2
  • Current agents struggle with real-world personalization, and memory designs yield markedly different outcomes despite memory’s importance for long-term user modeling.The authors also identify failure patterns and capability bottlenecks that help explain these difficulties.Source: §1 · p. 2

2 Related Work

Prior work develops personalization methods and evaluates personalized language models and agents, but existing benchmarks leave a gap in realistic settings that combine personalization with action. VitaBench 2.0 addresses this gap by jointly evaluating personalization and agentic execution.

2 Related Work

  • Personalization methods use retrieved context [23] [24] [25] or profiles [26] [27], model adaptation [28] [30], personalized objectives [31] [32] [33], and external memory [34] [35].External memory supports storing, retrieving, and updating user-relevant information over time.Source: §2 · pp. 2–3
  • Personalization benchmarks cover profiles [36] [37] [38], user documents [39] [40], and evolving interaction histories [41] [42] [43] [10] [9] [44], shifting from static inputs toward long-term memory and dynamic modeling.They nevertheless largely assess passive text-in/text-out generation rather than action.Source: §2 · p. 3
  • Agent benchmarks evolved from tool accuracy [46] [47] through stateful execution [48] [49] [50] [51] [52] to realistic environments [14] [53] [54] [15] [16] [17] [55] [56] [57], yet typically omit personalization.They often assume task-relevant information is available in the current context; this work jointly evaluates personalization and agentic execution in realistic interactive settings.Source: §2 · p. 3

3 VitaBench 2.0

VitaBench 2.0 evaluates personalization and proactiveness across temporal task sequences for individual users, requiring agents to infer evolving preferences from fragmented interactions and maintain them in memory. Its tasks also test whether agents recognize when information is missing and seek clarification or explore before acting.

3.1 Task Set

  • Tasks span multiple domains and require agents to combine user intent, tool use, executable environments, and fragmented interaction histories to make consistent personalized decisions.Task accuracy is evaluated at trajectory and outcome levels using an LLM and rubrics that decompose success into atomic criteria; the benchmark covers Delivery, In-store Consumption, and Online Travel Agency, with 66 tools.Source: §3.1 · pp. 3–4

3.2 Key Module

  • The benchmark curates 56 users with over 2,000 preferences across daily-life domains, including temporally grounded additions, deletions, and modifications that model preference drift.Profiles reflect dimensions such as demographics, geography, socioeconomic attributes, occupation, and social context; preferences are expressed in natural language and grounded in profiles.Source: §3.2 · p. 5
  • Agents receive fragmented histories of dialogues and behavior logs containing both preference signals and irrelevant, ambiguous, or misleading interactions, which they must distinguish.The benchmark supports persistent per-user memory through UPDATE and RETRIEVE operations, enabling comparison of agent-controlled structured memory with similarity-based RAG memory.Source: §3.2 · pp. 5–6

3.3 Proactiveness

  • Proactive tasks require agents to identify necessary context that cannot be inferred from memory or the query, then clarify with the user or explore the environment before acting.These tasks are interleaved with standard personalization tasks, requiring agents to decide adaptively when additional context is needed.Source: §3.3 · p. 6

4 Experiment

Experiments find that current agents struggle to reliably personalize decisions from user preferences, manage long interaction histories and memory, and proactively seek missing information. Even with ground-truth preferences or reasoning modes enabled, performance remains limited or inconsistent.

4.1 Experimental Setups

  • The evaluation covers proprietary and open LLMs in thinking and non-thinking configurations, using four deterministic runs per task and reporting Avg@4, Pass@4, and Passˆ4.Agents use function calling, interactions have no predefined step limit, and gpt-4.1-2025-04-14 serves as user simulator and evaluator. Avg@4 is mean performance, Pass@4 is success in at least one of four trials, and Passˆ4 is success in all four.Source: §4.1 · p. 7

4.2 Main Results

  • Even with full interaction history, state-of-the-art models achieve only around 0.5 Avg@4 and 0.3 Passˆ4, showing that real-world personalization remains challenging.The authors attribute the difficulty to agents’ struggles to infer and use preferences, even in simplified tasks, and describe personalization as a new bottleneck relative to reasoning-intensive domains.Source: §4.2 · p. 7

4.3 Analysis & Discussion

  • On VitaBench 2.0, proactive-task performance is consistently below personalization performance: Claude scores 27.4 vs. 46.0, DeepSeek 27.8 vs. 44.1, and GLM 19.3 vs. 36.4.Agents often proceed with incomplete information rather than initiating clarification.Source: §4.3 · p. 9
  • With ground-truth preferences, DeepSeek and Claude score 52.7 and 51.2, while Seed scores 49.3, Kimi 43.8, and MiniMax 42.6.Performance improves over the standard setting but remains far from optimal, indicating challenges in reasoning over, prioritizing, and consistently applying preferences even when profiles are accurate.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Preference-related errors dominate the analyzed failures: agents often miss or ignore user preferences, while stronger models show fewer tool errors but retain personalization difficulties.Claude-Opus-4.6 analyzed DeepSeek-V4-Pro and DeepSeek-R1 trajectories; a common failure was choosing popular or highly rated items instead of those aligned with inferred user preferences.Source: Figure 5 · p. 8, §4.3 · p. 9

4.2 Main Results

  • Most models perform worse with either agentic memory or pipeline-based RAG than with full context, exposing a persistent challenge in memory use for long-term personalization.The authors conclude that current agents do not yet use memory effectively and that memory design remains a key challenge.Source: §4.2 · p. 8
  • Enabling thinking modes does not consistently improve Avg@4 or reduce turns, and overall reasoning gains are modest and inconsistent across settings.The authors identify robust preference extraction, long-term consistency, and handling noisy or incomplete observations as capabilities needed beyond general reasoning.Source: §4.2 · p. 8

4.3 Analysis & Discussion

  • Performance generally declines as tasks progress through users’ interaction sequences, with memory-based settings further affected by information loss and accumulated errors.Repeated UPDATE and RETRIEVE operations can propagate early inaccuracies to later tasks; full-context settings also show difficulty extracting relevant signals from long histories.Source: §4.3 · pp. 8–9

5 Limitation

VitaBench 2.0 prioritizes controllability and comparability, but its constructed histories, abstracted memory interface, and rubric-driven evaluation do not capture every aspect of real-world personalization and proactiveness.

5 Limitation

  • Programmatically constructed preferences and interaction histories enable reproducible control but may not reflect the full diversity of real-world user behavior.The benchmark controls preference dynamics and task difficulty.Source: §5 · p. 9
  • The memory interface isolates memory through update and retrieval operations, but does not cover all possible end-to-end architectures.Source: §5 · p. 9
  • Rubric-driven assessment provides structured, interpretable signals over task trajectories, while open-ended measures of user satisfaction remain out of scope.Source: §5 · p. 9
  • These design choices prioritize controllability and comparability, positioning VitaBench 2.0 as a complementary testbed for core challenges in personalized and proactive agents.Source: §5 · p. 9

6 Conclusion · Appendix

VitaBench 2.0 evaluates personalization and proactiveness in LLM agents through user-centric sequences of evolving preferences, fragmented histories, and incremental exposure. Experiments find that agents struggle with preference modeling, while the benchmark provides a unified testbed for studying memory, personalization, and proactive behavior.

6 Conclusion

  • VitaBench 2.0 benchmarks personalization and proactiveness in LLM-based agents using user-centric task sequences.Preferences evolve within fragmented interaction histories that are incrementally exposed to agents.Source: §6 · p. 10
  • The benchmark tests whether agents can infer and update user preferences over time from fragmented histories.Its task design embeds evolving preferences and incrementally reveals interaction histories.Source: §6 · p. 10
  • An extensible memory interface enables controlled comparison of memory mechanisms within a unified framework.Source: §6 · p. 10
  • Experiments across frontier models find that agents struggle to reliably infer, use, and update user preferences.The difficulty is especially pronounced when preferences evolve or information is incomplete.Source: §6 · p. 10
  • Further analysis identifies failure modes and difficulties in long-term preference modeling.Source: §6 · p. 10
  • VitaBench 2.0 exposes a gap between current LLM agents and realistic personalized assistants, and offers a testbed for advancing research.The research areas include memory, personalization, and proactive behavior.Source: §6 · p. 10

A Benchmark Construction … A.1.4 Prompt Template

VitaBench 2.0 models long-term, user-centric interactions with temporally ordered tasks for 56 curated users, testing preference inference, preference evolution, and proactive decisions. Its construction combines manually curated user data with controlled synthesis of fragmented, evolving interaction histories.

A Benchmark Construction

  • The benchmark organizes tasks as temporally ordered sequences for 56 users across real-world domains to evaluate preference inference, evolution, and proactive decision-making.These sequences are designed to model agents continuously satisfying long-term user needs.Source: A Benchmark Construction · p. 19

A.1 User

  • Because LLM-generated user data can be biased or hallucinated, user profiles and preferences are manually annotated while interaction histories are controlled-synthesized.This construction aims to ensure realism and consistency.Source: §A.1 · p. 19

A.1.1 User Profile

  • Manually curated profiles capture users’ demographic attributes and background, providing the foundation for preference construction and downstream task generation.Profiles are inspired by real-world application scenarios to support diversity and realism.Source: §A.1.1 · p. 19

A.1.2 User Preference

  • The benchmark curates over 1,000 profile-grounded preferences with additions, modifications, and deletions over time; conditional constraints require clarification when relevant context is missing.For example, coffee caffeine level depends on meeting time, so the agent should ask whether the meeting is in the morning or afternoon.Source: §A.1.2 · pp. 19–21

A.1.3 Interaction History

  • Preferences are implicitly encoded in fragmented dialogues and platform behavior logs, with histories between tasks capturing explicit or implicit changes alongside irrelevant, ambiguous, or fluctuating signals.Histories are manually reviewed to remain coherent and avoid contradictions or unintended preference leakage.Source: §A.1.3 · p. 21

A.1.4 Prompt Template

  • The synthesis prompt takes date ranges, profiles, current preferences, preference changes, required chat fragments and behaviors, and noise as inputs for a daily interaction history.Personalized content must remain grounded in the supplied profile and preference information.Source: §A.1.4 · pp. 22, 24
  • Prompt rules require naturally distributed drift signals, developed multi-turn dialogues, required platform behaviors and noise, and chronological, profile-grounded records without invented preferences.Noise must not introduce preferences or contradict the profile; added records must remain grounded in the supplied inputs.Source: §A.1.4 · pp. 22–23

A.2 Task Environment … A.2.3 Prompt Template

VitaBench 2.0 provides structured, executable environments and domain tools for agent tasks, with difficulty centered on inferring and applying user preferences. Its synthesis and prompt templates create realistic candidate spaces with rubric- and preference-based distractors, then validate and refine those environments.

A.2 Task Environment

  • Each task uses a domain-specific environment with structured data and executable tools, enabling agents to interact with simulated real-world services.The environments are designed to expose realistic candidate spaces for tool-based decision-making.Source: §A.2 · p. 24

A.2.1 Toolset

  • The toolset adapts VitaBench [17] across delivery, in-store consumption, and online travel, with 66 service tools and two preference-memory tools.The tools support multi-step interactions while keeping task difficulty focused on preference inference and decision-making rather than tool usage.Source: §A.2.1 · pp. 24–25

A.2.2 Environment Synthesis

  • A multi-agent synthesis pipeline builds task-specific merchant and item databases with relevant candidates and distractors, then applies model checks and expert review for consistency and validity.The pipeline generates merchants before populating them with items and controls how many candidates satisfy all rubric constraints.Source: §A.2.2 · pp. 25–26

A.2.3 Prompt Template

  • Environments include query-relevant candidates that violate rubric constraints and query-compliant candidates that violate inferred user preferences, testing both filtering and preference use.The prompt specifies at least five one-constraint-off distractors for each rubric dimension, plus 3–5 distractors violating multiple dimensions.Source: §A.2.2 · p. 25, §A.2.3 · p. 28
  • The environment prompt specifies JSON records for domain entities and products, labeling targets, query distractors, preference distractors, and irrelevant items with reasons.The template supplies the user profile, current time, task type, query, and evaluation rubric as task inputs.Source: §A.2.3 · p. 27
  • Complexity controls require one globally unique target among 10–15 merchants and at least 40 products total, including at least 30 distractors; place the target after the first five merchants, not highest-scored.Source: §A.2.3 · p. 28
  • Noise injection aims to make literal-query-only agents unreliable while preserving the ability of agents with preference memory to distinguish the target.Injected distractors should satisfy query-derivable constraints, clearly violate at least one preference-specific constraint, and remain retrievable using the query’s keywords and relevant locations.Source: §A.2.3 · pp. 28–29
  • Noise distractors are designed to be highly confusable, using attractive brands, ratings, prices, and names while preserving the real samples’ field structure and diverse options.Each entry must state which preference constraint it violates; the prompt also requires exact schemas and, for store products, a matching parent store name and string-list attributes.Source: §A.2.3 · p. 29

A.3 Benchmark Pipeline … B.1.4 Position

VitaBench 2.0 evaluates long-term personalization and proactive behavior through sequential tasks that require agents to infer, maintain, and use preferences from fragmented histories. The paper also positions the benchmark as a shared framework for comparing memory designs, whose trade-offs include lossy compression, adaptive operations, and retrieval quality.

A.3 Benchmark Pipeline; A.3.1 Task Set

  • VitaBench 2.0 sequences concrete user requests, combining tool and environment interaction with explicit constraints and history-derived preferences while simplifying tool chains to foreground preference modeling.Tasks are grounded in user profiles and may require context-dependent decisions for proactive behavior.Source: §A.3 · p. 30, §A.3.1 · p. 30

A.3.2 User Simulator

  • The user simulator issues tasks and minimal feedback without access to underlying preferences, while proactive-task responses to requests for more information are predefined and controlled.Restricting simulator context is intended to prevent preference leakage that would undermine evaluation validity.Source: §A.3.2 · pp. 30–31

A.3.3 Task Agent

  • Across sequential tasks, agents receive new interaction histories and may update external memory; when memory is enabled, they must rely on it rather than access full histories during execution.Decisions integrate current observations, retrieved memory, and tool feedback under incomplete and noisy observations.Source: §A.3.3 · p. 31

A.3.4 Evaluation

  • Manually curated atomic rubrics support window-level scoring of intermediate actions and an outcome-level reward for whether final decisions align with user preferences.Window scores are aggregated into a trajectory-level reward, complementing the final-outcome assessment.Source: §A.3.4 · p. 31

A.3.5 Prompt Template

  • Prompt templates guide agents to use preference memory when requests lack information and direct memory updates to retain valid preferences, revise contradictions, and add new ones.Updated summaries are structured across dimensions such as food, spending, time, location, and service preferences.Source: §A.3.5 · pp. 32–33

B.1 Memory in LLM Agents; B.1.1 Context Memory

  • The paper distinguishes context, agentic, and RAG memory; context-based approaches preserve bounded context through compression but may discard information needed later.Context memory can use summarization or other long-context approaches, but lossy compression remains a fundamental limitation.Source: §B.1 · p. 34, §B.1.1 · p. 35

B.1.2 Agentic Memory

  • Agentic memory lets agents decide what to store, update, retrieve, or discard, enabling selective preservation but adding credit-assignment, training-complexity, and evaluation challenges.The passage contrasts prompted memory updates with newer approaches that learn memory policies.Source: §B.1.2 · p. 35

B.1.3 RAG Memory; B.1.4 Position

  • VitaBench 2.0 provides a unified, extensible interface to compare context, agentic, and retrieval-based memory for long-horizon personalization; RAG can scale and update continually, but depends on retrieval quality.The paper frames the interface as enabling controlled analysis of how memory design affects preference inference, use, and updating over time.Source: §B.1.4 · p. 35, §B.1.3 · p. 35

B.2 Code of Ethics … C Analysis

VitaBench 2.0 uses annotated or synthetic data without real user information, and is presented as an evaluation framework whose potential benefits and risks motivate safeguards for future systems.

B.2 Code of Ethics

  • The benchmark uses no human subjects, sensitive personal data, or real-world deployment, and releases no private or user-identifiable data.Its scenarios simulate realistic settings without exposing real individuals or proprietary information.Source: §B.2 · p. 35
  • Manual data follows strict guidelines, while synthetic content is reviewed by human annotators for quality, consistency, and harmful or inappropriate content.The work states that it complies with the NeurIPS Code of Ethics.Source: §B.2 · p. 35

B.3 Broader Impacts

  • Personalized and robust agents may improve user experience in recommendation systems, digital assistants, and decision support tools.The paper frames these as potential benefits of developing more reliable and user-aligned assistants.Source: §B.3 · p. 36
  • More effective personalization may be misused to manipulate users, reinforce bias, or enable overly persuasive systems.Errors in preference inference may also produce inappropriate or misleading recommendations, particularly in high-stakes scenarios.Source: §B.3 · p. 36
  • VitaBench 2.0 is an evaluation framework rather than a deployable system, uses no real user data, and constructs scenarios through controlled simulation.The paper encourages future work to use transparency, user control over personalization, and monitoring mechanisms to prevent misuse.Source: §B.3 · p. 36

B.4 Safeguards

  • Dataset safeguards combine manual annotation and LLM-based synthesis with human verification and refinement of generated samples.Interaction histories, profiles, and task environments are reviewed for unintended bias, sensitive content, and unrealistic artifacts; the final dataset contains no real user data.Source: §B.4 · p. 36

C.1 Experiments Compute Resources

Each evaluation covers 56 tasks across four independent trials, with compute costs driven by commercial API latency and rate limits rather than intensive local CPU or GPU demands. In the full-context setting, runtime ranges from approximately 3.7 to 8.6 hours per model-memory configuration, depending on model category.

C.1 Experiments Compute Resources

  • Four independent trials cover roughly 3,276 subtask-level interactions per model-memory configuration across 56 tasks.Each task averages 14.6 subtasks, including multi-turn follow-ups.Source: §C.1 · p. 36
  • Commercial API latency and rate limits bound runtime; evaluations cap requests at 200 RPM and at most 20 concurrent asynchronous user-agent rollouts.The API-based evaluation imposes no intensive local CPU or GPU demands.Source: §C.1 · p. 36
  • In full context, non-reasoning models take approximately 4.3–4.6 hours and reasoning-enabled models take approximately 3.7–8.6 hours per model-memory configuration.Full context provides the complete dialogue history across prior interactions.Source: §C.1 · p. 36

C.2 Benchmark Data Analysis

VitaBench 2.0 is built to evaluate personalized and proactive agents in realistic daily-life scenarios, using user profiles, preferences, and interaction histories guided by real-world application statistics. The section describes the benchmark’s scale and analyzes user profiles, preferences, and interaction patterns.

C.2 Benchmark Data Analysis

  • The benchmark comprises 56 users, 819 tasks, 41,414 environment entities, and 2,286 preference annotations.The passage reports these counts for the benchmark as a whole.Source: §C.2 · p. 37
  • On average, each user has 14.6 tasks, 739.5 environment entities, and 40.8 preferences.These averages indicate a rich and structured user representation.Source: §C.2 · p. 37
  • The dataset analysis covers user profiles, preferences, and interaction patterns, with Figure 6 overviewing user profile statistics.Benchmark construction draws on detailed statistics from real-world application scenarios.Source: §C.2 · pp. 36–37, Figure 6 · p. 37

C.2.1 User Profile Analysis

VitaBench 2.0 constructs user profiles across demographic, geographic, socioeconomic, occupational, and social dimensions using real-world statistics. The resulting population combines demographic concentration with structural diversity, requiring context-dependent personalization rather than simple demographic heuristics.

User profile construction

  • The profiles span demographic, geographic, socioeconomic, occupational, and social dimensions, with attributes designed to reflect real-world statistics and user heterogeneity.The analysis examines demographic distribution, regional and socioeconomic structure, and occupation and social context.Source: §C.2.1 · p. 37

Demographic distribution

  • 62.5% of users are female, while ages 20–29 and 30–39 account for 62.5% and 26.8%, respectively.Users born after 1990 make up 87.5% of the dataset, including 71.4% from Generation Z; the average age is approximately 31.Source: §C.2.1 · p. 37

Regional and socioeconomic structure

  • Approximately 50% of users come from emerging first-tier regions, 28.6% from first-tier cities, and 21.4% from lower-tier regions.Cross-region mobility affects 59% of users, adding geographic contexts that agents must generalize across when modeling preferences.Source: §C.2.1 · pp. 37–38

Occupation and social context

  • 60.7% of users are white-collar workers, 16.1% blue-collar, and 23.2% gray-collar, alongside varied family structures and life stages.Profiles include single people, couples, and families with children, as well as single-child and multi-child households; transitions such as becoming an employee or getting married affect preferences and decisions.Source: §C.2.1 · p. 38

Implications for personalization

  • Demographic concentration combined with regional, occupational, and family diversity means personalization requires fine-grained, context-dependent modeling and adaptation across profiles.The benchmark is designed so personalization cannot be reduced to simple demographic heuristics.Source: §C.2.1 · p. 38

C.2.2 User Preference Analysis … D.1 Case Study: Memory-Sensitive Delivery for User A891207

VitaBench 2.0 models personalized interaction as long-term inference over diverse, changing preferences, noisy histories, and distinct consumption contexts. Its memory configurations and A891207 case study illustrate how agents represent and use those evolving preferences.

C.2.2 User Preference Analysis

  • VitaBench 2.0 includes 17,928 preference annotations across 2,048 types and five categories; users average 40.8 preferences, with food the largest category at 35%.Shopping accounts for 18.1%, travel 20.2%, entertainment 21.0%, and special long-tail cases 5%.Source: §C.2.2 · pp. 38–39, Table 3 · p. 39
  • Preferences change over time—averaging 48.38 changes per user—and are highly individualized, with 92.4% user-specific and 7.6% shared.Changes average 36.07 additions, 7.52 deletions, and 4.79 modifications, so agents must update representations rather than accumulate every historical signal.Source: §C.2.2 · p. 39
  • Online delivery and in-store consumption make up 57.7% and 23.2% of orders, while lower-frequency hotels and travel tickets have higher average order amounts.The benchmark also stratifies users by consumption level; for example, online delivery users are 19.6% high-, 62.5% medium-, and 17.9% low-consumption.Source: §C.2.2 · pp. 39–40, Table 4 · p. 39, Table 5 · p. 40

C.2.3 Interaction Analysis

  • Interaction histories average 1,580 days and 2,093 events per user, span multiple service domains, and include only 221 successful purchase conversions on average.Timelines cover more than 10 years overall, and fragmented, non-contiguous evidence and varied decision trajectories complicate preference inference.Source: §C.2.3 · p. 40
  • Approximately 20% of interactions are noisy or misleading, requiring agents to distinguish stable preferences from irrelevant, exploratory, proxy, impulsive, or corrective behavior.The evaluation tests aggregation of fragmented evidence, handling conflicting signals, and maintaining consistent representations under uncertainty.Source: §C.2.3 · pp. 40–41

C.2.4 Implementation Configurations

  • Agentic Memory rewrites existing memory with each interaction batch, selectively retaining, updating, or adding preferences within a 4,096-token buffer.Its structured prompt organizes preferences by axes such as food taste, spending, time, location, and service requirements; retrieval returns the current memory.Source: §C.2.4 · p. 41
  • RAG Memory indexes fixed 512-token interaction chunks without LLM summarization and retrieves up to eight chunks using cosine similarity and a 0.3 threshold.Unlike Agentic Memory, it does not explicitly decide what to retain or discard; retrieval depends on similarity scores.Source: §C.2.4 · pp. 41–42

D.1 Case Study: Memory-Sensitive Delivery for User A891207

  • The A891207 case study follows three subtasks where Full Context, Agentic Memory, and RAG Memory diverge as dietary preferences shift from spicy to light Cantonese food, add durian, and exclude pork.The cases illustrate how a preference error in memory can cause rubric failure and how each setting represents preference drift.Source: §D.1 · p. 42

D.1.1 Part 1. One Subtask, Three Backends · user · D.1.2 Part 2. Preference Drift Across Three Subtasks

On one preference-laden delivery task, Agentic Memory and Full Context earned reward 1.0, while RAG Memory earned 0.0 after missing the merchant-rating threshold. Across this user’s trajectory, Agentic Memory’s evolving summary retained stable preferences more reliably than RAG retrieval, with a 0.18 reward gap over 20 subtasks.

D.1.1 Part 1. One Subtask, Three Backends

  • The evening dessert task’s seven criteria were under-specified: durian and the ≥4.3 merchant-rating threshold had to come from preference memory, not the instruction.The task also required Cantonese-style tong sui, room temperature, quantity one, delivery within 30 minutes, and the home address.Source: §D.1.1 · p. 42
  • The target and a distractor sold the same durian tong sui, so choosing correctly required applying the ≥4.3 preference and checking the merchant rating.The target was rated 5.0; the distractor was rated 4.0.Source: §D.1.1 · p. 43

user

  • Agentic Memory recalled the user’s durian preference and ordered the matching item; Full Context also selected the target after comparing stores and delivery times.Both approaches produced the same order from S00027, satisfying all seven rubric criteria.Source: user · p. 43
  • Agentic Memory and Full Context each earned reward 1.0, while RAG Memory earned 0.0 after ordering from a merchant rated 4.0, below the ≥4.3 threshold.The RAG order satisfied R1–R5 and R7 but violated R6.Source: user · pp. 43–44
  • RAG retrieved durian-related context but not the rating threshold, skipped rating verification, and fabricated a delivery estimate before selecting the 4.0-rated store.Its coherent-sounding recommendation nevertheless violated R6.Source: user · p. 44

D.1.2 Part 2. Preference Drift Across Three Subtasks

  • Table 6 traces dietary-preference snapshots before subtasks 13, 17, and 20, contrasting Agentic Memory’s evolving summary with RAG Memory’s query-selected records.Agentic Memory crossed out stale preferences, added new ones, and retained stable thresholds; RAG lacked a mechanism to promote stable but off-topic preferences or demote outdated chunks.Source: §D.1.2 · p. 44, Table 6 · p. 45
  • A 0.18 reward gap over the user’s 20 subtasks favored Agentic Memory, driven primarily by preferences that existed in memory but were not retrieved.RAG could be misled by query-matched outdated records or lose dietary context when a query matched no record.Source: §D.1.2 · p. 44

Related papers

Loading 2605.27141v1…