Source-linked AI summary
VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
Yuxin Chen, Yi Zhang, Zhengzhou Cai, Yaorui Shi, Zhiyuan Yao, Chenhang Cui, Jingnan Zheng, Yaqi Huo, Xi Su, Qi Gu, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua
TL;DR
Existing benchmarks largely miss the challenge of inferring and applying evolving user preferences from fragmented interactions. VitaBench 2.0 evaluates personalized and proactive agents in long-term user sequences, finding that current agents still struggle to reliably personalize decisions, especially with evolving or incomplete information.
Problem
Existing agent benchmarks emphasize explicit instructions, reasoning, and tool orchestration, but insufficiently evaluate inferring, maintaining, and updating user preferences in under-specified real-world interactions.
Method
VitaBench 2.0 organizes multi-domain tasks into temporally ordered user sequences with fragmented histories, evolving preferences, proactive information acquisition, and an extensible memory interface.
Results
Current agents struggle to reliably infer, utilize, and update user preferences, particularly when preferences evolve or information is incomplete.
Takeaways & Limitations
VitaBench 2.0 shows that real-world personalized decision-making remains substantially beyond current agent capabilities and practical requirements.
Takeaways & Limitations
The benchmark’s programmatically constructed preferences and interaction histories may not fully capture the diversity of real-world user behavior.
Abstract
from arXiv · showhide
Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.
1 Introduction
VitaBench 2.0 evaluates personalized and proactive agent behavior in long-term user interactions, addressing benchmarks that largely overlook implicit, under-specified user intent. It measures preference extraction, utilization, and updating, and finds real-world personalization remains highly challenging for current agents.
- Motivation: Existing agent benchmarks primarily assess multi-step reasoning and tool orchestration with explicit instructions and clearly stated task context.These settings mainly measure whether agents follow instructions and execute correct action sequences.
- Benchmark: VitaBench 2.0 evaluates personalized and proactive behavior in real-world long-term user interactions beyond reasoning and tool use.The benchmark targets settings where user intent is under-specified and must be inferred from prior interactions.
- Benchmark: The benchmark measures personalization through preference extraction, utilization, and updating.These dimensions respectively cover inferring implicit preferences, using them for user-specific decisions, and updating them over time.
- Method: Tasks are temporally ordered sequences for individual users spanning multiple domains, with dedicated tools and executable environments for realistic interaction.This interactive setup follows the general structure of existing agent benchmarks while supporting long-term user modeling.
- Results: Evaluations across frontier proprietary and open-source models show that real-world personalization remains highly challenging and exposes a substantial gap between current capabilities and practical requirements.The analysis also finds memory mechanisms essential for long-term user modeling, but existing approaches often fail to consistently translate stored information into improved behavior.
2 Related Work
Prior work has established personalization and agent benchmarks as important evaluation areas, while memory-augmented methods support storing, retrieving, and updating user information over time. Existing benchmarks vary in the user-specific information they evaluate and have progressed from isolated tool use toward more realistic interactive task execution.
- Personalized LLM: Personalized LLM research targets outputs aligned with individual user needs and preferences.As user interactions become longer and more informative, personalization increasingly relies on capturing user-specific characteristics.
- Personalized LLM: Memory-augmented personalization equips LLMs with external mechanisms for storing, retrieving, and updating user-relevant information over time.This direction is supported by advances in memory systems and LLMs’ increasing ability to use them.
- Benchmarks for LLM Personalization: Personalization benchmarks differ by both the form of user-specific information and the evaluation setting.Evaluated information includes explicit profiles, user-authored documents, and interaction histories with implicit or…
- Benchmarks for LLM Agents: Agent benchmarks have advanced from isolated tool-use evaluation toward increasingly realistic interactive task execution.Early benchmarks primarily measure API invocation and tool-use accuracy, including tool selection and valid argument generation for user requests.
3 VitaBench 2.0
VitaBench 2.0 evaluates personalized and proactive agents through temporally ordered, multi-domain tasks in which preferences are fragmented across interactions and evolve over time. It supports controlled memory comparisons and tests whether agents recognize missing information and proactively acquire it before deciding.
- Benchmark design: VitaBench 2.0 models each user with an evolving preference set and temporal task sequence, requiring agents to infer, maintain, and leverage preferences across tasks.Tasks simulate long-term user–agent collaboration and continuous satisfaction of user needs.
- Benchmark design: Tasks span multiple domains and model agent–environment interaction as partially observable decision processes involving domain-specific tools and executable environments.Each task includes states, actions, observations, transitions, and task rewards or evaluation functions.
- Benchmark design: Successful execution requires combining explicit query constraints, appropriate tool selection, and user-preference inference from fragmented interaction histories.Agents use histories exposed between tasks to identify intent and make consistent, personalized decisions.
- Benchmark implementation: The benchmark covers three domains—Delivery, In-store Consumption, and Online Travel Agency—and includes a total of 66 tools.Task accuracy is evaluated at trajectory and outcome levels by an evaluator LLM against rubric-defined atomic criteria.
- Memory and proactiveness: VitaBench 2.0 defines an extensible UPDATE-and-RETRIEVE memory interface and implements agentic and retrieval-augmented memory mechanisms for controlled personalization studies.Agentic Memory selectively retains, updates, or discards information, whereas RAG Memory uses similarity-based retrieval from an embedding-indexed memory bank.
- Memory and proactiveness: Proactive tasks test whether agents recognize when memory and the current query lack sufficient context, then seek user clarification or explore the environment.Successful completion depends on acquiring information that cannot be inferred from stored preferences or the query alone.
4 Experiment
VitaBench 2.0 evaluates diverse LLM agents under controlled function-calling implementations and four-run metrics. Results show that personalization, memory management, long-context handling, and proactive information acquisition remain substantial challenges.
- Models: The benchmark evaluates diverse proprietary and open LLMs, including non-thinking and thinking configurations when available.Evaluated families include OpenAI, DeepSeek, Anthropic Claude, and Google Gemini variants.
- Implementations: All agents use OpenAI-schema function calling, run without a predefined step limit, and terminate at “###STOP###” or failure.The user simulator and evaluator is gpt-4.1-2025-04-14, and each task is run four times at temperature 0.0.
- Metrics: Avg@4, Pass@4, and Passˆ4 summarize mean performance, at-least-one success, and all-four-trial success across four independent runs.These metrics are averaged over all tasks.
- Main results: Even with Full Context, state-of-the-art models achieve only around 0.5 Avg@4 and around 0.3 Passˆ4 on real-world personalization tasks.The result indicates difficulty reliably inferring and using preferences despite controlled tool complexity and instruction difficulty.
- Main results: Most models perform worse with memory than with Full Context, across both agentic memory and pipeline-based RAG memory.Longer task sequences also degrade performance, with memory settings suffering additional information loss and error accumulation from repeated UPDATE and RETRIEVE operations.
- Main results: Thinking modes do not consistently improve VitaBench 2.0 performance, indicating that reasoning advances alone are insufficient for personalized decision-making.The paper highlights robust preference extraction, long-term consistency, and handling noisy or incomplete observations as additional requirements.
- Main results: Proactive performance is consistently lower than personalization across model families, while ground-truth preferences improve results but remain difficult to leverage.DeepSeek and Claude reach 52.7 and 51.2 with ground-truth preferences, and failure analysis identifies personalization as the primary bottleneck.
5 Limitation
VitaBench 2.0 enables controlled, reproducible evaluation but has limitations in how closely its constructed interactions and abstracted memory interface represent real-world use.
- Limitations: Programmatically constructed user preferences and interaction histories may not capture the full diversity of real-world user behavior.This construction supports precise control over preference dynamics and task difficulty and facilitates reproducible evaluation.
- Limitations: The benchmark abstracts memory into update and retrieval operations.
6 Conclusion · Appendix
VitaBench 2.0 is introduced as a benchmark for evaluating personalization and proactiveness in LLM-based agents, using user-centric sequences with evolving preferences and an extensible memory interface. The appendix documents benchmark construction, task environments, pipeline components, memory approaches, ethics and safeguards, analyses, configurations, and case studies.
- 6 Conclusion: VitaBench 2.0 evaluates personalization and proactiveness in LLM-based agents through user-centric sequences with evolving preferences embedded in fragmented interaction histories.The benchmark incrementally exposes these histories to assess preference inference and updating over time.
- 6 Conclusion: The benchmark provides an extensible memory interface for controlled comparison of different memory approaches.This interface supports systematic analysis of how agents handle user information over time.
- Appendix: The appendix describes benchmark construction through user profiles, user preferences, interaction histories, and prompt templates.These components are organized under sections A.1, A.1.1, A.1.2, A.1.3, and A.1.4.
- Appendix: The appendix documents the task environment, including its toolset, environment synthesis, and prompt template.These components appear in sections A.2, A.2.1, A.2.2, and A.2.3.
- Appendix: The benchmark pipeline appendix covers the task set, user simulator, task agent, evaluation, and prompt template.These components are listed in sections A.3 through A.3.5.
- Appendix: Additional appendix sections provide compute-resource details, benchmark data analyses, implementation configurations, and two memory-focused case-study analyses.The listed analyses cover user profiles, user preferences, interactions, delivery across three backends, and preference drift across three subtasks.
- Appendix: The appendix further lists interaction analysis, implementation configurations, and case studies on memory-sensitive delivery and preference drift.These materials are organized in sections C.2.3, C.2.4, D.1, D.1.1, and D.1.2.
A Benchmark Construction … A.1.4 Prompt Template
VitaBench 2.0 models long-term user-centric interaction through 56 users, manually curated profiles and preferences, temporally ordered fragmented histories, and controlled synthesis. Its prompt template distributes preference drift across realistic dialogues and behaviors while enforcing chronological, traceable, non-invented records.
- A Benchmark Construction: VitaBench 2.0 covers 56 curated users with temporally ordered task sequences spanning diverse real-world domains.This design evaluates preference inference, preference evolution, and proactive decision-making in realistic settings.
- A.1 User: Realistic user data combines manually annotated profiles and preferences with controlled synthesis of interaction histories.The construction addresses biases and hallucination issues that make LLM-only user-data generation insufficient.
- A.1.1 User Profile: Each user profile is manually curated from demographic attributes and background information to support diverse, realistic preference and task construction.Profiles provide the foundation for preference construction and downstream task generation.
- A.1.2 User Preference: The benchmark manually annotates over 1,000 heterogeneous user-specific preferences spanning daily-life domains and representing additions, updates, and deletions.Preferences are expressed as natural-language statements grounded in corresponding user profiles, with change histories tracking preference evolution.
- A.1.2 User Preference: Proactive preference examples require agents to resolve missing context before acting, such as asking when a meeting occurs to select caffeine level.The example maps an afternoon meeting to low-caffeine coffee delivered to the user’s workplace.
- A.1.3 Interaction History: Interaction histories hide preferences in two modalities: multi-turn dialogues and platform behaviors such as browsing, ordering, reviewing, and searching.Histories are synthesized from manually designed preference signals and extended between consecutive temporally ordered tasks.
- A.1.4 Prompt Template: The synthesis prompt makes preference drift the primary target, requiring added, modified, and deleted preferences to emerge naturally and distribute across histories.It also specifies multi-turn dialogue and standalone behavior requirements, realistic noise, chronological daily records, and traceability to supplied inputs.
A.2 Task Environment … A.2.3 Prompt Template
VitaBench 2.0 provides executable, domain-specific environments with structured tools and synthesized candidate spaces for realistic agentic tasks. Its prompt templates enforce unique targets, adversarial distractors, explicit preference violations, and controlled complexity so agents must use tools and user preferences to identify correct actions.
- A.2 Task Environment: VitaBench 2.0 uses realistic, domain-specific environments with executable tool interfaces and structured data for tool-based decision making.The environments simulate real-world service scenarios in which agents interact with tools to fulfill user requests.
- A.2.1 Toolset: 66 tools span Delivery, In-store Consumption, and Online Travel Agency, exposing structured APIs for multi-step state retrieval and manipulation.The toolset also includes two personalization tools: read_preference_memory and query_preference_memory.
- A.2.2 Environment Synthesis: Environment synthesis uses specialized agents to generate merchants, items, valid candidates, and distractors from each user profile, query, and evaluation rubric.The benchmark contains thousands of tasks across food delivery, in-store services, and travel booking, requiring rich candidate spaces.
- A.2.2 Environment Synthesis: Top-down synthesis generates merchants before items and controls the number of products satisfying all rubric constraints while verifying logical consistency.This design ensures each task has a well-defined solution while remaining sufficiently challenging.
- A.2.3 Prompt Template: The environment-synthesis template requires a structured JSON database grounded in the user profile, current time, task type, query, and evaluation rubric.Its schema distinguishes target, distraction, query_distractor, preference_distractor, and irrelevant entries with explicit reasons.
- A.2.3 Prompt Template: The synthesis protocol makes the target globally unique and imposes 10–15 merchants, 4–8 products per merchant, at least 40 products, and at least 30 distractors.The target must not appear in the first five merchants, and its merchant must not be the highest-scored merchant.
- A.2.3 Prompt Template: Query-relevant distractors violate rubric constraints, while preference distractors satisfy the literal query and most rubric constraints but violate latent user preferences.The template requests at least five one-constraint-off distractors per rubric dimension, plus 3–5 multi-dimensional violations, and distributes preference distractors across merchants.
- A.2.3 Prompt Template: The noise-injector template creates plausible query-level distractors that satisfy every Set A constraint but crisply violate at least one Set B preference constraint.This is intended to make preference-free agents unreliable while allowing preference-aware agents to separate the target from distractors.
A.3 Benchmark Pipeline … B.1.4 Position
VitaBench 2.0 models personalized assistance as sequential user-agent interaction requiring agents to infer, use, and update preferences across fragmented histories. Its unified memory interface enables controlled comparison of context, agentic, and retrieval-based memory mechanisms, while evaluation uses rubric-based task success.
- A.3 Benchmark Pipeline: The benchmark presents temporally ordered tasks in which agents continuously fulfill user needs through domain-specific tools and an executable environment.New interaction histories appear between tasks, and agents may update internal memory to maintain user-specific information over time.
- A.3.1 Task Set: Tasks are manually grounded in user profiles, span diverse everyday domains, and require agents to capture, utilize, and maintain one or more preferences over time.The task set combines explicit constraints with preference-related difficulty.
- A.3.2 User Simulator: A user simulator issues task instructions and provides only minimal, controlled feedback, withholding preference-related signals that would reduce task difficulty.In proactive tasks, additional information is supplied upon request through predefined responses.
- A.3.3 Task Agent: The task agent is an evaluated LLM that solves sequential requests using tools, the environment, newly generated interaction histories, and optional external memory.These histories reflect fragmented user behaviors and evolving preferences.
- A.3.4 Evaluation: Task success is assessed with manually curated rubrics that decompose requests into atomic constraints and are applied to interaction trajectories by a strong evaluator model.Rubric constraints include item attributes, price ranges, and temporal conditions, supporting structured and interpretable evaluation aligned with user preferences.
- A.3.5 Prompt Template: The prompt templates coordinate simulator behavior, task-agent preference use, memory updates, and rubric-based evaluator judgments across the benchmark pipeline.They specify roles, current tasks, interaction and disclosure rules, memory-update requirements, and JSON-formatted evaluation outputs.
- B.1 Memory in LLM Agents: Prior memory research is organized into context memory, agentic memory, and RAG-based memory, distinguished by where memory is stored and how it is accessed.Context methods retain information in the working context, agentic methods actively store, update, retrieve, or discard information, and RAG methods retrieve from external embedding- or graph-based systems.
- B.1.4 Position: VitaBench 2.0 provides a unified, extensible interface to isolate and compare context-based, agentic, and retrieval-based memory for inferring, using, and updating preferences over time.Context-based approaches can lose information through lossy compression, motivating systematic comparison of memory paradigms in personalized behavior.
B.2 Code of Ethics … C Analysis
VitaBench 2.0 follows ethical safeguards for manually annotated and synthetically generated benchmark data without human subjects, sensitive personal data, or real-world deployment. The benchmark may support more reliable, user-aligned assistants, while improved personalization also creates risks such as behavioral manipulation.
- B.2 Code of Ethics: The study complies with the NeurIPS Code of Ethics and does not involve human subjects, sensitive personal data, or real-world deployment.VitaBench 2.0 data are manually annotated or synthetically generated.
- B.2 Code of Ethics: Manually annotated data follow strict internal guidelines excluding personally identifiable and sensitive information.
- B.3 Broader Impacts: The benchmark aims to advance evaluation of personalized and proactive agents in realistic settings, supporting more reliable and user-aligned AI assistants.Potential applications include recommendation systems, digital assistants, and decision support tools.
- B.3 Broader Impacts: Improved personalization and robustness may enhance user experience in recommendation systems, digital assistants, and decision support tools.
- B.3 Broader Impacts: More effective personalization could also be misused to manipulate user behavior.
- B.4 Safeguards: VitaBench 2.0 uses manual annotation and large language model-based synthesis, with multi-stage quality control for responsible data release.Synthetic samples undergo human verification and refinement after generation.
- B.4 Safeguards: Manually annotated content is screened to exclude sensitive, personal, or harmful information.
C.1 Experiments Compute Resources
VitaBench 2.0 evaluations span 56 tasks and approximately 819 subtask-level interactions, repeated across four trials, while relying on commercial APIs rather than intensive local compute. In the most demanding full-context setting, each model–memory configuration takes approximately 3.7–8.6 hours for reasoning-enabled models and 4.3–4.6 hours for non-reasoning models.
- Evaluation workload: Each evaluation covers 56 tasks and approximately 819 subtask-level interactions, averaging 14.6 subtasks per task.The tasks include multi-turn follow-ups.
- Evaluation workload: Four independent trials yield roughly 3,276 interactions per model–memory configuration in total.This repetition supports statistical rigor.
- Compute requirements: Evaluations impose no intensive local CPU or GPU demands because all models are accessed through commercial closed-source APIs.The reported wall-clock costs therefore reflect API-based evaluation rather than local model inference.
- Wall-clock cost: In the most demanding full-context setting, non-reasoning models require approximately 4.3–4.6 hours per model–memory configuration.Full context provides the complete sequence of prior interactions across turns.
- Wall-clock cost: In the most demanding full-context setting, reasoning-enabled models require approximately 3.7–8.6 hours per model–memory configuration.Examples include Claude-Sonnet-4.5, DeepSeek-V4-Pro, GLM-5.1, and Gemini-2.5-Pro.
C.2 Benchmark Data Analysis
VitaBench 2.0 comprises a structured benchmark of realistic user interactions, with statistics indicating rich per-user representations across tasks, environment entities, and preferences.
- Dataset Statistics: The benchmark includes 56 users, 819 tasks, 41,414 environment entities, and 2,286 preference annotations.These statistics are derived from real-world application scenarios and guide the construction of user profiles, preferences, and interaction histories.
- Dataset Statistics: On average, each user is associated with 14.6 tasks, 739.5 environment entities, and 40.8 preferences.The averages indicate a rich and structured representation for each user.
C.2.1 User Profile Analysis
VitaBench 2.0 models users across realistic demographic, geographic, socioeconomic, occupational, and social dimensions rather than simplified synthetic profiles. This combination creates a concentrated yet heterogeneous population requiring context-dependent personalization beyond demographic heuristics.
- Profile construction: User profiles span demographic attributes, geographic distribution, socioeconomic status, occupation, and social context, with attributes designed to follow observed real-world statistics.The profiles aim to approximate structural properties of real-world users in online life-service applications.
- Demographic distribution: 62.5% of users are female, while ages concentrate in 20–29 (62.5%) and 30–39 (26.8%) groups.The average age is approximately 31, and users born after 1990 account for 87.5% of the dataset.
- Regional and socioeconomic structure: Approximately 50% of users come from emerging first-tier regions, 28.6% from first-tier cities, and 21.4% from lower-tier regions.This regional distribution introduces variation in consumption frequency, spending power, and price sensitivity.
- Occupation and social context: The dataset includes 60.7% white-collar, 16.1% blue-collar, and 23.2% gray-collar workers across diverse industries and social backgrounds.Occupational diversity introduces variation in lifestyle patterns, time constraints, and consumption habits.
- Implications for personalization: Demographic concentration combined with regional, occupational, and family-context diversity makes personalization dependent on fine-grained, context-dependent user modeling.The design requires agents to adapt decisions across diverse user profiles rather than relying on simple demographic heuristics.
C.2.2 User Preference Analysis … D.1 Case Study: Memory-Sensitive Delivery for User A891207
VitaBench 2.0 combines highly individualized, evolving preferences with long-horizon, noisy, cross-domain interactions, requiring agents to maintain robust user representations. Its memory configurations compare active LLM-based consolidation with deterministic retrieval, while a delivery case study demonstrates preference drift over time.
- C.2.2 User Preference Analysis: VitaBench 2.0 contains 17,928 preference annotations spanning 2,048 types and five categories, averaging 40.8 preferences per user.The categories cover food, shopping, travel, leisure and entertainment, and long-tail preferences.
- C.2.2 User Preference Analysis: Food preferences comprise 35% of annotations, while shopping, travel, and entertainment account for 18.1%, 20.2%, and 21.0%, respectively.Food preferences include cuisine, taste, beverages, and dietary restrictions.
- C.2.2 User Preference Analysis: Each user experiences 48.38 preference changes on average, including 36.07 additions, 7.52 deletions, and 4.79 modifications.Additions dominate, but deletions and modifications create non-monotonic preference evolution.
- C.2.2 User Preference Analysis: Approximately 92.4% of preferences are user-specific and 7.6% are shared, limiting reliance on population-level shortcuts.Most preferences reflect individual lifestyle, health, social, or consumption contexts.
- C.2.3 Interaction Analysis: Users average 2,093 interaction events over timelines averaging 1,580 days, with approximately 20% of behaviors intentionally noisy.These interactions span browsing, searching, consulting, comparing, and purchasing, requiring aggregation of fragmented evidence over time.
- C.2.4 Implementation Configurations: Agentic Memory rewrites the current memory using each new interaction batch, whereas RAG Memory deterministically retrieves the top k = 8 chunks after applying a 0.3 similarity threshold.Agentic Memory actively retains, merges, updates, or discards information; RAG uses text-embedding-3-large with 512-token chunks and no LLM summarization.
- D.1 Case Study: Memory-Sensitive Delivery for User A891207: User A891207’s stable home address coexists with dietary drift from spicy to light Cantonese food, a later durian preference, and eventual pork exclusion.The case study samples subtasks sub_A891207_13, sub_A891207_17, and sub_A891207_20 from her 20-subtask trajectory.
D.1.1 Part 1. One Subtask, Three Backends · user · D.1.2 Part 2. Preference Drift Across Three Subtasks
The benchmark tests a preference-laden dessert-delivery task whose seven criteria require recovering latent preferences and verifying environmental attributes. Agentic Memory and Full Context achieve perfect reward, whereas RAG Memory fails on the rating threshold; across subtasks, Agentic Memory better preserves evolving and stable preferences.
- D.1.1 Part 1. One Subtask, Three Backends: The dessert-delivery subtask has seven binary criteria, most recoverable only from memory rather than the under-specified instruction.The criteria include Cantonese-style tong sui, room temperature, durian, quantity 1, delivery within 30 minutes, rating at least 4.3, and the home address.
- D.1.1 Part 1. One Subtask, Three Backends: Durian and the merchant-rating threshold are not stated in the instruction and must be recovered from preference memory.The relevant memories are that the user likes durian and prefers delivery merchants rated ≥4.3.
- D.1.1 Part 1. One Subtask, Three Backends: Separating the target merchant from an otherwise identical distractor requires applying the remembered ≥4.3 rating preference to the tool-output rating.S00027 and S00017 offer the same product, but their ratings are 5.0 and 4.0, respectively.
- user: Agentic Memory and Full Context both select S00027 and satisfy all seven rubrics, each receiving reward 1.0.Agentic Memory recalls the durian preference and orders one room-temperature Cantonese durian tong sui to the home address; Full Context produces the identical order.
- user: The backends diverge on the latent delivery-rating preference: Agentic Memory surfaces a hard threshold, Full Context applies it implicitly, and RAG Memory retrieves insufficiently relevant history.This difference explains why only the first two backends choose the qualifying merchant in the rollout.
- D.1.2 Part 2. Preference Drift Across Three Subtasks: Across three subtasks, Agentic Memory better preserves preference drift by removing stale preferences, adding new ones, and retaining stable thresholds such as rating ≥4.3 and delivery ≤30 minutes.The comparison traces ground-truth dietary preferences against what Agentic Memory and RAG Memory surface at each timestamp.