PaperlayerView exact revision on arXiv

Source-linked AI summary

LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation

Feiyu Duan, Xuanjing Huang, Zhongyu Wei

arXiv:2603.12152v1cs.CL

TL;DR

Existing personalized-assistant benchmarks poorly capture changing environments, cognitive states, and long-term interactions. The paper introduces LifeSim, a BDI- and environment-grounded user simulator, and LifeSim-Eval, which evaluates assistants across single-scenario and long-horizon settings. Experiments show persistent weaknesses in implicit-intention fulfillment and long-term preference modeling, while the benchmark remains limited in domain coverage and multimodal signals.

  • Problem

    Existing benchmarks rely on static or short-context data because privacy constraints make long-term, diverse interaction logs scarce, limiting evaluation of dynamic personalized assistance.

  • Method

    LifeSim combines BDI-based cognition, physical environments, event generation, and behavior modeling to create trajectories and interactions for LifeSim-Eval.

  • Results

    Current LLMs handle explicit requests effectively but show pronounced limitations in implicit-intention fulfillment and long-term user preference modeling.

  • Takeaways & Limitations

    Personalized-assistant evaluation and modeling need to move beyond surface-level instruction following toward sustained, cognition-aware personalization.

  • Takeaways & Limitations

    LifeSim-Eval excludes high-stakes domains such as health care, legal consultation, and financial decision-making, and currently models behavior primarily through text.

Abstract

from arXiv · show

The rapid advancement of large language models (LLMs) has accelerated progress toward universal AI assistants. However, existing benchmarks for personalized assistants remain misaligned with real-world user-assistant interactions, failing to capture the complexity of external contexts and users' cognitive states. To bridge this gap, we propose LifeSim, a user simulator that models user cognition through the Belief-Desire-Intention (BDI) model within physical environments for coherent life trajectories generation, and simulates intention-driven user interactive behaviors. Based on LifeSim, we introduce LifeSim-Eval, a comprehensive benchmark for multi-scenario, long-horizon personalized assistance. LifeSim-Eval covers 8 life domains and 1,200 diverse scenarios, and adopts a multi-turn interactive method to assess models' abilities to complete explicit and implicit intentions, recover user profiles, and produce high-quality responses. Under both single-scenario and long-horizon settings, our experiments reveal that current LLMs face significant limitations in handling implicit intention and long-term user preference modeling.

1 Introduction

Existing personalized-assistant evaluations underrepresent real interactions because users respond to changing environments and evolving cognitive states. LifeSim and LifeSim-Eval address this gap by simulating cognition-grounded life trajectories and testing long-term individualized assistance.

  • Real-world assistance must account for time, location, weather, life events, evolving experience, stable personality, and preferences.
  • Privacy constraints and scarce multi-year interaction logs force benchmarks toward static or short-context datasets.
  • LifeSim simulates diverse users across life trajectories constrained by internal cognition and external environments.Its cognitive modeling adopts the Belief-Desire-Intention model, while event and behavior engines support trajectory and interaction simulation.
  • LifeSim-Eval covers 1,200 scenarios across 8 life domains and evaluates intention recognition, preference-aligned responses, and preference recall during long-term interactions.
  • Current LLMs handle explicit intentions relatively well but struggle with implicit intentions and long-horizon user modeling.Simple profile memory offers limited benefits, suggesting that personalization requires stable preference reasoning beyond retention.

2 LifeSim Framework

LifeSim combines profile-based cognition, physical environments, event generation, and behavior modeling to produce coherent user trajectories and conversations. Its BDI engine filters and samples intentions, while downstream engines ground events and generate responses.

  • Framework architecture: The framework combines a BDI-based cognitive engine, event engine, user behavior engine, and user profile pool.
  • BDI-based Cognitive Engine: User beliefs combine long-term profile information with short-term cognition generated from environmental conditions and recent experiences.
  • BDI-based Cognitive Engine: Desires are candidate intentions retrieved from a domain-organized pool and conditioned on the user’s belief state.Nine candidate intentions are retrieved at each service point.
  • BDI-based Cognitive Engine: User intentions are future-directed commitments selected after belief- and environment-based reranking filters inconsistent candidates.The remaining candidates are converted into a rank-based probability distribution before final sampling.
  • BDI-Grounded Event Engine: The event engine grounds trajectories in time, location, weather, and 3,374 mobility trajectories spanning 251 points of interest.
  • BDI-Grounded Event Engine: An event trigger uses elapsed time since the previous event to probabilistically decide whether the current trajectory point produces an event.
  • User Behavior Engine: The behavior engine models memory perception, emotion inference, and action selection before generating each final user response.
  • User Profile Pool: Users are represented through demographics, Big Five personality traits, and stable long-term preferences.

3 LifeSim-Eval Benchmark

LifeSim-Eval benchmarks personalized assistants in dynamically evolving scenarios where both explicit requests and implicit, profile- or history-dependent intentions matter. It evaluates intent handling, response quality, and user modeling in single-scenario and long-horizon settings.

  • Benchmark purpose: LifeSim-Eval tests whether assistants satisfy explicit and implicit intentions in dynamically evolving interaction scenarios.
  • Single Scenario Setting: The single-scenario setting evaluates intent recognition and completion without interaction history, with implicit intentions induced by profile or environmental context.
  • Evaluation protocol: The benchmark uses an LLM-as-Judge protocol that averages scores from GPT-4o3, Qwen3-32B, and Llama3.1-it 70B.
  • Long-Horizon Setting: In the long-horizon setting, assistants infer current implicit intentions from belief states integrating stable preferences and historical dynamic states.
  • Benchmark construction: The dataset contains 120 users, 10 events per user, 1,200 scenarios, and eight evenly distributed life domains.

4 Main Experiments

LifeSim-Eval tests personalized assistance across single-scenario and long-horizon settings, revealing strong explicit-intention performance but persistent weaknesses in implicit intention completion and long-term preference modeling.

  • Evaluation setup: LifeSim-Eval assesses mainstream LLMs across dynamically evolving personalized-assistance scenarios using long-horizon interaction histories.The experiments evaluate multiple open-source and closed-source assistant models.
  • Single-scenario performance: Most models recognize explicit intentions well but show a performance gap of more than 20 points on implicit intent recognition.The gap indicates difficulty uncovering latent user needs in single-scenario interactions.
  • Single-scenario performance: Proprietary models generally outperform open-source models on implicit intention recognition and completion, except for DeepSeek models.Model scaling improves performance across most metrics, while DeepSeek-V3.2 Thinking improves intention-related tasks over its base model.
  • Long-horizon performance: Under longer interaction histories, explicit intention completion remains stable, whereas implicit intention completion declines as conversation length increases.The contrast reflects the difficulty of integrating long-term dialogue history and latent preferences.
  • Preference recovery: A simple profile-memory mechanism benefits some models but leaves others flat or slightly worse, so retention alone does not guarantee robust long-term preference modeling.Qwen3-14B and Gemma3-it-12B trend upward with memory, while Qwen3-8B and Gemma3-it-27B remain nearly flat or decline.

5 Further Analysis

Further analyses examine variation across intention themes, validate the user behavior engine and evaluator agreement, and identify recurring assistant failure modes in multi-turn interactions.

  • Theme and type analysis: Model effectiveness varies systematically across intention types and themes, especially between explicit task-driven needs and implicit affective inference.The results indicate uneven robustness across service domains.
  • User behavior engine: The user behavior engine achieves scores above 90 on most metrics across its evaluation scenarios.Removing memory or emotion modules causes performance drops across all evaluated dimensions.
  • Failure analysis: Case studies identify rigid reasoning, limited proactive inquiry, and weak personalization as recurring assistant failure modes.Assistants may repeat initial suggestions, fail to ask clarifying questions, and underuse user profiles and preferences.

6 Related Works

Related benchmarks address social simulation or preference-focused evaluation, but LifeSim is positioned around cross-temporal, cross-situational personalized assistance.

  • Individual simulation: Prior individual-simulation systems improve behavioral fidelity but generally overlook spatiotemporal dynamics.LifelongSotopia extends simulation to longer horizons, while the cited systems otherwise remain limited in temporal modeling.
  • Personalized assistant evaluation: Most personalized-assistant benchmarks focus on single-turn instruction following or conversations confined to one scenario.Other long-dialogue tasks often examine preference memorization through generation or selection.
  • Benchmark comparison: LifeSim is compared with existing personal-assistant benchmarks as a long-horizon, multi-scenario evaluation framework.The comparison is presented in Table 5.

7 Conclusion

LifeSim and LifeSim-Eval model cognition-aware user trajectories and evaluate personalized assistants in multi-scenario, long-horizon interactions, exposing limits in implicit intention fulfillment and long-term preference modeling.

  • Conclusion: LifeSim models user cognition, life trajectories, and interaction behaviors, while LifeSim-Eval evaluates personalized assistants in multi-scenario and long-horizon settings.The framework targets sustained, cognition-aware personalization beyond surface-level instruction following.

8 Limitations

LifeSim-Eval is limited in domain coverage and currently models user behavior primarily through text. The framework also faces privacy, misuse, bias, and ethical considerations, although its controllable, de-identified design supports inspection and auditing.

  • High-stakes domain coverage: LifeSim-Eval covers common daily-life scenarios but excludes high-stakes domains such as health-care, legal consultation, and financial decision-making.These domains require more rigorous knowledge, regulatory and ethical constraints, and carry higher costs for incorrect or misaligned responses.
  • Multimodal user signals: LifeSim primarily models user behavior through textual interactions, omitting multimodal cues such as visual context and physiological signals.Adding multimodal information could improve realism but introduces data-collection, cross-modal-alignment, and ethical challenges.
  • Ethical considerations: User simulation and long-horizon interaction modeling raise concerns about privacy, data misuse, and biased representations.LifeSim avoids real long-term conversational logs and instead uses controllable, de-identified profiles and event sequences for evaluation.
  • Ethical considerations: LifeSim is intended as a research-oriented simulation framework rather than a deployable system, reducing risks of direct user manipulation or unintended real-world impact.The structured design enables systematic inspection and auditing, while biases may still arise from modeling assumptions.

B.6 Error mode analysis

The error analysis identifies recurring failures in long-horizon assistance: rigid reasoning, insufficient proactive inquiry, and weak use of user profiles. Additional evaluation examines these behaviors across models, while the benchmark and data-generation pipeline support broader personalized-assistance testing.

  • Rigid reasoning without dynamic adaptation: Assistants often repeat an initial solution despite evolving constraints and explicit negative feedback, rather than revising their reasoning strategy.The reported case describes repeated recycling of previously rejected approaches with minor rephrasing.
  • Insufficient proactive inquiry: Assistants frequently provide default recommendations without asking about underspecified user constraints, producing systematic misalignment with stated preferences.The example concerns exercise recommendations that ignored a preference for low-impact, nonrunning activities.
  • Cross-model evaluation: Most LLMs show relatively low proactive-questioning rates, while proprietary and larger models generally exhibit less rigid reasoning or better performance.These observations are reported alongside the model-comparison ratios in Table 9.
  • Evaluation resources: The event-desire pool contains over 113K pairs across 8 categories, with semantic deduplication and filtering used to improve coverage and reduce redundancy.Queries are normalized into canonical intentions and paired with plausible triggering events.

C.3 Life Trajectory Generation

Life trajectory generation combines intention extraction, event-hypothesis generation, retrieval, reranking, and contextual refinement. The pipeline organizes intentions across three event dimensions and adapts candidates to user beliefs and physical context.

  • Event-hypothesis generation: The trajectory pipeline generates event hypotheses from physiological, psychological, and environmental aspects before retrieving candidate intentions.Each dimension corresponds to an event-hypothesis source.
  • Intention normalization: User questions are classified by domain and normalized into formal intention expressions, excluding code, academic-paper, and general-information requests.The normalization prompt targets daily-life requests relevant to an AI assistant or coach.
  • Retrieval and reranking: For each hypothesis, the system retrieves 3 candidates per dimension, producing 9 candidate intentions that are reranked under belief-state and environmental constraints.Retrieval uses embedding-based nearest neighbors, followed by reranking and refinement.
  • Contextual refinement: The reranking stage filters candidates inconsistent with the user's beliefs and physical environment, then adapts the selected intention and event context to current spatiotemporal conditions.The event engine and cognitive engine use separate prompts for hypothesis generation and candidate reranking.

D LifeSim-Eval Details

LifeSim-Eval evaluates personalized assistants on explicit and implicit intentions, conversational quality, preference recovery, and persona alignment. Its single-scenario and long-horizon settings operationalize both contextual inference and preference-based user modeling.

  • Intent definitions: Explicit intentions are directly expressed, whereas implicit intentions require contextual or user-state reasoning to infer underlying needs affecting a good response.The distinction follows relevance-theoretic explicatures and implicatures.
  • Evaluation settings: In the long-horizon setting, one explicit scenario intention is combined with implicit intentions derived from user preferences and life-event sequences.This setting tests whether assistants infer and apply preferences from extended interaction histories.
  • Core metrics: Intent recognition aggregates predicted intentions across an episode and reports separate accuracy values for explicit and implicit intention sets.Each checklist intention receives a binary correctness indicator before averaging across scenarios.
  • Core metrics: Intent completion measures whether each checklist intention is fulfilled and reports separate explicit and implicit completion scores.An LLM-based evaluator assigns binary completion indicators and averages scores across episodes.
  • Auxiliary metrics: Auxiliary metrics assess naturalness, coherence, stable preference recovery, and faithful persona alignment across interactions.Preference recovery predicts discrete values across preference dimensions, while persona alignment measures consistency with enduring characteristics.

E.1 User Behavior Engine Evaluation

The user behavior engine generates personalized multi-turn behavior by conditioning memory perception, emotion inference, action selection, and conversation on profiles, events, intentions, and dialogue context. Evaluation materials include life-event sequences, attribute distributions, and prompts for explicit–implicit intent assessment.

  • Behavior generation: The behavior engine decides whether conversational information becomes long-term memory, infers emotion, selects an action, and generates the user's next utterance.The prompts condition these stages on profile, recent events, intentions, dialogue history, and prior memory.
  • Memory and emotion: Emotion inference selects the next likely user emotion from candidate emotions using profile, memory perception, event, and dialogue context.The engine predicts the emotional state before generating each response.
  • Action selection: The action-decision stage selects the user's next action from candidate actions conditioned on conversation history, profile, event, intent, emotion, and memory perception.This selected action feeds the subsequent user-conversation generation process.
  • Conversation generation: User conversations are generated as short, natural, everyday messages conditioned on explicit and implicit intent lists alongside personalized context.The system prompt includes user profile, dialogue scene, recent life event, and primary intent.
  • Intent processing: The pipeline decomposes broad intentions into at most 6 concrete sub-intents and classifies them as explicit or implicit according to reasoning requirements.Explicit sub-intents are task-oriented and consciously expressed, whereas implicit ones are motivational, emotional, or value-driven.
Loading 2603.12152v1…