PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants

Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, Emine Yilmaz

arXiv:2506.09902v1cs.CLcs.AIcs.LG

TL;DR

Systematic evaluation of personalization in task-oriented assistants remains largely unexplored. PersonaLens combines rich user profiles with user and judge agents to assess personalization during task-oriented dialogues. Its analyses reveal variability in personalization, including stronger adaptability in larger models alongside continued cross-domain consistency challenges.

  • Problem

    Systematic evaluation of how task-oriented AI assistants personalize responses to user preferences while completing tasks remains largely unexplored.

    Source: §1 · p. 1

  • Method

    PersonaLens evaluates task-oriented assistants using rich user profiles and user and judge agents that simulate dialogues and assess personalization.

    Source: §1 · p. 2

  • Results

    Larger models show better adaptability but still struggle with cross-domain consistency, and interaction history is the most valuable contextual factor for improving personalization.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings reinforce the need for dynamic user modeling in personalized assistants.

    Source: §6 · p. 9

    The benchmark is text-only and evaluates vanilla LLMs; actions such as bookings and purchases are simulated rather than executed.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have advanced conversational AI assistants. However, systematically evaluating how well these assistants apply personalization--adapting to individual user preferences while completing tasks--remains challenging. Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture the complexities of personalized task-oriented assistance. To address this, we introduce PersonaLens, a comprehensive benchmark for evaluating personalization in task-oriented AI assistants. Our benchmark features diverse user profiles equipped with rich preferences and interaction histories, along with two specialized LLM-based agents: a user agent that engages in realistic task-oriented dialogues with AI assistants, and a judge agent that employs the LLM-as-a-Judge paradigm to assess personalization, response quality, and task success. Through extensive experiments with current LLM assistants across diverse tasks, we reveal significant variability in their personalization capabilities, providing crucial insights for advancing conversational AI systems.

1 Introduction

Systematic evaluation of personalization in task-oriented assistants remains largely unexplored, while existing benchmarks miss key aspects of these interactions. PersonaLens addresses this gap with a benchmark that evaluates personalization alongside task completion across diverse tasks.

1 Introduction

  • Systematic evaluation of personalization in task-oriented AI assistants remains largely unexplored despite advances in personalization.The paper frames personalization as tailoring responses to user preferences learned from past interactions while completing tasks.Source: §1 · p. 1
  • Existing benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, and often depend on costly human-in-the-loop methods.Source: §1 · pp. 1–2
  • PersonaLens combines rich user profiles and contextual information with a user agent and a judge agent for evaluating multi-turn task-oriented dialogues.It covers over 100 tasks across 20 domains; the user agent simulates users, while the judge assesses personalization using preferences, interaction histories, and situational context.Source: §1 · p. 2
  • PersonaLens shows high agreement with human judgments, supporting its reliability for assessing personalization capabilities.Source: §1 · p. 2
  • PersonaLens analysis reveals key patterns and challenges in balancing personalization and task completion across diverse tasks.The conclusion adds that larger models adapt better but still struggle with cross-domain consistency, and that interaction history is the most valuable contextual factor for improving personalization.Source: §1 · p. 2
  • The authors release PersonaLens to support research on more personalized, context-aware AI assistants.Source: §1 · p. 2

2 The PersonaLens Benchmark

PersonaLens evaluates personalization in multi-turn task-oriented dialogues using diverse user profiles, tasks across domains, and LLM-powered user and judge agents. Its validation examines profile diversity and consistency, preference distributions, dialogue language variation, and benchmark coverage.

2.1 User Profile

  • Profiles combine demographics, domain-specific preferences, and LLM-generated interaction summaries, with preferences conditioned on demographics and some domains optionally masked.Demographic attributes are derived from data covering 1,500 real users across 75 countries.Source: §2.1 · pp. 2–3

2.2 Task Generation

  • The benchmark contains 111 tasks across 20 domains—86 single-domain and 25 multi-domain—with task-specific situational contexts and tasks excluded when required domains are masked.Multi-domain tasks typically involve three to five domains; Table 1 counts each such dialogue under its constituent domains.Source: §2.2 · p. 3, Table 1 · p. 3

2.3 User and Judge Agents

  • A user agent conducts task dialogues with assistants, while a judge evaluates personalization, response quality, and task success against the profile and scenario.Assistants may receive full, partial, or no user profile or situational context, depending on the experimental setup.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 Benchmark Validation

  • The benchmark comparison covers scale, domain coverage, task-oriented and personalization evaluation, user preferences, and situational context.Table 2 marks benchmarks that include only self-dialogues with ‡.Source: §2.4 · p. 4, Table 2 · p. 5
  • Demographic distributions are reported as diverse and mirroring real-world populations; manual review of 100 profiles and an LLM checker found no contradictions requiring correction.The checker flagged 11 profiles, which manual review judged to be valid complex preferences.Source: §2.4 · p. 4
  • Preference distributions are balanced across most domains, while reported asymmetries match real-world patterns; PersonaLens also has higher lexical diversity than existing benchmarks.The travel domain, for example, has a predominance of window-seat preferences.Source: §2.4 · p. 5

3 Experimental Setup

The experiments compare open-source and proprietary LLM assistants on personalization and task completion. They use a sampled set of user profiles and measure task completion, personalization, and dialogue quality.

Experimental Setup

  • The evaluation includes Claude, Llama 3.1 Instruct, Mistral 7B, and Mixtral 8x7B; Claude 3 Sonnet serves as the user agent and Claude 3.5 Sonnet as the judge.The listed Claude models are Claude 3 Sonnet, Claude 3 Haiku, and Claude 3.5 Haiku; Llama 3.1 Instruct uses 8B and 70B variants.Source: §3 · p. 5
  • Experiments use a random sample of 50 profiles, comprising 3,283 single-domain and 813 multi-domain dialogues.The full benchmark contains 122,133 user-task scenarios.Source: §3 · p. 5
  • Metrics include binary task completion, task completion rate, 1–4 personalization, and 1–5 naturalness and coherence ratings.Personalization scores how well dialogue responses are tailored to the user.Source: §3 · p. 5

4 Experiments and Results

The experiments compare assistant quality across task-oriented settings, personalization contexts, domains, and dialogue turns, and assess the user and judge agents. Results show contextual and domain-dependent variation, with interaction history especially valuable for personalization.

4.1 Quality of User Agent

  • The vanilla prompt was selected for the user agent because preliminary experiments found chain-of-thought prompting produced unnatural dialogue with excessive reasoning; the resulting agent was rated highly natural and coherent.The study compared vanilla, chain-of-thought, and user-state-tracking prompts.Source: §4.1 · pp. 5–6

4.2 Evaluation of Assistant Models

  • Claude 3 Sonnet achieves the highest TCR at 95.98%, while maintaining coherence of 4.86; personalization scores remain around 2 out of 4, leaving room for improvement.The paper identifies personalization as the benchmark’s primary focus, with TCR and other metrics serving as secondary indicators.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Effect of Model Scaling on Personalization

  • Larger models generally improve TCR, personalization, and dialogue quality: Llama 3.1 70B Instruct exceeds its 8B counterpart on all four reported dimensions.For Llama 3.1 Instruct, TCR rises from 89.55% to 90.80%, personalization from 2.14 to 2.21, coherence from 4.68 to 4.86, and naturalness from 3.90 to 4.11.Source: §4.3 · p. 6, Table 3 · p. 6

4.4 Personalization in Multi-Domain Tasks

  • Most assistants show lower TCR and personalization in multi-domain tasks than in single-domain tasks, while larger models such as Llama 3.1 70B Instruct show smaller drops.The authors identify preference adaptation, cross-domain interaction consistency, and conflicting domain preferences as multi-domain challenges.Source: §4.4 · pp. 6–7, Table 3 · p. 6

4.5 The Contextual Hierarchy of Personalization

  • Interaction history raises personalization from 2.13 to 2.59 in TSD and from 2.01 to 2.32 in TMD; demographic and situational context add marginal gains alone but synergize with history, especially in TMD.The vanilla baseline reaches comparable personalization to the base setting but reduces TCR, suggesting explicit instructions primarily improve task completion.Source: §4.5 · p. 7, Table 4 · p. 7

4.6 Cross-Domain Personalization Dynamic

  • Recommendation domains achieve higher TCR and personalization than procedural domains; turn-level patterns also vary, with movies and music improving over dialogue turns while messaging declines later.The authors suggest recommendation tasks benefit from early preference elicitation, while procedural tasks may require prioritizing task completion before personalization.Source: §4.6 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 8

4.7 Comparison with Human Evaluation

  • Human annotators showed high agreement, and their Cohen’s Kappa scores aligned strongly with the judge agent, especially for task completion and coherence.Three annotators assessed 100 sampled dialogues using the same guidelines as the judge agent.Source: §4.7 · p. 8

5 Related Work

Prior work addresses personalization through dialogue methods, benchmarks, and automated evaluation, but existing benchmarks often omit task structure, rich user context, or broad domain coverage. PersonaLens combines task-oriented evaluation with contextual personalization and an LLM-based judge.

Personalization in Conversational AI

Benchmarks for Personalized Conversational Systems

  • Existing benchmarks separately cover personalized text generation, open-ended dialogue, task completion, or narrow-domain preferences, while PersonaLens combines diverse domains, rich user context, and personalization evaluation for task-oriented dialogue.The related-work discussion characterizes LaMP and LongLaMP as nonconversational, TOD benchmarks as overlooking personalization, and PENS and Cornell-Rich as domain-limited.Source: §5 · p. 8

User Simulation and Evaluation

  • PersonaLens uses a user agent and a multidimensional LLM judge to evaluate personalization, task success, and response quality, offering a scalable alternative to resource-intensive human evaluation.The passage notes that lexical metrics such as BLEU and METEOR do not capture contextual alignment.Source: §5 · p. 9

6 Conclusion

PersonaLens evaluates personalization in conversational assistants across diverse domains and user preferences through simulated task-oriented dialogue. Experiments identify interaction history as the most valuable contextual factor, while larger models show better adaptability but still struggle with cross-domain consistency.

6 Conclusion

  • PersonaLens measures task completion and personalization quality through user-assistant simulation across diverse task settings.The conclusion describes experiments on prompting strategies, contextual information, and cross-domain personalization dynamics.Source: §6 · p. 9
  • Larger models show better adaptability but continue to struggle with cross-domain consistency.The conclusion identifies multi-domain personalization as a key challenge in the reported experiments.Source: §6 · p. 9
  • Interaction history is the most valuable contextual factor for improving personalization, supporting the need for dynamic user modeling.The authors identify improved historical-interaction retrieval and user simulation as directions for future work.Source: §6 · p. 9

Limitations

The benchmark has scope and data-generation limitations that may constrain how well it represents some real-world personalization settings.

Limitations

  • Specialized or niche domains may require additional customization to capture their domain-specific personalization dynamics.Source: Limitations · p. 9
  • The benchmark is text-only, simulates rather than executes actions such as bookings or purchases, and uses semi-synthetic profiles and dialogues that may retain LLM-related biases despite mitigation efforts.The authors note possible skews in demographic representation, cultural assumptions, socioeconomic biases, and language preferences.Source: Limitations · p. 9

A Benchmark Details

This appendix details benchmark creation, covering user profiles, task generation, and the user and assistant agents.

A.1 User Profile Generation

  • User profiles combine generated preferences and past-interaction summaries, with demographic breakdowns and profile examples provided.Claude 3 Sonnet generated the profile components.Source: §A.1 · p. 13

A.2 Task Generation

  • Claude 3 Sonnet generated tasks and situational context for the benchmark.The appendix includes task-generation prompts and example tasks.Source: §A.2 · p. 13

A.3 User and Judge Agents

  • The appendix provides prompts for U’s initial and subsequent queries and for A’s responses.Source: §A.3 · p. 13

A.4 Benchmark Validation

The benchmark validation checks profile consistency and compares dialogue volume and lexical diversity with existing conversational datasets.

A.4 Benchmark Validation

  • Claude 3 Sonnet checks profile consistency, while Dist-1, Dist-2, Ent-4, and Self-BLEU assess lexical diversity and redundancy, following Joko et al. (2024).Dist-1 and Dist-2 use unique n-gram ratios, Ent-4 measures 4-gram frequency entropy, and lower Self-BLEU indicates greater diversity.Source: §A.4 · p. 13
  • The benchmark includes more dialogues and greater lexical diversity than existing task-oriented dialogue datasets.Table 10 compares conversational datasets and reports lexical-diversity scores.Source: §A.4 · pp. 13–14

B Experimental Setup Details

The appendix reports model inference settings and identifies the prompts used by the judge to score task completion and response qualities.

B Experimental Setup Details

  • U runs at temperature 0.5, while A and J run at temperature 0; other inference parameters use each model’s defaults.Source: B Experimental Setup Details · p. 14
  • The appendix provides J’s prompts for evaluating task completion, personalization, naturalness, and coherence.The personalization evaluation guideline is included with its prompts.Source: B Experimental Setup Details · p. 14

C Additional Experiment Details

Additional experiments report dialogue quality and statistics for generated interactions, alongside details of the human evaluation and benchmark materials.

C Additional Experiment Details

  • The user agent engages in natural and coherent interactions across assistant models on TSD tasks, with minor dialogue-quality variations.Table 12 reports naturalness and coherence results.Source: C Additional Experiment Details · p. 14, Table 12 · p. 33
  • Generated dialogues with Claude 3 Sonnet are reported as having 5.64 versus 4.74 turns per dialogue and roughly 149.32 versus 149.87 tokens per turn across the stated settings.The passage names the settings as TSD and TSD, respectively, and suggests the turn-count difference may indicate greater complexity in multi-turn reasoning.Source: C Additional Experiment Details · p. 14, Table 13 · p. 33
  • Three experienced researchers with personalization expertise conducted human evaluation using the annotation guide also given to J.Source: C Additional Experiment Details · p. 14
Loading 2506.09902v1…