PaperlayerView exact revision on arXiv

Source-linked AI summary

UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions

Mengze Hong, Zeyang Lei, Wenbo Shang, Xia Zeng, Xiying Zhao, Qi Zhu, Chen Jason Zhang, Di Jiang, Taiming Fu, Qiongyi Zhou, Qinghe Chang, Fubao Zhang, Chenxuan Ma, Minlong Peng, Jinfeng Huang, Zineng Zhou, Jindou Wu, Muge Qi, Sijun He, Xin Cui, Di Liang, Yuan Hua, Davey Chen

arXiv:2610.11638v1cs.CL

TL;DR

Binary feedback and user-agnostic judges provide limited insight into heterogeneous, evolving user experience. UXBench Pro pairs interaction instances with FACTORS profiles, personalized reward modeling, multi-turn simulation, and evaluator meta-benchmarks. Results indicate that personalization’s value depends on user and task context and profile access, while interaction signals capture distinct aspects of UX.

  • Problem

    Binary preference signals offer limited interpretability, user-agnostic reward models overlook heterogeneous preferences, and single-turn evaluations miss UX evolution across conversations.

    Source: §1 · p. 2

  • Method

    UXBench Pro pairs 1,000 interaction instances with seven-facet FACTORS profiles, combines personalized reward modeling with multi-turn simulation, and validates evaluators against real preferences and behavior.

    Source: §1 · pp. 2–3, §5 · p. 9, §5.1 · p. 9

  • Results

    Personalization’s apparent value depends on user and task context and profile access, while multi-turn signals capture distinct interaction properties rather than interchangeable measures of UX.

    Source: §6.4 · p. 15, §7 · p. 15

  • Takeaways & Limitations

    Conversational UX evaluation should use behaviorally grounded signals, report subgroup effects, and check evaluator sensitivity to user context before comparing assistants.

    Source: §6.4 · p. 15

    FACTORS dimensions are coarse proxies that may encode demographic or socioeconomic biases; subgroup differences are noncausal and should not guide consequential individual decisions.

    Source: Ethical Considerations · p. 16

Abstract

from arXiv · show

Evaluating user experience (UX) with automated computational methods has gained increasing attention, supported by empirical evidence from UXBench. However, binary preference prediction provides limited insight, while relying on a single user-agnostic reward model overlooks the inherent heterogeneity of users, whose expectations can differ substantially. In this paper, we present UXBench Pro, comprising 1{,}000 test instances derived from real user interactions across 12 task scenarios and 82 domains. Each instance is paired with a FACTORS user profile that characterizes the user through seven interpretable behavioral facets, differentiating user groups. To provide richer evaluation insights, we introduce a dual-perspective paradigm that combines a personalized User Reward Model (URM) for third-person judgment with Sim4Eval, a user simulator that enables multi-turn interactions and provides first-person evaluation across four cognitive state dimensions. To assess the reliability of these based evaluators, we further introduce two meta-benchmarks, URMBench and USimBench, that evaluate how faithfully they reproduce real human preferences and behaviors. Extensive experiments reveal seven key findings that highlight the importance of user modeling and multi-perspective evaluation, offering a fresh perspective on user-centric benchmarking and motivating personalized model optimization.

1 Introduction

Current UX evaluation offers limited insight into everyday experience because binary feedback is coarse, user-agnostic judgments obscure preference heterogeneity, and single-turn tests miss change across dialogue. UXBench Pro responds with personalized profiles, dual-perspective evaluation, and human-grounded meta-benchmarks.

1 Introduction

  • Binary like/dislike feedback is coarse and hard to interpret, user-agnostic judges collapse heterogeneous preferences, and single-turn evaluations miss UX changes across conversations.User satisfaction may accumulate over multiple turns.Source: §1 · p. 2
  • UXBench Pro pairs 1,000 high-quality queries with seven-facet FACTORS profiles and covers 12 task scenarios across work and life.The profiles characterize users through seven behavioral facets.Source: §1 · p. 2
  • The dual-perspective paradigm combines a personalized User Reward Model for third-person inspection with Sim4Eval for multi-turn, first-person evaluation across four cognitive state dimensions.Sim4Eval outputs user states to provide interpretable evaluation insights.Source: §1 · p. 3
  • URM-Bench and USimBench use real user preferences and interaction behavior as ground truth to evaluate judge models before deployment.The meta-benchmark principle treats the evaluator itself as the object of study.Source: §1 · p. 3
  • The work emphasizes generalizable conceptual and methodological principles rather than comparisons among particular models that may quickly become outdated.It aims to account for the heterogeneous and evolving nature of human-AI interaction.Source: §1 · p. 3

2 Related Work

Prior UX evaluation offers interpretable feedback but is costly to scale, while conversational benchmarks, reward models, and user simulators leave gaps in user-specific and multi-turn assessment.

2.1 User Experience Evaluation

  • Traditional UX studies and instruments provide rich, interpretable feedback but are costly to scale; a review of 548 studies reported a median of 20 participants per study (D´ıaz-Oreiro et al., 2021).Computational approaches have been explored to augment or automate parts of UX evaluation.Source: §2.1 · p. 3
  • UXBench grounds evaluation in real user feedback, but remains static, single-turn, and generic, leaving multi-turn evolution and individual user heterogeneity open (Hong et al., 2026).Prior work also models satisfaction from graded dialogue annotations and behavioral signals such as interruptions and rephrasing.Source: §2.1 · p. 3

2.2 Reward Modeling

  • Existing reward-model benchmarks mainly measure correctness or alignment utility using curated preferences, not whether models capture different users’ experiences and preferences.RMB reports weak correlation between pairwise accuracy and downstream alignment quality, while RewardBench 2 addresses contamination and ceiling effects.Source: §2.2 · p. 3

2.3 User Simulation for Multi-turn Dialogues

  • LLM user simulators span generic proxies, goal-state tracking, and persona or digital-twin approaches, but remain largely domain-specific and rely on weakly grounded profiles.Profiles are often hand-authored or coarse prompt prefixes rather than stable, fine-grained signals inferred from real interactions.Source: §2.3 · p. 4

3 UXBench Pro

UXBench Pro pairs real interaction instances with interpretable user profiles and balances task coverage while preserving within-task profile distributions. Its statistics describe broad task, profile, and multi-turn coverage, alongside varied failure and dissatisfaction signals.

3.1 Overview

  • UXBench Pro contains 1,000 test instances from interaction logs and pairs them with FACTORS profiles to support personalized UX evaluation.The benchmark is designed to test user-characteristic modeling, profile-informed response quality, and the human alignment of user-aware evaluation.Source: §3.1 · p. 4

3.2 User Profile Schema with FACTORS

  • FACTORS distributions span varied age, consumption, region, and education groups, while feedback disposition is dominated by silent and light users.The sampling procedure retains critical, lenient, and expressive users at informative proportions.Source: Figure 3 · p. 5, §3.2 · p. 5
  • FACTORS represents each user with seven interpretable behavioral dimensions, discretized into categories plus an unknown state, using sanitized logs or device-level information.The resulting persona cards condition the personalized reward model and user simulator; profiles are intended as contextual signals, not definitive characterizations.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 User x Task Grid

  • Each instance combines interaction context, a FACTORS profile, and a task scenario in a user-by-task evaluation space.Sampling balances task categories while preserving natural profile–task correlations; rare scenarios are retained and dense ones capped.Source: §3.3 · p. 6

3.4 Data Construction Pipeline

  • The construction pipeline cleans negative-feedback sessions, screens for safety and privacy risks, and filters low-value cases through multi-agent review.Human reviewers then double-cover sessions, assess safety and privacy, and indicate preferred responses for the URM-Bench subset; flagged safety or privacy cases are excluded.Source: §3.4 · pp. 6–7

3.5 Data Statistics

  • 63.2% of instances include preceding dialogue context, 85.0% resolve all seven FACTORS facets, and verbosity, intent mismatch, and task incompleteness account for 63.8% of instances.The benchmark spans 12 task scenarios and 82 fine-grained domains; factual errors account for 6.8%, explicit complaints occur in 11.5%, and recovery attempts in 39.7%.Source: Table 3 · p. 7, §3.5 · p. 7

4 Methodology

The methodology pairs a profile-conditioned reward model with a simulator that continues multi-turn dialogue and exposes evolving user states. The evaluators provide complementary views of response satisfaction, interaction-state alignment, and profile-dependent reactions.

4.1 User Reward Model

  • The URM estimates response satisfaction from an assistant response, authentic dialogue context, and the user's FACTORS profile.The study compares pointwise and pairwise generative reward models with a Bradley-Terry model, using regeneration preferences, dual-answer preferences, or pointwise feedback.Source: §4 · p. 8, §4.1 · p. 8

4.2 Sim4Eval Framework

  • At each turn, Sim4Eval generates a next user query, a four-dimensional cognitive state, and a continue-or-leave decision conditioned on the latest assistant response and injected profile.The state dimensions are goal progress, emotional fit, cognitive fit, and engagement preference.Source: §4.2 · p. 8
  • The retrieval-augmented simulator retrieves realistic user follow-ups by matching assistant responses and conversational context, then retains three diverse utterances as examples.It searches authentic follow-up turns, retrieves the top 12 candidates, and removes near-duplicates before selecting examples.Source: §4.2 · p. 8

4.3 Decomposing User Satisfaction

  • The framework distinguishes response capability, interaction-state alignment, and profile-dependent preference, with the URM assessing them and Sim4Eval exposing state alignment and profile-dependent reactions over turns.This decomposition is a lens for interpreting complementary evaluator signals rather than reducing satisfaction to one universal response-quality measure.Source: §4.3 · pp. 8–9

5 Meta-Benchmarking

The paper defines meta-benchmarking as validating automated evaluators against real human behavior before deployment. It constructs separate benchmarks for reward-model preference ranking and simulator fidelity to multi-turn user behavior.

5 Meta-Benchmarking

  • Meta-benchmarking evaluates an automated evaluator against real human behavior before deployment rather than assuming it is human-aligned.The authors warn that unvalidated divergence from human preferences can propagate into downstream evaluation biases.Source: §5 · p. 9

5.1 URM-Bench

  • URM-Bench uses 800 human-verified response pairs to test preference ranking and whether FACTORS conditioning improves over a population-level baseline.Pairs come from regeneration and dual-answer dialogue logs, and comparisons use pairwise accuracy with exact ties counted as errors.Source: §5.1 · p. 9

5.2 USimBench

  • USimBench tests whether simulated multi-turn continuations match real user behavior in feedback signals, semantic content, and interaction style.Its 150 instances use authentic continuations as ground truth, and the three fidelity dimensions are reported separately.Source: §5.2 · p. 9

6 Results and Discussion

Across the evaluations, user-aware judgments and multi-turn simulation reveal that conversational experience varies by user and cannot be reduced to response quality alone. Profile effects depend on the evaluator, user, task, and available dialogue history; simulator fidelity also varies across behavioral dimensions.

6.1.1 Overall Evaluation

  • Turn BT correlates with resolution (ρ = 0.857), but resolution and abandonment are nearly uncorrelated (r = 0.106), marking distinct aspects of conversational experience.Abandoned trajectories have low goal progress even when cognitive fit remains relatively high, and the table compares rankings, resolution, abandonment, and pair separability across seven assistants.Source: Table 4 · p. 10, §6.1.1 · p. 10

6.1.2 The Effect of User Profile

  • With responses held fixed, profile-aware evaluation reduces preference for profile-free responses, but personalization preferences vary across user groups and are strongest at cold start.Profile-blind evaluators prefer the profile-free response in 62.5% of comparisons, falling to 54.8% when shown the true profile; conditioned responses are preferred at 0.502 without prior dialogue versus 0.412 with history.Source: §6.1.2 · pp. 10–11

6.2.1 Training Strategies

  • The certified URM reaches 0.7688 PairAcc, above the best prompted judge at 0.6525; profile conditioning raises it to 0.8187, near the human reference of 0.8271.The pairwise GRM trained on regeneration preference was selected for UXBench Pro; profile information chiefly improves style adaptation, while state recognition and need inference do not improve overall.Source: Table 5 · p. 11, §6.2.1 · p. 12, Table 6 · p. 12

6.1.3 Repairing Real Failures

  • Repair rates are higher for factual errors (0.548) than for verbosity (0.328) or formatting failures (0.324), despite the latter two being more common.Verbosity and formatting together make up 34.7% of the benchmark, compared with 6.8% for factual errors; repair rates rise with overall model strength.Source: §6.1.3 · p. 11, Table 6 · p. 12

6.3.1 Linguistic and Behavioral Fidelity

  • Retrieval improves linguistic similarity for most tested simulators, but feedback-behavior gains are inconsistent and fidelity still depends on the base model.Retrieval improves dialogue-level and turn-aligned similarity for 11 of 12 models, yet improves each reported feedback metric for only three of ten models; style similarity and repetition behavior also rank models differently.Source: Table 8 · p. 13, §6.3.1 · pp. 13–14

6.2.1 Training Strategies

  • Profile-semantic perturbations affect URM judgments, but added thinking effort does not reliably substitute for user context.Reversing all seven profile facets changes accuracy by 7.26 points, while placebo text changes it by 0.1 points; higher thinking effort helps only one of four tested judges, with significant changes being declines.Source: §6.2.1 · p. 13, Table 7 · p. 13

6.3.2 Dynamic State and Profile Effects

  • Behaviorally anchored state ratings reduce joint state persistence to at most 5.7%, while simulator signals and profile groups correspond to distinct state and behavior patterns.The comparison of prompting schemes uses different base models and is diagnostic, not controlled; agreement between simulated states and behavior does not externally validate latent states.Source: Table 9 · p. 14, §6.3.2 · p. 14

6.4 Discussion: Implications for User-Centric Evaluation

  • The results support subgroup-level evaluation that distinguishes actionable user context from tasks where context adds little, rather than relying on a single universal score.Response quality, resolution, abandonment, user-simulator behavior, and state change capture different aspects of conversational UX.Source: §6.4 · p. 15

7 Conclusion and Future Work

UXBench Pro treats satisfaction as shaped jointly by the user and task, and its results highlight the value of evaluating both response fit and how experience changes during interaction. Future work is proposed to extend user representations and evaluator capabilities, and to validate FACTORS beyond a single platform.

Conclusion

  • UXBench Pro frames satisfaction as jointly shaped by user and task, with Sim4Eval, URM-Bench, and USimBench supporting evaluation of user fit and experience over time.The conclusion describes this as a broader HCI view that assesses more than response correctness.Source: §7 · p. 15

Future Work

  • Future work includes subgroup evaluation, more actionable profiles, richer reward objectives, adaptive evaluators, and longer-term preference memory.The proposed extensions also include richer behavioral signals and cross-lingual and cross-cultural validation.Source: §7 · p. 15
  • FACTORS is proposed for extension beyond a single platform, with cross-lingual and cross-cultural validation among the suggested next steps.The paper also identifies longer-term preference memory and richer behavioral signals as future directions.Source: §7 · p. 15

Ethical Considerations

The paper limits its data pipeline to voluntarily enrolled users who consent to research use of their interaction logs. It also treats FACTORS profiles as coarse, potentially biased proxies and restricts released data to evaluation.

Ethical Considerations

  • Only users who opt in and authorize research use contribute interaction logs, and a locally deployed LLM rewrites the data to reduce identifying information.Strict cleaning and de-identification filtering provide further protection.Source: Ethical Considerations · p. 15
  • FACTORS dimensions are coarse contextual proxies that may encode demographic or socioeconomic bias, so subgroup differences are not causal evidence or a basis for consequential individual decisions.Source: Ethical Considerations · p. 16
  • Released data is restricted to privacy-preserving evaluation and must not be used for model training.Source: Ethical Considerations · p. 16

A FACTORS Profile Encoding

FACTORS encodes each user through seven behavioral facets derived from logged interaction signals, using fixed vocabularies and a special marker for missing or contradictory evidence. The facets include usage, occupation, region, schooling, feedback disposition, age, and consumption capacity.

A FACTORS Profile Encoding

  • Each FACTORS profile uses seven behavioral facets derived from logged interaction signals, assigns values from fixed closed vocabularies, and marks absent or contradictory source signals with suffix X.Source: A FACTORS Profile Encoding · p. 20
  • Feedback disposition uses the preceding 30 days of positive and negative interactions relative to total prompts, distinguishing silent, critical, lenient, expressive, and residual light users.Source: A FACTORS Profile Encoding · p. 20
  • Age uses the platform-recorded account age interval, grouped into four bands from 18 or younger through 51 or older.Source: A FACTORS Profile Encoding · p. 20
  • Consumption capacity is inferred primarily from device price tier because platform wealth labels disagree with device price for roughly 20% of users; strong asset signals can override the tier upward.The categories range from low to flagship.Source: A FACTORS Profile Encoding · p. 20
  • Technology usage intensity uses recent AI activity and competing-product installation, with usage thresholds calibrated on the full user base.The categories distinguish light, regular, and heavy usage.Source: A FACTORS Profile Encoding · p. 20
  • Occupation role is inferred from dialogue-mined and platform life-stage signals, with priority ordering to avoid contradictory assignments.The categories include student, working adult, retired/senior, and parent with young children.Source: A FACTORS Profile Encoding · p. 20
  • Region is derived from the recent residential city, with a rural-group flag overriding the city-based classification.The profile distinguishes metropolis, town, and rural categories.Source: A FACTORS Profile Encoding · p. 20
  • Schooling maps platform education signals into primary/lower secondary, upper secondary, or tertiary-and-above categories.Source: A FACTORS Profile Encoding · p. 21

B Human Annotators and Ethical Considerations

Ten recruited employees annotated the data during regular work hours, supported by task-specific training and safeguards for uncertainty and sensitive cases.

B Human Annotators and Ethical Considerations

C Task Taxonomy and Scenario Assignment

The taxonomy assigns each trace to one of 12 scenarios through ordered rules and then rates simulated-user states independently on a five-point scale. The rubric defines dimensions spanning goal progress, emotional fit, cognitive fit, and engagement preference.

C Task Taxonomy and Scenario Assignment

  • Scenario assignment evaluates rules from top to bottom, prioritizes high-confidence intent fields over industry labels, assigns unmatched cases to LF-g, and uses each session’s modal trace-level scenario.Source: C Task Taxonomy and Scenario Assignment · p. 21

D User-State Scoring Rubric

  • Each simulated-user state dimension is rated independently on a 1–5 scale, with brief evidence requested before scoring and unparsable values recorded as missing.Source: D User-State Scoring Rubric · p. 21
  • Goal progress ranges from off-topic responses to completion without a necessary follow-up, while emotional fit ranges from discomfort or mismatch to recognizing unstated concerns.Source: D User-State Scoring Rubric · p. 21
  • Cognitive fit rates presentation rather than factual correctness, and engagement preference assesses alignment with the user’s desired interaction mode.Source: D User-State Scoring Rubric · pp. 21–22

E Experiment Configuration and Statistical Analysis

The experiments specify controlled generation and pairwise judging, then evaluate reward-model judgments and simulated-user outcomes using separate measures. Human reference accuracy and profile effects are reported separately from automated judges.

E Experiment Configuration and Statistical Analysis

  • Default generation uses a common template, temperature zero, no reasoning, an 8,192-token limit, and no tools; persona-card insertion is the only profile-condition difference.Prompted URMBench judges also use temperature zero and no reasoning, with a 16-token limit; the certified pairwise GRM compares both responses in one pass.Source: E Experiment Configuration and… · p. 22
  • URM-Bench PairAcc gives zero credit to ties, missing outputs, and unparsable verdicts while retaining them in the denominator; profile effects use exact two-sided McNemar, with ten-annotator mean as the separate human reference.Source: E Experiment Configuration and… · p. 22
  • USimBench reports feedback-signal fidelity, semantic similarity, and behavioral similarity separately, while Sim4Eval tracks resolution, time to resolution, abandonment, and valid-state outcomes.Resolution is the fraction of trajectories reaching S1=5; abandonment requires an early stop with final S1≤2.Source: E Experiment Configuration and… · p. 22
Loading 2610.11638v1…