PaperlayerView exact revision on arXiv

Source-linked AI summary

LikeBench: Evaluating Subjective Likability in LLMs for Personalization

Md Awsafur Rahman, Adam Gabrys, Doug Kang, Jingjing Sun, Tian Tan, Ashwin Chandramouli

arXiv:2512.13077v1cs.LG

TL;DR

Personalization benchmarks largely assess memory recall and adherence, leaving subjective likability under-measured. LikeBench evaluates likability and adaptation over dynamic, multi-session conversations with simulated users and fine-grained personas. Its findings show that memory alone does not guarantee likability, and GPT-5 has limited robustness in longer, noisier interactions.

  • Problem

    Existing personalization benchmarks emphasize memory recall and adherence, while subjective likability and whether models adapt to user preferences remain under-measured.

    Source: §1 · p. 1

  • Method

    LikeBench evaluates simulated-user interactions dynamically across sessions, using fine-grained personas and seven diagnostic likability metrics.

    Source: §1 · p. 2

  • Results

    Memory alone does not guarantee likability, most models show little to no adaptability over conversations, and GPT-5 has limited robustness in longer, noisier interactions.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Likability depends on adaptive, context-sensitive use of explicit and implicit knowledge, not memory recall alone.

    Source: §6 · p. 10

    The framework’s discrete personality dimensions may not capture the full spectrum of real-world users, and its user profiles are static.

    Source: A DISCUSSION & LIMITATIONS · p. 13

Abstract

from arXiv · show

A personalized LLM should remember user facts, apply them correctly, and adapt over time to provide responses that the user prefers. Existing LLM personalization benchmarks are largely centered on two axes: accurately recalling user information and accurately applying remembered information in downstream tasks. We argue that a third axis, likability, is both subjective and central to user experience, yet under-measured by current benchmarks. To measure likability holistically, we introduce LikeBench, a multi-session, dynamic evaluation framework that measures likability across multiple dimensions by how much an LLM can adapt over time to a user's preferences to provide more likable responses. In LikeBench, the LLMs engage in conversation with a simulated user and learn preferences only from the ongoing dialogue. As the interaction unfolds, models try to adapt to responses, and after each turn, they are evaluated for likability across seven dimensions by the same simulated user. To the best of our knowledge, we are the first to decompose likability into multiple diagnostic metrics: emotional adaptation, formality matching, knowledge adaptation, reference understanding, conversation length fit, humor fit, and callback, which makes it easier to pinpoint where a model falls short. To make the simulated user more realistic and discriminative, LikeBench uses fine-grained, psychologically grounded descriptive personas rather than the coarse high/low trait rating based personas used in prior work. Our benchmark shows that strong memory performance does not guarantee high likability: DeepSeek R1, with lower memory accuracy (86%, 17 facts/profile), outperformed Qwen3 by 28% on likability score despite Qwen3's higher memory accuracy (93%, 43 facts/profile). Even SOTA models like GPT-5 adapt well in short exchanges but show only limited robustness in longer, noisier interactions.

1 INTRODUCTION

LikeBench addresses personalization benchmarks’ focus on memory recall and adherence by evaluating subjective likability with fine-grained personas and multiple diagnostic metrics. It finds that memory alone does not guarantee likability and that most models show little to no adaptability as conversations progress.

  • Existing benchmarks emphasize memory recall and adherence, but do not assess whether interactions feel likable or adapted to individual users.Prior likability evaluations also often use coarse personas and a single aggregate score, limiting diagnostic insight.Source: §1 · pp. 1–2
  • LikeBench evaluates likability and adaptability dynamically through multi-session conversations with simulated users.Adaptability is improvement in likability over ongoing conversations.Source: §1 · p. 2
  • LikeBench decomposes likability into seven diagnostic metrics for assessing subjective user satisfaction.Source: §1 · p. 2
  • Its simulated users have psychologically grounded profiles spanning seven personality traits, 35 facets, and seven conversation-style dimensions.Source: §1 · p. 2
  • Most models show little to no adaptability as conversations progress, and memory alone does not guarantee likability.Source: §1 · p. 2

2 RELATED WORK

Prior personalization benchmarks mainly evaluate factual memory and preference adherence, while likability benchmarks vary in whether they support dynamic interaction, detailed personas, multiple metrics, and adaptation tracking. LikeBench combines these dimensions in dynamic, multi-session conversations.

2.1 BENCHMARKS ON MEMORY RECALL AND MEMORY ADHERENCE

  • Memory-focused benchmarks assess recall, adherence, and preference tracking across long contexts or multiple sessions, generally using objective correctness or compliance measures.Source: §2.1 · p. 3

2.2 BENCHMARKS ON LIKABILITY

  • Likability benchmarks differ in whether they use dynamic or static evaluation, fine-grained personas, decomposed metrics, and adaptation tracking.ALOE, ALIGNX, and CUPID use coarse personas and aggregate scores; their settings and adaptation measures differ.Source: §2.2 · p. 3
  • LikeBench combines dynamic multi-session conversations, psychologically grounded textual personas, diagnostic likability metrics, and explicit adaptation measurement.Source: §2.2 · p. 3

3 BENCHMARK: LIKEBENCH

LikeBench evaluates LLMs in multi-session conversations with simulated users whose profiles and session priors remain hidden from the models. It scores likability across seven dimensions, tracks change over sessions, and measures remembered user facts and preferences.

3 BENCHMARK: LIKEBENCH

  • Models interact with simulated users over multiple sessions; models see dialogue history, while the user agent also sees the persona and hidden session prior.The model replies to each user message, then the user agent scores the reply and the conversation history carries forward across sessions.Source: §3 · pp. 3–4

3.1.1 LIKABILITY METRICS

  • Seven per-turn likability dimensions are emotional adaptation, formality matching, knowledge adaptation, reference understanding, conversation-length fit, humor fit, and callback.Each dimension receives a 1–5 score or NA; non-applicable scores are excluded when scores are aggregated.Source: §3.1.1 · pp. 4–5, §3 · p. 4

3.1.2 ADAPTABILITY METRICS

  • Adaptability is measured by regressing session scores on session index; positive improvement rates indicate increasing likability, while negative rates indicate degradation.The benchmark also reports normalized improvement rate and R^2, which measures goodness of fit.Source: §3.1.2 · p. 5

3.1.3 MEMORY ACCURACY

  • Memory accuracy is measured by checking model-reported user facts against the full dialogue and hidden profile, with statistics also split between explicit facts and inferred preferences.The evaluation reports aggregated accuracy and the total number of correctly remembered facts.Source: §3.1.3 · p. 5

4 DATA GENERATION

LikeBench uses a two-stage pipeline to create detailed user profiles and session priors for multi-session conversations. Profiles combine psychologically grounded personality and conversation-style descriptions with type-specific interests and other contextual tiers.

4 DATA GENERATION

  • The data-generation pipeline first creates fine-grained profiles and then instantiates session priors for multi-session conversations.The benchmark uses 50 profiles and 10 priors per profile, generated with Claude 3.7 Sonnet; profile types are social and anti-social.Source: §4 · p. 5, §4.1 · p. 5

4.2.1 PERSONALITY TRAITS GENERATION.

  • Profiles use seven psychological traits represented by 35 facets, each described at five ordered intensity levels with behaviorally interpretable text.The traits include the Big Five, Honesty–Humility, and Humor Styles.Source: §4.2.1 · pp. 5–6

4.2.2 CONVERSATION STYLE GENERATION.

  • Conversation style is represented by nine dimensions, each with a curated menu of categorical options, rather than personality-style facets or intensity levels.The dimensions include formality, preferred response length, reference usage, and feedback style.Source: §4.2.2 · p. 6

4.2.3 PROFILE ASSEMBLY.

  • Profile assembly applies type-specific archetype biasing to personality and style, then attaches three to five interest paths that are correlated or partially decorrelated by profile type.An LLM completes the remaining tiers using personality, style, interests, and demographics, adding knowledge, behaviors, motivations, life events, and relationships.Source: §4.2.3 · p. 6, Figure 2 · p. 6

4.3 PRIOR GENERATION

  • Session priors specify agendas, constraints, disclosure timing, and callback opportunities, while remaining hidden from the evaluated model.Social profiles receive more linear agendas; anti-social profiles include delayed disclosures, contrarian shifts, and intermittent surprises.Source: §4.3 · p. 7

5 EXPERIMENTATION AND RESULTS

Across 50 profiles and 10 sessions, GPT-5 leads overall likability, but models generally struggle to sustain improvement in longer interactions. Memory accuracy alone does not ensure likability, and targeted human review aligns with the automated scores.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers five models across 50 profiles, 10 sessions, and five turns per session, totaling 2,500 turns.Source: §5.1 · p. 7

5.2 PERFORMANCE ACROSS METRICS

  • GPT-5 achieves the highest overall likability score (3.94), while humor fit and formality matching challenge all models and its conversation length fit trails Claude Sonnet 4.GPT-5’s long, comprehensive answers outperform the second-best model, Claude Sonnet 4, by over 12% on knowledge adaptation but can reduce satisfaction when users prefer brevity.Source: §5.2 · p. 7

5.3 ADAPTABILITY

  • Early-session likability rises, but all models decline in Sessions 3–6; GPT-5 alone rebounds late, and other models generally show declines over the full interaction.In Sessions 3–6, the decline is predictable (R2 > 0.7); late-stage trajectories are less predictable (R2 often < 0.25), while GPT-5’s late improvement rate is IR=0.038.Source: §5.3 · pp. 7–8, Figure 3 · p. 8, Table 3 · p. 8

5.4 MEMORY PERFORMANCE

  • Qwen3 recalls more facts with higher memory accuracy than DeepSeek R1 yet underperforms it in likability, while GPT-5 excels in both memory and likability.LikeBench’s memory comparison indicates that recalling user facts is not sufficient: models must also use remembered information fluidly and contextually.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
  • GPT-5 extracts the most user facts per profile, infers implicit facts conservatively with 98% accuracy, and actively uses 80–100% of explicit and almost 100% of implicit memories in dialogue.Qwen3 infers implicit memories more aggressively, but its implicit-memory accuracy is 93%.Source: §5.4 · p. 9, Figure 5 · p. 9

5.5 HUMAN VALIDATION

  • A targeted review of five profiles found that higher automated scores aligned with human judgments of more personalized, context-aware, engaging responses; weaker models often became generic or lost coherence in longer conversations.The authors note that occasional human review adds complementary value.Source: §5.5 · p. 10

6 CONCLUSION

The conclusion argues that personalization requires more than memory recall: models must use explicit and implicit knowledge adaptively and contextually. Even GPT-5 has limited robustness in longer, noisier conversations.

  • Memory accuracy alone does not capture effective personalization; user satisfaction hinges on adaptive, context-sensitive use of explicit and implicit knowledge.Source: §6 · p. 10
  • GPT-5 applies recalled facts in conversational context while balancing breadth of recall with relevance, but its robustness remains limited in longer, noisier interactions.Source: §6 · p. 10
  • Most other models steadily decline in extended dialogues, underscoring the difficulty current systems face in sustaining high likability across real-world-like exchanges.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT.

The reproducibility statement points readers to the paper’s data-generation, evaluation, metric, and experimental details, with additional results and examples in the supplement.

  • Section 4 and the Appendix describe the data-generation pipeline, personality facets, conversation-style dimensions, archetype biasing, and prior construction.Source: §7 · p. 10
  • Algorithm 1 specifies the evaluation framework, while Section 3.1 gives mathematical formulations for likability, adaptability, and memory metrics.Source: §7 · p. 10
  • Section 5.1 reports model versions and evaluation counts, while supplementary materials provide additional results, ablations, and profile examples.Source: §7 · p. 10

8 ETHICS STATEMENT.

LikeBench uses simulated users rather than real human participants, avoiding collection of personal or sensitive data. The authors acknowledge possible misuse of personalization research and frame the benchmark as controlled academic evaluation, not deployment.

  • All user interactions are simulated with psychologically grounded personas and conversation styles; the study involves no real human participants.Source: §8 · p. 10
  • The design avoids collecting personal, private, or sensitive data and the associated ethical risks of real human experimentation.Source: §8 · p. 10
  • The benchmark is intended for academic research and evaluation rather than deployment, with controlled conditions and documentation; the authors flag risks including profiling, bias amplification, and unfair treatment.Source: §8 · p. 10

A DISCUSSION & LIMITATIONS

LikeBench’s static persona-driven evaluation cannot represent the full spectrum of real-world personality or reciprocal changes between a user and model. The discussion also identifies specific scoring effects and character-consistency issues under this setting.

A DISCUSSION & LIMITATIONS

  • Static personas constrain the benchmark: its discrete traits may not capture real-world personality, and users cannot change reciprocally during an interaction.The authors describe dynamic, reciprocal adaptation as a broader challenge for current benchmarks.Source: A DISCUSSION & LIMITATIONS · p. 13
  • GPT-5’s humor fit appears penalized when it introduces jokes over time despite a humor-averse profile that remains fixed.The authors attribute this apparent penalty to the benchmark’s static user profiles.Source: A DISCUSSION & LIMITATIONS · p. 13
  • Claude 3.7 Sonnet sometimes broke character midway through evaluation, followed by a sharper performance decline after session 5.Source: A DISCUSSION & LIMITATIONS · p. 13
  • The authors found no evidence of self-bias when Claude 3.7 Sonnet evaluated its own model, which ranked fourth overall.Source: A DISCUSSION & LIMITATIONS · p. 13

B EXPERIMENTATION & RESULTS

The authors use turn-level preference tracking to address preference drift during longer conversations. Dynamic User Profiles improved likability for the two top-performing models, but not for the others.

B EXPERIMENTATION & RESULTS

  • GPT-5 improved from 3.939 to 4.055 (+2.95%) and Claude Sonnet 4 from 3.828 to 3.914 (+2.25%) with Dynamic User Profiles.The gains occurred without additional training.Source: §B.1 · p. 13
  • Dynamic User Profiles track inferred preferences after each turn in a compact, schema-guided summary of personality and conversation style.The ground-truth persona remains fixed; the model’s profile is an evolving approximation inferred from dialogue.Source: §B.1 · p. 13
  • Qualitative analysis found that accumulated context and off-topic noise dilute preference signals as conversations lengthen, while most models did not improve with Dynamic User Profiles.Source: §B.1 · p. 13

B.2 PROFILE WISE PERFORMANCE.

Likability varies with user profile type, though the strongest systems remain stable across social and anti-social profiles. Turn-level results also show likability rising as models calibrate to each session’s topic.

B.2 PROFILE WISE PERFORMANCE.

  • GPT-5 and Claude Sonnet 4 show similar medians and tight IQRs across social and anti-social profiles, while DeepSeek R1 rises and Claude 3.7 Sonnet falls on anti-social profiles.Qwen3 remains lowest overall but narrows its gap on anti-social cases; personality is reported to shape likability more than topical alignment.Source: §B.2 · pp. 13–14, Figure 6 · p. 14
  • Top models sustain high likability with both socially warm and analytically detached users.Source: Figure 6 · p. 14, §B.2 · p. 14

B.3 TURN-WISE LIKABILITY PERFORMANCE.

C.1 DATASET DISTRIBUTION

The appendices describe the profile population, its psychological and conversational-style taxonomy, and the procedures used to generate personas, conversation agendas, and evaluation prompts.

C.1 DATASET DISTRIBUTION

  • Profiles span the late teens through the late 60s, include multiple ethnic identities, and are roughly balanced by gender, with a small non-binary share.The profile visualizations show localized high-similarity pockets and relatively separable social and anti-social groups; some heatmap similarities reflect shared names.Source: §C.1 · pp. 14–15

C.2 PERSONALITY TRAITS & CONVERSATION STYLE

  • The taxonomy covers 35 personality facets and nine conversation-style dimensions, including Big Five traits, humor styles, and preferences such as formality and conversation length.The Big Five facets use five intensity levels.Source: §C.2 · pp. 15–16

D.1 PROFILE GENERATION

  • Persona generation samples demographic attributes and archetypes, and varies interest granularity to create profiles with distinct identifiers.Social users receive psychologically correlated interest paths, while anti-social users receive randomly selected interests.Source: §D.1 · p. 24

D.2 PRIOR GENERATION

  • Conversation agendas give social users structured, linear flows and anti-social users more diverse, sometimes ambiguous agendas; later priors build on earlier conversations and include callback opportunities.The prior-generation setup can vary the number of sessions, defaulting to 10.Source: §D.2 · pp. 24–25

D.3 METRICS

  • Each conversation prior targets specific combinations of seven likability metrics while preserving persona alignment and natural progression.Source: §D.2 · p. 24

D.4 LIKEBENCH

  • The appendix provides prompts for generating simulated-user queries and responses, evaluating assistant replies, and assessing the model’s recalled memories.Source: §D.4 · p. 26

Related papers

Loading 2512.13077v1…