Re-Centering Humans in LLM Personalization

Lechen Zhang; Jiarui Liu; Tal August

Source-linked AI summary

Re-Centering Humans in LLM Personalization

Lechen Zhang, Jiarui Liu, Tal August

arXiv:2606.06614v1Jun 4, 2026cs.CLcs.AIcs.HC

TL;DR

It remains unclear whether LLM personalization works for real users because evaluations mostly rely on synthetic data. Using human conversations and judgments across a three-stage pipeline, the paper finds limitations throughout personalization, with personalized responses judged no better than generic ones in 54.6% of cases.

Problem
Most LLM personalization evaluations use synthetic data, leaving unclear how well current systems work for real users.
Method
The paper compares human and synthetic data across attribute extraction, relevance matching, and personalized response generation using 550 conversations and human judgments.
Results
Across all stages, human data reveals noisy attribute extraction, over-selection of relevant attributes, and personalized responses judged no better than generic ones in 54.6% of cases.
Takeaways & Limitations
Human-aligned personalization evaluation remains difficult, and systems must cautiously select and incorporate user information to improve responses for people.
Takeaways & Limitations
The evaluated three-stage pipeline does not cover memory updates, outdated or conflicting attributes, or user control over retained information.

Abstract

from arXiv · show

Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

1 Introduction

This paper examines the gap between synthetic and human-data evaluations of LLM personalization through a three-stage pipeline: extracting user attributes, matching relevant attributes, and generating improved personalized responses. Human data exposes limitations at every stage, while lightweight training interventions improve alignment in the first two stages.

The study frames personalization as extracting stable user attributes, matching relevant attributes to prompts, and generating responses better than generic ones.
An additional 22% of extracted user attributes from real conversations are judged problematic compared to synthetic ones.
LLMs over-identify 20–40% more attributes as relevant than humans despite strong inter-LLM agreement.
In 54.6% of cases, humans judge LLM-generated personalized responses no better than generic responses.
A human-trained RoBERTa verifier safeguards attribute extraction, while supervised classification and GRPO reduce over-selection and improve relevance alignment.
The collected data supports deeper evaluation of personalization systems and study of how models can use information in ways humans find useful.

2 A Three-Stage Framework for LLM Personalization

The paper frames LLM personalization as a three-stage pipeline: extracting stable user attributes, matching relevant attributes to the current context, and generating a personalized response. This decomposition separates attribute quality, relevance, and contextually appropriate use for evaluation.

Motivation: Decomposing personalization makes it possible to diagnose where systems fail, unlike end-to-end benchmarks that treat quality as a single black-box task.Prior work motivates more explicit factorization of personalization into separate components.
Three-stage framework: The framework comprises attribute extraction, attribute relevance matching, and personalized response generation.The stages operate on conversation history, the current interaction context, and selected attributes, respectively.
Stage 1 — Attribute extraction: User attributes are stable, long-term, context-independent statements covering preferences and profile information.Examples include preferring bullet points or being a Java developer.
Stage 2 — Relevance matching: Attribute relevance concerns whether a user attribute should influence the response to a specific prompt.Semantic similarity may fail to capture personalization relevance, while lexically distant attributes can still affect a response.
Stage 3 — Personalized response generation: Successful personalized generation improves over a generic response after relevant attributes are selected.The framework evaluates whether attributes are used in a contextually appropriate and user-acceptable manner, rather than only whether they affect the output.

3 Re-Centering Humans in the Personalization Pipeline

The section argues that personalization research often substitutes synthetic personas, users, dialogues, or LLM judges for human input, leaving the faithfulness of these proxies unclear. It studies personalization with five recent open-weight and proprietary LLMs, including practical open-weight models with strong quality-cost tradeoffs.

Motivation: Existing personalization work commonly proxies human input with synthetic personas, simulated users, synthetic dialogues, or LLM-based judges.These designs enable large-scale evaluation, but it is unclear whether they faithfully simulate human users.
Models: The study focuses on five recent, widely used open-weight and proprietary LLMs.The models include Llama-3.3-70B, Qwen3.5-27B, Gemma-4-31B, Claude-Sonnet-4.6, and GPT-5.4.
Models: The open-weight models are practical choices for large-scale personalization experiments because they offer strong quality-cost tradeoffs.The open-weight models are Llama-3.3-70B, Qwen3.5-27B, and Gemma-4-31B.

4 Grounding personalization to human interactions (Stage 1)

Human conversations provide richer potential user diversity but make reliable attribute extraction substantially harder than synthetic data. Verification and refinement interventions improve alignment with human judgments, with supervised verification achieving the strongest practical verifier performance and refinement raising acceptance above 90%.

Real-user diversity: A 5,000-user WildChat subset becomes substantially more diverse than all compared synthetic datasets after filtering and diversity-based sampling.Diversity is measured by average pairwise inter-user cosine distance among sentence embeddings of extracted user attributes.
Attribute extraction difficulty: Human conversations yield lower attribute acceptance and higher uncertainty and rejection than synthetic conversations, making valid extraction considerably more challenging.The study compares human judgments of 1,983 extracted attributes from 77 users across real and synthetic conversations.
Extraction errors: Overgeneralization is the most common failure mode, occurring in 53.9% of 1,225 uncertain attributes.Other analyzed causes include missing evidence and task-context confusion, but the supplied passage truncates the latter’s percentage.
Verification intervention: Optimized verification prompts substantially improve recall, while a supervised RoBERTa verifier achieves the highest overall F1 and strongest practical precision–recall trade-off.After prompt optimization, GPT-5.4 and Claude achieve much higher recall than Llama-3.3-70B; Gemma-4-31B reaches an F1 score close to them with lower recall.
Verification intervention: A verify-and-refine step increases average human acceptance from 58% to above 90% on 250 attributes.Unsupported attributes are sent back to the extraction model for revision, indicating that many extraction errors are recoverable.

5 Aligning Personalized Relevance Selection with Human (Stage 2)

Humans and LLMs often disagree when selecting which user attributes should influence responses: LLMs over-select attributes despite strong internal agreement. Training-based alignment, especially GRPO, substantially improves agreement with human relevance judgments.

Human–LLM agreement: 0.426 versus 0.597: human annotators show substantial internal agreement, while LLMs agree even more strongly with one another but disagree systematically with humans.These are average pairwise Cohen’s κ values for humans and LLMs, respectively.
Relevance selection: 0.243 and 0.384: BM25 and semantic similarity achieve low F1, showing that lexical or semantic relatedness is insufficient for attribute relevance selection.LLM judges outperform both retrieval-based baselines, using the majority vote of three human annotators as ground truth.
Training-based alignment: More than 60% of attributes marked relevant by LLMs are judged irrelevant by humans, and prompt refinement provides only marginal alignment gains.This over-selection can introduce unnecessary or distracting personalization.
Training-based alignment: 0.641/0.611: GRPO raises Qwen3-4B’s F1/precision from 0.417/0.267, outperforming zero-shot methods and improving calibration toward human judgments.Both RoBERTa and GRPO improve over all zero-shot LLMs in F1 and precision, with GRPO achieving the strongest evaluated performance.
Training-based alignment: GRPO makes Qwen3-4B reject “the user is creative” as relevant to a basic factual definition, producing a more cautious decision boundary aligned with human judgments.Before training, the model justified relevance by suggesting the answer could connect to creative processes; afterward, it recognized the attribute would not change the factual answer.

6 Human-Judged Personalization Quality (Stage 3)

Human judgments show that personalized responses provide little consistent benefit over generic responses, while LLM judges systematically overestimate personalization quality. Learned reward models improve alignment only modestly, underscoring the difficulty of modeling subjective, user-specific preferences.

Human-Judged Personalization Quality: 54.6% of personalized responses receive human ratings no better than their generic counterparts, with proprietary models only slightly above the neutral baseline of 3.0.Open-source models such as Qwen3.5 and Gemma-4 can degrade response quality.
Human-Judged Personalization Quality: Humans rate personalization lower than all LLM judges, whose strongest Spearman correlation with human ratings reaches only about 0.37.Open-weight judges show very low correlations, making automated evaluation unreliable substitutes for human evaluation.
Human-Judged Personalization Quality: LLM judges often reward mechanical attribute invocation, whereas human annotators and GPT-5.4 show little sensitivity and Claude-S4.6 favors more implicit personalization.Explicitly mentioning attributes can appear personalized to models while feeling robotic, presumptuous, or intrusive to users.
Human-Judged Personalization Quality: Spearman r = 0.90, p = 0.04 between models’ explicit-mention rates during generation and their rewards for explicit mentions during evaluation.This association suggests that surface-level personalization preferences can transfer from generation to evaluation.
Human-Judged Personalization Quality: Learned reward models reach only around 0.3 Spearman correlation with human ratings, remaining far from reliable judges of personalization quality.This performance is comparable to stronger LLM judges such as Claude-S4.6 and GPT-5.4; moderate human agreement and subjective preferences motivate user-specific reward models.

7 Conclusion

Across attribute extraction, relevance matching, and personalized generation, evaluations using real human conversations and judgments expose limitations that synthetic data and LLM-based evaluation can obscure. Models infer noisy attributes, over-select relevant information, and often fail to produce responses humans prefer.

The study evaluates LLM personalization across three stages: attribute extraction, relevance matching, and personalized generation.
Real human conversations and judgments reveal limitations that synthetic data and LLM-based evaluation can obscure.
Models infer noisy attributes, over-select relevant attributes, and often fail to produce responses that humans prefer.

8 Limitations

The study’s limitations concern collapsed annotator variation, narrow cultural coverage, an incomplete abstraction of deployed personalization, and uncertain intervention generalization. These constraints motivate broader, more diverse, and multilingual human-centered evaluation.

Annotation and subjectivity: Consensus labels and average ratings simplify evaluation but collapse meaningful variation among annotators, despite personalization being inherently subjective.Users may reasonably prefer different adaptations, explicitness levels, or tones.
Cultural coverage: The primarily English-language data likely reflects mostly Western conversational norms, limiting conclusions about personalization across languages and cultures.What counts as helpful, intrusive, polite, or appropriately personalized may differ substantially across cultural contexts.
System scope: The evaluated pipeline abstracts personalization as attribute extraction, relevance selection, and response generation, omitting memory updates, outdated or conflicting attributes, and user control over retained information.The abstraction is useful for diagnosis but does not cover all aspects of deployed personalization systems.
Intervention generalization: The training interventions improve attribute verification and relevance selection on this study’s annotations, but their generalization across user populations, domains, and settings remains unestablished.Larger and more diverse human-centered datasets are needed to assess robustness beyond the evaluation setting.

9 Ethical Considerations · A WildChat Preprocessing Details

The paper highlights privacy, consent, overgeneralization, intrusiveness, and limited user control as ethical risks in personalization. Its WildChat preprocessing filters language, scripted accounts, user activity, and personal-assistant usage to construct real-user histories.

9 Ethical Considerations: Real conversations may contain sensitive information, so the study avoids identifying users and rejects treating inferred attributes as reliable memories without verification and user control.The analysis uses an existing public dataset and reports aggregate model behavior.
9 Ethical Considerations: Models can overgeneralize from limited evidence and use personal attributes intrusively, motivating evaluation of accuracy, respectfulness, and meaningful response improvement.The ethical standard concerns how information is used, not merely whether it is used.
9 Ethical Considerations: Human annotations involve subjective judgments about users’ preferences and may expose annotators to personal content, making such judgments useful but imperfect.Future systems should give users control over what is remembered, when it is used, and how it shapes responses.
A WildChat Preprocessing Details: 1,039,785 conversations form the WildChat source corpus for the real-user dataset.The corpus is attributed to WildChat (Zhao et al., 2024).
A WildChat Preprocessing Details: English filtering retains metadata-labeled English conversations and uses Gemma-4-31B to exclude conversations containing non-English content.This removes cases such as translation tasks with foreign text in user turns.
A WildChat Preprocessing Details: 5,777 scripted accounts are removed using hashed-IP user clustering and repeated 120-character window hashes, leaving 92,557 users.A user is flagged when a hash appears in five or more turns with coverage above 40%, or in eight or more turns absolutely.
A WildChat Preprocessing Details: 16,573 active user histories remain after retaining users with at least 3 source conversations and more than 15 total user messages.The activity threshold is applied across all conversations.
A WildChat Preprocessing Details: Users scoring below 0.6 on an LLM-based genuine-personal-assistant classifier are excluded during attribute extraction.The classifier score ranges from 0 to 1 and distinguishes personal-assistant use from API endpoints or jailbreak targets.

B Attribute Extraction Implementation · C User Diversity Sampling Details · D Uncertain Attribute Categorization

The paper specifies pipelines for extracting and consolidating attributes, sampling users across specialist–generalist diversity, and categorizing uncertain or rejected attributes. These procedures combine model-based filtering, embedding-based clustering and diversity sampling, and a two-stage GPT-5.4 categorization process.

B Attribute Extraction Implementation: Attributes are extracted with Llama-3.3-70B at temperature 0.8, using up to 4,096 output tokens per conversation.Conversations exceeding 50,000 characters are split into chunks of up to 50,000 characters before pooling and deduplication.
B Attribute Extraction Implementation: Attributes with confidence below 0.4 are discarded using self-reported confidence as a coarse first-pass filter.The authors note that more robust reliability estimates could improve this step, but leave that work for the future.
B Attribute Extraction Implementation: Remaining attributes are embedded with all-MiniLM-L6-v2 and clustered per user using agglomerative clustering with average linkage.Manual inspection of cosine similarity thresholds 0.6, 0.7, 0.8, and 0.9 found that 0.7 offered the best balance.
B Attribute Extraction Implementation: Cluster representatives are selected by proximity to the confidence-weighted centroid, while cluster confidence discounts correlated evidence sources.The aggregation uses an independence-aware formula.
C User Diversity Sampling Details: K = 17 merged-attribute embeddings define each user’s generalist score as mean pairwise cosine distance, capturing semantic breadth.K is set to the p50 of eligible users’ attribute counts; higher scores indicate broader attribute topics, while lower scores indicate specialist users.
C User Diversity Sampling Details: Users are divided into five equal-frequency generalist-score quintile bins, then sampled by greedy farthest-point selection within each bin.Slots are allocated proportionally across bins to span the specialist-to-generalist spectrum.
D Uncertain Attribute Categorization: Human annotators flagged 1,225 unique attributes as uncertain or rejected across WildChat, PersonaLens, CUPID, and PrefEval.The counts were 868 from WildChat, 134 from PersonaLens, 133 from CUPID, and 90 from PrefEval.
D Uncertain Attribute Categorization: A two-stage GPT-5.4 pipeline infers a free-form reason from annotation context and assigns one of six categories.The resulting category distribution and representative examples are presented in Table 5.

E RoBERTa Attribute Verifier: Training Details

The attribute verifier fine-tunes RoBERTa-base on human quality annotations to classify extracted attributes as accepted or rejected. It uses evidence-aware inputs, recall-focused training, and user-level data splits to reduce contamination.

RoBERTa-base is fine-tuned on Task 4 attribute-quality annotations to classify each extracted attribute as accepted or rejected.
Input format: Each example concatenates a conversation snippet and attribute as [CLS] {conversation snippet} [SEP] {attribute} [SEP], truncated to 512 tokens.The snippet is the evidence excerpt shown to human annotators.
Training setup: 10 epochs, batch size 16, and learning rate 2 × 10−5 define the training setup.
Training setup: Weighted cross-entropy with Fβ weighting (β = 2.0) prioritizes recall over precision because missed unsupported attributes are more harmful downstream than false positives.
Training setup: Data is split at the user level to prevent attribute-level contamination across train / validation / test sets.

F Relevance Alignment: Training Details … K Licenses

The paper details training and annotation procedures for relevance alignment, documents GRPO’s more calibrated reasoning, and specifies prompt, language-model-use, and licensing practices. Human relevance judgments were substantially stricter than LLM judges, while data and code are slated for public release with privacy safeguards.

F Relevance Alignment: Training Details: RoBERTa is fine-tuned on Task 5 human relevance annotations using 10 epochs, batch size 16, learning rate 2 × 10−5, and prompt-level train/validation/test splits.Examples use [CLS] {prompt} [SEP] {attribute} [SEP] and are truncated to 256 tokens.
F Relevance Alignment: Training Details: GRPO trains Qwen3-4B for relevance matching with 5 sampled rollouts per prompt and rewards derived from human majority-vote relevance labels.Training uses 4× A100 80 GB GPUs, 15 epochs, batch size 128, actor learning rate 1 × 10−6, KL coefficient 0.001, and specified prompt and response length limits.
G Human Annotation Details: Three Prolific-recruited annotators completed all tasks through a custom web interface, selected for quality and self-consistency and paid at an estimated $18 per hour.The study used task-specific interfaces and reports compensation as above common minimum-wage standards.
G Human Annotation Details: Approximately 20% of attributes were marked relevant by humans, versus 40–60% by LLM judges across 3,973 attribute–prompt pairs.Each pair received judgments from all three annotators.
H GRPO Relevance Reasoning Example: The base model treated any possible attribute–response connection as sufficient for YES, whereas GRPO self-corrected toward a more calibrated relevance standard.The comparison uses the same attribute–prompt pair and presents GRPO as more human-aligned in its reasoning.
I Prompts: The prompts were iterated sufficiently for fair comparisons, although the paper acknowledges that prompt quality can affect model performance and that exhaustive prompt testing was infeasible.Prompt optimization is explicitly not the paper’s main focus.
J Use of Large Language Models: LLMs were used only to check grammatical errors and improve clarity of expression in the paper.The stated use concerns writing assistance rather than analysis or evaluation.
K Licenses: All data and code will be released under CC BY-SA 4.0 after manual inspection for personally identifying or offensive content, with results analyzed only in aggregate.The paper avoids reporting user-identifying examples to reduce privacy and exposure risks.