Can LLM be a Personalized Judge?

• LLM personalization requires evaluation that reflects diverse user values, but current alignment often assumes homogeneous preferences and uses LLM-as-a-Judge without adequate validation.Source: §1 · p. 1
• Persona sparsity limits personalization because simple attributes may provide little task-relevant information about a user’s preferences.The paper contrasts profession-based personas with richer demographic, descriptive, and interaction-based profiles.Source: §1 · p. 1
• The study evaluates personalized judging against self-reported ground truth across multiple datasets and introduces verbal uncertainty estimation to identify judgments with insufficient predictive information.The workflow infers a preference from a persona, estimates certainty, and compares the prediction with the user’s reported preference.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Standard LLM-as-a-Judge reaches only around 70% agreement in binary personalization tasks, falling below 60% for some tasks, whereas certainty-enhanced judging exceeds 80% on high-certainty samples.The certainty-aware judge is comparable to third-person human evaluation and surpasses human performance on high-certainty samples.Source: §1 · p. 2
• First-person evaluation by the intended users remains the gold standard, while certainty-thresholded personalized judging is presented as a scalable alternative when such annotations are unavailable.First-person evaluation reflects the individuals’ own preferences and values.Source: §1 · p. 2

• Personalization tailors model outputs to individual preferences, needs, and behaviors, but LLM alignment increasingly must address pluralistic user needs.Source: §2 · p. 2
• Persona sparsity occurs when available persona variables are not relevant or informative for the task, making preference inference unreliable and detailed data collection impractical.The paper uses the term for insufficiently informative persona variables, distinct from earlier uses concerning scarce naturalistic dialogue data.Source: §2 · pp. 2–3
• The proposed response to persona sparsity is to use verbal uncertainty estimation to filter cases where the LLM cannot make a well-informed personalized judgment.Source: §2 · p. 3
• LLM-as-a-Judge is widely used because it often agrees with human annotators, yet its validity for realistic personalized settings remains insufficiently examined.Prior role-playing evaluation focused on simplistic professional personas rather than complex profiles containing demographics, descriptions, and prior interactions.Source: §2 · p. 3

• The judge receives demographic, socio-behavioral, descriptive, or other relevant persona information and selects a preferred response to a subjective question.The primary task is binary choice, with an additional setting that permits a tie response.Source: §3 · p. 3
• Verbal uncertainty estimation instructs the LLM to report how certain it is, targeting cases where persona information may not support accurate preference assessment.The workflow combines persona-conditioned preference inference with a certainty estimate.Source: §3 · p. 3

4.1 Datasets

• The datasets span conversations, surveys, empathetic responses, and Reddit posts, combining preference-related judgments with demographic, behavioral, or personal attributes.PRISM includes over 8,000 conversations from 1,500 participants across 75 countries; OpinionQA contains 1,498 survey questions; EC contains 1,000 essays; PR contains 500 Reddit samples.Source: §4.1 · pp. 3–4

4.2 LLM as Personalized Judge

• The study compares standard personalized judging, standard judging with 1–100 verbal certainty estimates, and judging with a third tie option.The tie-option setting does not allow verbal uncertainty.Source: §4.2 · p. 4
• Experiments evaluate GPT-4, GPT-3.5, Command R+, and Llama3 70B under shared generation settings, with limited regeneration when outputs are unparseable.Source: §4.2 · p. 4

Overall performance

• 72.5% average accuracy is achieved by the strongest personalized judge, falling to around 60% on challenging EC and OpinionQA tasks.This is below the 80+% agreement previously reported for simpler role-playing evaluations.Source: §5 · p. 4, Table 1 · p. 5

Task variation

• 94.6% accuracy on PR reflects a clear response-quality distinction, making the dataset closer to instruction-following or textual entailment than genuine personalization.EC is the most difficult task, with every model below 60%, likely because its persona information has limited predictive power and responses are broadly similar.Source: §5 · p. 4

Certainty estimation

• High-certainty GPT-4 and Command R+ samples reach approximately 80% agreement with human ground truth under a certainty threshold of 80.Higher self-assessed certainty is more predictive of correctness for stronger models, whereas GPT-3.5 often fails to quantify uncertainty effectively.Source: §5 · pp. 5–6, Table 2 · p. 5

Overall performance

• GPT-4 performs best across nearly all tasks, while GPT-3.5 trails by 5%–10% on average and performs poorly at estimating certainty.Adding a tie option generally preserves the model ranking but substantially harms performance on EC because models rarely select ties.Source: §5 · pp. 4–6

Certainty estimation

• Only around 1% of EC predictions are high-confidence, compared with around 50% for GPT-4 on PR and nearly 100% for Command R+ on PR.The larger share of low-confidence predictions on difficult tasks supports using confidence as an indicator of task difficulty and persona sparsity.Source: §5 · p. 6

Persona ablation

• On OpinionQA, GPT-4 high-confidence samples fall from 480 (16.0%) with full personas to 10 (0.3%) with one feature.Command R+ changes little because it already has very few high-confidence samples.Source: §5 · pp. 6, 8, Table 3 · p. 6
• On PRISM, GPT-4 high-confidence samples decrease only from 120 to 92 when moving from full to one-feature personas while retaining high accuracy.The small change is attributed to large inherent quality differences between response options, which make the preferred choice evident regardless of persona detail.Source: Table 4 · p. 6, §5 · p. 8

Human comparison

• GPT-4 reaches 62.3% overall accuracy versus 63.3% for human annotators, but rises to 79.2% on high-certainty samples versus 71.4% for humans.The evaluation used 300 OpinionQA instances with three annotators per instance and majority-vote human answers.Source: Table 5 · p. 6, §5 · p. 8

6 Conclusion

• Standard LLM-as-a-Judge shows low agreement with human ground truth on personalization tasks.The paper identifies persona sparsity as a major cause of this unreliability.Source: §6 · p. 8
• Verbal certainty estimation enables powerful LLMs to identify high-certainty samples with approximately 80% accuracy.This suggests that certainty-aware judging can serve as a proxy when first-person preference data are unavailable, provided personas are sufficiently fine-grained.Source: §6 · pp. 8–9
• LLM-as-a-Personalized-Judge matches third-person human accuracy overall and surpasses human performance on high-certainty samples.The paper advocates collecting more first-person personalization data while treating certainty-aware judging as a promising evaluation proxy.Source: §6 · pp. 8–9

Limitations

• Available personalization evaluation datasets are limited and predominantly English, preventing conclusive claims about non-English performance.The paper notes that cultural gaps in multilingual LLMs may further affect non-English evaluation.Source: Limitations · p. 9
• The study uses direct verbal uncertainty estimation without comprehensively evaluating other uncertainty-estimation methods.A broader comparison with existing approaches is left for future work.Source: Limitations · p. 9

Ethical Considerations

• Personalization should remain opt-in so users retain autonomy and consent without adverse consequences for opting out.This requirement applies throughout research and production.Source: Ethical Considerations · p. 9
• Biases and stereotypes encoded in LLMs may be reinforced during the LLM-as-a-Judge process.The paper calls for attention to these risks when developing personalized judges.Source: Ethical Considerations · p. 9
• Personal information used to fine-tune or condition models requires explicit consent and responsible data handling under data-protection regulations.The study relies on publicly available datasets that were anonymized and received institutional review board approval.Source: Ethical Considerations · p. 9

A.1 LLM Model Details

• GPT-4o uses gpt-4o-2024-05-13, while GPT-3.5-turbo uses gpt-3.5-turbo-0125.Source: §A.1 · p. 12

A.2 Experiment Details

• PRISM experiments use 1,000 first-turn samples, exclude ties in two settings, include them in a third, and shuffle response positions.Responses generated by the judging LLM and refusals are filtered to mitigate self-enhancement and safety-related effects.Source: §A.2 · p. 12
• OpinionQA uses one binary-choice question from each of 15 topics and 200 randomly selected respondent answers per question.This sampling design supplies the preference judgments used in the OpinionQA experiments.Source: §A.2 · p. 12
• EC evaluates 500 essay-response comparisons, treating responses as ties when empathy or distress scores differ by less than 2.The tie proportion is controlled to approximately match PRISM’s setting-3 ratio of around 20%.Source: §A.2 · p. 12
• PR constructs comparison pairs by matching personas with cosine similarity and asking the model which response was written by the target user.The similarity uses the all-MiniLM-L6-v2 model from Sentence Transformers.Source: §A.2 · p. 12

A.5 Results With and Without Persona

• The evaluation includes persona-variable comparisons, third-person human assessment against GPT-4 judging, and prompt variants with or without ties.PRISM prompts require an A/B choice and certainty level, while tie-enabled prompts add a C option; other datasets receive minor adaptations.Source: Table 8 · p. 13, Table 7 · p. 13, §A.7 · pp. 15–16

A.6 Number of Persona Variables Provided Influence Certainty Distribution

• Fewer persona variables reduce GPT-4 and Command R+ confidence on OpinionQA, while PRISM’s large response-quality gap leaves its certainty distribution largely unchanged.The study reports this comparison in Table 3 across the two datasets.Source: §A.6 · p. 13