PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

1 Introduction

• Evaluators judge whether a response uses history appropriately, omits applicable information, or uses information that should not apply.Scenarios specify whether prior information should persist, transfer to a related task, or cease to apply.Source: §1 · p. 2
• PERSEVALBENCH contains 403 scenarios and 806 responses spanning nine controlled error subtypes across under-personalization, over-personalization, and misapplied-state errors.Human audits validate both the generated errors and the prescribed uses of user information.Source: §1 · p. 2
• Under-personalization recognition was 94.4% for humans versus 63.9% for the strongest evaluator, and 26.6% of under-personalized responses were labeled OVER.Highlighting relevant history improves error recognition only slightly.Source: §1 · p. 2
• Evaluators often detect a problem but reverse its direction, treating failure to follow an applicable preference as use of an inapplicable one.The benchmark matches RIGHT and error responses while keeping the task fixed and changing how user information is used.Source: §1 · p. 2

2 Related Work

• Assistant-side benchmarks examine memory, preference following, selective use of stored information, and evolving user states, while generally assuming judge competence.The cited work includes persona-grounded dialogue, long-term memory, preference following, and contextually appropriate memory use.Source: §2 · p. 3
• LLM-judge meta-evaluations compare judges with human assessments or controlled labels across instruction following, reward modeling, and knowledge and reasoning tasks.Prior work also documents position, verbosity, and self-generation biases.Source: §2 · p. 3
• Unlike Personalized Judge, PERSEVALBENCH evaluates judgments of fixed responses using multi-episode histories and matched pairs isolating three personalization error families.The matched pairs constrain factual correctness, relevance, completeness, specificity, and style, leaving persona-information use as the difference.Source: §2 · p. 3

3.1 Task Definition

• Evaluators classify each response as RIGHT, UNDER, or OVER based on whether history is used appropriately, omitted despite being applicable, or used when it should not apply.For error verdicts, the evaluator also identifies a primary subtype.Source: §3.1 · p. 3

3.2 Controlled Scenario Specification

• Scenarios test whether prior information persists, transfers to a related task, or ceases to apply, with each state assigned an intended influence on the target response.States may be ignored, support the response, or dominate it; interventions represent under-personalization, over-personalization, or misapplied-state errors.Source: §3.2 · p. 4

3.3 Benchmark Realization

• Each error response differs from its matched RIGHT response by one controlled intervention, while generation and validation preserve the scenario’s prescribed personalization logic.Matched responses are constrained to be comparable in factual correctness, task relevance, completeness, specificity, and style.Source: §3.3 · pp. 4–5

3.4 Human Verification

• Human audits found 88% raw subtype fidelity, 86.3% agreement with prescribed state roles, and 83.5% agreement across 200 human-reference judgments.Flagged responses were revised and re-verified before inclusion.Source: §3.4 · p. 5

4 Experimental Setup

• The study evaluates five LLM judges on PERSEVALBENCH, which retains 403 of 456 instantiated scenario specifications after verification and quality control.Each retained scenario pairs one RIGHT response with one controlled error response.Source: §4 · p. 6
• Evaluation varies the available history, judgment formulation, and protocol, with Full history and End-to-end prompting as the default setting.Context conditions include Target Only, Recent, Full, Full + Gold Highlights, and Gold Evidence Only; formulations include End-to-end, Evidence-first, and State-table first.Source: §4 · p. 6
• Evaluator-specific results are partially omitted from the public version because of release restrictions.Source: §4 · p. 6
• Error Recognition Rate measures correctly directed error verdicts, while False Alarm Rate measures incorrect error verdicts on matched RIGHT responses.ERR alone can be inflated by indiscriminately predicting errors; pair accuracy additionally requires accepting the RIGHT response and recognizing its paired error.Source: §4 · p. 6

5.1 Overall Reliability and the Under-Personalization Failure

• A constant-OVER baseline reaches 67.7% ERR but falsely flags every RIGHT response; Kimi K2.5 pairs 66.5% ERR with 24.6% FAR, while Qwen3-32B pairs 32.0% ERR with 2.5% FAR.The evaluators occupy different operating regimes, so their error-recognition rates must be considered alongside false alarms.Source: §5.1 · pp. 6–7
• Humans recognize 94.4% of under-personalization errors, versus 63.9% for the strongest evaluator; evaluators label 26.6% of these responses OVER.The wrong-direction rate remains 19.5–21.1% after excluding human-flagged secondary OVER errors; rationales cite genuine history but assign it the wrong applicability, scope, or weight.Source: §5.1 · pp. 6–7
• Underweighting errors split evaluators between UNDER and OVER verdicts, reflecting insufficient influence from the applicable state or excessive influence from a competing consideration.The two verdicts capture complementary readings of the same error.Source: §5.1 · p. 7

5.2 Evidence Access Does Not Resolve Applicability

• Removing applicable-state turns raises FAR by 55.2 points and lowers pair accuracy by 44.4 points, while highlighting relevant evidence improves ERR only slightly.Removing the same number of unrelated turns changes FAR and pair accuracy by only 2.0 and 3.2 points; Gold Evidence Only leaves ERR nearly unchanged but raises FAR to 24.0%.Source: §5.2 · p. 7
• With Target Only, under-personalization recognition falls to 31.7%, while unsupported inference and underweighting remain at 74.9% and 70.9%.Response-visible inference or weighting errors can be recognized without history, but omission errors require recovering what the history should have contributed.Source: §5.2 · pp. 7–8
• Evaluators reach lower ERR on graded applicability decisions involving strength or transferability than on categorical boundaries involving holder, scope, or temporal status.The difficulty extends beyond finding relevant information to judging how strongly and broadly it should apply.Source: §5.2 · p. 8

5.3 Structured Prompting Changes Verdict Tendencies, Not Judgment Quality

• Evidence-first prompting raises ERR but also FAR, while the state-table formulation lowers ERR and generally lowers FAR; neither improves recognition while preserving acceptance of RIGHT responses.Source: §5.3 · p. 8

5.4 Independent Judgment Remains Difficult Without an Explicit Contrast

• Pointwise scoring separates RIGHT from error responses in 63.0% of matched pairs, with 35.9% ties; direct pairwise comparison reaches 89.0% accuracy with 90.5% order consistency.Pairwise comparison exposes responses side by side, unlike the independent target setting, and neither protocol identifies error direction.Source: §5.4 · p. 8

6 Conclusion

• PERSEVALBENCH tests whether evaluators can judge how user history should influence responses, finding persistent errors even with full history and a pronounced under-personalization direction reversal.The results suggest reliable evaluation requires interpreting the role of history, not merely retrieving it.Source: §6 · p. 8

Construction

• LLM-generated dialogues and responses may differ from organic interactions or retain artifacts, and restricted-context generation or upstream reliance on Claude may create broader construction-family effects.A Mistral Large 3 response-generation robustness check is reported in Appendix C.6.Source: Limitations · p. 8

Evaluation scope

• Each response targets one controlled error, while compound errors remain outside the benchmark’s intended scope and confidence calibration is left to future work.ERR and FAR are reported at each model’s observed operating point.Source: Limitations · pp. 8–9

Generalizability

• English-only evaluation grounded in SGD-derived task seeds may not generalize fully across domains, interaction styles, languages, or cultural norms of personalization.Source: Limitations · p. 9

• All dialogues, personas, and responses are synthetic and generated from structured specifications grounded in task seeds derived from Schema-Guided Dialogue.Source: Ethical Considerations · p. 9
• The benchmark uses no real user data or personally identifying information, and its graduate-student volunteers participated with informed consent.Source: Ethical Considerations · p. 9
• PERSEVALBENCH is intended for evaluator-reliability research, and its synthetic, English-only, task-oriented results should not certify deployment reliability.Overgeneralization could obscure failures involving real users, other languages, or culturally dependent personalization norms.Source: Ethical Considerations · p. 9

• Table 6 uses ✓ for explicit coverage, △ for partial or narrowly scoped coverage, and – for a capability that is not a primary evaluation target.Source: A Benchmark Comparison · p. 11

B.1 Scenario Grammar and Coverage

• The scenario grammar permits 19 of 27 relation–subtype combinations; a human-authored compatibility matrix excludes implausible pairings such as scope leakage across domains.Subtype precedence distinguishes explicit violation from complete omission and scope leakage from irrelevant use.Source: §B.1 · p. 11

B.2 State-Relation Inventory

• Each scenario contrasts applicable state A with competing state B through one primary relation, drawn from eight relations grouped into five applicability families.Source: §B.2 · p. 11

B.3 Benchmark Statistics

• Table 9 organizes benchmark statistics into scale, composition, and realized evidence form and distance.Source: §B.3 · p. 11

C.1 Evaluation Formulation

• The reported evaluators are tested under three elicitation formulations using zero-shot evaluation, without hyperparameter search or model selection.Source: §C.1 · p. 11

C.2 Underweighting Verdict Split

• Underweighting rationales split between a deficit reading of downplayed constraints and an excess reading that treats weakened preferences or dominant secondary considerations as unsupported or misapplied.Source: §C.2 · p. 11

C.3 UNDER→OVER Robustness

• Excluding items flagged for secondary OVER errors lowers the UNDER→OVER rate from 26.6% to 21.1%; requiring both annotators to confirm intended UNDER leaves 19.5%.Residual confusion ranges from 14.3% for complete omission to 26.4% for explicit violation; audited OVER rationales cite genuine history evidence and err on applicability, scope, or weight, so secondary errors amplify but do not explain the confusion.Source: §C.3 · p. 11

C.4 Relevant-History Ablation

• A manual audit traced 78% of Gold Evidence Only false alarms to context lost when surrounding dialogue was removed.Evaluators sometimes mistake correct use of the applicable state for importing information from another task or treat details supported by removed turns as unsupported.Source: §C.4 · p. 12

C.5 Alternative Judging Protocols

• Pairwise comparison reaches 89.0% accuracy with 90.5% order consistency, while pointwise scoring separates responses in 63.0% of pairs and ties 35.9%.Neither protocol assigns an error direction, so neither exposes the UNDER→OVER confusion measured by the three-way verdict.Source: §C.5 · p. 12

C.6 Response-Generator Robustness

• Regenerating matched responses with Mistral Large 3 changes pair accuracy by only −0.5 points on average across four evaluators outside the construction model family.Scenario planning, specification, and dialogue synthesis still use Claude, so broader construction-family effects cannot be excluded.Source: §C.6 · pp. 12–13

C.4 Relevant-History Ablation

• Removing relevant history leaves ERR largely intact but turns most RIGHT responses into false alarms; removing unrelated context barely changes either metric.The matched ablations delete the same number of evidence or non-evidence turns on a stratified sample of 50 scenarios.Source: §C.4 · pp. 11–12, Table 11 · p. 13

Human validation protocol

• Three studies assessed intervention fidelity, state-role policy, and human evaluator judgments on the same stratified sample of 100 scenarios.Study 3 used independent, unadjudicated labels; misapplied-state cases were judged separately, potentially benefiting from a narrower judgment space.Source: D Human Validation · pp. 13–14

Study 1: Intervention fidelity

• Annotators confirmed the intended error in 88% of raw generations; 12 intervention-realization mismatches were revised and re-verified for the final benchmark.Explicit violation and scope leakage had the lowest raw confirmation rates, at 67% and 70%, respectively.Source: D Human Validation · p. 14

Study 2: State-role validation

• After adjudication, 86.3% of state-role assignments match benchmark policy; on evaluator disagreements, both annotators support the benchmark verdict in 72.8% and both reject it in 7.8%.Independent exact three-level agreement is 70.0% (weighted κ=.709).Source: D Human Validation · p. 14

Study 3: Human evaluator reference

• Both human annotators recognize 94.4% of under-personalization errors and significantly outperform every evaluator on that family (paired tests, p ≤.004, n=36 paired scenarios).Inter-annotator agreement across 200 judgments is 83.5% (Cohen’s κ=0.72).Source: D Human Validation · p. 14

E Worked Examples

• Examples 1–5 illustrate all nine error subtypes across stale or underweighted state, scope leakage, omission, violation, and over-personalization cases.The examples show traps such as anchoring on an emphatic but superseded constraint, acknowledging a hard constraint while giving it no force, and importing a genuine state across an irrelevant role boundary.Source: E Worked Examples · pp. 15–16

F Prompt System

• The appendix presents prompts 1–9 for benchmark construction and prompts 10–12 for eliciting evaluator judgments, with each card specifying its purpose, input, and output contract.Source: F Prompt System · p. 17

F.1 Generation Pipeline

• The generation pipeline fixes scenario labels before dialogue generation and designs applicable State A against a plausible but inapplicable State B.Its controlled difficulty factors include implicit or accumulated evidence, competing states, distant evidence, and weighting errors; the error response either withholds or weakens A or actively applies B.Source: §F.1 · p. 17
• Subtype admissibility tests require independently verifiable errors, including hard-constraint failure for underweighting, a changed choice for stale-state use, and a materially consequential scope boundary crossing.The specification verifier then checks label soundness, a single primary error, family-specific logic, and structural separation between states.Source: §F.1 · pp. 17–19
• The pipeline expands accepted specifications into annotated, natural multi-episode conversations that express preferences explicitly, implicitly, or across accumulated turns.Scenario plans place applicable and decoy states in designated episodes and make the target request require personalization.Source: §F.1 · pp. 19–20
• Right responses apply applicable states, while error responses receive subtype-specific information to induce a single controlled intervention and are matched in quality and length.The specified subtype criteria define observable error behavior across omission, constraint violation, and inappropriate state-use patterns.Source: §F.1 · pp. 20–22
• Generated error responses undergo deterministic pre-checks and subtype-specific validation, with failures triggering targeted regeneration for up to three attempts.Pre-checks reject clarification requests, claims of missing context, duplicates of the matched right response, self-corrections, metadata leakage, and off-topic drift.Source: §F.1 · p. 22

F.2 Evaluation Formulations

• The end-to-end evaluator identifies request-relevant preferences, checks the response against them, and classifies its use of state as under- or over-personalized or appropriate.Prompt subtype identifiers may carry generation-time direction prefixes, but scoring follows the three error families and does not score the elicited subtype.Source: §F.2 · p. 23
• Prompt 10 distinguishes UNDER failures to use applicable information from OVER use of inapplicable information, classifying mixed cases by their primary mechanism.Its rubric includes omission, underweighting, and explicit violation for UNDER, and irrelevant use, scope leakage, stale or superseded information, unsupported inference, and over-dominance for OVER.Source: §F.2 · p. 23
• Prompt 11 first extracts relevant preferences, constraints, and states with their source turns and currency, then judges the response against that evidence using Prompt 10’s taxonomy and contract.The second stage receives the extracted evidence, user request, and assistant response.Source: §F.2 · pp. 23–24
• Prompt 12 searches dialogue history for candidate states and records each state’s holder, scope, status, strength, evidence turns, and policy before checking the response.The checks map missing DOMINATE states or underweighting them to UNDER, and use of IGNORE states or over-dominance of SUPPORT states to OVER; stale information is OVER, while simply forgetting a state is UNDER.Source: §F.2 · p. 24