PaperlayerView exact revision on arXiv

Source-linked AI summary

PrefDisco: Benchmarking Proactive Personalized Reasoning

Shuyue Stella Li, Avinandan Bose, Faeze Brahman, Simon Shaolei Du, Pang Wei Koh, Maryam Fazel, Yulia Tsvetkov

arXiv:2510.00177v2cs.CLcs.AI

TL;DR

Human-facing LLMs must address user preferences even in cold-start settings, but current development treats correctness and preference alignment separately. PrefDisco evaluates interactive preference discovery and reasoning adaptation; across 21 frontier models, 29.0% of personalization attempts have worse preference alignment than generic responses. The findings point to personalized reasoning as a capability requiring dedicated development.

  • Problem

    LLMs are typically optimized for task correctness and then aggregate preference alignment, while cold-start interactions require addressing individual needs without prior user history.

    Source: §1 · p. 1

  • Method

    PrefDisco transforms reasoning benchmarks into interactive personalization assessments with sparse persona preferences and evaluates preference discovery and response alignment.

    Source: §1 · p. 2

  • Results

    29.0% of personalization attempts produce worse preference alignment than generic responses across evaluations of 21 frontier models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Personalized reasoning requires dedicated development rather than emerging naturally from general language understanding.

    Source: §1 · p. 3

    The evaluation focuses on beneficial personalization, not over-personalization or sycophancy, and does not capture preference evolution or conflicts across contexts.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

Current large language model (LLM) development treats task-solving and preference-alignment as separate challenges, optimizing first for objective correctness, then for alignment to aggregated human preferences. This paradigm fails in human-facing applications where solving a problem correctly is insufficient if the response mismatches the user's needs. This challenge intensifies in just-in-time scenarios where no prior user interaction history exists due to cold-start conditions or privacy constraints. LLMs need to proactively identify what they don't know about the user, strategically elicit preference values through questioning, then adapt their reasoning processes and responses accordingly -- a complicated chain of cognitive processes which we term personalized reasoning. We introduce PrefDisco, an evaluation methodology that transforms static benchmarks into interactive personalization tasks using psychologically-grounded personas with sparse, context-dependent preferences, and define PrefAlign as a fine-grained rubric-based metric for measuring preference alignment. PrefDisco builds scenarios where identical questions require different reasoning chains depending on user context, as optimal explanation approaches vary by individual expertise and preferences while maintaining factual accuracy. Evaluation of 21 frontier models across 10 tasks reveals 29.0% of naive personalization attempts produce worse preference alignment than generic responses, yet generic responses also fail to serve individual user needs. These findings suggest personalized reasoning requires dedicated development rather than emerging naturally. PrefDisco provides a foundation for developing systems that can adapt to individual users in education, healthcare, and technical domains where personalization is critical.

1 INTRODUCTION

Current LLMs separate task correctness from preference alignment, leaving cold-start users underserved when generic answers miss their needs. PrefDisco turns reasoning benchmarks into interactive personalization assessments, and its evaluation finds that personalization often performs worse than generic responses.

1 INTRODUCTION

  • In cold-start settings, users may be unable to articulate their needs, so models must elicit preferences rather than rely on interaction history or user feedback.This is especially relevant to human-facing tasks where the same problem calls for different reasoning approaches across users.Source: §1 · p. 1
  • PrefDisco transforms existing reasoning benchmarks into interactive assessments using psychologically grounded personas with sparse preferences and strategic questioning.It evaluates preference discovery and response alignment against no-personalization and known-preference baselines.Source: §1 · p. 2
  • 29.0% of personalization attempts produce worse preference alignment than generic responses, while generic responses still fail to address user needs.Evaluation covers 21 frontier models across 10 tasks.Source: §1 · pp. 1–2
  • Mathematical reasoning shows a 3.5% accuracy loss under personalization constraints, while social reasoning shows a 3.1% gain.The introduction reports these domain-specific differences among task types.Source: §1 · pp. 2–3
  • Personalized reasoning requires proactively discovering user preferences and adapting reasoning, while PrefAlign measures response alignment with a fine-grained rubric.The approach is distinguished from static persona consistency and content recommendation.Source: §1 · p. 3

2 PERSONALIZED REASONING

Personalized reasoning is formalized as identifying task-relevant attributes, eliciting user-specific preferences, and adapting responses while jointly requiring correctness and preference alignment.

2.1 IDENTIFYING RELEVANT ATTRIBUTES

  • Personalization begins by identifying the small subset of global response attributes relevant to each user-task pair.Examples include visualization or analogies for a physics explanation; other attributes may be irrelevant in that context.Source: §2.1 · pp. 3–4

2.2 ELICITING PREFERENCE VALUES

  • A preference profile captures each relevant attribute’s preferred value and relative importance, which can vary by user and instance.The model elicits the unobserved profile through sequential choices to ask about an attribute or answer, under limited interaction.Source: §2.2 · p. 4

2.3 ADAPTING RESPONSES AND EVALUATING ALIGNMENT

  • Responses are adapted to inferred priorities, then assessed with attribute-specific rubrics while jointly requiring objective correctness and preference alignment.The benchmark pipeline samples sparse, context-dependent preferences and generates rubrics for evaluation.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 PREFDISCO BENCHMARK CONSTRUCTION

PREFDISCO converts static benchmarks into interactive tests of cold-start preference discovery. It combines psychologically grounded personas, sparse contextual preferences, attribute-specific rubrics, and passive user simulation.

Benchmark construction

  • PREFDISCO targets cold-start evaluation by turning static benchmarks into interactive tasks designed to isolate preference discovery.Its construction decomposes personalization into attributes to expose capabilities and failure modes that holistic preference ratings can obscure.Source: §3 · pp. 5–6

Psychologically-Grounded Persona Generation

  • Personas use demographics, Big Five traits, and domain expertise conditioned on the International Personality Item Pool, with rejection sampling for diversity.Consistent persona traits support testing whether models transfer discovered preferences across tasks in a session.Source: §3 · p. 5

Context-Dependent Preference Instantiation

  • Each persona-problem pair receives a sparse, context-dependent preference profile; human validation of attribute labels achieved Fleiss kappa 0.463 and 61.5% accuracy against majority voting.Profiles activate only context-relevant attributes, and weights reflect persona-specific priorities.Source: §3 · p. 6

Evaluation Rubric Generation

  • Attribute-specific rubrics score responses from 1 to 5 across 10K scenarios, enabling fine-grained assessment at a scale considered prohibitively expensive for human annotation alone.Explicit criteria are intended to reduce susceptibility to hallucination and bias relative to an opaque holistic score.Source: §3 · p. 6

User Simulation

  • Passive users answer elicitation questions with minimal information, and the five-turn limit reflects attention constraints; performance curves plateau around 3–5 turns.This setup is intended to isolate model questioning capabilities from user proactiveness and communication style.Source: §3 · p. 6

4 EXPERIMENTS

The experiments test proactive preference discovery across diverse reasoning benchmarks, personas, and controlled evaluation conditions. They compare task-only, interactive-discovery, and oracle settings using preference alignment and task accuracy.

Benchmarks and Models

  • The evaluation spans 10 reasoning benchmarks and 21 frontier models, with 10,000 scenarios and a five-turn interaction limit.Scenarios sample 100 problems per benchmark and assign 10 personas to each problem.Source: §4 · pp. 6–7

Evaluation Conditions

  • Baseline, discovery, and oracle conditions distinguish task-only performance, interactive preference elicitation, and personalization with the full preference profile known upfront.The baseline-to-discovery gap measures interactive discovery performance, while baseline-to-oracle indicates an upper bound on personalization quality.Source: §4 · p. 7

Evaluation Metrics

  • Evaluation pairs preference alignment with original benchmark accuracy, and normalized alignment compares discovery against baseline and oracle performance.The normalized scale assigns 0 to no improvement over baseline and 100 to oracle-matching discovery performance.Source: §4 · p. 7

5 RESULTS

Personalization often worsens preference alignment and task accuracy, with outcomes varying by task and interaction strategy. More questioning correlates with better alignment, but domain-specific accuracy losses persist under controlled interaction lengths.

Preference Discovery Performance

  • MATH and LogiQA show the most degradation, with 10 and 11 of 21 models performing worse, while SocialIQA benefits most from interactive personalization.Claude Opus 4 is reported as most consistently positive, while o3-high exhibits extreme variance.Source: §5 · pp. 7–8
  • 29.0% of model–task combinations show negative normalized alignment, meaning discovery responses align worse than baseline responses without personalization.The score is normalized between baseline and oracle conditions; negative values indicate worse alignment than generic responses.Source: §5 · p. 7, Table 2 · p. 8

Interaction Efficiency and Preference Alignment Tradeoff

  • Question volume correlates positively with alignment (r=0.445, p<0.001), while models average 1.48 questions despite a five-turn allowance.Regression coefficients are Gemini (β=0.474), OpenAI (β=0.379), and Claude (β=0.117).Source: §5 · p. 8, Figure 3 · p. 9

Accuracy-Personalization Trade-off

  • Accuracy falls from Baseline (65.2%) to Oracle (61.8%) to Discovery (60.1%), with AIME losing 12.1% and CommonsenseQA gaining 5.4%.The Baseline-to-Oracle decline occurs without interactive discovery, indicating a cost associated with processing preference constraints.Source: §5 · p. 9
  • Models can fail to solve correctly when required to use a different cognitive toolkit, as reasoning remains tied to pathways reinforced during training.The authors connect this brittleness to reinforcement-learning-optimized reasoning paths that may not accommodate personalization constraints.Source: §5 · p. 9

Question Quality vs. Termination Decision Isolation

  • Under fixed questioning of 2, 4, or 8 questions, math and science tasks remain degraded while social reasoning improves, so the domain pattern is not explained by stopping decisions.The authors interpret this persistence as incompatibility between preference processing and formal reasoning.Source: §5 · p. 9

6 RELATED WORK

Prior personalization benchmarks generally assume known or static preferences, while interactive elicitation work has focused on narrower settings. PrefDisco combines interactive preference discovery with reasoning adaptation across domains.

Static Personalization and Evaluation Benchmarks

  • Existing benchmarks focus on dialogue, recommendations, or multi-session profiling rather than cold-start preference elicitation across reasoning tasks.Prior preference-modeling approaches also do not determine which attributes matter for specific user-task pairs or how to discover them interactively.Source: §6 · p. 10

Interactive Preference Elicitation

  • With fixed interaction lengths, math and science preference alignment is hindered while social reasoning alignment improves.The figure caption describes this as a domain-dependent effect of fixed interaction length.Source: Figure 5 · p. 10
  • GATE and MediQ demonstrate interactive questioning, but target intent understanding or clinical information-seeking rather than adaptive reasoning across diverse domains.PREFDISCO combines interactive preference discovery with adaptation of the model’s cognitive approach.Source: §6 · p. 10

7 DISCUSSION AND FUTURE WORK

The discussion frames personalized reasoning as a distinct capability and reports domain-specific accuracy and alignment failures. PrefDisco offers an evaluation foundation, while the authors identify further research directions and applications.

Discussion

  • Personalized reasoning requires adapting cognitive processes to discovered preferences, not merely changing response style; 29.0% of personalization attempts align worse than generic responses.The discussion reports universal degradation for mathematical reasoning and robustness in social reasoning.Source: §7 · p. 10
  • PrefDisco turns static benchmarks into interactive personalization assessments and finds that questioning correlates with alignment while preference constraints carry accuracy costs.Models average 1.48 questions despite five-turn allowances.Source: §7 · p. 10

Future Work

  • Future work includes attribute-specific alignment analysis, reinforcement learning with multidimensional rewards, and cross-task preference transfer.The framework is presented as a technical foundation for adaptive systems in education, healthcare, and technical domains.Source: §7 · p. 10

LIMITATIONS

The evaluation covers beneficial personalization but leaves negative effects unexamined. Its simulated interactions and preference scope also limit what it assesses.

Limitations

  • The evaluation excludes over-personalization and sycophancy, including cases where preference agreement could reduce response quality or factual accuracy.It does not assess possible information bubbles or prioritizing agreement over helpful feedback.Source: LIMITATIONS · p. 11
  • Simulated interactions may not capture the complexity of real preference expression, and the framework omits content preferences, preference evolution, and contextual conflicts.The current evaluation focuses on communication preferences.Source: LIMITATIONS · p. 11

ETHICS STATEMENT

The paper flags possible misuse of personalization for manipulation or reinforcing harmful biases, and says its framework does not address when or how personalization should be applied.

ETHICS STATEMENT

  • Personalization could be misused for manipulation or to reinforce harmful biases; the evaluation measures technical capabilities, not when or how personalization should be applied.Future deployments should include safeguards against over-personalization, preserve factual accuracy despite preferences, and disclose how preferences are discovered and applied.Source: ETHICS STATEMENT · p. 11

A EVALUATION DETAILS

The evaluation spans 21 models, ten benchmarks, and three response conditions, with preference alignment scored by a configured LLM judge.

Model Configurations

  • The evaluation covers 21 frontier models across OpenAI, Google, and Anthropic, using temperature 0.7 and high reasoning effort.The model set includes seven models from each family.Source: A EVALUATION DETAILS · p. 14

Benchmark Selection

  • Ten benchmarks cover mathematical, logical, scientific, general-knowledge, and social reasoning tasks.The benchmarks include MATH-500, AIME, LogiQA, MascQA, ScienceQA, MedQA, MMLU, SimpleQA, CommonsenseQA, and SocialIQA.Source: A EVALUATION DETAILS · p. 14

Experimental Protocol

  • Each benchmark uses 100 personas and 100 problems, producing 1,000 scenarios per task and 10,000 overall, with interactions capped at five turns.Each problem is assigned to ten personas with partial overlaps.Source: A EVALUATION DETAILS · p. 14

Evaluation Conditions

  • Baseline, Discovery, and Oracle conditions distinguish responses without preferences, interactive elicitation, and access to complete preference profiles.The three conditions isolate interactive discovery from general personalization and establish performance bounds.Source: A EVALUATION DETAILS · p. 14

LLM Judge Configuration

  • GPT-4.1 scores each rubric criterion independently, and the final PREFALIGN score is the preference-profile-weighted average of those scores.The judge uses temperature 0.1, a 4,096-token limit, top-p 0.95, and high reasoning effort.Source: A EVALUATION DETAILS · p. 14

B HUMAN ANNOTATION DETAILS

Human annotators validate which preference attributes matter for task scenarios, while qualitative examples illustrate varied outcomes across baseline, discovery, and oracle personalization.

B HUMAN ANNOTATION DETAILS

  • For annotation, three expert annotators judged attribute relevance across 20 scenarios, with 20 candidate attributes per scenario and 400 labels per annotator.An attribute counted as relevant when it could change reasoning strategy or explanation style, rather than only superficial wording.Source: B HUMAN ANNOTATION DETAILS · p. 15
  • Annotators achieved Fleiss’ Kappa of 0.463, characterized as moderate agreement, and 61.5% average accuracy against majority vote.The authors consider this agreement acceptable for subjective judgments about communication and preference relevance.Source: B HUMAN ANNOTATION DETAILS · p. 16

C QUALITATIVE EXAMPLES

  • The qualitative examples show expected PREFALIGN gains from Baseline to Discovery to Oracle, discovery failures, and cases where personalization reduces answer correctness.Each vignette is sampled from one of the 21 evaluated models.Source: C QUALITATIVE EXAMPLES · p. 16

C.1 AIME

  • In the AIME example, Discovery has lower PREFALIGN than Baseline, while Oracle scores higher than both.The reported ordering is Discovery < Baseline < Oracle.Source: §C.1 · p. 16

Persona

  • The school-party example uses a 70-year-old Vietnamese floral designer’s persona and asks how many students dance after 20 boy dancers join the party.The baseline response is generic and asks no questions.Source: Persona · pp. 16–17

Assistant: claude-3.5-haiku

  • For the school-party problem, Baseline is correct with PREFALIGN 2.82, Discovery is correct at 2.56 after failing to elicit meaningful preferences, and Oracle scores 4.06.The discovery interaction focused on step-by-step explanation despite missing meaningful user preferences.Source: Persona · p. 17, Assistant: claude-3.5-haiku · pp. 18, 19, 21

Assistant: claude-3.5-haiku

In an AIME example, the baseline answers correctly while both personalized modes answer incorrectly; a separate oracle example illustrates preference-matched explanation choices.

Assistant: claude-3.5-haiku

  • The oracle response to the school-party problem uses a flower-arrangement analogy and receives Oracle PREFALIGN 4.06 with a correct answer.Its evaluation highlights analogy and practical-application criteria as strong matches for the user’s preferences.Source: Assistant: claude-3.5-haiku · pp. 21–22
  • In the AIME example, the baseline is correct, but both Discovery and Oracle produce incorrect responses.The example describes personalization as leading to the incorrect answers.Source: Assistant: claude-3.5-haiku · p. 23, Persona · p. 23

Persona

  • The AIME task asks for the sum of positive integers b < 1000 for which base-b 36b is a square and base-b 27b is a cube.The example’s persona is an archaeological site surveyor and GIS specialist from Yellowknife.Source: Persona · p. 23

Assistant: gemini-1.5-pro

After asking about Dmitri’s math background and desired solution style, the assistant gave a simplified search, but its Discovery Mode answer was marked incorrect. Its evaluation nevertheless rated several preference dimensions highly.

Assistant: gemini-1.5-pro

  • The assistant ended with 371, but Discovery Mode marked the answer incorrect and assigned a PREFALIGN score of 2.67.Source: Assistant: gemini-1.5-pro · pp. 24–25
  • The evaluation gave the response scores of 5 for listed dimensions including verification steps, explanation depth, and self-paced learning, while the baseline evaluation records several unmet preferences.Source: Assistant: gemini-1.5-pro · pp. 24–25
  • The report says the assistant inferred limited higher-level math experience, avoided higher-level math, and consequently got the answer wrong.Source: Assistant: gemini-1.5-pro · p. 25
  • The assistant asked about Dmitri’s math background and whether he wanted a quick solution or conceptual explanation; he replied that he had a geography degree and wanted a quick solution.Source: User · p. 25
  • The response translated the base-b numbers into square and cube equations, tested several cube values, then recommended a program or spreadsheet for the remaining search and reported a sum of 4131.Source: Assistant: gemini-1.5-pro · p. 26

Assistant: gemini-1.5-pro

In Oracle Mode, the assistant tailored its explanation to Dmitri’s preferences but stopped before completing the search. The evaluation describes strong preference alignment alongside an incorrect answer.

Assistant: gemini-1.5-pro

  • The response used surveying and mapping analogies, a stepwise explanation, and a table to connect the base-conversion search to Dmitri’s background.Source: Assistant: gemini-1.5-pro · pp. 27–28
  • The assistant left the remaining calculations to Dmitri, and the Oracle Mode evaluation marked its answer incorrect with a PREFALIGN score of 4.21.Source: Assistant: gemini-1.5-pro · pp. 27–28
  • The Oracle evaluation awarded 5s for listed dimensions including number-theory comfort, explanation depth, coordinate-system analogies, and practical examples.Source: Assistant: gemini-1.5-pro · p. 28

C.2 MEDQA

The MedQA examples show both favorable and unfavorable outcomes: personalization can improve preference alignment while answers remain correct, but personalized responses can also be incorrect. The discovery example also shows limited questioning.

C.2 MEDQA

User

In this clinical-question scenario, the assistant’s response changed after it elicited a preference for directness, and the final answer was incorrect. The paper’s account contrasts that outcome with an earlier, detailed answer that selected a different option.

User

  • The initial task described a three-month-old with feeding-related breathing difficulty, listlessness, and a holosystolic murmur.Source: User · p. 38

Assistant: gemini-2.5-pro

  • An earlier response reasoned through the infant’s symptoms as a ventricular septal defect and chose A. 22q11 deletion.Source: User · p. 38
  • The paper attributes the wrong answer to the assistant spending less time reasoning after the user requested a direct answer.Source: User · p. 40
  • Discovery Mode scored 2.98 on PREFALIGN and answered incorrectly.Source: User · p. 40
  • The assistant first asked whether the user preferred a direct answer or a detailed walkthrough, then selected D. Maternal alcohol consumption.Source: Assistant: gemini-2.5-pro · p. 40

Assistant: gemini-2.5-pro

The examples examine how personalized reasoning affects preference alignment and correctness across benchmark questions, including social-behavior and safety-critical medical scenarios. They show personalized responses reflecting user-specific explanation preferences and sometimes outperforming baselines, while alignment gains do not guarantee correctness or fully meet user needs.

Assistant: gemini-2.5-pro

  • Oracle mode scores PREFALIGN 4.67 but answers incorrectly, showing that preference-driven reasoning can still fail in a safety-critical clinical case.The paper describes the reasoning chain as brittle after it leads to the wrong answer.Source: Assistant: gemini-2.5-pro · p. 42

C.3 SOCIALIQA

  • On SOCIALIQA, PREFALIGN rises from Baseline through Discovery to Oracle, and all modes answer correctly.The example is presented as showing improved preference alignment after asking questions.Source: §C.3 · p. 45

Persona

  • The Baseline response scores PREFALIGN 1.85 despite a correct answer and is described as barely aligned with the user’s preferences.In this example, the model receives only the question, without additional user context.Source: Persona · p. 46

Assistant: claude-opus-4

  • Discovery scores PREFALIGN 1.92 versus Baseline’s 1.85, with both answers correct; its single question clarifies factual context but misses affective and metacognitive needs.The model asks what kind of help Sasha gave Jordan, then proceeds to answer without asking about preferred explanation style or a reasoning walkthrough.Source: Persona · p. 46, Assistant: claude-opus-4 · p. 47
  • Oracle mode raises PREFALIGN to 4.26 while keeping the answer correct, as the model analyzes each option against the user’s preferences.The response’s reasoning changes substantially when all preference attributes and values are revealed.Source: Assistant: claude-opus-4 · p. 49

User

  • The response to Lila uses volunteer-work and animal-behavior analogies, visual mappings, and empathetic, step-by-step reasoning to support answer C.Its evaluation rates empathy-based reasoning, animal-behavior analogies, and visual problem mapping at 5, while preferred format scores 3.Source: User · pp. 49–51
  • In SocialIQA Example 2, the baseline is incorrect, while discovery and oracle personalization both produce the correct answer.The example identifies this as a personalization success after the baseline abstains from choosing an option.Source: User · p. 52

Persona

  • For Remy’s test question, the baseline hedges instead of choosing an option, scoring 3.71 on PREFALIGN and answering incorrectly.It explains the options rather than committing to A, B, or C, despite the supplied answer being B, “like a winner.”Source: Persona · pp. 52–53

Assistant: gemini-1.5-flash

This section documents the prompts and evaluation procedures used for personalized reasoning, covering baseline, discovery-, and oracle-mode responses as well as persona- and problem-sensitive preference dimensions and rubrics. It describes how clarification and persona-primed reasoning shape answers and how responses are scored against user-specific criteria.

Assistant: gemini-1.5-flash

  • The baseline concludes that none of the options perfectly describes Remy: the prompt says Remy tried to do better but does not say whether they succeeded, suggesting determined, persistent, or ambitious instead.The evaluation summary scores the baseline 5 for Analytical Rigor and Individual Attention, and 1 for Authority Source Citations and Practical Observation Exercises.Source: Assistant: gemini-1.5-flash · pp. 53–54
  • Discovery asks about the test context and Remy’s goal, then selects C, “do well,” as the neutral descriptor for aiming at a higher math-test score.The final response avoids assuming Remy succeeded or felt relief, or that they achieved a significant victory.Source: Assistant: gemini-1.5-flash · p. 54, User · p. 54

User

  • Oracle Mode PREFALIGN Score: 3.96 and Discovery Mode PREFALIGN Score: 3.24; both responses were marked correct.The oracle response used persona-primed step-by-step elimination, while discovery used quick clarifiers to elicit context and commit to an answer.Source: Assistant: gemini-1.5-flash · p. 54, User · p. 56

D USER SIMULATOR PROMPTS

  • The user simulator prompts contrast passive users, who give minimal factual answers and reveal preferences only when directly asked, with collaborative users, who volunteer context and ask follow-ups.Both prompt formats request JSON with thought and response fields; the collaborative prompt also specifies a termination signal when the user is satisfied.Source: PASSIVE USER · pp. 57–58

E ASSISTANT PROMPTS FOR DIFFERENT MODES

F BENCHMARK GENERATION PROMPTS

F.1 LLM JUDGE SCORING PROMPT

  • The judge evaluates each criterion independently against five performance levels, cites response evidence, assigns an exact 1–5 score, and justifies it with the user’s context in mind.The prompt calls for objective, consistent scoring and precise reasoning another evaluator could follow.Source: §F.1 · p. 64, User Prompt · p. 65

User Prompt

  • Rubric performance levels are personalized at construction time to the user’s preferred value, so the judge scores how well a response matches that stated preference.For example, when terminology complexity is 2, the rubric contrasts inaccessible jargon, inconsistent simplicity, and consistently plain language.Source: User Prompt · p. 65

Related papers

Loading 2510.00177v2…