PaperlayerView exact revision on arXiv

Source-linked AI summary

Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

Zihan Chen, Lei Nico Zheng, Di Zhu

arXiv:2610.30763v1cs.HCcs.AIcs.CL

TL;DR

The paper asks how users’ stated preferences relate to the response qualities they reward, then compares preferences and ratings in PRISM. Stated priorities aligned more clearly with perceived-attribute profiles than with text proxies, while concrete requests tracked length weights.

  • Problem

    It remains unclear how people’s stated priorities relate to what they reward when rating AI responses, including whether weak correspondence reflects limited insight, measurement mismatch, or context.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes PRISM survey priorities and response ratings, comparing stated attribute importance with text proxies, perceived performance, concrete feedback, and reported choice reasons.

    Source: §3.1 · p. 4

  • Results

    Priorities aligned more clearly with aggregate perceived-attribute profiles than text proxies, while concrete length requests tracked length weights and short histories yielded small ranking gains from personal information.

    Source: §1 · p. 2, §6 · p. 16

  • Takeaways & Limitations

    Preference reports should be evaluated against their intended use, distinguishing information about perceived attributes from estimates of concrete feature weights or personalization value.

    Source: §6 · p. 16

    The observational design supports conditional associations, and post-conversation perceptions and reasons about chosen responses may reflect shared perceptions, wording, or post-choice reconstruction.

    Source: §5.4 · p. 16

Abstract

from arXiv · show

AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.

1 Introduction

The paper asks how users’ stated priorities relate to what they reward when rating AI responses, given that this correspondence may depend on how preferences are measured. In PRISM, priorities aligned more clearly with aggregate perceived-attribute profiles than with text proxies, while concrete requests tracked the weight on response length.

1 Introduction

  • The motivating question is whether stated priorities correspond to what people reward in rated responses, since weak correspondence may reflect limited insight, measurement mismatch, or context dependence.The paper focuses on whether that correspondence changes when attributes are measured as perceived qualities or as concrete response cues.Source: §1 · pp. 1–2
  • Stated priorities aligned more clearly with aggregate attribute profiles based on others’ perceptions than with designed text proxies; concrete requests tracked length weights.Stable individual differences in feature weights coexisted with small predictive gains from personal information.Source: §1 · p. 2
  • The paper distinguishes three contributions: measurement-dependent alignment, the relation between concrete requests and feature weights, and the stability, recoverability, and predictive value of individual differences.Together, these connect a psychological question about preference reports to personalization.Source: §1 · p. 2

2 Theoretical background

The theoretical background distinguishes response cues, perceived attributes, and reported preferences or reasons, motivating tests of whether alignment changes with the level at which attributes are measured.

2 Theoretical background

  • The framework links observable response cues, perceived attributes, and the preferences or reasons people report when evaluating responses.Judgment analysis, self-report research, and construal-level theory motivate examining how these elements relate.Source: §2 · p. 2

2.1 Cues, perceived attributes and judgment

  • In judgment analysis, people infer qualities from cues, so a mismatch between reported and captured weights can reflect limited insight, inconsistent judgments, or mismatched measurements.Self-insight is partial and depends on how it is measured.Source: §2.1 · pp. 2–3

2.2 What reports reveal

  • Choice explanations may rely on a priori theories, yet can accurately summarize a perceived attribute when report and behavior describe evaluation at the same level.This distinguishes reporting a perception from identifying the cues associated with it.Source: §2.2 · p. 3

2.3 Levels of description

  • Because surveys assess attributes generally while ratings assess particular responses, stated priorities may align more closely with perceived attributes than with concrete features.Concrete message properties such as readability can shape impressions independently of substance.Source: §2.3 · p. 3

2.4 Human feedback and personalization

  • Prior work describes systematic response-rating patterns, including preferences for longer or more assertive answers, while the relation between attribute priorities and people’s own ratings remains open.The paper asks whether this relation changes when the same attributes are represented through features versus perceptions.Source: §2.4 · p. 3

2.5 Competing explanations and the research questions

  • The research questions test measurement-dependent alignment, stability and recoverability of feature weights, links between concrete feedback and feature weights, and whether experience changes alignment.The planned hypotheses also examine text-proxy moderation, omitted features, choice reasons, and predictive value of other conversations versus stated priorities.Source: §2.5 · pp. 3–4

3 Method

The study uses PRISM’s repeated response ratings and participant feedback to examine how stated priorities relate to evaluations and personalization. Its methods compare multiple representations of response attributes, model within-conversation ratings and individual differences, and test whether participant information improves held-out choice prediction.

3.1 Data

  • The analysis covers 29,347 first-turn responses from 1,396 participants, who rated multiple model responses to shared prompts and later reported perceived performance and choice reasons.Participants completed importance ratings before a median of six conversations; the sample includes responses of at least three words in conversations with at least two eligible responses.Source: §3.1 · p. 4

3.2 Measures

  • Measures combine participants’ relative priorities with 12 text-derived response features: seven designed as proxies for surveyed attributes and five not elicited by the survey.Relative importance ratings are ipsatized within participant; feature values are demeaned within conversation and scaled across responses.Source: §3.2 · pp. 4–5
  • Each surveyed attribute is represented by a designed text proxy, a feature-weighted lens composite, and an aggregate profile of other participants’ perceptions of the producing model.Lens weights and aggregate profiles are estimated on one participant half and applied to the other; the rating participant does not see the profile.Source: §3.2 · p. 5
  • Open-ended conversation feedback is coded for directional requests and other concrete comments, with validation showing that the rules capture only a subset of requests.The rules distinguish requests for shorter or longer responses from praise of brevity or detail; estimated recall for detected shorter and longer requests was 0.22 and 0.18, respectively.Source: §3.2 · pp. 5–6

3.3 Estimation

  • Rating models use conversation fixed effects to estimate feature associations among responses to the same prompt, while separate regressions assess feature-to-perception links.The estimation also examines priority moderation and perceived attributes in reported reasons, and uses mixed models to estimate participant- and conversation-level feature slopes.Source: §3.3 · pp. 6–7
  • The analysis tests whether stated priorities and concrete feedback moderate feature weights, including whether length-related requests in other or earlier conversations relate to the length weight.Choice reasons are modeled against feature differences and perceived performance; the moderation analyses include comparisons across attribute representations.Source: §3.3 · p. 7
  • Held-out first-turn choices are predicted with pooled and participant-specific information, using participant folds and weights learned from other conversations.The primary metric is pairwise accuracy; personal weights are shrunk toward pooled weights, and a forward-only version uses earlier conversations only.Source: §3.3 · p. 7

3.4 Analysis plan and transparency

  • A time-stamped pre-analysis plan specified the sample, feature definitions, primary tests, and robustness checks, while several analyses were added later and designated exploratory.The plan was not publicly registered; the comparison of representations, feedback coding, individual-difference models, and full prediction protocol were among the later additions.Source: §3.4 · p. 8

4 Results

Ratings were associated with concrete response features, including features not elicited by the survey, while alignment between stated priorities and rating weights depended on measurement and was weak for designed text proxies.

4.1 Features associated with ratings

  • A one-standard-deviation increase in response length was associated with 5.56 more rating points (SE 0.31), the strongest feature association.Non-elicited features were jointly significant and added at least as much explained within-conversation variance as the seven text proxies.Source: §4.1 · p. 8
  • First-turn feature weights predicted preferences between same-model later-turn samples in 54.8% of 17,908 pairs, against 50% chance.Feature weights also varied with prompt context, including prompt length and controversial topics.Source: §4.1 · p. 8

4.2 Concrete features are associated with perceived attributes

  • Non-elicited features were often among the strongest correlates of perceived attributes, while most designed text proxies showed little partial association.Length correlated with perceived factuality and viewpoint coverage; lists and headings with helpfulness and creativity; refusal language with lower helpfulness and creativity.Source: §4.2 · p. 9, Figure 2 · p. 9

4.3 Stated priorities and the weights on text proxies

  • Mean moderation across text proxies was 0.09 (SE 0.08, p = 0.30), and their joint test was not significant (χ2(7) = 10.72, p = 0.15).The planned test of alignment between stated importance and designed proxies was not supported for the mean.Source: §4.3 · pp. 9–10

4.4 Alignment depends on how attributes are operationalized

Alignment between stated priorities and rating weights depended on how the attributes were represented: it was strongest for aggregate attribute profiles, intermediate for lens composites, and absent on average for text proxies.

4.4 Alignment depends on how attributes are operationalized

  • In the joint model, mean moderation was 0.00 for text proxies (SE 0.09), 0.33 for lens composites (SE 0.14), and 0.85 for aggregate attribute profiles (SE 0.23).Profiles exceeded proxies by 0.85 (SE 0.25, pHolm = 0.002); the corrected contrasts involving lens composites were not distinguishable from zero.Source: §4.4 · p. 10
  • Profiles exceeded proxies in all 50 random half-splits, and the participant-bootstrap 95% interval for their contrast was 0.16 to 1.12.The bootstrap interval was 0.15 to 1.13 for profiles and −0.10 to 0.56 for lens composites.Source: §4.4 · p. 10
  • After removing common perceived quality, attribute-specific deviations still moderated weights (mean 0.29, SE 0.08, p < 0.001); mismatched profiles averaged −0.06 versus 0.93 for correct pairings.Profiles were reproducible across rater halves, while lens composites predicted held-out perceptions weakly.Source: §4.4 · p. 10

4.5 Concrete feedback tracks the length weight

Written requests about response length tracked the weight attached to length, while choice reasons reflected perceived attributes more than measured features. Stable differences in feature weights existed, but individual estimates were poorly reproducible.

4.5 Concrete feedback tracks the length weight

  • A one-standard-deviation increase in the net share of conversations requesting shorter rather than longer answers was associated with a −0.78-point change in length weight per length standard deviation.The association held prospectively (−3.50, SE 1.05, pHolm = 0.002); relative stated priorities added no detectable information about length weight (p = 0.171).Source: §4.5 · pp. 10–11

4.6 Reasons track perceived attributes

  • Reasons were more strongly associated with perceived performance on the named attribute (0.20 to 0.58) than with measured feature differences.With perceived performance included, standardized associations were 0.013 for the matched feature and 0.026 for length.Source: §4.6 · p. 11, Table 5 · p. 12

4.7 Stable individual differences and their recoverability

  • Participant-level variation remained significant after Holm correction for 5 of 12 features, but individual slopes correlated only 0.021 on average across conversation halves.Simulations reproduced this low reliability at fitted heterogeneity levels, even when stable differences existed.Source: §4.7 · p. 12
  • The less assumption-dependent moment estimator was imprecise: its 95% interval for stable length-weight variance ran from −10.2 to 19.9 squared rating points.The authors state that evidence for stable differences therefore rests on the calibrated likelihood analysis.Source: §4.7 · p. 12

4.8 Predicting held-out rankings

Personal information yielded little improvement in held-out rankings, while strengthening the pooled prediction model produced a larger gain. Concrete feedback was the strongest personal-information addition, but improvements varied by prediction target and evaluation setup.

4.8 Predicting held-out rankings

  • 0.13 percentage points (0.02 to 0.24) was the gain for concrete feedback from other conversations, the only personal-information addition whose interval lay above zero.Intervals were not corrected for 12 comparisons, and paired comparisons did not distinguish feedback from wishes or model taste.Source: §4.8 · p. 13
  • Concrete feedback outperformed feature-interacting sliders by 0.36 (0.12 to 0.60), while adding those sliders to feedback lowered accuracy by 0.33 (−0.57 to −0.11).The direct comparison of watching with asking had limited support: only one of six comparisons excluded zero.Source: §4.8 · p. 13
  • Personalized feature weights and profile-interacting sliders reduced squared rating error by 2.1 and 1.3 squared rating points, respectively, without detectably changing response rankings.Individual information improved prediction of rating levels more than rankings.Source: §4.8 · p. 13
  • 1.11 points (0.73 to 1.52) was the pairwise-accuracy gain from replacing the pooled linear model with pooled gradient boosting, exceeding the largest personal-information estimate of 0.13 percentage points.A linear pairwise logistic model with the same inputs gained 0.35 (0.12 to 0.62), and boosting exceeded it by 0.76 (0.43 to 1.10).Source: §4.8 · pp. 12–13
  • Using only earlier conversations produced small accuracy changes whose intervals mostly included zero, with no clear increase as the number of earlier conversations grew.For model taste, the estimated trend was 0.04 points per earlier conversation (−0.09 to 0.16); with five or more conversations, the change was 0.32 (−0.17 to 0.79).Source: §4.8 · p. 14

4.9 Experience and item order

The analysis found no detectable alignment differences by language-model experience and no clear evidence that randomized item position affected stated importance.

4.9 Experience and item order

  • Experience-group contrasts were not significant (smallest p = 0.25), and randomized item positions shifted stated importance by at most 1.4 points without a significant joint effect.These checks provided no clear evidence of an experience-related alignment difference or item-order effect.Source: §4.9 · p. 14

5 Discussion

The discussion argues that stated priorities, concrete requests, and choice-specific reasons reveal different aspects of response evaluation. Their usefulness depends on how attributes are measured, and short histories limit the precision and predictive value of personal profiles.

5.1 The explanations revisited

  • Stated-priority alignment was near zero for designed text proxies but positive for aggregate attribute profiles, making the apparent preference gap dependent on measurement.Lens composites fell between these results, with neither pairwise contrast statistically resolved.Source: §5.1 · p. 14
  • Requests for shorter or longer answers tracked the weight participants gave length, even for earlier requests, while choice reasons tracked perceived attributes more than measured feature differences.The request association remained after accounting for relative priorities across the seven abstract attributes.Source: §5.1 · p. 15

5.2 Theoretical implications

  • Observable features, perceived attributes, and stated priorities are related but distinct, so a weak link between priorities and text features can coexist with stronger alignment to perception-based measures.The discussion applies the lens model of judgment to interpreting preference reports in human–AI evaluation.Source: §5.2 · p. 15
  • Stable population-level differences in evaluation did not make individual feature weights reliably recoverable from short histories.The discussion treats establishing heterogeneity and estimating a useful personal profile as distinct empirical tasks.Source: §5.2 · p. 15

5.3 Practical implications

  • Elicitation should match its intended use: abstract priorities aligned with aggregate perceived attributes, whereas concrete requests supplied information about length weighting.The authors identify direct questions about observable properties as candidates for evaluation against held-out choices.Source: §5.3 · p. 15
  • Because concrete features can covary with perceived attributes, ratings under attribute labels may reward properties the labels do not mention.The authors recommend analyzing concrete properties alongside attribute-level reasons when auditing what ratings reward.Source: §5.3 · p. 15
  • With a median of 6 conversations per participant, the largest personal-information gain in held-out pairwise accuracy was 0.13 percentage points, versus 1.11 points from a stronger pooled pipeline.The discussion recommends evaluating personalization against a strong pooled baseline, on held-out choices from the same person, using metrics suited to the intended purpose.Source: §5.3 · p. 15

5.4 Limitations and future research

  • The evidence is constrained by unvalidated text proxies, observational and post-choice measures, one late-2023 dataset of capped responses, and short histories; the authors propose independent attribute assessment and more judgments per participant.Priorities were measured once, and evidence for stable weights rests primarily on REML models; longer interactions, repeated elicitation, and newer models would test how patterns change.Source: §5.4 · pp. 15–16

6 Conclusion

The conclusion is that preference reports convey different kinds of information depending on what is measured and what they are intended to predict.

6 Conclusion

  • In PRISM, priorities aligned with aggregate attribute profiles but not designed text proxies; length requests tracked length weighting, while short histories yielded poorly reproducible personal estimates and small ranking gains.The conclusion recommends evaluating preference reports against their intended use: explaining perceived attributes, estimating concrete feature weights, or predicting an individual’s next choice.Source: §6 · p. 16

A Lexicons and feature details

The appendix specifies lexicons for concrete response features and rules for coding participants’ written feedback. It also describes how text measures and codes were processed and validated.

A Lexicons and feature details

  • The feature lexicons cover viewpoint, caution, refusal or disclaimer, validation, and coherence-defect markers.Informational density, lexical diversity, and readability were winsorized; lexical diversity below ten tokens was imputed with the conversation mean and a missing-value indicator.Source: A Lexicons and feature details · p. 19
  • Feedback codes distinguish requests for shorter or longer responses, praise for brevity or detail, formatting mentions, and refusal complaints.Negated matches and specified references to prompts, truncation, latency, links, or sources were excluded; requests and praise were not counted in the same category.Source: A Lexicons and feature details · p. 19
  • An independent coder checked six yes/no feedback questions, each with definitions and examples, using texts presented in random order.The coding rules also specify how feedback was lowercased, split into clauses, and matched against phrase lists.Source: A Lexicons and feature details · p. 19

B Analysis plan: deviations and exploratory additions

The appendix records changes from the analysis plan and identifies analyses added after it. These include changes to feature selection, variation thresholds, and the prediction baseline.

B Analysis plan: deviations and exploratory additions

  • Inspection before ratings were examined found frequent coherence failures, so coherence replaced fluency as the primary feature and readability became the alternative.The reported failures included 12.9% of analyzed responses ending mid-sentence, 1.3% fabricating a dialogue continuation, and 1.8% repeating themselves.Source: B Analysis plan: deviations and… · pp. 19–20
  • The analysis required feature variation in at least 1,000 conversations rather than the plan’s 80%-of-conversations threshold, which was mis-specified for sparse markers.Heavy-tailed features were winsorized and undefined lexical diversity was imputed.Source: B Analysis plan: deviations and… · p. 20
  • The main prediction analysis used a non-personal baseline; the planned baseline with self-description similarity was reported separately in Table 28.The appendix also lists exploratory analyses spanning perceptual correspondence, feedback coding, feature weights, random slopes, and prediction protocols.Source: B Analysis plan: deviations and… · p. 20

C Additional results

The additional-results tables document analyses of sample flow, attribute perception and moderation, feedback, individual feature weights, and prediction. The supplied table descriptions specify their scopes but do not report outcome values.

C Additional results

  • The sample-flow table documents inclusion and exclusion counts, while a separate table reports N/A-reason shares by attribute and conversation type.The table descriptions do not provide the underlying counts or shares.Source: Table 6 · p. 20, Table 7 · p. 20
  • Perceptual correspondence is assessed with within-participant contrasts, response feature levels, and the feature most associated with each perceived attribute.The related moderation table reports stated-importance coefficients for text proxies under three representations of importance.Source: Table 8 · p. 21, Table 9 · p. 21
  • Additional tables compare moderation across representations, their pairwise contrasts, attribute specificity, reliability, and aggregate-profile construction checks.The descriptions identify these analyses but do not state their results.Source: Table 10 · p. 21, Table 11 · p. 22, Table 12 · p. 22
  • Other tables examine whether stated priorities moderate weights on non-elicited features and whether reasons for choices correspond to perceived attributes.The table descriptions also cover context dependence of pooled feature weights and exploratory priority-feature pairings.Source: Table 15 · p. 22, Table 19 · p. 23, Table 20 · p. 24
  • The feedback analyses include independent validation of free-text codes and exploratory tests linking concrete feedback to matching feature weights.A common length-weight model enters feedback from other conversations alongside the seven stated priorities.Source: Table 16 · p. 23, Table 17 · p. 23, Table 18 · p. 23
  • The individual-differences and prediction tables cover random-slope estimates and calibration, held-out rankings, information-source comparisons, forward-only prediction, and planned prediction results.Additional tables describe experience contrasts and stated importance by randomized display position; the captions provide scopes, not findings.Source: Table 22 · p. 24, Table 23 · p. 25, Table 24 · p. 25

Related papers

Loading 2610.30763v1…