PaperlayerView exact revision on arXiv

Source-linked AI summary

Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models

Hritik Bansal, John Dang, Aditya Grover

arXiv:2308.15812v3cs.LGcs.AIcs.CL

TL;DR

Sparse feedback offers a choice between absolute ratings and relative rankings, but their implications for alignment and evaluation are underexplored. The paper compares feedback collected through both protocols and reports disagreement between them, alongside evaluation outcomes that vary with protocol choice.

  • Problem

    Sparse feedback can be collected as absolute ratings or relative rankings, but the impact of this design choice on LLM alignment and evaluation remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    The authors collect ratings and independent pairwise rankings for the same responses, then compare the preferences implied by ratings with direct rankings.

    Source: §1 · p. 2

  • Results

    60% of preference comparisons disagree for both humans and AI; under human rankings evaluation, rankings-trained Best-of-n has an 11.2% win-rate gap over SFT versus 5.3% for ratings-trained, while ratings evaluation gives 5% versus 4.3%.

    Source: §1 · p. 3, §5.2 · p. 9

  • Takeaways & Limitations

    Evaluation trends are significantly influenced by the choice of feedback protocol.

    Source: §7 · p. 9

    Human feedback came from Amazon Mechanical Turkers and may not represent all people, and findings from open-ended responses may differ in domains such as mathematics or coding.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Aligning large language models (LLMs) with human values and intents critically involves the use of human or AI feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural design choice between ratings (e.g., score Response A on a scale of 1-7) and rankings (e.g., is Response A better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMs. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena, such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments. To our surprise, we also observe that the choice of feedback protocol also has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?) but not with a rating-based evaluation protocol (score Rank X/Y's response on a scale of 1-7). Our findings thus shed light on critical gaps in methods for evaluating the real-world utility of language models and their strong dependence on the feedback protocol used for alignment. Our code and data are available at https://github.com/Hritikbansal/sparse_feedback.

1 INTRODUCTION

The introduction frames ratings versus rankings as an underexplored choice in sparse feedback for LLM alignment and evaluation. It reports substantial disagreement between protocols and shows that evaluation outcomes also depend on the protocol used.

1 INTRODUCTION

  • Sparse ratings and rankings are easier to acquire than dense feedback, but their effect on the alignment pipeline has been underexplored relative to alignment algorithms and dense feedback protocols.The alignment pipeline includes feedback acquisition, alignment algorithms, and model evaluation.Source: §1 · p. 1
  • The study analyzes how ratings and rankings affect LLM alignment and subsequent evaluation, including annotators’ differing assessments of response quality.The authors identify facets of perceived response quality as a focus of their analysis.Source: §1 · pp. 1, 3
  • Ratings give an absolute score for an individual response, while rankings express a relative preference between two candidate responses.Ratings can help gauge strengths and weaknesses but are hard to determine for complex instructions; rankings are easier to acquire for such instructions but do not quantify the preference gap.Source: §1 · pp. 1–2
  • Under human rankings evaluation, the rankings-trained Best-of-n policy has an 11.2% win-rate gap over SFT versus 5.3% for ratings-trained; under ratings evaluation, the gaps are 5% and 4%, respectively.The evaluation protocol favors the Best-of-n policy trained with the same feedback protocol.Source: §1 · pp. 2–3
  • 60% of preference comparisons disagree between ratings and rankings for both human and AI annotators.The detailed analysis reports 58% for humans and 59.4% for GPT-3.5-Turbo; inconsistent responses are perceived as closer to one another than consistent responses.Source: §1 · pp. 2–3

2 BACKGROUND

The background describes an alignment pipeline that gathers candidate responses and feedback, then uses reward models with Best-of-n selection to choose outputs.

2 BACKGROUND

  • Alignment proceeds from supervised fine-tuning to response generation, feedback acquisition, and a Best-of-n algorithm that trains reward models on ratings or rankings feedback.The feedback is collected from human or machine annotators for new instructions.Source: §2 · p. 3

2.1 INSTRUCTION-RESPONSE DATA COLLECTION

  • For each instruction, the method samples multiple candidate responses from the language model to form instruction-response data for feedback collection.The candidates are sampled from the model’s distribution for each instruction.Source: §2.1 · p. 3

2.2 FEEDBACK DATA

  • In ratings feedback, annotators independently assign each response an ordinal Likert value (1–7); in rankings, they choose a preferred response or “equal” for each candidate-response pair to an instruction.Both protocols instruct annotators to consider accuracy, coherence, harmlessness, and subjective judgment.Source: §2.2 · p. 3

2.3 REWARD MODELING

  • The ratings reward model regresses normalized scores, while the rankings model uses a negative log-likelihood objective after equal-labeled pairs are filtered out.Best-of-n samples n responses from the SFT model, scores them with the trained reward model, and returns the highest-scoring response.Source: §2.3 · p. 4

3 FEEDBACK DATA ACQUISTION

The section gathers ratings and rankings feedback from GPT-3.5-Turbo and humans, then examines score distributions, feedback patterns, and annotator agreement.

3 FEEDBACK DATA ACQUISTION

  • GPT-3.5-Turbo supplied 71K feedback instances, including 24.6K ratings and 46.5K unique rankings; human feedback comprised 6K annotation instances.The human data was used for analysis, not reward-model training, because it was small and costly to expand.Source: §3 · p. 5

3.1 DATA ANALYSIS

  • About 80% of responses received ratings above 4 from both humans and AI, while humans gave many more perfect 7s than GPT-3.5-Turbo, which assigned 7 to fewer than 5%.GPT-3.5-Turbo most often assigned scores of 5 or 6.Source: §3.1 · p. 5, Figure 2 · p. 5
  • Rankings feedback showed no discernible difference in average response length or unique-token count between preferred and unpreferred responses for humans or AI.The analysis tested whether length or number of unique words biased feedback judgments.Source: §3.1 · p. 5
  • Human-human and human–GPT-3.5-Turbo ratings differed by 1.08 and 0.9 on average, while rankings agreement was 62.7% and 60.5%, respectively.The analysis used the first three human annotations as gold labels and the fourth annotation or GPT-3.5-Turbo as the prediction.Source: §3.1 · p. 6, Table 1 · p. 6

4 FEEDBACK INCONSISTENCY PROBLEM

The section defines feedback consistency by comparing rankings from ratings with directly collected pairwise rankings. It finds substantial disagreement across both human and AI feedback and examines when and why protocols diverge.

4.1 CALCULATING CONSISTENCY

  • Feedback consistency is agreement between rankings converted from ratings and rankings directly collected for the same response pair.The comparison is made for a pair of responses to a given instruction.Source: §4.1 · p. 6

4.2 RESULTS

  • 60% of feedback comparisons yielded contradictory preferences across protocols, with consistency scores in a similar 40%–42% range for humans and AI.Table 2 reports inconsistency results for AI and human feedback.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • 47.1% of GPT-3.5-Turbo rankings and 57.4% of its ratings were equal; the corresponding human rates were 30.9% and 40.8%.Equal responses were more frequent under ratings than rankings for both annotator groups; the authors attribute GPT-3.5-Turbo’s rates partly to many individual scores of 5 or 6.Source: §4.2 · p. 7
  • When pairwise rankings were decisive, consistent assignments exceeded inconsistent ones for both GPT-3.5-Turbo and humans.The same pattern held when ratings converted to rankings were decisive.Source: §4.2 · p. 7
  • Pairs in inconsistent cases had smaller perceived quality gaps, and the authors suggest humans may prioritize factors such as accuracy over coherence when choosing decisively.Human rankings also varied more across annotators for inconsistent than consistent instances.Source: §4.2 · p. 7
  • Human annotators sometimes favored a denser response in rankings while penalizing its density in ratings and judging it less compliant with instructions.In the example, the other response received an average rating for only partial guideline adherence.Source: §4.2 · pp. 7–8

5 ALIGNMENT AND EVALUATION

The section evaluates Best-of-64 policies trained with ratings or rankings feedback against a reference model. Its results show that evaluation preferences depend on whether the evaluation protocol matches the alignment feedback protocol.

5.1 SETUP

  • The evaluation used Alpaca-7B reward models, 553 unseen instructions, and Best-of-64 selection from 64 generated candidate responses.Win-rates were measured against DaVinci-003; evaluators and evaluation protocol varied.Source: §5.1 · p. 8

5.2 RESULTS

  • Best-of-64 achieved win-rates of 39.2% with ratings-trained reward models and 45.1% with rankings-trained reward models, versus 33.9% for SFT, against DaVinci-003 under human pairwise judgments.The authors report a similar trend for ratings-trained reward models when humans and AI served as evaluators.Source: §5.2 · p. 9
  • 11.2% vs 5.3% was the gap over SFT for rankings- versus ratings-trained policies under human rankings evaluation; under ratings evaluation, the gaps were 5% vs 4.3% in the reverse order.GPT-3.5-Turbo evaluation showed a similar protocol-dependent pattern.Source: §5.2 · p. 9
  • Evaluation inconsistency also appeared with rejection-sampling finetuning, alternative reward models, and ratings data converted into ranking format.These checks are reported in appendices M, N, and O, respectively.Source: §5.2 · p. 9

6 RELATED WORK

Prior work established RLHF and RLAIF frameworks and used both ratings and rankings feedback. The section identifies a gap: prior work had not systematically compared the effects of collecting ratings versus rankings.

6 RELATED WORK

  • RLHF work learned reward models from human preferences and used them in optimization; later frameworks aligned language models with human preferences.Source: §6 · p. 9
  • Prior studies explored language models as feedback providers [16] [49] [12] and introduced RLAIF as an automatic alternative to human feedback [4].This paper compares human and AI feedback in both ratings and rankings settings.Source: §6 · p. 9
  • Earlier studies used ratings [19] [17] and rankings [37] [12], but had not systematically compared their effects.The prior work covered machine-written and human-written responses [3].Source: §6 · p. 9

7 CONCLUSION

The study finds inconsistency between ratings and rankings from both human and AI feedback providers, and shows that evaluation trends depend on the feedback protocol. It identifies richer feedback, cognitive underpinnings, and downstream alignment and evaluation as future research directions.

7 CONCLUSION

  • Ratings and rankings are inconsistent for both human and AI feedback providers.The conclusion identifies feedback inconsistency as a challenge in sparse protocols.Source: §7 · p. 9
  • Evaluation trends are significantly influenced by the feedback protocol used.The conclusion identifies this as a practical consequence for model evaluation.Source: §7 · p. 9
  • Future work could examine inconsistency's cognitive underpinnings, richer feedback, and effects on alignment algorithms and evaluation procedures.Source: §7 · p. 9

A LIMITATIONS

This material bounds the study to relative and absolute feedback, win-rate evaluation against DaVinci-003, and human annotators and open-ended response tasks. It also describes related work, data acquisition, robustness checks, and the alignment and evaluation setup.

A LIMITATIONS

  • The study focuses on ratings and rankings; it evaluates downstream performance by win-rate against DaVinci-003 and leaves other evaluation criteria for future work.The authors identify helpfulness, harmfulness, and hallucinations as outcomes for future investigation.Source: A LIMITATIONS · p. 14

B DETAILED RELATED WORK

  • Prior work covers RLHF and RLAIF, ratings and rankings, and richer feedback, but had not systematically compared ratings with rankings.This work focuses on disagreement between feedback protocols rather than annotator disagreement on a single entity.Source: B DETAILED RELATED WORK · p. 14

C COMPOSITION OF THE INSTRUCTIONS DATA

D FEEDBACK ACQUISITION FROM HUMANS

  • Human feedback came from 26 Mechanical Turk annotators: 13 each for ratings and rankings, with four workers assigned to each instance.The collection yielded 2,000 rankings and 4,000 ratings; the authors note these workers may not represent all people.Source: A LIMITATIONS · p. 14, D FEEDBACK ACQUISITION FROM HUMANS · p. 15

E.1 HUMAN ANNOTATORS

  • Human inconsistency estimates varied by 1.6% standard deviation across four splits of randomly selected annotator groups.Each split aggregated preferences from three of four annotators.Source: §E.1 · p. 15

E.2 AI ANNOTATOR

  • AI inconsistency varied by 2.7% standard deviation across GPT-3.5-Turbo temperatures, and other model choices also showed inconsistency.The reported temperature settings were 0.0, 0.2, and 0.5; the authors state results were not restricted to one AI annotator choice.Source: §E.2 · pp. 15–16

F PAIRWISE RATING

  • Pairwise ratings converted to rankings had 58% inconsistency across 500 comparisons and were not calibrated with individually collected ratings.The authors attribute this to annotators focusing on different response-quality attributes across the two feedback settings.Source: F PAIRWISE RATING · p. 16

G SETUP

  • Reward models trained on ratings or rankings select among 64 Alpaca-7B candidates, and their outputs are evaluated against DaVinci-003 on 553 unseen instructions.Evaluation collects both ratings and rankings from human and AI annotators; ties receive half a point in the win-rate calculation.Source: G SETUP · pp. 16–17

H ANALYSIS OF THE RESPONSE LENGTH AND NUMBER OF UNIQUE WORDS

The analysis finds no discernible length or unique-word bias in the collected feedback, while clarifying that this does not rule out a general preference for longer responses. The appendix also presents feedback prompts and human-study interface screenshots.

H ANALYSIS OF THE RESPONSE LENGTH AND NUMBER OF UNIQUE WORDS

  • Ratings did not increase with response length or unique-word count, and rankings showed no discernible average length or unique-token difference between preferred and unpreferred responses.The authors caution that this setup-specific result does not imply humans generally lack a preference for longer responses.Source: H ANALYSIS OF THE RESPONSE LENGTH AND… · p. 17

I LLM PROMPT FOR FEEDBACK ACQUISITION

J HUMAN EXPERIMENTS WITH UI SCREENSHOTS

J.1 FINE-GRAINED ANALYSIS OF CONSISTENT AND INCONSISTENT DATA

Fine-grained analyses find that inconsistent feedback pairs tend to be closer in perceived quality, while consistent pairs show larger rating gaps and greater human ranking variation. The section also reports reward-model training settings.

J.1 FINE-GRAINED ANALYSIS OF CONSISTENT AND INCONSISTENT DATA

  • Consistent instances had the largest average rating-score differences for both human annotators and GPT-3.5-Turbo; inconsistent differences were similar across decisive and indecisive rankings.The authors interpret the latter pattern as suggesting that inconsistent pairs can be qualitatively close despite decisive ratings.Source: §J.1 · pp. 18, 21
  • 0.5 vs. 0.36: consistent human-feedback instances had higher ranking variation scores than inconsistent instances, supporting qualitative closeness in inconsistent pairs.The variation score sums annotator labels per example and averages the result across the dataset.Source: Table 7 · p. 21, §J.1 · p. 22

K TRAINING SETTINGS

  • The reward models were trained for five epochs with validation-loss early stopping; ranking models used effective batch size 16 and rating models 64.Training used a single Nvidia RTX A6000 GPU with 48GB VRAM.Source: K TRAINING SETTINGS · p. 22

L TASK-SPECIFIC RESULTS

Task-specific results show evaluation inconsistency, and the same protocol preference persists in rejection-sampling fine-tuning and with RoBERTA-Large reward models. Across these settings, evaluation favors the feedback protocol used to train the reward model.

L TASK-SPECIFIC RESULTS

  • Across the specific tasks, rating evaluation favors the rating Best-of-64 policy and vice versa, indicating that feedback acquisition protocol biases evaluation.The task-specific evaluation covers the results summarized in Table 8.Source: L TASK-SPECIFIC RESULTS · p. 23

M REJECTION SAMPLING FINETUNING (RSFT)

N USING ROBERTA-LARGE AS REWARD MODEL

O CONVERTING RATING FEEDBACK INTO RANKING FORMAT FOR REWARD TRAINING

The paper compares reward models trained on rating and ranking feedback, including rating data converted to ranking format. Evaluation inconsistency remains when reward-model training data formats are identical, implicating the nature of the feedback protocol.

O CONVERTING RATING FEEDBACK INTO RANKING FORMAT FOR REWARD TRAINING

  • Rating feedback was modeled with regression, while ranking feedback used a negative log likelihood objective because it does not provide scalar scores.Source: O CONVERTING RATING FEEDBACK INTO… · p. 24
  • 57.4% of pairwise comparisons were labeled “equal” when rating data were converted to ranking format, requiring almost 60% of rating data to be filtered for ranking-model training.The authors use regression for rating data because it can use the complete dataset.Source: O CONVERTING RATING FEEDBACK INTO… · p. 24
  • Evaluation inconsistency remains when reward models use identical training data formats, indicating the pattern is not confounded by format but relates to the nature of rating versus ranking feedback.Table 11 presents win-rates against davinci-003 for pairwise rating feedback data collection.Source: Table 11 · p. 24, O CONVERTING RATING FEEDBACK INTO… · p. 25

P QUALITATIVE EXAMPLES WITH HUMAN EXPLANATIONS

Human explanations illustrate how annotators can apply different quality criteria across feedback protocols, while also showing cases where ratings and rankings agree. The appendix also identifies the Super-Natural Instructions tasks used in the feedback data.

P QUALITATIVE EXAMPLES WITH HUMAN EXPLANATIONS

Q LIST OF TASKS FROM SUPERNI-V2

Loading 2308.15812v3…