UltraFeedback: Boosting Language Models with Scaled AI Feedback

Ganqu Cui; Lifan Yuan; Ning Ding; Guanming Yao; Bingxiang He; Wei Zhu; Yuan Ni; Guotong Xie; Ruobing Xie; Yankai Lin; Zhiyuan Liu; Maosong Sun

Source-linked AI summary

UltraFeedback: Boosting Language Models with Scaled AI Feedback

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, Maosong Sun

arXiv:2310.01377v2Oct 2, 2023cs.CLcs.AIcs.LG

TL;DR

Human feedback is valuable for aligning LLMs but is limited by the cost, time, and capability required to collect it, especially for open-source research. The paper builds UltraFeedback by scaling and diversifying instructions, responses, and GPT-4 annotations, then uses it to develop models and improve open-source chat systems. It concludes that scaled AI feedback can support open-source alignment, while noting unresolved concerns about evaluator bias, precise preference modeling, and single-turn coverage.

Problem
Human preference data is costly and time-consuming to collect, limiting feedback learning for open-source LLMs.
Method
The paper constructs UltraFeedback by diversifying instructions and responses, applying bias-mitigation techniques, and collecting scalar scores and textual critiques from GPT-4.
Results
UltraFeedback enables reward and critique model development, while best-of-n sampling and PPO dramatically improve an open-source chat model.
Takeaways & Limitations
Scaled AI feedback provides a high-efficiency, low-cost source for developing and studying alignment methods for open-source LLMs.
Takeaways & Limitations
AI feedback may not precisely model human preferences in all situations, and UltraFeedback currently provides only single-turn dialogues because of time and budget restrictions.

Abstract

from arXiv · show

Learning from human feedback has become a pivot technique in aligning large language models (LLMs) with human preferences. However, acquiring vast and premium human feedback is bottlenecked by time, labor, and human capability, resulting in small sizes or limited topics of current datasets. This further hinders feedback learning as well as alignment research within the open-source community. To address this issue, we explore how to go beyond human feedback and collect high-quality \textit{AI feedback} automatically for a scalable alternative. Specifically, we identify \textbf{scale and diversity} as the key factors for feedback data to take effect. Accordingly, we first broaden instructions and responses in both amount and breadth to encompass a wider range of user-assistant interactions. Then, we meticulously apply a series of techniques to mitigate annotation biases for more reliable AI feedback. We finally present \textsc{UltraFeedback}, a large-scale, high-quality, and diversified AI feedback dataset, which contains over 1 million GPT-4 feedback for 250k user-assistant conversations from various aspects. Built upon \textsc{UltraFeedback}, we align a LLaMA-based model by best-of-$n$ sampling and reinforcement learning, demonstrating its exceptional performance on chat benchmarks. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models, serving as a solid foundation for future feedback learning research. Our data and models are available at https://github.com/thunlp/UltraFeedback.

1. Introduction

Feedback learning can align language models with human preferences, but open-source research is constrained by the cost, time, and limited scalability of human preference data. This paper addresses that gap with diversified AI feedback and demonstrates benefits for open-source models.

Human feedback data is central to feedback learning but is constrained by annotators’ financial resources, time, and knowledge.
AI feedback offers a more scalable alternative because it is easier to collect and expand at lower cost, while quality can improve with stronger LLM annotators.
UltraFeedback broadens feedback coverage using over 60,000 instructions and 17 models across diverse user-assistant interactions.
The annotation pipeline reduces ambiguity and bias through four assessment aspects, objective grading criteria, reference responses, and detailed GPT-4 critiques before scoring.
Using UltraFeedback, the authors train UltraRM and enhance UltraLM with best-of-n sampling and PPO, with both strategies improving the model dramatically.
The paper contributes a dataset, reward model, and critique model while demonstrating beneficial effects of scaled AI feedback on open-source chat LLMs.

2. ULTRAFEEDBACK

UltraFeedback is constructed by scaling and diversifying instructions, model completions, and GPT-4 annotations while reducing annotation ambiguity and bias. The resulting dataset combines scalar preferences and textual critiques and supports reward and critique model development.

2.1. Overview: Existing human feedback is difficult to scale, while prior AI feedback approaches are narrower in domain or form and lack broader-context diversity.
2.1. Overview: The pipeline prioritizes scalability and diversity across instruction collection, completion sampling, and comparison-pair annotation.
2.2. Instruction Collection: The instruction set targets instruction-following, truthfulness, honesty, and helpfulness using six publicly available datasets.
2.3. Response Collection: Different model series, sizes, architectures, and training data are sampled to produce dissimilar, well-distributed completions and reduce style-quality confounding.
2.3. Response Collection: Different principle prompts are added to system prompts to elicit more diverse model behaviors across data sources.
2.4. AI Feedback Annotation: 255,864 completions receive GPT-4 scalar scores and textual critiques, producing over 1 million feedback data in total.
2.4. AI Feedback Annotation: Annotation quality is improved by decomposing assessments into four aspects and combining standards, calibration, and rationales to yield four scalar scores and rationales per response.
2.4. AI Feedback Annotation: GPT-4 critiques are generated from an overall perspective to provide improvement suggestions rather than directly proposing answers.

3. Experiments

The experiments evaluate UltraRM against human preference benchmarks and test its use for best-of-n sampling and PPO-based alignment. UltraRM generalizes strongly, while both sampling and PPO substantially improve UltraLM performance.

3.1. Reward Modeling: UltraRM outperforms open-source reward-model baselines across four human preference datasets, except the much larger closed LLaMA2 model.Using only ULTRAFEEDBACK improves accuracy by over 6.3 percent on average over open-source baselines.
3.1. Reward Modeling: UltraRM reaches a 2.6% absolute-point improvement over baselines on the held-out WebGPT benchmark.WebGPT has no training/test split, and most compared models were not trained on it, making it a generalization test.
3.2. Best-of-n Experiments: Best-of-n sampling raises UltraLM-13B’s AlpacaEval win rate from 76.53% with one sample to 91.54% with 16 samples.The method samples multiple completions, scores them with UltraRM, and selects the highest-reward response.
3.3. PPO Experiments: UltraLM-13B-PPO achieves the highest average win rate across three benchmarks and outperforms the original UltraLM-13B by 16.8 percent.It also beats the larger LLaMA2-70B-Chat model and reaches the highest win rate on Evol-Instruct and UltraChat against gpt-3.5-turbo.

4. Agreement with Human Preferences

The paper compares GPT-4 and Claude-3 evaluations with human judgments to assess whether AI feedback tracks human preferences. Agreement is generally strong, but GPT-4 diverges more on reasoning tasks and can underestimate reasoning performance.

4.1. Annotation Consistency: GPT-4 agrees with majority votes from three human annotators on 68.6% of sampled comparisons.Across 400 pairs from four datasets, individual GPT-4 agreement averages 59.7%, while majority-vote agreement is higher.
4.2. Reliability of AI Evaluation: On 266 filtered pairs, human evaluations are mostly consistent with GPT-4, with average winning rates of 64.3% and 67.3%, respectively.Claude-3 follows the same trend, while humans assign more ties than GPT-4.
4.2. Reliability of AI Evaluation: Humans and GPT-4 both prefer the authors’ models on writing and QA tasks, but diverge on ties and losses for reasoning tasks.The reasoning category includes coding, math, and logic.
4.2. Reliability of AI Evaluation: The expert-grounded reasoning comparison yields a 42.1%/26.3%/31.6% win/tie/lose rate and closely matches human evaluations.The authors report that GPT-4 potentially underestimates their model’s reasoning performance.

5. Analysis

The analysis examines subject- and task-level effects of ULTRAFEEDBACK alignment. PPO improves broad chat performance while adding only marginal gains on conventional capability benchmarks, and math and code remain weaker areas.

5.1. Subject Analysis: UltraLM-13B-PPO overtakes ChatGPT on 22/29 subjects in Evol-Instruct, including writing-related tasks and human-value categories.It scores higher on toxicity, ethics, and TruthfulQA, while remaining on par on roleplay, reasoning, and counterfactual subjects.
5.2. Task Analysis: UltraLM-13B-PPO falls behind gpt-3.5-turbo on math and code-related tasks.The authors attribute this possibility to limitations in base-model ability and insufficient relevant data in ULTRAFEEDBACK.
5.2. Task Analysis: PPO produces about 1 absolute point of improvement on nine conventional question-answering and multiple-choice capability benchmarks.The comparison evaluates UltraLM-13B before and after PPO.

6. Related Work

Feedback learning has become central to aligning LLMs, but human supervision is difficult to scale, while existing AI-feedback approaches lack broad diversity. Preference-data construction therefore remains underdeveloped for general chat-model improvement.

Feedback Learning for LLMs: Human feedback is widely used to align LLMs but remains difficult to scale because it depends on human capabilities.
Feedback Learning for LLMs: AI feedback can reduce human involvement, but prior approaches are restricted to specific domains or forms.
Data for LLM Alignment: Instruction-tuning research highlights scalability, diversity, and quality as important data-construction factors.
Data for LLM Alignment: Existing preference datasets often target narrow tasks or remain small, limiting their usefulness for general chat models.

7. Conclusion

The paper constructs ULTRAFEEDBACK as a large-scale, diverse AI-feedback dataset and explores its uses for preference modeling, chat-model improvement, and critique-model training. It presents AI feedback as a basis for enhancing open-source LLMs and supporting future research.

7. Conclusion: ULTRAFEEDBACK is a large-scale and diverse AI-feedback dataset constructed to enhance open-source LLMs.
7. Conclusion: The dataset supports modeling human preferences, improving chat language models, and training critique models.
7. Conclusion: The authors analyze human agreement and model-capability evaluations, reporting nuanced insights about AI feedback.
7. Conclusion: The paper presents AI feedback as a scalable and reliable source for future AI oversight.

Impact Statement

Scaled AI feedback is presented as a way to support the safety and trustworthiness of advanced LLM applications while reducing human labor in alignment. The authors also identify bias and missing safety-oriented data as important concerns.

Impact Statement: Scaled AI feedback can reduce human labor and support more scalable alignment of open LLMs.
Impact Statement: AI feedback may be biased toward answer positions, response lengths, and particular styles, producing inaccurate or unfair annotations and evaluations.
Impact Statement: ULTRAFEEDBACK does not intentionally include safety-oriented conversations, so adversarial prompts may still elicit toxic or unethical behavior.

A. Limitations

ULTRAFEEDBACK assumes powerful LLMs can imitate human annotators and evaluate fairly, but the authors acknowledge that human preferences remain imperfectly modeled. The dataset is also limited to single-turn dialogues because of time and budget constraints.

A. Limitations: ULTRAFEEDBACK assumes powerful LLMs such as GPT-4 can imitate human annotators and act as fair evaluators.
A. Limitations: LLMs cannot model human preferences precisely in every situation, leaving preference collection and rigorous evaluation challenging.
A. Limitations: ULTRAFEEDBACK provides only single-turn dialogues because of time and budget restrictions.

B. Data Contamination

The evaluation reports contamination filtering, order-bias mitigation, and model- and aspect-dependent feedback quality. GPT-series models rank highest across aspects, while UltraCM nearly matches GPT-3.5-turbo overall but remains weaker on some datasets.

Data Contamination: 48 contamination samples were found through 13-gram matching across AlpacaEval, Evol-Instruct, and UltraChat test sets and filtered out.The authors recommend additional decontamination against researchers’ own evaluation datasets.
Model Statistics: GPT-series models rank highest across all four aspects, and larger models generally outperform smaller models.LLaMA2-Chat models score especially highly on honesty, consistent with their RLHF alignment.
Reward Modeling: Averaging scores across swapped response orders mitigates SteamSHP’s tendency to prefer the first response.This addresses position bias in its two-response text-to-text evaluation format.
Critique Modeling: UltraCM nearly approaches gpt-3.5-turbo overall but remains deficient on AlpacaFarm and code datasets.It is comparable to gpt-3.5-turbo on commonsense and mathematics reasoning.

E.4. Capability Experiments

Capability experiments evaluate models across world knowledge, commonsense reasoning, and reading comprehension using nine datasets. UltraLM-13B-PPO achieves the highest average score, while best-of-n and PPO examples show improved response quality without a simple length shortcut.

Evaluation Setup: Nine datasets cover world knowledge, commonsense reasoning, and reading comprehension in the capability evaluation.The evaluation uses NaturalQuestions, TriviaQA, PIQA, HellaSwag, OpenBookQA, ARC, BoolQ, RACE, and MultiRC.
Capability Results: UltraLM-13B-PPO achieves the highest average score, especially on commonsense, world knowledge, and professional knowledge questions.The model does not show advantages on math and reasoning tasks.
Best-of-n Sampling: Best-of-16 produces a correct Python explanation and receives a much higher reward than incorrect best-of-1 and partially correct best-of-2 samples.The longer best-of-2 sample receiving lower quality assessment indicates UltraRM does not rely on response length alone.
PPO Case Study: UltraLM-13B-PPO correctly answers an AlpacaEval question that the original UltraLM-13B answers incorrectly.The comparison is presented as a before-and-after PPO case study.

F.4. UltraCM-13B

UltraCM-13B is evaluated as a critique model using GPT-4 ratings and task-diverse cases. Its critiques can identify flaws and suggest improvements, though suggestions may be unnecessarily long and performance remains uneven across tasks.

Case Study: UltraCM-13B provides meaningful critiques of incorrect GSM8K answers by identifying flaws and offering improvement suggestions.The case study also notes that some suggestions are unnecessarily long.