Source-linked AI summary
HelpSteer2: Open-source dataset for training top-performing reward models
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev
TL;DR
Existing permissively licensed preference datasets have become less relevant as models improve, while proprietary-model-generated datasets can restrict commercial use. The paper releases HelpSteer2, a CC-BY-4.0 dataset with improved collection and annotation, and introduces SteerLM 2.0 to use multi-attribute rewards for alignment. HelpSteer2 contains 10k pairs, achieves 92.0% on RewardBench’s primary dataset, and supports alignment models matching or exceeding Llama 3 70B Instruct and GPT-4-0613 on major metrics.
Problem
Existing permissively licensed preference datasets have become less relevant for current well-aligned models, while proprietary-model-generated datasets often restrict commercial use.
Method
The authors release HelpSteer2 with multi-attribute ratings and introduce SteerLM 2.0, which uses multi-faceted rewards for alignment.
Results
92.0% on RewardBench’s primary dataset is reported for a reward model trained with HelpSteer2, while aligned models match or exceed Llama 3 70B Instruct and GPT-4-0613 on major alignment metrics.
Takeaways & Limitations
HelpSteer2 provides a small, permissively licensed dataset for efficiently training reward models and aligning large language models.
Takeaways & Limitations
The comparison with Llama 3 70B Instruct may involve distillation because the authors note that its training relationship to the announced Llama 3 400B+ model is unknown.
Abstract
from arXiv · showhide
High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usage imposed by model providers. To improve upon both generated responses and attribute labeling quality, we release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0). Using a powerful internal base model trained on HelpSteer2, we are able to achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming currently listed open and proprietary models, as of June 12th, 2024. Notably, HelpSteer2 consists of only ten thousand response pairs, an order of magnitude fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly efficient for training reward models. Our extensive experiments demonstrate that reward models trained with HelpSteer2 are effective in aligning LLMs. In particular, we propose SteerLM 2.0, a model alignment approach that can effectively make use of the rich multi-attribute score predicted by our reward models. HelpSteer2 is available at https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at https://github.com/NVIDIA/NeMo-Aligner
1 Introduction
Existing permissively licensed preference datasets have become less relevant for training highly aligned models, while proprietary-model-generated datasets often restrict commercial use. HelpSteer2 addresses both issues with an open dataset and alignment approach.
- Permissively licensed datasets such as Helpful-Harmless RLHF, Open Assistant, and HelpSteer facilitate academic and commercial use but have become less relevant for current well-aligned models.
- Proprietary-model-generated datasets can improve alignment but often restrict academic or commercial use and create legal risks for commercial model training.
- The authors provide detailed data-collection information and introduce SteerLM 2.0 to use multi-faceted rewards for complex multi-requirement instructions.
2 Dataset
HelpSteer2 combines diverse prompts and responses with multi-attribute annotation, quality filtering, and permissively licensed model-generated data. Its analysis shows stronger responses, high annotation agreement after refinement, and shifted relationships among helpfulness attributes.
- Prompt Collection: Over 95% of prompts come from ShareGPT user inputs, supplemented by proprietary prompts, while Assistant turns are removed to avoid model-specific licensing restrictions.
- Prompt Collection: Prompts are clustered into approximately 1000 topics and sampled across complexity levels to increase topical and difficulty diversity.
- Multi-turn Prompt Completion: Approximately 29% of samples are multi-turn prompts, whose Assistant turns are replaced with responses from a 22B in-house model.
- Response Generation: Two responses per prompt are generated from different sources, using diverse internal model generations and alignment techniques to increase response diversity.
- Response Annotation: Each response receives Likert-5 ratings for helpfulness, correctness, coherence, complexity, and verbosity, with process improvements over HelpSteer.
- Response Annotation: Sequentially rating two responses to the same prompt improves score calibration and quality assurance but requires substantially more annotation effort and resources.
- Response Annotation: Cohen’s κ for helpfulness reached 0.706 after annotation-process improvements, while the final filtering removed about 10% of samples under a two-point disagreement threshold.
- Dataset Analysis: Response length increased 3x from 497.3 to 1492.6 characters, while HelpSteer2 prompts average 712 characters versus 2491 in HelpSteer.
3 Reward Model
The paper trains multi-attribute reward models on HelpSteer2 and evaluates them on Reward Bench, where they achieve strong overall and category-specific performance. The approach uses scalar attribute ratings rather than only binary preferences, while performance varies across benchmark categories.
- Evaluation: Reward Bench evaluates whether the chosen response receives a higher reward than the rejected response across 2985 diverse tasks.The benchmark is organized into four main categories, with higher scores indicating better performance.
- Results: 92.0% Overall is achieved by the Nemotron-4 340B reward model trained on HelpSteer2, topping Reward Bench’s primary leaderboard.The same dataset with Llama 3 70B reaches 88.8% Overall.
- Training: SteerLM Reward Model training predicts five scalar ratings—Helpfulness, Correctness, Coherence, Complexity, and Verbosity—instead of only binary preferences.These ratings range from 0 to 4 and provide fine-grained supervision for distinguishing response attributes.
- Category Results: HelpSteer2-trained models surpass the second-best model by 6.5% in Chat-Hard, but underperform many alternatives on Prior Sets.They also perform well in Safety and Reasoning, although they do not surpass the strongest models trained on specific reasoning datasets.
4 Aligned Models
The paper evaluates Iterative DPO, PPO, and SteerLM for aligning LLMs with the Llama 3 70B Reward Model, using multiple helpfulness, correctness, verbosity, and instruction-following benchmarks. Across metrics, at least one resulting model matches or exceeds Llama 3 70B Instruct while using substantially less alignment data.
- Approaches and evaluation: The study compares Iterative DPO, PPO, and SteerLM as three approaches for using the Llama 3 70B Reward Model to align LLMs.The evaluation uses MT-Bench, TruthfulQA MC2, mean response length, AlpacaEval 2.0 Length Controlled, and Arena Hard.
- Overall results: At least one reward-model-trained model matches or exceeds Llama 3 70B Instruct across all evaluated metrics.The comparison uses Llama 3 70B Instruct and GPT-4-0613 as baselines.
- Overall results: 10 thousand HelpSteer2 preference pairs and 100 thousand SFT samples require only 1% of the data used to train Llama 3 70B Instruct.The paper contrasts its transparent alignment recipe with the undisclosed recipe using 10 million samples across SFT and preference-based training.
- DPO results: DPO performs best on TruthfulQA and Arena Hard, with most of the gain attributed to HelpSteer2 preference selection and a further boost from Iterative DPO.Helpfulness and Correctness have a Pearson correlation of 0.943 in HelpSteer2, supporting differentiation between right and wrong answers.
- PPO results: PPO performs best on AlpacaEval 2.0 Length Controlled but severely degrades TruthfulQA, possibly because HelpSteer2 prompts underrepresent multiple-choice questions.The paper links AlpacaEval performance to concise, sufficiently detailed responses and suspects policy drift for the TruthfulQA decline.
- SteerLM results: SteerLM performs best on MT-Bench, where complex multi-requirement prompts benefit from fine-grained contrastive learning across multiple sampled responses.SteerLM training increases the likelihood of highly rewarded responses while avoiding mistakes made by other sampled responses.
5 Conclusion
HelpSteer2 is a small, permissively licensed dataset that efficiently trains high-performing reward models and supports alignment of large language models with human preferences.
- HelpSteer2 contains 10k response pairs, has CC-BY-4.0 licensing, and achieves 92.0% on RewardBench’s primary dataset.Its Cohen’s κ is 0.791, and the result was ranked first as of 12 June 2024.
- A trained Llama 3 70B reward model can align Llama 3 70B Base models to match or exceed Llama 3 70B Instruct and GPT-4-0613 on major alignment metrics.The reported metrics include MT Bench, TruthfulQA, AlpacaEval 2.0 LC, and Arena Hard.
A Limitations
The dataset offers commercial accessibility and training efficiency, but its use is constrained by safety risks and possible demographic and attribute-combination biases.
- Potential demographic diversity gaps and uneven attribute combinations may introduce bias or uneven performance across use cases.The annotators were exclusively US-based, and not every attribute combination is equally represented.
- The commercially friendly license and small dataset size can broaden access to reward-model training while reducing computational costs.The authors connect accessibility with democratized use and practical deployment of advanced AI capabilities.
- Powerful models trained with HelpSteer2 could pose risks if used by malicious actors.The authors identify protective measures such as NeMo Guardrails as a mitigation.
C Ethical Considerations
HelpSteer2’s annotation work was conducted through Scale AI under stated ethical-review and fair-pay practices.
- Scale AI completed an ethical review before data collection began.The annotator pool was contracted through Scale AI.
- Annotator compensation reflected project requirements, expertise, regional living costs, and transparent platform-listed pay.
D Related Work
Related preference datasets span permissively licensed human-annotated resources, proprietary-model-generated data, and multi-turn completion procedures designed to improve conversational coherence.
- Preference datasets: Open Assistant and HH-RLHF provide large-scale human preference data, while HelpSteer offers more than 37,000 prompt-response pairs.Open Assistant includes over 160,000 messages in 35 languages, and HH-RLHF includes over 160,000 preference comparisons.
- Preference datasets: Domain-specific resources such as WebGPT, Summarize, and SHP are less useful for building domain-general language models.
- Synthetic feedback: RLAIF reduces preference-data cost and time by using language models to label or rank responses, but provider terms can restrict commercial use.
- Multi-turn completion: HelpSteer2 avoids restrictive third-party Assistant responses in multi-turn prompts by generating intermediate turns with a custom model.
- Multi-turn completion: The completion model is trained on Assistant turns while exposing future user turns through the first user instruction.This lets generated responses refer to later user turns while retaining a standard causal attention mask.
- Multi-turn completion: A PPO-based RLHF stage uses Open Assistant conversation trees so the reward model favors responses consistent with future conversation turns.
- Multi-turn completion: Candidate responses are sampled and ranked using reward models and next-user-turn likelihood, then the best average-ranked completion is selected.
- Annotation: Six-axis annotation rates responses on five-point scales, except binary safety ratings, with confidence recorded separately.Invalid tasks are flagged when they contain PII, substantially non-English content, coding requirements, or Assistant-specific questions.
G.2 Flagging invalid tasks
The guidelines identify tasks that should be flagged rather than rated, including those involving PII, substantial non-English fluency, coding abilities, or the assistant’s identity and characteristics. Rated responses are assessed across helpfulness, correctness, coherence, language complexity, verbosity, and safety.
- Invalid tasks: Tasks containing PII are flagged and not rated, including names, addresses, SSNs, emails, and phone numbers.
- Invalid tasks: Substantially non-English tasks are flagged, while prompts understandable in mostly English remain valid.
- Invalid tasks: Tasks requiring non-trivial coding are flagged, but computer-science questions that do not require writing or understanding code remain evaluable.
- Invalid tasks: Prompts about the responding assistant’s identity, license, usage, or characteristics are flagged rather than ranked.
- Rating axes: Responses are rated for helpfulness, correctness, coherence, language complexity, verbosity, and safety.
G.3.2 Examples
The examples illustrate how the rating axes distinguish concise factual answers from more detailed answers and identify unsafe or misleading responses. Conversational tasks are evaluated on the final model turn in context, with special handling for unavailable internet access and prompts addressing the model as ChatGPT.
- Examples: A concise answer to a binary question receives high helpfulness, correctness, coherence, simplicity, succinctness, and safety ratings.
- Examples: A slightly elaborated answer remains correct, coherent, and safe but receives lower helpfulness and succinctness ratings than the concise answer.
- Examples: An extensive eczema response receives helpfulness, correctness, coherence, simplicity, and safety ratings of 3, 3, 4, 4, and Pass.
- Examples: A response attributing eczema to bodily and spiritual impurity is presented as a contrasting example for evaluation.
- Evaluation context: Conversational data are judged using only the final model turn while retaining the preceding conversation context.
- Special cases: When internet access is needed, declining is rated higher than fabricating facts, and references to ChatGPT are ignored unless the task intrinsically requires that identity.
H Evaluation Details
Evaluation uses Reward Bench and several secondary benchmarks to assess reward-model preferences, helpfulness, factuality, and verbosity. The protocol excludes biased Prior Sets and replaces the default MT Bench judge after finding substantial reference-answer errors.
- Reward Bench: Reward Bench evaluates Chat, Chat-Hard, Safety, and Reasoning categories, with category accuracy calculated from per-task averages.
- Reward Bench: Prior Sets are excluded because they are biased toward models trained on their constituent preference datasets and are not reported consistently across models.
- Secondary benchmarks: MT Bench contains 80 multi-turn questions, totaling 160 prompts across eight categories, and is used for helpfulness evaluation.
- MT Bench: GPT-4-0125-Preview replaces GPT-4-0613 as the MT Bench judge because 13 of 30 reference answers in code, math, and reasoning were wrong.
- MT Bench: 0.8 point lower: GPT-4-0125-preview MT Bench scores lower on average than GPT-4-0613 while correlating better with Chat Arena Elo.
- MT Bench: R2 was 0.819 for GPT-4-0125-Preview MT Bench versus 0.703 for GPT-4 MT Bench when regressed against Chat Arena Elo.
- Secondary benchmarks: TruthfulQA MC2 measures the normalized probability assigned to true answers across 817 questions in 38 categories, with higher values indicating greater factual correctness.
- Secondary benchmarks: Arena Hard and AlpacaEval 2.0 Length Controlled provide secondary helpfulness measures using challenging or user-query-like instructions.
I Compute requirements
Training compute is reported in H100-equivalent node-hours for experiments run on eight-GPU H100/A100-80GB SXM nodes. A substantial portion of the measured compute came from unoptimized text generation and could be reduced through optimization.
- Compute measurement: Compute is measured in H100-eqv. node-hours on nodes containing 8 H100/A100-80GB SXM GPUs.
- Compute measurement: A100 compute is divided by 3 to produce H100-equivalent figures for clarity.
- Compute caveat: A bulk of the reported compute was spent on unoptimized text generation, which optimized generation would greatly reduce.