Source-linked AI summary
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, Oleksii Kuchaiev
TL;DR
Existing open-source helpfulness preference datasets do not specify what makes responses more helpful, and preference learning can favor length over substance. HelpSteer adds four attribute annotations to overall helpfulness, and training Llama 2 70B with SteerLM achieves a 7.54 MT Bench score, reported as highest among models trained with open-source datasets.
Problem
Open-source preference datasets do not specify which response attributes make answers more helpful, and black-box preferences can encourage length over substance.
Source: §1 · p. 1Method
HelpSteer annotates responses for correctness, coherence, complexity, and verbosity alongside overall helpfulness, and the dataset is used with SteerLM.
Source: §1 · p. 2, §4.4 · p. 6Results
7.54 on MT Bench: the HelpSteer-trained model scores highest among models trained with open-source datasets.
Source: §8 · p. 8Takeaways & Limitations
The resulting model is user-steerable in verbosity and complexity.
Source: §8 · p. 8HelpSteer contains English prompts and responses, and its US-based annotators may not represent helpfulness judgments in other countries.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Existing open-source helpfulness preference datasets do not specify what makes some responses more helpful and others less so. Models trained on these datasets can incidentally learn to model dataset artifacts (e.g. preferring longer but unhelpful responses only due to their length). To alleviate this problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated for the various aspects that make responses helpful. Specifically, our 37k-sample dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. Training Llama 2 70B using the HelpSteer dataset with SteerLM technique produces a model that scores 7.54 on MT Bench, which is currently the highest score for open models that do not require training data from more powerful models (e.g. GPT4). We release this dataset with CC-BY-4.0 license at https://huggingface.co/datasets/nvidia/HelpSteer
1 Introduction
The paper argues that black-box preferences leave the criteria for helpfulness opaque and may encourage models to value length over substance. It proposes four response attributes and introduces an annotated dataset and a model trained with it.
1 Introduction
- Black-box preferences can be inefficient in compute and data and risk encouraging longer responses over substance (Bai et al., 2022; Dong et al., 2023; Dubois et al., 2023; Singhal et al., 2023).Source: §1 · p. 1
- Helpfulness factors may vary across applications: creativity and humor can help in some settings but be counterproductive in formal business contexts.Task-specific work identifies quality factors such as accuracy, coverage, coherence, relevance, factuality, and completeness.Source: §1 · pp. 1–2
- Helpfulness can be assessed through correctness, coherence, complexity, and verbosity, a proposed foundational framework rather than an exhaustive set of factors.Correctness means pertinent facts without errors; coherence means clear, consistent expression; complexity reflects intellectual depth; verbosity reflects response detail.Source: §1 · p. 2
- The dataset contains 37k conversations, with each response annotated for those four attributes as well as overall helpfulness.Source: §1 · p. 2
- 7.54 MT Bench score: the HelpSteer-trained model is reported as highest among models that do not require training data from powerful proprietary models such as GPT4.The paper also reports releasing the dataset under CC-BY-4.0.Source: §1 · p. 2
2 Related Works
Related work includes preference-ranking datasets, response-level attribute ratings, and annotations generated with GPT-4. The paper also distinguishes its broad-task aim from task-specific preference datasets.
2 Related Works
- HH-RLHF contains preferred and rejected response pairs, while Open Assistant also rates response quality, creativity, and humor (Bai et al., 2022; Köpf et al., 2023).Source: §2 · p. 2
- GPT-4 has been used to annotate helpfulness-related aspects, but the paper questions trust in such annotations without clarity about biases and limitations or human validation (Sharma et al., 2023).It also raises a concern about using GPT-4 output to develop competing models under the cited Terms of Use.Source: §2 · p. 2
- Task-specific preference datasets cover summarization, question answering, code problems, and Reddit conversations, but are described as unlikely to improve performance across diverse tasks.Source: §2 · p. 2
3 Dataset
HelpSteer was collected to address shortcomings encountered with Open Assistant, especially response issues and limited representation of several tasks. Its analysis relates annotated attributes to helpfulness and reports their distributions and joint explanatory power.
3 Dataset
- The collection targets response problems and task gaps observed with Open Assistant, including factual inaccuracy, incoherence, oversimplification, excessive verbosity, and weaker performance on reference-text tasks.The dataset section describes its focus as collection methodology, prompt selection, response generation, and annotation quality.Source: §3 · p. 2, §3.1 · p. 3
3.1 Data collection
- The collection includes 10,459 single-turn prompts, with roughly 10% allocated to most categories and the remaining 20% to five tasks where SteerLM had diminished performance.Those five tasks were Rewrite, Summarization, Classification, Extraction, and Closed Question Answering; about half of prompts came from Scale AI and the rest were synthetically generated.Source: §3.1 · p. 3
- The finalized dataset contains 37,120 annotated samples, after annotation review and filtering.Responses were independently rated for helpfulness and the four attributes on a 0–4 Likert-5 scale; this was described as scaling linearly in response count, unlike comparative annotation’s quadratic workload.Source: §3.1 · pp. 3–4
3.2 Data Analysis
- Correctness and coherence correlate strongly with HelpSteer helpfulness (Pearson’s R > 0.6), while complexity and verbosity are weakly correlated (Pearson’s R > 0.2).In Open Assistant, creativity has Pearson’s R = 0.34 and humor has Pearson’s R = −0.099 with quality.Source: §3.2 · p. 4
- HelpSteer responses average 3.30 for coherence, 2.84 for correctness, 1.44 for complexity, 1.53 for verbosity, and 2.78 for helpfulness, on a 0–4 scale.Open Assistant responses average 2.57 for overall quality; its creativity and humor averages are 1.58 and 0.72.Source: §3.2 · p. 4, Table 2 · p. 4
- The four attributes jointly account for 73.0% of helpfulness variance, and each makes a significant contribution in the regression (p < 0.05).Source: §3.2 · p. 4
4 Experiments
Experiments assess whether HELPSTEER guides models toward helpfulness, factual accuracy, coherence, and appropriate complexity and verbosity, using automated and human evaluations and comparisons with baseline models.
4.1 Automatic Evaluation
- MT Bench measures helpfulness, TruthfulQA MC2 factuality, Llama 2 13B perplexity coherence, FKGL complexity, and mean response characters verbosity.MT Bench scores are GPT-4 evaluations averaged across prompts; higher scores indicate greater helpfulness.Source: §4.1 · p. 5
4.2 Human Evaluation
- Human evaluation ranks responses to 80 first-turn MT Bench prompts, with three annotators independently ranking each prompt’s responses and win rate and Elo used for comparison.Twelve volunteers with relevant undergraduate education evaluated responses blind; Fleiss’ κ was 0.383.Source: §4.2 · p. 5
4.3 Foundation models
- Experiments use Llama 2 70B as the main language model and Llama 2 13B for the STEERLM Attribute Prediction Model and RLHF reward model.Initial exploration found little benefit from the 70B attribute/reward model alongside substantially higher compute requirements.Source: §4.3 · p. 5
4.4 STEERLM
- STEERLM predicts semantic-attribute scores, annotates prompt-response data, trains attribute-conditioned supervised fine-tuning, and can bootstrap further data from generated responses.The optional final step samples diverse responses from the AC-SFT model for additional training.Source: §4.4 · p. 6
- Implementation uses OASST for AC-SFT, combines OASST and HELPSTEER labels for attribute prediction, replaces the language-model predictor with regression, and omits bootstrapping.Both models train for 800 steps with global batch size 128 and a constant 5e-6 learning rate; inference attributes are generally set to 4, except creativity, humor, and toxicity, set to 0.Source: §4.4 · p. 6
4.5 Baseline Models
- Baselines include Open Assistant supervised fine-tuning, RLHF on HH-RLHF, DPO on HH-RLHF or Open Assistant, and Llama 2 70B Chat trained with private data.Llama 2 70B Chat uses 27,540 private SFT samples and 1.4 million private comparison pairs for RLHF.Source: §4.5 · p. 6
5 Results
STEERLM outperforms the reported open-data baselines on MT Bench and leads the reported human comparisons. Its category-level results show gains over Llama 2 70B Chat in several instruction-following categories.
5 Results
- 7.54 MT Bench for STEERLM versus a 7.21 maximum for open-data RLHF/DPO baselines; the best RLHF baseline uses 5 times its compute, while similar-compute alternatives score ≤6.98.The comparison is against models trained with open-source datasets; Table 3 reports automatic evaluation against those models and Llama 2 Chat.Source: Table 3 · p. 7, §5 · p. 7
- 1050 Elo for STEERLM, with 57.5% wins against Llama 2 Chat and 62.9% against RLHF w. HH-RLHF in human pairwise comparisons.The human evaluation covered STEERLM, Llama 2 Chat, and RLHF w. HH-RLHF.Source: Table 4 · p. 7, §5 · p. 7
- STEERLM gains over Llama 2 70B Chat in Extraction, Coding, Math, Reasoning, and Roleplay; reported measures include TruthfulQA 0.5613, coherence 2.876 PPL, FKGL 8.658, and 1192.7 characters.The paper describes the responses as sufficiently detailed, contrasting them with succinct open-data models and highly verbose Llama 2 Chat.Source: §5 · p. 7, Figure 2 · p. 7
6 Ablation Studies
Ablations indicate that each HELPSTEER attribute contributes to MT Bench performance and to metrics measuring its corresponding attribute. Removing helpfulness or correctness also exposes trade-offs across helpfulness, factuality, and response complexity.
6 Ablation Studies
- Removing any HELPSTEER attribute lowers MT Bench and the metric measuring that attribute, indicating that the model uses each attribute to condition generation.The study excludes one or more attributes when training the Attribute-Condition SFT model.Source: §6 · p. 7, Table 5 · p. 8
- TruthfulQA MC2 rises from 0.5613 to 0.5754 when helpfulness is removed, which the authors say suggests that correctness can improve without explicit helpfulness optimization.The authors note this as a possible concern for mainstream RLHF, which optimizes helpfulness.Source: §6 · p. 8
- Removing correctness lowers MT Bench from 7.54 to 6.92 while allowing more complex-sounding responses, underscoring the paper’s concern about complexity without factual grounding.The authors argue that factuality should be trained explicitly alongside helpfulness.Source: §6 · p. 8
- Removing HELPSTEER attributes lowers MT Bench from 7.54 to 6.9, versus 7.54 to 7.36 when Open Assistant attributes are removed; using both yields 7.54, versus 6.29 for vanilla SFT.The authors interpret this comparison as suggesting that HELPSTEER attributes contribute more to helpfulness, while the two attribute sets complement each other.Source: §6 · p. 8
7 Steerability Demo
SteerLM allows users to customize responses at inference time by steering attributes such as complexity and verbosity. The demo links higher complexity to a higher reading grade level and higher verbosity to longer responses.
7 Steerability Demo
- SteerLM offers inference-time customization so end-users can adapt responses to different scenarios.The demo illustrates complexity steering with low- and high-complexity answers to “What is gravity?”Source: §7 · p. 8
- Increasing complexity raises the response’s reading grade level from 8.489 to 9.104.The passage presents this as an educationally relevant way to adapt answers for beginning learners and experts.Source: §7 · p. 8
- Increasing verbosity raises mean response length from 422.3 to 1125.2 characters.The passage connects different levels of detail to the needs of stakeholders such as executives, managers, and engineers.Source: §7 · p. 8
8 Conclusion
The authors report a strong MT Bench result for a Llama 2 70B model trained with HelpSteer and SteerLM, alongside user steering of response attributes. HelpSteer provides 37k samples with multiple helpfulness annotations and is released under CC-BY-4.0.
8 Conclusion
- HelpSteer contains 37k samples annotated for correctness, coherence, complexity, and verbosity as well as overall helpfulness, and is released under CC-BY-4.0.Source: §8 · p. 8
- 7.54 on MT Bench is the highest score among models trained with open-source datasets.The reported Llama 2 70B model was trained on HelpSteer using the SteerLM technique.Source: §8 · p. 8
- The model is reported as more truthful and coherent than Llama 2 70B Chat and other baselines, while allowing verbosity and complexity steering.Source: §8 · p. 8
Limitations
The dataset and evaluation have language and cultural scope boundaries: annotations cover English, and all annotators are based in the United States.
Limitations
- HelpSteer annotations cover English only, and the authors did not evaluate SteerLM on multilingual benchmarks.They say improvement on non-English prompts is unlikely, while noting that their collection methodology could be applied to other languages.Source: Limitations · p. 9
- The authors describe the collection methodology as applicable to building similar datasets in other languages and cultures.Source: Limitations · p. 9
- Because all annotators are based in the United States, the annotations are likely to reflect US understandings of helpfulness.The authors are uncertain whether the dataset reflects opinions from other countries and suggest the methodology could support further annotation in other cultures.Source: Limitations · p. 9
Ethics Statement
The authors describe contractor pay practices and channels for worker support and concerns.
Ethics Statement
- Scale AI calculates pay using factors including project specifics, required expertise, and regional cost of living, and lists it on its platform.Source: Ethics Statement · p. 9
- Contractors have multiple support channels, including 24/7 teams, moderated community discussions, and an anonymous hotline.Source: Ethics Statement · p. 9
- The Remotasks support team reviews worker concerns, while trained specialists review pay disputes.Source: Ethics Statement · p. 9
A.1 HELPSTEER Annotation Guidelines
The guidelines rate responses across five attributes on a five-point scale, defining what each attribute measures and illustrating rating levels with examples.
A.1 HELPSTEER Annotation Guidelines
- Annotators rate outputs on five 5-point Likert axes: helpfulness, correctness, coherence, complexity, and verbosity.The guidelines frame these ratings as fine-grained evaluation of prompt-response pairs.Source: §A.1 · p. 11
Correctness
- Correctness guidance emphasizes factual accuracy and completeness, while coherence concerns clarity, consistency, and whether a person can follow the response.The rating examples distinguish fully correct, partially correct, and incorrect responses, and describe degrees of clarity.Source: §A.1 · p. 11, Correctness · pp. 11–12
Complexity
- Complexity rates language and content from basic, readily understood responses to sophisticated or expert-level responses.The examples describe levels from simple wording through advanced terminology to expert-specific language.Source: §A.1 · p. 11, Complexity · p. 12
Helpfulness
- Helpfulness runs from 0 for a response that misses the user's intent to 4 for one fully aligned with the prompt's spirit.Intermediate ratings describe responses that are partly or mostly helpful but still fall short.Source: Helpfulness · p. 11
5. Verbosity: 1
- Verbosity is judged against what the prompt calls for, so equally long responses can differ in whether they are concise or excessive.The scale ranges from succinct answers to responses that are longer or more detailed than requested.Source: §A.1 · p. 11, Complexity · p. 13
5. Verbosity: 2
- The examples describe blue light as naturally present and short-wavelength, emitted by natural and artificial sources, and associate exposure—especially prolonged exposure from screens and televisions—with eye-damage risk.Source: §5 · p. 13
A.2 Compute requirements for training models
- Table 6 compares compute with MT Bench, where higher MT Bench and lower compute are better.The caption says experiments use 32 nodes, each with 8 A100-80GB SXM GPUs.Source: Table 6 · p. 14