PaperlayerView exact revision on arXiv

Source-linked AI summary

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, Oleksii Kuchaiev

arXiv:2505.11475v2cs.CLcs.AIcs.LG

TL;DR

Open preference data must improve in both quality and diversity to support RLHF for general-domain instruction-following models. HelpSteer3-Preference addresses this gap with specialist human annotations across STEM, coding, multilingual, and general tasks, producing reward models that reach 82.4% on RM-Bench and 73.7% on JudgeBench.

  • Problem

    Open preference datasets need greater quality and diversity to support RLHF across increasingly challenging, multilingual, and difficult-to-verify tasks.

  • Method

    The paper collects over 40,000 preference samples using specialist annotators and responses from 17 language models across General, STEM, Code, and Multilingual categories.

  • Results

    82.4% on RM-Bench and 73.7% on JudgeBench are achieved by reward models trained on HelpSteer3-Preference.

  • Takeaways & Limitations

    HelpSteer3-Preference provides a permissively licensed dataset for training reward models and supporting RLHF across diverse real-world tasks.

Abstract

from arXiv · show

Preference datasets are essential for training general-domain, instruction-following language models with Reinforcement Learning from Human Feedback (RLHF). Each subsequent data release raises expectations for future data collection, meaning there is a constant need to advance the quality and diversity of openly available preference data. To address this need, we introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0), high-quality, human-annotated preference dataset comprising of over 40,000 samples. These samples span diverse real-world applications of large language models (LLMs), including tasks relating to STEM, coding and multilingual scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This represents a substantial improvement (~10% absolute) over the previously best-reported results from existing RMs. We demonstrate HelpSteer3-Preference can also be applied to train Generative RMs and how policy models can be aligned with RLHF using our RMs. Dataset (CC-BY-4.0): https://huggingface.co/datasets/nvidia/HelpSteer3#preference Models (NVIDIA Open Model): https://huggingface.co/collections/nvidia/reward-models-68377c5955575f71fcc7a2a3

1 Introduction

HelpSteer3-Preference addresses limited diversity in open preference data by collecting specialist-annotated samples across STEM, coding, multilingual, and general tasks. Its reward models reach leading results on RM-Bench and JudgeBench.

  • Motivation: Preference datasets support RLHF for domains where response quality cannot be easily verified from ground truth.Human preferences remain important for qualities such as creativity, pace, and plot in tasks like story writing.
  • Prior preference datasets: Existing general-domain preference datasets improved quality through stronger model annotators, dataset combination, or stricter annotation and filtering.These approaches included GPT-4 annotation, combining high-quality datasets, and using 3–5 independent annotations per sample.
  • Motivation: Data diversity has lagged behind quality improvements as language models are applied to increasingly challenging coding and multilingual tasks.The paper highlights longer coding and debugging scenarios and limited language representation in publicly available preference datasets.
  • Contribution: HelpSteer3-Preference uses specialist annotator pools for STEM, Code, Multilingual, and General categories, covering real-world prompts with responses from 17 language models.The dataset contains over 40,000 samples, and specialist inclusion may require relevant degrees, work experience, or language proficiency.
  • Results: 82.4% on RM-Bench and 73.7% on JudgeBench are achieved by the best reward models trained with HelpSteer3-Preference.These results represent approximately 10% absolute improvements over previously reported top-performing reward models.

2 Dataset

HelpSteer3-Preference combines specialist annotation, diverse prompt and response sourcing, post-processing, and broad coverage of programming and natural languages. The resulting dataset shows strong inter-rater reliability and low position bias.

  • Prompt Collection: HelpSteer3-Preference sources prompts from ShareGPT for Code and Multilingual tasks and WildChat-1M for General and STEM tasks.The sourcing strategy was designed in part to avoid potential prompt overlap with HelpSteer2-Preference.
  • Response Generation: Responses were generated from 17 commercially permissive language models, while multi-turn annotations used generated intermediate turns and contexts capped at 2000 words.Annotators evaluated only the last assistant turn in the multi-turn setup.
  • Preference Post-Processing: Each sample received 3–5 independent preference annotations, and samples with invalid responses or large disagreement among the most agreeing annotators were removed.The retained annotations were the three that agreed most closely in preference strength.
  • Preference Annotation: More than 6400 annotators from 77 countries or regions participated, with higher inclusion standards for STEM, Code, and Multilingual subsets.Requirements included relevant degrees, software-engineering experience, and language fluency, depending on the subset.
  • Improvements over HelpSteer2-Preference: HelpSteer3-Preference contains 12.2% STEM, 21.9% Code, 19.9% Multilingual, and 46.0% General samples.The dataset contains 40,476 samples, exceeding HelpSteer2-Preference’s 9,125 samples by more than four times.
  • Diverse Programming and Natural Languages: The Code and Multilingual subsets contain 14 programming languages and 13 natural languages, respectively.Python represents 38.2% of Code tasks, while Chinese represents 30.2% of Multilingual tasks.
  • Dataset Analysis: Weighted Cohen’s κ exceeds 0.8 for every subset, indicating strong inter-rater reliability, while mean preferences indicate low position bias.The weighted metric penalizes larger annotation disagreements more heavily than smaller disagreements.
  • Preference Distributions: General, STEM, and Code preferences are bimodal, whereas Multilingual preferences are unimodal with a peak near equal preference.The paper hypothesizes that annotation-vendor differences may contribute to these distributional patterns.

3 Reward Models

The paper evaluates Bradley-Terry and Generative Reward Models trained on HelpSteer3-Preference across RM-Bench and JudgeBench. Performance varies by subset and response style: multilingual training leads RM-Bench, English training leads JudgeBench, and GenRMs improve accuracy but require substantially more compute.

  • Overall: 82.4% is the highest RM-Bench score among Bradley-Terry models, achieved by a Reward Model trained only on the Multilingual subset.The English model trained on General, STEM and Code achieves 73.7% on JudgeBench and 79.9% on RM-Bench.
  • English vs. Multilingual RM: 82.4% versus 79.9% overall on RM-Bench shows that the Multilingual RM outperforms the English RM, while difficulty reverses the pattern on Easy and Hard prompts.English RM scores 92.1% versus 86.5% on Easy prompts, whereas Multilingual RM scores 80.0% versus 71.1% on Hard prompts.
  • English vs. Multilingual RM: 24.8% is the English RM win-rate drop when a concise chosen response is compared with a verbose rejected response, versus 6.4% for the Multilingual RM.The English RM is therefore more influenced by response length in this comparison.
  • Code: 65.5% on RM-Bench and 54.8% on JudgeBench are obtained by training only on the Code subset.The Code-only model nevertheless reaches the highest RM-Bench Easy score, 94.0%, reflecting annotation attention to comments and coding style as well as correctness.
  • Generative Reward Models: 84.2% on RM-Bench and 75.1% on JudgeBench are achieved by the best Generative Reward Models, improving over the best Bradley-Terry scores.Voting@32 further raises these results to 85.5% and 78.6%, respectively, while requiring substantially more compute.

4 Aligned Models

The paper evaluates policy models aligned with reward models trained from HelpSteer3-Preference prompts, using MT Bench, Arena Hard, and WildBench. RLOO improves all three metrics over the initial Llama-3.3-70B-Instruct policy, with the English reward model showing the largest reported gains.

  • 4.1 Evaluation and Training: Policy models are aligned with trained reward models using HelpSteer3-Preference prompts.The evaluation follows prior policy-alignment work and uses MT Bench, Arena Hard, and WildBench.
  • 4.2 Results: RLOO with each of three reward models improves MT Bench, Arena Hard, and WildBench over the initial Llama-3.3-70B-Instruct policy.
  • 4.2 Results: 8.29 to 9.24 on MT Bench, 62.4 to 87.0 on Arena Hard, and 52.5 to 60.0 Overall on WildBench are the English RM’s gains after RLOO.The English reward model also performs well against the cited GPT-4o and Claude-3.5-Sonnet baselines.
  • 4.2 Results: The English RM performs best on WildBench Creative, Information Seeking, and Coding, close second on Planning, and behind GPT-4o on Data Analysis.The authors suggest the Data Analysis gap may reflect limited representation of related tasks in HelpSteer3-Preference.

5 Conclusion

The paper releases HelpSteer3-Preference as a permissively licensed, high-quality preference dataset covering diverse real-world STEM, coding, and multilingual tasks. Reward models trained on it achieve strong RM-Bench and JudgeBench performance, and the dataset supports generative reward modeling and RLHF policy alignment.

  • HelpSteer3-Preference is a CC-BY-4.0 high-quality preference dataset with diverse real-world STEM, coding, and multilingual prompts requiring specialist annotators.It is presented as a general-domain dataset.
  • 82.4% on RM-Bench and 73.7% on JudgeBench are achieved by reward models trained on HelpSteer3-Preference.The paper reports an approximately 10% absolute lead over the best existing reward models.
  • The dataset is also used to train Generative Reward Models and align policy models with RLHF using the trained reward models.

NeurIPS Paper Checklist

The checklist judges that the abstract and introduction accurately reflect the paper’s contributions and scope. It identifies the collection and release of a high-quality, diverse, commercially permissive preference dataset as the main claim.

  • Yes, the abstract and introduction accurately reflect the paper’s contributions and scope.
  • The main claim is collecting and releasing a high-quality, diverse, commercially permissive preference dataset.
  • Claims should state contributions, assumptions, limitations, and the extent to which results generalize to other settings.The checklist permits aspirational goals when clearly presented as motivation.

2. Limitations

The paper addresses limitations through disclosures on reproducibility, open access, experimental details, statistical significance, compute resources, societal impact, and responsible data release. It reports no theoretical results and states that limitations are discussed in Appendix A.

  • The paper discusses its limitations in Appendix A.
  • The paper reports no theoretical results, so theoretical assumptions and proofs are not applicable.
  • Reproducibility is supported by reporting experimental details and sharing data publicly, with code linked in Appendix H.Training and test details are specified in Sections 3.1, 3.2, 4.1, and 4.2.
  • The paper reports error bars where evaluations provide such information, while deterministic metrics do not receive error bars.
  • Compute-resource requirements are discussed in Appendix H.
  • The released data is described as posing no high misuse risk because potentially unsafe or personally identifying samples were removed before annotation.

A Limitations

HelpSteer3-Preference expands preference coverage but remains bounded by language, modality, conversation-naturalness, and potential misuse constraints.

  • Dataset scope: The dataset covers General, STEM, Code, and Multilingual areas but excludes samples at their intersections.It also includes only 14 programming languages and 13 natural languages.
  • Dataset scope: Vision and other modalities are omitted to preserve diverse, commercially permissive CC-BY-4.0 data under copyright constraints.The authors identify multimodal preference data as an area for future work.
  • Data construction: Multi-turn prompt infilling can occasionally produce unnatural conversations, although manual inspection suggested the issue was rare.The authors suggest such conversations may improve robustness to ambiguous queries and user mistakes.
  • Safety: Unsafe prompts, responses, bias, stereotypes, and personally identifiable information were filtered before annotation, but misuse cannot be guaranteed absent.The dataset therefore mitigates direct risks without eliminating all potential misuse.
  • Annotation process: Annotation was conducted through Scale AI and Translated, with consent obtained and ethical review completed before data collection.The vendors also implemented measures intended to prevent annotators from using LLMs during annotation.

D Geographical Locations

The annotation guidelines define geographic and task-screening procedures, including invalid-task flags, response ratings, and written justifications.

  • Annotation procedure: Annotators rate each response on five axes using a five-point Likert scale, then rank the responses by preference strength.They evaluate the final assistant turn in multi-turn conversations within the full conversational context.
  • Annotation procedure: Each response receives a moderate-length helpfulness justification, while the preference ranking receives a separate short explanation.Annotators are instructed not to reference the other response in a helpfulness justification.
  • Operational constraints: Annotators may use publicly accessible, non-paywalled search engines to validate factual accuracy, but copying AI-generated content is prohibited.The guidelines state that no further rating is required once a task receives a disqualifying flag.
  • Task screening: Tasks are flagged and skipped when they contain real-world PII, require substantial non-English fluency, or need coding abilities outside the applicable annotation scope.The guidelines give examples including identifiers, foreign-language content, and non-trivial code.
  • Task screening: Additional exclusion flags cover assistant-specific identity or license questions, length beyond 2000 words, post-July-2024 knowledge, and capabilities unavailable to the responding models.Unavailable capabilities include network access, local file interaction, sampling-parameter changes, current date knowledge, and image, audio, or video analysis.
  • Task screening: High-risk advice, potentially copyrighted requests, and unsafe-content requests are flagged regardless of user intent or explicitness.Examples include medical, legal, financial, or insurance advice and requests involving illegal, harmful, hateful, or sexually explicit material.

G Analysis of Preference Justifications

The paper analyzes preference justifications through keyword associations with HelpSteer attributes, highlighting differences between task subsets.

  • Analysis method: Keyword proportions in preference justifications are used to estimate associations with each HelpSteer attribute.The keywords come from the 500 most frequent words, making the measure high-precision but low-recall.
  • Subset differences: 38.0% of Code samples mention response verbosity, compared with 31.3% of General samples.The paper links this difference to Code-specific guidance about comments supporting code understanding.
  • Subset differences: Code justifications also mention complexity more often, consistent with guidelines emphasizing established coding styles.This comparison concerns keyword mentions in justifications rather than direct human ratings of the attribute.

H Training Details

The experiments train reward, aligned policy, and generative reward models with distinct optimization settings, validation splits, and sequence lengths.

  • Experimental setup: All experiments use NeMo-Aligner, split each dataset subset into 95% training and 5% validation, and use 4096-token sequences except 8192-token Generative Reward Models.The reported models include Llama-3.3-70B-Instruct and Llama-3.3-Nemotron-Super-49B-v1.
  • Reward Model: Reward models use global batch size 128, AdamW, 10 warm-up steps, learning-rate search over {1, 2, 3}e-6, and one training epoch.One epoch is used because Bradley-Terry-style losses are reported to overfit beyond one epoch.
  • Aligned Models: Aligned models use 64 prompts per global or rollout batch, four samples per prompt, AdamW, learning-rate search over {2, 5}e-7, and KL penalty 0.01.A lower learning rate was tested after the English reward-model policy collapsed around step 140 at 5e-7.
  • Generative Reward Model: Generative Reward Models are trained with GRPO to predict preference rankings and helpfulness scores from the annotation guidelines.Their reward penalizes format violations and absolute errors in predicted rankings and helpfulness scores.
  • Generative Reward Model: Generative Reward Model training samples eight responses per prompt, uses rollout prompt size 64, global batch size 256, and KL penalty 0.001.All HelpSteer3-Preference and English models train for two epochs, while smaller Multilingual and HelpSteer2-Preference models train for four.
  • Compute: Compute requirements and optimal hyperparameters are reported in Table 8 in H100-node-hours.The experiments use nodes containing eight H100-80GB SXM GPUs.

I Reasoning vs. Non-Reasoning Model to Initialize Generative RM Training

Generative reward models perform better when initialized from a reasoning-capable model rather than a non-reasoning instruction model. The evaluation uses response-scoring and ranking guidelines that assess helpfulness across several dimensions.

  • Initialization choice: Reasoning capability in the initial model substantially improves the final Generative RM’s judging accuracy on both benchmarks.The comparison uses Llama-3.3-Nemotron-49B-v1 versus Llama-3.3-70B-Instruct.
  • Scoring procedure: Generative RMs evaluate responses by producing critiques and then assigning scores based on those critiques.The prompt instructs the model to analyze responses before scoring them.
  • Scoring criteria: Helpfulness scoring considers correctness, clarity, instruction following, relevance, and appropriate detail or creativity.These criteria are applied to each response individually.
  • Ranking procedure: When two responses are present, the evaluator assigns a separate ranking score even if their individual helpfulness scores are identical.The ranking score ranges from 1 to 6 and must distinguish the responses strictly.

K Example Responses

The example compares ABC-notation melodies generated by an initial policy and models trained with multilingual, baseline, and English reward models. The responses explain notation conventions alongside the generated melodies.

  • Example Responses: The initial policy response provides a G-major folk melody in ABC notation with a 4/4 meter and eighth-note default length.Its notation uses X:1, T:Folk Tune, M:4/4, L:1/8, and K:G headers.
  • Example Responses: The English-RM response expands beyond the requested melody with explanations of ABC notation, playback options, and composition advice.The added material covers tools, notation basics, repetition, stepwise motion, and modal experimentation.
  • Example Responses: The ABC examples use headers for tune indexing, title, meter, default note length, and key, while note letters encode pitches and durations.Numbers after notes multiply the default note length.
  • Example Responses: The multilingual-RM response gives a G-major melody with repeated phrases, stepwise motion, and explanatory text about the notation.It also describes how the melody can be modified or played.
  • Example Responses: The baseline-RM response includes chord annotations such as [G] and [Em] in addition to the melody and explains their accompaniment role.It describes uppercase and lowercase notes as different octaves and explains duration multipliers.

L Stylistic Analysis Of HelpSteer3-Preference

Preferred responses are generally longer and use more Markdown than dis-preferred responses, but the size and pattern of these stylistic differences vary across subsets. The multilingual subset shows the smallest length gap and a distinct Markdown profile.

  • L Stylistic Analysis Of HelpSteer3-Preference: Preferred responses are longer and contain more Markdown features than dis-preferred responses across HelpSteer3-Preference subsets.The analysis covers headings, bold markers, and list items.
  • L Stylistic Analysis Of HelpSteer3-Preference: The multilingual subset has the smallest length difference, with preferred responses 3.6% longer than dis-preferred responses.Other subsets show gaps ranging from 8.7 to 19.4%.
  • L Stylistic Analysis Of HelpSteer3-Preference: Multilingual responses show a smaller gap in list items but larger gaps in headings and bold markers than the other subsets.This indicates that stylistic preference differences are not uniform across Markdown features.

M Stylistic Analysis Of Aligned Model Responses

Reward-model alignment changes response length and Markdown usage in model outputs. The English RM produces the largest increases, while the multilingual RM is less influenced by Markdown style but may still favor headings.

  • Length: The English-RM model substantially increases response length, the baseline-RM model moderately increases it, and the multilingual-RM model slightly decreases it.These comparisons are made against the initial Llama-3.3-70B-Instruct policy.
  • Length: The English-RM model adds substantial extra sections, including notation explanations, rationale, playback guidance, and composition tips.These additions extend beyond directly generating the requested melody.
  • Length: The authors believe the longer responses generally provide relevant information rather than merely exploiting reward through redundancy or unhelpful formatting.They support this interpretation with examples from melody generation and travel recommendations.
  • Limitations: The usefulness of elaborate responses depends on user knowledge, and additional information is not guaranteed to help every real-world user.The paper identifies personalization by user characteristics as future work.
  • Markdown: The English-RM model increases all measured Markdown features, whereas the baseline increases them moderately and the multilingual model increases headings while reducing or maintaining other features.The multilingual pattern is consistent with lower sensitivity to Markdown formatting.
Loading 2505.11475v2…