Source-linked AI summary
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, Yahui Zhou
TL;DR
Reward modeling for LLM alignment is challenged by complex human preferences and noisy, inconsistently labeled preference data. The paper develops data-selection and filtering techniques, combines them with training analyses, and builds Skywork-Reward models from a compact curated dataset. The resulting model series achieves top RewardBench rankings, while the authors note that contamination affects preference datasets and evaluations.
Problem
Complex human preferences and noisy, inconsistently labeled open-source preference data make reliable reward modeling difficult.
Method
The paper curates high-quality preference data through selection and filtering strategies, analyzes loss functions, and trains Skywork-Reward models on compact dataset mixtures.
Results
Skywork-Reward-Gemma-2-27B ranks first on RewardBench, while Skywork-Reward-Llama-3.1-8B surpasses all models except SFR-LLaMa-3.1-70B-Judge-I.
Takeaways & Limitations
The curated preference collection has been adopted in subsequent research, indicating practical applicability for reward-model training.
Takeaways & Limitations
Preference datasets, including Skywork Reward Preference 80K and Preference 700K, contain contamination overlapping with RewardBench test prompts.
Abstract
from arXiv · showhide
In this report, we introduce a collection of methods to enhance reward modeling for LLMs, focusing specifically on data-centric techniques. We propose effective data selection and filtering strategies for curating high-quality open-source preference datasets, culminating in the Skywork-Reward data collection, which contains only 80K preference pairs -- significantly smaller than existing datasets. Using this curated dataset, we developed the Skywork-Reward model series -- Skywork-Reward-Gemma-27B and Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top position on the RewardBench leaderboard. Notably, our techniques and datasets have directly enhanced the performance of many top-ranked models on RewardBench, highlighting the practical impact of our contributions in real-world preference learning applications.
1. Introduction
Reward modeling is a scalable way to align LLM outputs with user preferences, but its effectiveness depends heavily on the quality of preference data. Skywork-Reward addresses this challenge through targeted data curation and training analysis.
- Human preferences are complex and variable, making them difficult to represent exhaustively in reward models.
- Noisy open-source preference datasets can contain subtle response differences or inconsistent labels that degrade reward-model performance.
- The paper proposes lightweight, transparent preference-data collections using publicly available sources and selection strategies focused on performance-relevant pairs.
- The authors analyze loss functions and report that vanilla Bradley-Terry loss consistently outperforms alternative approaches in reward-modeling tasks.
- The resulting Skywork-Reward models show significant RewardBench improvements, with the model series ranking first and seventh as of October 2024.
2. Related Work
Reward modeling research spans discriminative, generative, and implicit-reward approaches. Skywork-Reward uses the discriminative approach and achieves top RewardBench rankings.
- RLHF uses reward models to learn preference- or objective-based reward functions that guide LLMs toward desired behaviors.
- Discriminative Models: Discriminative reward models commonly use Bradley-Terry loss to maximize reward differences between chosen and rejected responses for binary ranking.
- Discriminative Models: Related work improves reward modeling through regularization, dataset de-biasing, and other refinements addressing generalization and response-length bias.
- Generative Models: Generative reward models provide nuanced and interpretable assessments but often underperform discriminative models on reward-modeling tasks.
- Implicit Rewards via DPO Models: DPO derives a reward signal from the current and initial supervised policies without requiring an explicitly trained reward model.
- Skywork-Reward belongs to the discriminative-model category and achieves top rankings on RewardBench.
3. Method
Skywork-Reward constructs a lightweight, publicly sourced preference dataset through targeted composition, selection, and filtering, then trains reward models with pairwise ranking objectives. The process reduces an initial 378K-sample mixture to 80K pairs while emphasizing high-quality, math, code, and safety-relevant data.
- Method overview: The methodology combines a public-data mixture, selection and filtering procedures, and reward-model training objectives to improve effectiveness while preserving transparency.The approach covers dataset composition, Magpie and WildGuardMix filtering, and loss-function design.
- Dataset mixture: The proposed composition targets important RLHF abilities and domains, especially math and code, while reducing overall data requirements through exclusively public sources.The design is intended to support transparency, reproducibility, and broader adoption without proprietary datasets.
- Dataset mixture: The original seven-dataset mixture contains approximately 378K samples, with Magpie contributing about 93% and potentially diluting the influence of other datasets.This imbalance motivates subsequent filtering and composition changes.
- Data selection and filtering: Magpie selection uses generator identity and average ArmoRM scores, prioritizing data from stronger models after manually adjusting Air and Pro (Llama 3) scores.The adjustments subtract 0.1 from Air scores and 0.05 from Pro (Llama 3) scores to align the intended model ordering.
- Data selection and filtering: The Magpie filter independently retains the top 30% of Math and Code & debugging pairs and samples 10% from all remaining categories combined.This category-aware sampling emphasizes math and code while consolidating other tasks into one group.
- Training objective: For training, the method uses the standard Bradley-Terry pairwise ranking loss, while tested alternatives provided no consistent overall performance improvement and sometimes reduced effectiveness.The ranking objective compares scalar rewards for chosen and rejected responses; the loss-variant finding is reported across the tested alternatives.
4. Experiment
The experiments evaluate reward models trained with different preference datasets and loss functions on RewardBench, while also examining prompt contamination and further data filtering. Results favor compact, carefully curated datasets and show strong benchmark performance, though contamination complicates interpretation.
- Evaluation setup: RewardBench evaluates reward models across chat, reasoning, and safety using prompt-chosen-rejected trios.The benchmark measures whether models assign higher scores to chosen responses than rejected responses.
- Main results: Skywork-Reward-Gemma-2-27B ranks first on RewardBench, while Skywork-Reward-Llama-3.1-8B surpasses all models except SFR-LLaMa-3.1-70B-Judge-I.The models use the Skywork Reward Preference 80K dataset and demonstrate robust performance across all four categories.
- Main results: The 27B model is the only model above 90 on Chat Hard, exceeding Nemotron-4-340B-Reward by more than four points, with a score of 87.1.Chat Hard is the adversarial preference category highlighted in the reported comparison.
- Dataset comparisons: The complete 378K dataset outperforms Preference 700K for Llama 3 while providing a competitive Chat Hard advantage and balanced performance across four categories.This comparison supports the reported quality-over-quantity finding under matched training conditions.
- Further filtering: Refining the dataset from 80K to 66K pairs yields scores of 96.3 and 94.9 for the 27B and 8B models, while adding 20K selected Magpie samples raises them to 96.8 and 95.5.The enhanced models were not released because they required further RLHF-pipeline testing, and it remained unclear whether the scores reflected overfitting or improved reward signals.
- Loss ablations: Bradley-Terry loss achieves the highest average score of 93.8 and the best overall balance across Chat, Chat Hard, Safety, and Reasoning.Alternative losses show marginal gains in some categories but reduce performance in Chat and do not consistently outperform the baseline.
- Prompt contamination: RewardBench contamination affects Skywork and other preference datasets, motivating decontamination and stricter evaluation dataset selection.The authors removed contaminated Magpie Ultra pairs to create Skywork Reward Preference 80K v0.2, while noting that minor contamination may persist elsewhere.
5. Closing Remarks
The report introduces the Skywork-Reward Preference 80K collection and shows that carefully curated, smaller datasets can support state-of-the-art reward models despite fewer samples and straightforward training. Its findings emphasize data quality, targeted filtering, decontamination, and alignment between datasets and evaluation criteria.
- The results highlight prioritizing data quality over quantity and using targeted filtering and selection when constructing preference datasets.
- The decontaminated v0.2 dataset empirically improves scores across most RewardBench categories.
- Experiments reaffirm the Bradley-Terry loss as the most effective loss function in this setting across various tasks.
- The findings emphasize aligning datasets with evaluation criteria when developing and assessing reward models.