PaperlayerView exact revision on arXiv

Source-linked AI summary

Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy

Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, Yahui Zhou

arXiv:2507.01352v3cs.CLcs.AIcs.LG

TL;DR

Open reward models perform poorly across many benchmarks, raising questions about whether preference datasets adequately capture nuanced human preferences. The paper introduces SynPref-40M and a human-AI two-stage curation pipeline, then trains eight Skywork-Reward-V2 models on curated data. Across seven major benchmarks, the series achieves state-of-the-art performance, while the authors note limitations in single-benchmark evaluation and in generalizing Gemma-2-27B modification results.

  • Problem

    Existing open reward models perform poorly on many benchmarks, while preference datasets may be narrowly scoped, synthetically labeled, or insufficiently quality-controlled.

  • Method

    The paper constructs SynPref-40M through human verification and human-preference-guided LLM curation, using 26 million curated pairs to train eight Skywork-Reward-V2 models.

  • Results

    Across seven major reward-model benchmarks, Skywork-Reward-V2 achieves state-of-the-art performance, with its 8B model outperforming all existing open reward models.

  • Takeaways & Limitations

    The results support combining human annotation quality with LLM scalability when curating large preference datasets for open reward models.

  • Takeaways & Limitations

    RewardBench gains do not consistently transfer to other benchmarks, and the model-modification finding is limited to Gemma-2-27B variants.

Abstract

from arXiv · show

Despite the critical role of reward models (RMs) in Reinforcement Learning from Human Feedback (RLHF), current state-of-the-art open RMs perform poorly on most existing evaluation benchmarks, failing to capture nuanced human preferences. We hypothesize that this brittleness stems primarily from limitations in preference datasets, which are often narrowly scoped, synthetically labeled, or lack rigorous quality control. To address these challenges, we present SynPref-40M, a large-scale preference dataset comprising 40 million preference pairs. To enable data curation at scale, we design a human-AI synergistic two-stage pipeline that leverages the complementary strengths of human annotation quality and AI scalability. In this pipeline, humans provide verified annotations, while LLMs perform automatic curation based on human guidance. Training on this preference mixture, we introduce Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B parameters, trained on a carefully curated subset of 26 million preference pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile across a wide range of capabilities, including alignment with human preferences, objective correctness, safety, resistance to stylistic biases, and best-of-N scaling. These reward models achieve state-of-the-art performance across seven major reward model benchmarks, outperform generative reward models, and demonstrate strong downstream performance. Ablation studies confirm that effectiveness stems not only from data scale but also from high-quality curation. The Skywork-Reward-V2 series represents substantial progress in open reward models, demonstrating how human-AI curation synergy can unlock significantly higher data quality.

1 Introduction

The paper targets preference-data quality and scale as barriers to open reward-model development. It introduces a human-AI curation pipeline, SynPref-40M, and Skywork-Reward-V2, which achieves strong results across seven reward-model benchmarks.

  • Reward models are critical to RLHF and LLM post-training, motivating improved preference modeling.
  • The work introduces SynPref-40M, a 40-million-pair preference mixture with 26 million carefully curated pairs.
  • The 8B model achieves state-of-the-art performance and outperforms existing open reward models across seven major benchmarks.
  • Skywork-Reward-V2 comprises eight reward models ranging from 0.6B to 8B parameters.
  • The two-stage pipeline combines stringent human verification with human-preference-guided LLM judges for scalable curation.

2 The brittleness of current open reward models

The paper argues that single-benchmark reward-model evaluation is unreliable because gains on RewardBench do not consistently transfer to other benchmarks. It therefore evaluates multiple dimensions and identifies scope limits for some model-modification findings.

  • A comprehensive suite spans human preference, correctness, best-of-N, content variation, and style-bias evaluation dimensions.
  • RewardBench improvements from ∼80 to 90+ do not consistently improve performance on other benchmarks.Performance elsewhere may improve, worsen, or remain approximately unchanged.
  • For Gemma-2-27B variants, alternative losses or model modifications do not consistently improve average performance.The authors limit this observation to that specific model series rather than generalizing to all modifications.
  • Among the top 20 RewardBench models, 16 share a base model or highly similar training data, indicating stagnant progress since September 2024.

3 Scaling preference data curation via human-guided AI feedback

The pipeline combines iterative human verification with human-guided LLM annotation to curate preference data from a small seed set to tens of millions of pairs. Reward-model feedback drives adaptive retrieval and consistency-based filtering before large-scale data is retained.

  • 3.2 Stage 1: small-scale human-in-the-loop curation: Stage 1 iteratively collects unverified pairs and produces silver LLM labels and gold human labels to train and diagnose the reward model.Eight iterations accumulate roughly 1M preference pairs.
  • 3.2 Stage 1: small-scale human-in-the-loop curation: Human annotators verify seed preferences under a strict protocol using task-specific guidelines and permitted external tools, while retaining responsibility for final judgments.Full reliance on LLMs for labeling is prohibited.
  • 3.2 Stage 1: small-scale human-in-the-loop curation: The Stage 1 reward model is trained on silver data and selected by validation accuracy on gold data, then used to identify misclassified gold examples for adaptive retrieval.The retrieved samples support further annotation aligned with human preferences.
  • 3.2 Stage 1: small-scale human-in-the-loop curation: LLM judgments are guided by retrieved, similar gold examples with human labels, making preference-aware annotation scalable while conditioning decisions on verified data.The retrieved examples are inserted as few-shot demonstrations.
  • 3.3 Stage 2: large-scale automatic curation of preference data in the wild: Stage 2 scales to tens of millions of in-the-wild pairs, retaining consistent examples and sending mismatches to LLM annotation without human verification.Filtering uses agreement with reward models and human-preference-guided LLM judgments.

4 Experimental results

Skywork-Reward-V2 is evaluated across seven benchmarks and multiple dimensions, including correctness, best-of-N scaling, style-bias resistance, and instruction following. Ablations show that curated data quality, human curation, adaptive retrieval, and richer annotation information materially improve reward-model performance.

  • 4.1 Reward model training: Skywork-Reward-V2 models use Llama 3.1, Llama 3.2, and Qwen3 backbones with no more than 8B parameters and a 16K-token context length.Final training uses a large global batch size, while early experiments use one epoch and batch size 256.
  • 4.1 Reward model training: Across seven benchmarks, Skywork-Reward-V2 models outperform larger 70B models and generative reward models.The authors interpret this as evidence that SynPref-40M captures diverse preferences across multiple dimensions.
  • 4.2 A comprehensive evaluation of the Skywork-Reward-V2 series: Skywork-Reward-V2 performs strongly on correctness, best-of-N helpfulness and harmlessness, and five-task PPE Correctness scaling curves.All eight models outperform GPT-4o on the RMB best-of-N evaluation by up to 20 points, while the Llama-3.1-8B variants show superior scaling.
  • 4.2 A comprehensive evaluation of the Skywork-Reward-V2 series: Skywork-Reward-V2 models outperform baselines on RM-Bench while maintaining smaller performance gaps across stylistic conditions.The authors report that the gap shrinks rapidly as model size increases.
  • 4.3 Ablation studies on data quantity and quality: Uncurated data scaling yields minimal gains, whereas curated data produces consistent improvements as more pairs are added.Adding 12 million uncurated pairs during Stage 2 fails to surpass the initial seed model.
  • 4.3 Ablation studies on data quantity and quality: Flipping discarded preference labels produces consistent gains, showing that curation can identify data whose original chosen-rejected order is inconsistent with judgments.The corrected curve includes filtered data plus pairs with flipped labels.
  • 4.3 Ablation studies on data quantity and quality: Training on 1.8% of a 16M preference mixture surpasses the previous 70B open reward-model state of the art at 8B scale.The subset contains roughly 290K samples.
  • 4.4.1 Pipeline-level ablations: Human curation yields a 2.3-point relative gain over the seed model, compared with only a 0.1-point gain from simple LLM curation.Adaptive retrieval adds a further 0.9-point gain over raw LLM annotation in the human-curation variant.

5 Conclusion

The paper introduces SynPref-40M and Skywork-Reward-V2 through a human-AI curation pipeline that combines verified human supervision with scalable LLM judging. The resulting reward models perform strongly across diverse capabilities and seven major benchmarks.

  • SynPref-40M contains 40 million preference pairs, including 26 million curated pairs, and supports eight Skywork-Reward-V2 models from 0.6B to 8B parameters.
  • The curation pipeline combines human supervision for quality with human-preference-guided LLM judges for scalability.
  • Skywork-Reward-V2 demonstrates strong capabilities in general human preferences, objective correctness, style-bias resistance, safety, and best-of-N scaling.
  • The 1.7B variant surpasses the best existing 70B reward model on average, while the 8B variant ranks first across all seven benchmarks among open reward models.
  • The evaluation includes Bradley-Terry-trained models and compares them across seven major reward model benchmarks.

B Limitations

The paper identifies limitations involving preference diversity, data curation uncertainty, annotation constraints, benchmark interpretation, and evaluation coverage. These boundaries qualify how broadly the results should be interpreted.

  • A single reward model may not fully capture diverse and conflicting human preferences, especially for subjective prompts without one correct answer.
  • The improvement observed from re-annotated discarded data is purely empirical and was not further verified because of budget constraints.
  • LLM and reward-model filtering is not fully interpretable, so discarded pairs may contain useful data alongside biases and modeling errors.
  • The structured annotation protocol promotes consistency but reduces flexibility and may not fully capture minority preferences.
  • RewardBench scores may not directly correlate with downstream performance, motivating evaluation across additional benchmarks.
  • The evaluation suite spans human preference, correctness, best-of-N, style-bias robustness, and judge-oriented correctness benchmarks, but benchmark evidence remains scope-dependent.

C.2 Full evaluation results

The full evaluation reports SynPref-40M-trained reward models against seven benchmarks and finds higher average performance than previous models. The appendix also documents dataset composition, filtering, decontamination, and licensing practices.

  • Across all seven benchmarks, reward models trained on SynPref-40M outperform previous models on average.
  • The evaluation table categorizes compared systems as Bradley-Terry reward models, LLM-as-Judges, and generative reward models.
  • The dataset is globally deduplicated, decontaminated against evaluation prompts, and aggregated only from redistributable public sources.
  • Over 99% of SynPref-40M samples contain synthetic prompts and/or responses, while the remainder are written by real humans.
  • Information-seeking and coding tasks account for over 70% of the dataset, followed by advice-seeking, mathematics, and creative writing.
  • 73.8% of pairs are classified as low controversiality and 74.8% as objective.
  • Over 95% of preference pairs are in English, with roughly 2.5% in Chinese and the remainder in other languages.

D.3 Privacy and PII analysis

The paper analyzes privacy risks and preference inconsistency in SynPref-40M while describing metadata, adaptive retrieval, and consistency filtering used in curation. The reported checks find minimal genuine PII and improved agreement from the hybrid approach.

  • Approximately 0.07% of the dataset, or 28K samples, were flagged by the initial LLM-based PII scan.
  • Most flagged samples contained indirect identifiers rather than direct personal identifiers, and human checks found no genuine PII among score-0 or score-1 samples.
  • Reward models maintained near-perfect accuracy on pairs with PII removed or replaced by neutral placeholders, indicating limited reliance on PII.
  • The pipeline uses metadata to isolate risky regions, adaptive retrieval to target unstable examples, and dual-RM filtering to resolve contradictory signals.
  • The hybrid LLM-human-adaptive approach achieved 93% agreement on objective pairs and 84% on subjective pairs, exceeding pure human and pure LLM annotation.
  • The curated data and consistency filters are compatible with explicitly intransitive preference models, although the paper uses Bradley-Terry models for simplicity and comparability.

E.4 Lessons learned from verifying and annotating human preferences in-the-wild

The annotation experience shows that preference labeling is difficult for both humans and LLMs: reasoning tasks favor capable LLMs, while human preferences remain ambiguous and context-dependent. Clear protocols and verified, well-structured preference data substantially improve reward-model performance.

  • Lessons from annotation: LLMs are more efficient and reliable than human annotators for reasoning tasks such as mathematics and coding.The authors attribute this to human annotators’ limited expertise across all such problems and recommend advanced reasoning models.
  • Lessons from annotation: Different LLMs exhibit strong annotation bias, making human guidance important for curation quality.The authors observed this bias in early annotations and use human preferences to guide later annotation.
  • Lessons from annotation: Human preference pairs can be ambiguous, subjective, or context-dependent even for trained annotators.Uncertainty arises from tone, informativeness, safety expectations, and individual annotator biases.
  • Lessons from annotation: Mixing pairs with opposite preferences can make reward models learn spurious correlations such as pure text format.The authors also report improved pure-LLM annotation quality when preference specifications are provided.
  • Lessons from annotation: Well-structured, verified data guided by clear annotation protocols improves reward-model performance across all evaluation benchmarks.The authors hypothesize that benchmark preference pairs impose substantially stronger construction requirements, but report no quantitative results here.
  • Operational experience: Fewer than 20 trained annotators produced approximately 2,000 to 3,000 annotations per week, with each pair taking 10 seconds to 5 minutes.Authors participated voluntarily in the seed and Stage 1 annotation process.

F.1 What our scaling results reveal about cost

The scaling results indicate that curation quality, rather than raw preference-pair volume, determines cost-effective reward-model gains. A small human-verified core can guide automated expansion, while scaling should be monitored against target performance and diminishing returns.

  • Scaling findings: Adding 12M uncurated preference pairs yields almost no performance gain, whereas Stage 1 + Stage 2 curation produces steadily improving performance.The largest gains occur in Stage 2.
  • Scaling findings: 1.8% of the 16M curated mixture, or 290K pairs, surpasses the previous open SOTA 70B RM at the 8B scale.This result supports using carefully curated hundreds of thousands of pairs instead of tens of millions of new human labels.
  • Stage-wise costs: Fewer than 500K pairs receive full human verification, while the remaining tens of millions are curated automatically in Stage 2.Human effort comprises only a couple percent of the final training pool but drives most performance gains.
  • Budget planning: Practitioners can estimate a required curated fraction from the scaling curve after defining a target average score across six main benchmarks.Higher targets require larger fractions, with diminishing returns.
  • Stage-wise costs: Total curation cost is approximately B ≈ cH · |Dgold| + cL1 · |Dsilver| + cL2 · |DStage2|.The gold set dominates quality, while automatic stages dominate quantity because cH ≫ cL1 ≥ cL2 and |Dgold| ≪ |DStage2|.
  • Budget planning: Given a fixed budget, allocate a gold-labeling budget first, then use the remainder to scale Stage 2 while trading off curated volume against LLM quality.The authors suggest roughly O(10^5) carefully selected gold pairs can train strong gold and Stage-1 reward models.
  • Budget planning: Stop spending when incremental gains per additional curated million pairs fall below a user-defined threshold, such as 0.3 average benchmark points.The proposed monitoring signals are RM benchmark scores and downstream best-of-N curves.
  • Budget example: A $50K allocation can produce a curated dataset exceeding 1M pairs, sufficient to reach or exceed the target performance.The example balances gold labels, silver labels, and Stage 2 curation across the available budget.

G Training details and hyperparameters

The training setup follows prior hyperparameter choices while adjusting learning rates by model size during development and standardizing final-run settings.

  • Optimization: Development learning rates are 1e-6 for 8B models and 4e-6 for all other model sizes.Models use a global batch size of 256, linear learning-rate decay, one warmup epoch, and a maximum token length of 16,384.
  • Optimization: Final training runs use a learning rate of 3e-6.

H.1 Existing (uncurated) preference datasets are inadequate

Existing preference datasets do not match the performance of the curated mixture, and simply combining more datasets does not solve the scaling problem.

  • Benchmark comparison: 69.4 is the average score from olmo-2-0425-1b-preference-mix alone, while combining all datasets reaches only 68.9.The all-dataset combination is 0.5 points lower than the single-dataset result.
  • Benchmark comparison: None of the evaluated single preference datasets or their combination outperforms the curated mixture.The comparison supports the conclusion that preference scaling cannot be achieved by simply accumulating preference pairs.

H.2 Downstream RLHF evaluation and human evaluation

The evaluation examines downstream RLHF performance, human agreement, and whether Stage 2 filtering aligns with human judgments. The reported results support strong human alignment and reduced concern that dual-RM filtering merely imposes model-specific biases.

  • Downstream RLHF evaluation: The evaluation compares policies trained with the proposed reward model against previous state-of-the-art reward models of similar size.The supplied passage introduces the comparison but does not report the resulting policy scores.
  • Human evaluation: Human evaluation on an internal hold-out benchmark finds significantly higher human preference agreement for models trained on the curated preference mixture.
  • Filtering evaluation: The dual-RM filtering mechanism achieves the highest agreement for both kept and flipped pairs, indicating reduced rather than amplified systematic biases.
  • Filtering evaluation: 84% and 88% agreement rates are reported for the Gold RM on kept and flipped pairs, respectively, exceeding the baseline models.The Skywork-Reward-Llama-3.1-8B baseline reaches 69% and 57%, while combining two baseline models reaches 71% and 60%.

H.5 Baseline experiment: LLM + RM filtering without human guidance

The baseline experiments test whether LLM filtering without human guidance can match the full curation recipe. They also compare an LLM-as-a-Judge ensemble with trained reward models and report deployment-related ethical boundaries.

  • Baseline experiment: 83% versus 74–75% shows that human-guided curation outperforms LLM-plus-best-RM filtering by an 8–9 point gap at iteration 8.The LLM-only baseline plateaus around 74–75%, while the full recipe reaches 83%.
  • LLM-as-a-Judge comparison: 88.6 average versus 86.0 shows that the final trained reward model outperforms the LLM-as-a-Judge ensemble overall.The ensemble performs better on PPE Correctness and JudgeBench, while the trained reward model leads on most other benchmarks.
  • LLM-as-a-Judge comparison: 89.6% on PPE Correctness and 87.8% on JudgeBench are the ensemble’s strongest reported benchmark results.These benchmarks focus on objective correctness and code-related tasks.
  • Ethical considerations: The dataset may retain demographic or cultural biases from its annotator pool and underlying data sources despite validation and consistency checks.
  • Ethical considerations: Downstream deployment requires use-case-specific safety evaluations, safeguards, and continued monitoring.
Loading 2507.01352v3…