PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldPM: Scaling Human Preference Modeling

Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, An Yang, Binyuan Hui, Dayiheng Liu, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Bowen Yu, Jingren Zhou, Junyang Lin

arXiv:2505.10527v2cs.CL

TL;DR

Scaling laws are well established for language modeling, but manually annotated preference data constrains investigation of whether they extend to preference modeling. The paper trains WorldPM on public-forum preferences and finds improvements across preference fine-tuning and RLHF evaluations.

  • Problem

    The paper asks whether language-model scaling laws extend to preference modeling, where manually annotated preference data constrains scale.

    Source: §1 · p. 2

  • Method

    WorldPM uses public-forum preference pairs to study preference-model scaling and serves as a foundation for preference fine-tuning.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Results

    WorldPM broadly improves preference-fine-tuning generalization, with gains exceeding 5% on many key subtasks and 4% to 8% in in-house evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Preference-model pre-training's scaling properties merit direct investigation, while subjective-domain improvement calls for better annotation strategies and preference-modeling frameworks.

    Source: §5 · p. 13

    Noisy subjective preference annotations can introduce uncertainty into evaluation.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations.

1 Introduction

The paper proposes WorldPM to study scaling in preference modeling using public-forum preferences. It reports distinct scaling behavior across evaluation domains and improvements when WorldPM is used for preference fine-tuning and RLHF.

1 Introduction

  • 15 million preference samples from StackExchange, Reddit, and Quora informed training across 1.5B- to 72B-parameter models; StackExchange was selected for its generalization and cross-domain transfer.The evaluation used 12 test sets grouped into adversarial, objective, and subjective categories.Source: §1 · p. 2
  • Adversarial test losses decreased by a power law as WorldPM improved at detecting intentional errors and responses that were irrelevant or incomplete despite being well-written.The authors suggest large-scale training can mitigate subtle deficiencies in discriminative models.Source: §1 · p. 2
  • Objective evaluations showed emergent scaling: larger models reduced test losses by a power law across more benchmarks, while smaller models showed limited or no such improvement.The paper frames objective preference modeling as a challenging task for which larger models elicit preferences for objective knowledge.Source: §1 · p. 2
  • Subjective evaluations showed no scaling trend; the authors link some lower scores to style preferences that diverge from WorldPM's increasing style neutrality.Examples include preferences for longer or richer responses.Source: §1 · p. 2
  • Performance gains exceeded 5% on many key subtasks when WorldPM supported preference fine-tuning across datasets of 7K, 100K, and 800K samples.Improvements were more pronounced with limited preference data, and some subtasks gained over 10%.Source: §1 · p. 2
  • 4% to 8% gains in in-house evaluations followed WorldPM integration into a training pipeline using preference fine-tuning and GRPO.The pipeline also improved on open-sourced alignment evaluations compared with baselines without WorldPM.Source: §1 · p. 2

2 Background & Related Work

This section outlines reward modeling and reinforcement learning in RLHF, presents Best-of-N sampling as an alignment alternative, and situates WorldPM among preference-model pre-training work.

2.1 Reinforcement Learning from Human Feedback (RLHF)

  • RLHF has two stages: a reward model learns from pairwise preferences, then guides policy optimization that maximizes rewards while constraining divergence from a reference policy.The reward model uses Bradley-Terry preference probabilities and minimizes the negative log-likelihood of preference data.Source: §2.1 · p. 3

2.2 Best-of-N Sampling as an Alignment Alternative

  • Best-of-N samples candidate responses, ranks them with a reward model, and selects the highest-scoring response instead of using RL optimization.Prior studies report competitive performance with RLHF while avoiding optimization instability.Source: §2.2 · p. 3

2.3 Preference Model Pre-training

  • Earlier preference-model pre-training improved fine-tuning sample efficiency and transferred across datasets, while this work investigates pre-training scalability and generalization directly to test sets.The earlier work used 5K human preference samples and in-distribution evaluation; this paper uses larger datasets and broader benchmarks.Source: §2.3 · pp. 3–4

3 Modeling World Preference

WorldPM is trained on forum preference pairs and evaluated across preference benchmarks, with StackExchange selected for its cross-source generalization. Results vary by evaluation type: adversarial and objective tasks scale differently from subjective evaluations, while style control clarifies downstream alignment performance.

3.1.1 Data Collection

  • Forum posts, responses, and votes from StackExchange, Reddit, and Quora form preference pairs by ordering two responses according to their net votes.For each post, the response with more net votes is labeled preferred.Source: §3.1.1 · p. 4

3.1.2 Training Methods

  • Training uses the Bradley–Terry loss on preference pairs, with a 3e-6 learning rate and 10K batch size across model sizes.The study trains Qwen2.5 models from 1.5B to 72B parameters on approximately 15M StackExchange pairs.Source: §3.1.2 · p. 4, §3.2 · p. 6

3.1.3 Evaluation Methods

  • Evaluation groups benchmark capabilities into adversarial flaw detection, objective judgments with ground-truth answers, and subjective human or AI preferences.The evaluation uses BT loss and includes Alpaca Eval and Arena Hard for downstream alignment assessment.Source: §3.1.3 · pp. 4–5

3.1.4 Training Data Source Selection

  • StackExchange outperformed Reddit and Quora, matched or surpassed ArmoRM on named evaluations, and Math-board training performed similarly across human-preference benchmarks.Cross-source evaluation also found StackExchange had the best generalization among the forum sources and could represent different sources.Source: §3.1.4 · p. 5, Table 1 · p. 5, Figure 2 · p. 5

3.2 Scaling Trends

  • Adversarial test losses decrease by a power law across three test sets; objective test-loss scaling appears across more benchmarks at larger scales, while subjective test losses do not decrease with increased training data.The 1.5B model fails to generalize on any objective test set, while 72B shows consistent scaling across all aspects; subjective losses quickly converge or even increase, with no notable distinction between 7B and 72B.Source: §3.2 · pp. 6–7

3.3.1 Style-Content Separation in Evaluation

  • Style control leaves expert HelpSteer2 results stable but changes crowdsourced and GPT4 evaluations; the controlled-versus-uncontrolled gap widens with model and training scale.After style control, 72B outperforms 7B, though it slightly underperforms on GPT4 evaluations.Source: Figure 4 · p. 7, §3.3.1 · p. 8

3.3.2 Style Effects on Training Dynamics

  • As scale increases, model predictions become less correlated with style, yet remain more style-correlated than human labels; length correlates more strongly than markdown use.WorldPM learns Chosen Longer examples early and gradually improves on Chosen Shorter examples while maintaining the former performance.Source: §3.3.2 · pp. 8–9

3.4 Alignment Performance

  • Arena Hard performance continued improving through 10M training samples, while Alpaca Eval peaked at 0.6M and direct reward-model tests saturated around 3M.Style control reveals clearer 72B-versus-7B differences on Alpaca Eval; without it, scores strongly correlate with response length.Source: §3.4 · p. 9, Figure 6 · p. 10

4 Preference Fine-Tuning based on WorldPM

WorldPM initialization improves preference fine-tuning across human-preference datasets and domains, with gains varying by evaluation type and model scale. The paper also tests how training scale affects those gains and integrates WorldPM into an RLHF pipeline.

4.1 Experimental Setup

  • The experiments use HelpSteer2 (about 7K pairs), UltraFeedback (about 100K pairs), and RLHFlow (about 800K points), alongside RM benchmarks and best-of-64 alignment tests.The setup evaluates 7B and 72B models; HelpSteer2 uses batch size 128, while the other datasets use 512.Source: §4.1 · p. 10

4.2 Evaluation Results

  • Across human-preference datasets of different scales, WorldPM initialization improves fine-tuning generalization across diverse domains.The authors conclude that this makes large-scale WorldPM a useful preliminary step before training on costly human-preference annotations.Source: §4.2 · p. 12
  • PPE-Human and RMB-Helpful improve, while RewardBench-Chat decreases slightly but remains above 97% accuracy.The authors describe RewardBench-Chat as saturated and therefore limited as an indicator.Source: §4.2 · p. 12
  • Objective metrics improve broadly, with larger gains for 72B than 7B; 72B improves across objective domains while 7B struggles to generalize.The reported metrics include PPE, RM-Bench code and math, and RewardBench reasoning.Source: §4.2 · p. 12
  • Adversarial results improve on initially weaker datasets such as HelpSteer2, while other datasets show minimal change or slight decreases.WorldPM itself already reaches around 90% on RM-Bench Chat and OffsetBias; the authors suggest post-fine-tuning decreases may reflect simple features being disrupted.Source: §4.2 · p. 12
  • The 72B model shows mixed safety-evaluation trends without consistent improvement, possibly reflecting defensive behavior against pseudo-harmful content.Source: §4.2 · p. 12
  • Most Arena Hard and Alpaca Eval metrics improve, but RLHFlow Alpaca Eval declines for 7B in raw results and 72B after style control.The authors suggest RLHFlow’s larger scale and quality may make WorldPM’s benefits less pronounced.Source: §4.2 · p. 12

4.3 The Impact of WorldPM Training Scale on Fine-tuning

  • At 5M WorldPM samples, fine-tuning gains emerge; larger WorldPM scales consistently perform better, especially on objective PPE metrics.Subjective metrics fluctuate substantially, though the scale comparison uses 5M, 10M, and 15M-sample 72B checkpoints against a no-WorldPM baseline.Source: §4.3 · p. 12, Figure 7 · p. 13

5 Dicussion

The discussion frames WorldPM as an investigation of reward-model pretraining scalability, finding clearer scaling in objective and adversarial domains than in subjective evaluation. It argues that subjective progress depends on better annotation and preference-modeling approaches, while reward-model integration with other signals remains open.

5 Dicussion

  • The paper shifts the focus from reward-model pretraining’s downstream RLHF benefits to its inherent scalability properties.Source: §5 · p. 13
  • Objective and adversarial performance improves with model size and data volume, but subjective evaluations show no scaling trend.The authors identify style preference as a potential limiting factor: WorldPM mitigates it during training, while subjective human evaluations can retain it and misassess performance.Source: §5 · p. 13
  • The authors note that objective dimensions increasingly use rule-based, retrieval-augmented, or reference-based rewards, reducing reliance on reward models in those areas.Source: §5 · p. 13
  • How to integrate reward models with other reward signals remains open; in subjective domains, the authors call for better annotations and preference frameworks beyond surface-level preferences.They argue that minimizing dependence on conscious judgments from a small number of humans is important for scalable preference modeling.Source: §5 · p. 13

6 Limitaiton

The limitations section identifies a modest StackExchange preference corpus and incomplete control of subjective-evaluation biases. Its forum analysis also describes long-tailed data distributions that constrain which preference pairs can be formed.

6 Limitaiton

  • The study uses 15M StackExchange preference pairs, modest beside trillion-token next-token pretraining, while other forums and social media remain potential data sources.Source: §6 · p. 13
  • Subjective-evaluation controls cover only response length and Markdown formatting, leaving emotional preferences, cultural tendencies, and intertwined factors difficult to isolate.Source: §6 · p. 14

A.1 Forum Data Analysis

  • Forum data are long-tailed: most replies are under 1K characters, receive fewer than 10 upvotes, and posts have fewer than 6 replies; many posts lack the two differently rated replies needed for a preference pair.Source: §A.1 · p. 18
  • Reply lengths are relatively uniform across upvote counts but positively correlated with upvotes; the authors hypothesize that moderation filters out very short, low-information replies.Source: §A.1 · p. 18

A.2 Preference Data Construction

Preference pairs are derived from StackExchange voting, with author acceptance counted as an additional upvote. Tests find similar performance across vote-gap buckets and forum categories, supporting transfer across the sampled domains.

A.2 Preference Data Construction

  • Preference pairs form when replies to the same post have different net-vote scores, calculated as upvotes minus downvotes with author acceptance counted as an extra upvote.The paper treats the reply with the higher score as preferred.Source: §A.2 · p. 18
  • Performance is similar across the 1–2, 3–5, 6–10, and 11+ score-gap buckets, with no bucket showing significant variation.Table 5 summarizes the score-gap analysis.Source: §A.2 · p. 18, Table 5 · p. 18
  • Models trained on StackOverflow, Math StackExchange, and other StackExchange sections show no significant performance variation, and topic resampling yields similar results.The sections reflect the forum’s major topic areas and its many smaller sections.Source: §A.2 · p. 19, Table 6 · p. 19
  • The authors interpret consistent performance across discussion topics as evidence that preferences within the StackExchange community are domain-agnostic.This conclusion is bounded to users within the same forum community.Source: §A.2 · p. 19

A.3 Experimental Settings

The appendix details training and evaluation choices, then reports scaling patterns across test sets. Safety results vary by benchmark and model size, while HelpSteer2 dimensions show distinct correlations with response length.

A.3 Experimental Settings

  • Training uses Qwen2.5 models from 1.5B to 72B parameters; larger batches improve performance through 40K, while learning rates from 1e-6 to 1e-5 yield stable results.The selected settings are batch size 10K and learning rate 3e-6; the authors cite limited data and computational efficiency for the batch-size choice.Source: §A.3 · p. 19

A.4 Evaluation Settings

  • Evaluation uses PairWise for RMB and original, unmodified responses for RM-Bench; Claude-3-Haiku samples are excluded from MBPP-Plus because answer length and correctness correlate at -0.6.The authors identify this correlation as a severe style bias in that test set.Source: §A.4 · p. 20

A.5 Evaluator Prompts

B.1 Scaling Trends Across All Test Sets

  • HelpSteer2’s Helpfulness, Correctness, and Coherence correlate strongly, as do Complexity and Verbosity; the latter metrics associate with response length and show increasing loss as WorldPM scales.Figure 13 plots loss trajectories across evaluation benchmarks against training scale and model size.Source: §B.1 · p. 20, Figure 13 · p. 23
  • RewardBench-Safety accuracy exceeds 90% for 7B and 72B models after stabilizing, while RMB-Harmlessness declines for 1.5B, saturates for 7B, and rises for 72B.The authors report declining and saturating loss for detecting genuinely harmful content, but rising loss for pseudo-harmful queries later in training.Source: §B.1 · p. 23

B.2 Preference Fine-tuning Evaluation Without Style Control

Without style control, WorldPM initialization is compared with initialization without WorldPM for preference-model fine-tuning at 7B and 72B. The reported findings align with those from style-controlled evaluations.

B.2 Preference Fine-tuning Evaluation Without Style Control

  • Uncontrolled fine-tuning results for 7B and 72B models compare initialization with and without WorldPM, and the findings align with the style-controlled versions.Tables 8 and 9 cover evaluation categories for the two model sizes.Source: §B.2 · pp. 23–24, Table 8 · p. 24, Table 9 · p. 24

C.1 Ablation Study on Style Control Factors

The ablation finds that length matters more than markdown formatting in subjective evaluation, while controlling both factors improves style control. Across test sets, style-control effects vary by domain and training scale.

C.1 Ablation Study on Style Control Factors

  • Markdown formatting has less influence than length on subjective evaluation, and controlling both features enables more effective style control.Figure 14 compares controlling length alone, markdown alone, or both together.Source: §C.1 · pp. 24–25, Figure 14 · p. 25

C.2 Effect of Style Control Across Test Set Evaluation

  • Under style control, HelpSteer2’s Complexity and Verbosity change while their trends remain stable; objective and robustness performance gaps stay constant or narrow with scale, unlike widening subjective gaps.RM-Bench-Safety-Refuse has high accuracy even without style control; other safety benchmarks show declining accuracy later in training, potentially reflecting increased defense against pseudo-harmful queries.Source: §C.2 · p. 25

D.1 Impact of RM Filtering on Different Data Sources

Reward-model filtering helps Quora more than StackExchange, but gains on StackExchange remain limited. The analysis also warns that filtering can align retained data with a reward model’s discrimination patterns.

D.1 Impact of RM Filtering on Different Data Sources

  • RM-based filtering improves Quora performance across benchmarks, while StackExchange shows notable gains only on PPE-Objective, consistent with its stricter quality control.The filtering retained instances where Qwen-2.5-72B-RM scored the chosen response above the rejected response.Source: §D.1 · p. 25, Table 10 · p. 26

D.2 Impact of Various Reward Model Filtering

  • Independent and joint reward-model filtering yield modest benchmark improvements on StackExchange, with no significant gains even when both models are used.The tested reward models were Athene-70B-RM and Qwen-2.5-72B-RM.Source: §D.2 · p. 28, Table 11 · p. 28

D.3 Potential Biases in RM Filtering

  • Filtered-out data, evaluated using inverse scores, closely matches filtered data, suggesting similar modeling capability but opposing preference directions and a risk of inheriting RM discrimination patterns.Filtered data performs relatively consistently across score intervals, with only the 0–0.7 interval marginally worse.Source: §D.3 · p. 28, Table 12 · p. 28

E Case Study

The case study illustrates WorldPM identifying errors in some PPE-Human examples and presents StackExchange training examples as information-rich material for modeling preferences.

E Case Study

  • In PPE-Human examples, WorldPM identifies incorrect chosen answers in Figures 17–18, while Figure 19 illustrates low-quality, uninformative human data that can interfere with evaluation.Source: E Case Study · p. 28, Figure 17 · p. 29, Figure 18 · p. 30
  • StackExchange training examples span coding, mathematics, freelancing, and physics questions, and the authors say such data often carry rich information for learning complex representations.Source: E Case Study · pp. 28, 29, 31, 32, 33
Loading 2505.10527v2…