PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Trained Policy Discriminators are General Reward Models

Shihan Dou, Shichun Liu, Yuming Yang, Yicheng Zou, Yunhua Zhou, Shuhao Xing, Chenhao Huang, Qiming Ge, Demin Song, Haijun Lv, Songyang Gao, Chengqi Lv, Enyu Zhou, Honglin Guo, Zhiheng Xi, Wenwei Zhang, Qipeng Guo, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Tao Gui, Kai Chen

arXiv:2507.05197v2cs.CLcs.LG

TL;DR

Preference-based reward models face scalability and generalization challenges, and rule-based verifiers are hard to extend to open-ended tasks. POLAR pre-trains reward models to discriminate policies and reports stronger preference accuracy than larger SOTA baselines, alongside generalizable RLHF rewards.

  • Problem

    Preference-based reward models face scalability and generalization challenges, while rule-based verifiers are difficult to extend to open-ended domains such as writing and translation.

    Source: §1 · pp. 1–2

  • Method

    POLAR pre-trains an RM to distinguish trajectories from identical versus different policies, then fine-tunes it on human-ranked trajectories to align evaluations with human criteria.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    5.8% average improvement: POLAR-7B surpasses the SOTA 72B-parameter WorldPM in preference evaluation tasks, while POLAR RMs also provide reliable and generalizable RLHF reward signals.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors conclude that POLAR expands the potential applicability of RL algorithms such as RFT and offers promise for stronger, more generalizable reward models.

    Source: §1 · p. 3

    The paper identifies preference-based training as constrained by subjective judgments, costly data collection, and weak generalization beyond the training distribution.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Based on this conceptual insight, we propose a scalable pre-training method named Policy Discriminative Learning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between one policy and an arbitrary target policy, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. POLAR also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36% to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. The impressive performance, strong generalization, and scaling properties suggest that POLAR is a promising direction for developing general and strong reward models.

1 Introduction

Preference-based reward models face scalability and generalization challenges, while rule-based verifiers are difficult to extend to open-ended tasks. POLAR reframes reward modeling as criterion-agnostic policy discrimination, then adapts the pretrained model to human criteria.

1 Introduction

  • Traditional preference-based RMs rely on manually defined absolute criteria, while rule-based verifiers are difficult to extend to open-ended domains such as writing and translation.Preference methods also face scalability and generalization challenges, including difficulty acquiring large volumes of high-quality labeled pairs.Source: §1 · pp. 1–2
  • POLAR defines an RM as a policy discriminator that scores candidate policies by their similarity to a chosen target policy, providing a criterion-agnostic reward signal for policy training.Because target policies can be chosen arbitrarily, the objective does not rely on manually defined preferences.Source: §1 · p. 2
  • POLAR pre-trains an RM to distinguish trajectories from identical versus different policies, then fine-tunes it using a small set of reference and ranked candidate trajectories.The reference trajectories may be human-annotated or derived from high-performing LLMs.Source: §1 · p. 2
  • 5.8% average improvement: POLAR-7B surpasses the SOTA 72B-parameter WorldPM in preference evaluation tasks.The paper also reports that POLAR RMs achieve higher preference accuracy and better generalization than considerably larger RMs.Source: §1 · pp. 2–3
  • The paper presents POLAR RMs with parameter scales from 1.8 to 7 billion and reports higher preference accuracy and better generalization than considerably larger RMs.The authors identify scaling laws as a promising direction for stronger and more generalizable reward models.Source: §1 · pp. 2–3

2 Related Work

Related work describes limitations of preference-based and rule-based reward signals, reviews scaling-law research, and distinguishes POLAR from earlier discriminator-based methods. POLAR combines large-scale pre-training with policy discrimination and human-criteria fine-tuning.

2 Related Work

  • Preference-based reward models face scarce labeled pairs and limited out-of-distribution generalization, while rule-based verifiers struggle with tasks that predefined rules cannot readily verify.The paper also notes that generative reward models can reflect the biases and preferences of their underlying models.Source: §2 · p. 3
  • Prior scaling-law work studies power-law relationships between computation and performance, including applications to language models, other fields, and preference-based reward-model over-optimization.Source: §2 · p. 3
  • Unlike GAIL and DIRECT, which distinguish policy trajectories from expert or high-quality ones for mainly single-task imitation, POLAR pre-trains on large-scale unsupervised policy data.POLAR learns criterion-agnostic differences among policies rather than relying on expert demonstrations.Source: §2 · p. 3
  • POLAR's two-stage overview pairs criterion-agnostic policy-difference pre-training with human-ranked trajectory fine-tuning to align evaluations with human standards.In RFT, the fine-tuned RM compares candidate trajectories with human-preferred references to guide policy training.Source: Figure 2 · p. 4

3 Method

The method reframes reward modeling as policy discrimination, then combines criterion-agnostic pre-training with supervised fine-tuning on human rankings. Its motivation is that absolute preference modeling depends on subjective labels and can be costly to scale and weak beyond its training distribution.

3.1 Preliminary #1: Preference-based Reward Models and Their Limitations

  • Preference-based reward models encode absolute judgments, but subjective and inconsistent labels, costly data collection, and weak generalization beyond the training distribution constrain them.These limitations motivate exploring more scalable reward-modeling paradigms.Source: §3.1 · p. 4

3.2 Preliminary #2: Policy Optimization in Reinforcement Learning

  • Under KL constraints, the reward model encodes an operator mapping the initial policy toward its optimal form through reinforcement learning.The policy optimization objective includes a regularization coefficient controlling the KL-divergence penalty relative to the initial policy.Source: §3.2 · pp. 4–5

3.3 Unsupervised Reward Pre-training via Distributional Alignment

  • The pre-training rationale is that arbitrary criteria can impose opposing response rankings, making exhaustive enumeration computationally infeasible and theoretically ill-posed.The proposed unsupervised paradigm provides a criterion-agnostic initialization by measuring distributional differences between a desired target policy and an initial policy.Source: §3.3 · p. 5
  • POLAR pre-trains the reward model to quantify differences between training and target policies, assigning higher rewards when their difference is smaller.The pre-training objective uses Bradley-Terry loss to distinguish trajectories from the same policy from those generated by different policies.Source: §3.3 · p. 5

3.4 Supervised Fine-tuning with Human Criteria

  • Supervised fine-tuning adapts the pre-trained discriminator to human criteria by ranking three trajectories from the same policy and training against their relative judgments.The stage bridges criterion-agnostic discrimination and human-aligned evaluation, with the authors describing it as requiring minimal annotation overhead.Source: §3.4 · p. 6

4 Experiments

Across experiments, POLAR RMs outperform traditional baselines in preference prediction and RLHF, while validation loss decreases predictably with model size and compute. Ablations examine the roles of pre-training, reference trajectories, and reinforcement fine-tuning.

4.1 Model and Training Details

  • The evaluated POLAR-1.8B and POLAR-7B models use an InternLM2.5-aligned autoregressive Transformer and reward head, initialized from a pre-trained InternLM2.5 model for one additional pre-training epoch.The model input combines a prompt, reference trajectory, and candidate trajectory.Source: §4.1 · p. 6, §4 · p. 6

4.2 Performance in Preference Evaluation

  • On STEM preference prediction, POLAR-1.8B and POLAR-7B surpass the best baseline by over 24.9 and 26.2 percentage points, respectively, and POLAR consistently outperforms baseline reward models across most tasks.Source: §4.2 · p. 7
  • Standard preference datasets may inadequately reflect reward models’ full capabilities and nuanced distinctions, motivating more representative evaluation frameworks.POLAR-1.8B and POLAR-7B perform similarly in preference evaluation despite differing in downstream RL performance.Source: §4.2 · p. 7

4.3 Performance in RLHF Training

  • 9.0% average improvement over the initial policy and 6.7% over the policy model optimized by WorldPM-72B-UltraFeedback were achieved by LLaMa-3.1 fine-tuned with POLAR-7B across benchmarks.The RLHF experiments compare reward models across 20 benchmarks; POLAR scores candidate trajectories against references during RFT.Source: §4.3 · pp. 7–8, Table 1 · p. 8

4.4 Scaling Laws in POLAR

  • Validation loss follows power laws with model size and compute, with fit R2 values of 0.9886 and 0.9912, respectively (Figure 4).The experiments train five reward models from 50M to 1B parameters using up to 54B training tokens.Source: Figure 4 · p. 9, §4.4 · p. 9

4.5 Ablation Study

  • Fine-tuning-only reward models remain competitive in preference evaluation but decline substantially in RLHF, highlighting the contribution of POLAR pre-training.The comparison uses otherwise identical experimental settings.Source: §4.5 · p. 9, Table 2 · p. 10
  • With the same prompt-reference data, policies trained by POLAR RFT outperform those trained by straightforward SFT.The authors report that RL exploits the training data more robustly and that POLAR RMs provide supervision during policy optimization.Source: Table 3 · p. 10, §4.5 · p. 10
  • Without pre-training, policies fine-tuned with a reward model using references outperform those using a model without references across nearly all tasks.The authors attribute this to references clarifying reward-model evaluation and providing clearer, more stable policy-training directions.Source: §4.5 · pp. 9–10

5 Conclusion

POLAR reformulates reward modeling as policy discrimination and uses measured policy differences as reward signals for policy optimization. Evaluations report strong preference accuracy, RLHF generalization, and power-law scaling.

5 Conclusion

  • POLAR RMs provide reward signals by distinguishing diverse policies and quantifying their differences for subsequent policy optimization.Source: §5 · p. 10
  • Despite their small parameter counts, POLAR RMs surpass larger SOTA baselines in preference accuracy across multiple tasks.Source: §5 · p. 10
  • POLAR shows effectiveness in RLHF across diverse and potentially out-of-domain scenarios, alongside clear power-law scaling behavior.Source: §5 · p. 10

A Limitations and Future Work

The discussion identifies reference costs, POLAR’s pre-training focus, and the computational expense of policy sampling as limitations or areas for further study. It also describes POLAR as promising and reports no negative societal impacts.

A Limitations and Future Work

  • Reference trajectories improve reward-signal accuracy and reliability, but generating and curating them increases annotation costs.Multiple references per prompt could further reduce reward variance, particularly in open-ended scenarios.Source: A Limitations and Future Work · p. 20
  • Cross-prompt references are a preliminary exploration; systematic evaluation and integration with multiple-reference settings remain future work.The proposed strategy uses trajectories from prompts with similar underlying reasoning structures.Source: A Limitations and Future Work · p. 20
  • POLAR primarily addresses pre-training; integrating it with test-time scaling remains future work.Source: A Limitations and Future Work · p. 20
  • Scaling POLAR data requires substantial policy sampling, likely at considerably higher computational cost than traditional LLM data preparation.Source: A Limitations and Future Work · p. 20

B Broader Impact

  • The authors describe POLAR as a promising direction for general reward models and report that they see no negative societal impacts.Source: B Broader Impact · p. 20

C License For Artifacts and Data Consent

The paper describes the licenses and research-use status of the models and datasets used in its evaluations.

C License For Artifacts and Data Consent

D.1.1 Pre-training Data

POLAR’s pre-training data comprises synthetic trajectories generated from a diverse policy pool. The process uses paired policy samples and sampling controls to shape the dataset.

D.1.1 Pre-training Data

  • The synthetic pre-training corpus pairs prompts with trajectories generated by a diverse collection of LLMs.Prefixes are sampled from pre-training text; policies generate trajectories up to 4096 tokens, with human-instruction prompts included in a small proportion.Source: §D.1.1 · p. 21
  • The policy pool includes 53 base and 53 chat models, plus 78 intermediate InternLM3-8B-base checkpoints; the trajectory corpus totals 3.60T tokens.The models were selected for variation in pre-training data versions and generated-response diversity.Source: §D.1.1 · p. 21
  • Table 4 summarizes the composition of the pre-training dataset mixture.Source: Table 4 · p. 21, §D.1.1 · p. 21
  • To address repetitive loops and excessively long outputs, the authors truncate repeated sections and cap trajectories at 4096 tokens; sampling uses temperature 1.0, top-p 0.9, and top-k 50.The authors report that low temperatures provide insufficient contrast, while high temperatures introduce biases in policy characterization.Source: §D.1.1 · p. 21

D.1.2 Hyperparemeter and Implementations

POLAR models are initialized from pretrained InternLM2.5 models and tuned using data-driven hyperparameter scaling; supervised fine-tuning uses ranked three-output examples and one training epoch.

D.1.2 Hyperparemeter and Implementations

  • POLAR-1.8B and POLAR-7B start from pretrained InternLM2.5 models and receive one additional epoch of POLAR pretraining.The reported learning rates and batch sizes are 1.4e-5 and 1940 for POLAR-1.8B, and 1.67e-5 and 4343 for POLAR-7B.Source: §D.1.2 · pp. 21–22
  • Hyperparameters are selected through scaling experiments relating optimal learning rate and batch size to model size and pretraining-data scales.The experiments vary model and reward-model data scales, fit loss curves, and illustrate learning-rate and batch-size scaling in Figures 5 and 6.Source: §D.1.2 · p. 21, Figure 5 · p. 23, Figure 6 · p. 23
  • The pretraining trajectory corpus draws on base and chat models, mainly from the Llama, Qwen, Yi, ChatGLM, DeepSeek, Gemma, and InternLM series.Table 5 lists the models used to generate trajectories for pretraining data.Source: Table 5 · p. 22

D.2.1 Supervised Fine-tuning Data

  • The supervised fine-tuning set contains 150K prompts, each paired with three candidate outputs ranked by human annotators.For open-source prompts, a third candidate is generated with a selected state-of-the-art LLM; the top two outputs form a positive pair and the lower two a negative pair.Source: §D.2.1 · pp. 22–23

D.2.2 Hyperparemeter and Implementations

  • Supervised fine-tuning runs for one epoch with batch size 320, using learning rates of 1e-5 for 1.8B and 2e-5 for 7B.The w/o PT and w/o PT & Ref ablations use the same fine-tuning data and settings, with different pretraining and reference-trajectory conditions.Source: §D.2.2 · p. 23

D.3 Compute and Data Comparisons

The comparison highlights POLAR’s additional unsupervised pretraining compute alongside substantially lower labeled-data requirements than traditional preference-based reward modeling.

D.3 Compute and Data Comparisons

  • POLAR uses 150K labeled samples versus tens of millions for other baselines, reducing annotation costs while adding an unsupervised pretraining compute phase.Table 6 compares pretraining data, labeled data, and compute across reward-modeling approaches; undisclosed quantities are marked as missing.Source: §D.3 · pp. 23–24, Table 6 · p. 24

E.1 Baseline Reward Models

  • InternLM2-7B-Reward and InternLM2-20B-Reward are trained on over 2.4 million preference samples, including human-annotated and AI-generated data.They use InternLM2-Chat-7B-SFT and InternLM2-Chat-20B-SFT as their respective backbones.Source: §E.1 · p. 24
  • Skywork-Reward-Gemma-2-27B and Skywork-Reward-Llama-3.1-8B use 80K curated preference pairs, and the Gemma model had state-of-the-art results in prior studies.Their backbones are gemma-2-27b-it and Meta-Llama-3.1-8B-Instruct, respectively.Source: §E.1 · p. 24
  • WorldPM-72B-UltraFeedback is first trained on 15M public-forum preference pairs and then fine-tuned on 100K UltraFeedback pairs.The passage describes it as a recently released state-of-the-art reward model.Source: §E.1 · p. 24

E.2 Details of Preference Evaluation

Preference evaluation uses ranked candidate trajectories and excludes training overlap, while RLHF experiments compare reward models under PPO and the pretraining loss curves show stable convergence.

E.2 Details of Preference Evaluation

  • Preference evaluation uses three candidate trajectories per real-user query, excludes training-set overlap, and tests whether the model selects the better of two outputs given the top-ranked output as reference.Candidate trajectories are generated with listed or state-of-the-art LLMs, and human annotators rank them by quality.Source: §E.2 · p. 24

E.3 Details of RLHF Training and Evaluation

  • RLHF training uses 1.25 million prompt-reference pairs without human annotation, with prompts kept separate from those used for SFT.Reference trajectories are generated by selected state-of-the-art LLMs.Source: §E.3 · pp. 24–25
  • PPO experiments average results across random seeds and test baselines both with and without reference trajectories, reporting each baseline’s best setting.Standard scoring without references consistently performs better for baselines, which the authors attribute as likely closer to their original training objectives.Source: §E.3 · p. 25
  • PPO uses actor and critic learning rates of 1e−6 and 1e−5, respectively, with training and rollout batches of 1024.Most policy runs use 32 H800 GPUs for about 48 hours; Qwen2.5-32B-Instruct runs use 64 GPUs for about 72 hours.Source: §E.3 · p. 25

F.1 Pre-Training Loss Curve

  • Figure 7 shows steadily declining pretraining loss for POLAR-1.8B and POLAR-7B, indicating stable convergence.The figure plots loss against training steps.Source: §F.1 · p. 25, Figure 7 · p. 25

F.2 Reference-free vs. Reference-included Evaluation

POLAR outperforms traditional reward models across evaluation categories, including reference-based comparisons, and its policy-discrimination approach exceeds embedding similarity baselines. The supplementary results also test fine-tuning, illustrate reward scoring, and report RLHF comparisons and ablations.

F.2 Reference-free vs. Reference-included Evaluation

  • POLAR consistently achieves superior performance across categories, while providing a reference does not necessarily improve traditional reward-model accuracy and can reduce it.Table 7 compares POLAR-1.8B and two state-of-the-art baselines in reference-free and reference-included settings.Source: §F.2 · p. 25, Table 7 · p. 26

F.3 Embedding Similarity Baselines

  • POLAR significantly outperforms embedding-similarity preference prediction, measuring consistency between policies rather than surface-level response similarity.The comparison includes OpenAI’s Text-Embedding-3-Large; the authors characterize the policy-level measure as better aligned with RL objectives.Source: §F.3 · p. 26, Table 7 · p. 26

F.4 Impact of POLAR Supervised Fine-tuning

  • Without human-judgment alignment, the pre-trained-only model performs poorly on preference prediction; a small amount of labeled fine-tuning aligns its scoring with human preferences.The comparison evaluates pre-trained-only POLAR-1.8B against the fully trained model.Source: §F.4 · p. 26

F.5 Case Study

  • The case studies illustrate POLAR-7B scoring reasoning correctness despite phrasing variation and evaluating summary content against prompt and reference criteria.The summarization example considers multiple evaluation criteria rather than relying only on semantic similarity.Source: §F.5 · p. 26, Figure 8 · p. 27, Figure 9 · p. 28

F.6 Additional Results of POLAR Evaluation

  • Tables 9–12 report RLHF comparisons across four policy backbones, while Tables 13–15 cover ablations and RFT-versus-SFT results.Table 15 reports that reward-model-guided RFT is more effective than SFT and that its policy improvement cannot be attributed solely to shared prompt-reference data.Source: §F.6 · p. 29, Table 9 · p. 31, Table 15 · p. 34
  • Figure 10 shows the reference-included prompt format for baseline reward models, with baseline performance reported from the better of reference-included and traditional scoring.The prompt gives the assistant access to the reference when generating a response for baseline scoring.Source: Figure 10 · p. 29, §F.6 · p. 29

Related papers

Loading 2507.05197v2…