PaperlayerView exact revision on arXiv

Source-linked AI summary

RM-OPD: On-Policy Distillation for Generative Reward Models

Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu

arXiv:2610.20174v2cs.LGcs.CL

TL;DR

Mix-RL’s sparse outcome rewards and cross-domain interference constrain generalist reward models’ performance ceiling. RM-OPD trains domain specialists and consolidates them into a student with routed, dense on-policy supervision and domain reweighting. Across benchmarks and base models, it consistently outperforms Mix-RL, with an emergent teacher-surpassing effect.

  • Problem

    Sparse outcome rewards and cross-domain interference constrain the performance ceiling of mixed-domain training for generalist reward models.

    Source: §1 · p. 2, §8 · p. 9

  • Method

    RM-OPD trains domain-specialized teachers and consolidates them into a student using routed on-policy distillation, domain contribution reweighting, and criteria context distillation.

    Source: §1 · p. 2

  • Results

    Across eight benchmarks and two base families, RM-OPD improves macro average over matched Mix-RL by 4.60–5.55% relative and exhibits a teacher-surpassing effect across diverse domains.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The authors present RM-OPD as a promising paradigm for developing robust generalist reward models.

    Source: §8 · p. 10

    Hard routing on inherently mixed-domain prompts may cause the student to ignore relevant secondary criteria.

    Source: C LIMITATIONS · p. 20

Abstract

from arXiv · show

Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.

1 INTRODUCTION

Generalist reward models must combine distinct domain-specific evaluation capabilities, but Mix-RL’s sparse outcome rewards and cross-domain interference leave gaps relative to isolated specialists. RM-OPD separates specialist training from integration and improves benchmark performance over matched Mix-RL.

1 INTRODUCTION

  • Mix-RL’s sparse outcome rewards and cross-domain interference produce a seesaw effect and a consistent performance gap relative to single-domain specialists.The domains require different evaluation skills, including logical verification, functional correctness, and intent-sensitive judgment.Source: §1 · pp. 1–2
  • RM-OPD first trains domain-specialized teachers, then integrates them through routed on-policy distillation, domain contribution reweighting, and optional criteria context distillation.Routing supplies dense signed token-level supervision on student-generated critiques; reweighting aligns domain update shares with target allocations.Source: §1 · pp. 2–3
  • Across eight benchmarks and two base families, RM-OPD improves macro average over matched Mix-RL by 4.60–5.55% relative and surpasses specialists in some domains.Figure 1 reports faster convergence and performance that matches or exceeds specialists across Safety, Math, and Code.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 RELATED WORK

Prior GenRM reinforcement learning often optimizes verdict correctness, while on-policy distillation offers token-level supervision on student-generated trajectories. RM-OPD applies multi-teacher OPD to reward modeling, a use the authors describe as previously unstudied.

Reinforcement Learning for Generative Reward Models

  • RLVR approaches train GenRMs on verdict correctness, but verdict-only rewards leave intermediate reasoning tokens and heterogeneous domain criteria weakly supervised (Chen et al., 2026; Liu et al., 2025b; Wang et al., 2026b;a).RM-OPD instead uses dense token-level supervision from domain-specialized experts on student judgment rollouts.Source: §2 · p. 3

On-Policy Distillation

  • On-policy distillation aligns teacher guidance with student-generated rollouts, with prior work extending it to multi-teacher routing and other settings (Agarwal et al., 2024; Gao et al., 2026).The cited extensions also include flow-matching generators, speculative drafters, and self-evaluation.Source: §2 · p. 3
  • The authors describe RM-OPD as the first multi-teacher OPD framework for generative reward models, consolidating domain-specialized experts into one generalist GRM.They state that, to their knowledge, prior work had not systematically applied OPD to reward modeling.Source: §2 · p. 3

3 PRELIMINARIES

The paper formulates pairwise generative reward modeling as producing a textual judgment with reasoning followed by a preference verdict. It describes on-policy distillation as aligning the student and teacher at prefixes from the student’s own generated trajectories.

Generative reward models

  • A generative reward model produces a textual judgment for a pair of candidate responses, with a reasoning segment followed by a verdict predicting the preferred response.The reasoning may contain a rubric, solution check, or critique, and judgment accuracy measures agreement between the predicted and ground-truth preference.Source: §3 · p. 3
  • The paper focuses on math, code, and safety, which require distinct evaluation criteria and judgment skills.These include step-level logical verification, execution-grounded edge-case analysis, and intent-sensitive judgment.Source: §3 · p. 3

On-policy distillation

  • On-policy distillation samples student-generated judgments and aligns student and teacher distributions at the visited prefixes using a reverse-KL objective.The global objective minimizes expected local discrepancies over the student’s active generation distribution.Source: §3 · pp. 3–4

4 MOTIVATION

The motivation identifies cross-domain gradient interference and sparse outcome rewards as sources of Mix-RL capability gaps. It proposes on-policy, domain-specific, process-dense supervision, realized in RM-OPD by integrating specialized teachers.

4.1 WHY DOES MIX-RL LEAVE CAPABILITY GAPS?

  • When gradients for two domains are negatively aligned, optimizing one can counteract progress on the other; accumulated interference can increase a domain’s loss despite improving the joint objective.The authors connect repeated antagonistic updates across domains to the seesaw effect and capability erosion.Source: §4.1 · p. 4
  • Sparse outcome rewards depend only on the final verdict, spreading the same trajectory-level advantage across the critique without identifying which reasoning steps caused success or failure.The authors state that critiques with different reasoning quality can remain indistinguishable when they reach the same verdict, leaving Mix-RL with capability gaps relative to isolated training.Source: §4.1 · p. 4

4.2 HOW TO RAISE THE COEXISTENCE CEILING?

  • Raising the coexistence ceiling requires on-policy supervision of student-induced states, coherent domain-specific criteria, and process-dense guidance reaching the critique.OPD supplies token-level guidance over student trajectories while independent domain experts provide coherent supervision.Source: §4.2 · p. 4
  • RM-OPD answers the consolidation question with a two-stage framework separating domain expertise acquisition from capability integration.Independent domain-specialized reward teachers supervise a shared student on its own judgment trajectories.Source: §4.2 · pp. 4–5

5 RM-OPD

RM-OPD separates specialist acquisition from integration: single-domain GRPO teachers supervise one student on its own judgments through routed distillation, domain reweighting, and criteria context.

5 RM-OPD

  • RM-OPD trains domain-specialized reward teachers independently, then uses them to supervise one student that performs multi-domain judgment.The teachers are frozen, and the student is initialized from the shared base model.Source: §5 · p. 5, Figure 2 · p. 5, §5.1 · p. 5

5.1 ROUTED ON-POLICY DISTILLATION

  • Hard routing sends each student-sampled judgment to its input-matched teacher, which supervises every prefix of that judgment.The student samples judgments from a fixed policy snapshot.Source: §5.1 · p. 5
  • Positive teacher–student log-ratios favor teacher-preferred tokens, while negative values penalize tokens overestimated by the student.The signal provides dense guidance throughout both critique and verdict.Source: §5.1 · p. 5

5.2 DOMAIN CONTRIBUTION REWEIGHTING

  • Domain contribution reweighting replaces length-dependent token shares with target coefficients for domain-mean losses, separating domain allocation from judgment length.Each represented domain's mean loss receives its normalized target coefficient.Source: §5.2 · p. 6

5.3 CRITERIA CONTEXT DISTILLATION

  • Criteria context distillation conditions the matched teacher on domain-specific evaluation principles so the student can internalize them without external prompts.Setting the criteria context to empty recovers unguided distillation.Source: §5.3 · p. 6

6 EXPERIMENTS

Across two base families, RM-OPD matches or narrows gaps with specialist teachers and surpasses them on some reasoning benchmarks, while outperforming mixed-domain GRM baselines. Experiments compare it with specialists, Mix-RL, and public models across multiple benchmarks.

6 EXPERIMENTS

  • The evaluation spans safety, mathematics, and code, comparing RM-OPD with domain specialists, Mix-RL, and strong open-source GRMs.The study evaluates eight public benchmark suites and 16 domain-specific subsets using two base models and 74k cleaned pairwise preferences.Source: §6 · p. 6, §6.1 · p. 7

6.2 MAIN RESULTS

  • RM-OPD matches or narrows specialist-teacher gaps across domains and surpasses specialists on RewardBench 2 and RM-Bench Safety.MixRL-GRPO consistently lags the single-domain teachers across both Qwen2.5 and DeepSeek-R1-Distill bases.Source: §6.2 · pp. 7–8
  • 75.41 average accuracy across 16 subsets for Qwen RM-OPD exceeds RISE-Judge (74.12), RM-NLHF (71.76), RM-R1 (71.22), and RRM (68.88).DeepSeek RM-OPD also improves the overall macro average across four core benchmarks over matched RM-R1 and RM-NLHF variants.Source: §6.2 · p. 8

7 ANALYSIS

The analysis examines multi-domain capability coexistence, component ablations, and out-of-domain generalization. It reports that RM-OPD better preserves capabilities as domains are added, while ablations and OOD results reveal domain-specific effects and evaluation-setting variation.

7.1 MULTI-DOMAIN SCALING AND COEXISTENCE

  • With safety added, Mix-RL’s mathematical accuracy falls 2.96 points below its single-domain baseline and RM-OPD, while RM-OPD’s math capability drops only 0.97.Across scaling, the full RM-OPD model also outperforms single-domain specialists in Safety and Overall average.Source: Figure 3 · p. 8, §7.1 · pp. 8–9

7.2 ABLATION STUDY

  • Removing DCR reduces code and safety accuracy by 3.43 and 2.48 points, respectively, while mathematics remains nearly unchanged.The ablation reports domain means over 12 benchmark subsets.Source: §7.2 · p. 9, Table 3 · p. 9
  • With criteria context distillation, overall accuracy improves by 1.70 points as average response length rises from 3,488 to 4,395 tokens.The reported accuracy gains are largest in safety, followed by code and math.Source: Figure 4 · p. 9, §7.2 · p. 9

7.3 OUT OF DOMAIN GENERALIZATION

  • RM-OPD reaches 60.98% on RM-Bench Chat/Normal and exceeds Mix-RL by 5.30 and 6.75 points on two RMB-BoN subsets, but trails Mix-RL on pairwise creative writing.For pairwise creative writing, RM-OPD scores 64.96% versus Mix-RL’s 67.72%.Source: Figure 4 · p. 9, §7.3 · p. 9

8 CONCLUSION

The paper concludes that sparse rewards and cross-domain interference constrain mixed-domain RL for general-purpose GRMs, and presents RM-OPD as a two-stage alternative. It reports consistent gains over Mix-RL and a teacher-surpassing effect across diverse domains.

8 CONCLUSION

  • RM-OPD separates specialist acquisition from capability integration, combining routed on-policy distillation, domain contribution reweighting, and criteria context distillation.The framework uses dense token-level supervision to consolidate heterogeneous evaluation capabilities into a generalist GRM.Source: §8 · pp. 9–10
  • Experiments show that RM-OPD consistently outperforms Mix-RL baselines and exhibits an emergent teacher-surpassing effect across diverse domains.Source: §8 · p. 10
  • The authors present RM-OPD as a promising paradigm for developing robust generalist reward models.Source: §8 · p. 10

A THEORETICAL ANALYSIS AND IMPLEMENTATION DETAILS

The appendix details how RM-OPD scores student-generated judgments with domain teachers, aligns criteria-conditioned teacher scores, and applies token-level policy updates and domain weighting. It also describes training configuration and preference-data construction.

A THEORETICAL ANALYSIS AND IMPLEMENTATION DETAILS

  • Each prompt receives G student-generated judgments scored only by its domain teacher, whose causal token log-probabilities are masked to include response positions and exclude prompts and padding.Prompt quotas use largest-remainder allocation, recycling shuffled domain pools when exhausted.Source: A THEORETICAL ANALYSIS AND… · p. 15
  • Criteria context inserts domain criteria for teacher scoring, then removes inserted positions so teacher log-probabilities align with student-sampled tokens.Criteria context can be enabled independently by domain; the default enables it for all three.Source: A THEORETICAL ANALYSIS AND… · p. 15
  • The local reverse-KL derivation yields a sampled-token policy-gradient update using fixed student rollouts and teacher scores, with detached advantages and clipped signed log-ratios.The actor recomputes current student log-probabilities while sampled prefixes and teacher scores remain fixed.Source: A THEORETICAL ANALYSIS AND… · p. 15
  • Domain contribution weights are computed after dynamic micro-batching and reused across a judgment’s tokens; single-domain micro-batches receive unit weights.The appendix also specifies PPO clipping, dual clipping for negative advantages, and a token-mean distillation surrogate without task-reward, entropy, or reference-model KL terms.Source: A THEORETICAL ANALYSIS AND… · pp. 15–16, §B.1 · p. 16

B.1 TRAINING CONFIGURATION

  • Student distillation uses synchronous VeRL training with FSDP and vLLM across two eight-GPU H800 nodes, while criteria-conditioned teachers receive a 256-token prefix budget.The student objective uses the token-mean policy-gradient surrogate; outcome and format rewards are logged for monitoring.Source: §B.1 · p. 16, Table 4 · p. 16

B.2 TRAINING DATASET

  • The student pool combines the three specialist-training datasets, globally shuffles them with seed 42, and uses prompt-level domain quotas of 1:1:1; distillation uses teacher probabilities, not offline traces or critique SFT data.The pool includes mathematics, code, and safety preference data; A/B order is randomized approximately evenly.Source: §B.2 · pp. 16–17, Table 5 · p. 17

B.3 EVALUATION BENCHMARKS AND METRICS

The evaluation covers public reward-model and judge suites across mathematics, code, and safety, with benchmark-specific pairwise scoring and macro-averaged accuracy. The appendix also identifies the model families and baselines used for comparison.

B.3 EVALUATION BENCHMARKS AND METRICS

  • The final evaluation contains 35,184 official samples across 16 benchmark subsets spanning mathematics, code, and safety.The eight suites include RewardBench, RewardBench 2, RM-Bench, RMB and RMB-BoN, JudgeBench, PKU-SafeRLHF, and Themis-CodeRewardBench.Source: §B.3 · p. 17, Table 6 · p. 17
  • Evaluation converts datasets to a pairwise judge format, expands selected best-of-N examples into comparisons, and counts an official sample correct only when every expanded comparison is correct.RM-Bench retains its official 3 × 3 grid and reports hard, normal, and easy accuracy from corresponding cells.Source: §B.3 · p. 18
  • The final Average is the macro average over 16 benchmark subsets, excluding Overall columns; each benchmark’s Overall is the unweighted arithmetic mean of its displayed domains.Source: §B.3 · p. 18
  • Comparisons include each base family’s MixRL-GRPO, RL Teacher, and RM-OPD models alongside public reward-model baselines; MixRL-GRPO and RM-OPD share initialization and training data.This controls for initialization and data while contrasting sparse outcome optimization with routed on-policy distillation.Source: §B.3 · p. 18

B.4 ADDITIONAL BENCHMARK RESULTS

The additional results cover expanded benchmark comparisons, domain-wise scaling configurations, and a detailed domain contribution reweighting ablation.

B.4 ADDITIONAL BENCHMARK RESULTS

  • Table 7 adds RMB, RMB-BoN, PKU-SafeRLHF, and Themis to the evaluation and reports an average across all 16 benchmark subsets.Its caption defines Overall as the unweighted mean of displayed domains within each benchmark and Delta as RM-OPD minus MixRL-GRPO.Source: §B.4 · p. 18, Table 7 · p. 18

B.5 DOMAIN-WISE RESULTS FOR MULTI-DOMAIN SCALING

  • Table 8 compares three configurations that expand the training-domain set while retaining the same evaluation subsets.Domain means average four fixed subsets; the active RM-OPD specialist pool follows each configuration’s training-domain set.Source: §B.5 · p. 19, Table 8 · p. 19

B.6 DETAILED RESULTS FOR DOMAIN CONTRIBUTION REWEIGHTING

  • Table 9 gives a detailed DCR ablation across 12 fixed subsets, with changes measured relative to the configuration with DCR.Green and red values denote increases and decreases relative to w/ DCR.Source: §B.6 · p. 19, Table 9 · p. 20

C LIMITATIONS

This section outlines RM-OPD’s limitations, including untested scaling beyond 7B models, reliance on accessible and aligned teacher token probabilities, and risks from hard routing and teacher errors. It illustrates contextual safety assessment and explicit mathematical verification, including a case where RM-OPD catches a false factorization rewarded by MixRL-GRPO and selects Chatbot A.

C LIMITATIONS

  • Experiments cover only 7B models, leaving RM-OPD’s scaling to frontier models of 70B+ parameters unvalidated.The authors note that base capabilities and distillation dynamics may differ at larger scales.Source: C LIMITATIONS · p. 20
  • Dense token-level supervision requires specialist teachers’ causal token probabilities and shared teacher–student tokenizers and judgment formats.This excludes closed-source API judges that expose only text or scalar scores; cross-family expert pools would require token or representation alignment.Source: C LIMITATIONS · p. 20
  • Hard routing by domain metadata may miss secondary criteria in mixed-domain prompts, while confidently wrong teachers can transfer errors to the student.The authors identify soft routing for ambiguous prompts and extensions to process reward models and multimodal settings as future directions.Source: C LIMITATIONS · p. 20

E.1 SAFETY: CONTEXTUAL HARM ASSESSMENT

  • In the chess case, RM-OPD interprets “destroy my opponent” as conventional competitive language and selects standard chess advice rather than a refusal.The passage contrasts this contextual reading with MixRL-GRPO’s literal harm interpretation and identifies the case as an example of distinguishing genuine harmful intent from false-positive refusal triggers.Source: §E.1 · pp. 23–24

E.2 MATHEMATICS: EXPLICIT VERIFICATION

  • The mathematics case describes RM-OPD reconstructing the algebraic identity before issuing its verdict, rather than rewarding a shorter derivation.The case is framed as testing explicit verification of a decisive factorization.Source: §E.2 · p. 25
  • Substituting x = r/s gives x^2 + x + 1 = 0, whose solutions are the non-1 complex cube roots of unity, so (r/s)^3 = 1.The passages contrast this result with Chatbot B’s answer of -1, which follows from its incorrect factorization.Source: §E.2 · p. 28
  • RM-OPD identifies Chatbot B’s invalid factorization of r^2 + rs + s^2 as (r + s)^2 and selects Chatbot A by checking the decisive symbolic transformation.MixRL-GRPO rewards B’s shorter derivation without expanding the factorization; the case illustrates transfer of the mathematics expert’s criterion to verify each decisive step before judging concision.Source: §E.2 · p. 28

E.3 CODE: EXECUTION AND BOUNDARY TESTING

In this code case, RM-OPD checks the supplied boundary example against actual function behavior and locates a return-logic bug that makes Chatbot B fail digits(1).

E.3 CODE: EXECUTION AND BOUNDARY TESTING

  • For the boundary case digits(1), Chatbot B incorrectly returns 0 because multiplying the initial product by odd digit 1 leaves it at 1; Chatbot A correctly handles n=1.This violates the first test in the prompt.Source: §E.3 · p. 32
  • B uses product != 1 to infer that an odd digit was found, but multiplying the odd digit 1 leaves the product at 1 and triggers an incorrect 0.The product value 1 can mean either that no odd digits were found or that an odd digit contributed a product of 1.Source: §E.3 · pp. 31–32
  • RM-OPD traces the supplied tests and final conditional to locate the bug, while MixRL-GRPO rewards the shorter implementation without executing this boundary case.The code expert’s transferred criterion is to establish behavior on concrete tests before rewarding compactness.Source: §E.3 · p. 32

Related papers

Loading 2610.20174v2…