PaperlayerView exact revision on arXiv

Source-linked AI summary

The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement

Xiaobo Wang, Tong Wu, Min Tang, Jiaqi Li, Qi Liu, Zilong Zheng

arXiv:2605.30888v1cs.CL

TL;DR

Reward-model improvement is limited by costly, unreliable preference data and offline training that becomes stale as policies evolve. SAVE uses value-anchored on-policy feedback to train RMs continuously, and experiments report gains across six benchmarks, RL algorithms, policy backbones, and downstream policy evaluations.

  • Problem

    Reward-model training lacks scalable, diverse, reliable supervision because human or external-judge preference data is costly and offline data becomes stale during policy evolution.

  • Method

    SAVE uses a prompt-specific value anchor to compute RM advantages, filter ambiguous on-policy responses, and update the reward head with a contrastive objective.

  • Results

    SAVE improves average RM accuracy from 76.0 to 77.3 across six benchmarks and consistently improves results across GRPO, RLOO, GSPO, and two policy backbones.

  • Takeaways & Limitations

    On-policy feedback can serve as effective supervision for continuously improving reward models and strengthening downstream RLHF policies.

  • Takeaways & Limitations

    Experiments use 3B–4B policy models and a 3B RM, rely mainly on automatic benchmarks and LLM judges, and add computation through multiple rollouts and RM updates.

Abstract

from arXiv · show

Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.

1 Introduction

SAVE addresses costly, unreliable preference supervision and stale offline RM data by using evolving on-policy responses as self-supervised feedback. It combines value anchoring, adaptive filtering, contrastive RM updates, and a minimax interpretation, with gains across six benchmarks and downstream RLHF.

  • Motivation: RM construction is constrained by expensive preference annotation, reliance on external judges, and offline data that becomes stale as policies evolve.These bottlenecks limit scalable, reliable RM improvement during RL training.
  • Method: SAVE uses a prompt-specific value head to compute response-level RM advantages, filter ambiguous samples, and train the reward head contrastively.The retained responses are partitioned into positive and negative advantage subsets for self-supervised feedback.
  • Theory: SAVE formalizes policy optimization as adaptive generation of challenging samples while the RM minimizes worst-case ranking and calibration losses.This provides a reward-model-centric minimax interpretation of on-policy feedback.
  • Results: 76.0 to 77.3 average accuracy, with best scores on all six reward model benchmarks, validates SAVE across diverse evaluations.The benchmarks are RewardBench, RewardBench 2, RM-Bench, PPE Preference, PPE Correctness, and JudgeBench.
  • Results: SAVE improves downstream RLHF policy performance, raising AlpacaEval 2 length-controlled win rate from 51.68% to 54.24% and Arena-Hard-v2.0 win rate from 30.2% to 33.9%.The improvements are consistent across GRPO, RLOO, GSPO, and two policy backbones.

2 Related Work

Prior work addresses reward-model weaknesses through scaling, robustness, calibration, and joint reward-policy optimization. SAVE fits a broader shift toward online joint evolution while using on-policy feedback for RM improvement.

  • Reward Modeling for Preference Alignment: Scalar RMs trained on fixed offline data can suffer over-optimization and reward hacking when policies move beyond their training support.Existing responses include scaling data and capacity, robustness-oriented architectures, and auxiliary calibration signals.
  • Reinforcement Learning Algorithms for LLMs: Value baselines and grouped-rollout estimators support advantage estimation in both classical policy gradients and recent critic-free LLM RLHF methods.Examples include RLOO, GRPO, REINFORCE++, DAPO, and GSPO.
  • Joint Optimized Optimization of Reward and Policy Models: Recent methods jointly update reward and policy models through iterative preference optimization, hidden-state adaptation, or alternating implicit process reward modeling.Self-Rewarding LMs, R2M, PRIME, and iStar exemplify this direction.

3 Preliminaries

RLHF optimizes a language policy with a learned scalar reward while constraining divergence from a reference policy. The reward model learns from pairwise preferences, and value or group-based baselines support advantage estimation.

  • RLHF Objective: RLHF optimizes an autoregressive policy to maximize scalar reward while remaining close to a reference policy through KL regularization.The coefficient β controls the KL regularization strength.
  • Reward Modeling: The learned reward model is trained on pairwise preferences with the Bradley-Terry objective, where yw is preferred over yl.The logistic sigmoid σ(·) appears in the preference model.
  • Value Baselines: For a policy, the prompt value is the expected response reward and can serve as a reward baseline.This value-based framing underlies response-level advantage estimation.
  • Advantage Estimation: Recent LLM RLHF pipelines often use grouped estimators such as RLOO, GRPO, and GSPO instead of training a separate critic.Value baselines reduce policy-gradient variance without bias.

4 On-Policy Feedback for Reward Model Self-Supervised Improvement

SAVE turns on-policy RL rollouts into RM supervision using a prompt-specific value anchor and contrastive feedback. Its theory connects policy optimization with challenging-example generation under stated local conditions.

  • Framework: SAVE introduces a value-anchored RM objective and reframes RL from merely maximizing reward toward generating informative feedback.The framework is designed for self-supervised RM training during on-policy RL.
  • Value-Anchored Reward Model: The value-anchored RM computes response-level advantages by subtracting a prompt-specific expected-reward estimate from each scalar response reward.The value head shares the RM backbone and estimates the expected outcome for the prompt.
  • On-Policy Feedback: Sampled response groups are partitioned by advantage sign, while prompts lacking both positive and negative subsets are discarded because they provide no contrastive signal.This creates positive and negative feedback sets for RM training.
  • Value Calibration: The value anchor is calibrated against the sampled group’s mean reward using MSE, which estimates the policy-conditioned value with variance decreasing as O(1/K).The group size K determines the estimator’s variance.
  • Minimax Interpretation: Under Conditions (C1)–(C3), the standard policy-gradient direction is also a local ascent direction for the joint objective, so reward maximization generates hard on-policy examples.The interpretation requires RM sensitivity to epistemic errors and a KL term that does not dominate the effect.
  • Minimax Interpretation: The resulting minimax view has the policy expose RM weaknesses while the RM reduces ranking and calibration errors on the induced samples.Value-anchored feedback stochastically approximates this minimax problem through policy and RM updates.

5 Methodology

SAVE continuously improves reward models from on-policy responses by anchoring response rewards to prompt-specific value estimates, filtering ambiguous feedback, and updating the model before policy optimization.

  • Value-Anchored Reward Modeling: SAVE initializes a value-anchored reward model with a shared backbone, reward head, and prompt-specific value head.The backbone and reward head learn from pairwise preferences, while the value head is integrated using prompt-only data and sampled responses.
  • On-Policy Feedback: At each training step, the current policy generates response groups whose response-level RM advantages provide on-policy feedback.Advantages compare response rewards against the prompt-level value estimate.
  • Adaptive Feedback Filtering: A decaying curriculum threshold removes near-zero-advantage responses, emphasizing clearly separated feedback early and admitting broader feedback later.Only prompts containing both positive- and negative-advantage responses are retained for reward-model updates.
  • Self-Supervised Reward Model Improvement: The reward model is updated with a stop-gradient objective that treats the value anchor as fixed while separately updating the reward and value components.The reward loss updates the backbone and reward head, whereas the value loss updates the value head.
  • Policy Model Optimization: After reward-model improvement, the same sampled groups are re-evaluated and filtered before standard RL optimization produces the next on-policy distribution.This closes the feedback loop between reward-model updates and policy-generated data.

6 Experiments

Experiments evaluate SAVE across six reward-model benchmarks, multiple policy backbones and RL algorithms, and downstream policy tasks. SAVE consistently improves reward models and strengthens policy performance, while ablations support adaptive filtering and evolving on-policy data.

  • Experimental Setup: SAVE is evaluated with two instruction-tuned policy backbones and three RL algorithms: GRPO, RLOO, and GSPO.UltraFeedback supplies prompts, while responses and reward scores are generated online during RL training.
  • Experimental Setup: The reward model is assessed on six benchmarks covering preference ranking, robustness, preference alignment, correctness, and complex response judging.The benchmark suite includes RewardBench, RewardBench 2, RM-Bench, PPE Preference, PPE Correctness, and JudgeBench.
  • Reward Model Results: 77.3 average accuracy is achieved with Qwen3-4B-Instruct-2507, improving from 76.0 and obtaining the best score on all six reward-model benchmarks.With Qwen2.5-3B-Instruct, SAVE raises the average score from 76.0 to 76.7 and achieves the best results on most benchmarks.
  • Reward Model Results: Continual offline training yields only marginal gains, while Mean Reward degrades severely and HL-BT matches the initial average but underperforms SAVE.The comparison favors value decomposition, adaptive filtering, and on-policy feedback over static-data updates or extreme-response pairing.
  • Ablations: Removing curriculum filtering drops average scores by 0.7 with Qwen3-4B-Instruct-2507 and 0.2 with Qwen2.5-3B-Instruct.Freezing the policy also reduces averages from 77.3 to 76.5 and from 76.7 to 76.4, respectively.
  • RL Algorithm Effects: SAVE consistently improves reward-model performance across GRPO, RLOO, and GSPO under both policy backbones.The authors attribute this transfer to all three algorithms producing grouped on-policy responses usable as value-anchored feedback.
  • Downstream Policy Results: Both co-training and improved-RM settings outperform vanilla GRPO on downstream benchmarks, with the improved-RM setting achieving the best overall results.The improved reward model also transfers beyond the co-training process.

7 Conclusion

SAVE uses on-policy responses, value anchoring, adaptive filtering, and contrastive learning to improve reward models without extra human labels or external judges. Experiments associate these improvements with stronger downstream policy performance.

  • Conclusion: Prompt-specific value anchors, adaptive feedback filtering, and value-anchored contrastive learning convert policy rollouts into reward-model feedback.The framework uses these components to make on-policy supervision suitable for reward-model improvement.
  • Conclusion: Experiments show consistent reward-model gains across benchmarks together with stronger downstream policy performance.The conclusion presents on-policy feedback as effective supervision for improving reward models and policies.

Limitations

SAVE’s evaluation is limited by model scale, reliance on automated evaluators, and added computation from on-policy feedback during RL training.

  • Experiments use 3B–4B policy models and a 3B reward model, leaving substantially larger-model training dynamics unverified.
  • Evaluation mainly relies on automatic reward-model benchmarks and LLM-based judges, so human preference studies remain needed for safety-critical, culturally sensitive, or subjective instructions.
  • SAVE samples multiple responses per prompt and updates the reward model during RL training, introducing additional computation whose efficiency remains an open direction.

B Proof of Proposition 1

The training procedure uses minibatch estimates and alternating block updates to approximate SAVE’s minimax objective, with special handling for batches lacking contrastive samples.

  • When a minibatch contains eligible prompts, the contrastive term estimates conditional risk while the value term uses all sampled response groups.
  • Empty contrastive minibatches skip the contrastive sub-step and use only the value term, producing null contrastive updates rather than unbiased contrastive-risk estimates.
  • The reward model and value head descend their respective losses with stop-gradient block surrogates, holding the opposing targets fixed during each update.
  • The policy update performs approximate inner ascent toward responses increasing self-supervised reward-model risk, while KL regularization limits distributional drift.
  • Each iteration filters ambiguous on-policy responses, partitions retained samples by advantage sign, updates the reward and value heads, then optimizes the policy with updated rewards.

D.1 Training Dynamics

SAVE’s training dynamics show faster and higher reward-model improvement than HL-BT, with performance shaped by update rates, group size, and adaptive-margin filtering.

  • Training Dynamics: SAVE rises faster than HL-BT early on RewardBench 2 and RM-Bench, maintains a consistent lead, and converges higher after HL-BT plateaus around 200 steps.
  • Sensitivity Analyses: A smaller backbone-and-reward-head learning rate paired with a larger value-head rate performs best, allowing the anchor to track distributional changes without disturbing reward knowledge.
  • Sensitivity Analyses: Larger response groups generally improve reward-model performance by exposing richer on-policy variation and making the value anchor and adaptive filter more informative.
  • Sensitivity Analyses: The initial margin controls a precision–coverage trade-off: aggressive filtering increases confidence but discards potentially useful signals, while an intermediate setting balances both.
  • Case Study: SAVE avoids HL-BT’s spurious pairing of semantically equivalent answers by assigning both above-anchor responses to the positive subset rather than forcing a chosen/rejected gap.

F.4 Training Cost Analysis

SAVE increases training memory and time relative to GRPO, trading moderate overhead for stronger reward feedback while keeping total training time comparable.

  • 50GB versus 62GB peak GPU memory and 26 versus 29 hours of training compare GRPO with SAVE under identical ZeRO-2 CPU-offloading settings.
  • The extra memory mainly reflects maintaining and updating the reward model, including its value head, while the time increase remains limited.
  • SAVE therefore trades moderate memory growth for stronger reward feedback while keeping overall training time comparable to GRPO.
Loading 2605.30888v1…