PaperlayerView exact revision on arXiv

Source-linked AI summary

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training

Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, Haoyu Wang

arXiv:2602.01511v2cs.CLcs.LG

TL;DR

Non-verifiable domains expose limits of scalar reward models and static or decoupled rubric pipelines. Rubric-ARM jointly learns a rubric generator and judge with alternating reinforcement learning, achieving broad benchmark gains and stronger policy-alignment supervision. Its analysis and experiments support alternating updates as a stable approach, with scope qualified by an exploration-gradient assumption.

  • Problem

    Scalar or pairwise reward judgments miss multifaceted response quality in non-verifiable domains, while high-quality rubrics are costly or difficult to obtain at scale.

  • Method

    Rubric-ARM treats rubric generation as a latent action and jointly optimizes the rubric generator and judge through alternating reinforcement learning.

  • Results

    +4.7% average gain on reward-modeling benchmarks, with robust out-of-distribution generalization and improved supervision for offline and online policy alignment.

  • Takeaways & Limitations

    Alternating rubric and judge optimization provides a more reliable reward signal than static approaches within the evaluated non-verifiable post-training settings.

  • Takeaways & Limitations

    The early-training variance analysis assumes exploration-gradient sufficiency, with the generator’s gradient norm satisfying an exploration-dependent lower bound relative to the judge’s.

Abstract

from arXiv · show

Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.

1. Introduction

Scalar or pairwise reward judgments can miss multifaceted response quality in non-verifiable domains, while existing rubric construction is costly, fixed, or decoupled from judging. Rubric-ARM jointly trains rubric generation and judging with alternating reinforcement learning, improving benchmark reward modeling and downstream policy post-training.

  • Scalar or pairwise judgments often fail to capture multifaceted response quality in creative writing and open-ended instruction following.
  • Human-authored rubrics are expensive and difficult to scale, while prompting-based methods rely on fixed models and learning-based methods train generators and judges separately.
  • Rubric-ARM jointly optimizes a rubric generator and judge through alternating reinforcement learning, treating rubrics as latent actions that guide preference prediction.
  • The schedule first stabilizes the reward model under fixed rubrics before optimizing the rubric generator, reducing variance during joint optimization.
  • +4.7% average gain on reward-modeling benchmarks accompanies consistent improvements in downstream policy post-training across offline and online alignment settings.

2. Related Works

Prior work increasingly uses LLM judges and rubric-based reward models for evaluation, alignment, and reasoning. A central unresolved challenge is generating high-quality rubrics at scale.

  • LLM-based judges have expanded from foundational scoring toward chain-of-thought, self-critique, strategic evaluation, and inference-time reasoning.
  • Rubric-based approaches have emerged for LLM evaluation, alignment, and reasoning.
  • Generating high-quality rubrics at scale remains a distinctive challenge for rubric-based reward models.

3. Preliminaries

Rubric-based reward modeling uses pairwise preference feedback to evaluate responses in non-verifiable domains. Rubrics are prompt-conditioned criteria, generated as latent actions and judged through preference correctness optimized with reinforcement learning.

  • Rubrics are structured, prompt-conditioned sets of criteria, with each criterion specifying an aspect of response quality such as correctness, tone, or presentation.
  • Training uses pairs of candidate responses and a preference label indicating which response is preferred.
  • The rubric generator produces a rubric from the prompt, while the judge predicts a preference and reasoning chain conditioned on the prompt, responses, and rubric.
  • The learning objective maximizes expected preference correctness under generated rubrics.
  • Because rubrics and judging decisions are sampled actions, reinforcement learning is used to optimize the objective.

4. Rubric-ARM: Alternating RL for Rubric Generation and Judging

Rubric-ARM alternates judge and rubric-generator updates around a shared preference-correctness objective, using rubrics as latent actions and format-shaped rewards for stable rubric-conditioned judging. The trained components then supervise offline and online policy optimization.

  • 4. Rubric-ARM: Alternating RL for Rubric Generation and Judging: Rubric-ARM addresses nonstationary targets from simultaneous updates by decoupling rubric-generator and judge learning through alternating reinforcement learning.
  • 4.1. Stage I: SFT Warmup: Stage I warm-starts both policies with synthetic rubric and judging trajectories from open-source datasets using next-token prediction.
  • 4.2. Stage II: Alternating Reinforcement Learning: The judge is updated with the rubric generator fixed to recover dataset preferences under sampled rubrics.
  • 4.2. Stage II: Alternating Reinforcement Learning: Judge training combines preference correctness with a format reward requiring per-criterion explanations, an overall justification, and a final decision.
  • 4.2. Stage II: Alternating Reinforcement Learning: The rubric generator is then updated with the judge fixed to produce rubrics that lead to correct decisions, with cached rubrics reducing sampling cost during judge updates.
  • 4.2. Stage II: Alternating Reinforcement Learning: The default cycle updates the judge before the rubric generator, and the alternating procedure is described as a generalized EM process with rubrics as latent variables.
  • 4.3. Policy Model Post-training with Rubric-ARM: For policy post-training, Rubric-ARM samples rubrics and labels response preferences, supporting DPO, iterative DPO, and online RL supervision.
  • 4.3. Policy Model Post-training with Rubric-ARM: Online comparison uses additional rollouts and queries both response orders under the same rubric to mitigate positional bias.

5. Theoretical Analysis

The analysis compares fixed-rubric judge updates with rubric-generator updates and shows that exploration-induced variance creates greater instability. This supports stabilizing the judge under fixed rubrics before alternating optimization.

  • Strategy A freezes and reuses the rubric during judge updates, so estimator variance is determined solely by binary classification uncertainty.
  • Strategy B’s generator-gradient variance decomposes into multiplicative reward noise and cross-rubric inconsistency.
  • Multiplicative reward noise reflects judge uncertainty amplified by the high-dimensional generator gradient, while cross-rubric inconsistency causes gradient-direction oscillation.
  • The analysis assumes an exploration-dependent lower bound linking generator-gradient magnitude to judge-gradient magnitude during early training.
  • Active exploration introduces a positive buffer that can satisfy the assumption without requiring strict generator-gradient dominance, because both components typically produce comparable-length sequences.
  • Under the assumption, Strategy B has strictly higher gradient variance than Strategy A, making exploration stochasticity the governing source of instability.
  • The resulting schedule trains the judge first with fixed rubrics, then the generator, and finally alternates updates to reduce variance and improve training stability.

6. Experiment

Rubric-ARM is evaluated across reward-model and downstream policy-alignment benchmarks, including out-of-distribution writing and instruction-following tasks. It achieves strong results through alternating judge–rubric optimization, with gains extending to offline and online policy training.

  • Reward-model evaluation: Rubric-ARM achieves the best average performance among white-box methods, improving Rubric-RM from 70.1 to 74.8 and reaching 76.2 with voting@5.It also outperforms API-based judges, including 76.2 versus 71.3 for Rubric+Judge API and 64.9 for direct Judge API.
  • Reward-model evaluation: On the out-of-distribution WritingPreferenceBench, Rubric-ARM achieves the best overall score of 63.2, exceeding Rubric-RM at 60.3 and RM-R1-Qwen2.5-7B at 59.8.The improvements span Functional, Promotional, Non-Fiction, and Poetry genres.
  • Ablation study: Reversing the default judge-first optimization order lowers the average from 74.8 to 72.4 without voting and from 76.2 to 74.9 with voting@5.RewardBench2-Precise IF declines from 41.9 to 24.4, supporting the use of a stronger judge signal before rubric optimization.
  • Ablation study: Removing the format reward lowers the average from 74.8 to 72.6 without voting and from 76.2 to 75.5 with voting@5.The largest reported gain is on RewardBench2-Precise IF at +16.3, where format reward improves rubric adherence and prevents missing criteria checks.
  • Offline policy alignment: Rubric-ARM-trained rewards improve downstream instruction following, reaching 80.8 on IFEval and 85.0 on InfoBench with iterative DPO.On IFBench, Rubric-ARM reaches 35.4 with IterDPO, above RLCF at 32.0 and iterative Rubric-RM at 33.7.

7. Conclusion

Rubric-ARM jointly optimizes rubric generation and judging through alternating reinforcement learning, using gradient-variance analysis to stabilize training. It reports gains across benchmarks and stronger supervision for policy alignment in offline and online RL.

  • 7. Conclusion: Rubric-ARM treats rubric generation as a latent action and jointly optimizes a rubric generator and judge through alternating reinforcement learning.The framework uses an alternating update schedule grounded in gradient-variance analysis.
  • 7. Conclusion: 4.7% gains across diverse benchmarks accompany robust out-of-distribution generalization.
  • 7. Conclusion: Rubric-ARM provides superior supervision for policy alignment in both offline and online reinforcement learning settings.The conclusion characterizes its reward signal as more reliable than static approaches.
  • 7. Conclusion: GRPO reduces variance by using the within-prompt average reward as a baseline during actor-only policy optimization.The method samples response groups, computes group-normalized advantages, and applies a PPO-style clipped update.

B. Detailed Theoretical Derivations

The derivations compare gradient estimators for fixed-rubric judging and rubric-generator updates. They use conditional variance and expectation terms to decompose total variance under alternating optimization.

  • B. Detailed Theoretical Derivations: The analysis defines reward, judge-correctness, generator-score, and judge-score quantities for the rubric-based reinforcement-learning estimators.
  • B. Detailed Theoretical Derivations: The Law of Total Variance decomposes vector variance into expected conditional variance plus variance of the conditional expectation.The derivation states Var(Y) = E_X[Var(Y|X)] + Var_X(E[Y|X]).
  • B. Detailed Theoretical Derivations: With a fixed rubric, the judge gradient estimator is a scaled Bernoulli variable determined by whether the sampled response is correct.Correct responses produce the judge score, while incorrect responses receive zero reward.
  • B. Detailed Theoretical Derivations: The fixed-rubric estimator’s first and second moments are used to obtain its variance expression.
  • B. Detailed Theoretical Derivations: For rubric-generator updates, conditioning on the sampled rubric separates reward randomness into conditional variance and variance of the conditional expectation.The rubric score is constant when conditioned on a rubric, while the reward follows Bernoulli(p(r)).

B.4. Proof of Theorem 5.5

The proof studies the variance difference between rubric-generator updates and the expected conditional variance of fixed-rubric judge updates. Under the stated assumption, it establishes strict positivity.

  • B.4. Proof of Theorem 5.5: The proof defines Δ as the variance of Strategy B minus the expected conditional variance of Strategy A.
  • B.4. Proof of Theorem 5.5: The variance difference is expanded using the expressions from Propositions 5.1 and 5.2.
  • B.4. Proof of Theorem 5.5: The proof incorporates the exploration coefficient C1 through Var_r(p(r)u_r(r)) = C1 E_r[p(r)^2∥u_r(r)∥^2].
  • B.4. Proof of Theorem 5.5: The assumed inequality makes the bracketed term strictly positive, and p(r) ∈ (0, 1) then makes the full integrand positive.Therefore, the expectation defining the variance difference is strictly positive.

C. Implementation Details

The implementation uses established libraries for Rubric-ARM, DPO, and iterative DPO training, with three alternating-RL iterations. Supplementary tables specify training and inference settings.

  • C. Implementation Details: GRPO training is implemented with the ms-swift library.
  • C. Implementation Details: DPO and iterative DPO are implemented with LLaMA-Factory.
  • C. Implementation Details: Tables 9 and 10 report hyperparameters for Rubric-ARM and policy-model training.
  • C. Implementation Details: Rubric-ARM alternating reinforcement-learning training is implemented for three iterations.
  • C. Implementation Details: Table 11 reports Rubric-ARM inference sampling parameters, while Table 12 compares judges and reward models on WritingPreferenceBench.

D.2. Position Bias Analysis

The section evaluates position bias by changing response-pair ordering across three settings and reports results on RewardBench and IF evaluation benchmarks. Rubric-ARM shows lower sensitivity to response ordering and avoids a judging error illustrated for Rubric-RM.

  • Position-bias settings: The evaluation compares original ordering, globally flipped ordering, and randomly flipped ordering for each response pair.These settings test whether predicted preferences depend on the relative order of the responses.
  • Position-bias results: Rubric-ARM shows much lower sensitivity to response-pair ordering than the other judge and reward models.Table 13 presents the position-bias analysis on RewardBench and IF evaluation benchmarks.
  • Case study: In an IFBench case requiring specific keywords and exactly two paragraphs, Rubric-RM incorrectly judged a valid response as having three paragraphs.The error is described as a judging hallucination caused by the baseline rubric output.
  • Case study: Rubric-ARM accurately extracts the required keywords and paragraph structure in the same IFBench example.The case study contrasts its judgment with Rubric-RM's incorrect paragraph-count claim.
  • Judge procedure: The judge prompt separates objective compliance checks from subjective quality criteria when analyzing each response.Objective gatekeeper examples include word or paragraph limits and required output formats, while criteria such as creativity and tone are treated as subjective.
Loading 2602.01511v2…