PaperlayerView exact revision on arXiv

Source-linked AI summary

RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains

Haoxiang Jiang, Zihan Dong, Tianci Liu, Wanying Wang, Ran Xu, Tony Yu, Linjun Zhang, Haoyu Wang

arXiv:2605.29156v1cs.LGcs.CL

TL;DR

Pointwise reward modeling remains difficult in subjective, non-verifiable settings, while rubric evaluators often depend on frontier models and hard aggregation. RUBRIC-ARROW jointly trains a rubric generator and conditioned judge from pairwise preferences, and outperforms strong baselines by 3.0% while improving downstream policy post-training.

  • Problem

    Rubric-based pointwise reward models need jointly trainable, consistent rubric generators and judges that can learn from pairwise preferences without frontier-LLM supervision.

  • Method

    RUBRIC-ARROW alternates reinforcement-learning updates for a rubric generator and rubric-conditioned judge, using probability-based scoring and pairwise preference rewards.

  • Results

    3.0%: Rubric-ARROW consistently outperforms strong baselines across reward-modeling benchmarks and downstream policy post-training.

  • Takeaways & Limitations

    Rubric-ARROW attains the best average reward-modeling performance and the best downstream policies under offline and online training.

  • Takeaways & Limitations

    Theoretical guarantees assume a human-preferred response pair and analyze each alternating phase with the other module frozen.

Abstract

from arXiv · show

Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.

1. Introduction

RUBRIC-ARROW addresses unreliable pointwise scoring in non-verifiable domains by alternatingly training a rubric generator and rubric-conditioned judge from pairwise preferences alone. It combines probability-based scoring with alternating reinforcement learning to jointly optimize both evaluation components.

  • Pointwise reward modeling assigns independent scalar scores but is unreliable for holistic evaluation in non-verifiable tasks such as open-ended instruction following and chat.
  • Rubric-based evaluation requires both high-quality criteria and a judge that applies them consistently, while frontier-LLM prompting makes reward estimation expensive, difficult to control, and hard to deploy.
  • Existing learning-based rubric evaluators often train only the rubric generator or judge and retain frontier-LLM dependence for supervision.
  • RUBRIC-ARROW treats rubric evaluation as two-module learning: a rubric generator proposes criteria, and a rubric-conditioned judge produces criterion-level scores aggregated into a pointwise reward.
  • Both modules are jointly optimized through alternating reinforcement learning using only pairwise preference data during the RL stage, with a probability-based scoring rule reducing ties.

2. Related Work

Reward modeling has progressed from scalar pairwise-preference models toward generative and judge-based evaluators, including pointwise and pairwise judges. Rubric-based methods further structure open-ended evaluation through explicit criteria, rubric learning, policy guidance, and rubric-conditioned judging.

  • Reward modeling and LLM judges: Reward modeling is central to RLHF and LLM post-training, with early approaches learning scalar rewards from pairwise preferences using Bradley–Terry-style objectives.More recent work has shifted toward generative and judge-based evaluators that may produce rationales or chain-of-thought judgments.
  • Reward modeling and LLM judges: Pointwise judges score responses independently and are efficient, directly reusable reward models, but their coarse scores miss fine-grained preference differences in subjective tasks.Pairwise judges instead compare responses directly.
  • Rubric-based reward modeling: Rubric-based reward modeling replaces holistic judgments with structured criteria or checklists for open-ended evaluation.Prior work spans rubric extraction and synthesis, rubric-guided policy learning, and rubric-conditioned judges.

3. Preliminaries

The pointwise rubric reward model evaluates instruction–response pairs by generating explicit rubrics, judging each criterion, and aggregating the judgments into a scalar score. Rubric items are sampled for each instruction, while the judge uses binary satisfaction judgments and predefined positive weights.

  • Evaluator: The evaluator first generates a rubric for an instruction–response pair, then queries a rubric-conditioned judge and aggregates criterion-level judgments into a scalar score.This defines the evaluator as a two-stage rubric-generation and judgment pipeline.
  • Rubric generator: For instruction x, the rubric generator πr samples a rubric set r = {rk}K_k=1, where K is the number of rubric items.Each rk denotes the k-th rubric item.
  • Rubric-conditioned judge: Given instruction x, candidate response y, and rubric set r, the judge πj produces criterion-level judgments a = {ak}K_k=1.The judge also produces explanations e through its judgment distribution.
  • Pointwise score: Each judgment ak is binary: ak = 1 when y satisfies rubric item rk and 0 otherwise; criterion judgments are aggregated using predefined weights wk > 0.The weights represent prior importance assigned to rubric items.

4. Method

RUBRIC-ARROW trains a pointwise rubric reward model through SFT warm-up followed by alternating GRPO on pairwise preference data alone. Probability-based scoring supplies smoother evaluator rewards, while phase-specific judge and generator updates support downstream DPO and GRPO.

  • Training framework: The method first performs brief SFT, then alternates RL updates for the rubric generator and rubric-conditioned judge using pairwise preference data alone.The warm-up uses OpenRubrics rubrics and GPT-5-mini-generated judge labels, retaining pairs where the chosen response scores higher.
  • Probability-based scoring: Replacing Boolean rubric judgments with probability margins converts discrete scores into continuous, more discriminative pointwise reward signals while preserving the original aggregation structure.The margin is defined from the judge’s probabilities for the “true” and “false” tokens at the verdict position.
  • Alternating RL objectives: In the judge phase, pairwise labels become preference-consistency rewards by encouraging higher scores for the preferred response and lower scores for the other response.The judge is optimized with the rubric generator fixed, using stochastic rollout scores compared against the opposite response’s average score.
  • Alternating RL objectives: In the generator phase, sampled rubrics are rewarded when the frozen judge ranks the preferred response higher, with penalties for unnecessarily long or redundant rubrics.The judge and generator are optimized in alternating phases until convergence.
  • Downstream policy training: After training, the evaluator constructs DPO pairs from highest- and lowest-scoring candidates offline and supplies pointwise rewards to GRPO online.The highest-scoring response is selected as chosen and the lowest-scoring response as rejected for offline DPO.

5. Theoretical Analysis

The theoretical analysis establishes preference consistency and variance-reduction properties for the judge, alongside phase-wise GRPO convergence guarantees for the alternating algorithm. These results apply under stated rollout-noise and optimization assumptions, with rollout-dependent error scaling as O(n^-1) in both phases.

  • Judge-side guarantees: Jn > 1/2 if and only if Δ > 0, while Jn = 1/2 when Δ = 0, establishing preference consistency under symmetric rollout noise.The result calibrates the judge-side reward to human preference recovery.
  • Judge-side guarantees: For Δ > 0 and n > 1, opposite-side averaging achieves strictly higher preference recovery probability than a single pairwise comparison.Under Gaussian rollout noise, Jn increases monotonically with n.
  • Phase-wise GRPO guarantees: With the other module fixed, GRPO attains first-order stationarity for the phase-specific surrogate under smoothness, unbiased-gradient, bounded-variance, and diminishing-step-size assumptions.The phase b may be either judge updates or generator updates.
  • Phase-wise GRPO guarantees: With a bounded surrogate-to-true gradient gap, each phase reaches a stationary neighborhood of the true objective, and the rollout-dependent term is O(n^-1) in both phases.The effective group sizes are Gj = 2n for the judge phase and Gr = n for the generator phase.

6. Experiment

Rubric-ARROW is evaluated as a white-box reward model and as a source of rewards for policy post-training across broad benchmark suites. It achieves strong reward-modeling accuracy, robustness under best-of-N selection, and consistent downstream gains, with ablations supporting its alternating RL, length penalty, and probability-based scoring components.

  • Reward-model evaluation: Rubric-ARROW achieves the best average performance among white-box methods, improving from 70.1 for Rubric-RM to 73.1, or 75.6 with voting@5.Voting@5 also achieves the best results on Chat Hard, PPE-IFEval, InfoBench, and IFBench.
  • Reward-model evaluation: Rubric-ARROW-voting@5 outperforms prompting-based judges, while its improvement over the no-RL variant supports jointly optimizing the rubric generator and judge.The results indicate that learned rubric-conditioned evaluation provides more reliable signals than direct prompting.
  • Robustness evaluation: Rubric-ARROW continues to closely track the oracle upper bound in best-of-N selection, whereas Rubric-RM and Skywork begin degrading after N = 8.The evaluation uses 100 WildBench prompts from instruction-following and chat categories with frontier-LLM rubric scores as the oracle reference.
  • Ablation studies: Removing the length penalty reduces the average score from 73.1 to 72.3 and voting@5 from 75.6 to 74.3.The drop is most visible on stricter instruction-following benchmarks such as Chat Hard and PPE-IFEval.
  • Ablation studies: Probability-based scoring resolves 98/140 tied cases on InfoBench and 68/115 on IFBench while preserving 305/312 and 234/255 originally correct predictions.A random-probability control preserves only 160/312 and 130/255 correct decisions, indicating that meaningful judge confidence drives the improvement.
  • Downstream policy evaluation: Policies trained with Rubric-ARROW-derived rewards show consistent gains across instruction-following, preference-alignment, creative-writing, and online RL evaluations.Reported results include 80.7 on IFEval with DPO, 53.0 on Arena-Hard and AlpacaEval with IterDPO, 40.5 on Creative Writing Benchmark v3 with IterDPO, and 65.4 average with GRPO.

7. Conclusion · A. Experiment · A.1. Implementation Details

RUBRIC-ARROW jointly trains a rubric generator and rubric-conditioned judge for pointwise reward modeling in non-verifiable domains using alternating reinforcement learning with pairwise preference data. Its implementation combines weighted rubric scoring, three alternating RL iterations, and established training libraries.

  • 7. Conclusion: RUBRIC-ARROW is an alternating framework for training a pointwise rubric reward model in non-verifiable domains.
  • 7. Conclusion: After a brief SFT warm-up, the alternating RL stage jointly trains the rubric generator and rubric-conditioned judge.
  • 7. Conclusion: The RL stage uses a probability-based scoring rule to reduce ties and phase-specific preference-based rewards with alternating GRPO.
  • A. Experiment: The method trains using only pairwise preference data, without further frontier-LLM annotation.
  • A. Experiment: Theoretical analysis establishes preference consistency, variance reduction from opposite-side averaging, and phase-wise convergence guarantees.
  • A.1. Implementation Details: Rubric-based scoring assigns weight 3 to each Hard Rule and weight 1 to each Principle when computing the final score.
  • A.1. Implementation Details: GRPO models are trained with ms-swift, while DPO and IterDPO training use LLaMA-Factory.
  • A.1. Implementation Details: Rubric-ARROW is trained for three alternating RL iterations.

A.2. Evaluation Details · A.3. Baselines · A.4. Length-Aware Rubric Reward

The evaluation constructs pairwise benchmark comparisons from sampled responses, follows official splits and metrics, and compares Rubric-ARROW with generative, reward-model, and downstream-training baselines. Its length-aware rubric reward favors concise preference-consistent rubrics while preventing overly short solutions.

  • A.2. Evaluation Details: For FollowBench and InfoBench, each prompt yields two responses from Qwen-3-8B or Qwen-3-14B, with constraint violations identified by official verifiers.The original single-response evaluation is converted into pairwise comparison.
  • A.2. Evaluation Details: Evaluation follows each benchmark’s official splits and scoring rules, reporting accuracy, win rate, or the benchmark-specific metric.
  • A.3. Baselines: Reward-model comparisons include RM-R1, RRM, and JudgeLRM, alongside black-box references such as Claude-3.5-Sonnet4 and Gemini-2.5-Flash.The black-box references also include API-based rubric+judge pipelines with pairwise, pointwise, and direct-judge variants.
  • A.3. Baselines: Downstream policy training uses Qwen2.5-7B-Instruct with signals from Skywork, ArmoRM, UltraFeedback, and RLCF.Skywork and ArmoRM are scalar Bradley–Terry reward models trained on pairwise human preferences, while UltraFeedback supplies multi-aspect AI-feedback annotations.
  • A.4. Length-Aware Rubric Reward: The rubric-generator reward combines preference consistency with a length-aware bonus over preference-consistent rubric rollouts.The supplied formulation defines the reward through the set of preference-consistent rollouts and their rubric-item counts.
  • A.4. Length-Aware Rubric Reward: When no preference-consistent rubric rollout exists, every rubric rollout receives reward −1.
  • A.4. Length-Aware Rubric Reward: The length-aware bonus selects the shortest rubric set that correctly recovers human preference, while ¯Ki ≥5 prevents rewards for overly short rubrics.

A.5. Full Version of Policy Model Performance on IFBench and Creative Writing Benchmark v3 … B.2. Proof of Theorem 5.1

The appendix expands policy-model comparisons, presents a case study with full rubric and judge outputs, and supplies assumptions and proof details for the theoretical results. The proofs establish Theorem 5.1 through continuity, symmetry, monotonicity, and averaging arguments.

  • A.5. Full Version of Policy Model Performance on IFBench and Creative Writing Benchmark v3: Figs. 4 and 5 provide full IFBench and Creative Writing v3 policy-model comparisons, including two baselines omitted from the main figures for readability.The IFBench baseline results, except Rubric-RM (IterDPO), come from OpenRubrics; Creative Writing v3 baseline results, except Rubric-RM, come from RuscaRL.
  • A.6. Case Study: Table 12 expands the main-text case study with the prompt, two candidate responses, and complete rubric and judge outputs for RIFL and Rubric-ARROW.
  • B. Proofs of Theoretical Results: The proofs appendix provides technical assumptions and full proofs omitted from the main text.
  • B.1. Auxiliary setup: The auxiliary setup defines judge-side results using averages of n i.i.d. copies of S+ and S−.
  • B.1. Auxiliary setup: For optimization, each phase b ∈ {j, r} holds the other module θ−b fixed and uses a phase-specific population objective with a one-step on-policy GRPO surrogate.The GRPO iterates use a stochastic gradient estimator for the active phase.
  • B.1. Auxiliary setup: Theoretical optimization results assume phase-wise smoothness and upper boundedness, unbiased gradient estimates, bounded conditional variance, a specified step size, and bounded surrogate-to-true gradient gaps.The phase sample counts are Gj = 2n for the judge phase and Gr = n for the generator phase.
  • B.2. Proof of Theorem 5.1: Theorem 5.1’s proof uses continuity and a median of 0 to derive the needed probability relation, with the second term handled analogously.
  • B.2. Proof of Theorem 5.1: Symmetry about 0 yields the corresponding relation, and averaging Equations (24) and (26) inside Equation (14) proves the claim.

B.3. Proof of Theorem 5.2

The proof derives the multi-comparison result by combining Gaussian branches and shows that, for n > 1, its performance exceeds a single pairwise comparison. In the equal-variance case, the resulting expression is strictly increasing in n when ∆ > 0.

  • Gaussian closure under averaging and subtraction enables the proof’s distributional derivation.
  • Combining the two branches yields the expression used to compare n comparisons with the single-pairwise baseline.
  • For n > 1 and ∆ > 0, strict monotonicity of Φ implies J_n > J_pair.
  • When σ+ = σ− := σ, the two terms coincide, and the resulting expression is strictly increasing in n whenever ∆ > 0.

B.4. Proof of Theorem 5.3 · B.5. Proof of Theorem 5.4 · C. Prompts

The paper proves convergence and stationary-neighborhood guarantees for the alternating phases, then specifies the experimental prompts and a rubric-level error case study. The proofs establish an O(T^-1/2) rate and an O(n^-1) rollout-dependent term in both phases.

  • B.4. Proof of Theorem 5.3: Theorem 5.3 follows by combining phase-wise smoothness with identities, taking full expectations, summing over t, and dividing the resulting bound.The proof fixes a phase b ∈ {j, r} while freezing θ−b.
  • B.4. Proof of Theorem 5.3: Theorem 5.3 yields the stated O(T−1/2) convergence rate.
  • B.5. Proof of Theorem 5.4: Theorem 5.4 decomposes the phase-wise true gradient, averages its expectation over iterations, and controls its terms using the preceding bound and the uniform gradient-gap assumption.
  • B.5. Proof of Theorem 5.4: Gj = 2n and Gr = n imply an O(n−1) rollout-dependent term in both phases of the stationary-neighborhood bound.
  • C. Prompts: The prompts section reports the experimental prompts and states that baseline prompts follow official implementations and original papers.
  • C. Prompts: The rubric-generation prompt requires universal rubric principles and separates hard rules derived from explicit request requirements from another rubric category.Topic-specific references such as names, places, myths, numbers, and historical facts invalidate a rubric item.
  • C. Prompts: The judge-generation prompt scores the last assistant response against rubric items and returns explanations plus boolean criteria_met_i fields in a single JSON object.The required keys are explanation_i and criteria_met_i for each rubric item, with no additional text.
  • C. Prompts: Table 12 describes a case study where Rubric-ARROW rejects a misleading comparison that the baseline incorrectly accepts through surface-level criteria.
Loading 2605.29156v1…