PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric Rewards from Item Response Theory

Milad Yazdani, Yaser Souri, Xiren Zhou, Pranit Chawla, Dena Shahriari, Subhojit Som, Xia Song

arXiv:2609.35646v1cs.CL

TL;DR

Rubric-based reinforcement learning needs scalar rewards from criterion verdicts, but point totals can collapse distinct verdict patterns and judging every criterion increases judge requests. RRT infers quality from verdict patterns and selects criteria using Fisher information; across four datasets it exceeds Vanilla GRPO by 1.7 points on Qwen3.5-4B and remains close with half-budget judging.

  • Problem

    Weighted point totals can assign the same reward to distinct verdict patterns, while judging the full rubric requires more requests as criteria are added.

    Source: §1 · p. 1

  • Method

    RRT uses an item response model to infer rollout quality from rubric verdicts and Fisher information to select criteria, with online updates to its parameter network.

    Source: §1 · p. 2

  • Results

    Across Medical, Science, Rubrics as Rewards Science, and RubricBench, RRT exceeds Vanilla GRPO by 1.7 points on Qwen3.5-4B; at half the criterion budget, its macro score is within 0.1 points of Vanilla GRPO with full judging.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For rubrics whose criteria are monotone indicators of a shared target, RRT reduces judge usage while retaining Vanilla GRPO gains, and the deployed policy adds no parameters or inference components.

    Source: §5 · p. 10

    RRT assumes criteria are monotone indicators of shared quality and verdicts are conditionally independent given quality and criterion parameters.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Many language tasks have no single answer that can be checked automatically. Rubrics provide criteria for judging responses to these tasks. For reinforcement learning, the resulting verdicts must be combined into a scalar reward. A common approach sums the points assigned to satisfied criteria. Distinct verdict patterns can thus receive the same reward, and the fixed points encode how much each criterion should count, not how strongly its verdict distinguishes the current rollouts. Beyond this aggregation problem, judging the full rubric needs more judge requests as the criterion count grows. To address these limitations, Rubric Response Theory (RRT) measures quality and selects criteria when rubric criteria are monotone indicators of a shared target. Rather than adding assigned points, RRT uses a two parameter item response model that treats the verdict pattern as evidence about scalar quality specific to the rubric. Under this model, its likelihood score maximizes the local signal-to-noise ratio for quality. Its Response Parameter Network (RPN) reads the prompt and criterion text to predict criterion difficulty and discrimination. As the policy distribution changes during training, RRT uses online expectation maximization to update the RPN from current rollout verdicts. With Qwen3.5-4B as the policy, RRT's macro criterion score across Medical, Science, Rubrics as Rewards Science, and RubricBench is 1.7 points above that of group relative policy optimization (GRPO). On hard and very hard criteria in Medical and Science, RRT gains 2.8 to 5.6 points over GRPO. At half the criterion budget, adaptive Fisher selection with a frozen RPN keeps the macro criterion score across four datasets within 0.1 points of GRPO with full judging. These results show RRT can reduce judge requests while remaining competitive with GRPO.

1 INTRODUCTION

Rubric-based reinforcement learning must turn criterion verdicts into scalar rewards, but weighted point totals can erase differences between verdict patterns and judging every criterion increases judge requests. Rubric Response Theory (RRT) instead infers rollout quality from verdict patterns and uses criterion information to guide judging.

1 INTRODUCTION

  • Weighted rubric points can give distinct verdict patterns the same reward, and full-rubric judging requires more requests as criteria are added.Assigned points encode how much each criterion counts, not how strongly its verdict distinguishes the current rollouts.Source: §1 · p. 1
  • RRT uses an item response model to infer rubric-specific rollout quality from verdicts and uses that quality as the GRPO reward.Its Response Parameter Network predicts criterion difficulty and discrimination from prompt and criterion text, then online hard EM updates the network from current rollout verdicts.Source: §1 · p. 2
  • 1.7 points: RRT exceeds Vanilla GRPO in macro criterion score on Qwen3.5-4B; at half the criterion budget, its score across four datasets is within 0.1 points of Vanilla GRPO with full judging.Figure 1(b) normalizes adaptive Fisher-selection scores to full judging for each dataset.Source: Figure 1 · p. 2, §1 · p. 2
  • Under its item response model, the rubric likelihood score maximizes local signal-to-noise ratio, while adaptive Fisher selection ranks unjudged criteria by information at inferred rollout qualities.Rollout qualities are updated after each selected criterion is judged.Source: §1 · p. 2

2 RELATED WORK

Prior rubric approaches use fixed weights, judge-provided aggregation, or weighted sums adapted to current rollouts, while item response theory has mainly measured fixed examinees or models. RRT applies learned criterion measurement to changing policy rollouts and uses it for rewards and criterion selection.

2 RELATED WORK

  • Gunjal et al. (2026) and Viswanathan et al. (2025) use preset criterion weights or judge aggregation and judge the full rubric for every response.RRT complements rubric generation by learning from current rollouts how much each verdict reveals about quality and which criteria to judge.Source: §2 · pp. 2–3
  • POW3R (Tyagi et al., 2026) and DIVA adapt weighted sums to current rollouts but retain one weight per criterion and judge every criterion.RRT replaces the weighted sum with inferred quality from the full verdict pattern, enabling different rewards for equal point totals and criterion selection.Source: §2 · p. 3
  • Prior IRT applications measure fixed examinees or models, whereas RRT measures policy-training rollouts and predicts criterion parameters from text for unseen rubrics.Current training verdicts refine those estimates as the policy changes, and the same estimates select informative criteria.Source: §2 · p. 3

3 METHOD

RRT models rubric verdicts as evidence about a shared rollout-quality variable, estimates that quality as a Bayesian reward, and uses Fisher information for aggregation and criterion selection. Its item-response assumptions and online calibration support these operations as the policy changes.

3.1 IRT FOR RUBRIC CRITERIA

  • RRT assumes criteria are monotone indicators of shared quality and conditionally independent given quality and criterion parameters.The Response Parameter Network predicts each criterion’s difficulty and discrimination from the prompt and criterion text.Source: §3.1 · p. 3
  • The two-parameter item response model assigns each criterion difficulty and discrimination, which shape its pass probability as a function of rollout quality.Difficulty is the quality at which pass probability reaches 0.5; discrimination controls the slope there.Source: §3.1 · p. 3

3.2 BAYESIAN REWARD INFERENCE AND ONLINE EM

  • RRT uses the posterior mode of quality under a Gaussian prior as the GRPO reward, finding the unique maximizer of the strictly concave log posterior.The E-step computes rollout posterior modes, and one online EM sweep per policy step calibrates the Response Parameter Network from new verdicts.Source: §3.2 · pp. 3–4

3.3 FISHER INFORMATION FOR AGGREGATION AND SELECTION

  • Under the probit item response model, the rubric likelihood score is the unique scalar verdict signal, up to affine transformation, attaining the local information bound for quality.The bound applies at fixed quality and criterion parameters with conditionally independent verdicts; fixed point weights attain it only under a specific parameter-dependent proportionality.Source: §3.3 · pp. 4–5
  • RRT ranks unjudged criteria by total Fisher information at the inferred rollout qualities and updates those qualities after each selected criterion is judged.Fisher information measures expected information before judging, whereas the likelihood score measures evidence from an observed verdict.Source: §3.3 · p. 5

4 EXPERIMENTS

Across four datasets, policy comparisons test criterion satisfaction and training behavior, while verdict-prediction and criterion-selection experiments assess reward quality and judging efficiency. RRT improves macro criterion scores over Vanilla GRPO on Qwen3.5-4B, and adaptive Fisher selection retains near-baseline scores with half the criteria judged.

4.1 EXPERIMENTAL SETUP

  • The experiments use Qwen3.5-4B by default across Medical, Science, RaR Science, and RubricBench, with transfer comparisons on Qwen3.5-2B and Llama-3.1-8B-Instruct.Each condition has one training run; policy scores average three evaluation sampling seeds.Source: §4.1 · p. 5

4.2 VERDICT PREDICTION AND REWARD STABILITY

  • RRT gains 10.1 points in pooled macro ROC-AUC over the mean of other verdicts and exceeds the RPN without rollout evidence on every dataset.Table 1 reports leave-one-criterion-out verdict prediction by ROC-AUC.Source: §4.2 · p. 6, Table 1 · p. 6
  • The adopted RPN model improves macro ROC-AUC by 0.7 points over the model with a_j = 1, while predicted and empirical criterion difficulty correlate at 38.0% to 47.9%.Online updates increase macro Pearson correlation in empirical pass-rate prediction by 1.7 points on the next policy step.Source: §4.2 · p. 6
  • Relative to rubric points, RRT has 1.2 to 2.2 times the within-prompt variance share and 2% to 58% fewer tied pairs across all 12 dataset-policy cells.RRT also raises rank correlation over rubric points by 15.1 points on RaR Science and 0.5 to 3.6 points on the other datasets.Source: §4.2 · pp. 6–7
  • At corruption level 0.20, RRT with marginal calibration has the highest similarity on every dataset, with gains of 3.0 to 7.0 points over criterion score.Under repeated verdicts, marginal calibration increases the macro stable nonzero ordering share by 5.4 points over rubric points.Source: §4.2 · pp. 6–7

4.3 POLICY TRAINING RESULTS

  • RRT + online RPN exceeds Vanilla GRPO by 1.7 points and POW3R by 0.8 points on both macro metrics, and is highest or tied in every Table 2 column.The policy comparison reports criterion and normalized points scores.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 CRITERION SELECTION BASED ON FISHER INFORMATION

  • At 95.0% reward fidelity on trained-policy rollouts, adaptive Fisher leaves 11.0 points more criteria unjudged than random selection and 1.7 more than static Fisher in the macro mean.Table 3 reports unjudged criteria at the smallest budget reaching the stated correlation with GRPO advantages from full judging.Source: §4.4 · p. 8, Table 3 · p. 8
  • At criterion budget 0.50, adaptive Fisher keeps both RRT macro scores within 0.1 points of Vanilla GRPO with full judging and reduces RRT judge requests by 49.0% on Medical and Science.Compared with full judging, random selection at the same budget lowers macro criterion scores by 2.3 points for Vanilla GRPO and 1.6 for RRT.Source: §4.4 · p. 8, Table 4 · p. 8

5 DISCUSSION AND CONCLUSION

RRT combines rubric verdicts through an item-response model and uses criterion parameters for reward inference and selection. The discussion reports improved criterion satisfaction, more robust reward ordering, and lower judging costs, with no added deployment components.

5 DISCUSSION AND CONCLUSION

  • Under the item-response model, the rubric likelihood score maximizes local SNR; posterior curvature and even unanimous criteria can affect MAP reward gaps and GRPO advantages.Fewer ties preserve verdict-pattern distinctions across policies and rollout group sizes.Source: §5 · p. 9
  • Text-derived criterion parameters complement rollout evidence, and online EM improves next-step pass-probability correlation while the hard E-step has lower criterion loss than tested soft variants.The reported results support an RPN warm start followed by online calibration from current rollout verdicts.Source: §5 · p. 9
  • Marginal calibration preserves reward distinctions under judge variation; RRT has fewer ties and better order preservation in repeated judging, but rubric points fare better under symmetric or response-length-linked lenient errors.With online RPN parameters, RRT is less sensitive to criterion sampling on all tested datasets, with the largest gain in the lowest quintile of rank correlation under rubric points.Source: §5 · p. 9
  • RRT + online RPN improves criterion satisfaction over Vanilla GRPO, POW3R, and DIVA in the primary Qwen3.5-4B comparison, while scores remain close to Vanilla GRPO on Qwen3.5-2B and Llama-3.1-8B-Instruct.The gains on other benchmarks support generalization beyond the training rubrics.Source: §5 · p. 9
  • Fisher information unifies reward inference and criterion selection; difficulty and discrimination support savings over random selection, with the greatest savings shifting from static selection on base-policy rollouts to adaptive selection on trained-policy rollouts.The discussion says this shift supports adapting judge allocation as the policy changes.Source: §5 · p. 9
  • At about half the judge requests, adaptive Fisher selection keeps RRT macro scores close to Vanilla GRPO with full judging, while larger budgets recover more of RRT’s full-judging gains.Since judging is the longest measured stage of a policy step, selecting fewer criteria shortens steps with nearly unchanged generation time; online RPN updates add little computation.Source: §5 · p. 10
  • RRT’s savings and criterion-satisfaction gains are training-time changes: the deployed policy has no added parameters or inference components.This conclusion is stated for rubrics whose criteria are monotone indicators of a shared target.Source: §5 · p. 10

A.1 RESPONSE FUNCTION ASSUMPTIONS AND PROPERTIES

The response function maps a signed quality margin to criterion-satisfaction probability under monotonicity and centering assumptions. The logistic and Gaussian choices differ in how criterion difficulty affects reward ordering and verdict contributions.

A.1 RESPONSE FUNCTION ASSUMPTIONS AND PROPERTIES

  • The response function is strictly increasing, approaches 0 and 1 at opposite tails, and therefore makes a criterion’s pass probability nondecreasing with rollout quality.It maps the signed margin to a probability and is centered so that F(0) = 0.5.Source: §A.1 · p. 12
  • Criterion difficulty is the rollout quality where its satisfaction probability equals 0.5.The model also assumes a positive discrimination parameter for each criterion.Source: §A.1 · p. 12
  • With a logistic response function, MAP reward ordering depends on the weighted pass total and not criterion difficulties, although difficulties can still change reward gaps and normalized GRPO advantages in groups with at least three distinct weighted verdict totals.The difficulty-independence of ordering holds for centered logistic response functions, including scaled logistic functions.Source: §A.1 · p. 13
  • With the Gaussian CDF, criterion difficulty changes signed verdict coefficients, and changing one difficulty can reverse the MAP reward order of two fixed verdict patterns.The passage illustrates the reversal by varying one criterion’s difficulty while keeping the verdict vectors fixed.Source: §A.1 · pp. 13–14
  • For the Gaussian CDF, raising criterion difficulty increases the positive score for a satisfied criterion and decreases the magnitude of the negative score for an unsatisfied criterion.Thus a hard pass receives a larger positive likelihood score, while a hard failure receives a negative score with smaller magnitude.Source: §A.1 · p. 13
  • Under the Gaussian CDF, an unexpected hard pass or easy failure can have unbounded likelihood-score magnitude, whereas logistic criterion scores are bounded by their discrimination parameter.For a mastered criterion, the expected pass score tends to zero while the magnitude of an unexpected failure score grows with the margin.Source: §A.1 · p. 14

A.2 MAP REWARD DERIVATION AND PROPERTIES

RRT infers rollout quality from the full rubric verdict pattern rather than treating assigned points as fixed contributions. Under the Gaussian-CDF model, the MAP reward is uniquely defined, and the likelihood score is locally efficient while point-based rewards can tie distinct verdict patterns.

A.2 MAP REWARD DERIVATION AND PROPERTIES

  • The Gaussian-CDF log posterior is strictly concave and tends to −∞ in both tails, so the MAP quality is its unique maximizer.The maximizer is differentiable in the criterion parameters.Source: §A.2 · p. 14

A.3 CRITERION INFORMATION AND REALIZED EVIDENCE

  • Criterion difficulty locates the information peak, while discrimination raises its height quadratically and narrows its quality-scale width in proportion to 1/aj.Expected information peaks at zi = bj; realized likelihood evidence instead grows when the observed verdict has low fitted probability.Source: §A.3 · pp. 15–16, Figure 3 · p. 16

A.4 LOCAL EFFICIENCY OF REWARD AGGREGATION

  • Under conditional independence, the likelihood score is the minimum-variance linear verdict statistic with unit local response to quality.For F = Φ, the efficient coefficient is aj sΦ(uij), not aj alone; duplicate or dependent criteria do not have additive information.Source: §A.4 · p. 17

A.5 ALIGNMENT WITH THE REWARD BASED ON RUBRIC POINTS

  • Rubric-point reward is strictly increasing in quality, but its training signal is guaranteed not to decrease after updates only under first-order stochastic dominance.Outside that condition, the point-based and other training signals can disagree; a location shift is one sufficient special case.Source: §A.5 · pp. 17–18
  • RRT can separate rollouts with equal rubric-point totals, including under the Gaussian CDF when their discrimination-weighted totals also agree.The MAP reward can rank incomparable verdict vectors differently from the point-based reward.Source: §A.5 · p. 19

B METHOD AND OPTIMIZATION DETAILS

The method alternates MAP quality inference from rollout verdicts with RPN calibration using current rollouts. The accompanying analyses examine criterion prediction, reward separation and variation, and how often verdicts are unanimous across rollouts.

B.1 ONLINE EM OBJECTIVE AND ALGORITHMS

  • The online EM loop infers MAP quality from each rollout’s verdicts, applies GRPO, then updates the RPN with detached quality targets.The RPN predicts criterion parameters from prompt and criterion text; the online update uses policy-step rollouts.Source: §B.1 · p. 20

B.2 GRPO OBJECTIVE

  • GRPO centers scalar rollout rewards within each prompt group and uses a clipped PPO-style objective with a KL penalty toward a fixed reference policy.A positive stabilizer is included when reward variance is near zero.Source: §B.2 · p. 21

C.1 TEXT PREDICTION OF CRITERION PARAMETERS

  • The Gaussian-CDF two-parameter model reaches 87.1% macro ROC-AUC, compared with 86.4% for one parameter, 87.0% for three, and 86.6% for four.Across all tested configurations, macro ROC-AUC ranges from 85.6% to 88.7%.Source: §C.1 · p. 22, Table 5 · p. 22

C.2 RPN REPRESENTATION GEOMETRY

  • The RPN representation analysis finds stronger local structure for predicted difficulty and fitted pass probability than for empirical difficulty and observed verdicts.Predicted-versus-empirical difficulty correlations range from 38.0% to 47.9%; Moran’s I is 74.0% to 85.8% for fitted pass probability and 32.9% to 45.9% for observed verdicts.Source: §C.2 · pp. 22, 24, Figure 4 · p. 23, Table 6 · p. 24

C.3 RESPONSE FUNCTION SEPARATION OF TIED ROLLOUTS

  • With matched verdicts and criterion parameters, the Gaussian CDF separates 83.3% of equal-pass-count pairs on Medical and 39.3% on Science, versus 0.0% for the logistic CDF on both.Neither CDF produces count-order reversals.Source: §C.3 · p. 24, Table 7 · p. 24

C.4 REWARD SIGNAL ACROSS POLICIES AND GROUP SIZES

  • Across 12 policy and dataset cells, RRT’s within-prompt variance share is 1.2 to 2.2 times the point-based reward’s, with tied-pair reductions of 2% to 58%.The matched comparison uses the same rollouts and a frozen RPN.Source: §C.4 · p. 24, Figure 5 · p. 25
  • At n = 8, RRT has lower tied-pair shares and higher within-prompt variance shares than point-based reward on both Medical and Science.Medical tied-pair shares are 3.6% versus 5.8%, and Science shares are 19.1% versus 20.6%.Source: §C.4 · p. 24, Table 8 · p. 25

C.5 UNANIMOUS CRITERIA

  • Across the 12 dataset-policy cells, unanimous criterion shares range from 52.3% to 80.5%.The measure is the share of criteria receiving identical verdicts for all rollouts in a group.Source: §C.5 · p. 26, Table 9 · p. 26

C.6 ONLINE EM CALIBRATION

Online RPN calibration improves agreement between predicted pass probabilities and empirical pass rates relative to a frozen RPN. The section also compares hard and soft E-steps and reports repeated-judge agreement and disagreement.

C.6 ONLINE EM CALIBRATION

  • The online RPN gains 1.3 points on checkpoint aggregate macro mean and 1.7 points on next-policy-step macro mean over the frozen RPN.For the next policy step, gains are 2.7 points on Medical and 0.7 points on Science.Source: §C.6 · p. 26, Table 10 · p. 26

C.7 COMPARISON OF HARD AND SOFT E-STEPS

  • On RubricBench, hard-E-step RPNs reach 91.0% to 91.9% ROC-AUC, versus 90.4% to 90.6% for soft-E-step RPNs.Criterion losses are 0.364 to 0.369 for hard E-steps and 0.380 to 0.381 for soft E-steps.Source: §C.7 · p. 26, Table 11 · p. 27

D.1 JUDGE AGREEMENT AND NOISE CALIBRATION

  • Across three independent verdicts, pairwise agreement is 94.6% to 95.2% and unanimous triple share is 91.9% to 92.9%.Verdict prevalence is skewed toward PRESENT, and the reported chance-adjusted agreement measures account for chance agreement differently.Source: §D.1 · p. 27, Table 12 · p. 27
  • In the larger repeated-judging sample, mean disagreement is 1.27% and unanimous triple share is 96.19%.Disagreement is measured against each triple’s majority verdict.Source: §D.1 · pp. 27–28, Table 13 · p. 28

D.2 REWARD STABILITY UNDER JUDGE VARIATION

The section evaluates whether RRT’s reward ordering remains stable across repeated or corrupted judge verdicts. Marginally calibrated RRT improves stable nonzero ordering share in the reported replicate comparison and often has lower order-flip rates under corruption.

D.2 REWARD STABILITY UNDER JUDGE VARIATION

  • Across replicate verdicts, RRT with marginal calibration raises macro stable nonzero ordering share from 54.1% to 59.5%.Its tied-pair share decreases by 5.0 points, and order preservation among separated pairs increases by 2.5 points.Source: §D.2 · p. 28, Table 14 · p. 28
  • The corruption comparison matches channels on expected flips and compares advantages from corrupted verdicts with those from the uncorrupted consensus.It tests three corruption levels and includes a control that spreads the same expected number of flips over all triples.Source: §D.2 · pp. 28–29
  • At corruption level 0.10, marginally calibrated RRT is ahead in 6 of 12 cells for split-verdict corruption and 8 of 12 for all-verdict corruption.Its order-flip rate is lower in 11 and 9 cells, respectively; mean stable-ordering gains range from 0.99 to 2.00 points at every tested level.Source: §D.2 · p. 29, Table 15 · p. 29

D.3 CONTROLLED JUDGE NOISE ROBUSTNESS

The appendix tests reward robustness to structured judge noise, criterion sampling, and residual dependence, then examines policy gains by scale and criterion difficulty. Results show where RRT’s advantages hold and how they vary across settings.

D.3 CONTROLLED JUDGE NOISE ROBUSTNESS

  • At corruption level 0.20, RRT with marginal calibration is ahead in 19 of 19 cells for symmetric corruption on 30% of criteria, but behind in 19 of 19 for lenient response-length corruption.When symmetric corruption covers all criteria, it is behind in 17 cells and ahead in none.Source: Table 16 · p. 30, §D.3 · p. 30
  • At corruption level 0.20 on a fixed 30% criterion subset, RRT with marginal calibration gains 6.0 points on Medical, 7.0 on Science, 4.2 on RaR Science, and 3.0 on RubricBench over criterion score.Source: Table 17 · p. 30, §D.3 · p. 30

D.4 EMPIRICAL CRITERION PASS RATE RELIABILITY

  • Reliability at eight rollouts is 93.6%–96.0% across all criteria and 70.9%–71.9% for uncertain criteria.Two blocks of eight have the same pass count for 61.4%–77.1% of criteria, while uncertain criteria have majority disagreement in 25.5%–31.7% of cases.Source: Table 18 · p. 31, §D.4 · p. 31

D.5 RANK CORRELATION ACROSS CRITERION SPLITS

  • RRT has higher rank correlation than rubric points on all four datasets, gaining 15.1 points on RaR Science and 0.5–3.6 points on the other datasets.Source: Table 19 · p. 31, §D.5 · p. 31
  • RRT’s rank-correlation gain falls from 14 points in the lowest rubric-point rank-correlation quintile to 1 point in the highest.Source: Table 20 · p. 32, §D.5 · p. 32

D.6 LOCAL INDEPENDENCE DIAGNOSTICS

  • Fitted quality explains 72.8%–85.3% of pairwise mutual information; residual redundancy is higher among the most text-similar criteria than the bootstrap null.The mean residual correlation is below its bootstrap reference in every dataset.Source: Table 21 · p. 33, §D.6 · p. 33

E.1 POLICY COMPARISON ACROSS SCALES AND FAMILIES

  • On Qwen3.5-2B, RRT and Vanilla GRPO reach macro criterion scores of 59.2% and 59.4%; on Llama-3.1-8B-Instruct, they reach 53.8% and 53.7%.Their RubricBench criterion scores are 60.8% and 60.6% on Qwen3.5-2B, and 68.8% and 67.8% on Llama-3.1-8B-Instruct.Source: §E.1 · p. 33, Table 22 · p. 34

E.2 POLICY GAINS BY CRITERION DIFFICULTY

  • RRT + online RPN exceeds Vanilla GRPO by 2.8–5.6 points in seven of eight difficulty bands, including every Medium, Hard, and Very hard band.Source: Figure 7 · p. 34, §E.2 · p. 34

E.3 EVALUATION ACROSS BENCHMARKS

The appendix evaluates transfer benchmarks, response length, and criterion selection across budgets. RRT’s macro criterion-score differences from Vanilla GRPO are small across the tested policies, and adaptive Fisher selection reaches the specified reward-fidelity target while leaving some criteria unjudged.

E.3 EVALUATION ACROSS BENCHMARKS

  • Selected checkpoints are evaluated on HealthBench after Medical training and ResearchQA after Science training to measure performance across benchmarks.Source: §E.3 · p. 34
  • RRT’s macro criterion-score gains over the base policy are 1.6, 2.2, and 2.2 points for Qwen3.5-4B, Qwen3.5-2B, and Llama-3.1-8B-Instruct; its macro differences from Vanilla GRPO are 0.7, 0.1, and 0.5 points.Source: §E.3 · p. 34

E.4 RESPONSE LENGTH

  • RRT has lower median response length than Vanilla GRPO in all 12 dataset-policy combinations.On Qwen3.5-2B, the mean of dataset medians grows by 21.7% under RRT and 133.7% under Vanilla GRPO; on Llama-3.1-8B-Instruct, it grows by 5.0% and 18.1%, respectively.Source: Figure 8 · p. 35, §E.4 · p. 36

E.5 CRITERION SELECTION ACROSS CRITERION BUDGETS

  • At the 95.0% reward-fidelity target, adaptive Fisher selection leaves 14.7%–21.7% of criteria unjudged across datasets, versus 4.9%–11.5% for random selection.Static Fisher selection leaves 18.2%–20.7% unjudged across datasets and 19.6% in the macro mean.Source: Figure 9 · p. 36, §E.5 · pp. 36–37

F.1 DATA SOURCES AND SPLITS

The appendix describes the datasets and splits, policy and judge prompts, training configuration, and baseline reward aggregation procedures.

F.1 DATA SOURCES AND SPLITS

  • Medical and Science use RubricHub datasets, RaR Science uses the Science dataset, and RubricBench supplies human-annotated rubrics across five domains; splits keep prompts separate across training, validation, and test.Every training dataset provides multi-criterion rubrics for each prompt, and RubricBench has fewer prompts than the other datasets.Source: §F.1 · p. 37

F.2 POLICY GENERATION AND CRITERION JUDGE PROMPTS

  • The policy receives each dataset row’s message sequence, while judges assess one criterion against the prompt and response without seeing rubric point values.Pitfalls are encoded inversely: PRESENT maps to Gij = 0, and NOT_PRESENT to Gij = 1.Source: §F.2 · pp. 37–38

F.3 TRAINING AND EVALUATION CONFIGURATION

  • The appendix tabulates training and evaluation settings and the selected policy steps; comparisons use a shared training-step limit across conditions.Source: §F.3 · p. 39

F.4 BASELINE REWARD AGGREGATION

  • Algorithm 4 summarizes DIVA and POW3R aggregation for binary verdicts, using criterion points and categories with method-specific weighting and update parameters.For POW3R, λ blends the contrast factor with one, and βema controls its exponential moving average.Source: §F.4 · p. 39

G.1 JUDGE USAGE AND INTERFACE RELIABILITY

The appendix measures judge usage and reliability, policy-step and warm-start costs, and deployment requirements. Partial judging reduces measured costs, while RRT adds no deployment parameters or inference components.

G.1 JUDGE USAGE AND INTERFACE RELIABILITY

  • At criterion budget 0.50, judge requests fall by 49.0% on Medical and Science; all 11,712 completed requests return parseable verdicts, with no retries or ungraded criteria.Endpoint failover adds 11.0% of attempts on Medical and 0.2% on Science.Source: Table 29 · p. 40, §G.1 · pp. 40–41, Table 30 · p. 41

G.2 RRT COMPUTATION AND TIME PER POLICY STEP

  • At criterion budget 0.50, adaptive Fisher selection, relative to full judging with the same frozen RPN, reduces median judging time from 1,457 to 735 seconds on Medical and 1,516 to 771 on Science; total step time falls 23.5% and 28.7%, respectively.Source: §G.2 · p. 42

G.3 COST OF THE RPN WARM START

  • An RPN warm start costs 0.89–5.30 GPU hours with the 0.6B embedder and 1.57–7.64 GPU hours with the 8B embedder.Source: Table 33 · p. 43, §G.3 · p. 43

G.4 DEPLOYMENT REQUIREMENTS

Related papers

Loading 2609.35646v1…