PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-wise Rubric Rewards for LLM Reasoning

Weichu Xie, Haozhe Zhao, Wenpu Liu, Yongfu Zhu, Liang Chen, Minghao Ye, Zirong Chen, Yuqi Xu, Shuai Dong, Ziyue Wang, Xinbo Xu, Kean Shi, Ruoyu Wu, Xiaoying Zhang, Wenqi Shao, Baobao Chang, Nan Duan, Jiaqi Wang

arXiv:2605.17291v1cs.LG

TL;DR

Response-level rubric rewards lose criterion and step structure, while giving uniform credit to correct and incorrect steps and allowing self-correction loops to inflate rewards. SRaR attributes rubric items to steps, normalizes step scores across rollouts, and decouples rubric and outcome advantages. It improves benchmark accuracy and AIME 2025 faithful reasoning while reducing looping; its judgments still depend on an external LLM judge.

  • Problem

    Rubric-based RL aggregates multiple criteria into a response-level scalar, losing criterion and step structure and applying uniform rewards to correct and incorrect steps.

    Source: §1 · p. 2

  • Method

    SRaR uses an LLM judge to attribute rubric items to reasoning steps, normalizes per-step scores across rollouts, and adds them as bounded adjustments to a separately normalized outcome advantage.

    Source: §1 · p. 3, §3.3 · p. 6

  • Results

    3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B: SRaR outperforms the state-of-the-art in average accuracy across six mathematical benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    On AIME 2025, Faithful Reasoning Rate rises from 34.5% to 46.7% and self-correction looping falls from 48.1% to 26.5%, alongside improved final-answer accuracy and intermediate reasoning quality.

    Source: §1 · p. 3

    SRaR depends on an external LLM judge, whose individual-sample attribution or satisfaction errors may introduce noise into training.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.

1 Introduction

The introduction identifies weaknesses in response-level rubric rewards and presents SRaR as a step-level alternative. It reports accuracy gains across benchmarks, alongside improvements in faithful reasoning and reduced self-correction looping.

1 Introduction

  • Response-level rubric rewards discard criterion and step structure, apply the same signal to correct and incorrect steps, and can reward self-correction loops.In 1,000 rollouts, 18.2% of steps in answer-correct responses were wrong, while 49.9% of steps in answer-incorrect responses were correct.Source: §1 · pp. 2–3
  • SRaR attributes rubric items to reasoning steps, normalizes their scores across rollouts, and adds them through a decoupled advantage estimator.The estimator keeps rubric noise out of the outcome baseline, bounding rubric influence on the overall advantage.Source: §1 · p. 3
  • The authors construct a 16K-problem rubric corpus by contrasting correct and flawed reasoning paths sampled from GPT-5.The resulting rubric items are classified as SUGGEST, PITFALL, or BONUS and target observed failure modes.Source: §1 · p. 3
  • Across six mathematical benchmarks, SRaR improves average accuracy over the strongest baseline on both Qwen3-8B and Qwen3-32B.The reported gains are consistent across benchmarks and model scales.Source: §1 · p. 3
  • On AIME 2025, step-attributed judging with per-step token assignment raises Faithful Reasoning Rate from 34.5% to 46.7%, while step-level credit assignment reduces self-correction looping from 48.1% to 26.5%.Source: §1 · p. 3

2 Related Work

The related work contrasts process-supervision signals with interpretable rubric evaluation, then identifies trajectory-level aggregation as a remaining limitation of rubric-based RL.

2.1 Process Supervision for Reasoning

  • Process-supervision methods offer either step scores that require extensive human annotation or step signals that remain opaque.PRMs require massive human annotations; GRPO-VPS uses the policy’s conditional probability of the correct answer at step boundaries but provides an opaque scalar.Source: §2.1 · p. 4

2.2 Rubric-Based Evaluation and Rewards

  • Rubric-based RL uses interpretable criteria, but existing methods aggregate them into a trajectory-level scalar that loses step-level credit.SRaR addresses this by attributing rubric items to reasoning steps and normalizing rewards at the step level.Source: §2.2 · pp. 4–5

3 Method

The method combines outcome-based RL with rubric rewards attributed to individual reasoning steps. It defines typed rubric criteria, normalizes step rewards across rollouts, decouples outcome and rubric advantages, and builds rubrics through contrastive distillation.

3.1 Preliminary

  • GRPO normalizes scalar rewards across rollouts and broadcasts the resulting advantage to every token, while SRaR formulates training around problems, rubrics, and response step spans.Each problem carries a ground-truth answer and a rubric; responses are organized into token spans for reasoning steps.Source: §3.1 · p. 5

3.2 Rubrics Design

  • The rubric schema separates SUGGEST items for standard reasoning, PITFALL items for known errors, and BONUS items for exceptional insights.Each type contributes a distinct, bounded policy-gradient signal.Source: §3.2 · p. 5

3.3 Step-wise Rubrics Rewards in RLVR (SRaR)

  • An LLM judge links each rubric item to the reasoning step it evaluates, producing per-step, per-criterion verdicts rather than one response-level score.The policy uses structured step formatting, and tokenizer offsets recover the step token spans.Source: §3.3 · p. 5
  • Rubric verdicts become type-budgeted signed deltas, which are summed into raw rubric rewards for their attributed steps.The per-type budget bounds each category’s contribution regardless of rubric-list length.Source: §3.3 · p. 6
  • Step rewards are normalized across rollouts of the same prompt and broadcast only to tokens in the judged step.Consistently good or bad steps receive near-zero advantage, while steps whose quality varies receive a stronger signal; a step represented by only one rollout receives zero.Source: §3.3 · pp. 5–6
  • The decoupled estimator adds a bounded per-step rubric offset to the separately normalized outcome advantage, keeping rubric noise out of the outcome baseline.The outcome reward combines accuracy and format rewards; the paper states this design prevents rubric gaming from inflating overall advantage.Source: §3.3 · p. 6

3.4 Contrastive Rubric Distillation

  • Contrastive distillation builds a 16K-problem rubric corpus by sampling GPT-5 trajectories, verifying them against ground truth, and contrasting correct and flawed paths.The resulting criteria target errors a policy is likely to commit during training.Source: §3.4 · p. 6

4 Experiment

The experiments compare SRaR with established RL and rubric-reward baselines across six mathematical benchmarks, then examine performance by benchmark length and problem difficulty, intermediate-step quality, looping, and design ablations.

4.1 Experimental Setup

  • The experimental setup uses Qwen3-8B and Qwen3-32B, six math benchmarks, and comparisons with GRPO, DAPO, RaR, GRPO-VPS, and RGR-GRPO.Training follows DAPO’s setting except for the reward signal; GPT-OSS-20B serves as the rubric judge, with Qwen3-30B-A3B-Instruct used to validate judge robustness.Source: §4.1 · p. 7

4.2.1 Main Results

  • SRaR reaches average accuracy of 72.53 on Qwen3-8B and 75.81 on Qwen3-32B, leading RaR by 3.57 and 2.75 points, respectively.It also exceeds GRPO-VPS by 10.85/14.41 points and RGR-GRPO by 9.29/8.08 points across the two scales.Source: §4.2.1 · p. 7, Table 1 · p. 8

4.2.2 Think Longer and Better

  • At 8B, gains are largest on AIME 2025 (+5.83), Olympiad Bench (+5.64), and AMC 2023 (+4.53), while MATH500 improves by +1.80.The benchmark-level gain over RaR correlates with average reasoning steps (Pearson r = 0.91, p = 0.012); on Olympiad Bench, the gap grows from +0.7 pp in Q1 to +13.4 pp in Q5, where SRaR responses are shorter (3,645 vs. 5,933 tokens).Source: §4.2.1 · p. 8, §4.2.2 · p. 8, Figure 3 · p. 9

4.3.1 Process-Outcome Alignment of Step-wise Rewards

  • On AIME 2025, SRaR achieves 85.2% step accuracy and 46.7% answer accuracy, with no correct-answer trajectories containing erroneous steps.Its Faithful Reasoning Rate is 46.7% vs. 34.5% for RaR, and its Misaligned Trajectory Rate is 10.0% vs. 20.7%.Source: §4.3.1 · p. 9, Table 2 · p. 9

4.3.2 Reduction of Repetitive Looping

  • On AIME 2025, SRaR reduces repetitive looping from RaR’s 48.1% to 26.5%.Looping responses have 42.5% accuracy under SRaR versus 24.2% under RaR; SRaR also has fewer mean self-correction phrases (14.8 vs. 25.4).Source: §4.3.2 · p. 9, Table 3 · p. 10

4.4 Ablation Study

  • On Qwen3-8B, Suggest-only SRaR improves over RaR by +2.86, while multi-aspect SRaR improves by +3.57 and raises accuracy from 71.39 to 72.53.Removing step-structured formatting yields 67.85 average accuracy; removing cross-rollout normalization causes collapse to 56.76, while merging the decoupled advantage yields 72.06 versus full SRaR’s 72.53.Source: §4.4 · p. 10

5 Conclusion

SRaR applies rubric rewards at the reasoning-step level, using step-attributed judging, cross-rollout normalization, and a decoupled advantage estimator. Across six mathematical benchmarks, it improves accuracy over the state of the art and raises faithful reasoning while reducing self-correction looping.

5 Conclusion

  • Step-attributed rubric judging and cross-rollout normalization provide step-wise rewards, while a decoupled advantage estimator keeps training stable against rubric noise.Together, these designs deliver fine-grained supervision during RL training.Source: §5 · p. 10
  • Across six mathematical benchmarks, SRaR exceeds the state of the art by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B.It raises Faithful Reasoning Rate from 34.5% to 46.7% and reduces self-correction looping from 48.1% to 26.5%.Source: §5 · p. 10
  • The conclusion emphasizes that the granularity at which supervision is applied matters as much as its content.Source: §5 · p. 10

A Limitations

The section lists limitations concerning judge reliability, inference cost, domain scope, rubric generation, and step boundaries, then provides prompts and training hyperparameters.

A Limitations

  • SRaR depends on an LLM judge, adds per-rollout inference cost, is tested only on math, relies on GPT-5 for rubric generation, and requires explicit step headers.The authors note individual judge errors can add reward noise; costs scale with rollouts and rubric items, and rubric generation may need domain adaptation.Source: A Limitations · p. 13

B.1 Rollout Generation Prompt

  • Rollout-generation instructions request step-by-step solutions with numbered step headers and a final answer enclosed in tags.Source: §B.1 · p. 13

B.2 Rubric Distillation Prompt

  • The rubric prompt requires concise, verifiable checkpoints, with SUGGEST and ANSWER mandatory and PITFALL and BONUS optional.It limits rubrics to eight lines and excludes formatting checks.Source: §B.2 · p. 14

B.3 Training Format Prompt

  • The training-format prompt asks for numbered reasoning steps and a final answer inside \boxed{}.Source: §B.3 · p. 14

B.4 LLM Judge Prompt

  • The LLM judge prompt evaluates each rubric item and assigns it to the most closely associated numbered step, or to a whole response or no step.Its verdict rules distinguish satisfied SUGGEST, PITFALL, and BONUS items.Source: §B.4 · p. 15

B.5 Step-Level Judge Prompt

  • The step-level judge prompt asks whether the current reasoning step has correct operations and logic, and requires a CORRECT or INCORRECT verdict.Source: §B.5 · p. 16

C Training Hyperparameters

  • Table 4 is identified as the presentation of training hyperparameters.The supplied caption does not specify individual settings.Source: Table 4 · p. 16

D Training Dynamics

The section reports training dynamics, format compliance, rubric quality, judge agreement, and associations between rubric scores and answer correctness.

D Training Dynamics

  • At convergence, SRaR reaches validation accuracy of ∼0.47, compared with RaR (∼0.43) and DAPO (∼0.42), despite similar response-length trajectories to RaR.All three methods converge to similar training reward accuracy (∼0.80–0.83).Source: D Training Dynamics · pp. 16–17, Figure 4 · p. 17

E Format Compliance During Training

  • The step-header format stays near 100% throughout training, while occasional boxed-answer compliance drops are attributed to truncation at maximum generation length.Source: Figure 5 · p. 17

F Rubric Quality Evaluation

  • 98.0% of 51,583 rubric items were judged valid, with type-specific validity rates of 96.3%–98.5%; human–AI agreement on a 100-item cross-check was 92%.Source: F Rubric Quality Evaluation · pp. 17–18, Table 5 · p. 18
  • Suggest, Bonus, and Pitfall scores rise during training, with the Pitfall score moving toward zero.The figure describes these trends as increased adherence to suggested steps, discovery of alternative strategies, and avoidance of common errors, respectively.Source: Figure 6 · p. 18

G LLM Judge Agreement Analysis

Agreement on Rubric Satisfaction

Rubric Scores and Answer Correctness

  • Correct trajectories satisfy 74.2% of Suggest items versus 24.7% for incorrect ones, trigger 2.5% of Pitfall items versus 24.2%, and satisfy 12.5% of Bonus items versus 0%.All three rubric aspects are significantly correlated with answer correctness (p < 0.01).Source: Rubric Scores and Answer Correctness · p. 19, Table 7 · p. 19

H Judge Model Robustness

SRaR retains gains with an alternative LLM judge, and case studies contrast its structured reasoning with RaR’s looping or unstructured responses. The rubric examples show the structure of the distilled criteria.

H Judge Model Robustness

  • +1.90 average improvement over RaR (70.86 vs. 68.96) with Qwen3-30B-A3B-Instruct as judge; judge choice affects improvement magnitude, not viability.The comparison is on Qwen3-8B-Non-Thinking.Source: H Judge Model Robustness · p. 20, Table 8 · p. 20

I Rubric Examples

  • Representative training-set examples pair each problem with Suggest, Pitfall, Bonus, and Answer rubric items from the three-stage pipeline.The examples illustrate the generated rubrics’ structure and quality.Source: I Rubric Examples · p. 20

J.1 Process--Outcome Alignment Case Study

  • Across two AIME 2025 cases, RaR loops or truncates without an answer, while SRaR produces structured solutions, including a correct answer in one case.The cases include a geometry solution and a chair-selection problem; RaR’s chair-problem response contains 63 self-correction phrases and no final answer.Source: §J.1 · pp. 21–22

J.2 Looping Case Study

  • On a difficult AIME 2025 problem neither model solves, RaR repeatedly restarts and produces no answer, while SRaR maintains a two-step structure and reaches a definite but incorrect answer.The case illustrates that structured convergence can occur without solving the problem.Source: §J.2 · p. 23

K Data-Size Sensitivity

Accuracy rises as synthetic rubric training data scales from 1k to 16k across three benchmarks. An ablation also finds gains from both step-wise credit assignment and multi-aspect rubrics.

K Data-Size Sensitivity

  • At 16k samples, accuracy reaches 48.12% on AIME 2024, 37.19% on AIME 2025, and 87.34% on AMC 2023, with upward trends across all three benchmarks.The 1k-to-16k trend is reported under a one-epoch training setup using fixed-seed random subsets.Source: K Data-Size Sensitivity · p. 23, Figure 7 · p. 23

L Rubric Aspect Ablation

  • SRaR improves over RaR by +2.86 with Suggest-only rubrics and +3.57 with full rubrics; full rubrics raise SRaR from 71.39 to 72.53 (+1.14).The ablation is on Qwen3-8B-Non-Thinking; the authors attribute the primary improvement to step-wise credit assignment and find PITFALL and BONUS complementary.Source: L Rubric Aspect Ablation · p. 24, Table 9 · p. 24

M Method Design Ablation

Ablations show that step normalization and decoupled advantage estimation matter to SRaR’s performance, while explicit step formatting adds no measurable penalty. Removing normalization can collapse training.

M Method Design Ablation

  • Without step normalization, performance collapses to 56.76 average as the model exploits uniformly rewarded rubric items by putting its reasoning into one step.Cross-rollout normalization zeros advantages for rubric items whose scores do not vary across rollouts.Source: M Method Design Ablation · pp. 24–25, Table 10 · p. 24
  • Explicit step formatting yields 67.85 average versus 67.26 for the full DAPO baseline, a comparison described as virtually identical and without measurable performance penalty.Step boundaries are needed to attribute rubric items to individual reasoning steps.Source: M Method Design Ablation · p. 24, Table 10 · p. 24
  • Decoupled advantage estimation reaches 72.53 average versus 72.06 for the merged-scalar variant; on AIME 2025, the scores are 49.06 versus 46.67.The authors suggest merging outcome and rubric signals before allocation loses the benefit of a stable outcome baseline.Source: Table 10 · p. 24, M Method Design Ablation · p. 25

N Case Study: Training Collapse without Step Normalization

A case study links unnormalized rubric rewards to single-step collapse, while the theoretical analysis shows that outcome and step-wise rubric advantages are independently zero-sum across rollouts.

N Case Study: Training Collapse without Step Normalization

  • Without cross-rollout normalization, uniformly satisfied rubric items retain positive advantages, and a representative collapsed response puts more than 4,000 tokens of reasoning into one step.The case analysis contrasts this with full SRaR, where normalization leaves only discriminative rubric items to shape the policy gradient.Source: N Case Study: Training Collapse… · p. 25

O.1 Zero-Sum Properties

  • The SRaR advantage decomposes into independently zero-sum outcome and per-step rubric components across rollouts.The outcome component sums to zero over all rollouts, and each step’s normalized rubric signal sums to zero over rollouts containing that step.Source: §O.1 · p. 26

O.2 Bounded Reward Analysis

The appendix bounds raw rubric rewards, analyzes how step-wise signals combine with outcome rewards, and establishes conditions for variance control and estimator unbiasedness. It also reports the composition of the 16,560-problem rubric dataset and distributions of rubric items per sample.

O.2 Bounded Reward Analysis

  • Each rollout’s total raw rubric reward is bounded by [RPIT, RSUG + RBON], from the PITFALL, SUGGEST, and BONUS reward budgets.SUGGEST and BONUS contributions lie in [0, RSUG] and [0, RBON], while PITFALL lies in [RPIT, 0].Source: §O.2 · pp. 26–27

O.3 Reduction to Standard GRPO

  • With empty rubrics, SRaR’s advantage estimator reduces to the standard GRPO outcome advantage broadcast across tokens.No rubric deltas are produced, leaving only the base reward at the final token.Source: §O.3 · p. 27

O.4 Policy Gradient Decomposition

  • The policy-gradient decomposition separates the standard response-wide GRPO outcome term from a step-selective rubric correction.The correction scales token log-probabilities within each evaluated step according to that step’s normalized rubric signal.Source: §O.4 · pp. 27–28

O.5 Variance Analysis

  • Step-wise normalization gives each step’s rubric signal unit population variance, matching the outcome advantage’s scale and limiting any one step’s gradient contribution.Without normalization, raw rubric-delta magnitudes can vary with rubric-item number and type, potentially dominating or becoming negligible relative to the outcome advantage.Source: §O.2 · p. 27, §O.5 · p. 28

O.6 Unbiasedness of the Policy Gradient Estimator

  • The estimator is unbiased when its baseline is independent of the rollout’s action; including that rollout in group statistics yields bias O(1/n), which vanishes as group size grows.The leave-one-out interpretation computes baseline statistics from the other n − 1 rollouts.Source: §O.6 · pp. 28–29

P Dataset Statistics

  • The dataset contains 16,560 math problems with SUGGEST, PITFALL, BONUS, and ANSWER rubric tags; accompanying tables cover overall statistics, tag counts, and Pitfall–Bonus co-occurrence.The supplied table captions identify these scopes but do not provide cell values.Source: P Dataset Statistics · p. 29, Table 11 · p. 29, Table 12 · p. 29

P.3 Distribution of Total Rubric Item Count; P.4 Distribution of Suggest Count; P.5 Distribution of Pitfall Count; P.6 Distribution of Bonus Count

  • Most samples contain 7–8 rubric items; pitfall warnings usually number 1–2, bonus insights usually number one, and suggest-item counts indicate 4–5 solution steps.The reported shares are 84.6% for 7–8 total items, 97.3% for 1–2 pitfall warnings, and 67.1% for one bonus insight.Source: Figure 8 · p. 30, Figure 9 · p. 30, Figure 10 · p. 31
Loading 2605.17291v1…