Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

1 Introduction

• GRPO typically rewards only completed responses, leaving little information about why a reasoning trajectory succeeds or fails and creating a credit-assignment bottleneck.The passage gives final-answer correctness and successful execution as examples of outcome-level checks.Source: §1 · p. 1
• Rubrics decompose response quality into explicit criteria, but existing rubric-based training often aggregates those judgments into a scalar reward and discards their detail during optimization.The cited passage notes that rubric-based evaluation can extend post-training beyond strictly verifiable tasks.Source: §1 · pp. 1–2
• Existing on-policy distillation methods provide denser guidance but tie the teacher to privileged model outputs that may not reveal the underlying criteria for a strong response.A rationale can specify one valid reasoning trace without identifying the properties that an answer should satisfy.Source: §1 · p. 2
• RCSD conditions a teacher on criterion-level rubrics to provide token-level guidance on the student’s own sampled trajectories, rather than reducing rubric feedback to scalar rewards.The introduction contrasts this with RL’s sequence-level reward and OPSD’s guidance toward a reference trajectory.Source: §1 · p. 2, Figure 1 · p. 2
• 70.6 overall average; RCSD surpasses GRPO by 1.4 points and OPSD by 0.9 points across diverse reasoning benchmarks.The reported gains are especially pronounced on scientific and rubric-based reasoning tasks.Source: §1 · p. 2

2 Method

• RCSD uses rubrics as privileged teacher-side supervision, converting criterion-level feedback into dense token-level guidance on student-generated trajectories.This preserves distinctions across evaluation dimensions rather than compressing feedback into a scalar reward.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Preliminaries

• Off-policy distillation trains on teacher-generated prefixes, creating a mismatch with inference on student-generated prefixes; on-policy distillation samples student rollouts, while OPSD uses privileged information such as a reference answer.OPSD can still favor a single preferred response and restrict valid reasoning paths.Source: §2.1 · pp. 3–4

2.2 Motivation: Beyond Reward Optimization and Reference-Conditioned Distillation

• Scalar reward optimization offers sparse feedback, while reference-conditioned distillation can encourage global revision when a student deviates from one reference path; RCSD instead conditions the teacher on rubric criteria.The paper also notes that separate judge-based rewards add evaluator cost and may introduce judge bias.Source: §2.2 · p. 4

2.3 Problem Setup

• A rubric is a question-specific set of evaluation criteria, each with a title, natural-language description, and importance weight; rubrics are privileged during training and unavailable to the student at inference.The two-stage setup automates rubric generation because obtaining high-quality instance-specific rubrics can be costly.Source: §2.3 · pp. 4–5

2.4 Stage I: Learning a Rubric Generator

• Stage I trains a rubric generator from the question alone, using a teacher with access to the reference answer to guide rubric generation on the student’s sampled rubric prefixes.The reference answer serves as privileged context rather than a target for direct copying.Source: §2.4 · p. 5

2.5 Stage II: Rubric-Conditioned Reasoning

• In Stage II, the student reasoner sees only the question, while the teacher also receives the learned rubric and provides criterion-aware guidance along the student’s rollout.The rubric lets the teacher signal multiple dimensions of response quality rather than a scalar reward or single reference trajectory.Source: §2.5 · pp. 5–6
• Forward KL distillation is used to cover the rubric-conditioned teacher’s support, retaining probability on alternative valid derivations that more mode-seeking objectives may underrepresent.Rubric criteria define what solutions should satisfy without fixing a single reference path.Source: §2.5 · p. 6

2.6 Training Procedure

• The training procedure first learns a rubric generator and then uses its criteria for teacher-side token-level correction on student rollouts, preserving structured feedback during on-policy learning.Stage I learns what a strong response should satisfy; Stage II learns how to realize those criteria along the student’s trajectory.Source: §2.6 · p. 6

3 Experiments

• The evaluation spans verifiable and non-verifiable science reasoning benchmarks, with additional tests on medical QA and rubric-generation questions.Training data comprise approximately 10k rubric-generation samples and 30k reasoning samples; comparisons include SFT, GRPO, Rubric-GRPO, and OPSD.Source: §3 · pp. 6–7

3.1 Main Results

• RCSD improves 4.7 points over Qwen3-8B and 0.9 point over OPSD on overall average, and reaches 70.8 on SCIBENCH.It also improves over the base model by 8.2 points on RESEARCHQA and 4.9 points on RUBRICHUB.Source: §3.1 · p. 7, Table 1 · p. 7
• Forward KL achieves the highest average at 70.6; reverse KL averages 69.9 and JSD averages 69.6.Forward KL's entropy rises steadily, reverse KL reduces entropy, and JSD produces a milder increase; token lengths are noisy without a clear systematic advantage.Source: §3.1 · pp. 8–9, Table 3 · p. 8, Figure 3 · p. 8
• RCSD improves over its corresponding base model at 1.7B, 4B, and 8B across RAR, RESEARCHQA, and RUBRICHUB.Gains range from 2.7 to 6.6 points at 1.7B, 3.2 to 5.6 at 4B, and 4.8 to 8.9 at 8B.Source: §3.1 · p. 9, Figure 4 · p. 9
• Generated rubrics remain competitive with reference rubrics, with small reported gaps on GPQA-D, SCIBENCH, and PIQA.Generated rubrics average 8.4 criteria versus 7.5 for reference rubrics, with similar average token lengths of 236.6 and 248.7.Source: §3.1 · p. 9, Table 4 · p. 9
• Generic, random, noisy, and reduced rubric variants all improve over the base model, but +Learned achieves the best overall average.The strong +Reduced result suggests concise relevant criteria retain much of the benefit, while the +Learned advantage indicates rubric relevance and coherence matter.Source: §3.1 · p. 10, Table 6 · p. 10

LLM Post-training

• Outcome-based RL assigns one score to an entire response, losing information about where and how reasoning failed.Critique-based supervision and process reward models offer finer-grained alternatives, though they are mainly developed for domains such as mathematics.Source: §4 · p. 10
• On-policy distillation uses token-level teacher guidance on student-sampled trajectories, while prior self-distillation typically conditions the teacher on privileged reference solutions or textual feedback.RCSD instead conditions the teacher on structured rubric feedback rather than a single reference trajectory.Source: §4 · p. 11

Reinforcement Learning with Rubrics

• Rubric-based RL aggregates criterion judgments into scalar rewards for completed responses, so rubric structure does not guide token-level learning on the model’s own trajectory.Some approaches also rely on predefined rubrics, often produced by frontier LLMs, which are costly to obtain.Source: §4 · p. 11

5 Conclusion

• RCSD uses rubrics as privileged teacher supervision for on-policy self-distillation, converting criterion-level feedback into dense token-level guidance on student-generated rollouts.The paper presents this as a verifier-free approach for open-ended reasoning when exact-match answers, executable tests, or other automatic outcome verifiers cannot always judge response quality.Source: §5 · p. 11

A Implementation Details

• Standard GRPO uses a fixed Qwen3-14B judge conditioned on the prompt, completion, and reference answer.The judge returns a 1–10 quality score normalized to [0, 1] as the reward.Source: A Implementation Details · p. 15
• GRPO-Rubrics first generates rubrics offline from each training example’s question and reference answer, then conditions the reward judge on the prompt, completion, and rubric.The judge supplies a holistic score reflecting rubric satisfaction rather than direct comparison with the gold text.Source: A Implementation Details · p. 15
• Table 7 is titled “Training Configurations.”The supplied passage identifies the table’s subject but gives no configuration values.Source: Table 7 · p. 15

B.1 Rubric Generator Prompt

• Rubrics are generated in structured JSON, with title, description, and weight fields and category-prefixed descriptions.Descriptions should not copy large blocks of the question or reference answer; the generator receives both as inputs.Source: B Prompt Templates · p. 15, §B.1 · pp. 15–16
• Each rubric item should be self-contained for non-expert readers, and the reference answer is guidance rather than necessarily an exhaustive response.The rubric prompt asks for the ideal answer, including specific facts and explanations, alongside the full question.Source: §B.1 · pp. 15–16
• The generator is instructed to produce 5–15 rubric items, with weights of 1–5 for positive categories and -1 or -2 for pitfalls.Essential criteria cover critical facts or safety checks; the prompt also distinguishes important, optional, and pitfall criteria.Source: §B.1 · pp. 15–16
• The prompt specifies answer-choice phrasing, when to require a stated conclusion or prior reasoning, and when to reward concise responses or mention of specific findings.It also requests a JSON array containing exactly three evaluation criteria.Source: §B.1 · p. 16

B.2 LLM-as-a-Judge Prompt

• The evaluator is prompted to rate overall response quality from 1 to 10 according to the supplied rubrics, considering them holistically.Responses that violate multiple rubrics should score lower, while those satisfying all rubrics should score higher.Source: §B.2 · p. 16
• The response should be a JSON object with a single key "rating" whose value is an integer between 1 and 10; it rates overall response quality on a scale of 1 to 10 based on rubric satisfaction.Source: §B.2 · pp. 16–17
• Figure 5 tracks performance across training checkpoints on GPQA-D and SciBench.The caption specifies the benchmarks and checkpoint comparison but no outcome values.Source: Figure 5 · p. 17

• On GPQA-D, performance peaks at 120 steps; on SCIBENCH, it improves early and remains relatively stable across later checkpoints.The authors describe the GPQA-D pattern as suggesting moderate training is most effective, and the SCIBENCH pattern as showing no substantial degradation with continued training.Source: C Experiments · p. 17

• RCSD produces the strongest trajectory: it is shortest, settles on a coherent numerical setup early, and reaches a correct answer without revisiting earlier choices.Source: §C.1 · p. 19
• OPSD also reaches a correct answer, but repeatedly rechecks established arithmetic, increasing token use without improving solution quality.Source: §C.1 · pp. 18–19
• The base QWEN3-8B response repeatedly changes the gas constant and oscillates between final answers, despite using the correct formula and unit conversions.Source: §C.1 · pp. 18–19
• All three responses identify the ideal gas law, but differ substantially in verbosity, consistency, and numerical discipline.Source: §C.1 · pp. 17, 19

• In the phonon example, the learned rubric expands to 10 criteria versus 7 in the reference rubric and repeats overlapping temperature and frequency requirements.The reference rubric instead consolidates related content into a smaller set of criteria, including Temperature Analysis, Mathematical Derivation, and Frequency Trends.Source: §C.2 · p. 21
• In the soda-lime example, learned rubrics include broadly reasonable but weakly task-critical items, while the reference focuses on chemically essential checks.The reported examples include Step-by-Step Explanation and Application Context versus Separate Reactions, Stoichiometry Accuracy, Final Volume Accuracy, and Unit Consistency.Source: §C.2 · p. 21
• Across these examples, learned rubrics usually identify the topic but can be verbose, overlapping, and less aligned with the true grading objective.Source: §C.2 · p. 21

D Additional Theoretical Details

• The proposition assumes that −log q_t(y) is bounded by B_t for tokens y in the support of p⋆.Source: D Additional Theoretical Details · p. 21
• The proof applies the per-step bound across decoding steps, sums it, and takes expectation over student rollouts to obtain the stated result.Source: D Additional Theoretical Details · p. 22
• Generated rubrics need only induce similar rubric-conditioned teacher distributions on student-visited prefixes; their wording need not match reference rubrics.This preserves the teacher-side guidance used in Stage II.Source: D Additional Theoretical Details · p. 22