PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers

MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang, Utkarsh Tyagi, Advait Gosai, Razvan-Gabriel Dumitru, Aakash Sabharwal, Bing Liu, Yunzhong He

arXiv:2606.12507v1cs.LG

TL;DR

Open-ended rubric-based training relies on LLM verifiers, which add training cost and provide sparse feedback. RGSD distills a rubric-conditioned teacher into an unconditioned student, achieving quality parity with judge-based GRPO across medical and science evaluations. It is a complementary option when verifier cost or reliability is a concern.

  • Problem

    Open-ended tasks lack programmatic verifiers, and LLM rubric grading adds per-rollout cost, inherits grader biases, and provides weak token-level credit assignment.

    Source: §1 · p. 1

  • Method

    RGSD trains an unconditioned student to match token-level distributions from a frozen copy of the base model conditioned on the prompt and rubric.

    Source: §3.2 · p. 5, §6 · p. 11

  • Results

    RGSD achieves quality parity with GRPO across domains: +6.1 versus +5.9pp on medical evaluations and +4.9 versus +4.5pp on science.

    Source: §5 · p. 7

  • Takeaways & Limitations

    RGSD is complementary to verifier-based RL and is most useful when rubric conditioning gives the base model a meaningful response lift.

    Source: Limitations · p. 11

    Most main-table runs use a single seed, so small margins are interpreted as parity rather than statistically significant wins.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.

1 Introduction

Rubric-based training for open-ended tasks relies on LLM verifiers, which add cost and provide sparse feedback. RGSD instead distills a rubric-conditioned teacher into an unconditioned student, achieving quality comparable to judge-based GRPO across the evaluated models and domains.

1 Introduction

  • Open-ended tasks lack applicable programmatic verifiers, while LLM grading adds per-rollout cost and reduces multi-criterion feedback to a scalar reward.Verifier-derived rewards also inherit grader biases, and their weak token-level guidance makes credit assignment across long responses difficult.Source: §1 · pp. 1–2
  • RGSD samples one prompt-only student rollout and trains it against next-token distributions from a frozen, rubric-conditioned copy of the base model.A clipped Jensen–Shannon distillation loss replaces the external judge in the training loop.Source: Figure 1 · p. 2, §1 · p. 2
  • Rubric-conditioned base responses score 30–45pp higher than unconditioned responses, motivating the conditioning lift as a signal RGSD can transfer.The rubric-conditioning gap compares the same base model with and without the prompt-specific rubric.Source: §1 · p. 2, Table 1 · p. 3
  • RGSD improves over the base by +6.1 percentage points versus +5.9 for GRPO on medical evaluations, and +4.9 versus +4.5 on science.The authors interpret the results as quality parity, not a consistent accuracy advantage for either method.Source: §1 · pp. 2–3
  • On Qwen-2.5-7B-Instruct, RGSD reaches peak quality at roughly half GRPO’s response length.The reported length comparison is specific to this model.Source: §1 · p. 2
  • Ablations find that raw rubrics enrich the teacher more than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings.The stronger-judge comparison retains the per-rollout verifier loop.Source: §1 · pp. 2–3

2 Related Work

Related work spans rubric-based evaluation and reward optimization, judge-specific vulnerabilities, and on-policy distillation. RGSD combines rubric conditioning with self-distillation as an alternative to verifier-based rewards.

2 Related Work

  • Rubric-based RL uses multi-criterion judgments as scalar rewards, while related work also automates rubric construction or adjusts criteria during training.Rubrics as Rewards (Gunjal et al., 2025) aggregates rubric judgments for GRPO; Viswanathan et al. (2025) combines LLM judges with verifier programs.Source: §2 · p. 3
  • Mahmoud et al. (2026) report that policies optimized against one rubric judge can exploit its preferences and score worse with a cross-family reference panel.Their verifier-free diagnostic uses a self-internalization gap correlated with reference-panel scores; RGSD instead uses rubric conditioning as a training signal.Source: §2 · p. 3
  • On-policy distillation trains on student-visited states, with prior work extending it from external teachers and arbitrary divergences to self-distillation under privileged contexts.Lu and Thinking Machines (2025) use an external stronger teacher; Zhao et al. (2026) describe On-Policy Self-Distillation with differently conditioned teacher and student copies.Source: §2 · p. 4
  • RGSD uses rubrics as the teacher’s enrichment signal, combining dense on-policy distillation with self-distillation for open-ended domains.The authors position it as an alternative to verifier-based rewards where external stronger teachers or ground-truth reasoning traces are unavailable.Source: §2 · p. 4

3 Method

RGSD uses rubric-conditioned predictions from a frozen copy of the base model to train an unconditioned student token-by-token, without an LLM judge in the training loop. Its dense supervision is applied to final-response tokens to limit potential rubric leakage from reasoning traces.

3.1 Preliminaries

  • Rubrics comprise weighted criteria, and an LLM judge returns binary criterion verdicts that are aggregated into a rubric score.The paper uses a single batched judge call for the full set of criterion verdicts.Source: §3.1 · p. 4

3.2 Rubric-Guided Self-Distillation

  • RGSD samples student rollouts, then trains the student to match a frozen, rubric-conditioned teacher’s per-token distributions under clipped Jensen–Shannon divergence.The teacher and student start from the same base checkpoint; only the student receives gradients.Source: §3.2 · p. 5
  • Every rollout token contributes a divergence term, replacing sparse trajectory-level reward with dense supervision and requiring no judge or reward model during training.The clipped Jensen–Shannon divergence interpolates between forward and reverse KL; the paper uses β = 0.5.Source: §3.2 · p. 5
  • Reasoning-model teacher traces may refer to rubric criteria, so the method masks tokens between <think> and </think> from the loss.On Qwen3-Thinking, masking improved results by 2–4pp versus applying loss to the entire trace.Source: §3.2 · pp. 5–6

4 Experimental Setup

The experiments compare RGSD and judge-based GRPO on medical and science tasks using four base models, in-domain and out-of-distribution evaluations, and matched training settings. Figure 2 tracks rubric satisfaction and response length during medical training.

4 Experimental Setup

  • Training covers 12,519 medical and 19,806 science prompts, with Qwen-2.5 Instruct models (3B, 7B) and Qwen3-Thinking models (4B, 8B).Evaluation uses 300-prompt held-out RubricHub subsets and 300-prompt out-of-distribution subsets from HealthBench and ResearchQA.Source: §4 · p. 6
  • GRPO uses 16 rollouts per prompt and a gpt-4o-mini training judge; RGSD uses one rollout and no training-time judge calls.The methods are otherwise matched on data, base model, optimizer, and schedule.Source: §4 · p. 6
  • Figure 2 plots rubric satisfaction above mean response length by base model; RGSD avoids severe Qwen-2.5 verbosity drift while reaching comparable or higher medical scores.For Qwen3-Thinking, RGSD often scores higher but is not uniformly shorter.Source: Figure 2 · p. 7

5 Results and Discussion

RGSD achieves quality parity with judge-based GRPO across the reported medical and science evaluations while eliminating training-time judge calls. The section also examines response length, rubric enrichment, and the conditions under which a stronger GRPO judge can outperform RGSD.

5 Results and Discussion

  • RGSD and GRPO are at quality parity: medical gains average +6.1pp vs. +5.9pp, and science gains +4.9pp vs. +4.5pp, respectively.Per-model differences are mixed, rather than showing a consistent accuracy advantage for either method.Source: §5 · p. 7
  • GRPO requires 80 batched judge calls per training prompt in the five-epoch, 16-rollout setup, while RGSD makes no training-time judge calls and samples one rollout per prompt per epoch.The reported setup corresponds to approximately 1.0M calls for the medical split and 1.6M for the science split.Source: §5 · p. 7
  • Length effects are family-dependent: on science, RGSD scores higher while remaining longer than the base, whereas GRPO shortens relative to the Qwen3-Thinking base.The Qwen3-Thinking models already produce longer base responses because they include reasoning traces; RGSD is therefore not uniformly shorter on these models.Source: Figure 3 · p. 8, §5 · p. 8
  • On the Qwen-2.5-7B medical run, GRPO rose from 15.8 to 38.4 claims per response and from 30.5% to 45.1% false-claim rate; RGSD rose from 15.8 to 25.8 claims and 30.5% to 35.1%.Because the audit is LLM-based, the authors treat it as supporting evidence rather than a primary factuality benchmark.Source: §5 · p. 8
  • On Qwen-2.5, GRPO peaks before training ends and then regresses, while RGSD continues improving on medical and stays at or near its science peak.Qwen-2.5-7B GRPO drops 1.3pp on medical and 2.7pp on science from peak to final; RGSD stays within 0.8pp of its science peak. The authors note RGSD’s gains remain bounded by teacher quality.Source: §5 · pp. 8–9
  • The rubric-conditioning gap is larger in medical than science, and RGSD recovers a correspondingly larger share of that lift in the reported benchmarks.RGSD gains average +8.8pp on RubricHub-med, ∼20% of the conditioning lift, and +3.5pp on RubricHub-sci, ∼11% of the lift; the authors describe this as a lightweight, imperfect diagnostic.Source: §5 · p. 9

5.1 Ablations

  • On Qwen-2.5 medical, raw-rubric RGSD peaks at 27.4/36.8 rubric satisfaction for 3B/7B, versus 25.3/34.0 for the self-golden variant.The authors interpret this as a coverage effect: a single generated response may omit rubric criteria, while the rubric preserves the full criteria set; both variants avoid GRPO’s severe late-stage length growth, but self-golden is longer than raw-rubric RGSD.Source: §5.1 · pp. 9–10, Figure 4 · p. 9
  • With gpt-oss-120b, GRPO nearly matches RGSD on Qwen-2.5-7B medical (36.7 vs. 36.8 at peak) and surpasses it on science (39.2 vs. 35.9).The stronger-judge runs still require per-rollout verifier calls and produce longer final responses: roughly 1,200 versus 700 tokens on medical and 1,200 versus 900 on science, compared with RGSD.Source: §5.1 · p. 10, Figure 5 · p. 10

6 Conclusion

RGSD uses a rubric-conditioned base model as a frozen teacher to distill token-by-token into an unconditioned student, replacing external rollout scoring. Across four models and two domains, it reaches performance comparable to judge-based GRPO with fewer rollouts and no training-time judge calls, while its relationship to GRPO depends on the setting.

6 Conclusion

  • RGSD distills the rubric-conditioned teacher distribution into the unconditioned student token-by-token, turning sparse scalar rubric rewards into dense per-token signals without expensive verifier calls.Source: §6 · p. 11
  • Across four base models and two domains, RGSD achieves comparable performance to judge-based GRPO with one on-policy rollout per prompt and no training-time judge calls, versus sixteen rollouts for GRPO.The authors describe the methods as complementary rather than claiming RGSD uniformly wins.Source: §6 · p. 11
  • On Qwen-2.5 models, RGSD reaches comparable rubric satisfaction with substantially shorter responses than GRPO; an auxiliary audit suggests GRPO’s additional tokens can introduce more unsupported claims.The authors note that length behavior is more domain-dependent on Qwen3-Thinking models, whose base responses already contain long reasoning traces.Source: §6 · p. 11
  • RGSD is attractive when verifier calls are costly or difficult and the base model shows a large rubric-conditioning response, while stronger-judge GRPO may be preferable when its additional verifier cost is justified.Source: §6 · p. 11

Limitations

The evaluation spans multiple models and domains, but most main-table runs use a single seed because exhaustive replication is expensive. RGSD’s suitability also depends on rubric-conditioning behavior, evaluation remains judge-dependent, and reasoning-trace models require a specific masking adjustment.

Limitations

  • Most main-table runs are single-seed because exhaustive GRPO replication is expensive; the authors therefore interpret small margins as parity, not statistically significant wins.A three-seed envelope was run for the headline Qwen-2.5-7B medical configuration alongside a grid of four models, two domains, and in- and out-of-distribution benchmarks.Source: Limitations · p. 11
  • RGSD transfers behavior the base model can express with rubric context, so it is most useful when rubric-conditioning response is meaningful; a stronger affordable verifier can make GRPO the better choice.The conditioning gap can be measured with a forward-only validation pass before training.Source: Limitations · p. 11
  • Evaluation still uses a stronger held-out LLM judge, so absolute rubric-satisfaction scores may retain judge-specific biases.Source: Limitations · pp. 11–12
  • The HealthBench factual-claim audit supports the response-length analysis and is not presented as a standalone factuality benchmark.Source: Limitations · p. 12
  • For Qwen3-Thinking, the teacher may reference rubrics in its thinking trace while the student lacks rubrics at inference, so the method masks thinking-block tokens and distills final-answer tokens.The authors describe this as a practical recipe choice for reasoning-trace bases, not a conceptual requirement for standard instruction-tuned models.Source: Limitations · p. 12

A Implementation Details

The appendix documents the training and evaluation setup used to reproduce the main results, including the data, matched method configurations, and computing environment.

A Implementation Details

  • RGSD and GRPO match effective batch size, learning rate, schedule, and optimizer steps; both use full fine-tuning, batch size 128, learning rate 4.2e-6, and five epochs.The listed optimizer steps are 490 for medical and 775 for science; method-specific entries are marked not applicable where needed.Source: A Implementation Details · pp. 15–16
  • RGSD uses distillation β 0.5, JSD clip τ 0.05, and top-128 teacher vocabulary entries for the loss; GRPO uses 16 rollouts per prompt and KL coefficient 0.01.For Qwen3-Thinking, thinking-block tokens are excluded from the loss; top-128 restriction is required for Qwen3-8B-Thinking to fit the stated hardware setup.Source: A Implementation Details · p. 16
  • Each run uses one 8×NVIDIA H100 node with ZeRO-3; RGSD uses vLLM-backed on-policy rollouts, while the GRPO training judge is called per rollout through a LiteLLM gateway.The evaluation judge is gpt-5.4; the default GRPO training judge is gpt-4o-mini, with gpt-oss-120b used in the judge-strength ablation.Source: A Implementation Details · p. 16
  • Training uses full RubricHub train splits: 12,519 medical prompts and 19,806 science prompts, with disjoint 300-prompt held-out validation subsets for evaluation.The self-golden ablation additionally uses one rubric-conditioned base response per prompt, generated at temperature 1.0 and top-p 0.95.Source: A Implementation Details · p. 16

B Additional Results

The appendix gathers additional analyses covering reasoning-trace masking, seed variation, cross-family generalization, and factual-claim auditing.

B Additional Results

  • The appendix includes a thinking-token mask ablation and a case study of rubric leakage in the teacher’s thinking trace.Source: B Additional Results · p. 17
  • It reports a multi-seed envelope for the headline base-model and domain configuration and a cross-family generalization study.Source: B Additional Results · p. 17
  • It includes a factual-claim audit of RGSD and GRPO responses on HealthBench-300.Source: B Additional Results · p. 17

B.1 Thinking-Token Mask Ablation

The thinking-token mask improves the primary score by excluding rubric-leaking traces from distillation. The appendix also reports tight seed variation on the headline configuration and improvement on two additional model families.

B.1 Thinking-Token Mask Ablation

  • Masking the thinking trace outperforms leaving it in on the primary score for both Qwen3-Thinking sizes.Without masking, roughly 70% of generated tokens are thinking traces, pulling the student toward the teacher’s thinking style rather than its answer distribution.Source: §B.1 · p. 17
  • Rubric references appear in 70.2% of Qwen3-4B-Thinking and 73.0% of Qwen3-8B-Thinking medical teacher traces, versus 0.7% and 0.3% in unconditioned controls.The corresponding science rates are 41.1% and 44.5% for conditioned traces, versus 0.0% for both unconditioned controls.Source: §B.1 · p. 17
  • Because students never see the rubric, distilling traces that enumerate criteria can teach them to fabricate references to criteria they cannot access.The reported leakage is common despite the teacher template instructing the model to address criteria naturally without explicitly referencing them.Source: §B.1 · pp. 17–18, Figure 6 · p. 18
  • The mask-on/off comparison is clean within each model, but enrichment differs across sizes: Qwen3-4B uses rubric enrichment and Qwen3-8B uses self-golden enrichment.No rubrics-without-mask Qwen3-8B run was conducted.Source: §B.1 · p. 18

B.2 Multi-Seed Envelope on the Headline Cell

  • On Qwen-2.5-7B-Instruct medical, the three-seed envelope has peak primary-score variability of about 0.4 percentage points.The HealthBench band is wider but remains well above the base.Source: §B.2 · p. 18, Figure 8 · p. 19

B.3 Cross-Family Generalization

  • Both OLMo-2-1124-7B-Instruct and Mistral-7B-Instruct-v0.3 improve in rubric satisfaction from their unconditioned starting points under RGSD.Matching GRPO baselines were not run, so this experiment tests transfer beyond Qwen families rather than comparing methods.Source: §B.3 · p. 19

B.4 Factual-Claim Audit on HealthBench-300

The factual-claim audit compares claim volume and false-claim rates in RGSD and GRPO responses on HealthBench-300. GRPO’s claim count and false-claim rate rise more sharply, while RGSD remains lower on both measures after step 0.

B.4 Factual-Claim Audit on HealthBench-300

  • The audit extracts distinct verifiable claims from each response and has gpt-5.4 verify them against parametric medical knowledge without web search.Incorrect and fabricated claims are pooled to calculate the false-claim rate across 300 HealthBench prompts.Source: §B.4 · pp. 19–20
  • After step 0, RGSD produces fewer claims, fewer false claims, and a lower false-claim percentage than GRPO at every checkpoint.GRPO’s false-claim rate rises from 30.5% to 45.1%; RGSD’s rises from 30.5% to 35.1%. GRPO’s mean claims per prompt grows from 15.8 to about 38–40, while RGSD saturates around 25–28.Source: Table 4 · p. 20, §B.4 · p. 20
  • The authors interpret the divergence as GRPO’s training judge rewarding rubric satisfaction even when claims are unsupported, while RGSD distills the rubric-conditioned base distribution.They report that RGSD inherits the base model’s claim-density and accuracy profile rather than receiving a scalar judge reward that can favor additional claims.Source: §B.4 · pp. 20–21

C Prompts and Data Examples

The appendix provides the teacher and judge prompts alongside sampled training instances. It specifies what each model receives and how rubric judgments become per-prompt rewards.

C Prompts and Data Examples

  • The appendix reproduces teacher and judge prompts and includes one sampled training instance from each domain for inspection of possible conditioning-channel leakage.Source: C Prompts and Data Examples · p. 21

C.1 Teacher Input Template

  • The student receives only the user question, while the teacher receives that question with the full rubric in a single user-turn message.The same domain-agnostic template is used for medical and science; its closing transition prompt asks the teacher to generate a fresh rubric-conditioned response.Source: §C.1 · p. 21
  • The self-golden ablation replaces the teacher’s rubric block with a pre-generated reference response from the same base model.Source: §C.1 · p. 21, Figure 11 · p. 22

C.2 Judge Prompt

  • At evaluation, gpt-5.4 judges each criterion; GRPO training uses gpt-4o-mini by default and gpt-oss-120b in the judge-strength ablation.Both use the weight-normalized fraction of satisfied criteria as the per-prompt reward.Source: §C.2 · p. 21
  • The judge prompt structure is shared between evaluation and GRPO training, with per-criterion verdicts aggregated by Equation 1 into a per-prompt reward.Source: §C.2 · p. 21, Figure 12 · p. 22

C.3 Sample Training Instances

  • The sampled medical and science instances show the question and highest-weighted rubric criteria; the student sees only the question, while the teacher gets the full rubric.The displayed instances contain 23 medical criteria and 22 science criteria, respectively.Source: §C.3 · p. 22, Figure 13 · p. 23, Figure 14 · p. 23
Loading 2606.12507v1…