Rubric-based On-policy Distillation

Motivation

• Logit-based OPD requires full teacher output logits, restricting distillation to open-source teachers and excluding high-performance proprietary models.This limitation motivates retaining OPD’s on-policy nature without relying on logit-based signals.Source: §1 · p. 2

Method

• ROPD contrasts teacher and student rollouts to synthesize prompt-specific rubrics, then scores student rollouts against them for on-policy optimization.A Rubricator creates the rubrics and a Verifier provides the scores; the teacher typically performs both roles.Source: §1 · p. 2, Figure 2 · p. 2

Evaluation

• ROPD is evaluated on AIME24/25, HMMT25, GPQA-Diamond, HealthBench, and IFEval with Qwen3-4B and Gemma3-4B students and GPT-5.2 and Qwen3-30B teachers.The evaluation spans diverse benchmarks and model configurations.Source: §1 · p. 2

Results

• In black-box settings, ROPD consistently outperforms existing black-box distillation methods, while in white-box settings it remains highly competitive and often surpasses them.The passage describes these results as establishing a new performance frontier in black-box settings.Source: §1 · p. 2

Interpretation

• ROPD acts as a semantic filter by distilling behavioral gaps into structured rubrics and isolating task-level reasoning principles instead of stochastic token-level phrasing.This semantic guidance also supports cross-architecture distillation without tokenizer alignment.Source: §1 · p. 2

Contribution

• The work presents high-level semantic rubrics as an efficient and robust alternative to fine-grained logits within a complementary framework requiring minimal hyperparameter.It positions clearer semantic guidance as complementary to the prevailing logit-centric distillation landscape.Source: §1 · p. 2

• Traditional white-box OPD uses the teacher’s next-token distribution for dense supervision, but such internal access is often unrealistic for proprietary or API-governed teachers.Source: §2.1 · p. 3
• ROPD has two stages: Rubric Induction extracts criteria from teacher and student responses, and Rubric-based Verification scores student rollouts against them.Source: §2.2 · p. 3
• Each prompt-specific rubric contains textual criteria and positive importance weights, and the same rubric set is shared across all n student rollouts to maintain consistent group rewards.Source: §2.2 · p. 3
• The Verifier evaluates every rollout against every criterion and computes a response-level weighted pass-rate score, which ROPD uses as the reward for on-policy optimization.Source: §2.2 · p. 3
• The teacher typically serves as both Rubricator and Verifier, while replacing these roles with an auxiliary LLM has a marginal impact on final results.Source: §2.2 · p. 3

Black-box setting

• ROPD ranks first across all 14 benchmark configurations and reaches 68.75 on AIME25 (thinking), surpassing the GPT-5.2-chat-latest teacher’s 67.08.The result suggests rubric-augmented optimization can elicit reasoning capabilities beyond direct teacher imitation.Source: §3.2 · p. 4

Efficiency

• 1.6k versus 15.4k samples let ROPD recover LOPD’s best performance, while its 6.3× wall-clock speedup reaches the threshold in 5.5h versus 34.4h.The efficiency advantage persists despite higher per-step overhead from the Rubricator and Verifier.Source: Figure 3 · p. 5, §3.4 · p. 5

White-box setting

• 74.1% of the student-teacher gap is closed by ROPD, compared with 42.1% for LOPD, representing a 1.8× improvement in the white-box setting.ROPD outperforms white-box baselines despite using text-only information.Source: §3.3 · p. 5

Cross-architecture generalization

• ROPD raises Gemma3-it-4B’s AIME24 performance from 6.67% to 10.00%, a +50% relative improvement under identical experimental conditions.This demonstrates robust cross-architecture transferability when replacing the Qwen3-4B student with the less capable Gemma3-it-4B.Source: §3.5 · p. 6

4.1 Qualitative Case Study

• A case study contrasts a rollout with the correct conclusion but no general parity proof against one that fabricates a derivation to guess 337.The example concerns the contradiction n3 +3n2 +2n+1 ≡0 (mod 2024), where parity precludes solutions.Source: §4.1 · p. 6

4.2 Reward Alignment

• ROPD evaluates rubric rewards, teacher logits, and top-24 token overlap against ground-truth correctness using 3,120 AIME24 rollouts.The analysis frames teacher logits as a potentially misaligned proxy for correctness.Source: §4.2 · p. 6
• Rubric rewards filter token-level stochastic noise so optimization prioritizes logical fidelity over surface-form mimicry.ROPD initially increases token overlap rapidly, then shifts toward reasoning correctness, whereas LOPD persistently mimics the teacher distribution.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 Ablations

• ROPD’s core design choices are multi-teacher seeding, cross-rollout rubric sharing, and blind verification.These choices are evaluated with a leave-one-out ablation on AIME24.Source: §4.3 · p. 7, Table 6 · p. 8
• 17.9 points: reducing teacher rollouts from m = 4 to m = 1 causes a catastrophic Pass@1 drop.A single teacher answer over-anchors rubrics to one solution trajectory, whereas diverse strategies support criteria that check correctness across reasoning paths.Source: §4.3 · p. 8
• +3.75 points: using one shared rubric per prompt instead of one per {teacher, student} pair improves Pass@1.Sharing aggregates cross-rollout contrasts and exposes systematic reasoning gaps across the rollout distribution.Source: §4.3 · p. 8

On-policy Distillation

• On-policy distillation replaces sparse rewards with dense feedback on student-generated trajectories, mitigating exposure bias and improving sample efficiency.Existing work advances OPD through objective design, reward extrapolation, training efficiency, signal calibration, and cross-tokenizer methods.Source: §5 · p. 9

Rubric-based Methods

• RLER and SibylSense pioneer evolving rubrics grounded in search evidence or adversarial memory to capture emergent behaviors.These approaches treat rubrics primarily as evaluation instruments.Source: §5 · p. 9
• ROPD repurposes rubrics as a dynamic distillation interface rather than treating them solely as evaluation instruments.This positions rubrics as part of the distillation process instead of only as assessment tools.Source: §5 · p. 9

Evaluation scope

• ROPD has been evaluated mainly on formal reasoning tasks, including Mathematics, Medicine, and Science.IFEval results indicate that general instruction-following is preserved.Source: §6 · p. 9
• Its performance on subjective or creative tasks remains to be established.The paper identifies this as a primary limitation despite preserved general instruction-following on IFEval.Source: §6 · p. 9

System dependencies

• ROPD depends on the instruction-following capabilities of both the Rubricator and Verifier.The paper identifies this dependency as its second primary limitation.Source: §6 · p. 9

7 Conclusion

• ROPD shifts OPD supervision from probabilities to high-level rubrics while maintaining competitive performance and accessibility.The framework is characterized as minimalist yet potent.Source: §7 · p. 9
• 10× greater data-utilization efficiency is reported for ROPD.The conclusion describes this as a boost in data utilization efficiency.Source: §7 · p. 9
• ROPD exhibits superior robustness across disparate model capabilities.The conclusion presents robustness across differing model capabilities as a central finding.Source: §7 · p. 9

Appendix Overview

• The complete related-work discussion is provided in Appendix A.Appendix A is titled “Related Work (Complete Version).”Source: Appendix Overview · p. 13
• Appendix B presents qualitative analysis and case studies.The appendix overview identifies qualitative analysis and case studies as its subject.Source: Appendix Overview · p. 13
• Appendices C and D document hyperparameters, training configuration, and prompt templates.Appendix C covers hyperparameters and training configuration, while Appendix D covers prompt templates.Source: Appendix Overview · p. 13
• Appendix E contains additional figures and analysis.The overview lists this material under Appendix E.Source: Appendix Overview · p. 13
• Appendix F provides algorithm pseudocode and method details.The overview identifies algorithm pseudocode and method details as the focus of Appendix F.Source: Appendix Overview · p. 13

Knowledge distillation and on-policy distillation

• Classical KD matches teacher soft targets on fixed data, while SeqKD substitutes teacher-decoded sequences for label-level targets; both are offline and suffer from exposure bias.SeqKD extends distillation from label-level targets to generation.Source: A Related Work (Complete Version) · p. 14

Black-box On-policy Distillation

• Black-box distillation uses response-level supervision, including mixed-policy preference pairs, discriminator rewards, and discrete verbal trajectory scores.ORPO-Distill uses teacher and student reasoning traces; GAD co-evolves a discriminator-based reward; OVD avoids token alignment.Source: A Related Work (Complete Version) · p. 14

Rubric-based Reinforcement Learning

• RLVR delivers gains in math and code but relies on binary correctness, whereas rubrics decompose response quality into structured multidimensional criteria for open-ended tasks.Rubrics-as-Rewards formalized instance-specific rubrics as on-policy RL rewards and treats RLVR as a special case of rubric-based reinforcement learning.Source: A Related Work (Complete Version) · p. 14

Case Studies

• Teacher disagreement is surfaced explicitly, preventing students from overfitting to one teacher’s style.For example, the Rubricator can report a criterion split evenly between proof by induction and direct computation.Source: B Qualitative Analysis and Case Studies · p. 15
• < 2% of rollouts exhibit rubric exploitation, mainly before step 1k, and the behavior self-corrects with explicit correctness checks.The failure involves high-scoring but substantively incorrect responses, such as formatting tricks or keyword stuffing.Source: B Qualitative Analysis and Case Studies · p. 15

Rubric Item Examples

• The Rubricator generates K = 12 rubric items per instance, with 4 representative examples shown per domain.Table A1 presents examples for different prompt types.Source: Table A1 · p. 15, B Qualitative Analysis and Case Studies · p. 15

C Hyperparameters and Training Configuration

• ROPD experiments use a complete hyperparameter configuration, documented in Table A2.The paper identifies Table A2 as the source of all hyperparameters used in ROPD experiments.Source: C Hyperparameters and Training… · p. 16, Table A2 · p. 16
• Checkpoints are evaluated every 500 steps and selected by track-specific pass@1 metrics, while IFEval uses the math checkpoint without instruction-following fine-tuning.Selection uses AIME24 pass@1 for math, GPQA-Diamond pass@1 for science, and HealthBench pass@1 for medical evaluation.Source: C Hyperparameters and Training… · p. 16
• Sampling uses temperature = 1.0, top-p = 0.95, a 32,768-token output limit, k = 16 responses per problem, and pass@1 reporting.Think mode prepends a standard chain-of-thought prompt, whereas no-think mode generates answers directly.Source: C Hyperparameters and Training… · p. 16

D Prompt Templates

• The Rubricator receives a question, multiple teacher responses, and multiple on-policy student rollouts, then generates one shared rubric applicable to all student responses.Teacher responses collectively represent high-quality behavior despite possible errors or differing approaches, while student rollouts provide actionable improvement targets.Source: D Prompt Templates · p. 17
• Rubric criteria target teacher-strong and student-weak dimensions while remaining response-level, independently verifiable, and safe across valid alternative methods.The design avoids rewarding teacher-specific wording, methods, final answers, or criteria requiring teacher comparison at verification time.Source: D Prompt Templates · p. 17
• The rubric taxonomy covers Task Completion, Observable Quality, and General Reasoning, prioritizing task completion and observable correctness signals that distinguish teachers from students.The prompt requires at least one high-weight criterion for the requested target and emphasizes criteria likely satisfied by teachers but lacking in students.Source: D Prompt Templates · p. 18
• The verifier judges each binary criterion independently from the question, response, and criterion text, accepts equally valid methods, and computes weighted_score and pass_rate.The output schema is black_opd.verifier.v1, with weighted_score = sum of satisfied criterion weights and pass_rate = weighted_score / sum of all weights.Source: D Prompt Templates · pp. 19–20

Leaderboard comparisons

• Figures A1 and A2 provide leaderboard-style comparisons under think and no-think decoding, respectively.Both use a DeepSeek-v4 leaderboard-style horizontal bar-chart format.Source: E Additional Figures and Analysis · p. 22, Figure A1 · p. 22, Figure A2 · p. 22

Analysis protocol

• 3,120 responses form the independent offline analysis pool used for supplementary checkpoint and signal analyses.The pool contains 30 AIME24 prompts × 8 rollouts × 13 checkpoints, sampled independently from the main benchmark.Source: E Additional Figures and Analysis · pp. 22–23

Per-category transitions

• ROPD improves more and regresses less than LOPD in every rubric category.Improvement measures failed-to-passed cells, while regression measures passed-to-failed cells at the final checkpoint.Source: Figure A3 · p. 23, Table A3 · p. 23

Checkpoint dynamics

• ROPD’s accuracy and rubric reward rise together while teacher likelihood falls; LOPD shows weaker coupling among these quantities.Figure A4 normalizes accuracy, rubric reward, and teacher log-likelihood relative to each method’s earliest checkpoint.Source: Figure A4 · p. 24, Table A5 · p. 24

F Algorithm Pseudocode and Method Details

• ROPD accesses the teacher, Rubricator, and Verifier only through text prompts and JSON outputs, without internal logits or hidden states.This establishes the method’s fully black-box operating regime.Source: F Algorithm Pseudocode and Method… · p. 24
• For each prompt, GRPO samples n responses from the old policy and uses response-level rewards to optimize the student on its self-generated rollouts.Teacher-derived supervision constructs the rewards supplied to GRPO.Source: F Algorithm Pseudocode and Method… · pp. 24–25
• Group-relative advantages are normalized within each rollout group, avoiding a separate value model and making updates depend on relative quality for the same prompt.The advantage normalization is A_i = (r_i − r̄) / σ_r.Source: F Algorithm Pseudocode and Method… · p. 24, Algorithm 1 ROPD: Black-box On-policy… · p. 27
• Rubric rewards strongly separate correct from incorrect responses across all three families, whereas teacher average log-likelihood separates weakly or in reverse and top-24 overlap is nearly identical.The signal distributions show rubric reward providing the strongest correctness-conditioned separation.Source: Figure A5 · p. 25
• +0.179 is ROPD final’s accuracy delta, +0.135 its rubric-reward delta, and −0.089 its teacher-log-likelihood delta versus Black step 250 and White step 543.All three reported deltas have bootstrap 95% confidence intervals excluding zero; the comparison uses 30 AIME24 prompts.Source: Figure A6 · p. 26
• Mean teacher top-24 overlap remains between 0.9986 and 0.9996 across checkpoints and families, leaving negligible within-group dynamic range for advantage computation.This saturation is reported to explain why top-24 overlap AUC is near 0.5 for ROPD.Source: Figure A7 · p. 26

Algorithm 1 ROPD: Black-box On-policy Distillation via On-policy Rubrics

• Algorithm 1 samples m teacher answers and n student rollouts, then has the Rubricator generate K weighted criteria shared across rollouts.The rubric is generated from the question, teacher responses, and student responses.Source: Algorithm 1 ROPD: Black-box On-policy… · p. 27
• The Verifier scores every student rollout against the shared rubric, after which GRPO computes group-relative advantages and accumulates the per-question policy gradient.The algorithm then updates parameters as θ ← θ + η · Δθ.Source: Algorithm 1 ROPD: Black-box On-policy… · p. 27