PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric-based On-policy Distillation

Junfeng Fang, Zhepei Hong, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Xiang Wang, Tat-Seng Chua

arXiv:2605.07396v1cs.LGcs.AI

TL;DR

Logit-based on-policy distillation requires teacher logits, limiting it to white-box teachers. ROPD replaces those signals with prompt-specific semantic rubrics and outperforms logit-based methods across most scenarios, with up to a 10× gain in sample efficiency.

  • Problem

    Logit-based OPD requires full teacher logits, restricting distillation to open-source white-box teachers and motivating alternatives based only on teacher-generated responses.

  • Method

    ROPD contrasts teacher and student rollouts to induce prompt-specific rubrics, then verifies student rollouts against them to provide rewards for on-policy optimization.

  • Results

    ROPD outperforms existing black-box methods and often surpasses logit-based OPD across diverse benchmarks and model configurations, achieving up to a 10× gain in sample efficiency.

  • Takeaways & Limitations

    Rubric-based OPD provides a flexible, efficient, and black-box-compatible alternative to logit-based distillation for proprietary and open-source LLMs.

  • Takeaways & Limitations

    Evaluation mainly covers formal reasoning tasks, so performance on subjective or creative tasks remains unestablished.

Abstract

from arXiv · show

On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.

1 Introduction

ROPD addresses the white-box limitation of logit-based on-policy distillation by using prompt-specific semantic rubrics derived from teacher–student rollout contrasts. It validates this approach across diverse benchmarks and model configurations, outperforming black-box distillation methods while remaining competitive in white-box settings.

  • Motivation: Logit-based OPD requires full teacher output logits, restricting distillation to open-source teachers and excluding high-performance proprietary models.This limitation motivates retaining OPD’s on-policy nature without relying on logit-based signals.
  • Method: ROPD contrasts teacher and student rollouts to synthesize prompt-specific rubrics, then scores student rollouts against them for on-policy optimization.A Rubricator creates the rubrics and a Verifier provides the scores; the teacher typically performs both roles.
  • Evaluation: ROPD is evaluated on AIME24/25, HMMT25, GPQA-Diamond, HealthBench, and IFEval with Qwen3-4B and Gemma3-4B students and GPT-5.2 and Qwen3-30B teachers.The evaluation spans diverse benchmarks and model configurations.
  • Results: In black-box settings, ROPD consistently outperforms existing black-box distillation methods, while in white-box settings it remains highly competitive and often surpasses them.The passage describes these results as establishing a new performance frontier in black-box settings.
  • Interpretation: ROPD acts as a semantic filter by distilling behavioral gaps into structured rubrics and isolating task-level reasoning principles instead of stochastic token-level phrasing.This semantic guidance also supports cross-architecture distillation without tokenizer alignment.
  • Contribution: The work presents high-level semantic rubrics as an efficient and robust alternative to fine-grained logits within a complementary framework requiring minimal hyperparameter.It positions clearer semantic guidance as complementary to the prevailing logit-centric distillation landscape.

2 Method

ROPD enables black-box on-policy distillation by converting teacher and student responses into shared, prompt-specific rubrics, then using rubric-based verification to reward student rollouts. The verified scores support on-policy optimization, while Rubricator and Verifier roles can be fulfilled by the teacher or an auxiliary LLM.

  • Traditional white-box OPD uses the teacher’s next-token distribution for dense supervision, but such internal access is often unrealistic for proprietary or API-governed teachers.
  • ROPD has two stages: Rubric Induction extracts criteria from teacher and student responses, and Rubric-based Verification scores student rollouts against them.
  • Each prompt-specific rubric contains textual criteria and positive importance weights, and the same rubric set is shared across all n student rollouts to maintain consistent group rewards.
  • The Verifier evaluates every rollout against every criterion and computes a response-level weighted pass-rate score, which ROPD uses as the reward for on-policy optimization.
  • The teacher typically serves as both Rubricator and Verifier, while replacing these roles with an auxiliary LLM has a marginal impact on final results.

3 Main Result

ROPD consistently outperforms competing distillation approaches across black-box and white-box settings, while achieving substantial sample and compute-efficiency gains. It also transfers across student architectures and can exceed the teacher on AIME25 (thinking).

  • Black-box setting: ROPD ranks first across all 14 benchmark configurations and reaches 68.75 on AIME25 (thinking), surpassing the GPT-5.2-chat-latest teacher’s 67.08.The result suggests rubric-augmented optimization can elicit reasoning capabilities beyond direct teacher imitation.
  • Efficiency: 1.6k versus 15.4k samples let ROPD recover LOPD’s best performance, while its 6.3× wall-clock speedup reaches the threshold in 5.5h versus 34.4h.The efficiency advantage persists despite higher per-step overhead from the Rubricator and Verifier.
  • White-box setting: 74.1% of the student-teacher gap is closed by ROPD, compared with 42.1% for LOPD, representing a 1.8× improvement in the white-box setting.ROPD outperforms white-box baselines despite using text-only information.
  • Cross-architecture generalization: ROPD raises Gemma3-it-4B’s AIME24 performance from 6.67% to 10.00%, a +50% relative improvement under identical experimental conditions.This demonstrates robust cross-architecture transferability when replacing the Qwen3-4B student with the less capable Gemma3-it-4B.

4 Analysis

Section 4 analyzes why ROPD succeeds, showing that rubric rewards provide more discriminative, correctness-aligned supervision than scalar or token-level signals. Ablations further identify multi-teacher coverage and cross-rollout rubric sharing as essential design choices.

  • 4.1 Qualitative Case Study: A case study contrasts a rollout with the correct conclusion but no general parity proof against one that fabricates a derivation to guess 337.The example concerns the contradiction n3 +3n2 +2n+1 ≡0 (mod 2024), where parity precludes solutions.
  • 4.2 Reward Alignment: ROPD evaluates rubric rewards, teacher logits, and top-24 token overlap against ground-truth correctness using 3,120 AIME24 rollouts.The analysis frames teacher logits as a potentially misaligned proxy for correctness.
  • 4.2 Reward Alignment: Rubric rewards filter token-level stochastic noise so optimization prioritizes logical fidelity over surface-form mimicry.ROPD initially increases token overlap rapidly, then shifts toward reasoning correctness, whereas LOPD persistently mimics the teacher distribution.
  • 4.3 Ablations: ROPD’s core design choices are multi-teacher seeding, cross-rollout rubric sharing, and blind verification.These choices are evaluated with a leave-one-out ablation on AIME24.
  • 4.3 Ablations: 17.9 points: reducing teacher rollouts from m = 4 to m = 1 causes a catastrophic Pass@1 drop.A single teacher answer over-anchors rubrics to one solution trajectory, whereas diverse strategies support criteria that check correctness across reasoning paths.
  • 4.3 Ablations: +3.75 points: using one shared rubric per prompt instead of one per {teacher, student} pair improves Pass@1.Sharing aggregates cross-rollout contrasts and exposes systematic reasoning gaps across the rollout distribution.

5 Related Work

Prior on-policy distillation improves feedback density, exposure-bias mitigation, and sample efficiency, while recent rubric-based methods use evolving rubrics to capture emergent behaviors. ROPD differs by repurposing rubrics from evaluation instruments into a dynamic distillation interface.

  • On-policy Distillation: On-policy distillation replaces sparse rewards with dense feedback on student-generated trajectories, mitigating exposure bias and improving sample efficiency.Existing work advances OPD through objective design, reward extrapolation, training efficiency, signal calibration, and cross-tokenizer methods.
  • Rubric-based Methods: RLER and SibylSense pioneer evolving rubrics grounded in search evidence or adversarial memory to capture emergent behaviors.These approaches treat rubrics primarily as evaluation instruments.
  • Rubric-based Methods: ROPD repurposes rubrics as a dynamic distillation interface rather than treating them solely as evaluation instruments.This positions rubrics as part of the distillation process instead of only as assessment tools.

6 Limitation and Future Work

ROPD’s evaluation primarily covers formal reasoning tasks, leaving its effectiveness on subjective or creative tasks unresolved. The framework also depends on the instruction-following abilities of its Rubricator and Verifier.

  • Evaluation scope: ROPD has been evaluated mainly on formal reasoning tasks, including Mathematics, Medicine, and Science.IFEval results indicate that general instruction-following is preserved.
  • Evaluation scope: Its performance on subjective or creative tasks remains to be established.The paper identifies this as a primary limitation despite preserved general instruction-following on IFEval.
  • System dependencies: ROPD depends on the instruction-following capabilities of both the Rubricator and Verifier.The paper identifies this dependency as its second primary limitation.

7 Conclusion · Appendix · Appendix Overview

The paper concludes that ROPD is a minimalist rubric-based OPD framework that replaces probability supervision with high-level rubrics, achieving 10× greater data-utilization efficiency and robustness across differing model capabilities. The appendices provide complete related work, analyses, configurations, prompts, figures, and algorithmic details.

  • 7 Conclusion: ROPD shifts OPD supervision from probabilities to high-level rubrics while maintaining competitive performance and accessibility.The framework is characterized as minimalist yet potent.
  • 7 Conclusion: 10× greater data-utilization efficiency is reported for ROPD.The conclusion describes this as a boost in data utilization efficiency.
  • 7 Conclusion: ROPD exhibits superior robustness across disparate model capabilities.The conclusion presents robustness across differing model capabilities as a central finding.
  • Appendix Overview: The complete related-work discussion is provided in Appendix A.Appendix A is titled “Related Work (Complete Version).”
  • Appendix Overview: Appendix B presents qualitative analysis and case studies.The appendix overview identifies qualitative analysis and case studies as its subject.
  • Appendix Overview: Appendices C and D document hyperparameters, training configuration, and prompt templates.Appendix C covers hyperparameters and training configuration, while Appendix D covers prompt templates.
  • Appendix Overview: Appendix E contains additional figures and analysis.The overview lists this material under Appendix E.
  • Appendix Overview: Appendix F provides algorithm pseudocode and method details.The overview identifies algorithm pseudocode and method details as the focus of Appendix F.

A Related Work (Complete Version)

Prior work spans offline knowledge distillation, black-box on-policy distillation, and rubric-based reinforcement learning. These approaches address teacher-student transfer, response-level supervision, and structured rewards for open-ended tasks through distinct mechanisms.

  • Knowledge distillation and on-policy distillation: Classical KD matches teacher soft targets on fixed data, while SeqKD substitutes teacher-decoded sequences for label-level targets; both are offline and suffer from exposure bias.SeqKD extends distillation from label-level targets to generation.
  • Black-box On-policy Distillation: Black-box distillation uses response-level supervision, including mixed-policy preference pairs, discriminator rewards, and discrete verbal trajectory scores.ORPO-Distill uses teacher and student reasoning traces; GAD co-evolves a discriminator-based reward; OVD avoids token alignment.
  • Rubric-based Reinforcement Learning: RLVR delivers gains in math and code but relies on binary correctness, whereas rubrics decompose response quality into structured multidimensional criteria for open-ended tasks.Rubrics-as-Rewards formalized instance-specific rubrics as on-policy RL rewards and treats RLVR as a special case of rubric-based reinforcement learning.

B Qualitative Analysis and Case Studies

The qualitative analysis highlights how ROPD handles disagreement among teachers and mitigates a rare rubric-exploitation failure mode. Rubric examples illustrate the Rubricator’s prompt-specific outputs.

  • Case Studies: Teacher disagreement is surfaced explicitly, preventing students from overfitting to one teacher’s style.For example, the Rubricator can report a criterion split evenly between proof by induction and direct computation.
  • Case Studies: < 2% of rollouts exhibit rubric exploitation, mainly before step 1k, and the behavior self-corrects with explicit correctness checks.The failure involves high-scoring but substantively incorrect responses, such as formatting tricks or keyword stuffing.
  • Rubric Item Examples: The Rubricator generates K = 12 rubric items per instance, with 4 representative examples shown per domain.Table A1 presents examples for different prompt types.

C Hyperparameters and Training Configuration · D Prompt Templates

The paper specifies complete ROPD training and evaluation settings, including checkpoint selection, sampling, and reasoning modes. Its prompt templates construct shared, black-box-compatible rubrics from teacher–student contrasts and verify each rollout independently with binary criteria and weighted scores.

  • C Hyperparameters and Training Configuration: ROPD experiments use a complete hyperparameter configuration, documented in Table A2.The paper identifies Table A2 as the source of all hyperparameters used in ROPD experiments.
  • C Hyperparameters and Training Configuration: Checkpoints are evaluated every 500 steps and selected by track-specific pass@1 metrics, while IFEval uses the math checkpoint without instruction-following fine-tuning.Selection uses AIME24 pass@1 for math, GPQA-Diamond pass@1 for science, and HealthBench pass@1 for medical evaluation.
  • C Hyperparameters and Training Configuration: Sampling uses temperature = 1.0, top-p = 0.95, a 32,768-token output limit, k = 16 responses per problem, and pass@1 reporting.Think mode prepends a standard chain-of-thought prompt, whereas no-think mode generates answers directly.
  • D Prompt Templates: The Rubricator receives a question, multiple teacher responses, and multiple on-policy student rollouts, then generates one shared rubric applicable to all student responses.Teacher responses collectively represent high-quality behavior despite possible errors or differing approaches, while student rollouts provide actionable improvement targets.
  • D Prompt Templates: Rubric criteria target teacher-strong and student-weak dimensions while remaining response-level, independently verifiable, and safe across valid alternative methods.The design avoids rewarding teacher-specific wording, methods, final answers, or criteria requiring teacher comparison at verification time.
  • D Prompt Templates: The rubric taxonomy covers Task Completion, Observable Quality, and General Reasoning, prioritizing task completion and observable correctness signals that distinguish teachers from students.The prompt requires at least one high-weight criterion for the requested target and emphasizes criteria likely satisfied by teachers but lacking in students.
  • D Prompt Templates: The verifier judges each binary criterion independently from the question, response, and criterion text, accepts equally valid methods, and computes weighted_score and pass_rate.The output schema is black_opd.verifier.v1, with weighted_score = sum of satisfied criterion weights and pass_rate = weighted_score / sum of all weights.

E Additional Figures and Analysis

The appendix extends the main analysis with leaderboard comparisons, per-category transition results, and supplementary reward-signal and checkpoint-dynamics analyses. ROPD improves more and regresses less than LOPD across every rubric category, while its accuracy and rubric reward rise together as teacher likelihood falls.

  • Leaderboard comparisons: Figures A1 and A2 provide leaderboard-style comparisons under think and no-think decoding, respectively.Both use a DeepSeek-v4 leaderboard-style horizontal bar-chart format.
  • Analysis protocol: 3,120 responses form the independent offline analysis pool used for supplementary checkpoint and signal analyses.The pool contains 30 AIME24 prompts × 8 rollouts × 13 checkpoints, sampled independently from the main benchmark.
  • Per-category transitions: ROPD improves more and regresses less than LOPD in every rubric category.Improvement measures failed-to-passed cells, while regression measures passed-to-failed cells at the final checkpoint.
  • Checkpoint dynamics: ROPD’s accuracy and rubric reward rise together while teacher likelihood falls; LOPD shows weaker coupling among these quantities.Figure A4 normalizes accuracy, rubric reward, and teacher log-likelihood relative to each method’s earliest checkpoint.

F Algorithm Pseudocode and Method Details · Algorithm 1 ROPD: Black-box On-policy Distillation via On-policy Rubrics

ROPD performs fully black-box on-policy distillation by generating prompt-specific rubrics from teacher and student responses, then using verifier scores as group-relative rewards for GRPO updates. Algorithm 1 alternates teacher sampling, student rollouts, rubric construction, verification, advantage normalization, and policy optimization.

  • F Algorithm Pseudocode and Method Details: ROPD accesses the teacher, Rubricator, and Verifier only through text prompts and JSON outputs, without internal logits or hidden states.This establishes the method’s fully black-box operating regime.
  • F Algorithm Pseudocode and Method Details: For each prompt, GRPO samples n responses from the old policy and uses response-level rewards to optimize the student on its self-generated rollouts.Teacher-derived supervision constructs the rewards supplied to GRPO.
  • F Algorithm Pseudocode and Method Details: Group-relative advantages are normalized within each rollout group, avoiding a separate value model and making updates depend on relative quality for the same prompt.The advantage normalization is A_i = (r_i − r̄) / σ_r.
  • F Algorithm Pseudocode and Method Details: Rubric rewards strongly separate correct from incorrect responses across all three families, whereas teacher average log-likelihood separates weakly or in reverse and top-24 overlap is nearly identical.The signal distributions show rubric reward providing the strongest correctness-conditioned separation.
  • F Algorithm Pseudocode and Method Details: +0.179 is ROPD final’s accuracy delta, +0.135 its rubric-reward delta, and −0.089 its teacher-log-likelihood delta versus Black step 250 and White step 543.All three reported deltas have bootstrap 95% confidence intervals excluding zero; the comparison uses 30 AIME24 prompts.
  • F Algorithm Pseudocode and Method Details: Mean teacher top-24 overlap remains between 0.9986 and 0.9996 across checkpoints and families, leaving negligible within-group dynamic range for advantage computation.This saturation is reported to explain why top-24 overlap AUC is near 0.5 for ROPD.
  • Algorithm 1 ROPD: Black-box On-policy Distillation via On-policy Rubrics: Algorithm 1 samples m teacher answers and n student rollouts, then has the Rubricator generate K weighted criteria shared across rollouts.The rubric is generated from the question, teacher responses, and student responses.
  • Algorithm 1 ROPD: Black-box On-policy Distillation via On-policy Rubrics: The Verifier scores every student rollout against the shared rubric, after which GRPO computes group-relative advantages and accumulates the per-question policy gradient.The algorithm then updates parameters as θ ← θ + η · Δθ.
Loading 2605.07396v1…