PaperlayerView exact revision on arXiv

Source-linked AI summary

PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang

arXiv:2603.11178v3cs.AIcs.LG

TL;DR

Standard distillation wastes training on problems at competence extremes, where gradient SNR collapses. PACED estimates pass rates from student rollouts and applies Beta-kernel weighting, achieving broad benchmark gains with low forgetting; a two-stage KL schedule improves results further.

  • Problem

    Standard distillation weights all problems uniformly even though mastered problems are redundant and intractable problems yield noisy gradients.

  • Method

    PACED weights each problem by a Beta-kernel function of the student’s empirical pass rate, using student rollouts without architectural changes or hyperparameters.

  • Results

    PACED generalizes across Qwen3, Qwen2.5, and Llama-3 families, improving over unweighted distillation by up to +8.2 while keeping forgetting at 1.4% and 0.6% in distillation and self-distillation.

  • Takeaways & Limitations

    A two-stage forward-KL-then-reverse-KL schedule pushes gains to +5.8 over standard forward KL on the hardest benchmark.

  • Takeaways & Limitations

    Pass-rate estimation adds rollout overhead, and with K=8 the discretized estimates can assign zero weight to problems whose true pass rate is near zero.

Abstract

from arXiv · show

Standard LLM distillation treats all training problems equally -- wasting compute on problems the student has already mastered or cannot yet solve. We empirically show that this inefficiency has a precise gradient-level signature: the cross-problem gradient signal-to-noise ratio (SNR) follows a bell curve over student pass rate, collapsing at both extremes. We propose PACED, which weights each problem by $w(p) = p(1{-}p)$ where $p$ is the student's empirical pass rate -- concentrating training on the zone of proximal development. This requires only student rollouts, no architectural changes, and no hyperparameters. We prove the Beta kernel $w(p) = p^α(1{-}p)^β$ is the leading-order optimal weight family arising from the SNR boundary-collapse structure, and is minimax-robust under misspecification (worst-case efficiency loss $O(δ^2)$). Across Qwen3, Qwen2.5, and Llama-3 families, PACED sets a new state of the art in our experimental setting on MATH-500, AIME~2024, and AIME~2025, improving over unweighted distillation by up to $\mathbf{+8.2}$ and over the strong AKL baseline by up to $\mathbf{+3.6}$, while reducing forgetting to $\mathbf{1.4\%}$ and $\mathbf{0.6\%}$ in distillation and self-distillation. A two-stage forward-then-reverse KL schedule pushes gains further to $\mathbf{+5.8}$ over standard forward KL on the hardest benchmark.

1 Introduction

Standard distillation wastes training on mastered and intractable problems. PACED uses competence-dependent weighting, theoretically motivated by bell-shaped gradient SNR, and reports gains with low forgetting.

  • Uniform distillation wastes compute because mastered problems provide redundant signal, while intractable problems produce noisy gradients that can erode learned knowledge.
  • Cross-problem gradient SNR forms a bell curve over student pass rate, collapsing near both p≈0 and p≈1 and peaking at intermediate competence.At low pass rates gradients are directionally incoherent; at high pass rates problem-specific refinements disperse and cancel.
  • PACED weights problems with w(p) = p(1−p), concentrating training on the zone of proximal development using only student rollouts.The approach requires no architectural changes or hyperparameters.
  • The Beta kernel w(p) = p^α(1−p)^β is derived as the leading-order optimal family and is minimax-robust under bounded misspecification.The stated worst-case efficiency loss is O(δ^2).
  • PACED improves over unweighted distillation by up to +8.2 on AIME 2025 while keeping MMLU forgetting ≤1.4% across Qwen and Llama families.
  • A two-stage forward-then-reverse KL schedule adds gains of +4.6, +4.9, and +5.8 over standard forward KL on MATH-500, AIME 2024, and AIME 2025.

2 Related Work

Prior work commonly treats samples uniformly or relies on fixed curricula, whereas PACED adapts problem priority from student pass rates and operates orthogonally to loss and generation choices.

  • Knowledge-distillation methods share a common tendency to treat all samples alike despite differing student competence.
  • Curriculum and self-paced methods typically use fixed difficulty annotations or predetermined schedules, unlike PACED’s pass-rate-based adaptation.
  • PACED’s closed-form pass-rate weighting avoids per-sample gradient computation and is orthogonal to parameter-constraint approaches for forgetting mitigation.
  • PACED determines which problems to prioritize independently of the distillation loss or generation policy.
  • Table 1 summarizes the design features distinguishing PACED from representative prior methods.

3 Methodology

PACED estimates student competence from rollouts, derives Beta-kernel weights from boundary-collapsing gradient SNR, and applies them to forward- or reverse-KL distillation.

  • 3 Methodology: PACED directs distillation toward problems where the student can benefit most, using competence-aware problem weighting as its core idea.
  • 3.1 Problem Setup: The teacher is frozen while the student is updated in both distillation and self-distillation settings.
  • 3.1 Problem Setup: An expert solution is regenerated by the teacher to produce a same-family target with token-level distillation signals.
  • 3.2 Pass-Rate Weighting: Student competence is estimated by sampling K rollouts per problem and computing its empirical pass rate p∈[0,1].
  • 3.2 Pass-Rate Weighting: Low-SNR extremes arise from incoherent gradients on intractable problems and dispersed problem-specific refinements on mastered problems.
  • 3.4 Theoretical Guarantees: The Beta kernel peaks at p* = α/(α+β); the default α=β=1 gives w(p)=p(1−p), while misspecification incurs only O(δ^2) worst-case efficiency loss.
  • 3.3 Overall Algorithm: Each problem’s distillation loss is scaled by its weight, with forward KL used for Qwen3 distillation and reverse KL for Qwen2.5 self-distillation.
  • 3.3 Overall Algorithm: The algorithm prepares teacher targets when needed, estimates and normalizes weights, then applies weighted distillation with optional periodic recomputation.

4 Experiments

PACED is evaluated in distillation and self-distillation across Qwen and Llama settings, using pass-rate weighting with forward or reverse KL and reasoning, retention, and baseline comparisons.

  • Experimental setup: PACED is tested in Qwen3 distillation with forward KL, Qwen2.5 self-distillation with reverse KL, and cross-family Llama-3.1 distillation.The distillation setup uses Qwen3-8BGRPO → Qwen3-1.7B, while self-distillation uses Qwen2.5-Math-7B-Instruct.
  • Experimental setup: Pass rates are estimated from K=8 student rollouts, with default weight w(p) = p(1 −p); PACED is compared with unweighted training, Hard Filter, and AKL.Hard Filter retains problems with 0.2 ≤p ≤0.8, while AKL adapts the per-token KL coefficient from teacher–student logit discrepancy.
  • Plasticity: +2.6/ + 3.9/ + 3.6 on MATH-500/AIME 2024/AIME 2025 is PACED’s improvement over unweighted forward KL in distillation.These results are reported for the Qwen3 distillation track.
  • Plasticity: +4.8/ + 6.3/ + 8.2 on MATH-500/AIME 2024/AIME 2025 is PACED’s improvement over the unweighted baseline in self-distillation.The corresponding gains relative to the base model are +9.8/ + 12.0/ + 13.6.
  • Stability: PACED reduces forgetting from 2.9 to 1.4 percentage points in distillation, while self-distillation reports 0.6% forgetting versus 0.5% for Hard Filter on retention alone.In distillation, PACED matches Hard Filter on retention while outperforming it on the reasoning benchmarks.

5 Discussion: Limitations and Future Work

The paper identifies rollout cost, discrete pass-rate estimates, and dependence on explicit correctness judgments as limitations of PACED.

  • Limitations: With default K=8, pass-rate estimation adds roughly one inference pass over the training set before optimization.K=4 captures most of the benefit while halving this cost.
  • Limitations: K=8 yields only nine discrete pass-rate values, limiting fine competence distinctions despite the Beta kernel’s smoothing of nearby weights.Larger K improves resolution but adds rollout cost, with diminishing returns beyond K=8.
  • Limitations: A problem with true p ≈0.05 can be estimated as p̂ = 0 and receive zero weight, eliminating weak learning signal because w(0) = 0.This hard-zero effect becomes more pronounced at small K.
  • Limitations: Pass-rate estimation depends on an explicit binary correctness verdict for each rollout.

6 Conclusion

PACED concentrates distillation on problems near the student’s competence frontier, motivated by bell-shaped cross-problem gradient SNR and supported by theoretical and empirical results. The method requires only pass-rate estimates and remains robust to stale or misspecified weights.

  • Results: +4.6/+4.9/+5.8 gains over standard forward KL are reported for the two-stage schedule on MATH-500/AIME 2024/AIME 2025.The reported pattern generalizes across Qwen3, Qwen2.5, and Llama-3.1 model families.
  • Practical implications: Single-pass pass-rate estimation is sufficient even when weights become stale as the student improves.The minimax guarantee limits worst-case efficiency loss to O(δ^2).
  • Theory: The Beta kernel p^α(1−p)^β is derived as the leading-order weight family under power-law SNR decay and is minimax-robust under bounded misspecification.The stated worst-case efficiency loss is O(δ^2).
  • Empirical motivation: The cross-problem gradient SNR is bell-shaped over student pass rate, collapsing near both incompetence and mastery.At low pass rates, gradients are directionally incoherent; at high pass rates, problem-specific corrections disperse across parameter space.

A.3 Gradient Boundary Conditions and Representation Theorem

This section establishes that distillation gradient SNR vanishes at both pass-rate boundaries for distinct coherence-related reasons. Under power-law boundary decay, the SNR decomposes into a Beta leading term with bounded remainder, motivating the Beta weighting family.

  • Boundary conditions: Gradient SNR collapses as p→0 and p→1 because cross-problem gradients become incoherent at both boundaries.At mastery, individual KL gradients may remain nonzero while their directions disperse across problems.
  • Boundary conditions: At p→1, remaining distributional corrections are problem-specific calibration refinements rather than systematic capability improvements.Their dispersion causes the mean gradient norm to shrink relative to per-problem gradient energy.
  • Boundary conditions: At p→0, diverse intractable problems generate destructively interfering corrections because the student lacks a coherent representational foundation.The resulting mean gradient is small relative to individual gradient magnitudes.
  • Representation theorem: Any continuous positive function with power-law decay at both boundaries admits a Beta leading term multiplied by a bounded remainder.The representation captures boundary decay rates while excluding additional shape modulation from the leading-order kernel.
  • Learning signal: The learning signal Q(p) = SNR(p)·(1−p) is unimodal and reaches its maximum at an interior pass rate.Under the leading-order representation, its maximizer is p∗= (a′/2)/((a′/2)+(b′/2+1)).

A.4 Complementary Derivation: Per-Problem Descent Maximization

The complementary derivation obtains Beta-kernel weighting by maximizing a per-problem guaranteed descent bound. The resulting optimal weight is a saturating function of SNR and has a unimodal Beta profile with zero weight at both boundaries.

  • Per-problem optimization: Maximizing the per-step descent lower bound yields w∗(p) proportional to ∥E[g(p)]∥^2/E[∥g(p)∥^2].This optimization is performed independently for each pass rate under the stated per-problem descent surrogate.
  • Beta kernel: Under power-law boundary decay, the low-SNR optimal weight takes the Beta form p^α(1−p)^β with (α,β)=(a′,b′).The peak occurs at p∗=α/(α+β).
  • SNR connection: The optimal weight can be written as w∗(p) = SNR^2/(1+SNR^2), a saturating transformation of gradient SNR.The transformation inherits SNR’s boundary zeros and interior maximum.
  • Kernel shape: The Beta profile increases from p=0 to p∗ and then decreases to p=1, automatically filtering both extremes.The default α=β=1 gives w(p)=p(1−p) with p∗=0.5.
  • Batch setting: Exact additive decomposition into independent per-sample subproblems is not generally available at the batch level unless gradients at different pass rates are orthogonal.The paper gives batch-level variance reduction and other routes as complementary justification for the Beta form.

A.5 Pointwise Minimax Robustness under Model Misspecification

The minimax analysis studies Beta weighting when the true SNR profile differs from its leading-order form by bounded multiplicative perturbations. It shows that the Beta kernel remains pointwise minimax-optimal in the low-SNR surrogate with only quadratic efficiency loss.

  • Minimax solution: The Beta kernel is pointwise minimax-optimal under the first-order low-SNR approximation.The result applies over the uncertainty set of bounded log-scale perturbations.
  • Uncertainty model: The analysis models misspecification through an unknown perturbation ϕ(p) satisfying |log ϕ(p)|≤δ.This permits bounded multiplicative deviations from the Beta-family SNR^2 profile.
  • Minimax solution: The minimax weight is proportional to p^a′(1−p)^b′, with a common scale factor sech(δ) that does not alter its shape.The adversarial perturbation can vary independently across pass rates in the stated uncertainty set.
  • Guarantee: The pointwise efficiency guarantee is sech^2(δ)≥1−δ^2, so misspecification causes at most quadratic efficiency loss.The same lower bound extends to aggregate efficiency under the theorem’s conditions.
  • Quantitative robustness: When SNR^2 differs from the Beta model by at most a factor of 2, the Beta kernel retains at least 64% pointwise worst-case descent efficiency.For δ≤0.3, the stated lower bound exceeds 91%.

A.6.1 Effective Gradient Variance

The variance analysis formalizes when Beta-kernel weighting reduces effective gradient variance: suppressing extreme-pass-rate problems can outweigh the variance penalty from non-uniform weights. Under stated assumptions, the reduction is quantified through a closed-form ratio and standard convergence bound.

  • Variance condition: Beta-kernel weighting can reduce effective gradient variance when downweighted boundary problems are sufficiently noisy.The sufficient condition requires the negative covariance between squared weights and per-problem gradient second moments to exceed the weight-variance penalty.
  • Closed-form analysis: The variance ratio R has a closed-form expression in Beta-function moments.The expression applies to the normalized Beta-kernel estimator and compares its variance with the uniform estimator.
  • Convergence scope: The non-convex SGD result is a standard convergence bound for optimizing the weighted objective L_w, not a direct objective-level comparison with unweighted SGD.This scope distinction limits what the convergence analysis establishes.
  • Quantitative regimes: R ≈0.84 for α = β = 1 and a_s = 1/4, corresponding to about a 1.19× variance reduction.The symmetric case reaches R ≈1.00 near a_s ≈0.34 and R > 1 for a_s ≥1/2.
  • Quantitative regimes: For bimodal pass-rate distributions concentrated near p ≈0 and p ≈1, R can fall substantially below 1.Boundary variance dominates in this regime, while Beta weights vanish at both boundaries.

A.7 Data-Driven Exponent Selection

The data-driven exponent procedure fits a Beta kernel to the empirical pass-rate distribution within the zone of proximal development. Moment matching determines kernel location, concentration, and asymmetry, while the robustness guarantee carries over to the fitted exponents.

  • Method: Data-driven exponent selection fits Beta(α+1, β+1) to the first two moments of pass rates inside the ZPD.The ZPD is defined by pass rates bounded away from 0 and 1, and the procedure requires only pass rates rather than gradients.
  • Conditions: The fitted exponents require v_Z < p̄_Z(1−p̄_Z)/3; under an exactly uniform distribution, the method yields the flat kernel w(p) = 1.For a moderately concentrated symmetric distribution with v_Z = 1/20, it recovers the default α = β = 1.
  • Robustness: Theorem 6 applies to the fitted exponents, giving pointwise and aggregate worst-case efficiency at least sech^2(δ) under bounded multiplicative misspecification.The same guarantee is inherited when the fitted Beta profile approximates the true SNR profile within e^±δ.
  • Interpretation: The fitted kernel peak approximates the mean ZPD pass rate, while α + β controls concentration around that location.Lower empirical variance yields a more peaked kernel; broader pass-rate distributions yield a flatter one.
  • Interpretation: The exponent ratio adapts emphasis to student competence: α < β favors harder problems when mean ZPD pass rate is below 0.5, whereas α > β favors consolidation above 0.5.This asymmetry follows the relationship α/β ≈ p̄_Z/(1−p̄_Z) for large concentration.

B.2 Implementation Details and Hyperparameters

The implementation estimates pass rates from student rollouts and applies smooth Beta-kernel weighting within established distillation pipelines. Ablations show that the default rollout budget and single-pass estimation retain most gains, while periodic recomputation offers additional improvements.

  • Implementation: PACED uses publicly available training and evaluation data, while some internal orchestration code remains unreleased.The paper states that the method, algorithmic changes, and full experimental configuration are described despite this restriction.
  • Baselines: The Hard Filter baseline keeps problems with 2 through 6 correct rollouts out of 8 and drops the remaining pass-rate estimates.It uses the same pass-rate estimates as PACED but replaces continuous weighting with the rule 0.2 ≤ p ≤ 0.8.
  • Rollout sensitivity: K=4 costs 1.4 points on MATH-500 and 1.1 on AIME 2025, while forgetting rises to 1.7%.K=16 provides modest gains of +0.7 on MATH-500 and +1.2 on AIME 2025, indicating diminishing returns.
  • Robustness: The smooth Beta kernel absorbs pass-rate drift, whereas hard thresholds can amplify estimation errors near their decision boundary.This behavior is consistent with the stated O(δ^2) efficiency loss under bounded stale-weight misspecification.

C.1.4 Two-Stage KL Schedule: Full Results and Budget Ablation

The two-stage schedule first uses forward KL for mode coverage and then reverse KL for consolidation, with the 50/50 allocation producing the strongest reported balance. The experiments also track competence migration and compare PACED’s problem-level adaptation with token-level AKL adaptation.

  • Schedule order: KL → RevKL improves over single-loss PACED KL by +1.7/+0.5/+1.7 on MATH-500/AIME 2024/AIME 2025.The reversed order underperforms both single-loss references and incurs 2.5% forgetting.
  • Budget allocation: The 50/50 split reaches 81.4% MATH-500, 26.1% AIME 2024, and 22.8% AIME 2025.It achieves the strongest MATH-500 and AIME 2025 performance while matching the lowest forgetting in the budget ablation.
  • Curriculum evolution: As the student improves, problems migrate from p < 0.2 through the ZPD into p > 0.8, while average pass rate rises from 0.61 to 0.84.Over 300 steps, the mastered fraction grows from 32% to 74% and the medium-pass-rate bin shrinks from 51% to 21%.
  • AKL comparison: PACED adapts at problem level through pass rate, whereas AKL adapts at token level through teacher–student logit discrepancy.The approaches are described as orthogonal: PACED selects problems, while AKL controls learning within selected problems.
  • Cross-family generalization: +4.4 on MATH-500 and +4.8/+4.1 on AIME 2024/2025 over unweighted forward KL transfer to the Llama family.Forgetting decreases from 3.5% to 1.4%, and gains over hard filtering are +1.5/+2.5/+1.4.
Loading 2603.11178v3…