PaperlayerView exact revision on arXiv

Source-linked AI summary

CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

Satyam Kumar, Saurabh Jha

arXiv:2607.16955v1cs.LGcs.AI

TL;DR

Deploying large reasoning models on resource-constrained hardware is impractical, while existing distillation methods leave important coverage and reward gaps. CADENCE addresses these gaps with coverage-adaptive per-token KL surrogates and targeted training components, closing 63.2% of the teacher–student gap on GSM8K and outperforming the strongest matched-compute baseline by 4.4 points on GSM8K and 3.4 points on MATH-500.

  • Problem

    Deploying billion-parameter reasoning models on edge or resource-constrained hardware remains impractical, motivating compact students that inherit teacher capabilities.

  • Method

    CADENCE combines DRIFT’s per-token forward/reverse-KL surrogate mixture with six targeted components addressing coverage, entropy, reward sparsity, brevity, calibration, and stabilization.

  • Results

    +4.4 ± 0.7 points on GSM8K and +3.4 ± 0.8 on MATH-500 versus the strongest matched-compute label-using baseline.

  • Takeaways & Limitations

    On the corrected evaluation protocol, CADENCE reaches strong reasoning quality while closing 63.2% of the teacher–student gap with a 1.5B teacher.

  • Takeaways & Limitations

    DRIFT optimizes per-token surrogate objectives that are not equivalent to sequence-level KL gradients, so its theoretical claims apply only to those surrogates.

Abstract

from arXiv · show

On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive $β$ schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 $\pm$ 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 $\pm$ 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 $\pm$ 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.

1 Introduction

CADENCE targets three persistent failures in on-policy knowledge distillation: cold-start collapse, state-agnostic divergence scheduling, and binary reward sparsity. It combines DRIFT with six components and reports strong GSM8K gains using matched-compute evaluation on a single Apple Mac Studio.

  • Motivation: Large language models offer strong reasoning, but their deployment on edge or resource-constrained hardware remains impractical, motivating compact students through knowledge distillation.Knowledge distillation transfers teacher capabilities to a smaller student.
  • Failure modes: CADENCE addresses cold-start collapse, state-agnostic divergence scheduling, and binary reward sparsity, which respectively impair support bootstrapping, KL scheduling, and partial-credit learning.Cold-start collapse leaves teacher-preferred reasoning tokens with near-zero student probability; time-only schedules ignore prompt-specific coverage growth; binary rewards treat partially correct and incoherent traces identically.
  • Method: CADENCE’s DRIFT combines forward-KL and reverse-KL per-token surrogate signals on student-sampled trajectories, extended by six components targeting coverage, entropy, reward density, brevity, calibration, and self-distillation.COVA adapts β after a measured coverage gate, while FTB emphasizes high teacher-entropy positions using a globally scaled entropy reference.
  • Results: 63.2% of the teacher–student gap closes on GSM8K with a 1.5B teacher, improving the 0.5B student from 48.7% to 69.8% pass@1; a 3B teacher closes 76.2%.The reported results use 5 seeds with standard deviations and outperform the strongest matched-compute label-using baseline by +4.4 points.

2 Related Work

Prior work spans classical forward-KL, reverse-KL and on-policy distillation, reasoning-oriented reinforcement and rejection-sampling methods, adaptive divergence schedules, token weighting, and self-distillation. CADENCE differs through data-adaptive per-token divergence mixing, entropy-based token weighting, correctness-gated supervision, and consistency-filtered self-distillation.

  • Distillation objectives: Classical softened-output and sequence-level distillation minimize forward KL, whereas MiniLLM uses reverse KL and GKD demonstrates on-policy training outperforms off-policy training.DistiLLM combines skew KL with adaptive off-policy mechanisms; CADENCE instead uses a data-adaptive per-token convex mixture through COVA.
  • Reasoning distillation and baselines: Reasoning distillation uses teacher-trace SFT, outcome-reward methods such as GRPO, PPO-inspired KTR, and rejection-sampled SFT; CADENCE compares matched-compute label-using baselines.CADENCE’s CCD, LAP, and BSD apply gold-answer correctness gates, motivating baseline matching to isolate component contributions from label access.
  • Divergence scheduling: CADENCE’s COVA makes divergence interpolation state-adaptive, extending prior α-divergence paths, fixed skew-KL interpolation, and Warmup-Distill’s distribution-mismatch bridging.The related work traces α-divergence interpolation to variational inference and contrasts CADENCE with fixed interpolation.
  • Token weighting: Prior token-weighting methods use teacher–student disagreement or token importance, while CADENCE’s FTB applies globally normalized entropy weighting for cross-sequence comparability.High-entropy tokens are described as carrying disproportionate learning value in reinforcement-learning-with-verifiable-rewards settings.
  • Self-distillation and consistency training: CADENCE’s BSD builds on self-distillation and self-consistency by selecting high-consistency correct rollouts with an explicit correctness gate against confidently-and-consistently-wrong traces.The related work connects self-distillation across vision and language to self-consistency decoding applied during training.

3 CADENCE: Framework and Methodology … 3.3 CADENCE Novel Components

CADENCE combines DRIFT’s per-token forward/reverse-KL surrogate mixture with coverage-adaptive scheduling and six targeted components addressing coverage, entropy, reward density, brevity, calibration, and self-distillation. The framework uses student-sampled trajectories and explicitly does not estimate sequence-level KL gradients.

  • 3.1 Problem Formulation: CADENCE formulates on-policy distillation from a frozen teacher πϕ to a smaller trainable student πθ sharing vocabulary V and operating autoregressively.
  • 3.2 The DRIFT Mechanism: Per-Token KL Surrogate Mixture: DRIFT mixes forward-KL and reverse-KL per-token surrogate signals on student-sampled trajectories rather than estimating sequence-level KL divergence gradients.Its β schedule anneals from 1 to 0 via cosine scheduling, while the true sequence-level gradients require reward-to-go or sequence-level importance ratios absent from DRIFT.
  • 3.3.1 (A) COVA: Coverage-Adaptive β Scheduling: COVA accelerates β’s forward-to-reverse transition only after measured teacher-token coverage exceeds a validation-selected gate, while not addressing premature sharpening.Before the gate, βCOVA equals the baseline cosine schedule; COVA uses k = 20, τ = 10^-3, and an EMA coverage measure.
  • 3.3.2 (B) FTB: Forking-Token Boost with Global Entropy Reference: FTB boosts advantage at high teacher-entropy forking positions using a global entropy reference, but entropy also captures stylistic forks weakly correlated with correctness.The fixed reference Href = 2.0 provides cross-sequence comparability, making the target set a superset of ideal reasoning-critical positions.
  • 3.3.3 (C) CCD: Correctness + Numerical-Proximity Partial Credit: CCD adds numerical-proximity partial credit to incorrect-but-close trajectories, raising the nonzero-reward fraction from ∼49% to ∼55%.Across the first 100 steps of Experiment 1, ∼12% of the 51% incorrect trajectories received nonzero partial credit with pi > 0.1; CCD changes the objective by design rather than correcting DRIFT bias.
  • 3.3.4 (D) LAP: Brevity-Preferential Reinforcement: LAP is a length-weighted correct-rollout SFT term that gives shorter correct outputs higher weight without imposing a direct single-token length penalty.Using fixed Gmax = 192 removes prompt-length dependence from the brevity weight; the effect is validated via RLD and separated from generation-cap truncation.
  • 3.3.5 (E) EMR: Entropy-Matching Regularizer at Forking Tokens: EMR is applied only at forking positions to match entropy and calibrate confidence, with its ECE benefit contingent on a well-calibrated teacher.The implementation uses λemr = 0.10 and η = 1.0 nat, and calibration is evaluated using 10-bin expected calibration error.
  • 3.3.6 (F) BSD: Bootstrapped Self-Distillation: BSD follows main training by filtering consistent, correct rollouts for SFT, with the correctness gate preventing reinforcement of high-consistency wrong groups.The consistency threshold τbsd is selected on the validation split.

3.4 Stabilization Mechanisms · 3.5 Total CADENCE Objective

CADENCE stabilizes on-policy distillation with teacher-forced warm-up and a soft per-token trust region, then combines DRIFT, dense-credit, brevity, calibration, and trust-region losses while keeping BSD separate. The total objective explicitly uses FTB-boosted, COVA-modulated advantages.

  • 3.4 Stabilization Mechanisms: 20 teacher-forced SFT steps on teacher-generated traces precede on-policy sampling, conditioning importance weights from the start.This warm-up is the TFW stabilization mechanism.
  • 3.4 Stabilization Mechanisms: KTR applies a soft trust region using λ_ktr = 0.005 and δ_ktr = 3.0.The threshold bounds per-token log ratios.
  • 3.4 Stabilization Mechanisms: Per-token |ˆk_t| can exceed 3.0 at outlier positions even when trajectory-average reverse KL peaks near 1.7, activating KTR there.The reported ablation impact is correspondingly modest.
  • 3.5 Total CADENCE Objective: L_CADENCE = L_DRIFT + L_CCD + L_LAP + L_EMR + L_KTR.This is the stated total CADENCE objective.
  • 3.5 Total CADENCE Objective: BSD is a separate post-phase rather than a term in L_CADENCE.The objective equation excludes BSD from the summed losses.
  • 3.5 Total CADENCE Objective: L_DRIFT uses the FTB-boosted, COVA-modulated advantage within the total objective.Thus, the stabilization and component mechanisms enter through the DRIFT loss.

4 Experimental Setup

Experiments distill a shared 0.5B Qwen2.5 student from 1.5B or 3B teachers under fixed, validation-selected protocols. Evaluations use corrected task-specific caps and answer extraction, with five-seed statistical reporting on a single Apple Mac Studio.

  • Model configurations: Two configurations distill the same Qwen2.5-0.5B-Instruct student from either a Qwen2.5-Math-1.5B-Instruct or Qwen2.5-3B-Instruct teacher.Both configurations share the Qwen2.5 tokenizer with vocabulary size 151,936.
  • Training setup: Training uses 400 DRIFT steps with four prompts per step, 192-token generation, temperature annealing from 1.0 to 0.7, cosine β decay from 1.0 to 0.0, and importance-sampling clipping c=10.The student is trained with LoRA adapters on all attention and MLP projections.
  • Evaluation protocol: Evaluation uses a 512-token cap for MATH-500, retains 192 tokens for GSM8K, and applies a Qwen2.5-Math–matched extractor with boxed-format parsing and a final-numeric-expression fallback.The corrected protocol was introduced after identifying truncation and regex extraction errors under the earlier setup.
  • Validation protocol: CADENCE hyperparameters are selected on a validation split, while final test-set results use the resulting fixed settings.The validation sweeps include COVA γ, BSD τbsd, learning rate, EMR λemr, and LAP αlap.
  • Compute and reporting: All main-text results use five seeds with reported standard deviations, and method deltas use pooled standard deviation.The experiments run on a single Apple Mac Studio with Apple silicon, 16-core CPU, 40-core GPU, and 64GB unified memory.

5 Results

CADENCE outperforms matched-compute label-using baselines while combining forward and reverse KL effectively. Diagnostics and ablations attribute gains to complementary components, positive interactions, dense correctness-gated rewards, and genuine compression.

  • Overall performance: +4.4 ± 0.7 points on GSM8K and +3.4 ± 0.8 on MATH-500 versus DRIFT+binary in Experiment 1.The gaps exceed pooled standard deviation by approximately 6σ and 4σ, respectively.
  • Overall performance: CADENCE maintains its lead against fair baselines under both 1.5B and 3B teachers, without claiming a teacher-size scaling trend.Absolute pass@1 rises with teacher size across all methods, while margin differences remain within pooled standard deviation.
  • Diagnostic metrics: 58.6% FTA and 11.7 SAG show that CADENCE’s scheduled per-token KL mixture combines reverse-KL coverage with forward-KL stability.Reverse KL alone reaches 47.6% FTA with 27.1 SAG, while forward KL alone reaches 40.9% FTA with 17.5 SAG.
  • Compute efficiency: 4.15 CNI is highest among multi-rollout methods, indicating CADENCE adds accuracy per compute unit rather than relying only on approximately 4× sampling compute.The comparison includes STaR/RFT 4.35, GKD+GRPO 3.44, DRIFT+binary 3.61, and CADENCE 4.15.
  • Behavioral diagnostics: 26-token genuine compression reduces pretrained median rollout length from 134 to 108 tokens, while CCD’s correctness gate keeps Jaccard at 0.44 ± 0.03 instead of 0.71.Removing the correctness gate causes Jaccard to rise to 0.71 by step 400 with pass@16 degradation.
  • Ablations: Removing all components costs 6.7 points versus 18.5 summed leave-one-out deltas, indicating positive interactions; CCD has the largest single impact at −3.5.Removing EMR worsens ECE from 0.078 to 0.113, a 45% relative degradation, while accuracy drops 1.4 points.

6 Discussion and Limitations

The discussion limits CADENCE’s theoretical scope to per-token surrogate objectives and notes that CCD’s numerical-proximity reward provides only modest partial-credit gains. It also explains that corrected evaluation protocols shift absolute results while preserving relative conclusions, alongside potential consumer-hardware and calibration benefits.

  • Scope and limitations: CADENCE’s theoretical claims apply only to per-token surrogate objectives, which are not equivalent to sequence-level KL gradients.DRIFT uses per-token surrogates standard in on-policy distillation practice; Proposition 1 and Remark 1 do not establish sequence-level KL-gradient results.
  • Scope and limitations: CCD raises the nonzero-reward fraction from ∼49% to ∼55%, making its numerical-proximity partial-credit fix meaningful but modest.The passage states that larger gains would require step-level p…
  • Evaluation protocol: 65% →70% absolute CADENCE numbers result from a corrected 512-token MATH-500 cap and improved answer extraction, while relative comparisons and conclusions are preserved.The revised protocol reproduces Qwen2.5 published performance within a few points.
  • Societal impact: CADENCE enables reasoning-capable models on consumer hardware, while EMR’s calibration focus helps deployed models communicate uncertainty appropriately.These are presented as societal-impact implications of the framework.

7 Conclusion

CADENCE is a unified on-policy distillation framework that addresses three diagnosed failure modes through per-token surrogate theory, six targeted components, and two stabilization mechanisms. Under corrected GSM8K evaluation, it substantially improves a 0.5B student’s performance with both 1.5B and 3B teachers.

  • Framework: CADENCE addresses three diagnosed failure modes with honestly stated per-token surrogate theory, six novel components, and two stabilization mechanisms.The framework is presented as a unified on-policy distillation approach.
  • Results: 69.8% GSM8K performance raises a 0.5B student from 48.7% with a 1.5B teacher, closing 63.2% of the teacher gap.The result uses the corrected evaluation protocol.
  • Results: 72.1% GSM8K performance with a 3B teacher closes 76.2% of the teacher gap for the 0.5B student.This result is also reported under the corrected evaluation protocol.

A Diagnostic Metrics: Definitions and Experimental Role

The section defines diagnostic metrics for selection, reasoning-critical positions, path efficiency, compute-normalized improvement, and response-length compression. These metrics validate CADENCE’s mechanisms and enable comparison with pretrained and baseline systems.

  • Selection and Reasoning Diagnostics: SAG fell from 24.6 ± 0.9 pretrained to 11.7 ± 0.7 with CADENCE, separating capability from selection; removing BSD increased SAG to 16.9.SAG = pass@k − pass@1 with k = 16; the ablation identifies BSD as its primary driver.
  • Selection and Reasoning Diagnostics: High-teacher-entropy positions increased from 34.8% pretrained to 58.6% with CADENCE, validating FTB as a proxy for reasoning-critical positions.The proxy is not exclusive.
  • Efficiency Diagnostics: KLPE was approximately 0.64 for CADENCE, combining sharpening of 0.965 with forward-KL stability of 0.667.KLPE is clipped to [0, 1] and constructed from unitless ratios; CADENCE’s peak reverse KL was approximately 1.7 and final reverse KL approximately 0.06.
  • Efficiency Diagnostics: 4.15 was CADENCE’s CNI versus 3.61 for DRIFT+binary, while multi-rollout methods with ng=4 incurred approximately 4× sampling cost.CNI measures pass@1 improvement points per PFLOP and enables apples-to-apples compute comparison.
  • Response-Length Diagnostics: Un-truncated RLD was 108 tokens for CADENCE versus 134 pretrained, a genuine 26-token compression attributed to LAP.RLD is the median response length among correct answers; with a 192-token cap, both raw and un-truncated RLD are reported.

B Teacher Calibration

Both teachers are reasonably calibrated, supporting EMR’s premise that matching student and teacher entropies at forking tokens can calibrate the student.

  • B Teacher Calibration: Both teachers exhibit reasonable calibration, with ECE < 0.07.This supports EMR’s operational premise of matching student-teacher entropies at forking tokens.

C BSD Sweep Details

The BSD validation sweep shows that accepted-subset accuracy rises monotonically with τbsd, while τbsd = 0.80 delivers the best validation performance by balancing acceptance volume and per-trace quality.

  • BSD sweep: τbsd = 0.80 achieves peak validation performance by balancing acceptance volume and per-trace quality.This value was selected after the validation sweep.
  • BSD sweep: Accepted-subset accuracy is monotone non-decreasing in τbsd because the correctness gate admits only correct traces.The consistency threshold further filters accepted traces for robustness.
Loading 2607.16955v1…