PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers

Yihao Ai, Weilong Yan

arXiv:2609.18321v1cs.LGcs.AI

TL;DR

Offline OPD efficiently reuses teacher supervision, but failures persist in the training set and a failed response does not make every associated student-trajectory target useless. The paper estimates learnability through successful-only learning and trajectory-level weights. Across mathematical reasoning and code generation, it improves offline OPD and matches or exceeds online variants on multiple evaluations, with lower reported compute use.

  • Problem

    Offline OPD reuses supervision from teacher responses, so imperfect guidance persists, while a teacher’s problem-level failure does not establish that all targets on the associated student trajectory are unhelpful.

    Source: §1 · p. 1

  • Method

    The method measures signed observed-token likelihood changes on teacher-failed trajectories after successful-only learning, then smooths and normalizes the signal into bounded positive trajectory-level weights.

    Source: §1 · p. 2

  • Results

    Across mathematical reasoning and code generation, the method improves offline OPD and matches or exceeds online variants on multiple evaluations; Qwen3-4B-Base raises offline OPD AIME24 from 66.4% to 69.1%.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    Teacher-failed supervision can be reweighted rather than uniformly kept or discarded, because failed responses do not show that every conditional target is useless.

    Source: §1 · pp. 1–2

    The teacher outcome records an independently generated response and labels neither the stored student trajectory’s correctness nor the quality of every teacher target on student-visited states.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision persistent. Since even strong teachers can fail, we ask \emph{what remains learnable from imperfect teacher supervision?} Teacher failure is only a coarse problem-level signal and does not imply that all supervision along the associated student trajectory is unhelpful. A natural alternative is to estimate teacher recoverability along the trajectory, but repeated continuations largely erase the efficiency advantage of offline distillation. We instead use teacher-successful problems to define a cheap reference for what the student can learn. We train on teacher-successful problems and measure how the likelihood of each observed token in trajectories from teacher-failed problems changes. We use these signed likelihood changes as an operational \emph{learnability signal}: larger increases indicate behavior more strongly promoted by successful-only learning. We aggregate this signal into trajectory-level weights for the original distillation loss. Unlike continuation-based estimates, our learnability requires no additional generation and can be computed once from stored trajectories and model checkpoints. Across mathematical reasoning and code generation, our method improves an offline OPD baseline by up to 2.7 percentage points and matches or outperforms online OPD variants on multiple benchmarks. Despite the additional successful-only distillation stage, it uses 2 GPUs and about 22 GPU hours, compared with 3 GPUs and 36--48 GPU hours for representative online OPD methods.

1 INTRODUCTION

Offline OPD reuses teacher supervision efficiently, but that persistence makes failures consequential; a failed teacher response does not show that every target along a student trajectory is useless. The paper estimates learnability from successful-only learning and uses trajectory-level weights, improving results across reasoning and code tasks while retaining lower compute costs than representative online OPD methods.

1 INTRODUCTION

  • Teacher failure is a problem-level label, not proof that every conditional target on the student trajectory is useless; uniformly keeping or discarding failed cases can respectively retain unhelpful supervision or remove useful supervision.Student trajectories sharing the same failure label may warrant different amounts of OPD supervision.Source: §1 · p. 1, Figure 1 · p. 2
  • Repeated continuations from many trajectory prefixes could estimate recoverability, but their generation and verification costs largely sacrifice offline distillation’s efficiency advantage.The paper therefore seeks a cheaper signal for how much useful supervision remains in each teacher-failed trajectory.Source: §1 · pp. 1–2
  • Successful-only learning measures signed changes in observed-token likelihoods on teacher-failed trajectories; larger positive changes indicate behavior more strongly promoted by learning from teacher-successful problems.The signal can be used at token, span, or trajectory level; trajectory-level weighting was empirically most effective while preserving token-level OPD structure.Source: §1 · p. 2
  • Learnability-Weighted Distillation smooths the signal, normalizes trajectory statistics relative to teacher-failed examples, and assigns bounded positive weights to coarse learnability tiers.The trajectory-level scheme aggregates local signals into tiers to reallocate supervision while reducing sensitivity to local fluctuations.Source: §1 · p. 2
  • Qwen3-4B-Base raises offline OPD from 66.4% to 69.1% on AIME24; the method matches or exceeds online OPD variants on multiple evaluations using 2 GPUs and about 22 GPU hours.Online OPD methods use 3 GPUs and 36–48 GPU hours in the reported comparison.Source: §1 · pp. 2–3

2 RELATED WORK

The related work situates the paper within on-policy distillation, selective teacher-supervision methods, and outcome-based or adaptive reweighting. Its distinction is using successful-only learning as a low-cost reference for learnability after teacher failure.

2.1 KNOWLEDGE DISTILLATION AND ON-POLICY DISTILLATION

  • Offline OPD extends on-policy distillation by reusing precomputed student trajectories and teacher supervision, and this paper studies what remains worth learning when that supervision is imperfect.The cited background traces distillation from predictive distributions and sequence-level supervision to explanations and chain-of-thought reasoning.Source: §2.1 · p. 3

2.2 SELECTIVE AND RELIABILITY-AWARE ON-POLICY DISTILLATION

  • Selective OPD work reweights or filters trajectories, models reliability, or uses teacher probes; this paper instead uses teacher-successful problems as a low-cost reference after teacher failure.The cited examples include ReNIO, FiRe-OPD, RA-OPD, ReOrder-OPD, and TGOPD.Source: §2.2 · p. 3

2.3 OUTCOME SUPERVISION AND ADAPTIVE REWEIGHTING

  • Outcome selection and process supervision provide coarse or denser feedback, while adaptive example weighting spans curriculum, robust learning, and offline policy learning; this method derives its signal from successful-only learning while preserving token-level supervision structure.The related work notes that unsuccessful trajectories can contain reusable structure that binary rejection discards.Source: §2.3 · p. 3

3 PRELIMINARIES

On-policy distillation supervises states induced by student generations with a stronger teacher, typically through a reverse-KL objective. Online OPD interleaves collection and optimization, whereas offline OPD stores a fixed collection for repeated optimization, reducing repeated generation and teacher inference while making supervision quality more consequential.

3.1 ON-POLICY DISTILLATION

  • OPD generates student responses and queries a stronger teacher for dense supervision on the student-visited states; its reverse-KL objective matches the student policy to the teacher distribution at those states.Practical implementations may use sampled-token estimators, while the underlying target remains the same.Source: §3.1 · pp. 3–4

3.2 ONLINE AND OFFLINE ON-POLICY DISTILLATION

  • Online OPD interleaves current-student trajectory generation, teacher queries, and updates, preserving evolving on-policy coverage but incurring repeated generation and teacher-inference costs.Both the student state distribution and teacher queries evolve throughout training.Source: §3.2 · p. 4
  • Offline OPD separates collection from optimization: a fixed student checkpoint generates trajectories, teacher distributions are stored once, and training repeatedly optimizes on that fixed collection.The supplied passages describe the fixed-collection process; no new generations or teacher queries occur during those repeated optimization steps.Source: §3.2 · p. 4
  • Removing repeated generation and online teacher inference reduces offline OPD’s training cost, but reusing the same supervision makes its quality more consequential.This trade-off motivates studying imperfect teachers in the offline setting.Source: §3.2 · p. 4

4 METHODS

LWD uses teacher outcomes to train a successful-only reference, measures how that update changes likelihoods of observed tokens in teacher-failed trajectories, and uses the resulting learnability signal to weight offline distillation.

4.1 OFFLINE RECORDS AND TEACHER OUTCOMES

  • Records pair student trajectories and frozen-teacher token supervision with outcomes from independently generated teacher responses, which are problem-level signals rather than token-quality judgments.The teacher outcome does not label the stored student trajectory as correct or judge every teacher target on its states.Source: Figure 2 · p. 5, §4.1 · p. 5

4.2 SUCCESSFUL-ONLY REFERENCE

  • The reference checkpoint π+ comes from distilling only teacher-successful records, whose trajectories remain student-generated; teacher-failed records are excluded from this update.This update provides a reference for assessing how successful-only learning changes behavior on teacher-failed problems.Source: §4.2 · p. 5

4.3 TOKEN-LEVEL LEARNABILITY SIGNAL

  • For teacher-failed trajectories, the token signal is the likelihood change for the same observed student tokens under π+ versus ρ; positive and negative values mark increases and decreases.The signal reflects how observed student behavior responds to successful-only learning, not whether a token is correct or whether its teacher target is useful.Source: §4.3 · pp. 5–6

4.4 TRAJECTORY-LEVEL LEARNABILITY WEIGHTING

  • Smoothed token signals yield trajectory-level mean and variation, standardized within teacher-failed examples and mapped to three learnability tiers with bounded positive weights shared across each trajectory.This reallocates overall supervision strength while preserving the relative token-level structure within each trajectory.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

The experiments compare learnability-weighted distillation with offline and online OPD baselines on mathematical reasoning and code generation. LWD improves the offline baseline across benchmarks, while ablations support retaining teacher-failed supervision with trajectory-level weights.

5 EXPERIMENTS

  • The evaluation covers mathematics and code generation, with Qwen3-8B supervision for 4B and 1.7B students and a fixed final checkpoint under matched training conditions.The reported metrics are avg@32 for mathematics and avg@4 for code; comparisons within each student scale use the same initialization, data, platform, optimization budget, and steps.Source: §5 · p. 7

5.1 MAIN RESULTS.

  • +2.7 AIME24, +2.0 AIME25, +1.6 HMMT25, +1.9 LCB v5, and +1.7 LCB v6 over Lightning-OPD; LWD is best among compared 4B methods on every benchmark.The gains are reported for the protocol-matched Lightning-OPD baseline.Source: §5.1 · p. 8
  • At 1.7B, LWD improves the offline baseline on all five benchmarks and leads the mathematics benchmarks, while online OPD remains slightly stronger on code.The authors report that LWD narrows the smaller-scale code gap while retaining lower-cost offline training.Source: §5.1 · p. 8
  • LWD uses 2 GPUs and 21.90 GPU hours, versus 3 GPUs and 36.28–47.69 GPU hours for online selective OPD methods.LWD and Lightning-OPD take 2.14 and 2.13 minutes per optimization step, respectively; the extra cost mainly comes from the successful-only distillation stage.Source: §5.1 · p. 8, Table 2 · p. 8

5.2 ABLATION STUDIES

  • Teacher-failed supervision performs best with learnability-based weighting: its average is 57.1, versus 56.4 for shared reduced weighting and 55.5 for full weighting.Removing failed-case supervision slightly improves on full weighting, but learnability weighting improves all three benchmarks.Source: §5.2 · p. 8, Table 3 · p. 8
  • Tiered trajectory weighting reaches a 57.1 average, above continuous trajectory weighting at 56.7, while token- and span-level variants lack consistent cross-task wins.Dense span weighting reaches the best HMMT25 result of 43.1, but the overall comparison favors trajectory-level weighting.Source: §5.2 · p. 9, Table 4 · p. 9
  • Teacher-failed examples vary by learnability tier: high-tier trajectories can solve the problem, mid-tier ones make progress with checking, and low-tier ones remain unresolved.The examples illustrate why a shared teacher-failure label can correspond to different reasoning behaviors.Source: §5.2 · p. 9, Figure 3 · p. 9

6 CONCLUSION

The conclusion frames teacher failure as a coarse signal rather than a reason to discard all associated supervision. LWD uses successful-only learning to estimate trajectory-specific learnability and improves offline OPD results while avoiding online teacher interaction during training.

6 CONCLUSION

  • A failed teacher rollout does not establish that every conditional target on a student-visited state is useless.The conclusion treats teacher failure as a coarse signal, not a binary supervision filter.Source: §6 · p. 9
  • Successful-only learning supplies a cheap learnability signal that is aggregated into trajectory-level weights for the original OPD objective.The conclusion contrasts this approach with repeated continuation-based recoverability estimation, which would largely sacrifice offline efficiency.Source: §6 · p. 9
  • Across mathematics and code, LWD improves a protocol-matched offline baseline and matches or exceeds online OPD variants on multiple benchmarks with fewer GPUs and GPU hours.The conclusion says the method retains failed-case supervision while varying how much supervision each trajectory receives.Source: §6 · p. 9

A.1 LEARNABILITY ESTIMATION AND WEIGHT CONSTRUCTION

Learnability is estimated by comparing stored failed-case responses under the collection policy and a successful-only model. The resulting trajectory statistics determine bounded weights for teacher-failed records.

A.1 LEARNABILITY ESTIMATION AND WEIGHT CONSTRUCTION

  • The collection policy ρ initializes both the auxiliary successful-only model and the final student; π+ is trained for 150 optimizer updates on D+.This constructs the successful-only reference model from teacher-successful records.Source: §A.1 · p. 12
  • For each teacher-failed response, the token signal is the log-likelihood under π+ minus that under ρ, then clipped to [−4, 4] and smoothed with a 33-token moving average.Trajectory mean and variation are standardized over D− using response-length-weighted statistics.Source: §A.1 · p. 12
  • Low-, mid-, and high-learnability tiers receive nominal weights 0.15, 0.25, and 0.35, globally scaled and clipped to [0.15, 0.35].The response-token-weighted average over D− is calibrated to ω0 = 0.25; teacher-successful records retain weight 1.Source: §A.1 · p. 13

A.2 FINAL LWD TRAINING

Final LWD training initializes the student from ρ and trains on the full offline collection. It preserves the protocol-matched baseline's sampled-token OPD estimator and applies trajectory weights to its token losses.

A.2 FINAL LWD TRAINING

  • The final LWD student is initialized from ρ and trained on the full offline collection.Source: §A.2 · p. 13
  • LWD uses the same sampled-token OPD estimator as the protocol-matched Lightning-OPD baseline.Source: §A.2 · p. 13
  • LWD multiplies all token losses from trajectory τi by its trajectory weight wi, leaving the token-level OPD structure intact.Source: §A.2 · p. 13

A.3 TRAINING CONFIGURATION

The experiments use Qwen3 students with a Qwen3 teacher across mathematical reasoning and code generation. Training uses 150 updates with a shared optimization setup, while teacher-success labels are defined differently by task.

A.3 TRAINING CONFIGURATION

  • The setup uses Qwen3-1.7B and Qwen3-4B SFT students with a Qwen3-8B teacher, on DAPO-Math-17K and the EpiCoder function subset.The datasets contain 17,398 mathematical prompts and 30,000 code prompts; 29,943 are uniquely aligned in the 1.7B run.Source: §A.3 · p. 13
  • Mathematical teacher success is determined with the DAPO answer verifier, while code success is approximated by normal termination and strict extraction of a complete Python program.The code labels indicate format validity, not functional correctness.Source: §A.3 · p. 13
  • Both the auxiliary policy and final student receive 150 updates using a 4,096-token response limit, batch size 256, and constant learning rate 2×10−6.Optimization uses Adam, tensor and sequence parallelism, activation recomputation, dynamic batching, and disabled dropout; only update 149 is evaluated.Source: §A.3 · p. 13

A.4 MATHEMATICAL REASONING EVALUATION

Mathematical reasoning is evaluated on three 30-problem benchmarks using 32 sampled completions per problem. Code generation is evaluated with the official LiveCodeBench execution evaluator, using four completions per problem and execution-based scoring.

A.4 MATHEMATICAL REASONING EVALUATION

  • Each 30-problem math benchmark is evaluated with 32 completions per problem at temperature 0.6 and top-p 0.95, with reasoning enabled.The primary run uses seed 42 and a 32,768-token generation limit; scores use avg@32 after extracting a valid boxed expression or, if needed, the final “Answer:” line.Source: §A.4 · p. 13

A.5 CODE EVALUATION

  • LiveCodeBench versions 5 and 6 contain 167 and 175 problems, respectively, and use four completions per problem with temperature 0.6 and top-p 0.95.Generation uses seed 42 and a 40,960-token limit.Source: §A.5 · pp. 13–14
  • The code evaluation retains the official CodeQwenInstruct template and requires one complete Python program in a single code block immediately after any reasoning.The output contract disallows restating the problem or adding text after the reasoning block.Source: §A.5 · p. 14
  • Code completions are graded by executing official benchmark tests, with avg@4 reported and valid-program extraction recorded as a diagnostic.The displayed scoring expression counts completions that pass all tests.Source: §A.5 · p. 14

Related papers

Loading 2609.18321v1…