PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang

arXiv:2610.08448v1cs.CLcs.AI

TL;DR

Cross-tokenizer OPD must align both response sequences and vocabularies, and it remains an open question whether recovering excluded targets improves learning. The paper compares strict-position and span supervision, finding that a compact top-16 shared-vocabulary objective retains most learning gains while span MSE lowers accuracy.

  • Problem

    Cross-tokenizer OPD requires sequence- and vocabulary-level alignment, and the learning value of strict matching versus recovered targets remains to be established.

    Source: §1 · p. 2

  • Method

    Across heterogeneous teacher–student pairs, the study compares strict shared-vocabulary OPD with student-selected top-k subsets and span-log-probability MSE, using probability-mass and gradient diagnostics.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    At k = 16, training preserves at least 96% of full shared-vocabulary OPD’s full-average improvement over the undistilled student, while all 18 tested positive span-MSE settings lower full-average accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings motivate assessing additional supervision by its reliability and interaction with strict distribution matching, rather than by alignment coverage alone.

    Source: §1 · p. 3

    Although the datasets are official, anonymized, and contain no offensive information, the authors cannot guarantee they are free of socially harmful or toxic language.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher--student pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

1 INTRODUCTION

The introduction asks whether broader alignment coverage improves cross-tokenizer OPD, distinguishing comparable response and vocabulary coverage from whether teacher targets guide learning. It previews evidence that compact strict-position supervision retains learning benefits, whereas span MSE adds coverage but lowers accuracy.

1 INTRODUCTION

  • The motivating question is how much useful supervision strict matching retains and what recovering excluded targets contributes to learning.Cross-tokenizer comparison must handle both sequence alignment and differing vocabulary distributions.Source: §1 · p. 2
  • Static vocabulary overlap can coexist with frequent strict alignment and concentrated probability mass on shared entries because vocabulary frequency and predictive probability vary across student trajectories.Whether supervision on unmatched spans improves learning depends on the objective applied to those groups.Source: §1 · p. 2
  • The study evaluates supervision through training comparisons, probability-mass measurements, and gradient diagnostics across three heterogeneous teacher–student pairs on mathematical reasoning and code generation.It varies span-log-probability MSE weight and the number of shared-vocabulary entries used at strict positions.Source: §1 · p. 2
  • Every positive span-MSE weight gives complete structural supervision coverage, yet all 18 tested settings lower full-average accuracy.Strict student-token coverage is 85.57–96.98% across studied runs, despite static vocabulary Jaccard overlap of 39.49–64.87%.Source: §1 · p. 2
  • At k = 16, training preserves at least 96% of full shared-vocabulary OPD’s full-average improvement over the undistilled student, and both strict objectives outperform evaluated baselines on all three pairs.The selected subset is drawn from the shared vocabulary at each strict position.Source: §1 · p. 2
  • Span gradients show weak or negative directional agreement with strict gradients, while their norm ratio grows across measured checkpoints and may help explain the accuracy drop.Their agreement remains below a reference formed by splitting strict-loss positions into two subsets.Source: §1 · pp. 2–3

2 PRELIMINARIES

These preliminaries define student-generated trajectories and OPD, then specify how cross-tokenizer responses are grouped and compared using strict alignments and shared-vocabulary distributions.

2.1 NOTATION

  • A trajectory is a prompt–response pair, with the response sampled autoregressively from the policy.The student and frozen teacher define distributions over their respective vocabularies.Source: §2.1 · p. 3

2.2 ON-POLICY DISTILLATION

  • OPD samples student trajectories and aligns the student with teacher feedback at the prefixes the student visits.With a shared tokenizer, the objective minimizes KL divergence between student and teacher distributions.Source: §2.2 · p. 3

2.3 CROSS-TOKENIZER ON-POLICY DISTILLATION

  • Shared token offsets delimit response intervals whose student and teacher token groups concatenate to the same span; a strict 1:1 group has one token on each side.Mismatch groups require multiple tokens on at least one side to cover the same span.Source: §2.3 · p. 3
  • At strict positions, the objective sums reverse KL between student and teacher distributions restricted and renormalized over the shared vocabulary.Shared entries are identified by their underlying tokens, excluding special tokens and ambiguous matches.Source: §2.3 · p. 4

3 REVISITING SUPERVISION COVERAGE

Across three heterogeneous teacher–student pairs, this section tests whether broader cross-tokenizer supervision improves learning and measures strict alignment and predictive mass. Strict-only training scores best across the mismatch-weight sweep, while compact student-selected supports retain most predictive mass.

3.1 STRICT ALIGNMENT ON STUDENT TRAJECTORIES

  • Strict student-token coverage is 85.57–96.98%, versus 39.49–64.87% static Jaccard overlap; teacher-token coverage is 82.82–97.26%.Granite→Phi has the lowest Jaccard overlap but the highest coverage under both tokenizations.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 LEARNING VALUE OF MISMATCH SUPERVISION

  • All three pairs score highest at λ = 0; the 18 positive-weight settings fall 0.27–1.20 percentage points below their strict baselines.Every positive weight gives complete supervision coverage, but accuracy decreases across the tested sweep.Source: Figure 2 · p. 5, §3.2 · p. 6

3.3 PREDICTIVE MASS CONCENTRATES ON THE SHARED VOCABULARY

  • At strict positions, the full shared vocabulary retains 99.69–99.90% of teacher mass and 98.99–99.81% of student mass on average.The measurements use responses sampled from students before distillation.Source: §3.3 · p. 6, Figure 3 · p. 6
  • At k = 16, student-selected shared-vocabulary entries retain at least 93.54% of teacher mass and 94.55% of student mass on average in each pair.Expanding the subset from 16 to 128 entries adds at most 3.44 percentage points of teacher mass and 2.70 points of student mass.Source: §3.3 · p. 7

4 LEARNING WITH STUDENT-SELECTED SUBSETS

The section tests whether student-selected subsets of the shared vocabulary can retain the learning gains of full-vocabulary strict supervision. It compares subset sizes with full shared-vocabulary supervision and evaluated cross-tokenizer alternatives.

4 LEARNING WITH STUDENT-SELECTED SUBSETS

  • The experiment restricts strict reverse KL to a student-selected top-k shared-vocabulary subset, renormalizing both distributions on that set.It compares k = 16 and k = 128 with the full shared vocabulary across three model pairs and evaluates math and code benchmarks.Source: §4 · p. 7
  • Strict full and both compact variants improve math and code accuracy over Base in every model pair.These results provide the reference for assessing retained learning benefit on student-selected subsets.Source: §4 · p. 7
  • At least 96% of full-average improvement over Base is retained by top-16 in each pair, with no consistent further gain from increasing subset size.The reported improvement is from Base to strict full.Source: §4 · p. 7
  • 0.51–1.05 percentage points: Strict top-16 leads the strongest alternative in full average, while all three strict variants exceed all four alternatives in math and full averages across every model pair.On ALFWorld, Strict top-16 also has the highest evaluated success rate on seen and unseen splits.Source: §4 · p. 7

5 OPTIMIZATION DIAGNOSTICS OF SPAN SUPERVISION

The diagnostics compare strict-loss and span-loss gradients on the same sampled responses. Across measured checkpoints, span gradients are weakly or negatively aligned with strict gradients and increase in relative magnitude.

5 OPTIMIZATION DIAGNOSTICS OF SPAN SUPERVISION

  • Gradient cosine measures directional agreement, while the norm ratio measures relative gradient magnitude.Both losses are evaluated on the same sampled responses held fixed during differentiation.Source: §5 · p. 8
  • Diagnostics use steps 0, 20, 60, and 100 from Strict full runs, with medians across replay units and a split-half strict-gradient reference.Each checkpoint uses responses to the same 512 prompts; raw accumulated gradients cover all trainable student parameters.Source: §5 · p. 8
  • Mismatch–strict gradient cosine remains below the split-half strict reference at every measured checkpoint in all three pairs.Mismatch cosines are near zero for Qwen→Llama and Granite→Phi; Granite→Qwen starts negative and approaches zero.Source: §5 · p. 8
  • The span-to-strict gradient norm ratio increases across measured checkpoints in all three pairs, so a fixed positive span-loss weight yields increasing relative gradient norm.The growing relative magnitude of weakly aligned span gradients may contribute to the accuracy drop from mismatch-group supervision.Source: §5 · p. 8

6 RELATED WORK

Related work covers on-policy distillation and methods for comparing predictions across tokenizers. This paper positions its study around whether concentrated student-selected supports remain useful when tokenizers constrain direct comparison.

6 RELATED WORK

7 CONCLUSION

Across three heterogeneous teacher–student pairs on math and code, strict-position top-16 supervision retains most of full shared-vocabulary OPD's improvement, while adding span supervision expands coverage but reduces accuracy. The conclusion recommends judging added supervision by its contribution to student learning.

7 CONCLUSION

  • Top-16 strict supervision retains at least 96% of full shared-vocabulary OPD's full-average improvement over the undistilled student and outperforms evaluated cross-tokenizer baselines.The study covers three heterogeneous teacher–student pairs on mathematical reasoning and code generation.Source: §7 · p. 9
  • Span log-probability MSE on mismatch groups completes structural supervision coverage but reduces downstream accuracy at every tested positive weight.At measured strict-only checkpoints, span gradients grow relative to strict gradients while directional agreement remains weak or negative; this combination may help explain the accuracy drop.Source: §7 · p. 9
  • The authors conclude that added supervision should be assessed by its contribution to student learning, not coverage alone.The findings motivate prioritizing supervision reliability over maximizing alignment coverage.Source: §7 · p. 10

AI USE STATEMENT

  • Generative AI tools were used solely for language polishing, not experimental design, implementation, execution, or analysis.The authors take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Human annotations assisted methodological feasibility analysis at the beginning of the work and were not used to evaluate the method.Annotators consented to research use, and the paper states that privacy was protected and payment met local standards.Source: ETHICS STATEMENT · p. 10
  • The authors cannot guarantee that the anonymized datasets contain no socially harmful or toxic language.The datasets were obtained from official sources and are described as not containing offensive information.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper reports that its code is available and directs readers to appendices for data preparation, training, baselines, evaluation, diagnostics, and additional results.Appendix B covers probability-mass and gradient diagnostic protocols; Appendices C–E provide benchmark-level and additional experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL DETAILS

The appendix records the prompt, training, and evaluation procedures, then specifies how vocabulary overlap, retained probability mass, and gradients are measured.

A EXPERIMENTAL DETAILS

  • The study draws 20,000 prompts—10,000 mathematics and 10,000 code—and filters training samples to at most 2,048 student prompt tokens.The probability-mass probe uses the complete, unfiltered prompt pool.Source: A EXPERIMENTAL DETAILS · p. 14
  • Students generate one response per prompt; student and teacher re-tokenize it with their own prompt encodings for loss computation.Response-position masks are derived from joint input length and separately tokenized response length; paired EOS positions enter strict losses.Source: A EXPERIMENTAL DETAILS · p. 14
  • Training runs for 100 on-policy iterations, with checkpoints every 10 iterations and a global batch of 512 responses.Optimization uses FSDP2 on eight H20 GPUs, bfloat16 computation, and SGLang rollouts.Source: A EXPERIMENTAL DETAILS · p. 14, Table 3 · p. 14
  • The four baselines use KDFlow implementations with unit KD weight and temperature 1.0; ULD, Extended ULD, and GOLD omit terminal EOS.SimCT selects its simct implementation.Source: A EXPERIMENTAL DETAILS · p. 14
  • Evaluation reports mean@32 mathematics and mean@8 code accuracy, with Math and Code averaged across their benchmarks and Full equally weighting both domains.Distilled models are evaluated at step 100; evaluation covers the benchmarks listed in the appendix.Source: A EXPERIMENTAL DETAILS · pp. 14–15

B DIAGNOSTIC PROTOCOLS AND STATISTICS

  • Static vocabulary overlap is measured using the Section 2 vocabulary convention, and Table 4 reports vocabulary sizes and shared-vocabulary fractions.The table covers all three teacher–student pairs and reports overlap ratios as percentages.Source: B DIAGNOSTIC PROTOCOLS AND STATISTICS · p. 15, Table 4 · p. 15
  • The probability-mass probe averages teacher and student mass retained by the same student-selected top-k subset over scorable strict positions.It excludes empty or unscorable responses and tokenizations with different response byte streams; Table 5 reports mean retained mass before distillation.Source: B DIAGNOSTIC PROTOCOLS AND STATISTICS · pp. 15–16, Table 5 · p. 16
  • Gradient diagnostics compare strict reverse-KL and mismatch span-log-probability MSE gradients at checkpoints from strict-only training.They use fixed saved rollouts and raw accumulated gradients without clipping, optimizer preconditioning, or parameter updates; Table 6 defines the reported cosine and norm-ratio statistics.Source: B DIAGNOSTIC PROTOCOLS AND STATISTICS · p. 16, Table 6 · p. 16

C COMPLETE DOWNSTREAM RESULTS

This section reports benchmark-level comparisons for the three main model pairs, alongside accuracy and averaging conventions.

C COMPLETE DOWNSTREAM RESULTS

  • Scores are accuracies in percent, using mean@32 for mathematics and mean@8 for code; distilled models are evaluated at step 100.Math, Code, and Full follow the Appendix A settings.Source: C COMPLETE DOWNSTREAM RESULTS · p. 16
  • Table 7 reports benchmark-level scores for the undistilled students and frozen teachers across the three main model pairs.Source: Table 7 · p. 17
  • Table 8 gives a benchmark-level comparison for Qwen2.5-7B-Instruct→Llama-3.2-3B-Instruct.Source: Table 8 · p. 17

D ADDITIONAL RESULTS WITH A LARGER TEACHER

The section evaluates Strict top-16 with a larger teacher and an 8B student, reporting results in benchmark-level tables.

D ADDITIONAL RESULTS WITH A LARGER TEACHER

E ADDITIONAL RESULTS ON ALFWORLD

The section evaluates Cross-Tokenizer OPD on ALFWorld and describes the compared methods and reported measures.

E ADDITIONAL RESULTS ON ALFWORLD

Related papers

Loading 2610.08448v1…