PaperlayerView exact revision on arXiv

Source-linked AI summary

SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Yilin Cheng, Bichuan Feng, Pengfei Liu, Junfeng Fang, Xiang Wang

arXiv:2605.07711v2cs.CL

TL;DR

Cross-tokenizer OPD faces vocabulary and sequence mismatches that make direct token-level comparisons problematic, while shared-vocabulary supervision discards teacher signal. SimCT expands the common supervision space with minimal aligned units and consistently improves over SFT and representative cross-tokenizer baselines on math and code benchmarks.

  • Problem

    Different vocabularies and text segmentations leave teacher and student predictions without a one-to-one token alignment, while shared-vocabulary OPD discards supervision represented by different tokens.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Method

    SimCT compares teacher and student over a common supervision space containing shared tokens and minimal aligned units, while retaining the standard reverse-KL OPD loss.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Across three model families on math and code benchmarks, SimCT consistently improves over SFT and representative cross-tokenizer OPD baselines, averaging a 6.0% relative gain over SFT.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Minimal aligned units preserve useful local teacher–student distinctions that coarser supervision units can erase, with larger coarsening associated with weaker downstream gains.

    Source: §4.4 · pp. 8–9

    Evaluation covers math and code reasoning with three teacher–student pairs, and minimal aligned units may need adaptation for tokenizers with substantially different segmentation behavior.

    Source: §5 · p. 9

Abstract

from arXiv · show

On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \textbf{\underline{Sim}ple \underline{C}ross-\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at \href{https://github.com/sunjie279/SimCT-}{https://github.com/sunjie279/SimCT-}.

1 Introduction

Tokenizer mismatch makes standard token-by-token OPD ill-defined and limits shared-vocabulary supervision. SimCT expands the supervision space without changing the OPD loss, and improves results across the reported benchmarks.

1 Introduction

  • OPD queries the teacher on student-generated prefixes to address the train–test mismatch of training on fixed contexts, requiring comparable prediction units at each prefix.Early generation errors can move the student into contexts rarely supervised during training.Source: §1 · p. 1
  • Different vocabularies leave most of the teacher distribution unused under shared-token matching, while different segmentations make next-token units incomparable.Standard OPD assumes teacher and student predict the same text unit at each supervised position.Source: §1 · pp. 1–2
  • SimCT consistently improves over SFT and representative cross-tokenizer baselines, with a 6.0% average relative gain over SFT.Figure 1 reports the best average Pass@1; absolute results and standard deviations are reported in Table 1.Source: Figure 1 · p. 2, §1 · p. 2
  • Existing alignment, projection, and token-bypassing methods leave substantial teacher signal beyond reach, at added computational, parameter, or granularity cost.These approaches include distribution alignment, learned cross-model projections, and coarser chunk, BPE, mapping, or byte-level interfaces.Source: §1 · p. 2
  • SimCT adds minimal aligned units to shared tokens while retaining the reverse-KL OPD loss and requiring no extra parameters, learned projections, or transport computation.The unchanged loss form attributes gains over shared-vocabulary OPD to the supervision interface.Source: §1 · p. 2

2 Related Work

Prior work extends on-policy distillation to heterogeneous tokenizers through alignment, representation-transfer, and token-bypassing approaches.

2 Related Work

  • OPD queries the teacher on student-generated prefixes, extending offline distillation on fixed corpora to student-visited contexts.Related work also studies sequence-level distributional compatibility [20] [38] and variation in token-level supervision quality [39] [40].Source: §2 · p. 3
  • Optimal-transport methods align heterogeneous logits between token distributions, across token and sequence levels, or layer-wise with chain-of-thought augmentation [41].Source: §2 · p. 3
  • Other methods unify output spaces through cross-model attention or bypass tokens with chunk likelihoods, dynamic vocabulary mapping, and byte-level interfaces [26].Related directions also extend cross-tokenizer transfer to preference alignment and embedding-model representations.Source: §2 · p. 3

3 Methodology

Cross-tokenizer OPD must compare teacher and student predictions despite different vocabularies and segmentations. SimCT constructs a common supervision space from shared tokens and minimal aligned units, then applies OPD over that space.

3.1 Problem Setup

  • Teacher and student predict over their own vocabularies from the same student-generated prefix, but different vocabularies and segmentations prevent direct token-by-token comparison.Cross-tokenizer OPD seeks a common interface for comparable teacher feedback and student predictions on those prefixes.Source: §3.1 · p. 3

3.2 Generalized OPD over a Common Supervision Space

  • Generalized OPD maps teacher and student predictions into a common supervision space U, where a distillation divergence compares the induced distributions.SimpleOPD restricts U to the shared vocabulary, discarding supervision for comparable text represented by different token segments.Source: §3.2 · pp. 3–4

3.3 SimCT: Recovering Supervision via Minimal Aligned Units

  • SimCT forms its space from shared tokens and minimal aligned units: the finest boundary-consistent text units both tokenizers can express.The shared-token portion retains 1:1-aligned tokens, while minimal aligned units recover supervision from differently segmented text.Source: §3.3 · p. 4
  • For multi-token units, SimCT averages autoregressive token log-likelihoods by 1/k, normalizes scores over candidates, and applies OPD without future ground-truth tokens.The length-normalized scores form an operational supervision distribution, not a mass-preserving marginalization of the model’s next-token distribution.Source: §3.3 · p. 5

3.4 Properties of the SimCT Supervision Space

  • Theoretical results characterize minimal aligned units as jointly tokenizable units with no finer jointly tokenizable decomposition, making them the finest comparable interface.This interface is more expressive than exact shared-token matching while avoiding unconstrained character-level comparison.Source: §3.4 · p. 5
  • Coarsening removes the within-unit teacher–student discrepancy; the analysis and ablation support preserving minimal units to retain fine-grained distillation signal.The practical objective uses normalized continuation scores over a finite candidate set rather than an exact mass-preserving marginal.Source: §3.4 · p. 5

4 Experiments

SimCT improves cross-tokenizer OPD across teacher–student pairs and math and code benchmarks by recovering supervision lost to tokenization mismatch. The experiments find that both recovering more mismatched-unit supervision and preserving it at minimal aligned-unit granularity matter for downstream gains.

4.1 Experimental Setup

  • The study evaluates three large-to-small teacher–student pairs on GSM8K, MATH-500, MBPP, and LiveCodeBench-v6.All methods start from a common SFT-warm-started checkpoint and follow the same OPD training protocol; comparisons include shared-vocabulary OPD and representative cross-tokenizer baselines.Source: §4.1 · p. 6

4.2 Overall Effectiveness

  • SimpleOPD improves only marginally over the SFT warm start, while adding minimal aligned units yields consistently stronger performance.The main-results table reports mean pass@1 across four benchmarks for each teacher–student pair.Source: §4.2 · p. 6, Table 1 · p. 7
  • SimCT achieves the best average performance for all three teacher–student pairs and the strongest result on every evaluated benchmark.The reported gains span different teacher and student families and both math and code benchmarks; prior cross-tokenizer baselines do not rank uniformly strongest across settings.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Supervision Recovery under Tokenizer Mismatch

  • Figure 3 identifies non-1:1 sequence units and high-probability out-of-vocabulary predictions as discarded signals; recovering either helps, while recovering both performs best.Figure 4 also shows that increasing recovered mismatch-unit supervision tends to improve performance for both students.Source: §4.3 · pp. 7–8, Figure 3 · p. 8
  • SimCT expands the supervision space while leaving the original OPD objective unchanged.The expansion reincorporates teacher feedback excluded by shared-vocabulary matching.Source: §4.3 · p. 8

4.4 Ablation on Supervision-Space Construction

  • Coarsening minimal aligned units removes within-span KL signal and is associated with weaker downstream gains for both students.The ablation supports preserving supervision at the finest granularity jointly expressible by both tokenizers.Source: Figure 4 · p. 9, §4.4 · p. 9

5 Limitations

The evaluation is limited to math and code reasoning with three teacher–student pairs, and the method’s scope depends on tokenizer and teacher-access assumptions. The study also leaves normalization calibration and extensions to broader settings for future work.

5 Limitations

  • Evaluation covers math and code reasoning with three teacher–student pairs across two teacher families; additional domains, scales, and tokenizer families remain future work.The stated evaluation scope sets the current evidence boundary.Source: §5 · p. 9
  • Mapping mismatches to minimal aligned units is natural for standard BPE-style tokenizers and may require adaptation for substantially different segmentation behavior.Source: §5 · p. 9
  • The study uses 1/k average log-likelihood normalization but does not systematically compare alternatives or calibrate the induced candidate-space distribution.Source: §5 · p. 9
  • SimCT targets white-box OPD with teacher next-token distributions available; extending it to sample-only or scalar-feedback teachers remains future work.Source: §5 · p. 9

6 Conclusion

The conclusion frames cross-tokenizer OPD as a supervision-space construction problem and proposes SimCT to recover feedback beyond exact shared tokens. It reports consistent gains across model families and math and code benchmarks.

6 Conclusion

  • Cross-tokenizer OPD loses useful teacher feedback when heterogeneous token-level predictions are matched only on exact shared tokens.Source: §6 · p. 9
  • SimCT expands the common supervision space with minimal aligned units while preserving the standard OPD objective.The conclusion characterizes minimal aligned units as the finest jointly expressible interface and notes that coarsening can erase useful teacher–student distinctions.Source: §6 · p. 9
  • Across model families and math/code benchmarks, SimCT consistently improves over SimpleOPD and representative cross-tokenizer distillation baselines.Source: §6 · p. 9

A Training Data Construction and Curation

The appendix constructs a teacher-generated warm-start corpus from filtered public math and code prompts, then documents its composition, response selection, attribution, and release.

A.1 Composition

  • The corpus contains 10,000 examples: 5,800 mathematical reasoning examples and 4,200 code-generation examples, with multiple sources sampled within each task family to avoid overfitting to any single dataset’s style.Math sources are GSM8K, Orca-Math 200K, OpenMathInstruct-1, and MATH; code sources are OpenCodeInstruct, KodCode, TACO, and CodeContests.Source: §A.1 · p. 16
  • The corpus combines math and code prompt sources in a 5,800-to-4,200 split, roughly matching the evaluation suite’s task-family emphasis.MATH-500 evaluation instances are excluded from the MATH prompt sample; MATH level 5 is also excluded because solutions often exceed the SFT context budget and show high teacher-side variance.Source: §A.1 · p. 16

A.2 Prompt Filtering and Deduplication

  • Prompts are filtered for noise, duplication, length, and verifiability, with overlap checks against held-out evaluation data.Math filtering excludes malformed or non-text problems and repeated templates; code filtering retains executable Python tasks and removes unsupported task formats.Source: §A.2 · p. 16

A.3 Teacher Response Generation and Selection

  • For each retained prompt, the teacher samples eight responses and one is selected using verification when available or quality filters otherwise.When multiple candidates pass, selection favors the shortest valid response; public datasets provide prompts, while targets come from the teacher-response pipeline.Source: §A.3 · pp. 16–17

A.4 Licenses, Attribution, and Release

  • Per-example source attribution and license metadata are retained, and downstream users must comply with each source asset’s terms.The appendix says the corpus has no single license and that the full corpus is reproducible from documented sources and pipeline materials, subject to redistribution permissions.Source: §A.4 · p. 17

B Training Details

All OPD methods share a warm-start checkpoint and training setup, differing only in how teacher and student predictions are compared under tokenizer mismatch. SimCT generates student rollouts, constructs a common supervision space, and applies reverse-KL distillation there.

B Training Details

  • All compared OPD methods start from the same warm-start checkpoint and share prompts, hyperparameters, and compute settings, differing only in prediction comparison under tokenizer mismatch.The common initialization is intended to prevent SFT data or checkpoint differences from confounding OPD comparisons.Source: B Training Details · pp. 17–18, §B.1 · p. 18

B.1 Warm-Start SFT

  • Each student is SFT-trained on the 10K teacher-generated corpus for two epochs to produce the shared warm-start policy.The SFT stage uses AdamW, a peak learning rate of 2 × 10^-6, warmup followed by cosine decay, and bfloat16 on 8×H20 96GB.Source: §B.1 · p. 18

B.2 On-Policy Distillation

  • During OPD, the student generates prefixes under its current policy, the teacher provides next-token supervision, and the student is updated under the method-specific loss.SimCT and SimpleOPD use reverse KL in their respective supervision spaces: SimCT combines shared tokens with minimal aligned units, while SimpleOPD uses exact vocabulary overlap.Source: §B.2 · p. 18

B.3 SimCT Algorithm

  • SimCT tokenizes each student rollout with both tokenizers, extracts minimal aligned units, maps teacher and student distributions into a common space, and accumulates reverse-KL loss.The algorithm repeats this comparison at student-generated prefixes and updates the student parameters from the accumulated loss.Source: §B.3 · pp. 18–19

C.1 Benchmarks

The evaluation spans math reasoning and Python code generation, using zero-shot prompts and repeated sampling with shared decoding and scoring procedures. The theory section defines minimal aligned units from tokenizer boundaries and establishes their joint tokenizability and minimality.

C.1 Benchmarks

  • GSM8K and MATH-500 assess mathematical reasoning, while MBPP and LiveCodeBench-v6 assess Python generation across function-level and contemporary hidden-test settings.The benchmark mix includes symbolic answers and rationale tokens in math, and identifiers, indentation, and syntax fragments in code.Source: §C.1 · pp. 19–20

C.2 Decoding

  • All benchmarks are evaluated zero-shot with fixed prompts, nucleus sampling at temperature 0.6 and top-p 0.95, and five independent runs.Maximum generation lengths are 4096 tokens for GSM8K, MATH-500, and LiveCodeBench-v6, and 2048 for MBPP.Source: §C.2 · p. 20

C.3 Scoring

  • The reported score is pass@1 averaged over five runs, with math answers checked for symbolic equivalence and code evaluated using benchmark test harnesses.Postprocessing and sandbox settings are identical across compared methods.Source: §C.3 · pp. 20–21

D.1 Tokenizer-Induced Spans and Minimal Aligned Units

  • Minimal aligned units are connected components induced by overlaps between teacher-token and student-token spans, yielding the finest units that remain valid under both tokenizers.Atomic fragments define the construction but need not themselves be valid supervision units.Source: §D.1 · p. 21

D.2 Proof of Theorem 1

  • Each minimal aligned unit is a union of complete consecutive tokens for both tokenizers, and no strictly finer jointly tokenizable decomposition exists.The proof uses the fact that any boundary splitting such a unit would have to be a boundary in both tokenizers, contradicting its connected-component construction.Source: §D.2 · pp. 21–22

D.3 Proof of Proposition 1: Coarsening and Removed KL Signal

The proposition characterizes coarsening as removing within-span teacher–student KL differences, while the discussion connects that loss to weaker supervision and the design of SimCT.

D.3 Proof of Proposition 1: Coarsening and Removed KL Signal

  • Coarsening minimal aligned units preserves each merged span’s total probability but removes conditional teacher–student differences among units within that span.The removed quantity is the expected within-span KL signal across student-generated prefixes.Source: §D.3 · pp. 22–23
  • The right-panel ablation tests the theory’s prediction: coarsenings that remove more within-span discrepancy provide weaker supervision and smaller downstream gains.The passage identifies the figure as a direct test of the theoretical prediction.Source: §D.3 · p. 23

D.4 Discussion

  • The theory relies on tokenizer boundaries and probability aggregation, not learned semantic alignment, embedding similarity, or optimal transport, matching SimCT’s unchanged OPD objective.SimCT changes the supervision space to the finest comparable text units while keeping the OPD objective unchanged.Source: §D.4 · p. 23

E Computational Resources

Reproducing the reported main experiments is estimated to require approximately 14.8 days on a single 8× H20 server. Runs take about 2.5–3 hours, and SimCT adds only a small overhead over SimpleOPD while remaining faster than the other reported baselines.

E Computational Resources

  • About 2.5–3 hours per training run; SimCT adds only a small overhead over SimpleOPD and is faster than ALM, GOLD, and DSKD.The measurements are for cross-tokenizer OPD runs across the reported teacher–student pairs and methods.Source: E Computational Resources · pp. 23–24
  • Approximately 14.8 days on a single 8× H20 server are estimated to reproduce the reported main experiments.The estimate covers warm-start SFT (about 1.0 day), cross-tokenizer OPD training (about 9.4 days), and evaluation (about 4.4 days), with each configuration repeated over five seeds.Source: E Computational Resources · p. 24, Table 6 · p. 24
  • The compute estimate excludes preliminary studies, hyperparameter exploration, debugging, ablations, and failed or repeated development jobs, so full project compute is larger.Source: E Computational Resources · p. 24
  • SimCT adds no trainable projection module or global vocabulary-level alignment and keeps the standard OPD pipeline unchanged.The passage attributes its low overhead relative to more complex baselines to operating at the distillation-interface level.Source: E Computational Resources · p. 24

F Case Study

The case studies illustrate local math and code errors across supervision interfaces, while SimCT avoids the specific errors shown; the examples are illustrative rather than causal evidence. The broader discussion frames supervision-interface design as a question for longer, more structured settings as well as cross-tokenizer distillation.

F Case Study

  • In the modular-arithmetic example, methods identify the arithmetic sequence but diverge at the final remainder calculation; SimCT computes remainder 4.The authors present the case studies as illustrative examples, not causal evidence; quantitative downstream gains are established in Table 1.Source: F Case Study · pp. 25–26
  • In the square-seating example, SimCT divides total permutations by rotational symmetries, while each baseline makes a different symmetry-counting error.SimpleOPD misapplies Burnside’s lemma, ALM double-divides by circular and square symmetries, DSKD confuses seating with pairing, and GOLD adds intra-side arrangements already counted.Source: F Case Study · p. 27
  • In the factorial-code example, SimCT produces an O(log X) iterative solution, while the shown baselines include repeated input reads and brute-force or costly factorial computation.The reported baseline flaws lead to timeout or stack overflow on large inputs.Source: F Case Study · p. 28
  • For the largest-number function example, all baselines use a one-parameter signature despite tests passing two arguments, whereas SimCT infers the required interface.The baselines therefore raise a TypeError when the test harness supplies the second argument.Source: F Case Study · p. 29

G Broader Scope of Supervision Interface Design

  • The broader-scope discussion asks how teacher feedback can remain informative in longer, more structured, tool-dependent agent trajectories.It presents SimCT as addressing the complementary question of how to express teacher and student distributions under tokenizer heterogeneity.Source: G Broader Scope of Supervision… · pp. 29–30
  • SimCT changes the comparison space for predictions, whereas cited supervision-routing and preference-training approaches change the source, strength, or routing of supervision.The passage describes the directions as orthogonal and says they could be combined in future OPD systems.Source: G Broader Scope of Supervision… · p. 30
  • The paper relates SimCT’s shared supervision space to the broader principle of preserving task-relevant information while reducing irrelevant representation differences.The passage contrasts tokenizer-specific spaces with aligned text units both models can realize.Source: G Broader Scope of Supervision… · p. 30
Loading 2605.07711v2…