PaperlayerView exact revision on arXiv

Source-linked AI summary

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Mingyu Yang, Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Nima Tajbakhsh, Ashwath Aithal, Pavlo Molchanov

arXiv:2610.28895v1cs.CLcs.LG

TL;DR

Cross-tokenizer distillation must align predictions despite differences in vocabulary and segmentation, which complicate comparisons across model families. CSD constructs shared supports of text-equivalent events and matches normalized distributions with one objective; across the evaluated settings, it achieves the highest average performance among cross-tokenizer methods and improves results for larger students. Its evaluation focuses primarily on English tasks, and residual events aggregate probability outside explicit support.

  • Problem

    Different tokenizers segment text differently and assign probability to different token outcomes, complicating alignment of prediction contexts and corresponding outcomes for token-level distillation.

    Source: §1 · p. 1

  • Method

    CSD constructs shared supports of text-equivalent events for aligned chunks and matches normalized teacher and student distributions with one objective.

    Source: §1 · p. 1, §5 · p. 9

  • Results

    Across four 1B teacher–student settings, CSD achieves the highest average accuracy among evaluated methods, and it outperforms continued training and ALM for base and instruction-tuned 8B students.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    CSD’s support construction and matching objective apply across chunk shapes and tokenizer pairs, with alternative events represented where available.

    Source: §5 · p. 9

    The evaluation focuses primarily on English tasks, and residual events aggregate probability without distinguishing teacher preferences among outcomes outside explicit support.

    Source: §5 · p. 9

Abstract

from arXiv · show

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to 4.2 points for the base student and 1.6 points for the instruction-tuned student.

1. Introduction

Cross-tokenizer distillation must align prediction contexts and corresponding outcomes despite differences in tokenization and vocabularies. CSD builds a shared support of text-equivalent events and applies one distribution-matching objective across chunk types and tokenizer pairs.

1. Introduction

  • Existing logits-based methods handle vocabulary and segmentation mismatches with different compromises, including token-identity loss, manual loss-mode selection, or aggregation of alternative continuations.GOLD compares unmatched probabilities by rank; X-Token requires a tokenizer-pair coverage audit, while ALM aggregates alternatives into a residual event [3] [4].Source: §1 · p. 1
  • CSD constructs a shared support of text-equivalent continuation events and matches the induced distributions with one objective across chunk types and tokenizer pairs.One-to-one chunks use common-token pairs; other alignments use sequence events scored by the autoregressive chain rule, with residual mass outside the support.Source: §1 · pp. 1–2
  • Across four 1B teacher–student settings, CSD achieves the highest average accuracy, surpassing continued pretraining and same-tokenizer distillation.It outperforms X-Token by 0.66–5.38 points and ALM by 1.33–3.30 points.Source: §1 · p. 2
  • For Llama-3.1-8B students, CSD improves over evaluated baselines for both base and instruction-tuned models.For the base student, it beats continued pretraining by 2.17–4.43 points and ALM by 2.61–4.23 points; for the instruction-tuned student, it improves over continued training and ALM by 3.90 and 1.60 points.Source: §1 · p. 2

2. Related Work

Prior cross-tokenizer distillation methods align representations, transfer logits through token correspondences, or compare transformed probability distributions. CSD instead constructs shared supports for both models, matching alternative continuations when available while retaining a residual event.

2. Related Work

  • Representation-based methods align hidden-state output spaces, while logits-based methods use tokenizer information to align or transform output probabilities.DSKD uses projections and cross-model attention [5]; BLD adds a byte-level prediction head [6].Source: §2 · p. 2
  • Logits-based methods include token-correspondence approaches and methods that compare transformed or sorted probability distributions.MinED, CDM, and SeDi use token or span mappings [7] [8]; ULD discards token identity [10], and MultiLevelOT extends optimal transport across token and sequence levels [11].Source: §2 · p. 2
  • GOLD combines KL matching for common tokens with sorted-probability matching for unmatched tokens.X-Token addresses GOLD’s limitations with projection and relaxed-hybrid modes, but requires a manual tokenizer-pair coverage audit [3].Source: §2 · p. 2
  • ALM matches the observed aligned chunk and its complement [4], but does not individually match alternative-token probabilities.CSD’s Figure 1 illustrates shared-support construction for one-to-one, one-to-many, and many-to-many chunks, including the residual event.Source: §2 · p. 2, Figure 1 · p. 3
  • On-policy methods score student-generated trajectories using synchronized chunks, shared continuations, or byte-prefix routing.DPCA synchronizes chunks [12], SimCT supervises shared continuations [13], and BPM routes teacher-token probability to student tokens with a residual category [14].Source: §2 · p. 2
  • CSD’s shared support uses common-token pairs for one-to-one chunks and chain-rule events over alternative final tokens for one-to-many and many-to-one chunks.Unlike ALM’s observed-chunk support or BPM’s byte-prefix routing, this construction requires text equivalence and applies unchanged to on-policy trajectories.Source: §2 · pp. 2–3

3. Common-Support Distillation

CSD aligns student and teacher token chunks that cover the same text, then matches normalized distributions over shared events and residual mass. Its support construction varies with chunk shape, while a common distribution-matching rule applies across accepted chunks.

3. Common-Support Distillation

  • CSD aligns token ranges covering the same text span and matches predictions conditioned on the same preceding text.Character offsets and decoded-text checks identify aligned ranges; unmatched ranges and candidates failing the checks are excluded.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1. Sequence alignment

  • A chunk is formed between consecutive common boundaries, with its M:N shape recording the student and teacher token counts.The two histories at a common boundary cover the same prefix of the original text.Source: §3.1 · p. 3

3.2. Common-support construction

  • The common support pairs student and teacher realizations of the same decoded text for one-to-one chunks, and uses virtual events for non-one-to-one chunks.The explicit events and a residual event define the common support for each chunk.Source: §3.2 · p. 4

3.2.1. One-to-one chunks

  • For one-to-one chunks, common-token pairs include alternative tokens that decode to the same text, and the relation is shared across chunks for that tokenizer pair.The residual aggregates unmatched-token probability mass, avoiding potentially inaccurate token-level correspondences as in GOLD.Source: §3.2.1 · p. 4

3.2.2. Non-one-to-one chunks

  • For non-one-to-one chunks, CSD fixes observed prefixes and pairs final tokens whose completions decode to the same text, scoring sequence events by the chain rule.One-to-many and many-to-one support can include alternatives; many-to-many chunks retain only the observed pair to avoid costly chunk-specific matching, a computational choice rather than an objective requirement.Source: §3.2.2 · pp. 4–5

3.2.3. Top-𝑘support selection

  • Top-k selection restricts candidate tokens entering the common support, while excluded probability mass remains in the residual event.Retained probabilities come from the original full-vocabulary distributions without renormalization.Source: §3.2.3 · p. 5

3.3. Distribution matching

  • Accepted chunks have at least one explicit event; CSD matches their normalized event distributions using the same selected divergence across chunk shapes and tokenizer pairs.Each accepted chunk contributes a loss term, and the configurable β weights non-one-to-one chunk losses without changing their support.Source: §3.3 · p. 5

4. Experiments

Across the evaluated 1B and 8B settings, CSD improves average performance over the reported cross-tokenizer baselines and, at 1B, over continued pretraining and same-tokenizer distillation. The experiments also examine support selection and teacher top-k truncation, alongside evaluations of base and instruction-tuned students.

4.1. Experimental Setup

  • The 1B experiments use 30,000 steps, while the 8B scaling experiments use 2,000; evaluation uses fixed task protocols and compares cross-tokenizer methods with CPT and same-tokenizer KD.The experiments use Llama-3.2-1B and Gemma3-1B-PT students, with Qwen3-4B-Base and Phi-4-mini as cross-tokenizer teachers; the 8B evaluations include base and instruction-tuned checkpoints.Source: §4.1 · p. 6

4.2. Main Results at 1B Scale

  • CSD achieves the highest Avg.-5 in all four 1B teacher–student settings, beating X-Token by 0.66–5.38 points and surpassing continued pretraining and same-tokenizer distillation.For Llama-3.2-1B, its five-task averages are 33.84 with Qwen and 35.84 with Phi; for Gemma3-1B-PT, they are 27.71 and 28.98, respectively.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Support Selection for Non-One-to-One Chunks

  • Shape-dependent support raises Avg.-5 from 33.58 to 33.84 with Qwen and from 35.22 to 35.84 with Phi versus realized-only support, at nearly unchanged step time.Extending full support to many-to-many chunks lowers Avg.-5 and costs 2.86× and 3.92× more step time than CSD.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4. Top-𝑘Robustness and Training Efficiency

  • With Qwen, k=128 scores 33.59 Avg.-5, 0.25 points below full logits, while cutting step time from 7.45 to 3.66 seconds, a 2.04× speedup.The evaluation compares teacher top-k values 16, 128, and 8192 with full-logit CSD; k=16 and k=128 also exceed ULD, GOLD, and X-Token in Avg.-5 at lower step time.Source: §4.4 · p. 8, Figure 2 · p. 8

4.5. Scaling to Larger and Instruction-Tuned Students

  • For the 8B base student, CSD reaches Avg.-4 scores of 56.72 with Qwen and 58.98 with Phi, exceeding both CPT and ALM with either teacher.Its largest gains over ALM are on MATH: 6.66 points with Qwen and 7.42 points with Phi; these results use a 2,000-step budget.Source: Table 4 · p. 8, §4.5 · p. 9
  • For Llama-3.1-8B-Instruct, CSD achieves the highest student Avg.-6 of 67.79, exceeding the initial checkpoint, continued training, and ALM by 2.00, 3.90, and 1.60 points.It exceeds ALM on four of six metrics, including gains of 6.65 and 5.39 points on IFEval-p and IFEval-i.Source: §4.5 · p. 9, Table 5 · p. 9

5. Conclusion

CSD matches teacher and student distributions over a shared support with residual mass, using one objective across chunk shapes and tokenizer pairs. It achieves the highest average score among evaluated baselines in four 1B settings and outperforms continued training and ALM for both 8B students.

5. Conclusion

  • Its shared-support distributions combine common-token and multi-token events with residual mass, using one objective across chunk shapes and tokenizer pairs.Source: §5 · p. 9
  • CSD achieves the highest average score among evaluated baselines across four 1B settings and outperforms continued training and ALM for both base and instruction-tuned 8B students.Source: §5 · p. 9
  • The evaluation focuses primarily on English, while the residual event aggregates outside-support probability without distinguishing teacher preferences among those outcomes.Source: §5 · p. 9

A. Tokenizer and Alignment Statistics

A.1. Tokenizer Overlap

  • Common pairs cover 79.64%–84.66% of Llama student vocabulary versus 36.21%–49.86% for Gemma; these are vocabulary-entry coverage, not probability mass.Source: §A.1 · p. 11

A.2. Task-Relevant Tokens Outside the Common Map

The uncommon-token audits identify task-relevant mapping gaps in GSM8K and MBPP examples and suggest plausible explanations for some GOLD results.

  • Tables 7 and 8 list the three most frequent uncommon-token entries, distinguishing token appearances from distinct examples containing each token.Source: §A.2 · p. 12
  • Audited GSM8K gaps include numerals for Qwen–Llama and Phi–Gemma, and unit or rate fragments elsewhere; numeral gaps plausibly explain GOLD’s low GSM8K and MATH scores in those settings (Tables 1 and 2).Source: §A.2 · p. 12
  • Unmatched whitespace and syntax fragments in Gemma MBPP examples encode code structure; these mapping gaps plausibly explain GOLD’s zero MBPP accuracy with both Gemma pairs (Table 2).Source: §A.2 · p. 12

A.3. Chunk-Shape Distribution

The appendix reports chunk-shape frequencies and common-support sizes, then describes how aligned chunks are formed and verified using offsets and decoded text.

A.3. Chunk-Shape Distribution

  • Non-one-to-one chunks are more frequent with Gemma (9.45–10.25%) than Llama (1.83–2.95%); many-to-many chunks are at most 1.58% of recorded chunks.Source: §A.3 · pp. 12–13, Table 9 · p. 13

A.4. Common-Support Size Distribution

  • One-to-many median support sizes range from 3 to 10; many-to-one medians are 8 for Llama–Qwen, 6 for Llama–Phi, 1,740 for Gemma–Qwen, and 175 for Gemma–Phi.Gemma many-to-one maxima reach 5,294 events with Qwen and 4,362 with Phi.Source: Figure 3 · p. 13, §A.4 · pp. 13–14

B.1. Inputs and outputs

  • Offset alignment yields chunks spanning matching source intervals, with shape determined by token counts rather than prescribed in advance.Source: §B.1 · p. 14

B.2. Decoded-text verification

  • Candidate ranges are decoded with their own tokenizers and compared using decoded text; whitespace and decoding-artifact checks help determine validity.Source: §B.2 · pp. 14–15

B.3. Special tokens and the KD interface

Special tokens are handled by semantic role, while aligned chunks provide the ranges and histories used for common-support construction and KD.

  • BOS, EOS, and padding tokens are matched by semantic role rather than character span; EOS and padding may be equivalent when the role configuration permits.Source: §B.3 · p. 15
  • Alignment supplies valid token ranges, histories, and chunk shapes; a valid chunk with no retained explicit event contributes no KD term.Source: §B.3 · p. 15
  • Table 10’s pair-specific loss-choice comparison concerns selecting between loss modes, not constructing tokenizer-dependent mappings.Source: Table 10 · p. 16

C. CSD Training Algorithm

CSD’s training algorithm builds bidirectional token-conversion indices to retrieve paired completions for aligned chunks. It computes normalized distributions with residual mass and backpropagates student gradients through retained chain factors and residual probabilities.

C. CSD Training Algorithm

  • AlignOffsets identifies shared character boundaries and retains chunks that pass decoded-text checks.Source: C. CSD Training Algorithm · p. 16
  • Bidirectional conversion indices map ordinary tokens by surface match or decode-and-retokenize, match special tokens by semantic role, and cache deterministic mappings.The indices are built once per process; EOS is matched by semantic role.Source: C. CSD Training Algorithm · p. 16
  • For non-one-to-one chunks, prefix-and-length keys retrieve retained paired completions in expected O(1) time, while gathering and scoring K candidates adds O(K) work.One-to-many chunks query the student-to-teacher index; many-to-one chunks query the reverse index.Source: C. CSD Training Algorithm · p. 16
  • The residual contains mass outside retained support; student gradients flow through retained chain factors and residual probabilities, while the teacher remains fixed.The denominator counts accepted chunks without weighting by β, and sg stops gradients through the scaling ratio.Source: C. CSD Training Algorithm · p. 16

D. Comparison with Logits-Based Distillation Methods

CSD and ALM both use a residual event without selecting a pair-specific loss mode, but differ in the explicit support they match. CSD can include alternative events across chunk types, whereas ALM matches only the observed chunk and its complement.

D. Comparison with Logits-Based Distillation Methods

  • Table 10 compares logits-based methods by their distribution-matching rule and how they construct supervision for non-one-to-one chunks.The comparison follows published formulations; experimental adaptations are described separately.Source: D. Comparison with Logits-Based… · p. 17
  • CSD matches multiple alternative events for one-to-one, one-to-many, and many-to-one chunks, while ALM matches only the observed aligned chunk and its complement.Under CSD’s default policy, many-to-many chunks retain only the observed sequence pair and the residual.Source: D. Comparison with Logits-Based… · p. 17

E. More Evaluations

Standalone teacher accuracy does not by itself determine student transfer effectiveness: Qwen leads the teacher averages, while Phi yields higher CSD student averages for both students. In the divergence and chunk-weight study, JSD has the highest reported average among the tested divergences, and increasing the weight from 1 to 5 improves the average in this setting.

E. More Evaluations

  • Qwen has the highest standalone teacher Avg.-5 at 64.94 and leads on four tasks, but Phi’s 60.25 is best on WinoGrande; with CSD, Phi yields higher student averages than Qwen for both Llama and Gemma.Teacher scores are separate from student-method rankings and use the same 3-shot evaluation protocol as the 1B students.Source: E. More Evaluations · p. 17

E.1. Divergence and Chunk Weighting

  • At β=5, JSD achieves Avg.-5 of 33.84, versus 32.65 for FKL and 33.77 for RKL; it is the highest reported average, close to RKL.The common-support construction accommodates all three divergence choices.Source: §E.1 · p. 17
  • With JSD, Avg.-5 rises from 32.88 at β=1 to 33.84 at β=5 and 33.91 at β=20, supporting upweighting non-one-to-one chunks in this setting without establishing a universal optimum.The default is β=5, which remains configurable.Source: §E.1 · p. 17

E.2. Detailed Support-Selection Ablation

The support-selection ablation favors full support for one-to-many and many-to-one chunks while retaining realized-only support for many-to-many chunks in the evaluated setting. Teacher top-k truncation reduces step time while keeping average accuracy close to full logits.

E.2. Detailed Support-Selection Ablation

  • The proposed support policy yields the highest Avg.-5 and MBPP with both teachers; against realized-only support, Avg.-5 rises from 33.58 to 33.84 with Qwen and from 35.22 to 35.84 with Phi.The proposed policy uses full support for one-to-many and many-to-one chunks, and realized-only support for many-to-many chunks.Source: §E.2 · p. 18
  • Full support for all costs 2.86× and 3.92× as much time per step as the proposed policy with Qwen and Phi, respectively, while producing mixed task-level changes.With Qwen, full support for all raises GSM8K and MATH relative to the proposed policy but lowers MBPP from 32.60 to 31.00, resulting in a lower Avg.-5.Source: §E.2 · p. 18

E.3. Detailed Teacher Top-𝑘Results

  • At k=128, Avg.-5 is 33.59, 0.25 points below full logits, while step time falls from 7.45 to 3.66 seconds, a 2.04× speedup.All three top-k settings remain within 0.59 average points of full-logit performance while reducing training-step time; task scores are not monotonic in k.Source: §E.3 · p. 18
  • Production timing uses 256 H100 GPUs across 32 nodes with online teacher predictions and includes forward and backward passes, support computation, communication, updates, and data loading.Offline data preparation, tokenizer-table construction, startup, checkpoint writing, and extra measurement synchronization are excluded.Source: §E.3 · p. 19

Related papers

Loading 2610.28895v1…