Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching

• Existing distillation methods generally require equivalent or similar teacher and student tokenizers, restricting applicable teacher–student pairs.Source: §1 · p. 1
• ALM enables effective distillation across fundamentally different tokenizers, including subword-to-byte transfer and pure cross-tokenizer distillation.Source: §1 · p. 2
• ALM tokenizes text with both models, aligns chunks encoding the same text, and approximates their likelihood difference with a binarised f-divergence.Optional debiasing and hidden-state distillation extend the objective.Source: §1 · p. 2
• Tokenizer transfer as self-distillation retains performance across subword tokenizers, enables rapid byte-level transfer, and supports ensembles that combine transferred models.Source: §1 · p. 2
• ALM also improves large-to-small distillation across tokenizers and provides a state-of-the-art objective for zero-shot tokenizer-transfer hypernetworks.Source: §1 · p. 2

2.1 Tokenizers

• A tokenizer is a vocabulary and tokenization function mapping byte strings to token sequences, with detokenization recovering the original text.Source: §2.1 · pp. 2–3
• Subword tokenization can leak future text information because token boundaries depend on subsequent characters, creating tokenization bias.Source: §2.1 · p. 3

2.2 Generative Language Models

• Autoregressive language models assign next-token probabilities and compose them into probabilities for multi-token continuations and texts.Source: §2.2 · p. 3

2.3 Distillation

• Pure distillation matches student predictions to teacher predictions, whereas hybrid distillation adds distillation to a ground-truth task objective.Pure self-distillation can preserve knowledge, while hybrid distillation may avoid teacher hacking and capacity-gap issues.Source: §2.3 · p. 3
• Prior cross-tokenizer methods heuristically incorporate teacher information and therefore do not provide a pure objective across fundamentally different tokenizers.Source: §2.3 · p. 4

3 Methodology

• ALM enumerates aligned teacher–student chunks encoding the same prefix and continuation, then compares their chunk-level probabilities.Greedy alignments are used, with optional constraints on comparable chunks.Source: §3 · p. 4
• Because all possible continuation chunks are infinite, ALM computes a binarised f-divergence over a sampled chunk probability and its complement.This is an upper bound that remains minimal exactly when student and teacher probabilities match.Source: §3 · p. 4

3.1 Outcome Chunk Debiasing

• Outcome-chunk debiasing multiplies chunk probabilities by marginal probabilities of shared pretoken-boundary bytes, reducing tokenization bias.The method excludes continuations that would otherwise encode boundary-dependent information; debiasing substantially improves performance.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Distilling Hidden States across Tokenizers

• The chunk-probability signal is sparse relative to same-tokenizer logits, so ALM can add an auxiliary loss aligning hidden states through a learned dimensionality projection.This auxiliary signal is particularly useful when teacher and student share an underlying architecture.Source: §3.2 · pp. 5–6

3.3 Combining Loss Components

• GradMag combines distillation and next-token objectives by weighting tasks inversely to their last-layer gradient norms.It directly equalizes last-layer gradient magnitudes and performs on par or better than GradNorm with lower implementation complexity.Source: §3.3 · p. 6

Use Case 1: Tokenizer Transfer via Self-Distillation

• The experiments frame tokenizer transfer as cross-tokenizer self-distillation, enabling transfer to a shared subword tokenizer or byte-level tokenization instead of training byte-level models from scratch.The study uses Gemma2 2B IT and Llama3.2 3B IT transferred to Qwen2, with Qwen2.5 1.5B IT serving as an ensemble model.Source: Use Case 1: Tokenizer Transfer via… · p. 7
• ALM consistently improves tokenizer transfer over MinED and SFT, including substantial gains for the harder subword-to-byte setting where prior distillation methods are inconsistent.Pure ALM outperforms ALM + SFT on Qwen2 transfer, while both ALM variants improve substantially over SFT for byte transfer; a residual gap to the original model remains for byte transfer.Source: Use Case 1: Tokenizer Transfer via… · pp. 7–8
• Transferred models can be ensembled with Qwen by averaging predictions, improving tasks where constituents perform similarly and moving toward the strongest constituent on other tasks.The reported ensemble combines transferred Gemma2 and Llama3 with Qwen2.5; the comparison includes probability and log-probability averaging and a MinED baseline.Source: Use Case 1: Tokenizer Transfer via… · p. 8, Table 2 · p. 8

Use Case 2: Large-to-Small Distillation Across Tokenizers

• ALM improves large-to-small mathematics distillation, outperforming SFT and prior methods by up to 3% points while reaching 53.3 / 74.6 of teacher performance at 4x fewer parameters.The experiment distills OpenMath2-Llama3.1-8B into Gemma2 2B and evaluates zero-shot accuracy on GSM8K and MATH.Source: Use Case 2: Large-to-Small… · p. 9, Table 3 · p. 9

Use Case 3: Improving Zero-Shot Tokenizer Transfer Hypernetworks via Self-Distillation

• ALM improves ZeTT hypernetwork training, raising Mistral v2 BoolQ performance from 67.5 to 74.0 for zero-shot tokenizer transfer.Unlike MinED, ALM requires no pre-computation of Levenshtein distances, supporting online hypernetwork training with newly sampled tokenizers.Source: Use Case 3: Improving Zero-Shot… · p. 9, Table 4 · p. 10

4.1 Scaling ALM to Larger Models

• ALM’s transfer-original gap reduction improves with model size, from approximately 55% for Gemma2 2B to approximately 70% for Gemma3 12B transferred to Qwen2.The authors leave scaling to truly large models for future work.Source: §4.1 · p. 10, Table 5 · p. 10

• ALM outperforms prior heuristic methods across diverse use cases while enabling effective subword-to-byte distillation for the first time.It also supports conventional large-to-small distillation and state-of-the-art zero-shot tokenizer transfer.Source: §5 · p. 10

• ≈0.6B training tokens leave ALM’s comparison with SFT and other distillation methods at larger data scales unresolved.Source: §6 · p. 10
• The study evaluates one chunk-level f-divergence and one chunk-selection strategy, leaving categorical divergences and alternative chunk structures unexplored.Source: §6 · p. 10
• ALM is tested only in the seminal supervised KD setting, leaving rejection-sampling and on-policy distillation applications for future work.Source: §6 · p. 10

A.1 Sensitivity to the Learning Rate

• Learning rates from 2e−6 to 5e−5 are generally robust, although high values occasionally cause loss spikes; the main experiments use 1e−5.This sensitivity analysis trains transfer from Gemma 2 2B IT to the Qwen2 tokenizer for 5k steps.Source: §A.1 · p. 15, Table 6 · p. 15

A.2 Impact of the Choice for the Distance Function f and the Temperature τ

• τ > 1 is beneficial for both KL-divergence and Total Variation Distance, and the main experiments use KL-divergence with τ = 100.Low temperatures emphasize high-likelihood chunks, whereas higher temperatures increase attention to lower-likelihood and longer chunks.Source: §A.2 · pp. 15–17, Table 7 · p. 17

A.3 Ablating Outcome Chunk Debiasing, the threshold γ and Hidden States Distillation

• Outcome Chunk Debiasing is consistently highly beneficial, while hidden-state distillation helps Subword →Byte transfer but slightly harms Subword →Subword transfer.A positive threshold γ can marginally improve debiasing.Source: §A.3 · p. 17

A.4 Comparing Loss Combination Strategies: Fixed β vs. GradNorm vs. GradMag

• GradMag performs on par with or slightly better than GradNorm, and both vastly outperform fixed loss-combination coefficients β.GradMag incurs only a small overhead from per-task backpropagation through the last layer.Source: §A.4 · p. 17, Table 9 · p. 18
• GradMag avoids the extra optimizer and associated hyperparameter tuning required by GradNorm.Source: §A.4 · p. 17

B Training Details

• ALM experiments generally use 64-text batches, 512-token sequences, Adam without weight decay, 20k steps, and LoRA with α = r = 64.The default peak learning rate is 1e−5, with 2k-step warmup and linear decay; GradMag balances loss components.Source: B Training Details · p. 18
• The largest experiments run on 32 v4 TPU chips, taking approximately 24 hours for Llama3-to-byte transfer and 12 hours for Gemma2-to-byte transfer.Gemma2 hypernetwork training takes approximately two days, while remaining experiments take roughly 5 to 10 hours each.Source: B Training Details · p. 18

C Adjustments for Transfer to Bytes in Use Case 1

• Byte transfer uses 2048-token student sequences, batch size 32, learning rate 3e-5, full-model training, and hidden-state distillation.Source: C Adjustments for Transfer to Bytes in… · p. 19
• Byte transfer unties embedding matrices and adds the longest matching subword embedding to each byte embedding to mitigate the parameter mismatch at the input layer.The strategy does not correct the output-layer mismatch, so performance may still decrease.Source: C Adjustments for Transfer to Bytes in… · p. 19

D Hypernetwork Architecture Modifications for Use Case 3

• The residual hypernetwork Eout = Ein[0] + Hθ(Ein) enables single-stage training without the auxiliary embedding-similarity loss.It avoids learning to mimic length-one inputs initially and reduces catastrophic drift in embedding space; removing the auxiliary loss then causes only minor degradation in the residual setting.Source: D Hypernetwork Architecture… · p. 19, Table 10 · p. 19

E Relation of the Binarised f-divergence to the Categorical f-divergence

• The binarised f-divergence is a lower bound to the categorical f-divergence because merging outcomes through post-processing cannot increase divergence.The same relation also follows from Jensen’s inequality when the relevant function is convex.Source: E Relation of the Binarised… · p. 20

F A Variant of ALM Which Explicitly Takes Tokenization Bias into Account

• The authors did not adopt the explicit bias-selection variant because it requires MinED-like precomputation, adds complexity, prevents online applications, and performs roughly equally well.The main ALM version therefore remains the practical choice despite the variant’s formal motivation.Source: F A Variant of ALM Which Explicitly… · p. 20
• The tokenization-bias-aware ALM variant estimates bias differences using cover encodings, last-token implied exclusions, and unigram likelihoods, then constrains aligned chunks by a threshold γ.These approximations make the metric efficiently precomputable, but they are crude because evaluating every alternative token sequence with a large language model is intractable.Source: F A Variant of ALM Which Explicitly… · pp. 20–21

G Overview of Existing Cross-Tokenizer Distillation Methods

• ULD combines next-token prediction with a Wasserstein loss between teacher and student logits at every position.This keeps the absolute differences between sorted teacher and student logits small.Source: G Overview of Existing Cross-Tokenizer… · p. 21
• Table 11 compares explicit tokenization-bias-aware ALM with main ALM on Gemma2 2B IT transferred to the Qwen tokenizer after 5k training steps.The supplied caption identifies the comparison scope but does not provide cell values or a winner.Source: Table 11 · p. 22
• MinED greedily aligns student vocabulary tokens to teacher tokens by Levenshtein distance and applies KL-divergence only at one-to-one sequence alignments.Its main objective remains next-token cross-entropy.Source: G Overview of Existing Cross-Tokenizer… · p. 22
• DSKD projects teacher and student representations into each other’s sequences with cross-attention, then adds bidirectional KL-divergence losses to next-token prediction.The method compares teacher predictions with both student-to-teacher and teacher-to-student projected predictions.Source: G Overview of Existing Cross-Tokenizer… · p. 22

H Analysing the Choice of Ensemble Pivot & Special Tokens

• Ensembles average 60.8 with Qwen as pivot and 60.3 with Llama as pivot, showing robustness to pivot choice.The MinED baseline instead degrades catastrophically with Llama as pivot, performing worse than its individual constituents.Source: H Analysing the Choice of Ensemble… · p. 22
• Transferring special tokens preserves performance for the Llama pivot but catastrophically degrades Qwen-pivot performance, especially for Gemma2 2B.Qwen2 lacks a <bos> token, forcing models that used one to relearn how attention is distributed; the authors therefore preserve original special tokens.Source: H Analysing the Choice of Ensemble… · p. 23

I Implementing ALM

• ALM aligns teacher and student chunks in one pass by tracking shared byte-ending positions, avoiding edit-distance alignment when both sequences encode the same bytes.The resulting binary alignment matrices record which tokens belong to each chunk and can be computed in a separate data-loading thread.Source: I Implementing ALM · p. 23
• ALM computes chunk log-probabilities with matrix gathers, applies an element-wise f-divergence, and adds negligible overhead relative to Transformer forward and backward passes.Alignment constraints and chunk debiasing add implementation complexity but are cheap and optional in principle.Source: I Implementing ALM · p. 23