Source-linked AI summary
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia, Davide Buffelli
TL;DR
Cross-tokenizer distillation is difficult because standard logit matching assumes shared vocabularies, while existing alignment strategies add complexity. BLD converts teacher outputs to byte probabilities, adds a lightweight byte-level decoder to the student, and distills through this shared interface. It performs competitively with, and sometimes surpasses, more complex methods, although gains are not consistent across benchmarks and tasks.
Problem
Cross-tokenizer distillation lacks a simple way to match teacher and student distributions when their tokenizers and vocabularies differ.
Method
BLD converts the teacher’s token distribution to byte-level probabilities and trains a lightweight parallel byte-level decoder head on the student.
Results
BLD performs competitively with, and on several benchmarks outperforms, substantially more sophisticated methods across tokenizer transfer and cross-model distillation settings.
Takeaways & Limitations
The byte level provides a shared interface for cross-tokenizer knowledge transfer, supporting a simple baseline across models ranging from 1B to 8B parameters.
Takeaways & Limitations
Experiments cover 3B-parameter tokenizer transfer and an 8B-teacher, 2B-student CTD setting, leaving behavior at larger scales underexplored.
Abstract
from arXiv · showhide
Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.
1 Introduction
Large language models offer powerful capabilities but their scale makes training and inference costly, motivating distillation. Cross-tokenizer distillation remains difficult because standard logit matching requires shared vocabularies, while BLD uses bytes as a common interface and performs competitively without consistent dominance.
- LLM scale makes training prohibitively expensive and inference often too slow for real-time or on-device applications.
- Knowledge distillation trains a compact student to mimic a larger teacher using dense output distributions rather than only hard labels.
- Standard logit-level distillation requires teacher and student models to share a vocabulary, making different tokenizers produce incompatible output spaces.
- Existing cross-tokenizer approaches use vocabulary or hidden-state mappings that can be inefficient, lossy, or theoretically unprincipled.
- BLD converts teacher outputs to byte probabilities, adds a lightweight student byte-level decoder, and distills through the shared byte-level interface.
- BLD performs competitively with more complex CTD methods across tasks, but no method consistently dominates across benchmarks.
2 Related Work
Related work addresses cross-tokenizer transfer through vocabulary alignment, byte-level probability estimation, and byte-level language modeling. BLD instead uses the byte level as a shared interface for distillation across tokenizers.
- Cross-Tokenizer Distillation: Cross-tokenizer distillation seeks to transfer knowledge between models whose different tokenizers obstruct standard distillation.
- Cross-Tokenizer Distillation: Prior methods align heterogeneous output spaces through optimal transport, knowledge fusion, dual-space distillation, or approximate likelihood matching.
- Byte-Level Probability Estimation: Byte-level probability estimation converts token-level model distributions into equivalent character- or byte-level distributions.
- Byte-Level Language Models: BLD leverages byte-level probability conversion to distill information from token-based models into byte-level interfaces.
- Byte-Level Language Models: Byte-level language models include pure byte-sequence models such as ByT5, which can remain competitive with token-level models and show greater noise robustness.
3 Our Method
BLD enables distillation between models with different tokenizers by converting teacher outputs to byte-level probabilities and adding a shared byte-level interface to the student. Distillation combines byte-level and token-level objectives, after which the added interface can be removed.
- Preliminaries: Standard logit-level distillation requires teacher and student to share a vocabulary, making tokenizer differences a central obstacle.The KL divergence transfers knowledge over a common vocabulary, while different tokenizers generally produce different output spaces.
- Preliminaries: Teacher byte-level probabilities are obtained by summing probabilities over token coverings of a byte sequence and deriving conditional probabilities for individual bytes.A fast approximation from Vieira et al. (2025) is used because the direct computation can be expensive.
- Preliminaries: A naive alternative reconstructs student-token probabilities from teacher byte probabilities, but it requires |V_S| computations per token and compounds approximation errors.The vocabulary size is described as typically ranging from 30,000 to 250,000 probabilities for each sequence token.
- Byte-Level Interface for Distillation: BLD adds a learnable byte-level decoder to the student in parallel with its original token-level decoder, creating a shared interface without tokenizer alignment heuristics.The byte-level decoder maps hidden representations to byte distributions, while the original token-level output layer remains available.
- Byte-Level Interface for Distillation: During distillation, the loss combines next-byte cross entropy, byte-level KL divergence, and next-token cross entropy.The next-token loss also updates the token-level decoder, and the byte-level output layer must be updated unless it was pretrained first.
- Byte-Level Interface for Distillation: After distillation, BLD removes the byte-level interface and retains the token-level output layer, although byte-level generation can optionally be preserved.The experiments use a linear byte-level projection with N_b fixed to 10; an autoregressive alternative is left for future work.
4 Experiments
The experiments evaluate BLD on BPE-to-BPE transfer, BPE-to-byte transfer, and cross-tokenizer distillation. BLD is competitive but does not uniformly dominate, with performance varying substantially by task and transfer setting.
- Experimental setup: The evaluation covers BPE tokenizer transfer, BPE-to-byte transfer, and cross-tokenizer distillation using shared LoRA training and held-out validation data.The experiments use Tulu-3 for tokenizer-transfer tasks and OpenMathInstruct-2 for cross-tokenizer distillation.
- BPE tokenizer transfer: In BPE-to-BPE transfer, BLD achieves the highest PiQA score (75.68) and AGI-ZH score (35.97), but ALM + SFT leads on four of seven benchmarks.BLD’s IFEval score is 30.58, below MinED (62.83) and ALM + SFT (58.51).
- Overall findings: Across the evaluated settings, benchmark leadership is fragmented and BLD’s strengths do not generalize uniformly across tasks.The results show competitive performance without uniform dominance.
- BPE-to-byte transfer: BPE-to-byte transfer causes large degradations across every benchmark, including approximately 21 points on MMLU and 13 points on ARC-C relative to the original model.BLD ranks first on PiQA (67.52), but its margin over MinED (67.41) is negligible and no method establishes a clear advantage.
- Cross-tokenizer distillation: In cross-tokenizer distillation, BLD reaches the highest GSM8K score (62.55), while SFT leads on MATH (22.40) and the gap to the teacher remains large.The teacher scores 87.26 on GSM8K and 37.60 on MATH, compared with BLD’s 62.55 and 20.08.
5 Limitations
The study’s limitations concern scale and optimization: experiments use practical but limited model sizes, and LoRA rather than full-parameter optimization.
- Scale: The experiments use 3B models for tokenizer transfer and an 8B teacher with a 2B student for cross-tokenizer distillation.The behavior of CTD methods at larger scales remains underexplored.
- Optimization: The distillation experiments use LoRA to reduce computational requirements, while full-parameter optimization may achieve higher performance.This leaves the performance of BLD under full-parameter training unresolved.
6 Conclusions
The paper concludes that BLD offers a simple byte-level interface for CTD and performs competitively, but consistent gains remain elusive. The authors identify CTD as an open problem and suggest richer byte-level architectures as a future direction.
- Contribution: BLD converts teacher outputs to byte-level probabilities and adds a lightweight byte-level decoder head to the student.This shared interface avoids complex vocabulary-alignment procedures.
- Findings: BLD performs competitively with, and sometimes outperforms, more sophisticated methods across tokenizer-transfer and cross-model distillation settings.The conclusion covers both evaluated task families rather than a single benchmark.
- Future direction: A byte-level transformer could enhance the approach by capturing sequential dependencies that MLP byte-level heads do not explicitly model.The paper presents this architecture as a possible improvement rather than an evaluated result.
- Open problem: No method, including BLD, achieves consistent improvements across all benchmarks and tasks, leaving CTD fundamentally open.Performance leadership shifts with the benchmark, transfer target, and model pair.
B Training Hyperparameters
The byte-level probability approximation uses parameters that trade computational efficiency against accuracy, with beam search and pruning controlling the tokenization paths considered.
- Byte-probability approximation: The implementation uses Vieira et al.’s fast approximation to compute byte-level probabilities from token-level teacher outputs.The approximation is applied to a teacher operating over a token vocabulary.
- Approximation parameters: K and ϵ control the trade-off between computational efficiency and approximation accuracy when estimating byte-level probabilities.Together they determine how many tokenization paths are considered.
- Beam width (K): Beam width K sets the maximum number of hypotheses retained during beam search over token sequences compatible with a byte prefix.Larger K explores more paths, improving approximation accuracy but increasing computational cost.
- Pruning threshold (ϵ): Pruning threshold ϵ removes hypotheses whose probability falls below a threshold relative to the highest-probability beam.This eliminates tokenization paths contributing negligibly to the final byte probability distribution.
C.2 Algorithm for Byte Probability Computation
The byte-probability procedure maintains tokenization alternatives in a beam while incrementally computing byte distributions. It balances approximation quality and computational cost through beam width, pruning, and token-boundary handling.
- The procedure initializes a beam with width K and pruning threshold ϵ to track candidate tokenization paths and their probability weights.
- At each byte position, it computes probabilities over 256 byte values by marginalizing across the tokenization paths currently in the beam.
- Each observed byte advances candidate paths, after which paths below the relative threshold ϵ are removed and at most K paths are retained.
- When a path completes a token, the beam starts a new token using the teacher model’s next-token probabilities.
- Teacher-model inference at token boundaries is the key computational bottleneck, while K and ϵ jointly determine accuracy and computational cost.
C.3 Evaluating Approximation Quality
Approximation quality is evaluated against a high-precision reference using Jensen–Shannon divergence, with additional checks of runtime and downstream distillation performance.
- The approximation is measured by Jensen–Shannon divergence against a reference distribution computed with K = 100 and ϵ = 10^-6.
- 0.0045 Jensen–Shannon divergence is achieved with K = 10 and ϵ = 0.01.
- Runtime is primarily affected by the pruning threshold ϵ, whereas beam width K has minimal impact because token queries are efficiently batched on GPUs.
C.4 Experimental Setup
The experiments evaluate approximation settings across two Llama-3.2 teacher models using Tulu-3 distillation data, measuring both runtime and divergence from a reference configuration.
- Experiments use Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct as teacher models with the Tulu-3 dataset for distillation.
- Beam widths K ∈ {2, 5, 10, 20, 50, 100} and pruning thresholds ϵ ∈ {10^-1, 10^-2, 10^-3, 10^-4, 10^-6} are evaluated.
- Each configuration is assessed by runtime and Jensen–Shannon divergence relative to the reference configuration.
C.5 Parallel Implementation
The implementation uses layered parallelism and batching to make byte-probability computation practical. A selected beam configuration trades memory and accuracy effectively, but full-dataset processing remains time-consuming.
- The dataset is partitioned into shards, with one independent worker assigned to each GPU.
- Within each worker, a process pool parallelizes across samples, while trie operations batch up to 1000 token-probability queries per forward pass.
- CPU preprocessing is overlapped with GPU computation using Python’s asyncio framework.
- 10.4 seconds per sample is achieved for 100–150 byte sequences with K = 10 and ϵ = 0.01 on four NVIDIA RTX 3090 GPUs.
- The K = 10, ϵ = 0.01 configuration provides JSD < 0.005, uses 10× less memory than K = 100, and enables greater sample-level parallelism.
- Computing byte probabilities for the entire Tulu-3 dataset requires approximately 2 days with this configuration and parallelism.