PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer, Srinivasan Iyer

arXiv:2609.12303v1cs.CLcs.AIcs.LG

TL;DR

The study asks how student tokenization changes distillation scaling as compute grows, then compares byte and token models using two token-to-byte logit methods. Token models lead early, but byte models improve more steeply, and End-Of-Token distillation has the highest predicted asymptotic performance.

  • Problem

    It remains unclear how student tokenization interacts with distillation and overtraining as compute scales, despite serving costs sometimes outweighing scale's benefits.

    Source: §1 · p. 1

  • Method

    The study compares byte and token distillation scaling and introduces Marginalize-It and End-Of-Token methods for converting token logits to byte logits.

    Source: §1 · p. 1, §2.1 · p. 4, §10 · p. 20

  • Results

    Token-1B models lead at low compute but plateau, while byte models improve more steeply; End-Of-Token distillation asymptotically outperforms the other five models.

    Source: §1.1 · p. 3, §10 · p. 20

  • Takeaways & Limitations

    End-Of-Token-1B can surpass distilled Token-1B with up to one-sixth the training data, while byte logits avoid top-k truncation and use roughly one-fifth the storage.

    Source: §1.1 · p. 3

    End-Of-Token-1B remains significantly expensive at inference, and matched-inference-cost comparisons are beyond this study's scope.

    Source: §10 · p. 21

Abstract

from arXiv · show

Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.

1 Introduction

The paper asks how student tokenization interacts with distillation as compute scales, and introduces two ways to convert token logits into byte logits. Its large-scale study finds byte models improve more with compute and that End-Of-Token distillation has a higher predicted downstream-performance ceiling.

1.1 Summary of Contributions

  • The study introduces single-pass conversion of token logits to byte logits: approximate Marginalize-It renormalizes prefix-matching tokens, while exact End-Of-Token adds an <eot> token.The exact method preserves the teacher distribution without losing probability mass.Source: §1.1 · p. 3
  • The experiments vary tokenization and training objective in layer-parameter-matched decoder-only transformers with about 1 billion layer parameters, scaling data to 1 trillion bytes.The objectives are distillation and supervised cross-entropy; tokenizations are tokens, bytes, and bytes with <eot>.Source: §1.1 · p. 3
  • Across eight benchmarks, byte models initially lag Token-1B but improve more steeply with compute, while Token-1B performance plateaus.The evaluation spans multiple-choice QA, language generation, and machine translation.Source: §1.1 · p. 3
  • Asymptotically, distilled End-Of-Token-1B is predicted to outperform distilled Token-1B by up to 4% and distilled Bytes-1B by up to 1.9%.The comparison uses downstream-error versus validation-BPB scaling laws; End-Of-Token uses approximately 30.94% more compute per fixed amount of data than Marginalize-It.Source: §1.1 · p. 3
  • End-Of-Token-1B is predicted to match distilled Token-1B using one-sixth of the training data, while its 256-byte vocabulary avoids top-k logit truncation and reduces logit storage to roughly one-fifth.Its asymptotic predictions also exceed Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.Source: §1.1 · p. 3

2 Logit Conversion Methods

The paper presents two single-pass methods for converting token logits into byte logits. Marginalize-It approximates later-byte distributions, whereas End-Of-Token preserves the full distribution by accounting for token endings.

2.1 Marginalize-It Logits

  • Marginalize-It forms byte distributions by marginalizing over tokens sharing the observed byte prefix, then renormalizing over the matching subset for later bytes.The first-byte distribution is exact, but subsequent-byte distributions are approximate because possible continuations are dropped.Source: §2.1 · p. 4

2.2 End-Of-Token Logits

  • End-Of-Token appends <eot> after each BPE token and expands the byte vocabulary to represent token endings, absorbing otherwise-missing continuation probability.This yields an exact conversion without additional teacher inference.Source: §2.2 · p. 4

2 Logit Conversion Methods

  • Figure 2 contrasts sequential prefix-based aggregation in Marginalize-It with End-Of-Token’s assignment of residual probability to <eot>.The example follows the byte sequence of “Tiramisu” and shows why Marginalize-It can omit later-token continuations.Source: Figure 2 · p. 5

3 Scaling Study Design

The study compares token and byte models across tokenization and training-objective choices using roughly billion-parameter transformers. It evaluates their scaling behavior on multiple-choice QA, language generation, and machine translation benchmarks.

3.1 Experimental Setup

  • The models use the transformer architecture and are layer-parameter-matched at 1.28B parameters, although Token-1B totals 1.81B parameters because of its larger vocabulary.Training uses the Llama-2 data mixture and AdamW, with learning rates of 1e-3, 4e-3, and 8e-3.Source: §3.1 · p. 6

3.2 Scaling Along Two Axes: Tokenization and Training Objectives

  • Six experiments vary tokenization—tokens, bytes, or bytes with <eot>—and objective—supervised cross-entropy or distillation.The study overtrains layer-parameter-matched models to examine their scaling trends.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Benchmark Evaluations

  • Evaluation covers four multiple-choice QA benchmarks, two language-generation tasks, and English-to-German and German-to-English translation on Flores.The benchmarks include ARC-Easy, ARC-Challenge, HellaSwag, PIQA, MBPP, and Natural Questions.Source: §3.3 · p. 6

4 Scaling Trends I: Validation BPB vs Training FLOPS

The section fits validation bits-per-byte scaling against training compute and compares the resulting curves across objectives and tokenizations. End-Of-Token starts with the worst BPB but has the best asymptotic value, while token models start ahead at low compute and saturate sooner.

4 Scaling Trends I: Validation BPB vs Training FLOPS

  • The analysis plots held-out validation BPB against training FLOPs and fits a power law whose asymptote is c.Figure 3 fixes the learning rate at 4e-3 and shows data-scale curves for approximately 1-billion-parameter models.Source: §4 · p. 7, §4.1 · p. 7, Figure 3 · p. 7

4.1 Validation BPB vs Training FLOP Plots

  • Both approximate Marginalize-It and exact End-Of-Token distillation improve validation BPB over their supervised counterparts.End-Of-Token distillation yields consistent improvements over supervised training.Source: §4.1 · p. 8
  • Token-1B has the lowest BPB at low compute but saturates sooner; End-Of-Token-1B starts with the worst BPB and has the best infinite-compute asymptote.Bytes-1B falls between the other two near the beginning and end of training.Source: §4.1 · p. 8
  • Asymptotically, BPB ranks End-Of-Token-1B below Bytes-1B below Token-1B for both supervised and distilled models.Models with the same tokenization show only small differences in asymptotic BPB across objectives.Source: §4.1 · p. 8
  • At approximately 1.8 × 10^21 FLOPs, Token Supervised appears to have the highest BPB among the six runs, prompting downstream benchmark evaluation.The BPB curves alone therefore do not settle the downstream-performance comparison at that compute budget.Source: §4.1 · p. 8

5 Scaling Trends II: Task Performance vs Training FLOPs

Across eight benchmarks in multiple-choice QA, language generation, and machine translation, token models lead at low compute while byte models improve faster as training compute grows. Distilled models generally outperform supervised counterparts, though token models can retain a downstream advantage at the observed 220B-token scale.

5 Scaling Trends II: Task Performance vs Training FLOPs

  • The evaluation covers eight benchmarks across multiple-choice QA, generative tasks, and machine translation to test whether validation-BPB scaling trends generalize to task performance.Source: §5 · p. 8

5.1 Results on Multiple Choice Question Answering Benchmarks

  • Distilled models consistently outperform supervised models on averaged multiple-choice accuracy, with the same trend on individual benchmarks.Source: §5.1 · p. 9
  • Token-1B models lead at low compute and appear to plateau, while byte models start worse but improve faster and eventually catch up on multiple-choice QA.Within byte models, End-Of-Token variants eventually surpass Bytes-1B counterparts; at 220B tokens, however, byte models remain much worse than distilled Token-1B.Source: §5.1 · p. 9, Figure 4 · p. 9

5.2 Results on Language Generation Benchmarks

  • On language generation, token models perform better at lower compute and appear to plateau less clearly, while byte models continue improving without clear saturation.Distilled models appear better at the same compute budget, but at 1.8 × 10^21 FLOPs only the distilled Token model performs better in most cases.Source: Figure 5 · p. 10, §5.2 · p. 10

5.3 Results on Machine Translation Benchmarks

  • Byte-model BLEU follows an S-shaped trend with training FLOPs, improving slowly at low compute, more steeply later, and then saturating.End-Of-Token Distilled models show significant improvements over Bytes w/ <eot> Supervised models.Source: §5.3 · p. 10, Figure 6 · p. 11

5.4 Concluding Remarks

  • Across the reported data points, distilled models achieve lower BPB and better overall downstream performance than supervised counterparts, but Token-1B can outperform byte models despite worse BPB at 220B tokens.The section notes that all four byte models remain much worse than Token Supervised at approximately 1.8 × 10^21 FLOPs, despite better validation BPB.Source: §5.4 · p. 11

6 Scaling Trends III: Task Performance vs Validation BPB

The section compares downstream task performance across tokenization schemes and training objectives as a function of validation BPB. It finds different scaling behavior across these choices, with End-Of-Token distillation achieving the strongest asymptotic performance.

6.1 Results on Multiple Choice Question Answering Benchmarks

  • Across multiple-choice benchmarks, tokenization and training objective affect scaling trends; equal BPB can correspond to different error rates and compute budgets.Iso-error lines intersect the End-Of-Token and Token curves at different compute budgets.Source: §6.1 · p. 12, Figure 7 · p. 12

6.2 Results on Language Generation Benchmarks

  • For language generation, the same BPB can correspond to different average downstream performance across tokenization schemes and training objectives.The plotted MBPP and Natural Questions curves show differing trends for Token-1B versus byte models and for distilled versus supervised models.Source: §6.2 · p. 13, Figure 8 · p. 13

6.3 Results on Machine Translation Benchmarks

  • Machine-translation scaling differs across tokenization schemes, while End-Of-Token distillation appears monotonic compared with noisy supervised Bytes w/ <eot> curves.Source: §6.3 · p. 13, Figure 9 · p. 14

6.4 Fitting the Downstream Task Error vs Validation BPB Scaling Law

  • The authors fit downstream-error scaling laws against validation BPB for six experiments, averaging top-1 error across four multiple-choice and two language-generation benchmarks.The fitted curves are bounded by asymptotic BPB values and visualized through interpolations and extrapolations.Source: §6.4 · p. 14, Table 4 · p. 15, Figure 10 · p. 15

6.5 Downstream Task Performance Upper Bound

  • End-Of-Token distillation has the lowest asymptotic average top-1 error, outperforming Token distillation and supervised Bytes w/ <eot> models.It also outperforms supervised Bytes models asymptotically.Source: Figure 10 · p. 15, §6.5 · p. 16

6.6 When Lower BPB Does Not (Always) Imply Better Models

  • Lower BPB does not ensure better downstream performance across different tokenization schemes and training objectives.Bytes-1B and End-Of-Token-1B surpass the Token Supervised BPB curve yet show worse downstream performance across benchmarks.Source: §6.6 · p. 16

7 The Case for End-Of-Token Logits

The section uses Feather Plots and scaling-law extrapolations to compare distilled token and byte-model performance across compute budgets. Token distillation leads at low compute, while End-Of-Token distillation has the stronger asymptotic outcome and can reach Token distillation’s asymptotic performance with less data.

7.1 Feather Plot : Visualizing Iso-FLOP Point Connections

  • Distilled Token-1B leads at low compute, but End-Of-Token distillation reaches the lowest asymptotic downstream error and improves toward that asymptote as compute grows.The Feather Plot also shows End-Of-Token distillation overtaking Bytes-1B as compute increases.Source: Figure 11 · p. 17, §7.1 · pp. 17–18

7.2 Calculating Downstream Error Cross-Over Point with Token Distillation

  • The authors use downstream-error and validation-BPB scaling laws to estimate the compute crossover where byte methods match Token distillation’s asymptotic performance.Table 6 describes the calculation by mapping Token distillation’s asymptotic accuracy through each method’s benchmark and BPB–FLOPs laws.Source: §7.2 · p. 18, Table 6 · p. 18

7.3 Results

  • End-Of-Token-1B asymptotically outperforms Token-1B while requiring one-sixth as much text data to match Token distillation’s performance.End-Of-Token Byte Logits can reduce storage costs to approximately one-fifth compared with token distillation.Source: §7.3 · pp. 18–19, Table 7 · p. 18

8 Comparisons with Open-Weight models

The section compares extrapolated asymptotic predictions with three open-weight models. The predicted End-Of-Token-1B model surpasses each on averaged downstream tasks.

8 Comparisons with Open-Weight models

  • End-Of-Token-1B is predicted to surpass Llama 3.2-1B by up to 6.5%, Gemma-3-1B-pt by 8.1%, and Gemma 2B by 2.1% on averaged downstream tasks.The comparisons use asymptotic predictions extrapolated from the scaling trend.Source: Table 8 · p. 19, §8 · p. 19

9 Related Work

The related work situates this study among research on scaling laws, downstream-task prediction, distillation, and byte-level modeling. It distinguishes this work by studying fixed-size overtraining across tokenization schemes and training objectives.

Scaling Laws with Overtraining

  • Prior scaling-law studies examine effects of architecture, data distribution, modality, resolution, and noise; this work varies tokenization and training objective during fixed-size overtraining.The authors report that both axes affect the power-law exponent.Source: §9 · p. 19

Scaling Laws for Downstream Tasks

  • Prior downstream-task work links validation loss to aggregate benchmark performance, while also finding that data distributions and task-specific setups can affect predictive reliability.Gadre et al. (2024) establish a relationship between average top-1 error and validation cross-entropy loss; Owen (2024) finds aggregate performance more predictable than individual performance.Source: §9 · pp. 19–20

Pretraining Distillation

  • Earlier distillation research studies teacher probabilities and compute allocation; this paper studies pretraining distillation for both token and byte models.Source: §9 · p. 20

Token to Bytes transfer

  • Prior token-to-byte transfer methods can require multiple teacher-inference passes; Marginalize-It marginalizes prefix-matched distributions, while End-Of-Token adds an <eot> token.The paper also places its decoder-only variants among byte-level modeling approaches and reports that End-Of-Token distillation asymptotically outperforms Token distillation.Source: §9 · p. 20

10 Discussion

The discussion reports an asymptotic advantage for End-Of-Token distillation while noting inference-cost and evaluation limits. The supplied setup details describe the models, training objectives, validation metric, and logit-storage trade-off.

10 Discussion

  • End-Of-Token distillation asymptotically outperforms the other five models on downstream tasks, though it adds about 30.94% compute per unit of data.The method preserves the teacher distribution exactly and improves the supervised End-Of-Token-1B baseline's asymptotic performance.Source: §10 · p. 20
  • Adding an <eot> token about every 4.5 bytes changes power-law coefficients, and supervised End-Of-Token-1B has better asymptotic downstream performance than Bytes-1B.Source: §10 · p. 21
  • End-Of-Token-1B remains significantly more expensive at inference than the study's asymptotic comparison establishes.The study does not compare it with a larger Token-1B model at matched inference cost.Source: §10 · p. 21
  • Validation BPB does not fairly compare models across objectives and tokenizations: lower BPB does not necessarily mean better downstream performance.Source: §10 · p. 21

10.1 Future work

  • The study is limited to dense models, approximately 1.28 billion layer parameters, and eight benchmarks; larger models, more benchmarks, and byte-student distillation dynamics remain future work.Source: §10.1 · p. 21

A.1 Model Architecture and FLOP Calculations.

  • The experiments use Transformer models with SwiGLU, Llama-2 pretraining data, Llama3-8B as teacher, and either BPE-Tiktoken or a byte tokenizer with 260 vocabulary entries.Source: §A.1 · p. 24, §A.2 · p. 24

A.3 Loss Function and Optimization Settings

  • The mixed distillation and cross-entropy loss uses α to control the balance; byte runs use longer sequence lengths than token runs after initial byte runs showed loss spikes.Token sequence length is 2048; Bytes-1B uses 9216 and End-Of-Token-1B uses 11264.Source: §A.3 · p. 24, §A.4 · p. 25

A.5 Bits Per Byte Calculation.

  • BPB measures validation cross-entropy per byte, while byte logits use a roughly 260-value vocabulary instead of 128256 token logits, reducing the storage burden of offline distillation.The paper describes the token-logit storage cost as a motivation for its controlled token-versus-byte study.Source: §A.5 · p. 25, §A.6 · p. 25

B Additional Results using Validation Dataset 1

Additional curves and analyses examine downstream and validation-BPB scaling, fit sensitivity, and the effect of validation data choice. They show distinct token and byte trends and caution that fitted crossovers depend on the validation set.

B Additional Results using Validation Dataset 1

  • The reported asymptotic BPB ordering is Bytes w/ <eot> supervised, Bytes supervised, then Token supervised; for distillation it is End-Of-Token, Marginalize-It, then Token.Source: Table 12 · p. 28
  • Across QA, language-generation, and translation curves, token models lead at low compute and plateau, while byte models start worse and improve faster with more training.The translation curves show this pattern for both byte variants, with very low initial BLEU scores.Source: Figure 13 · p. 29, Figure 14 · p. 30, Figure 15 · p. 31
  • Downstream-error and BLEU-versus-BPB curves vary by tokenization and training objective; MBPP is noisier than NQ, and some distillation curves are more monotonic.Source: Figure 17 · p. 33, Figure 18 · p. 34

D.1 Leave One Out Error Analysis

  • Leave-one-out fits show less spread for token curves on average than for Bytes and Bytes w/ <eot> curves.Source: Figure 21 · p. 38

D.2 Residual Weighting Scheme Sensitivity Analysis

  • Alternative residual-weighting schemes are more sensitive for Bytes and Bytes w/ <eot> than the unweighted method used in the plots.Source: §D.2 · p. 39

D.3 Choice of the validation data

  • Changing the validation dataset can shift curve crossovers, feather plots, and the amount of overtraining needed to predict asymptotes.At the stated 220B Token Supervised FLOP budget, all five other models have lower BPB on validation dataset 1, but only distilled token does on dataset 2.Source: §D.3 · p. 40, Figure 23 · p. 40

E.1 BPB vs Training FLOPs power law equations

  • The supplied dataset-2 figures and tables cover BPB-versus-FLOPs fits, downstream-error fits, feather plots, and compute/data crossover estimates, but their captions alone do not establish outcomes.Source: Figure 24 · p. 41, Table 19 · p. 42, Table 20 · p. 42

F.1 Marginalize-It and End-Of-Token Pseudocode

  • The byte-probability pseudocode filters teacher-token probabilities by matching the ground-truth byte prefix, accumulates probability by next byte, and normalizes each distribution.Source: §F.1 · p. 46

F.2 Comparing Marginalize-It method with End-Of-Token

The conversion comparison identifies when Marginalize-It preserves teacher byte probabilities and when alternative token paths make it approximate. End-Of-Token handles the illustrated shorter- and longer-path cases by adding an end marker.

F.2 Comparing Marginalize-It method with End-Of-Token

  • For the first byte, both Marginalize-It and End-Of-Token preserve the teacher distribution exactly.Source: Figure 27 · p. 46, §F.2 · p. 46
  • At intermediate positions, both methods give exact byte probabilities when alternative nonzero-probability tokens have the same length as the ground-truth token.Source: §F.2 · pp. 46–47, Figure 28 · p. 47
  • When shorter alternative tokens share a prefix, Marginalize-It misses paths, while adding <eot> lets End-Of-Token preserve the distribution exactly.Source: Figure 29 · p. 48, §F.2 · p. 48
  • When longer alternative tokens share a prefix, Marginalize-It misses their contributions, while End-Of-Token avoids that problem with the added <eot> marker.Source: Figure 30 · p. 49, §F.2 · p. 49

G.1 Why the same BPB may imply different downstream performance?

BPB fixes the probability assigned to each target token but ignores how the remaining probability mass is ranked. Consequently, identical BPB can accompany sharply different decoding accuracy.

G.1 Why the same BPB may imply different downstream performance?

  • BPB depends on target-token probability and ignores how the residual mass is arranged, while downstream accuracy depends on full-distribution ranking.Greedy decoding selects the arg max, so identical target probabilities do not ensure identical outputs.Source: §G.1 · p. 49
  • At pcorrect=0.4 across seven token positions, both models have a total loss of 9.24 bits despite their different rankings.The example phrase is segmented into seven Llama3-8b tokens spanning 20 bytes.Source: §G.1 · p. 49
  • With identical BPB 0.463, the example models achieve 100% versus 0% token accuracy because the target ranks differ.Model A ranks the target first at every step; Model B places a distractor above it.Source: §G.1 · pp. 49–50

Related papers

Loading 2609.12303v1…