PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperBPE: Space Travel for Language Models

Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, Yejin Choi

arXiv:2503.13423v3cs.CLcs.LG

TL;DR

Whitespace-bounded subword tokenization may miss multi-word semantic units and cross-lingual differences in word boundaries. SuperBPE addresses this by training BPE first on subwords and then on cross-whitespace superwords; under matched 8B-model conditions, it improves downstream performance while reducing inference compute.

  • Problem

    Whitespace is an unreliable delimiter of meaning, while conventional subword tokenization cannot bridge whitespace and may miss multi-word semantic units.

  • Method

    SuperBPE uses a two-stage BPE pretokenization curriculum that first learns subwords with whitespace boundaries and then learns superwords without that restriction.

  • Results

    +4.0% average improvement over BPE across 30 downstream tasks, with 27% less inference compute under matched 8B-model, vocabulary-size, and training-compute conditions.

  • Takeaways & Limitations

    SuperBPE is a local tokenization modification that improves encoding efficiency and downstream performance without changing model architecture, training framework, or decoding strategy.

  • Takeaways & Limitations

    SuperBPE tokenizer training without whitespace pretokenization requires more system memory and CPU time.

Abstract

from arXiv · show

The assumption across nearly all language model (LM) tokenization schemes is that tokens should be subwords, i.e., contained within word boundaries. While providing a seemingly reasonable inductive bias, is this common practice limiting the potential of modern LMs? Whitespace is not a reliable delimiter of meaning, as evidenced by multi-word expressions (e.g., "by the way"), crosslingual variation in the number of words needed to express a concept (e.g., "spacesuit helmet" in German is "raumanzughelm"), and languages that do not use whitespace at all (e.g., Chinese). To explore the potential of tokenization beyond subwords, we introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm to first learn subwords, then superwords that bridge whitespace. This brings dramatic improvements in encoding efficiency: when fixing the vocabulary size to 200k, SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average. In experiments, we pretrain 8B transformer LMs from scratch while fixing the model size, vocabulary size, and train compute, varying *only* the algorithm for learning the vocabulary. Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks (including +8.2% on MMLU), while simultaneously requiring 27% less compute at inference time. In analysis, we find that SuperBPE results in segmentations of text that are more uniform in per-token difficulty. Qualitatively, this may be because SuperBPE tokens often capture common multi-word expressions that function semantically as a single unit. SuperBPE is a straightforward, local modification to tokenization that improves both encoding efficiency and downstream performance, yielding better language models overall.

1 Introduction

SuperBPE challenges whitespace-bounded subword tokenization by learning superword tokens that bridge words, motivated by the fact that whitespace often fails to mark semantic units. Across controlled 8B-model experiments, it improves downstream performance and inference efficiency while preserving the existing model stack.

  • Motivation: Whitespace can separate words that function as single semantic units and varies substantially across languages, including languages without whitespace.These observations motivate testing tokens that can cross whitespace boundaries.
  • Approach: SuperBPE first learns subwords with whitespace pretokenization, then disables that restriction to learn superword tokens spanning multiple words.This pretokenization curriculum extends BPE without changing the model architecture or decoding strategy.
  • Results: +4.0% average improvement over BPE across 30 downstream tasks, including +8.2% on MMLU, while inference requires 27% to 33% less compute.The experiments fix model size, vocabulary size, and training compute while varying only the vocabulary-learning algorithm.
  • Analysis: SuperBPE produces more uniform per-token difficulty, often because its tokens capture common multi-word expressions as semantic units.Examples include fixed prepositional phrases whose individual words are easy to predict under BPE.
  • Implication: SuperBPE is a local tokenization change that improves encoding efficiency and language-model performance without modifying the underlying architecture.The method is designed to integrate with existing language-model ecosystems.

2 SuperBPE

SuperBPE modifies BPE's pretokenization curriculum: it learns subwords first and then permits cross-whitespace merges. This yields higher encoding efficiency than BPE and avoids the suboptimal choices of disabling whitespace pretokenization from the outset, while tokenizer training incurs extra memory and CPU costs.

  • 2.1 Background on BPE: BPE greedily merges the most frequent neighboring token pair until reaching the target vocabulary size, after splitting text into pretokenized chunks.Canonical whitespace pretokenization prevents merges across chunks, so learned tokens remain within word boundaries.
  • 2.2 SuperBPE tokenization: SuperBPE trains in two phases: it learns subwords with whitespace pretokenization up to transition point t, then learns superwords after lifting that restriction.t = T reproduces BPE, while t = 0 removes whitespace pretokenization throughout training.
  • 2.2 SuperBPE tokenization: Tokenizer training without whitespace pretokenization requires more system memory and CPU time because text is less deduplicated in the training dictionary.This overhead occurs during tokenizer construction, which is performed only once.
  • 2.4 Language-model evaluation: 27% less compute at inference time for the SuperBPE model evaluated against BPE under fixed 8B parameters, 200k vocabulary, and training FLOPs.The comparison differs only in the algorithm used to learn the vocabulary.
  • 2.3 Encoding efficiency: 6.63 bytes-per-token at vocabulary size 200k, versus BPE's 4.45 bytes-per-token at the same size.BPE plateaus near 50K and has a theoretical upper bound of 4.68 bytes-per-token, whereas SuperBPE exceeds that bound near 12K vocabulary size.
  • 2.3 Encoding efficiency: SuperBPE is more efficient than disabling whitespace pretokenization entirely because early greedy merges can otherwise join word fragments rather than complete word sequences.The curriculum preserves initial subword learning while still enabling later superword formation.

3 Experiments

The experiments compare BPE and SuperBPE under matched model, vocabulary, training-budget, and context conditions. SuperBPE models outperform BPE across downstream tasks, while transition-point choice trades encoding efficiency against performance.

  • Setup: All compared models use the same vocabulary size and train budget, while SuperBPE transition points vary across t = 80k, 160k, and 180k.The experiments fix vocabulary size at 200,000 and vary only the tokenizer-learning algorithm in the main comparison.
  • Results: SuperBPE models consistently outperform the BPE baseline across 30 downstream tasks under matched vocabulary size and training budget.The strongest 8B SuperBPE model wins on 25 of 30 tasks.
  • Results: 4.0% average improvement over BPE is achieved by the strongest 8B SuperBPE model, which wins on 25 of 30 individual tasks.The largest gains occur on multiple-choice tasks, reaching +9.7% when considered alone.
  • Results: 35% lower inference compute accompanies a +3.1% task improvement when using the most encoding-efficient transition point, t = 80k.Later transition points sacrifice some encoding efficiency for further performance gains.

4 Analysis

The analysis shows that SuperBPE and BPE have similar average BPB but distribute loss differently, with SuperBPE producing more uniform per-token difficulty. Scaling results further show that compute-matched SuperBPE achieves lower BPB across tested model sizes and training budgets.

  • Language modeling: BPB normalizes cross-entropy by encoded bytes, enabling comparisons between models with different tokenizers.Longer tokens generally contain more information and are harder to predict, so raw per-token loss is not directly comparable.
  • Language modeling: 0.7465 BPB, 0.7482 BPB, and 0.7445 BPB were achieved by BPE 8B, SuperBPE 8B, and SuperBPE 11B, respectively.Despite similar BPB values, BPB rankings do not consistently match downstream task performance.
  • Loss distribution analysis: SuperBPE makes fewer predictions with either very high or very low loss than the BPE baseline.The histogram measures per-token losses over a large corpus and indicates a more consistent loss distribution.
  • Loss distribution analysis: Excluding the three most common words reverses the BPB ranking, giving SuperBPE 0.02 lower BPB than BPE.Under BPE, “the,” “of,” and “to” appear an order of magnitude more often because SuperBPE often merges them into larger superword tokens.
  • Loss distribution analysis: SuperBPE superword tokens often capture fixed multi-word expressions whose component words have little contextual variation.Examples include “by accident,” “of course,” “for a living,” “depend on,” and “distinction between.”
  • Scaling: In the under-trained regime, both SuperBPE variants achieve lower BPB than BPE, while compute-matched SuperBPE is best in the over-trained regime.The separation between models increases with further over-training; scaling comparisons use BPB because small-model downstream evaluations are too noisy.

5 Related Work

Related work has explored multi-granularity, multi-word, tokenizer-free, and tokenizer-transfer approaches, but these methods differ in scope, effectiveness, or required model changes. SuperBPE is positioned within this broader effort to move beyond conventional subword tokenization.

  • Tokenization beyond subwords: Prior work explored multiple text granularities and frequency-based multi-word tokens, mainly in limited machine-translation settings with mixed effectiveness.Naively disabling BPE pretokenization has also been reported to severely degrade model performance.
  • Multi-token prediction: Multi-token prediction lets language models predict multiple tokens per time step but fixes the number predicted and requires architectural modifications.Its extra module can be discarded at inference time, unlike a tokenizer change that directly alters segmentation.
  • Tokenizer-free language modeling: Tokenizer-free methods model bytes directly and introduce alternative architectures or patching strategies to address longer sequences and increased compute.These approaches remove conventional tokenization rather than extending BPE with superword tokens.
  • Tokenizer transfer: Tokenizer-transfer methods adapt trained models to new tokenizers through pretraining interventions, continued training, self-distillation, heuristics, or hypernetworks.The paper identifies upgrading existing models to SuperBPE while maintaining performance as future work.

6 Conclusion

The conclusion presents SuperBPE as a local extension of BPE that allows superword tokens and improves downstream performance while reducing inference cost. The method changes tokenization rather than the model architecture, training framework, or decoding strategy.

  • Conclusion: SuperBPE extends BPE beyond subwords to produce superword tokens that can bridge word boundaries.The conclusion frames tokenization as the interface through which language models process text.
  • Conclusion: Replacing subword BPE with SuperBPE during pretraining improves performance across a large suite of downstream tasks while reducing inference cost.The conclusion characterizes SuperBPE as a straightforward modification to tokenization.

A.1.1 Tokenizer training data

Tokenizer training uses sampled olmo-mix data with safeguards against duplicated long documents and implementation choices for pretokenization, token limits, and model configuration.

  • Tokenizer training data: 10 GB of olmo-mix data is sampled for tokenizer training because larger datasets showed no encoding-efficiency benefit beyond approximately 10 MB.Documents are sampled uniformly from OLMO2 stage 2 pretraining data.
  • Tokenizer training data: The longest 1% of documents are truncated because extreme duplication can create unusual tokens and disproportionately affect the small tokenizer-training dataset.The authors report that the longest 1% comprised 15% of the data and recommend attention to deduplication.
  • Tokenizer training data: SuperBPE token length is capped at 4 words to limit disclosure of duplicated boilerplate or training data, without measurable effects on encoding efficiency or trained-LM results.This issue persisted even after truncating the longest documents.
  • Pretokenization rules: Whitespace pretokenization uses the GPT-2 tokenizer’s default regex, excluding its contraction-splitting rule because that choice is not universal and creates edge cases.Digits are independently grouped into blocks of three from the right, and colon-space sequences are excluded from learned tokens for question-answering compatibility.
  • Model configuration: Inference-compute-matched models are configured by expressing flops per token as a polynomial and grid-searching hidden dimension and layer count.The selected configuration expands transformer parameters by the most uniform factors, measured through the standard deviation of log parameter ratios.

A.4 Evaluation Suite

The evaluation suite combines 22 DataComp-LM tasks with 8 additional tasks and scores decoded outputs using exact-match, answer-containment, or coding pass@10 criteria.

  • Evaluation suite: 30 downstream tasks form the evaluation suite, combining DataComp-LM’s 22 core tasks with 8 additional tasks including MMLU, GSM8K, and string-manipulation benchmarks.The added tasks include categories known to challenge language models through tokenization.
  • Evaluation protocol: Multiple-choice tasks are scored by exact answer match, open-ended tasks by exact ground-truth containment, and coding tasks by pass@10.Pass@10 estimates whether at least one of ten sampled solutions is correct; coding generations use 20 samples.
  • Evaluation protocol: Evaluations use five in-context examples except for CoQA and coding tasks, with up to 5,000 examples per dataset.HumanEval and MBPP are evaluated zero-shot, while CoQA’s conversational context supplies examples naturally.
  • Task coverage: The suite spans reasoning, knowledge, language understanding, coding, arithmetic, commonsense, and string-manipulation tasks.Examples include ARC, GSM8K, MMLU, HumanEval, CUTE, DROP, and BIG-bench tasks.
  • Context analysis: At fixed token indices, SuperBPE has lower average next-token loss, but this advantage disappears when context is matched by preceding bytes.Figure 6 compares average bits-per-byte across token and byte positions.

B.3 BPB evaluation

Bits-per-byte during pretraining is reported for BPE and SuperBPE models, with the 11B SuperBPE model reaching the lowest listed final value.

  • BPB evaluation: 0.7445 BPB is reached by SuperBPE 11B at the end of training, versus 0.7465 for BPE 8B and 0.7482 for SuperBPE 8B.Figure 7 reports these bits-per-byte values for the compared models.

B.4 Additional scaling experiments

Tokenizer efficiency reduces the token-based context and compute burden, enabling either more training text at fixed model size or larger models at matched inference compute.

  • Fixed model size: 1/α^2 attention compute, 1/α non-attention compute, and 1/α activation memory or KV-cache size follow from a tokenizer efficiency ratio α.These savings assume the effective context length in bytes is held constant.
  • Fixed model size: For short contexts, total compute savings approach 1/α, while longer contexts can approach 1/α^2.The scaling depends on how much attention compute contributes to the total.
  • Fixed model size: At fixed model size, a more efficient tokenizer lets training process more text within the same budget and reduces inference compute and memory.The paper notes that seeing more training text may improve downstream performance by increasing exposure to relevant examples.
  • Scaling experiments: The spread between BPE and SuperBPE scaling settings decreases as model and training-token scale increase.Figure 8 compares a parameter-matched SuperBPE model and an inference-compute-matched SuperBPE model with the BPE baseline.
  • Matched inference compute: When inference compute is the constraint, model size can be increased to absorb tokenizer-efficiency savings.The paper considers this regime separately from applications where model size must remain controlled.
  • Matched inference compute: Scaling model size by up to α^2 can reduce the per-step token-to-parameter ratio by as much as 1/α^3.The paper relates this trade-off to diminishing gains reported when the ratio of training tokens to parameters becomes too low.

B.4.3 Experiments

Experiments compare SuperBPE models with BPE baselines under matched parameter or inference-compute settings, finding that inference-compute matching is generally strongest. SuperBPE’s second-stage vocabulary contains many superword tokens, especially with later transition points.

  • Experimental setup: SuperBPE models are trained alongside BPE baselines at 680M and 1.9B parameter scales across varying training-token budgets.For each baseline, one SuperBPE model matches parameter count and another matches inference compute cost.
  • Experiments: Matching inference compute is almost universally best, while matching parameter count is generally worse than BPE except in undertrained regimes.Differences between settings increase with overtraining but decrease when model size and training tokens scale together.
  • Superword vocabulary: For transition points above 100k, almost all tokens learned in Stage 2 are superword tokens.At transition points below 80k, the number of superword tokens remains around 120k.

C.2 Analysis of token frequencies in encoding

SuperBPE produces a more evenly distributed token-frequency profile and uses more of its vocabulary than BPE. This analysis also relates tokenization choices to prompt-boundary distortions and broader representational effects.

  • Token frequency: SuperBPE token frequencies decay more slowly with rank than BPE frequencies, reducing the proportion of very low-count tokens.The authors suggest this may reduce the prevalence and severity of glitch tokens.
  • Vocabulary coverage: 90% of SuperBPE tokens cover the data at t = 80k, compared with 57% of BPE tokens covering 99% of the data.At t = 180k, SuperBPE reaches 70% for the corresponding coverage statistic, while BPE retains a long tail of infrequent tokens.
  • Prompt-boundary distortion: Pretokenization can prevent BPE tokens from crossing prompt-completion boundaries, but this mitigation does not apply reliably to Chinese, Japanese, programming languages, or constrained generation.The underlying boundary problem is shared by BPE tokenizers, while chat models with special input-response tokens do not have this issue.
  • Semantic units: Prior work suggests that language models may aggregate multi-token entities internally, motivating more semantically cohesive input segments such as single-token multi-word expressions.This connection is presented as a possible rationale for analyzing superword tokenization.
Loading 2503.13423v3…