Source-linked AI summary
Byte Pair Encoding is Suboptimal for Language Model Pretraining
Kaj Bostrom, Greg Durrett
TL;DR
The paper addresses the lack of direct evidence about how subword tokenization affects language-model pretraining. It compares BPE and unigram LM tokenization through morphology analyses and controlled masked-language-model experiments, finding that unigram LM matches or outperforms BPE across English and Japanese downstream tasks.
Problem
The literature lacks a direct evaluation of how subword tokenization affects language-model pretraining, despite pretrained models requiring a fixed vocabulary for downstream use.
Method
The authors compare BPE and unigram LM tokenization, then train otherwise controlled ROBERTA-BASE masked language models from scratch on English and Japanese data for downstream evaluation.
Results
Across downstream tasks and English and Japanese, unigram LM tokenization matches or outperforms BPE, with up to 10% improvement in Japanese question answering.
Takeaways & Limitations
The results suggest that tokenization encodes substantial inductive bias and that unigram LM may be preferable for developing future pretrained language models.
Abstract
from arXiv · showhide
The success of pretrained transformer language models (LMs) in natural language processing has led to a wide range of pretraining setups. In particular, these models employ a variety of subword tokenization methods, most notably byte-pair encoding (BPE) (Sennrich et al., 2016; Gage, 1994), the WordPiece method (Schuster and Nakajima, 2012), and unigram language modeling (Kudo, 2018), to segment text. However, to the best of our knowledge, the literature does not contain a direct evaluation of the impact of tokenization on language model pretraining. We analyze differences between BPE and unigram LM tokenization, finding that the latter method recovers subword units that align more closely with morphology and avoids problems stemming from BPE's greedy construction procedure. We then compare the fine-tuned task performance of identical transformer masked language models pretrained with these tokenizations. Across downstream tasks and two languages (English and Japanese), we find that the unigram LM tokenization method matches or outperforms BPE. We hope that developers of future pretrained LMs will consider adopting the unigram LM method over the more prevalent BPE.
1 Introduction
Pretrained language models rely on subword tokenization to handle open-vocabulary text, but the vocabulary cannot be changed for downstream use. This makes comparing tokenization methods directly important; the paper contrasts BPE with unigram LM tokenization and evaluates their effects on English and Japanese tasks.
- Subword tokenization breaks rare words into units ranging from characters to full words, helping language models handle open-vocabulary text.
- A pretrained model’s subword vocabulary cannot be altered, so downstream applications must use its original vocabulary for input and output.
- Existing pretrained language models use varied tokenization methods, including WordPiece, character-level BPE, byte-level BPE, and unigram LM tokenization.
- The paper characterizes BPE and unigram LM tokenization, comparing their vocabularies with morphological references and evaluating models pretrained with each method.
- Up to 10% improvement over BPE was observed with unigram LM tokenization in Japanese question answering, alongside performance differences across tasks and languages.
2 Algorithms
BPE builds vocabularies bottom-up through ordered frequency-based merges, whereas unigram LM starts with a larger vocabulary and prunes tokens using language-model criteria. Their tokenization procedures differ accordingly, with unigram LM using Viterbi decoding and producing examples that more closely reflect meaningful subword structure.
- BPE: BPE constructs a vocabulary by repeatedly merging the most frequent adjacent token pair until reaching the target vocabulary size.
- BPE: BPE tokenization applies the learned merges to new text in the same order used during vocabulary construction.
- Related methods: WordPiece resembles BPE but scores potential merges using an n-gram language model, and the authors cannot compare against it because its implementation is unavailable.
- Unigram LM: Unigram LM begins with a vocabulary superset and prunes tokens to the target size based on unigram language-model perplexity.
- Unigram LM: Unigram LM uses Viterbi inference to select the maximum-likelihood segmentation under its vocabulary and language-model parameters.
- Tokenization examples: Example tokenizations show unigram LM recovering units such as nano technology and morphological pieces such as -ly, -s, pre-, and tri-.
- Vocabulary profiles: The unigram LM method produces longer English tokens on average and uses its vocabulary space more effectively, with more tokens at moderate frequencies.
3 Comparison of Segmentations
Compared with BPE, unigram LM tokenization produces subword units that align more closely with morphology and may allocate vocabulary space more economically. Its advantage reflects BPE’s greedy frequency-based construction, which absorbs affixes and creates low-frequency intermediate tokens.
- 3.1 Morphology: Unigram LM tokenization recovers common affixes such as -ly, -s, pre-, and tri- more often than BPE.BPE instead absorbs affixes into adjacent units and can produce meaningless single-character units.
- 3.2 Vocabulary Allocation: BPE’s greedy frequency-based construction absorbs common affixes and punctuation into other tokens, despite those affixes remaining in its vocabulary.When three tokens commonly occur together, BPE can leave an intermediate token that rarely occurs independently.
- 3.1 Morphology: Unigram LM tokenization tends to produce longer subword units than BPE, closer to gold-standard English morphs averaging approximately 6 characters.This comparison concerns mean subword-unit length across English Wikipedia and the gold morph length distribution.
- 3.1 Morphology: For both English and Japanese, unigram LM segmentations correspond more closely to morphological reference segmentations.English comparisons use CELEX2 morpheme boundaries, while Japanese comparisons use MeCab and UniDic references.
- 3.2 Vocabulary Allocation: Unigram LM tokenization may allocate vocabulary more economically by avoiding BPE’s excess intermediate junk tokens and reducing embedding redundancy from repeated affix merges.The BPE dead zone contains about 1500 more entries, while unigram LM can represent recurring linguistic functions in affix embeddings.
4 Downstream Task Experiments
The authors pretrained matched masked language models with BPE and unigram LM tokenizations under controlled conditions, then fine-tuned them across English and Japanese tasks. Unigram LM tokenization performed better on every evaluated task, with larger gaps on SQuAD and MNLI and the largest gap on Japanese TyDi.
- Experimental Setup: The study pretrained four ROBERTA-BASE masked language models from scratch while controlling data, objective, and optimization procedure.Each model pair used either BPE or unigram LM tokenization, with a vocabulary of 20,000 tokens.
- Evaluation: The models were fine-tuned on SQuAD, MNLI, CoNLL 2003 English NER, and Japanese minimal-answer TyDi QA.These tasks cover question answering, textual entailment, and named-entity recognition across English and Japanese.
- Results: Unigram LM tokenization produced better fine-tuned performance than BPE on all evaluated tasks.The comparison used models pretrained with the two tokenizations and then fine-tuned using the study’s task implementations.
- Results: The largest performance gap appeared on Japanese TyDi, while larger gaps were also evident on SQuAD and MNLI.Japanese TyDi had approximately 5k training examples, compared with approximately 88k available for SQuAD fine-tuning.
5 Conclusion
The paper concludes that input encoding affects pretrained language models’ end-task performance, indicating that tokenization encodes substantial inductive bias. It therefore suggests unigram LM tokenization as a better choice for developing future pretrained models.
- 5 Conclusion: The choice of input encoding affects how well pretrained language models perform on end tasks.The authors interpret this result as evidence that tokenization encodes a surprising amount of inductive bias.
- 5 Conclusion: The authors suggest unigram LM tokenization may be the better choice for developing future pretrained models.
A Hyperparameters
The section lists the pretrained model, software, optimizer settings, and training and fine-tuning hyperparameters used in the experiments.
- ROBERTA-BASE was used for pretraining.
- Pretraining used polynomial decay, peak learning rate 0.0005, 10000 warmup steps, batch size 2048, and 125000 total updates.
- Fine-tuning used linear decay, peak learning rate 5e-5, batch size 32, and three epochs.
- Fine-tuning sequence lengths were 512 for SQuAD and TyDi QA and 128 for MNLI and NER, with passage stride 192 for the question-answering tasks.
- SentencePiece used a vocabulary size of 20000 with unigram LM α 0.25.
B Japanese vocabulary comparison
In Japanese, unigram LM tokenization produces longer tokens and recovers morphologically meaningful verb-conjugation units, while BPE can merge common particles and punctuation into less meaningful tokens.
- BPE merges common Japanese tokens, including particles and punctuation, even when they do not form meaningful units.
- The unigram LM method recovers ていく and てしまう, productive components of the Japanese verb conjugation system.
- The unigram LM method produces longer Japanese tokens, while Japanese token-frequency profiles resemble English and show a less pronounced “dead zone” effect.