Tokenizer Choice For LLM Training: Negligible or Crucial?

1 Introduction

• Excessive token fragmentation can prevent learning long-range dependencies within a fixed context window.Prior work also reports translated passages tokenized into sequences differing in length by up to a factor of 15, affecting inference cost and latency.Source: §1 · p. 1
• No extensive study had holistically examined intrinsic and downstream tokenizer performance in monolingual and multilingual decoder-only LLMs.The authors identify this gap as the motivation for their study.Source: §1 · p. 1
• The study investigates intrinsic and downstream tokenizer performance and whether the two are correlated.Intrinsic evaluation concerns tokenizer characteristics and outputs; extrinsic evaluation measures impact on downstream components such as LLMs.Source: §1 · pp. 1–2

2.1 Tokenization Approaches

• Word tokenization uses whitespace-separated words, subword methods decompose words and handle out-of-vocabulary items, while character tokenization lengthens sequences and raises transformer compute costs.Subword algorithms include Word-Piece, BPE, Byte-Level BPE, and Unigram.Source: §2.1 · p. 2

2.2 Tokenizers in Transformers Models

• Encoder-model studies report tokenizer effects across multilingual BERT, machine translation, Turkish language models, and code-related tasks.Results include roughly equivalent subword approaches versus worse word- and character-level approaches for Turkish, and up to 40% shorter sequences or up to 2% downstream improvements for code tokenizers.Source: §2.2 · p. 2
• Decoder-model work includes intrinsic evaluation and preprocessing ablations, but major LLM publications had not studied extrinsic tokenizer performance.The cited preprocessing ablations held algorithm, vocabulary size, and implementation fixed.Source: §2.2 · p. 2

3 Approach

• The ablation trains BPE and Unigram tokenizers and 2.6B decoder-only models with remaining dataset and model hyperparameters fixed to isolate tokenizer impact.The study uses dedicated datasets for tokenizer and model training.Source: §3 · p. 2

3.1 Data

• Two 70B-word datasets use matched domain mixtures: English-only and multilingual English, German, French, Italian, and Spanish.The data are filtered and deduplicated; the multilingual web-crawled portion is equally distributed across languages by word count.Source: §3.1 · p. 3

3.2 Tokenizer

• The study trains 24 tokenizers across BPE and Unigram, two libraries, language composition, and vocabulary sizes, omitting Hugging Face Unigram.Hugging Face Unigram lacks preprocessing options that cause a large vocabulary discrepancy relative to SentencePiece.Source: §3.2 · p. 3

3.3 Models

• One 2.6B decoder-only model is trained per tokenizer on up to 52B tokens, with monolingual and multilingual GPT-2-tokenizer models as baselines.All models use causal language modeling.Source: §3.3 · p. 3

3.4 Evaluation

• Evaluation first measures intrinsic tokenizer output and vocabulary overlap, then downstream task performance and average training compute per word.Intrinsic metrics include fertility on a 10,000-document held-out set and parity across languages using FLORES-200 parallel sentences.Source: §3.4 · pp. 3–4

4.1 Fertility & Parity

• Monolingual tokenizers on multilingual data yield significantly higher fertility and parity; multilingual tokenizers sharply lower fertility on non-English text but slightly worsen English fertility.Source: §4.1 · p. 4
• Increasing vocabulary size reduces fertility and parity; English fertility for monolingual tokenizers depends less on vocabulary, suggesting 33k may suffice.Source: §4.1 · p. 4

4.2 Vocabulary Overlap

• BPE implementations from Hugging Face and SentencePiece have low, roughly vocabulary-size-constant overlap; multilingual overlap is lower than English.Source: Table 1 · p. 4, §4.2 · p. 4

5.1 Experimental Setup

• The evaluation trains 2.6B decoder-only models for 52.6B tokens and tests them zero-shot across a range of monolingual and multilingual tasks.The tasks span multiple languages and categories, including NLI, reading comprehension, question answering, commonsense reasoning, and classification.Source: §5.1 · p. 4, Table 2 · p. 5

5.3 Computational Costs

• Up to 68% higher training costs occurred for German documents when comparing Multi-UNI-SP-50 with EN-UNI-SP-100; multilingual tokenizers cost less on multilingual documents.For English documents, the compute-cost difference between multilingual and monolingual English tokenizers is marginal.Source: Figure 3 · p. 6, §5.3 · p. 7
• Increasing vocabulary size beyond 50k raises compute costs in all cases, and lower fertility does not offset the additional vocabulary costs.The cost analysis measures average GFLOPS per word over a full training pass, including the backward pass.Source: Figure 3 · p. 6, §5.3 · p. 7

5.2 Downstream Performance

• BPE-SP-33 performs best on average for monolingual tasks, while BPE-SP-100 leads multilingual tasks, followed by BPE-SP-82.For ARC-Easy, the gap between the best- and worst-performing tokenizer is 9%.Source: Table 3 · p. 6, §5.2 · p. 6, Table 4 · p. 6

5.2.1 Impact of the Tokenizer Library

• BPE-SP outperforms BPE-HF on average across languages in both monolingual and multilingual settings.The authors suggest implementation-specific pre- and postprocessing differences may affect vocabulary creation and downstream performance.Source: §5.2.1 · p. 6, Table 5 · p. 7

5.2.2 Impact of the Tokenizer Algorithm

• BPE benefits German and English, while Unigram benefits French and Spanish; Italian follows a different pattern from the other Romanic languages.Source: §5.2.2 · p. 7, Table 5 · p. 7

5.2.3 Impact of the Tokenizer Vocabulary

• Smaller or medium vocabularies of 33k or 50k perform better for monolingual English, while larger vocabularies generally help multilingual settings except German.The best-performing average tokenizer sizes reported are 33k for monolingual English and 100k for multilingual models.Source: §5.2.3 · p. 7, Table 5 · p. 7

• For non-English tasks, low fertility is mainly associated with higher performance, but correlations across tasks are mixed and multilingual task coverage is smaller.Some multilingual tasks show correlation behavior similar to their English translations.Source: §6 · p. 8, Figure 4 · p. 8
• The authors hypothesize that fertility predicts performance only within language-specific vocabulary-size bounds: English fertility gains level off near 33k, while multilingual fertility improves with larger vocabularies.They suggest additional English vocabulary tokens may not capture morphological segmentations and may harm performance while increasing compute; for multilingual tokenizers, larger vocabularies support stronger performance within the non-convergence range.Source: §6 · p. 8
• Fertility and parity are not consistently predictive of downstream performance; low fertility may be necessary but is not sufficient.Source: §6 · p. 8, Figure 4 · p. 8

Conclusion

• Up to 68% higher fertility is associated with greater training costs and prevents learning long-range dependencies within limited context windows.Tokenizers trained with balanced language shares achieve comparably low fertility and parity scores across languages.Source: §7 · p. 8
• English models need a 33k vocabulary, while multilingual models for the five considered languages require up to three times more; SentencePiece outperforms Huggingface.BPE applies in both mono- and multilingual settings.Source: §7 · p. 8
• Intrinsic tokenizer performance has no clear correlation with downstream performance, and any correlation is task-specific.Low fertility may be necessary but is not sufficient for good downstream performance.Source: §7 · p. 8

Future Work

• The authors plan to study more languages, including very diverse ones, and alternative approaches such as SAGE that use context during tokenizer training.Source: §7 · p. 8

Limitations

• Tokenizer-specific hyperparameters were not optimized, a deliberate choice given the computational expense of training all 26 models once, which required approximately 59,000 GPU hours.Source: §8 · p. 9
• The study did not test random-seed effects on performance for a given tokenizer, leaving robustness for future work.Source: §8 · p. 9
• Whether results extrapolate to larger models was not investigated, although use of BPE-SP-33 in state-of-the-art models up to 65B might indicate transfer.Source: §8 · p. 9
• Evaluation covers zero-shot performance only; the authors leave whether results translate to few-shot evaluation for future work.They chose zero-shot partly to avoid introducing few-shot example selection as an additional variable.Source: §8 · p. 9

9 Ethical And Broader Impact

• Neglecting multilingualism in tokenizer training can disadvantage language communities through increased training costs and decreased downstream performance, while raising ethical concerns.The increased costs also translate into a larger carbon footprint and environmental impact.Source: §9 · p. 9

A Corpora

• The curated data draws on The Pile, RedPajama, and standalone datasets, with web documents from Oscars1 generated from three Common Crawl WET archives.Source: A Corpora · p. 14

B Tokenizer

• Experiments use the Huggingface and SentencePiece tokenizer libraries, with SentencePiece standard settings unless otherwise specified.The Huggingface settings that deviate from standard values are listed in Table 9.Source: B Tokenizer · p. 15

C LLM Architecture and Hyperparameters

• The 2.6B-parameter models closely follow the GPT-3 architecture.Source: C LLM Architecture and Hyperparameters · p. 15
• Table 10 provides an overview of the training hyperparameters.Source: Table 10 · p. 15, C LLM Architecture and Hyperparameters · p. 15
• Model training used a fork of Megatron-LM.Source: C LLM Architecture and Hyperparameters · p. 15

D Intrinsic Tokenizer Evaluation

• Multilingual and English vocabularies overlap by 24–34%, similarly across vocabulary sizes; SentencePiece Unigram and BPE overlap slightly more than SentencePiece Unigram and Huggingface BPE.The authors suggest library-specific preprocessing and more similar hyperparameters may explain the algorithm overlap pattern.Source: D Intrinsic Tokenizer Evaluation · p. 15, Figure 5 · p. 16

D.1 Computational Costs Per Word During Training

• Table 11 reports average forward- and backward-pass computational training costs per word, in GFLOPs, for different tokenizers.Source: §D.1 · p. 17, Table 11 · p. 17

E Infrastructure & Computational Costs

• Training 26 models on NVIDIA A100 GPUs took up to 2,304 GPU hours per model and approximately 59,000 GPU hours in total.Source: E Infrastructure & Computational Costs · p. 17