Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

• Inefficient tokenization reduces effective learning per training step and can lower inference throughput, increase cost, and accelerate context-window exhaustion.Source: §1 · p. 1
• The community lacks a standardized, multi-metric framework for comparing tokenizers across languages, while prior work typically examines individual metrics or a narrow language set.Source: §1 · p. 1
• Tokka-Bench provides a reusable benchmark and tool covering 100 natural languages across 30+ scripts and 20 programming languages, with an interactive dashboard.Source: §1 · p. 1
• The framework evaluates five complementary metrics spanning compression efficiency, vocabulary coverage, subword granularity, word-boundary alignment, and vocabulary composition.Source: §1 · p. 1
• Its language-aware segmentation adapts the definition of a linguistic unit to each writing system.Source: §1 · p. 1
• The demonstration analyzes seven BPE tokenizers across three generations.Source: §1 · p. 1

• Prior studies introduced subword fertility for comparing BERT tokenizers and reported that monolingual performance correlates with tokenizer quality across 22 typologically diverse languages (´Acs, 2019; Rust et al., 2021).Source: §2 · p. 1
• Prior work reports low-resource languages needing up to 15× more tokens for equivalent content and speakers being overcharged under token-based API pricing (Petrov et al., 2023; Ahia et al., 2023).Source: §2 · pp. 1–2
• Tokka-Bench extends this work with a unified multi-metric framework covering an order of magnitude more languages and using language-aware segmentation.Source: §2 · p. 2

3.1 Datasets

• Evaluation text comes from FineWeb for English, FineWeb-2 for 99 other natural languages, and StarCoder for 20 programming languages, with a contiguous 2 MB UTF-8 block sampled per language.Source: §3.1 · p. 2

3.2 Tokenizers Evaluated

• The study evaluates seven tokenizers spanning vocabulary sizes from 50K for GPT-2 to 262K for Gemma 3.Source: §3.2 · p. 2

3.3 Metrics

• Five per-language metrics are computed in one tokenization pass; comparisons are most reliable within a language because UTF-8 widths and linguistic-unit definitions vary.Source: §3.3 · p. 2

3.4 Language-Aware Segmentation

• Units vary by writing system: whitespace-delimited words, non-whitespace characters for logographic and abugida scripts, or tsheg-delimited syllables with a character fallback for Tibetanfamily scripts.Source: §3.4 · p. 2

4 Results

• All comparisons are within-language; cross-language differences are confounded by encoding overhead and differing unit granularity.Source: §4 · p. 2

4.1 Tokenization Efficiency

• 5.5×: Gemma 3 reaches 9.3 bytes/token versus GPT-2’s 1.7 for Hindi; larger vocabulary does not uniformly improve efficiency.GPT-2 consistently trails modern tokenizers, with the smallest efficiency gains for English.Source: §4.1 · pp. 2–3, Figure 1 · p. 3

4.2 Vocabulary Coverage

• Kimi K2 has 42K unique Chinese tokens versus GPT-2’s 6.5K, while Gemma 3 activates 5.3% of 262K per language versus GPT-2’s 16.8% of 50K.Within each language, unique-token coverage reflects vocabulary allocated to that script.Source: Figure 2 · p. 3, §4.2 · p. 3

4.3 Subword Fertility

• gpt-oss fertility is 1.3 for English, 1.5 for French, and 1.7 for German; GPT-2 reaches 7.5 for Hindi, while Thai and Chinese character-unit values can fall below 1.0.Fertility is measured in words for alphabetic scripts and characters for CJK and Southeast Asian scripts.Source: §4.3 · p. 3

4.4 Programming-Language Convergence

• Post-2023 Llama 3.1, Kimi K2, and gpt-oss have nearly identical bytes-per-token across eight programming languages; GPT-2 lags by 29–51%.The proposed explanations are overlapping code corpora and programming languages’ constrained ASCII character set.Source: §4.4 · p. 3

4.5 Vocabulary Size vs. Efficiency

• Gemma 3 (262K vocabulary) and gpt-oss (200K) achieve 4.30 and 4.25 bytes/token, respectively, while Kimi K2 (164K) achieves 3.19.Kimi K2’s result is associated with concentrating vocabulary on CJK subwords rather than maximizing breadth.Source: §4.5 · p. 3

5 Conclusion

• Tokka-Bench’s demonstration analysis suggests vocabulary allocation strategy matters more than raw vocabulary size, while programming-language tokenization has converged among recent models; it also provides language-aware segmentation and an interactive dashboard.Source: §5 · p. 4
• The demonstration analysis finds that vocabulary allocation strategy matters more than raw vocabulary size and programming-language tokenization has converged among recent models.Source: §5 · p. 4
• Future work includes correlating tokenizer metrics with downstream performance, cross-linguistic normalization, multiple samples per language, and evaluation of non-BPE approaches.Source: §5 · p. 4

6 Limitations

• Cross-language comparisons are confounded because fixed 2 MB samples represent different text amounts across scripts, while bytes-per-token and fertility reflect encoding and unit differences.The metrics are most reliable for comparing tokenizers within a single language.Source: §6 · pp. 4–5
• Metrics use one 2 MB block per language without confidence intervals, and topic or domain shifts could affect results, particularly unique token counts.Source: §6 · p. 5
• All evaluated tokenizers use BPE variants, and the study does not test whether tokenizer-intrinsic metrics predict downstream task performance.WordPiece, Unigram, and byte-level approaches are outside the demonstration's scope.Source: §6 · p. 5