PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

Ivan Dobrovolskyi

arXiv:2608.21384v1cs.CLcs.AI

TL;DR

Cyrillic-script languages can incur higher tokenization costs and reduced context capacity, but evidence across languages and production tokenizers remains limited. This paper benchmarks the disparity and tests inference- and tokenizer-level mitigations, finding substantial Ukrainian overhead alongside effective compression and balanced-vocabulary remedies.

  • Problem

    Evidence is limited on how production tokenizers allocate efficiency across Cyrillic and Latin scripts and what this means for Ukrainian tokenization cost and context capacity.

  • Method

    The study benchmarks nine production tokenizers across five languages and evaluates prompt compression and balanced BPE vocabulary allocation as mitigation strategies.

  • Results

    Across five languages, efficiency favors the web-dominant script; Ukrainian overhead reaches 68–121% on modern tokenizers, while mitigations reduce input length 47–49% and the UK/EN ratio from 2.22× to 1.30×.

  • Takeaways & Limitations

    Training-data composition appears to contribute to Cyrillic overhead, while prompt compression and proportional vocabulary allocation offer practical mitigation without architectural changes.

  • Takeaways & Limitations

    The balanced tokenizer was not evaluated for downstream LLM performance, and the e-commerce benchmark may not represent other domains.

Abstract

from arXiv · show

Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations, covering 8.37 million word forms. On a corpus benchmark, Ukrainian shows 68-121% token overhead on modern tokenizers and 220% on the older cl100k, measured through full-text fertility on the BrUK and Brown corpora. Overhead is negatively associated with Cyrillic vocabulary allocation in the subset with independently verified English baselines, although the association is not statistically significant (Spearman rho = -0.536, p = 0.215, n = 7). We evaluate two mitigation strategies. LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark of 1,536 products and 145 queries, with no compression-induced value losses among 80 retrievable cases. A balanced byte-level BPE tokenizer trained with a 200K vocabulary cap, converging at 158,184 actual entries, reduces the held-out UK/EN ratio from 2.22x to 1.30x. Romanization increases Ukrainian token counts by 2-19% on most tokenizers. Across the five languages, tokenization efficiency favors the script more prevalent in web data. These findings indicate that training data allocation contributes to Cyrillic tokenization overhead and that mitigation is possible at both inference and tokenizer-design stages.

1 Introduction

This introduction frames Cyrillic tokenization overhead as a consequence of English- and Latin-dominated BPE training and the higher UTF-8 byte cost of Cyrillic code points. It presents a five-language analysis and balanced-tokenizer intervention, including a reduction in the Ukrainian-to-English token ratio from 2.22× to 1.30×.

  • Problem: Cyrillic code points require two UTF-8 bytes, compared with one byte for ASCII Latin letters, and BPE training corpora are predominantly English and Latin-script.The resulting Cyrillic byte sequences appear less frequent in training data.
  • Related work: Up to 15× more tokens are required by low-resource languages than by English, while Ukrainian overhead reaches 1.7–3.3× across foundational models.Prior work also proposed post-hoc vocabulary additions for languages whose individual characters are absent from tokenizer vocabularies.
  • Contributions: 8.37 million word forms across Ukrainian, Serbian, Kazakh, Azerbaijani, and Moldovan support a cross-linguistic analysis of overhead relative to standardized Cyrillic and Latin representations.The analysis shows overhead consistently tracks each language’s dominant web script.
  • Contributions: 2.22× to 1.30× is the reduction in the UK/EN ratio achieved by balanced BPE tokenizers under a 200K vocabulary cap.This is presented as the second direction of the study’s contribution.
  • Motivation: Ukrainian is the primary case study because AI assistants are actively deployed commercially and script reform through romanization is publicly debated.The authors state that the findings generalize to other underrepresented Cyrillic-script languages and inform multilingual tokenizer design.

2 Related Work

Prior work established that multilingual BPE tokenization can create language-based efficiency and cost disparities, including Ukrainian-specific measurements. This study builds on that benchmark while extending evaluation across scripts, prompt compression, and balanced tokenizer training.

  • Tokenization fairness: BPE fertility varies by an order of magnitude across languages and increases non-English users’ cost per semantic unit.These findings frame tokenization as both an efficiency and economic-fairness issue.
  • Ukrainian benchmark: The first Ukrainian-specific measurement benchmarked nine tokenizers on BrUK, providing this study’s methodological starting point for Study 1.The present work extends that benchmark to newer 2026 tokenizers and adds cross-script evaluation, prompt compression, and balanced tokenizer training.
  • Vocabulary adaptation: Churchill and Skiena proposed per-character tokens for scripts lacking individual Unicode characters in the vocabulary, improving hidden-state similarity on Llama 3.2 1B for 12 low-resource languages.Related work also includes multilingual subword-vocabulary specialization and embedding initialization for new languages.
  • Prompt compression: LLMLingua-2 reformulated prompt compression as token classification with XLM-RoBERTa, achieving 2–5× compression with minimal quality loss on English benchmarks.Other approaches include perplexity-based pruning, self-information token selection, and extractive or abstractive compression.
  • Information-theoretic context: Ukrainian character-level entropy was measured at 1.20 bits per character with 76.4% redundancy, comparable to English at approximately 1.0–1.3 bits.The estimates provide information-theoretic context but are not directly comparable because they use different corpora and procedures.

3 Data and Methods

The study evaluates tokenization across Ukrainian and four other languages using large standardized word-form dictionaries and balanced Ukrainian–English corpora, alongside retrieval-compression and tokenizer-training experiments. It defines corpus fertility and script-comparison metrics, then assesses factual accuracy on deterministic e-commerce queries and held-out data.

  • Metrics: Fertility is defined as F(t,c) = N_tokens(t,c) / N_words(c), with Ukrainian corpus overhead R(t) = F(t,BrUK) / F(t,Brown).Romanization and script comparisons use Δ_rom and Δ_cyr, respectively, with positive Δ_rom indicating increased token counts after romanization.
  • Corpus benchmarking: 1.34 million Ukrainian words across 805 balanced-genre texts were tokenized with nine commercial encodings, using a genrematched 1.23-million-word Brown corpus for available English baselines.The encodings include o200k, Llama 4, Gemma 4, Qwen 3/3.5, Mistral Small, Grok, Claude 4.5/4.6, and cl100k.
  • Cross-linguistic comparison: 8.37 million word forms across Ukrainian, Serbian, Moldovan, Kazakh, and Azerbaijani were tokenized in native Cyrillic and officially romanized Latin representations.Ukrainian romanization used KMU 55, DSTU 9112:2021 System A, and DSTU 9112:2021 System B.
  • Retrieval-compression evaluation: 1,536 products and 145 deterministic queries supported retrieval-compression evaluation across seven query types using structured API records and expected answers.The records contained product titles, prices, brands, sellers, availability, technical specifications, review counts, and ratings.
  • Tokenizer training: 821K-word BrUK and Brown training corpora supported held-out evaluation of balanced 50/50 English–Ukrainian byte-level BPE tokenizers at 32K, 50K, 100K, and 200K vocabulary sizes.Training used an 80/20 train/test split with seed 42; the 100K setting also varied the Ukrainian corpus share from 0% to 100%.
  • Evaluation: 80 retrievable queries received end-to-end validation by comparing Gemini 2.5 Flash answers from raw and compressed contexts against expected values.Deterministic factual checks covered all 145 queries, while retrieval depth varied by query type, including top_k=3 for comparisons and 5 for price lookups.

4 Results

Results show substantial Ukrainian tokenization overhead on production tokenizers, linked to Cyrillic vocabulary allocation and web-script prevalence. Compression and balanced tokenizer training reduce this overhead while preserving retrieval-relevant information.

  • Tokenization overhead: 1.68×–3.20× Ukrainian overhead occurs among tokenizers with independently verified English baselines, with modern systems clustering at 1.68×–2.21× and cl100k reaching 3.20×.Qwen 3 produces 2.82 tokens per Ukrainian word and has a small Cyrillic vocabulary.
  • Tokenization overhead: Spearman ρ = −0.536, p = 0.215, n = 7 indicates a nonsignificant association between greater Cyrillic vocabulary allocation and lower Ukrainian overhead.Claude and Qwen 3 were excluded because comparable complete-case measurements were unavailable.
  • Romanization and scripts: +9.5% to +19.0% token counts result from Llama 4 romanization, while five of six modern tokenizers increase tokens after romanization.Modern byte-level BPE tokenizers rely on learned Cyrillic byte merges, which romanization disrupts.
  • Romanization and scripts: Five-language comparisons show that the cheaper script matches the language’s dominant web script: Cyrillic for Ukrainian and Kazakh, and Latin for Serbian and Azerbaijani.The comparison used the o200k tokenizer.
  • Mitigation: 46.8%–49.0% reductions in Ukrainian e-commerce RAG context length were achieved by LLMLingua-2 across all six tested tokenizers.Among 80 retrievable cases, no compression-induced losses affected prices, ratings, brands, or availability; retrieval succeeded for 80 of 145 queries.
  • Mitigation: 1.30× versus 2.22× held-out UK/EN fertility was achieved by balanced training with a 200K cap and 158,184 actual entries.Single-token words rose from 22.7% to 34.1%, while words requiring at least four tokens fell from 30.3% to 9.4%.

5 Discussion

The discussion attributes Cyrillic tokenization overhead primarily to training-data composition and shows that mitigation can reduce Ukrainian input length without observed retrieval losses. It also cautions that the vocabulary–overhead association is suggestive but statistically nonsignificant, while romanization can be counterproductive.

  • 5 Discussion: Training-data composition, rather than UTF-8 byte length alone, appears to contribute importantly to Cyrillic tokenization overhead.Cross-linguistic results indicate tokenizers favor the script most prevalent in available web data, while the balanced-tokenizer experiment supports proportional training-data effects.
  • 5 Discussion: 47–49%: Ukrainian contexts were reduced by LLMLingua-2, with no compression-induced loss in checked values or end-to-end accuracy across 80 retrievable cases.The discussion notes that benchmark and evaluation differences prevent direct cross-linguistic entropy comparison.
  • 5 Discussion: Romanization is counterproductive for languages such as Ukrainian whose modern tokenizers have already learned Cyrillic byte merges.Across all five tested languages, efficiency favors whichever script dominates the web, with implications for proposed script changes.
  • 5 Discussion: ρ = −0.536, p = 0.215: the correlation between Cyrillic vocabulary size and tokenization overhead is suggestive but not conventionally significant across seven tokenizers.The causal interpretation is limited because the analysis uses only tokenizers with complete UK/EN ratio data.
  • 5 Discussion: −41%: OpenAI cl100k → o200k showed within-lab generational improvement alongside Cyrillic-vocabulary expansion from 729 to 14,208 tokens.The discussion presents this comparison and the balanced-tokenizer experiment as complementary evidence to the correlation analysis.

6 Conclusion

Ukrainian tokenization overhead is substantial across tested tokenizers, while prompt compression and balanced vocabulary allocation provide effective mitigation. Compression reduced input length without observed losses among retrievable cases, and vocabulary balancing improved the UK/EN ratio under a 200K cap.

  • 68–121% overhead occurred on modern tokenizers, reaching 220% on the older cl100k tokenizer.
  • 47–49% prompt compression reduced Ukrainian input length without observed losses among retrievable cases.
  • 2.22× to 1.30×: balanced vocabulary allocation reduced the UK/EN ratio under a 200K vocabulary cap, with 158,184 actual entries.

7. Limitations

The study’s limitations include estimated English costs, a single-domain e-commerce benchmark, and an unevaluated proof-of-concept tokenizer. It used public or research-oriented data without collecting personal or sensitive individual-level information.

  • Scope and evaluation: English cost estimates apply corpus-level fertility ratios to RAG-specific Ukrainian counts rather than using parallel translations.This avoids translation artifacts but makes the English figures estimates.
  • Scope and evaluation: The e-commerce benchmark covers one domain, so legal or medical applications may show different Cyrillic/Latin content ratios.The passage specifically identifies domain variation as a limitation.
  • Scope and evaluation: The balanced tokenizer remains a proof-of-concept because downstream LLM performance was not evaluated.The supplied passage ends while stating this limitation.
  • Data and ethics: The study used public or research-oriented corpora, tokenizer outputs, and structured product information without collecting personal data, private messages, profiles, or sensitive individual-level information.Its stated aim is to support more equitable AI access for underrepresented-script language communities.

Savings

A small set of frequent Ukrainian words accounts for substantial tokenizer-level savings if represented as single tokens. These estimates are upper bounds because integrating new tokens requires embedding adaptation or retraining.

  • Savings: −7.6% of total tokenization is the estimated savings from making the top 100 Ukrainian words single tokens.The estimate uses Qwen 3 with a Rust profiler on BrUK and defines savings as freq × (tokens − 1).
  • Savings: −18.1% of total tokenization is the estimated savings from making the top 500 Ukrainian words single tokens.The top 500 are selected from 24,719 unique words with frequency ≥ 5 in the full BrUK corpus.
  • Savings: These savings are tokenizer-level upper-bound estimates, because adding tokens to an existing model requires embedding adaptation or retraining.The calculations use current cost = freq × tokens and savings = freq × (tokens − 1).
Loading 2608.21384v1…