Source-linked AI summary
Language Model Tokenizers Introduce Unfairness Between Languages
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, Adel Bibi
TL;DR
The paper examines unequal treatment across languages despite multilingual language models, locating a key disparity at tokenization. It systematically measures tokenization differences across translated texts and finds substantial disparities in costs, latency, and context capacity, motivating multilingually fair subword tokenizers.
Problem
Multilingual language models can provide unequal performance and accessibility across languages, making comparable access important.
Method
The paper defines tokenizer parity and measures tokenization premiums across 2,000 parallel Wikipedia sentences translated into 200 languages.
Results
Tokenization disparities reach 15 times for some language pairs, while cost can be at least 2.5 times higher, processing time twice as long, and context capacity more than an order of magnitude different.
Takeaways & Limitations
The paper advocates multilingually fair subword tokenizers that produce similar encoded lengths for the same content across languages.
Takeaways & Limitations
A balanced training dataset alone is insufficient because languages can share tokens, favoring languages from more numerous language families and disadvantaging isolated languages and scripts.
Abstract
from arXiv · showhide
Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models. Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.
1 Introduction
Multilingual language models can provide unequal access across languages, with disparities emerging before model inference at tokenization. These differences affect cost, latency, and usable context, motivating multilingual tokenization parity.
- Language models often become multilingual through training on internet content across languages, even without explicit multilingual design.
- Performance is generally lower in non-target languages, especially low-resource languages, while unequal access to safety tools has caused societal consequences.
- 15 times more tokens can be required for Shan than English, while ChatGPT and GPT-4 use about 1.6 times more for Italian, 2.6 times for Bulgarian, and 3 times for Arabic.
- Over 4 times longer byte-level encodings occur for Burmese or Tibetan than Chinese for the same text.
- At least 2.5 times higher costs, twice the processing time, and more than an order-of-magnitude context differences can result across languages.
- The paper advocates multilingual tokenization parity, requiring similar encoded lengths for the same content across languages.
2 Intriguing Properties of Tokenization Across Languages
Tokenization vocabularies can favor obscure corpus-derived terms while fragmenting common words and non-Latin writing. These patterns reflect corpus biases and produce unequal token counts across languages and scripts.
- Corpus artefacts can give technical or rare words dedicated tokens while common words and non-Latin characters require multiple tokens.
- GPT-2 assigns dedicated tokens to obscure strings such as BuyableInstoreAndOnline and rawdownloadcloneembedreportprint.
- The Arabic word “لماذا” is split into letters, while Bulgarian “защо” requires 6 tokens for its 4 letters.
- More than half of Japanese kanji characters require three tokens in GPT-2.
- Tokenizer vocabularies are heavily influenced by corpus-source biases, so balanced tokens across languages are needed for equal treatment.
3 Measuring Tokenizer Parity
The paper introduces tokenizer parity to measure whether equivalent translated sentences receive similar tokenized lengths. It defines parity through a length ratio near 1 and calls that ratio a language-specific premium.
- Tokenizer parity assesses whether equivalent sentences in different languages receive similar tokenized lengths.
- For translations sA and sB, parity holds when |t(sA)|/|t(sB)| ≈1, where t(s) is a tokenization and |t(s)| its length.
- The ratio |t(sA)|/|t(sB)| is defined as the premium for language A relative to language B.
4 Tokenization Length Differences Across Languages
Across English-centric, language-targeted, multilingual, and byte-level tokenizers, equivalent content receives substantially different tokenization lengths, with no approach achieving parity across all languages. The disparities reflect vocabulary, corpus, script, character-length, and UTF-8 encoding effects.
- Evaluation: FLORES-200 evaluation compares English-centric, language-targeted, multilingual, and byte-level tokenizers using premiums relative to English or each model’s target language.The corpus contains 2,000 Wikipedia sentences human-translated into 200 languages.
- English-centric models: 66% more tokens are required for Pangasinan than English with GPT-2 and RoBERTa, while ChatGPT and GPT-4 still require about 50% more tokens for their cheapest languages.Shan has encodings up to 15 times longer than English for the ChatGPT and GPT-4 tokenizer; FlanT5 exceeds 10% UNK tokens for 42% of languages.
- Other target languages: English remains closest to parity for GottBERT, CamemBERT, and PhoBERT, while ArabicBERT, RoCBert, and BERT Japanese benefit more from languages sharing their scripts.MuRIL, despite focusing on 16 Indian languages, is also most token-efficient for English.
- Other target languages: Language varieties can receive much longer encodings than their standard languages, including Swiss German dialects and Mauritian and Haitian Creoles.ArabicBERT and BERT Japanese are closer to parity across dialects, likely reflecting better dialect representation and linguistic proximity to the standard languages.
- Multilingual models: None of the intentionally multilingual models uniformly reaches parity, although BLOOM and NLLB encode all FLORES-200 languages with less than 10% UNK tokens.XLM-R is much closer to parity than RoBERTa, but all five multilingual models still include languages with premiums above 2.5.
- Byte-level models: Over 4 times separates some language pairs in byte-level models, because languages differ in character counts and UTF-8 uses different byte widths across scripts.CANINE’s Shan premium relative to Yue Chinese is 4.58, while ByT5 ranges from 0.87 for Yue Chinese to 3.94 for Shan.
5 Fairness Implications of Tokenization Length Differences
Tokenization disparities create unequal costs, latency, and long-context capacity across languages, affecting both text and speech systems. These disparities persist across subword, multilingual, and byte-level tokenizers.
- None of the evaluated subword, multilingual, or byte-level tokenizers approaches parity across all FLORES-200 languages.The paper identifies tokenization parity as a cross-tokenizer fairness problem rather than an issue confined to one tokenizer family.
- Cost: At least 50% higher processing costs affect German and Italian than English for ChatGPT and GPT-4, while some languages cost more than twice as much.
- Cost: More than 4 times as many Unicode characters can be required for Burmese, Dzongkha, Shan, Tok Pisin, or Tumbuka than for Yue Chinese.Per-character pricing therefore also produces proportional cost differences across languages.
- Latency: Processing time correlates strongly with tokenization length, with Shan requiring almost twice the execution time of English in the RoBERTa analysis.The measured relationship is linear in sequence length, although transformer attention itself has quadratic complexity and the full architecture includes other submodules.
- Latency: Tokenization disparities also affect speech recognition and synthesis because these systems produce or consume tokenized sequences.
- Long context processing: Longer tokenized inputs reduce the amount of content that fits within fixed model context windows and may impair reasoning and automated-system performance for high-premium languages.The paper specifically connects this constraint to content moderation, where reduced capabilities could have severe real-world impacts.
6 Towards Multilingual Tokenization Fairness
The paper argues for multilingual tokenization parity through fair subword tokenizers trained with Unicode-based representations and carefully balanced multilingual data. It proposes design strategies while noting that vocabulary allocation and tokenizer choice create trade-offs.
- Training language models from scratch with a multilingually fair subword tokenizer is presented as the only approach addressing cost, latency, and long-context unfairness together.
- Character-level and byte-level representations cannot achieve tokenization parity, making a subword-tokenization variation necessary.The proposed allocation can give Chinese characters individual tokens, average Latin tokens of about three characters, and combine Burmese characters with diacritics.
- Using a separate tokenizer for billing does not fix latency or long-context limitations because the model’s processing tokenizer remains unchanged.Therefore, comparable processing times and context capacity require retraining the language model with a fair tokenizer.
- A Unicode-based starting representation is needed to support all Unicode codepoints, and UTF-8 is suggested as a basis for fair subword vocabulary construction.
- A balanced parallel corpus must represent topics and named entities fairly, because otherwise parity may hold only for a subset of topics or favor English.The paper therefore emphasizes a well-curated and diverse corpus for development and evaluation.
- Subword tokenizers cannot simply be trained on balanced data because shared tokens favor languages with overlapping vocabulary, while isolated scripts such as Korean receive larger premiums.
- Merging monolingual tokenizers is proposed as a two-stage strategy: train one per target language, then repeatedly add tokens for the language with the highest premium.The process starts from the 256 byte-value tokens while maintaining parity.
- Allocating one-third of the vocabulary away from English would make English sequences only 10% longer for ChatGPT/GPT-4, reflecting diminishing vocabulary returns.Replacing rare dominant-language tokens with frequent tokens in other languages is argued to offer an overall net benefit.
7 Related Works
Related work frames multilingual fairness through model bias, multilingual performance, and tokenization-length measurement. These strands motivate examining language-specific disparities at the tokenizer level.
- Fairness and bias in language models: Prior studies document stereotypes, intersectional bias, social bias, toxic generation, and English-centric quality judgments in language models.The literature also argues that non-English languages may require distinct fairness analyses and solutions.
- Multilingual performance: Multilingual-model research includes treating languages as entities and training separate vocabularies for language clusters to balance cross-lingual and language-specific tokens.Despite these approaches, lower-resource languages continue to face weaker transfer and uneven performance.
- Measuring tokenization lengths: Existing tokenization-length metrics based on tokens per characters or tokens per word may be unsuitable for cross-language comparison because sentence and word lengths differ across languages.
8 Conclusion
The paper finds substantial cross-language tokenization disparities that produce unequal treatment in language technology access. It advocates multilingually fair tokenizers to improve comparable performance and accessibility.
- Up to 13-fold tokenization-length variation remains even among tokenizers explicitly trained for multilingual support, while character- and byte-level models exceed 4-fold differences for some language pairs.
- These disparities increase commercial-service costs, processing time, and limitations on the contextual information available to language models.
- The paper proposes multilingually fair tokenizers that achieve comparable tokenization lengths across supported languages.It presents tokenization parity as a way to mitigate language-based inequalities and promote fairer access to language technologies.
A Background on Tokenization
Tokenization converts language into finite-vocabulary token sequences that language models process as integers. Modern systems mainly use subword methods, with Unicode and byte-level representations extending coverage across scripts while affecting sequence length.
- Tokenization methods: Tokenization turns natural language into sequences of tokens from a finite vocabulary, which models process as integers rather than characters or words.
- Tokenization methods: Subword tokenization breaks complex or novel words into multiple parts, enabling open-vocabulary handling of unfamiliar and misspelled words.
- Subword tokenization: BPE learns subwords from large corpora by repeatedly merging frequently occurring pairs, making it widely used in popular tokenizers.
- Unicode support: Unicode assigns 1,114,112 codepoints, while UTF-8 uses one to four bytes per codepoint and UTF-32 uses four bytes for every codepoint.
- Unicode support: Byte-level tokenization commonly encodes UTF-8 bytes first and then applies BPE, allowing a 256-token vocabulary to represent all texts.
- Sequence length: Character-level and byte-level approaches can lengthen sequences substantially, challenging transformers whose attention layers have quadratic complexity in input length.
B Parity for Linguistic Varieties
Tokenization parity varies across dialects, regional varieties, and creoles rather than following only their relationship to a high-resource standard language. Arabic and Japanese dialects show relatively narrow premiums, whereas Swiss German dialects and French-related creoles can be substantially less aligned.
- Scope: The study examines Swiss German varieties, Arabic and Japanese dialects, and Haitian and Mauritian creoles for differences from corresponding standard languages.
- Swiss German dialects: Swiss German premiums relative to High German range from 1.38 for Zürich dialect to 1.59 for Bernese dialect, exceeding English’s 1.35 premium.
- Swiss German dialects: GottBERT achieves better parity for English, Swedish, and Norwegian Bokmål than for Swiss German dialects, suggesting unstable linguistic constructs across varieties.
- Arabic dialects: ArabicBERT premiums across 25 Arabic dialects range from 0.91 for Jeddah to 1.17 for Rabat, a narrower range than Swiss German dialects.The authors associate this parity with dialectical Arabic in the training corpus and vocabulary, while also noting possible linguistic relatedness.
- Japanese dialects: Japanese dialect premiums range from 1.01 for Saitama to 1.15 for Morokata-ben and Okayama-ben, below English’s 1.49 premium.
- Creoles: CamemBERT premiums are 1.20 for Mauritian Creole and 1.64 for Haitian Creole, showing poor alignment despite both languages’ French basis.Haitian Creole’s premiums are 1.58 in FLORES-200, exceeding those reported for English, Pangasinan, and Nigerian Fulfulde.
C Extended Tables of Tokenization Premiums
The extended tables include tokenizer-premium results for additional multilingual models beyond those presented in the main text.
- Additional models: Extended tables report tokenizer premiums for LLAMA, MBart50, SeamlessM4T, and Qwen-VL in addition to the main-text models.