Source-linked AI summary
The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype
Thomas Serval
TL;DR
Token counts for the same content vary across languages, affecting services billed per token and context windows measured in tokens. The paper measures premiums across current tokenizers and evaluates a French-oriented prototype. French uses 1.31 to 1.58 times the English token count on NTREX, while the prototype comparisons show a specialization trade-off.
Problem
Earlier token-premium studies predate 2025–2026 tokenizers, leaving current measurements for French and regional languages, and their comparison with Chinese, to be updated.
Source: §1 · pp. 1–2Method
The paper measures translated-to-English token-count ratios on parallel corpora across seven current tokenizers and evaluates a French-oriented prototype against existing tokenizers.
Source: §1 · p. 2, §3 · pp. 2–3Results
French uses 1.31 (Tekken) to 1.58 (Llama 3) times the English token count on NTREX, and Chinese is cheaper than French on six of seven tokenizers.
Source: §4.1 · p. 3Takeaways & Limitations
Baracoda FR v1.2 uses fewer tokens than Tekken on the reported final French and English test, but does not outperform CroissantLLM at comparable vocabulary size.
Source: §6 · p. 7
Abstract
from arXiv · showhide
LLM services are billed per token and context windows are measured in tokens, yet the number of tokens needed for the same content varies across languages. We measure this token premium on seven tokenizers of widely used 2026 models (OpenAI o200k, Llama 3, Qwen3, DeepSeek V3/V4, Gemma 3, Mistral Tekken, and the Claude generation-5 tokenizer via Anthropic's counting API) on NTREX-128 (124 non-English reference translations) and on the Universal Declaration of Human Rights for regional languages. French requires 31% to 58% more tokens than English, whereas Simplified Chinese ranges from 5% fewer to 40% more and is cheaper than French on six of the seven tokenizers. Regional and overseas languages of France pay roughly 1.6 to 3.3 times the English count. We discuss how history re-sending, tiered pricing and fixed context windows amplify the absolute gap in agentic use. In a controlled experiment (BPE, Europarl, 50k vocabulary), adding French to tokenizer training data quickly reduces the premium, with diminishing returns and a growing cost for English. Finally, we present Baracoda FR v1.2, a byte-level BPE prototype with Tekken's vocabulary size. On a final test of six corpora never consulted during design, with a protocol declared fixed beforehand, it uses 11.5% fewer tokens than Tekken on French and 3.7% fewer on English; results hold after removing test sentences overlapping the training data and with an equal ordinary-token budget. It is worse on other languages and, at comparable vocabulary size, does not outperform CroissantLLM. These are segmentation results only; effects on model quality and task cost remain to be shown.
1 Introduction
Token premiums matter because LLM billing and context windows are token-based, while English-dominated tokenizer training can make equivalent content require more tokens in other languages. The paper updates measurements for current tokenizers, examines economic consequences, and tests a French-oriented tokenizer, including where it does not help.
1 Introduction
- English-dominated tokenizer training can split the same content into more pieces in other languages, despite billing and context windows being token-based.Earlier studies reported premiums up to 15× for some languages and linked them to API cost disparities and lower model utility; both predate 2025–2026 tokenizers.Source: §1 · p. 1
- The paper updates token-premium measurements for 2026 tokenizers, focusing on French, France’s regional and overseas languages, and Chinese.It also examines long-context economic consequences and tests whether a French-designed tokenizer can reduce the premium, reporting where it does not.Source: §1 · p. 2
- Seven current tokenizers are evaluated on 124 non-English NTREX-128 translations, with regional-language premiums, a French-premium decomposition, and a controlled training-data experiment.Baracoda FR v1.2 is compared with Mistral Tekken and CroissantLLM, including a final test and a comparable-vocabulary-size comparison where it does not win.Source: §1 · p. 2
2 Related work
Prior work measured tokenization disparities, their economic and performance associations, and tokenizer designs intended to improve parity or adapt vocabularies. The paper positions its study as an update using 2026 tokenizers.
2 Related work
- Petrov et al. [1] found closely related European languages can pay about 50% more than English, while per-character pricing does not remove disparity; Ahia et al. [2] linked over-fragmentation with worse results.Later work expressed premiums as lost context capacity, reported per-character cost multipliers across providers, and found 1.3–1.8 premiums on nine languages [4] [5] [6]. Design work covers a bilingual 32k tokenizer, parity-aware BPE, cross-word token learning, and vocabulary transfer [8] [9] [10] [11] [12] [13].Source: §2 · p. 2
3 Measuring the token premium
The study measures token premiums as translated-corpus token counts relative to English-source counts, using parallel corpora and current tokenizers. It reports corpus-based uncertainty intervals and distinguishes those from generalization to other text types.
3 Measuring the token premium
- Seven tokenizers support the main measurement, with CroissantLLM reserved as a specialized French–English comparator.Local counts were cross-checked against official implementations; Claude generation-5 counts came from Anthropic’s counting endpoint, with a measured 7-token message overhead subtracted.Source: §3 · pp. 2–3
- The premium is the translated corpus’s total token count divided by the English source’s total token count for a tokenizer.NTREX-128 supplies 124 non-English reference translations after excluding a duplicate Spanish reference; regional-language comparisons use the UDHR, and additional Universal Dependencies sets support Section 6.Source: §3 · pp. 2–3
- NTREX 95% intervals bootstrap over 123 source documents, while Universal Dependencies paired comparisons resample blocks of 10 consecutive sentences.The intervals describe uncertainty within these corpora, not generalization to other text types; texts are NFC-normalized and counted without special tokens.Source: §3 · p. 3
4 Results
French requires more tokens than English and is more token-expensive than Chinese on most tested tokenizers; regional-language counts are also elevated. The French premium reflects both translated-text length and tokenization per character, an accounting decomposition rather than a causal explanation.
4.2 Regional and overseas languages of France
- On the UDHR, Breton, Corsican, Wolof, and Bambara require about twice the English count, while Picard, Walloon, and Tahitian require 2.5 to 3.5 times.Relative to French, the extra cost ranges from +13% for Catalan and +20% for Occitan to +45% for Breton, +90% for Walloon, and +130% for Tahitian; the values are orders of magnitude because sentence-level correspondence is unverified.Source: §4.2 · p. 3
4.1 French pays more than Chinese
- French uses 1.31 (Tekken) to 1.58 (Llama 3) times the English token count, while Chinese ranges from 0.95 (DeepSeek) to 1.40 (Tekken) and is cheaper than French on six of seven tokenizers.The median of 124 non-English references pays 1.69–2.15 times the English count; Claude’s finer segmentation means cross-provider prices should be compared per document, not per token.Source: §4.1 · p. 3, Table 1 · p. 4
4.3 Decomposition
- 1.18 is the translated-text character-length ratio on NTREX, while the per-character token ratio ranges from 1.11 (Tekken) to 1.33 (Llama 3) and reaches 1.23 for Claude 5.This is an accounting decomposition, not a causal explanation; the character-length ratio is an observation on translated news, not a constant of the language.Source: Table 2 · p. 4, §4.3 · p. 4
5 Economic implications
In long-context use, token premiums widen absolute costs through repeated history, earlier tier crossings, and reduced context capacity. Dense-attention operations also rise quadratically with sequence length, but the paper does not infer total latency or energy effects.
5 Economic implications
- With Gemma 3’s 1.41 premium, French crosses the $2-to-$4-per-million-token threshold at turn 29 versus 41 for English, making each intervening French turn 2.8 times costlier.In the simulation, the cumulative surcharge rises from 41% to 112% before receding to 51% at turn 80.Source: §5 · p. 4
- When full history is re-sent at each turn without prompt caching, cumulative cost and the absolute language-cost gap grow quadratically with turns.The token and price ratios per turn remain constant except when a tariff threshold is crossed.Source: §5 · p. 4
- Dense-attention operations grow quadratically with sequence length, reaching 1.85–2.5 times for premiums of 1.36–1.58; total latency and energy effects are not inferred.Other layers, caches, and local-attention architectures can mitigate this component of compute.Source: §5 · p. 5
6 A French-optimized prototype
Baracoda FR explores whether a French-oriented tokenizer can reduce token counts, while comparing its results with Mistral Tekken and CroissantLLM. A declared-fixed-before-test protocol and controls assess its final-test performance and robustness.
6 A French-optimized prototype
- On NTREX, v1.2 used fewer tokens than every tested tokenizer in French and English, but its French–English premium was 1.20 versus v1’s 1.18.English improved faster than French, so the lower token counts did not produce better parity.Source: §6 · p. 6
- In the controlled 50,000-token Europarl experiment, increasing French’s share from 0% to 25% reduced the PUD premium from 1.81–1.83 to 1.17.Beyond 25%, French token gains were marginal (−2.3% from 25% to 75%) while English tokens increased by 3.8%; the lower 0.74–0.75 premium came at English’s expense.Source: §6 · p. 6
- The controlled mix experiment remains to be replicated on web data and at other data sizes, and does not explain commercial tokenizers’ results.Commercial tokenizers cover many languages and code in one vocabulary, and their training data are not public.Source: §6 · p. 7
- At comparable vocabulary sizes, the recipe did not outperform CroissantLLM, which was trained on a larger, more varied corpus.The 131,072-token v1.2 used 11.1% fewer French and 12.4% fewer English tokens on UD PUD, an advantage attributed mainly to vocabulary size.Source: §6 · p. 7
6.1 Final test with a protocol fixed in advance
- All three declared hypotheses held: v1.2 used fewer tokens on each French corpus, at least 8% fewer on pooled French, and at most 2% more on pooled English than Tekken.The protocol fixed the tokenizer and six Universal Dependencies corpora before computation; its SHA-256 establishes file identity, not the protocol’s date.Source: §6.1 · p. 7
- After removing 503 overlapping final-test sentence occurrences and retraining with Tekken’s 130,072 ordinary tokens, the reported gains remained essentially unchanged.The equal-budget retraining changed gains by 0.012 points in French and 0.026 points in English.Source: §6.1 · pp. 7–8
7 Limitations
The reported evidence is limited to token segmentation rather than invoices, model quality, or task success. The prototype also carries retraining costs and benchmark-scope limitations.
7 Limitations
- The study measures token counts, not invoices, model quality, or task success.Using Baracoda FR requires retraining at least model embeddings, with compute cost and quality risk still to be measured.Source: §7 · p. 8
- Only the final prototype test is held out; other prototype benchmarks are development data, and the final test covers French and English only.The Claude counts use a 7-token message-overhead estimate and an endpoint Anthropic describes as an estimate.Source: §7 · p. 8
- The UDHR comparison uses one document per language, without verified sentence-level correspondence, and tokenizer vocabularies range from 32k to 262k tokens.The UDHR versions match the English structure at 95% or more, except Wolof at 85%.Source: §7 · p. 8
8 Conclusion
In 2026, French still requires substantially more tokens than English, while the paper reports that French-focused tokenizer training narrows the gap on segmentation benchmarks. It identifies task-level cost evaluation as the next step and makes the study materials available.
8 Conclusion
- French requires 31% to 58% more tokens than English, and regional French languages pay about double; the prototype narrows the gap on segmentation benchmarks.The final test used a protocol fixed in advance.Source: §8 · pp. 8–9
- Figure 4 presents absolute NTREX-128 token counts, with French as bars and English as ticks.Source: Figure 4 · p. 9
- Table 6 reports final-test token counts with 95% paired block-bootstrap intervals using blocks of 10 sentences.Its caption states that the protocol was declared fixed before the test.Source: Table 6 · p. 9
- The next step is testing whether the tokenizer lowers the cost of correctly completed tasks, starting with banking and insurance French and a control branch.The proposed work includes established vocabulary-transfer methods and a publicly deposited protocol digest for the final test.Source: §8 · p. 9
- Tokenizer files, scripts, counts, controls, and library versions are released, while training data and evaluation corpora are not redistributed.Scripts download the corpora at pinned versions.Source: §8 · p. 9