Source-linked AI summary
Removable and Irreducible: A Token-Cost Ledger for the Multilingual Tokenization Tax
Madhulatha Mandarapu, Sandeep Kunkunuru
TL;DR
Non-English text can incur a large tokenization tax, but the paper asks how much reflects coding choices rather than intrinsic content. It builds a token-cost ledger on fixed parallel content and finds that matched coding removes about two-thirds of the production tax while intrinsic content differs by under 6% across Indic languages. The study is limited to compute-and-memory accounting and does not establish a cross-lingual orthographic direction or model-quality benefit.
Problem
Non-English text can cost substantially more tokens than equivalent English content, motivating the question of how much of this multilingual token tax is removable.
Method
The paper constructs a token-cost ledger on parallel content, separating removable coding redundancy, residual coding slack, intrinsic content, and an orthogonal grapheme-to-phoneme term.
Results
64%: on FLORES-200, a script-matched code trained on 1,012 sentences removes the median production-tokenizer tax, while intrinsic content varies by under 6% across Indic languages.
Takeaways & Limitations
The multilingual token tax is mostly removable in the compute ledger, while the grapheme-to-phoneme cost belongs to a separate multimodal axis.
Takeaways & Limitations
The matched code is a small-data demonstration and the study measures compute and memory, not model accuracy or loss; the cross-lingual orthographic direction is not established.
Abstract
from arXiv · showhide
Large language models pay a well-documented tax on non-English text: the same content costs several times more tokens, and because attention is quadratic in sequence length, far more compute. We ask how much of this tax is removable. Framing the token layer as source coding -- transformer compute is monotone in sequence length, whose per-atom floor is the Shannon rate $H/\log_2 V$, an object already applied to tokenizers in prior work -- we assemble a token-cost ledger that splits each language's cost, at fixed parallel content, into a removable coding redundancy, a residual coding slack, an intrinsic-content term, and an orthogonal, irreducible grapheme-to-phoneme term that governs the multimodal rather than the text cost. On FLORES-200 across eight languages, a production tokenizer costs up to $8.9\times$ more tokens for Indic scripts than for English; a script-matched code trained on $1,012$ sentences removes a median $64\%$ of that excess (bootstrap 95\% CI $[0.638, 0.647]$), and a script-fair information floor shows the intrinsic content differs by under $6\%$ -- the tax is representational, not informational. A constructed code removes $98\%$ of a controlled source's redundancy, and the token tax implies up to $79\times$ attention cost. We are explicit about scope and failure: this is compute-and-memory accounting, not a model-quality claim; we neither measure nor claim the cross-lingual direction of the orthographic term; and our matched code is a conservative small-data demonstration. We contribute the unifying ledger, the removable-versus-intrinsic attribution, and an open one-command harness.
1 Introduction
The paper asks how much of the multilingual token tax is removable through better coding versus intrinsic to the language, and answers with a token-cost ledger scoped to compute and memory. On parallel data, matched coding removes most of the excess while intrinsic content remains nearly constant across Indic languages.
- Motivation: 8.9×: a production tokenizer emits up to 8.9× as many tokens for Indic scripts as for English on identical translated content.Because attention scales quadratically with sequence length, token inflation also increases attention compute and reduces effective context.
- Ledger: The ledger separates removable coding redundancy, residual coding slack, intrinsic content, and an orthogonal grapheme-to-phoneme term.The first three terms govern text compute; the grapheme-to-phoneme term governs the multimodal speech ledger rather than text sequence length.
- Contributions: The paper unifies these components into one token-cost ledger and provides an open, one-command harness for reproducing its measurements.Its contribution is attribution of observed multilingual costs to removable versus intrinsic components while keeping the analysis scoped to compute and memory.
- Scope: The analysis is a cost accounting, not a model-quality claim, and reports FLOPs, KV-cache bytes, and context-window occupancy rather than downstream accuracy.The paper explicitly does not claim that fewer tokens improve models.
- Empirical attribution: 64%: a script-matched code trained on 1,012 sentences removes the median production-tokenizer tax, with bootstrap 95% CI [0.638, 0.647].The script-fair information floor finds intrinsic content varies by under 6% across Indic languages, so the measured tax is representational rather than informational.
- Constructed limit: 98%: a constructed code removes a controlled source’s redundancy, while the token tax implies up to 79× attention cost.The construction sits 0.036 bits above the entropy floor.
2 The token-cost ledger
The ledger treats tokenization as source coding, separating sequence-length costs into removable coding effects, intrinsic content, and an orthogonal grapheme-to-phoneme term.
- Cost monotonicity: Transformer compute and KV-cache memory are non-decreasing in sequence length, with attention becoming strictly increasing once N is sufficiently large relative to model width.In the quadratic regime, a sequence-length ratio r implies an attention-cost ratio r^2.
- The floor: The Shannon fertility floor is F⋆ = H/log2 V, while coding for a wrong distribution adds redundancy R = DKL(p ∥ q)/log2 V.Refitting the code to the true distribution drives the removable redundancy toward zero.
- The ledger: Holding parallel-corpus content fixed, the ledger decomposes normalized sequence length into coding redundancy, residual coding slack, intrinsic content, and an orthogonal grapheme-to-phoneme term.The first three terms affect text compute; the grapheme-to-phoneme term leaves text sequence length untouched and governs the speech ledger.
- The ledger: The removable fraction ρ measures the share of production-tokenizer tax removed by a language-matched code: ρ→1 indicates a code artifact, while ρ→0 indicates intrinsic cost.This is the ledger’s single headline estimand for how synthetic a language’s tax is.
- Vocabulary: The compute-optimal vocabulary discussion is inherited prior work rather than a contribution of this paper.The paper notes an interior vocabulary-size minimizer balancing shrinking sequence terms against a growing vocabulary term.
- Caveat: For the shallow-orthography German control, residual coding slack is marginally negative because matched-code sequence length is 1.13 versus a floor of 1.15.The paper attributes this to distinct normalized instruments agreeing within noise near the English baseline; all five Indic languages have positive terms.
3 Experimental setup
The experiments use professionally translated, sentence-aligned FLORES-200 data and compare byte, grapheme, production-tokenizer, matched-code, and script-fair information measurements with deterministic tooling and controls.
- Real parallel data: FLORES-200 provides 1,012 professionally translated, sentence-aligned devtest sentences across eight languages spanning Latin, Devanagari, Bengali, and Dravidian scripts.The parallel alignment holds semantic content fixed across languages.
- Encoders: The encoders include UTF-8 bytes, Unicode extended grapheme clusters, three production byte-level BPE tokenizers, and held-out per-language matched BPE codes.The matched BPE is trained on the FLORES dev split and evaluated on devtest.
- Information floor: The script-fair content estimate applies LZMA to grapheme-cluster-ID streams rather than UTF-8 bytes, avoiding a charge for script-specific byte assignments.A constructed Zipf source with known entropy provides the controlled-source apparatus.
- Apparatus and controls: The harness uses single-thread Python, deterministic seeds, tiktoken, and standalone tokenizers without GPU or model weights, and regenerates results with one command.A preregistered uniform-32 calibration control recovered 5.0000 bits before language results were read.
4 Results: the removable tax
On FLORES-200, production tokenizers impose a large Indic-script tax, while matched codes remove most of it and script-fair floors place intrinsic content near the English baseline.
- Production tax: 8.86× is the maximum cl100k_base production tax reported here, for Kannada; Telugu reaches 8.29× and Hindi 4.76×.The older GPT-2 tokenizer reaches 16.4–20.3× fertility per word for Dravidian scripts.
- Matched code: 2.89×, 2.95×, and 2.54× are the matched-code ratios for Telugu, Kannada, and Hindi after training on only 1,012 sentences.The learned matched-code vocabularies contain 2,000–3,000 entries.
- Intrinsic content: 1.02–1.06× is the script-fair information-floor range, placing intrinsic content within 6% across all Indic languages.The preregistered content-invariance control found no language above the 1.5× threshold.
- Removable fraction: ρ = 0.642, with 95% CI [0.638, 0.647], is the median removable fraction across the five Indic languages.The matched code removes about two-thirds of the production tax in this conservative small-data demonstration.
- Replication: Spearman ρs = 0.83 (p = 0.01) links UTF-8 bytes-per-character with production tax across the eight languages.This replicates the reported correlation from prior work; the paper’s addition is the decomposition.
5 Decomposition, a constructed code, and quadratic cost
The Indic token tax is dominated by removable coding redundancy rather than intrinsic content, while a matched code approaches the entropy floor and sharply reduces the quadratic compute penalty.
- Decomposition: The removable vocabulary-mismatch term dominates the Indic excess, while intrinsic content is at most 0.06 in every case.Residual coding slack is also removable in principle, so it is a lower bound on total removability.
- Constructed code: 98% of redundancy is removed on a controlled Zipf source by a matched Huffman code landing 0.036 bits above the entropy floor.The matched code pays 5.80 bits/concept versus the source entropy of 5.77 bits/concept.
- Constructed code: The no-sub-floor control rejects reporting codes below entropy, and finite-sample in-sample block coding is treated as biased rather than as a valid result.The residual sub-bit gap is closed by block or arithmetic coding as a theorem, not a measurement.
- Quadratic cost: Up to 79× attention cost for the same content under production tokenization collapses to 8–11× under the matched code.The proxy uses English = 1 and amplifies token inflation quadratically through Θ(N2) attention.
6 Related work
The paper builds on information-theoretic analyses of tokenizers, orthographic depth, and vocabulary selection, while positioning its contribution as a multilingual cost accounting rather than a new theorem.
- Information theory of tokenizers: Prior work supplies tokenizer efficiency measures and the additive entropy-floor-plus-redundancy decomposition used here.The paper explicitly distinguishes its multilingual accounting from those underlying theoretical results.
- Adjacent lines of work: Related research also addresses orthographic depth, compute-optimal vocabulary size, and the non-equivalence of compression and model quality.The paper uses orthographic depth for an irreducible axis but scopes its own claims to compute and memory.
7 Limitations and honest negatives
The paper reports two important negatives: it does not establish the cross-lingual direction of orthographic ambiguity, and its matched-code removability estimate is conservative.
- Honest negatives: The cross-lingual direction of grapheme-to-phoneme ambiguity is not established because only English has a pronunciation lexicon in the instrument.The reported English homograph entropy is 0.070 bits/type; Indic pronunciation data are unavailable.
- Method limitations: The matched code is trained on approximately 1k sentences with a 2–3k vocabulary, making ρ = 0.64 a lower bound on removability.The paper also states that the LZMA information floor is an upper bound on intrinsic content.
- Scope: The study measures compute and memory rather than model quality, and covers eight languages, one parallel benchmark, and text only.Speech and multimodal cost are argued rather than measured.
8 Conclusion
The conclusion attributes most of the multilingual tokenization tax to removable representation, separates orthographic cost onto a multimodal axis, and provides an open reproducibility path.
- Conclusion: A script-matched code trained on about a thousand sentences removes about two-thirds of the tax, while intrinsic content differs by under 6%.The conclusion frames the result as a compute-ledger finding about real parallel text.
- Conclusion: The grapheme-to-phoneme cost of a deep orthography remains on a different multimodal axis and can reverse the ranking.The paper does not claim this orthographic direction was established experimentally.
- Contributions: The paper contributes a unifying ledger, removable-versus-intrinsic attribution, a floor-reaching constructed code, and an open one-command harness.Code, data pointers, and preregistration support one-command reproduction.