PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

Clara Meister, Gül Sena Altıntaş, Antoine Bosselut

arXiv:2610.12144v1cs.CL

TL;DR

Existing evidence does not establish how tokenizer choice affects individual languages, despite finite shared vocabulary capacity that can create trade-offs. The paper trains 123 language models across 54 tokenizers under controlled conditions and finds that effects vary by language, with lower-resource languages generally more sensitive. Per-language intrinsic metrics can also screen tokenizer rankings before language-model training.

  • Problem

    Existing results are often aggregated too coarsely to establish how tokenizer choice affects each language, even though shared finite vocabulary capacity can make language outcomes move in opposite directions.

    Source: §1 · pp. 1–2

  • Method

    The authors train 123 language models across 54 tokenizers, holding architecture, training corpus, training-token budget, and optimization fixed in the main comparison.

    Source: §1 · p. 2

  • Results

    BPB spread across tokenizers rises as language-model training share falls (Spearman ρ = −0.52), and per-language intrinsic metrics rank held-out tokenizer pairs by BPB with 0.89 accuracy.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Tokenizer choice and the intrinsic properties associated with better BPB differ by language, while per-language intrinsic metrics can screen tokenizer candidates before model training.

    Source: §1 · p. 2

    The study uses one 1.27B-parameter decoder-only architecture and one training recipe, and its conclusions concern language-modeling performance rather than downstream task ability.

    Source: §8 · p. 11

Abstract

from arXiv · show

Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others. We therefore ask whether tokenizer choice matters equally across languages, a question that the current literature leave unanswered. To this end, we train 123 language models spanning 54 tokenizers. In the main comparison, architecture, training corpus, training-token budget, and optimization are held fixed, so the models differ only in their tokenizer. We find that tokenizer choice matters more for languages with less language-model training data: across the 54 tokenizers, the standard deviation of a language's bits-per-byte (BPB) increases as its model training-data share decreases (Spearman rho = -0.52 over the 31 trained languages and -0.69 over the 28 written with word boundaries). Leaving a language out of tokenizer training raises its BPB in every language we study, and the penalty tends to be larger for languages with less language-model training data. Giving lower-resource languages a larger share of tokenizer-training data, however, does not unconditionally help those languages: both equal weighting and an allocation inverting the shares with respect to the language model training data increase their BPB, particularly when language-model training repeats data. Finally, which intrinsic tokenizer properties are associated with better BPB differs across languages, providing further evidence that what makes a good tokenizer depends on the language. We find that the metrics quantifying these properties can be successfully used to predict downstream models' pairwise BPB rankings, suggesting a practical strategy for screening tokenizer candidates before training language models.

1 INTRODUCTION

The paper asks how tokenizer choice affects languages individually, given that vocabulary capacity is finite and multilingual improvements can trade off across languages. It studies 123 language models and 54 tokenizers under controlled training conditions, finding language-specific effects and opportunities for screening tokenizer choices.

1 INTRODUCTION

  • Because multilingual vocabulary capacity is shared and finite, aggregate outcomes can conceal languages whose performance changes in opposite directions.The paper therefore argues that tokenizer choice must be evaluated separately for each language.Source: §1 · pp. 1–2
  • The study trains 123 language models across 54 tokenizers, holding architecture, training corpus, training-token budget, and optimization fixed to isolate tokenizer effects.The tokenizer panel varies pretokenization, tokenization algorithm, vocabulary encoding, and tokenizer-training data.Source: §1 · p. 2
  • Tokenizer choice affects every language, but its impact is greatest for languages with less language-model training data; per-language intrinsic metrics predict held-out tokenizer-pair BPB rankings with 0.89 accuracy.The study also finds that equal weighting or an allocation favoring lower-resource languages can raise their BPB.Source: §1 · p. 2
  • Earlier studies establish that tokenizer choice affects model behavior, but cover at most five languages and do not isolate a broad range of tokenizer design choices.Prior per-language evidence also has not established whether intrinsic tokenizer metrics are useful for ranking tokenizers separately by language.Source: §1 · p. 2

2 EXPERIMENTAL SETUP

The study compares a panel of language models that differ only in tokenizer, evaluating language-modeling performance and selected downstream tasks across languages. Its setup varies tokenizer design, defines per-language and aggregate outcomes, and accounts for training noise and tokenizer-family dependence.

2 EXPERIMENTAL SETUP

  • The experimental setup trains a language model for each tokenizer and evaluates outcomes intrinsically and on downstream tasks across languages.Source: §2 · p. 3

2.1 TOKENIZER AND MODEL TRAINING

  • The 54-tokenizer panel spans algorithms, pretokenization and normalization, text encoding, and tokenizer-training data; models use a fixed architecture, corpus, and optimization recipe.Most tokenizers are trained on mixtures of the same multilingual corpus; two are off-the-shelf tokenizers.Source: §2.1 · p. 3

2.2 EVALUATION METRICS

  • Per-language FLORES+ bits-per-byte (BPB) is the primary outcome, supplemented by bits per sentence for cross-language comparisons, downstream benchmarks, aggregate means, and nine intrinsic tokenizer metrics.The 31-language FLORES+ mean weights languages equally, whereas Validation BPB reflects the training distribution's unequal language proportions.Source: §2.2 · pp. 3–4

2.3 STATISTICAL PROTOCOL

  • The analysis estimates seed variation from five tokenizers and groups near-duplicate tokenizers into families, using family-clustered standard errors to address dependence.P-values are corrected within relevant test sets using the Benjamini–Hochberg procedure.Source: §2.3 · p. 4

3 HOW MUCH DOES TOKENIZER CHOICE MATTER ACROSS LANGUAGES? (RQ1)

Tokenizer effects exceed seed variation for every language on FLORES+ BPB, but tokenizer rankings only partially agree across languages. Languages differ in sensitivity, and cross-language comparisons of sensitivity depend on the outcome unit.

3.2 RESULTS

  • ρ = −0.52 links BPB spread negatively to log LM share, while tokenizer changes exceed seed variation for every language on FLORES+ BPB.The figure plots BPB spread and exclusion penalty against log LM share; the correlation for exclusion penalty is ρ = −0.33, p = 0.068.Source: §3 · p. 4, Figure 1 · p. 5

3.1 METHOD

  • The method compares each language's outcome spread across 54 tokenizers with seed variation and measures cross-language ranking agreement using Kendall τ.Source: §3.1 · p. 5

3.2 RESULTS

  • Language identity accounts for 99.4% of per-(tokenizer, language) BPB variance, and Chinese has the largest tokenizer spread at 0.062.The tokenizer effect is visible in the within-language remainder of the variance.Source: §3.2 · p. 5
  • Tokenizer spread exceeds seed variation for all 31 languages on FLORES+ BPB and for 21 of 24 languages on MultiBLiMP accuracy.For 16 of the 24 MultiBLiMP languages, seed variation is below uncertainty from the benchmark's finite number of minimal pairs.Source: §3.2 · p. 5
  • Tokenizer rankings agree only partially across languages: mean pairwise Kendall τ is 0.418, and 16 of 54 tokenizers rank first for at least one language.Ukrainian has the lowest ranking agreement, at 0.099.Source: §3.2 · p. 5
  • Cross-language comparisons of spread depend on the outcome unit, and seed variation is estimated from replicates of only five tokenizers.Source: §3.2 · p. 5

4 WHICH LANGUAGES ARE MOST TOKENIZER-SENSITIVE? (RQ2)

Languages with smaller language-model data shares are generally more sensitive to tokenizer choice, though resource level and writing system are confounded in this 31-language set. Excluding a language from tokenizer training raises its BPB for every language, and broader tokenizer coverage can improve outcomes for nearly all languages.

4.1 METHOD

  • Sensitivity is measured by each language’s BPB spread across tokenizers and its mean BPB penalty when excluded from tokenizer-training data.The study compares both measures with log language-model share using Spearman correlation and tests writing-system and bytes-per-character confounds.Source: §4.1 · p. 6

4.2 RESULTS

  • BPB spread across 54 tokenizers falls as language-model share rises (Spearman ρ = −0.52), with Chinese an exception under this measurement strategy.The correlation remains negative and significant under several tested variants; the ranking is sensitive to measurement strategy.Source: §4.2 · p. 6
  • Excluding a language from tokenizer-training data raises its BPB for all 31 languages; the penalty–share correlation is negative but not significant without controls (Spearman ρ = −0.33).The correlation is significant when restricting to languages excluded by at least two coverage tokenizers or adjusting share for bytes per character.Source: §4.2 · pp. 6–7
  • Moving from a tokenizer trained on six high-resource languages to one trained on all 31 lowers BPB for 30 languages, while English and validation shifts remain below seed variation.The tokenizer ranking is not uniform, but the 31-language-mean winner ranks no lower than 11th of 54 among the ten smallest-share languages.Source: §4.2 · p. 6

5 DO DESIGN CHOICES CHANGE PER-LANGUAGE OUTCOMES? (RQ3)

Controlled design changes show that tokenizer training-data coverage and allocation can affect language-level outcomes. In particular, increasing lower-resource languages’ tokenizer shares does not reliably lower their BPB, and the inverted-allocation association depends on repeated language-model training data.

5.2 RESULTS

  • Under both equal weighting and inverted allocation, giving lower-resource languages larger tokenizer shares raised their mean BPB rather than lowering it.The inverted-allocation association was not significant for either permuted allocation (correlations +0.08 and −0.28); the authors suggest added entries may be hard to fit with limited LM data.Source: §5 · p. 7, §5.2 · pp. 7–8

5.1 METHOD

  • The study compares 25 single-choice configuration changes and reallocations of tokenizer-training shares, with selected comparisons matched on bytes or token counts.It also repeats the reallocation comparison without data repetition.Source: §5.1 · p. 7

5.2 RESULTS

  • Restricting tokenizer-training data to English produced the largest of 25 controlled configuration differences: +0.0386 in the 31-language FLORES+ mean.Eighteen of 25 design choices differed beyond run-to-run variation on that mean.Source: §5.2 · p. 7
  • Equal weighting raised mean BPB across the 15 languages with the smallest proportional tokenizer-training weights in all five tested pairs.The 31-language mean also rose in all five, with four increases larger than run-to-run variation.Source: §5.2 · p. 7
  • The inverted-allocation tokenizer raised the 31-language mean BPB by up to +0.0256, with larger rises among smaller-share languages (Spearman correlation −0.59).Tamil, which received the largest tokenizer share, had the largest rise (+0.267); Figure 2 plots per-language change against log language-model share.Source: §5.2 · p. 8, Figure 2 · p. 8
  • Sampling tokenizer-training data from the full FineWeb2 language pool raised mean BPB for the 183 languages outside the LM mixture in all four pairs (+0.20 to +0.42).For the 31 trained languages, the full-pool mean was higher in three of four matched pairs.Source: §5.2 · p. 8
  • With single-pass LM training, the inverted-allocation rise for Tamil was +0.14 rather than +0.267, and the BPB-change/share correlation was positive (+0.34) and not significant.The association between language-model share and BPB rise under inverted allocation therefore appeared only when training repeated data.Source: §5.2 · p. 8
  • At a fixed token budget, tokenizer changes also change training bytes; the inverted-allocation share–rise correlation appeared only in the repeated-data setup.Retraining pairs matched on training bytes reverses one of the 25 configuration comparisons.Source: §5.2 · p. 8

6 WHICH TOKENIZER PROPERTIES MATTER FOR WHICH LANGUAGE? (RQ4)

Intrinsic tokenizer properties relate to BPB differently across languages, so one metric does not provide a uniform account of tokenizer quality. Per-language metric measurements can nevertheless rank tokenizer pairs accurately enough to screen candidates before language-model training.

6.2 RESULTS

  • No intrinsic metric trends with BPB the same way for every language; significant correlations vary substantially across languages.Rényi efficiency correlates with Greek BPB at −0.72 and with Tamil BPB at +0.17, near zero.Source: §6 · p. 9, §6.2 · p. 10

6.1 METHOD

  • The analysis uses per-language metric and outcome values, mixed-effects models with language-specific intercepts, and pairwise rankers evaluated with tokenizer-family holdout.The random intercept accounts for language-specific BPB baselines; ranker accuracy is the fraction of held-out pairs ordered consistently with model BPB.Source: §6.1 · p. 9

6.2 RESULTS

  • Per-language intrinsic metrics support held-out pairwise ranking at 0.907 accuracy for validation BPB and 0.890 for 31-language FLORES+ mean BPB.Using one value per tokenizer instead yielded 0.700 and 0.735 accuracy; individual-language rankers ranged from 0.738 for Czech to 0.878 for Tamil.Source: §6.2 · p. 9
  • Within-language FLORES+ BPB is significantly related to eight of nine metrics: higher entropy, Rényi efficiency, or compression rate accompanies lower BPB, while higher fertility or split and crossing rates accompanies higher BPB.Vocabulary utilization is not significant in this analysis; the eight significant MultiBLiMP relationships have reversed signs.Source: §6.2 · p. 10, Figure 3 · p. 10
  • After controlling for training bytes, mixed-effects coefficients shrink and about three-fifths of significant per-language metric–BPB correlations remain significant.The authors also note possible artifacts in what intrinsic metrics measure, with SuperBPE tokenizers as a measured case.Source: §6.2 · p. 10

7 RECOMMENDATIONS

The authors recommend evaluating tokenizers per language, treating capacity reallocation as an intervention to validate, screening with per-language intrinsic metrics, and computing evaluation metrics against original text bytes.

7 RECOMMENDATIONS

  • Per-language evaluation should cover each intended language, or at least resource tiers and writing systems, because tokenizer rankings only partly agree across languages.Aggregate or English-only evaluations give little weight to the languages with the least language-model training data, which are most sensitive to tokenizer choice.Source: §7 · p. 11
  • Every tested reallocation raised BPB for the lower-resource languages it targeted and increased mean BPB across the 31-language FLORES+ set.Because this is one training setting, the authors recommend proportional tokenizer-training weights as the default, while validating targeted allocations rather than presuming they help.Source: §7 · p. 11
  • Per-language intrinsic metrics can screen candidate tokenizers before model training, but should be used as screening statistics rather than training objectives..Source: §7 · p. 11
  • Compute BPB against the original text's byte count, follow the training formatting protocol, and report bits per sentence when parallel data are available.Decoded-output normalization can change byte counts; parallel sentences hold content fixed across languages.Source: §7 · p. 11

8 LIMITATIONS

The study's conclusions are bounded by its model and training setup, its emphasis on language-modeling performance, and its 31-language training mixture.

8 LIMITATIONS

  • The main results use one 1.27B decoder-only architecture, one training recipe, a vocabulary near 128K, and mostly one run per tokenizer.The authors note that aggregate rankings change little without repetition or at twice the token budget, while some per-language findings interact with repetition.Source: §8 · p. 11
  • The conclusions concern language-modeling performance, since XCOPA and Belebele accuracies remain near chance at this model scale.MultiBLiMP and XNLI distinguish the models among the evaluated task benchmarks.Source: §8 · p. 11
  • The 54-tokenizer panel is observational, so metric correlations cannot be fully separated from ablated design choices, and imperfect family partitions may shrink standard errors.The authors warn this could lead to results being identified as significant when they are not.Source: §8 · p. 11
  • Claims about which languages are most sensitive apply only to the 31 languages in the training mixture..Source: §8 · p. 11

REPRODUCIBILITY STATEMENT

The paper documents its experimental specifications and evaluation procedures, reports public evaluation data, and commits to releasing research artifacts after de-anonymization.

REPRODUCIBILITY STATEMENT

  • The appendices specify tokenizer configurations, architecture and optimizer settings, training-mixture shares, BPB computation, seed-variation estimates, and ranking procedures.The ranking protocol is deterministic and uses no random seeds.Source: REPRODUCIBILITY STATEMENT · p. 11
  • All evaluation datasets are public, and the authors say they will release tokenizers, model checkpoints, result artifacts, and code upon de-anonymization..Source: REPRODUCIBILITY STATEMENT · p. 11
  • Reported numbers are bound in the paper source to named keys in programmatically computed result artifacts.The authors report checking these bindings against the underlying artifacts.Source: REPRODUCIBILITY STATEMENT · p. 11

THE USE OF LARGE LANGUAGE MODELS

The appendix describes the use of language-model-based tools, tokenizer configurations and pretokenization patterns, and model architecture and training details.

THE USE OF LARGE LANGUAGE MODELS

  • Language-model-based tools assisted with writing and experiment management; the authors attribute the research questions, design, analyses, and conclusions to themselves.They report checking the bindings between reported numbers and programmatically computed artifacts for errors.Source: THE USE OF LARGE LANGUAGE MODELS · pp. 11–12

A.1 TOKENIZER CONFIGURATIONS

  • Tokenizer families group configurations differing only in minor hyperparameters, while larger changes in algorithm, regex, normalization, encoding, or training data define separate families.The appendix also describes vocabulary configurations, 10 GB tokenizer-training data, retained implementation defects, and coverage and reallocation variants.Source: §A.1 · p. 17, Table 1 · p. 18

A.2 PRETOKENIZATION REGEXES

  • A pretokenizer splits text into chunks that subword algorithms cannot merge across; the Whitespace pattern splits only on whitespace, unlike the Unicode-aware GPT-4o pattern.The appendix also details punctuation, GPT-2, Claude, Clean, and RightAlign variants and their segmentation differences.Source: §A.2 · pp. 17, 19

A.3 MODEL ARCHITECTURE AND TRAINING DETAILS

  • All main models use the 24-layer, 1.27B-parameter nanochat d24 architecture, with a 128K vocabulary and shared training setup.The appendix gives the architecture and per-parameter-group optimizer settings.Source: §A.3 · p. 19, Table 2 · p. 20
  • The main runs use about 9.2B tokens and 2.0–2.9 loader passes over the training mixture, while document cropping means trained text totals about 1.1–2.0 times the mixture.The appendix defines a pass as one data-loader read of the mixture.Source: §A.3 · pp. 19–20
  • Smaller d8, d12, and d16 models share the d24 data, tokenizers, and optimizer recipe, but only their induced tokenizer rankings are compared with d24.Their smaller size and token budget make their absolute BPB incomparable to d24.Source: §A.3 · p. 20

A.4 MODEL TRAINING DATA

The model-training mixture fixes language shares independently of tokenizer choice, but tokenization affects how many drawn bytes are retained. The appendix also specifies BPB and seed-variation calculations and notes greater run-to-run noise among languages outside the training mixture.

A.4 MODEL TRAINING DATA

  • 25 GB of training text combines 36.9% English, 33.4% multilingual web text, 16.1% math, and 13.6% code.The multilingual portion is sampled by language in proportion to its presence in FineWeb2; realized language byte shares define the resource variable.Source: §A.4 · p. 20
  • 30.4%–43.5% of drawn text is discarded by the tokenizer-dependent packer, with a 35.5% median across main census runs.This redistributes language shares in the bytes actually used for training; across five balanced-data tokenizers, the training-to-drawn-byte share ratio varies by up to +38% for Chinese and −23% for Hungarian, and is −41% for Tamil under the English-only tokenizer.Source: §A.4 · pp. 20–21

A.5 EVALUATION AND ANALYSIS DETAILS

  • BPB is computed as model loss over the tokenized text divided by its UTF-8 byte count; FLORES+ devtest sentences are untruncated across tokenizers.Validation documents may be cropped, and three script-class pretokenized tokenizers are excluded from Validation BPB because they lack exact per-token byte tables.Source: §A.5 · p. 21
  • Seed variation pools within-tokenizer standard deviations across five tokenizers, with seed replicates differing only in weight initialization.The estimates are 0.0023 for a single language’s FLORES+ BPB, 0.0012 for the 31-language mean, and 0.00041 for validation BPB.Source: §A.5 · pp. 21–22
  • 0.0146 is the per-language seed variation across 214 FLORES+ languages, about six times the 0.0023 across the 31 trained languages.The additional 183 languages are mostly unseen during training, though misidentified or code-switched text may occur in the training corpora.Source: §A.5 · p. 22
  • Standard errors and p-values cluster by tokenizer family because the 54 tokenizers belong to 42 potentially dependent families.Clustering increases standard errors by factors of 1.0 to 3.2 compared with model-based estimates, and clustered p-values are used throughout.Source: §A.5 · p. 22

B ROBUSTNESS OF DESIGN CHOICES

The robustness checks find that the lower-resource sensitivity pattern survives alternative spread and normalization choices, while several comparisons expose limits on interpreting it. Controlled configuration changes and tokenizer-data reallocations also show that performance effects depend on the choice and training setup.

B ROBUSTNESS OF DESIGN CHOICES

  • −0.54 is the correlation between log language-model training share and BPB spread across 42 tokenizer families, compared with −0.52 across 54 tokenizers.The share relationship remains −0.54 for bits per sentence and −0.38 for BPB coefficient of variation; over 28 languages written with word boundaries, the corresponding correlations are −0.69, −0.69, and −0.51.Source: B ROBUSTNESS OF DESIGN CHOICES · pp. 22–23
  • The resource–spread relationship is descriptive rather than causal, and its estimates vary with script and word-boundary controls.The non-Latin indicator alone explains R2 = 0.17 versus R2 = 0.08 for log training share, but adding a no-word-boundary indicator changes which predictor is significant; the limited language pool does not cleanly isolate writing system.Source: B ROBUSTNESS OF DESIGN CHOICES · p. 23
  • 0.82 is the median ratio of training bytes under excluding versus including tokenizers; lower ratios accompany larger exclusion penalties (Spearman ρ = −0.47).Thus the exclusion penalty cannot be attributed cleanly to tokenizer coverage: excluding tokenizers generally expose models to fewer bytes of the language.Source: B ROBUSTNESS OF DESIGN CHOICES · pp. 23–24
  • +0.93 is the Spearman correlation between 31-language FLORES+ means under single-pass and repeated-data training, and +0.94 before versus after doubling the token budget.Per-language rank agreement can be lower, including −0.11 for Chinese between single-pass and repeated data.Source: B ROBUSTNESS OF DESIGN CHOICES · p. 24

C CONTROLLED COMPARISONS

  • The byte-alphabet defect lowers measured BPB: rebuilding the English-data GPT-4o tokenizer raises the 31-language mean by 0.009.The rebuild also raises per-language FLORES+ BPB for 30 of 31 languages; the defect omits 32 of 128 tested byte values.Source: C CONTROLLED COMPARISONS · pp. 25–26

D.2 REALLOCATION EFFECTS

  • Equal weighting raises both the 31-language mean and the mean for the 15 lowest-proportional-weight languages in all five pairs.In four pairs, the rise in the 31-language mean exceeds run-to-run variation.Source: Table 6 · p. 26, §D.2 · p. 27
  • Only inverted allocation has a significant correlation between per-language BPB change and log language-model training share; its difference from permutation 1 is −0.67 and significant.Its difference from permutation 2 is −0.30 and not significant.Source: §D.2 · p. 27, Table 8 · p. 27

D.3 TESTING FOR FURTHER CONFOUNDERS

The confounder checks show that inverted allocation’s effects depend partly on repeated-data training, while token-matched comparisons still leave a substantial Tamil BPB increase. The section also compares coverage effects for languages outside model training and describes intrinsic tokenizer metrics used to relate tokenization to BPB.

D.3 TESTING FOR FURTHER CONFOUNDERS

  • At matched Tamil training-token counts, inverted allocation remains +0.233 BPB above the reference tokenizer on the reweighted mixture.This is close to the +0.267 difference at matched steps on the original mixture, suggesting fewer Tamil training tokens are not the main cause of the higher BPB with repeated data.Source: §D.3 · p. 28, Table 9 · p. 28
  • Tamil’s inverted-allocation BPB change is +0.1386 on single-pass data versus +0.2674 on repeated data, meeting the preregistered repetition-effect criterion.The correlation between BPB change and log training share is −0.59 on repeated data (significant) and +0.34 on single-pass data (not significant); repetition-related change correlates with log share at −0.89.Source: §D.3 · p. 28, Table 10 · p. 29
  • Rare Tamil vocabulary entries account for at most 13.6% of the +0.277 rise; losses instead rise across matched vocabulary entries under inverted allocation.The paper hypothesizes embedding overfitting but says the experiments do not isolate that mechanism; about half of Tamil’s rise appears only with repeated data.Source: §D.3 · p. 29, §D.4 · p. 29

D.4 EXCLUSION AND REALLOCATION COMPARED

  • +0.20 to +0.42 is the higher mean FLORES+ BPB for full-pool tokenizers over their counterparts on 183 languages outside model training.Across the four pairs, at most 56 of the 183 languages have lower BPB with the full-pool tokenizer; the authors conclude coverage lowers BPB only when the language model trains on that language.Source: §D.4 · pp. 29–30

E.1 INTRINSIC METRIC DEFINITIONS

  • For Tamil, three of four metrics significantly correlated with BPB have signs opposite to the average within-language model coefficients.The metrics include compression rate, unigram entropy, and fertility, illustrating that a single language can benefit from different tokenizer properties than the average relationship suggests.Source: §E.1 · p. 30

E.2 PER-LANGUAGE RESULTS ON IN-DISTRIBUTION FINEWEB2 TEXT AND THE FULL FLORES+ LANGUAGE SET

Per-language tokenizer metrics predict held-out tokenizer-pair BPB rankings, and their relationships with performance persist across evaluation settings. The section also shows that training-byte adjustment and resource level qualify some metric–BPB relationships.

E.2 PER-LANGUAGE RESULTS ON IN-DISTRIBUTION FINEWEB2 TEXT AND THE FULL FLORES+ LANGUAGE SET

  • Eight of nine intrinsic metrics have the same coefficient sign on FineWeb2 as on FLORES+, seven with larger magnitudes; all nine are significant on FineWeb2 versus eight on FLORES+.Vocabulary utilization is not significant for FLORES+ BPB but is significant on FineWeb2, with coefficient −0.0076.Source: §E.2 · p. 31, Table 12 · p. 32
  • 211 of 279 metric-by-language correlations are significant on FineWeb2, versus 104 on FLORES+, while tokenizer-ranking disagreement persists across both texts.Mean off-diagonal Kendall τ is 0.455 on FineWeb2 versus 0.418 on FLORES+; Figure 6 reports the FineWeb2 per-language correlations.Source: §E.2 · pp. 31–32, Figure 6 · p. 32
  • For most metrics, the aggregate correlation’s sign flips when averaging over all 214 FLORES+ languages rather than the 31 trained languages, largely tracking the 183 untrained languages.Untrained-language correlations use global tokenizer metrics, not measurements on those languages’ text, and have greater seed variation.Source: §E.2 · pp. 32–33, Table 13 · p. 33

E.3 CONTROLLING FOR CONFOUNDERS

  • After adjustment for training bytes, segmentation metrics remain significant under all four estimates, while compression rate is significant only under the smallest estimate.The adjustment leaves coefficient signs unchanged; Table 14 presents the adjusted coefficients for 51 tokenizers with measured training bytes.Source: Table 14 · p. 33, §E.3 · p. 34
  • About three-fifths of identified significant per-language correlations remain significant after controlling for training bytes, and about one in ten change sign.Thirteen of 276 entries are not identified because the metric and training-byte count are too highly correlated; Figure 5 marks those entries.Source: Figure 5 · p. 31, §E.3 · p. 34
  • Rényi efficiency’s negative association with BPB is significant for low-resource languages but near zero for the highest-share languages; character-split-rate associations are larger for higher-share languages.The interactions with training-data share are significant for Rényi efficiency (+0.0061) and character-split rate (+0.0165), but not for the other seven metrics.Source: Table 15 · p. 34, §E.3 · p. 35

E.4 TOKENIZER RANKING

  • 0.829 mean pairwise accuracy is achieved when metrics are measured per language, versus 0.713 with one value per tokenizer, for predicting each language’s FLORES+ BPB rankings.Per-language accuracies range from 0.738 (Czech) to 0.878 (Tamil); the target language’s own metrics are excluded, and evaluation holds out tokenizer families.Source: §E.4 · p. 36

F TASK ACCURACIES AS THE OUTCOME: MULTIBLIMP, XNLI, XCOPA, TOKSUITE, AND XWINOGRAD

Intrinsic-metric relationships with task accuracy vary across benchmarks: they are more evident for MultiBLiMP and XNLI than for XCOPA, TokSuite, or XWinograd. Several benchmark estimates also have limited precision or remain near chance.

F TASK ACCURACIES AS THE OUTCOME: MULTIBLIMP, XNLI, XCOPA, TOKSUITE, AND XWINOGRAD

  • Eight of nine intrinsic metrics are significant for MultiBLiMP accuracy, while 67 of 216 metric-by-language correlations are significant after adjustment.For 16 of 24 languages, benchmark-item uncertainty is at least as large as observed run-to-run variation; Bengali has only 21 minimal pairs.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 36, Table 17 · p. 36
  • 16 of 117 XNLI metric-by-language correlations are significant after adjustment, and four metrics remain significant in the family-clustered within-language model.The surviving metrics are fertility, trigram entropy, UTF-8 boundary crossing, and UTF-8 character splitting; several languages remain close to chance across tokenizers.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 37, Table 18 · p. 37
  • Mean XCOPA accuracy correlates with validation BPB (ρ = −0.49) and the 31-language FLORES+ mean (ρ = −0.52), despite weak per-language metric relationships.Only 3 of 63 metric-by-language correlations are significant, and none of the metrics is significant in the family-clustered model; sampling uncertainty is at least as large as run-to-run variation.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 37
  • TokSuite shows no association between mean accuracy and either BPB outcome, and only 2 of 36 metric-by-language correlations are significant after adjustment.No metric is significant in the family-clustered within-language model.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 37
  • Mean XWinograd accuracy is not significantly associated with validation BPB or the 31-language FLORES+ mean.Across six languages, per-language accuracies span 0.434 to 0.680; only English, Russian, and Chinese average clearly above chance.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 38
  • Belebele accuracy across 27 evaluated runs ranges from 0.2259 to 0.2715 around chance level 0.25, so it is not used in the analyses.Source: F TASK ACCURACIES AS THE OUTCOME:… · p. 38
Loading 2610.12144v1…