PaperlayerView exact revision on arXiv

Source-linked AI summary

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel

arXiv:2512.20757v2cs.CLcs.LG

TL;DR

Tokenizer effects on language-model behavior are difficult to isolate from architecture and training differences. TokSuite releases 14 matched models and a multilingual robustness benchmark, finding that tokenizer design is associated with distinct robustness patterns across perturbations and languages.

  • Problem

    Tokenizer effects are difficult to disentangle from model architecture, training data, and training duration, limiting evidence about their influence on language-model behavior.

    Source: §1 · pp. 1–2

  • Method

    TokSuite pairs 14 otherwise matched models using different off-the-shelf tokenizers with a native-curated benchmark of real-world perturbations across five languages.

    Source: §1 · p. 2, To what extent does linguistic variety… · p. 2

  • Results

    Unconventional tokenizers are more robust than standard subword methods in the reported evaluations, noise disproportionately affects non-English languages, and robustness trends persist across model scales.

    Source: To what extent does linguistic variety… · p. 2

  • Takeaways & Limitations

    Robustness to surface-level variations is a meaningful criterion to consider alongside compression-oriented metrics when selecting or evaluating tokenizers for multilingual deployment.

    Source: §7 · p. 9

    The benchmark excludes coding and math tasks and focuses on base models with multiple-choice evaluations; generation, instruction-following, code, and larger-scale evaluations remain future work.

    Source: §8 · p. 9

Abstract

from arXiv · show

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

1. Introduction

TokSuite addresses the difficulty of isolating tokenizer effects by releasing matched language models and a multilingual benchmark of real-world perturbations. Across the tested settings, tokenizer design shapes robustness, with patterns varying by language and tokenizer type.

1. Introduction

  • TokSuite addresses a gap: existing model comparisons confound tokenizer effects with differences in architecture, training data, and training duration.Tokenization also combines interrelated choices such as segmentation algorithm, granularity, vocabulary constraints, and vocabulary-training data.Source: §1 · pp. 1–2
  • TokSuite releases 14 language models matched in initialization, architecture, and training data composition, varying only in tokenizer.The selected tokenizers span byte-level and subword approaches, English-only and multilingual designs, and differing vocabulary and preprocessing choices.Source: §1 · p. 2

To what extent does linguistic variety and the resulting

  • The benchmark covers five orthographically and morphologically diverse languages and perturbations curated to reflect real-world changes to text.It includes English, Turkish, Italian, Farsi, and Mandarin Chinese, with examples such as optional Farsi diacritics, keyboard substitutions, and Unicode formatting.Source: To what extent does linguistic variety… · p. 2
  • Unconventional tokenizers are more robust than standard subword methods in reported evaluations, while noise disproportionately affects non-English languages and all tokenizers struggle with Unicode formatting and technical text.The reported robustness trends persist across model scales.Source: To what extent does linguistic variety… · p. 2

2. Background

Tokenization maps input text to vocabulary entries, which become token IDs and then model embedding vectors. Vocabulary construction and preprocessing choices shape how text is segmented, with consequences for what tokenizers can represent and how models process inputs.

Tokenizers

  • Tokenization segments input symbols into strings from a vocabulary, whose entries map to integer IDs and then to embedding vectors.Natural-language vocabularies are estimated from data because language is open-ended and ambiguous.Source: §2 · pp. 2–3
  • Different segmentations can change what a model can reuse or must compose, affecting learnability and downstream information processing.For example, “dogs” can be represented as “dog” plus “s” or as one token.Source: §2 · p. 3

Training Data; Learning Algorithm

  • Tokenizer vocabulary composition depends on training data and learning algorithms, which primarily select vocabulary items by frequency.An algorithm trained on text containing a typo may include that typo as a token more readily than one trained on edited text.Source: §2 · p. 3
  • Larger vocabularies can include more rare words as individual tokens, while byte-fallback includes all 256 bytes to represent any Unicode character.Multilingual vocabularies may require substantially more entries when they cover scripts that do not share an alphabet.Source: §2 · p. 3

Preprocessing

  • Pre-tokenization can block merges across boundaries, while contraction, number, and whitespace handling vary across tokenizer pipelines.These choices involve trade-offs; for instance, grouping digits by thousands may suit some math inputs better than dates.Source: §2 · p. 3

3. The TokSuite Models

TokSuite trains matched models using 14 diverse off-the-shelf tokenizers, aligns shared tokens for initialization, and applies a common architecture and training setup. Standard benchmark evaluations show performance differences among the otherwise equivalent models.

3.1. Tokenizer Selection and Characteristics

  • The 14 selected tokenizers vary in algorithm, vocabulary size, multilingual coverage, preprocessing, and out-of-vocabulary handling.Their vocabularies range from 259 tokens for ByT5 to over 256,000 for Aya or XGLM.Source: §3.1 · p. 4

3.2. Cross-Tokenizer Vocabulary Alignment

  • A unified super vocabulary and token-ID mappings align shared token strings so their embeddings can receive the same initialization across models.The mapping accounts for tokenizer-specific conventions and uses UTF-8 byte representations.Source: §3.2 · p. 4

3.3. Model Architecture and Training Configuration

  • All models use the Lingua Llama-3.2-1B configuration and train for 100,000 steps on approximately 100 billion tokens.The fixed token budget means models see different amounts of raw information; the authors validate the choice with same-text-budget experiments.Source: §3.3 · pp. 4–5
  • The otherwise equivalent TokSuite models show notable performance differences on HellaSwag, ARC, PIQA, and XNLI, attributed by the authors to tokenization.The evaluations are presented as a sanity check of the trained base models.Source: §3.3 · p. 5

4. The TokSuite Benchmarks

TokSuite benchmarks tokenizer robustness to realistic input variations across five languages and technical domains, using canonical questions as a baseline. Its framework measures perturbation-related accuracy drops, tokenizer efficiency, and statistical uncertainty.

4. The TokSuite Benchmarks

  • The benchmark uses simple canonical questions with strong baseline accuracy to measure degradation under realistic input variations in Chinese, English, Farsi, Italian, and Turkish.Questions are selected through model-in-the-loop testing; over 70% of models answer each canonical question correctly.Source: §4 · p. 5

4.1. Multi-lingual Parallel Dataset

  • Native speakers translate 40 English multiple-choice questions into four languages and curate perturbations spanning orthography, register, morphology, noise, grammar, and linguistic variation.Perturbations include keyboard-related substitutions, diacritics, spelling variations, contractions, and dialects.Source: §4.1 · p. 5

4.2. Math & STEM Datasets

  • Math and STEM evaluations include formatted notation and structural representations, while multilingual arithmetic questions are translated into Chinese, Farsi, Turkish, and Italian.Examples include LaTeX expressions, molecular diagrams, tree structures, and flowcharts.Source: §4.2 · p. 6

4.3. The TokSuite Evaluation Framework

  • Robustness is measured with byte-length normalized log-likelihood and relative accuracy drop; tokenizer efficiency uses fertility, parity, and continued-word metrics, with bootstrap intervals and pairwise significance tests.Lower relative accuracy-drop values indicate greater robustness; the framework estimates 95% confidence intervals using 10,000 bootstrap trials.Source: §4.3 · p. 6

5. Findings

TokSuite finds substantial differences in robustness across tokenizers and perturbation types. TokenMonster and ByT5 perform strongly in different settings, while multilingual noise and technical or formatted content expose shared vulnerabilities.

Impact of Tokenization Algorithm Design on Multilingual Robustness

  • TokenMonster, despite its English-only 32,000-token vocabulary, had the lowest average relative performance drop across multilingual perturbations at 17%.The passage attributes this performance to its ungreedy algorithm and consistency constraints, which limit alternate token sequences for a word.Source: §5 · p. 6
  • ByT5 outperformed 9 models despite its 259-token vocabulary, while its byte-level robustness came with the highest subword fertility and PCW scores.The passage reports 0.04/0.06 drops for English/non-English orthographic errors and a 0.18 average drop for multilingual noise.Source: §5 · p. 6, Table 1 · p. 7

Amplification of Tokenization Vulnerabilities under Multilingual Noise

  • Noise caused larger average performance drops in non-English languages (0.21) than in English (0.15), with subword fragmentation especially damaging in morphologically complex languages.A Turkish spacing error fragmented the phrase into less meaningful subword sequences, whereas byte-level errors yield altered sequences of known bytes.Source: Table 1 · p. 7, §5 · p. 7

Structural Limitations in Mathematical and STEM Content

  • Models showed average performance drops of 0.22 for LaTeX and 0.28 for STEM content, where whitespace, symbol placement, and structural conventions matter.The passage identifies these formatting demands as parallel to challenges in coding tasks.Source: Table 1 · p. 7, §5 · p. 7

Universal Challenges Across Tokenizers

  • Unicode styling and character transformations produced the highest average drop, 0.53; XGLM’s NFKC normalization mitigated this degradation but limited faithful representation of Unicode formatting.Source: §5 · p. 7

Scaling Effects on Tokenization Robustness

  • Larger industry-scale models trained much longer showed only modest robustness improvements, and the authors conclude tokenizer design matters more than simply increasing model scale, training duration, and data.Source: §5 · pp. 7–8

6. Related Work

Prior work has studied tokenization’s effects on performance, cost, multilingual coverage, and robustness, but often under different controls or on narrower perturbations. TokSuite relates to this literature through its controlled comparisons and native-speaker-curated real-world variations.

Tokenization Design Factors

  • Ali et al. (2024) and Schmidt et al. (2024) found that intrinsic compression metrics do not reliably predict downstream performance; TokSuite evaluates pretrained tokenizers with controlled initialization across five languages.Source: §6 · p. 8

Multilingual Coverage; Algorithm Choice

Byte-level Models

Tokenization Robustness and Vulnerabilities; Decoding Algorithms

  • Earlier vulnerability studies examined trap words, non-canonical segmentations, and noise; TokSuite instead covers real-world, language-specific variations curated by native speakers (Wang et al., 2025; Geh et al., 2025; Dhole et al., 2023; Wang et al., 2021a;b).Token Healing and byte-level probability computation provide limited improvement on TokSuite (Phan et al., 2025).Source: §6 · p. 8

7. Discussion

Across several perturbation categories, robustness scores converge, but tokenizer choice still matters: TokenMonster combines high robustness with a small vocabulary and distinctive tokenization consistency. Robustness differences are not explained by standard compression-oriented intrinsic metrics, while off-the-shelf tokenizer comparisons cannot isolate individual design choices.

7. Discussion

  • The authors study off-the-shelf tokenizers as practitioners encounter them, but their differences across several dimensions prevent attributing performance to any single design choice.A fully controlled factorial study that varies one tokenizer design factor at a time is beyond the work’s scope.Source: §7 · p. 8
  • Robustness scores converge across several perturbation categories, particularly in English, and standard compression-oriented intrinsic metrics do not explain the differences that emerge.The paper names fertility, vocabulary size, parity, and PCW as examples of these metrics.Source: §7 · p. 8
  • TokenMonster achieves the best robustness with a 32,000-token English-only vocabulary, but at the cost of lower canonical performance.The authors speculate that its tokenization consistency contributes to this result: an ungreedy algorithm limits variation in token sequences for the same word, including leading-space variants.Source: §7 · pp. 8–9
  • Tokenizer choice has a measurable, statistically significant impact on robustness across models of different scales, and text budget does not explain the robustness differences.Similar scores are reported across 300M, 1B, and 7B models trained with the same tokenizer; text-matched experiments confirm the text-budget finding.Source: §7 · p. 9

8. Future Work & Limitations

The authors identify scope boundaries in the training languages and benchmark tasks that may constrain how findings extend to other multilingual settings and evaluation types.

8. Future Work & Limitations

  • Training on five languages with higher mixing rates than massive multilingual models may underestimate multilingual interference in more realistic settings.The authors note that cross-lingual interference could degrade performance in those settings.Source: §8 · p. 9
  • Tokenizers provide a cost-free inductive bias, while compression rates constrain information processing within token budgets and can leave some languages under-served.The passage contrasts this constraint with the possibility that additional training data may alleviate some vulnerabilities.Source: §8 · p. 9
  • TokSuite targets base models and multiple-choice questions; coding and math tasks were excluded because performance was inconsistent at the considered scale.The authors identify generation-focused evaluations, instruction-following settings, code domains, and larger scales as directions for future work.Source: §8 · p. 9

9. Conclusion

The conclusion presents TokSuite as evidence that tokenizer choice affects language-model robustness and capability, addressing the limited systematic understanding of commonly used off-the-shelf tokenizers.

9. Conclusion

  • Practitioners commonly adopt off-the-shelf tokenizers without a systematic understanding of their impact on model behavior.Source: §9 · p. 9
  • TokSuite releases 14 otherwise-identical models and a native-speaker-curated benchmark covering natural variations across five languages and technical domains.The models differ only in tokenizer, and the benchmark targets orthographic and morphological challenges.Source: §9 · p. 9
  • Tokenizer design can matter more than vocabulary size: an English-only tokenizer outperformed larger multilingual ones on certain perturbations.The conclusion names TokenMonster as the English-only tokenizer and notes that byte-level models were more robust to multilingual noise and subword fragmentation.Source: §9 · p. 9
  • Technical-content analysis found critical vulnerabilities in which trivial formatting differences caused catastrophic performance degradation.Source: §9 · p. 9

Impact Statement

The impact statement frames TokSuite as an initial step toward standardized study of tokenizer effects in multilingual real-life settings, with coverage of five languages and character-level perturbations.

Impact Statement

  • TokSuite aims to support a standardized methodology for studying off-the-shelf tokenizer effects across English, Chinese, Farsi, Italian, and Turkish.The authors encourage future multilingual and language-specific benchmarks targeting linguistic nuances and character-level perturbations.Source: Impact Statement · pp. 9–10

Appendix Contents

  • The appendices list supporting material on tokenization, training, benchmark composition, and detailed benchmark results.Source: Appendix Contents · p. 16

Pretokenization

  • Pretokenization strategies differ in how they split text: GPT-2 uses whitespace and transitions between letters, numbers, and punctuation, while GPT-4 also splits after every three contiguous digits.The supplied entries also describe variants used by Qwen 3, GPT-4o, Tekken, BLOOM, and SentencePiece.Source: Pretokenization · pp. 16–17

Numbers Processing

  • Number processing ranges from splitting every digit to letting the learning algorithm determine boundaries jointly for letters and numbers.The glossary also describes grouping numbers during pretokenization, including groups of three contiguous digits.Source: Numbers Processing · p. 17

Contractions Processing

  • Contraction handling may split contractions manually or compositionally, or leave their tokenization to the learning algorithm.The entries distinguish GPT-2 and GPT-4 handling from learned and composed approaches.Source: Contractions Processing · p. 17

Unicode Normalization

  • Unicode handling ranges from applying no normalization to decomposed, composed, or compatibility-composed forms.The glossary notes that compatibility mappings can make detokenization lossy.Source: Unicode Normalization · p. 18

Whitespace Treatment

  • Whitespace may be normalized, learned as segments, represented by predefined tokens, or preserved as individual character tokens.The glossary says normalized whitespace causes lossy detokenization, whereas learned, manual, and individual handling preserve whitespace.Source: Whitespace Treatment · p. 18

3 Bytes

  • Zero-width characters may be maintained in their original three-byte representation, assigned vocabulary tokens, normalized, or removed.Source: §3 · p. 18

B. Intrinsic Tokenization Efficiency Metrics

The section compares intrinsic tokenization efficiency across languages, then describes shared model initialization, training behavior, and downstream performance. Tokenizers differ on these measures, while multilingual-specialized tokenizers show advantages on some intrinsic metrics.

B. Intrinsic Tokenization Efficiency Metrics

  • The metrics quantify tokens per real word, parity of tokenized lengths across parallel languages, and the proportion of words split into multiple tokens.Fertility, parity, and continued-word proportion are evaluated using parallel Flores200 samples.Source: B. Intrinsic Tokenization Efficiency… · pp. 18–19
  • mBERT has the lowest average subword fertility at 1.54, while XGLM has near-optimal average parity at 1.18; intrinsic metrics do not significantly correlate with TokSuite performance.ByT5 needs 7.72 tokens per Farsi word versus 4.40 per English word, and larger vocabularies alone do not guarantee efficiency.Source: B. Intrinsic Tokenization Efficiency… · p. 19, Table 4 · p. 19

C. Model Training

C.1. Model Initialization

  • A shared super-vocabulary initialization lets each model select its tokenizer-specific embedding rows while reusing parameter values for most other layers.The embedding table is initialized from the corresponding rows of the shared vocabulary embedding table.Source: §C.1 · p. 21

C.2. Model Performance

  • All models converge to similar BPB values without loss spikes or divergence, supporting comparable training stability.BPB normalizes cross-entropy loss by log2 |V| to account for vocabulary-size differences.Source: §C.2 · p. 21, Figure 6 · p. 21
  • Models with larger vocabularies tend to perform better on downstream tasks, while advanced multilingual reading comprehension and reasoning remain slightly above random and uncompetitive.The reported pattern covers English reasoning, multilingual NLI, Belebele reading comprehension, and INCLUDE reasoning benchmarks.Source: §C.2 · p. 22, Figure 7 · p. 22

C.3. Training Data Consumption and Fairness

A fixed token budget leads different tokenizers to consume different amounts of raw text, but byte-normalized training preserves the reported robustness patterns. The benchmark spans multilingual, math, and STEM questions, with detailed results presented across canonical and perturbed inputs.

C.3. Training Data Consumption and Fairness

  • ByT5 consumed 100 GB of text under the fixed token budget, versus approximately 215–477 GB for other models; byte-normalized training preserved robustness patterns.Models consuming more bytes were not necessarily better performers, and the controlled experiment trained four models on identical text budgets.Source: Table 5 · p. 22, §C.3 · p. 23

D.1. Question Style and Difficulty

  • Canonical-question accuracy is strong in English and Italian, but slightly lower in Farsi, Turkish, and Chinese.The authors suggest higher fertility, continued-word, and parity scores may mean less raw-byte information is consumed in those languages.Source: §D.1 · p. 23, Figure 8 · p. 25

D.2. Benchmark Composition

  • The multilingual parallel dataset makes up 80% of TokSuite, with the remainder covering math and STEM.Source: §D.2 · p. 24

E. Detailed Benchmark Results

  • Figure 8 compares canonical and perturbed-question accuracies across five languages; Tables 8–9 report robustness measures for writing-system and input-medium perturbations.The captions describe Acccan and relative performance drop; lower values indicate greater robustness where specified.Source: Figure 8 · p. 25, Table 8 · p. 26, Table 9 · p. 26

E.1. Orthographic & Script Challenges

This section examines tokenizer robustness to orthographic, morphological, and noisy-input challenges, including changes in writing system, diacritics, spelling, grammar, and linguistic variation. It also considers how robustness profiles vary across model scales, training budgets, and random seeds, and how exact-byte evaluation affects Gemma-2’s robustness gap.

E.1. Orthographic & Script Challenges

  • Average relative performance drops range from 0.11 for Traditional Chinese to 0.60 for Chinese romanization, with keyboard and script constraints also testing robustness.Pinyin segmentation aids do not prevent the degradation on romanized Chinese text.Source: §E.1 · p. 27
  • Models average a 0.11 drop on homoglyphs and -0.01 on Chinese zero-width characters, but a 0.24 drop on Farsi zero-width characters.The passage attributes Farsi sensitivity to invisible characters disrupting both whitespace segmentation and character-joining rules.Source: §E.1 · p. 27
  • Diacritics cause average drops of 0.49 on Chinese and 0.41 on Farsi, while incorrect Italian accents cause a 0.10 drop; BLOOM performs best overall at 0.26 average drop.TokenMonster has a 0.21 drop on Farsi, GPT-2 improves by -0.02 on Italian wrong accents, and Llama-3.2 drops 0.60 on Farsi.Source: §E.1 · pp. 27–28
  • TokenMonster and character-level ByT5 show the strongest performance on orthographic and grammatical errors, while changing “week” to “weak” breaks 6/14 models despite both words being tokens.This example suggests robustness depends on semantic stability of token representations, not only vocabulary coverage.Source: §E.1 · p. 28
  • In Turkish, a single /t/-to-/d/ change can reorganize token boundaries, whereas English grammatical errors produce less striking performance degradation.The passage links this brittleness in agglutinative languages to multilingual tokenizers trained predominantly on English.Source: §E.1 · p. 28
  • TokenMonster has a 0.08 average drop across linguistic variety, vocabulary size shows little to no correlation with robustness, and colloquial phrasing breaks 3 models.ByT5 shows superior consistency without traditional vocabulary constraints; the “turns out to be” phrasing example suggests informal discourse can disrupt factual recall.Source: §E.1 · p. 30

E.2. Morphological Challenges

  • Morphological variants receive inconsistent segmentations, including contractions and Italian elisions, which can give semantically equivalent forms disparate tokenizations.For “Google’s,” most listed models separate “Google” and “’s,” while XGLM and mBERT fragment the marker and TokenMonster lowercases it; “dell’Italia” and “d’Italia” also tokenize differently across models.Source: §E.2 · p. 30

E.3. Noise

  • Tokenizers using complete word tokens are more vulnerable to noise because character perturbations can turn familiar words into unfamiliar subword combinations, while smaller subword units preserve more consistent segmentation.Source: §E.3 · p. 31

E.4. Mathematical & Scientific Expressions

  • Only 8/14 models identified “volume” with basic LaTeX formatting, 2/14 with braces, and 2/14 with spaced braces; TokenMonster and Qwen-3 remained correct through the spaced version.All models identified “volume” for plain text. In chemical notation, all models identified CH4 and H2SO4, but only Llama and GPT-2 interpreted the ASCII methane diagram, and systematic nomenclature reached 65% accuracy. In multilingual arithmetic, Gemma-2 matched canonical English on Italian but degraded 53% on Farsi; Aya showed the greatest cross-language robustness.Source: §E.4 · pp. 33, 35, 36

E.5. Styling & Unicode Challenges

  • Unicode and styling perturbations degraded performance across all models, and normalization such as XGLM’s NFKC may mitigate some changes but is irreversible.Some tokenizers retain distinct tokens for styled characters yet still fail frequently; Figure 9 distinguishes changes that NFKC normalizes from those it cannot.Source: §E.5 · p. 36, Figure 9 · p. 37

F. Decoding Algorithms

  • Exact-byte evaluation mitigated Gemma-2’s robustness drops but did not close the gap to TokenMonster, and it degraded canonical performance.The method was evaluated on Gemma-2, which performed worst among the TokSuite SentencePiece models; the available implementation also required substantial multilingual adaptation and took hours on 40 examples.Source: F. Decoding Algorithms · p. 38

G.1. Model Scale

  • Llama-3.2’s robustness profile remained largely unchanged from 300M to 7B parameters, with only modest improvement on noise at larger scales.The 7B model nevertheless had superior canonical performance to all fourteen baseline TokSuite models.Source: §G.1 · p. 39, Table 22 · p. 39, Table 23 · p. 39

G.2. Text-matched Models

  • Text-matched models had average performance drops of 0.24 under both text-budget and token-budget training, while Comma’s 70% larger token budget left robustness similar at 0.22 → 0.24.Gemma, despite consuming 5% fewer tokens, changed from 0.26 to 0.24; the authors conclude that changing token budget did not directly translate into robustness gains.Source: §G.2 · p. 39, Table 25 · p. 40

G.3. Sensitivity to Random Initialization

  • Across three Llama-3.2 tokenizer initialization seeds, individual perturbation-category scores remained stable, with average robustness of 0.25.The main-text robustness values fell within the bootstrapped confidence intervals reported for the additional seeds.Source: §G.3 · pp. 39–40

G.4. Evaluating Industry-level Models on TokSuite Benchmark

External pre-trained models improve on some perturbation categories, but Unicode formatting remains difficult and larger model size does not yield significant robustness gains. Their comparisons with TokSuite are confounded, while bootstrap and Wilcoxon analyses assess variability and significance.

G.4. Evaluating Industry-level Models on TokSuite Benchmark

  • External-model comparisons are not definitive because architecture, scale, training duration, data composition, and objectives differ; models also need reasonable canonical accuracy for fair robustness comparisons.Poor-multilingual-performance models were excluded, while English-only models were evaluated on English questions separately.Source: §G.4 · pp. 40, 42
  • Larger models achieved better canonical accuracy within families but showed no significant robustness gains; Gemma-2-9B and Gemma-2-2B had similar robustness profiles.Aya-Expanse-8B also performed comparably to much smaller models, while instruction tuning brought only modest robustness improvements for models at or below 2 billion parameters.Source: §G.4 · p. 42
  • Average degradation fell from 0.22 (TokSuite, LaTeX) to 0.09 (external models, LaTeX), and from 0.28 (TokSuite, STEM) to 0.12 (external models, STEM); Unicode remained hardest, improving from 0.53 to 0.46.These are averages across models; lower degradation indicates greater robustness.Source: §G.4 · p. 42
  • Unlike ByT5, BLT and Bolmo showed no comparable overall robustness; BLT’s diacritic gains may partly reflect low non-English canonical performance, while Bolmo underperformed the weakest TokSuite model on English noise.BLT improved on English noise but remained brittle to non-English noise; Bolmo was evaluated only on English perturbations.Source: §G.4 · pp. 42–43
  • Byte-level architectural innovations did not automatically confer robustness, and latent transformers may retain perturbation weaknesses because their learned representations and segmentation patterns reflect training data.The authors describe this as a possible explanation for weaknesses on perturbations not well represented during training.Source: §G.4 · p. 43

H. Statistical Significance

  • Figure 10 plots robustness distributions with 95% bootstrap intervals and significance colors; Figures 11–13 regroup the views by vocabulary bucket, algorithm, and pre-tokenization splits.Figure 10 orders models by vocabulary size and marks results more than one standard deviation from the mean.Source: H. Statistical Significance · p. 43, Figure 10 · p. 45, Figure 11 · p. 45
  • A 10,000-trial bootstrap estimated variability, and paired Wilcoxon tests found most observed robustness differences statistically significant at α = 0.05.The pairwise tests compare tokenizers on the same tasks; their results are reported in Table 32.Source: H. Statistical Significance · p. 43, Table 32 · p. 44
Loading 2512.20757v2…