PaperlayerView exact revision on arXiv

Source-linked AI summary

Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun

arXiv:2608.26449v1cs.CLcs.LG

TL;DR

The paper shows that a letters-only pre-tokenizer regex structurally splits abugida words at vowel signs, imposing a fertility floor that corpus composition cannot lift. Matched experiments, corpus sweeps, and ecosystem analysis quantify the effect and show that a one-character-class repair improves tokenizer outcomes.

  • Problem

    A character class chosen for English can split abugida words at vowel signs, while the usual data-shortage diagnosis does not explain Nepali fertility remaining unchanged as Nepali data increases.

  • Method

    The paper derives a training-free fertility bound from pre-tokenizer boundaries, tests matched tokenizer pairs and corpus-share sweeps, and classifies pre-tokenizers across 3,479 HuggingFace repositories.

  • Results

    Nepali fertility drops from 4.78 to 1.58 tokens per word, while the broken tokenizer varies 1.7% and the fixed tokenizer varies 33.9% across a 5%–95% Nepali-content sweep.

  • Takeaways & Limitations

    The pre-tokenizer can impose a design-time ceiling on fertility, and checking whether fertility responds to corpus composition helps distinguish structural constraints from data shortages.

  • Takeaways & Limitations

    The trained experiments cover 5 language pairs, all with a 50/50 English-target-language byte mixture, so treatment fertilities exceed realistic multilingual-mixture values.

Abstract

from arXiv · show

Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters. In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign. Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition. We formalise this effect as a training-free lower bound on fertility. Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai), whereas Latin, Cyrillic, Hangul, and Han show exactly 1.00x. For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali. When the Nepali share of the training corpus is swept from 5% to 95%, the broken tokenizer barely shifts at all (1.7%) while the fixed one shifts 33.9%, which separates a structural ceiling from a data shortage without needing to inspect any code. We train three 268M models that differ only in their tokenizer; the fixed variant achieves 4.43% lower held-out Nepali bits per byte at equal compute, and it still leads when given the same bytes with 1.59x the compute. A census of 3,479 HuggingFace repositories finds the letters-only word class present in 63.3% of the most-downloaded text-generation models, accounting for 72.5% of their downloads. GPT-4o's o200k pattern already uses a mark-aware word class, making the repair itself prior art. We quantify its value, show how to recognise its absence from symptoms alone, map which scripts it reaches, measure how widely it is deployed, and release a 65,536-entry Nepali-English tokenizer with a harness that regenerates every number here from public data on a laptop.

1 Introduction

The paper identifies a structural tokenizer defect: GPT-2’s letters-only pre-tokenization splits abugida vowel marks before BPE can merge them. It quantifies the resulting fertility floor, diagnosis, cross-language scope, ecosystem prevalence, and practical repair.

  • 1 Introduction: 4.4 tokens per word is the observed Nepali fertility that motivated testing whether low-resource data, rather than tokenization structure, was responsible.Increasing Nepali’s corpus share repeatedly failed to change the number under the standard tokenizer.
  • 1 Introduction: \p{L}+ splits Nepali words at combining vowel marks, while [\p{L}\p{M}]+ preserves the word for within-pre-token BPE merges.Devanagari vowel signs and the virama belong to Mn or Mc rather than Unicode category L.
  • 1 Introduction: 1.7% versus 33.9%: Nepali fertility changes little under \p{L}+ but shifts substantially under [\p{L}\p{M}]+ as Nepali content rises from 5% to 95%.This sweep distinguishes an upstream structural constraint from sensitivity to corpus composition.
  • 1 Introduction: 4.78 versus 1.58 tokens per word: matched tokenizers differing only in one character class produce markedly different Nepali fertility.The paper repeats the controlled comparison across Hindi, Bengali, Tamil, and Malayalam.
  • 1 Introduction: 63.3% of the most-downloaded text-generation repositories use the letters-only pre-tokenizer, accounting for 72.5% of their downloads.The study therefore treats the defect as broadly deployed rather than an isolated implementation mistake.
  • 1 Introduction: The paper releases a 65,536-entry Nepali–English tokenizer and a laptop-runnable harness that regenerates its reported numbers from public data.It also notes that the mark-aware repair is already used by GPT-4o’s o200k pattern.

2 Background

Byte-level BPE first applies regex pre-tokenization and then learns merges only within the resulting chunks. Because abugida vowel signs are combining marks rather than letters, a letters-only class creates a training-independent fertility floor.

  • 2 Background: Byte-level BPE maps text to UTF-8 bytes, then repeatedly merges frequent adjacent pairs only within regex-defined pre-token chunks.Pre-token boundaries prevent merges from spanning word boundaries.
  • 2 Background: Abugidas encode non-inherent vowels as Mn or Mc combining marks attached to consonants, separate from Unicode category L.This makes a letters-only word class unsuitable for preserving many abugida words.
  • 2 Background: Three of six characters in nep¯al¯ı are marks, so \p{L}+ creates six single-character pre-tokens while [\p{L}\p{M}]+ creates one.The code-point example makes the segmentation difference visible before any BPE training.
  • 2 Background: |T(s)| ≥ |P(s)|: every BPE tokenizer’s output length is at least the number of pre-tokens produced by its pre-tokenizer.The inequality holds regardless of vocabulary size, corpus, mixture, or merge count.
  • 2 Background: 4.67 versus 1.14 tokens per word: the Nepali pre-tokenization floors under \p{L}+ and [\p{L}\p{M}]+ differ before training.The letters-only regex produces 4.09 times as many Nepali pre-tokens.
  • 2 Background: Figure 1 plots Nepali fertility against Nepali training-corpus share for seven independently trained tokenizers per curve, contrasting control and treatment character classes.The control remains on its pre-tokenization bound, while the treatment declines as Nepali content rises.

4 Diagnosis: a sweep that does not respond

The corpus-share sweep tests whether poor Nepali fertility is caused by insufficient Nepali data. The letters-only tokenizer is nearly insensitive to that input, whereas the mark-aware tokenizer responds strongly.

  • 4 Diagnosis: a sweep that does not respond: The sweep varies only Nepali byte share at 65,536 vocabulary and 2,000 MB corpus under the full configuration.This isolates corpus composition as the changing experimental factor.
  • 4 Diagnosis: a sweep that does not respond: 5%–95% Nepali content changes fertility by 1.7% under \p{L}+ and 33.9% under [\p{L}\p{M}]+.The control remains within 3.6% of its training-free bound throughout the sweep, while the treatment declines monotonically.
  • 4 Diagnosis: a sweep that does not respond: A flat fertility response is evidence against the mixture hypothesis because data-limited quantities should respond to added data.The paper locates the remaining upstream candidates in the normaliser, pre-tokenizer, and vocabulary budget.

5 Matched pairs

Matched tokenizer pairs isolate the word-class effect by changing one character class while holding the training recipe fixed. Across five languages, the letters-only control approaches its training-free floor, while the mark-aware treatment substantially lowers target-language fertility at a small English cost.

  • Controlled setup: The two tokenizers use identical data, normalization, mapping, settings, special tokens, and 65,536 vocabulary, differing only in word_class.Both are produced by the same training function, making the control the repaired production recipe with the repair removed.
  • Nepali headline pair: 4.78 versus 1.58 tokens per word gives Nepali a 3.03× control-to-treatment fertility ratio.The control is within 2.2% of its training-free bound of 4.67.
  • Trade-off: 2.1% is the English fertility cost of the repair at a fixed vocabulary.English fertility rises from 1.276 to 1.303 because intact Nepali words win merge slots from English.
  • Prediction test: The four additional languages were selected before training to span 2.2× in predicted control fertility, from Hindi lowest to Malayalam highest.The identical experiment used the same corpus size, target-language share, vocabulary, and training function.
  • Prediction test: 1.8% above the training-free floor is the average control gap, never exceeding 2.2% across the five languages.The treatment arms settle 18–67% above their own bounds because the binding constraint shifts to vocabulary budget.
  • Released-tokenizer comparison: The released-tokenizer comparison is lossiness-aware: character deletion invalidates fertility comparisons, while IndicBERTv2 is lossless but uses a much larger vocabulary.IndicBERTv2 reaches 1.58 tokens per word versus the paper tokenizer’s 1.69, but it is not a generative drop-in.
  • Released-tokenizer comparison: Every letters-only tokenizer scores 3.76 or worse on Nepali, but the word class is necessary rather than sufficient.DeepSeek-V3 is mark-aware yet reaches 4.29 because its training text contains little Devanagari.

7 How much of the ecosystem inherits the pattern

A census of 3,479 HuggingFace repositories shows that letters-only pre-tokenization remains widespread among text-generation models and dominates their recent downloads. The classification treats omitted ByteLevel regex fields as enabled, matching library behavior, while the authors limit the conclusion to what the counts establish.

  • Census method: 3,479 distinct repositories were classified across the top-download and text-generation listings, replacing a selected tokenizer shortlist with a broader census.The classifier inspects tokenizer metadata and file layouts without fetching vocabulary and merge bulk.
  • Census results: 63.3% of 1,345 text-generation repositories carry a letters-only pre-tokenizer.These repositories represent 52.9% of the 412 distinct tokenizers behind them.
  • Census results: 72.5% of 30-day downloads come from repositories carrying a letters-only pre-tokenizer.Repository, distinct-tokenizer, and download denominators address different weighting objections and agree on the pattern.
  • Classification rule: Absent use_regex fields are classified as enabled because the library fills them with true, causing the canonical GPT-2 pattern to split Nepali into six pieces.Treating absence as disabled would misclassify older files and the GPT-2 tokenizer itself.
  • Scope: The census supports the narrower conclusion that letters-only word classes are still shipped by default, not that every deployed tokenizer is affected.The paper explicitly defers the census’s unsupported claims to its limitations discussion.

8 Which scripts the defect reaches

The defect reaches every tested abugida, with severity determined by how much word information is encoded as combining marks; it is dormant in unvocalised Arabic and Hebrew and absent in several non-mark-heavy scripts.

  • Script coverage: All 17 tested abugidas are affected, with shatter ratios from 1.47× in Tibetan to 9.02× in Thai.The ratio compares pre-tokens under \p{L}+ with those under [\p{L}\p{M}]+ on identical parallel content.
  • Script coverage: Latin, Cyrillic, Hangul, and Han have exactly 1.00× shatter ratios because ordinary NFC text in these scripts carries no combining marks.
  • Script coverage: Tibetan is milder because TSHEG punctuation splits syllables under both word classes, leaving marks to fragment already short units.The defect therefore tracks the proportion of word information classified as marks, not simply whether a script is Brahmic.
  • Script coverage: Vocalised Arabic and pointed Hebrew reach 6.50× and 6.14× shattering, despite appearing unaffected when evaluation text omits diacritics.Modern newswire can therefore conceal a failure that appears on marked text.
  • Evaluation context: The downstream evaluation uses bits per byte because different tokenizations make per-token loss incomparable.The matched 268M runs compare broken and fixed tokenizers at equal token budgets and with a compute-normalised control.
  • Evaluation context: At equal compute, the fixed tokenizer reaches 0.3899 Nepali bits per byte versus 0.4080 for the broken tokenizer, a 4.43% improvement.On English, the fixed tokenizer improves by 1.84% despite costing 2.1% of English fertility.
  • Evaluation context: On Nepali, the fixed tokenizer reaches 0.3899 bits per byte versus 0.3925 for the broken tokenizer given 1.59× more compute.The paper reports one seed per condition and does not establish downstream task accuracy.

10 Reproducibility

The paper provides a laptop-runnable harness, manuscript source, and released tokenizer that regenerate its reported results from public data without GPU access or an account.

  • A single command regenerates every number in the paper, including numbers quoted in prose.
  • The harness, manuscript source, and released tokenizer are publicly available through the project repository and Hugging Face.
  • The workflow requires no GPU, Hugging Face account, or private data, and streams public Nepali–English data while producing machine-readable results and tables.A quick configuration takes about fifteen minutes and reproduces the 26-language bound and vocalisation probe exactly.
  • Tokenizer experiments run on a laptop CPU, while downstream evaluation requires a GPU for three 268M-parameter runs totaling 14.36B tokens.The downstream runs use 19.1 GPU-hours on one NVIDIA H100 80GB.

11 Related work

The paper builds on work documenting multilingual tokenizer costs and pre-tokenization inequity, while distinguishing its one-character-class repair from new segmentation algorithms. It also situates the defect within existing tokenizer designs and states the study’s scope and limitations.

  • Prior work: Earlier studies link tokenizer design to multilingual performance, cost, equity, and vocabulary allocation.The cited work includes monolingual-versus-multilingual tokenizer comparisons, cross-language cost analyses, vocabulary-allocation studies, and downstream evaluations.
  • Closest work: Velayuthan and Sarveswaran identify pre-tokenization as a source of unfair representation for Tamil, Sinhala, and Hindi and propose Grapheme Pair Encoding.Their approach segments on graphemes rather than bytes.
  • This paper: This paper instead measures an existing pre-tokenizer defect under a one-character-class control, derives its bound, provides a code-free diagnostic, and measures downstream and ecosystem effects.The repair leaves byte-level BPE intact and is already present in known tokenizer patterns.
  • Scope: The reported scope includes 26-language bounds, five trained language pairs, 50/50 English mixtures, and a Nepali–English corpus-share sweep.The trained pairs cover Hindi, Bengali, Nepali, Tamil, and Malayalam; the sweep is Nepali–English only.
  • Scope: The census is an exposure snapshot that excludes gated repositories and repositories without tokenizer.json, while metadata language tags are unevenly maintained.It does not establish that every letters-only tokenizer is harmful, because the class is irrelevant for Latin-only models.

Ethical Considerations

The defect is asymmetric: it burdens abugida and diacritic-bearing languages, can be mistaken for a data shortage, and may redirect scarce effort toward text acquisition that cannot help. The released tokenizer also inherits source-data biases and is not claimed deployment-ready.

  • Distributional impact: The measured cost falls on abugida scripts and, with diacritics, Arabic and Hebrew, while Latin, Cyrillic, Hangul, and Han have shatter ratio 1.00×.Fertility affects context length, training cost, and per-token API price.
  • Misattribution: The failure can present as a data shortage, directing effort toward acquiring text that cannot lift the structurally constrained metric.The paper identifies this misattribution as most expensive where budgets are smallest.
  • Release considerations: The released tokenizer uses public web text and a private custom corpus, inherits their biases, and is not claimed suitable for deployment without further evaluation.All reported experimental results use only public data and do not depend on the private corpus.

A The bound in full

The full bound is presented across FLORES-200 with writing-system annotations and supplementary character-level measures. The section also documents reproducible artifacts, computational requirements, and where fertility is intentionally undefined.

  • Reproducibility: The harness records resolved tokenizer commits, corpus hashes, library versions, and its own commit, with disjoint corpus partitions by document index modulo 20.These records support reproducibility across baseline loading and evaluation/training splits.
  • Computation: Tokenizer measurements run on a laptop CPU, while the 26-language bound and vocalisation probe require no training and finish in seconds.A 65,536-entry BPE over 2,000 MB takes one to eight minutes; the census is network-bound.
  • The bound in full: Table 6 gives the complete training-free pre-tokenization bound across FLORES-200 and groups languages with identical 1.00× bounds by writing-system family.Per-language values are supplied in table_bounds.csv.
  • The bound in full: Table 7 illustrates the mechanism code point by code point: three of Nepali’s six characters are marks, producing six versus one pre-token.The comparison is between \p{L}+ and [\p{L}\p{M}]+.
  • The bound in full: Table 8 presents the same mechanism with combining marks stripped and intact, showing where vocalisation activates the effect.The cited description identifies the comparison as unvocalised versus vocalised text.

D The Nepali pair in bytes per token

The byte-level comparison connects tokens per word to bytes per token and downstream bits per byte under a controlled Nepali–English setup. The census tables then classify deployed pre-tokenization patterns and relate letters-only syntax to measured behavior.

  • Nepali pair in bytes per token: Table 9 reports bytes per token for the controlled Nepali tokenizer pair, the quantity used to connect tokenization to downstream bits per byte.The comparison uses 2,000 MB at 50% Nepali by bytes and vocabulary 65,536, with all other settings identical.
  • Nepali pair in bytes per token: At a fixed token budget, bytes per token determines how much text the downstream arms read.This is why the two tokenizers can expose different amounts of text to the model.
  • Nepali pair in bytes per token: Table 10 contains the data behind the Nepali mixture sweep, with control \p{L}+, treatment [\p{L}\p{M}]+, and a training-free bound row.The two arms are identical apart from the word class.
  • Census patterns: Letters-only patterns are inherited GPT-2 descendants applied by ByteLevel when use_regex remains at its default, rather than deliberate new choices by repository authors.The paper characterizes this as an inherited default.
  • Census patterns: Table 11 ranks common distinct pre-tokenization patterns in the repository census and provides an example repository for each.Patterns are truncated in the table, with full text in table_census_pat.csv.
  • Behavioral validation: Table 12 checks that census classifier labels correspond to behavior: letters-only repositories occupy the top of the measured Nepali-fertility table.No lower row is letters-only.
  • Census methodology: The census denominator excludes repositories without tokenizers, gated repositories, and patterns too large for an 8 MB request, but retains older or nontokenizers formats.The exclusions avoid shifting affected or unaffected counts without evidence.

G Predicted against measured, across five languages

Across five matched tokenizer pairs, the training-free bound correctly separates control arms that remain at the floor from treatment arms that exceed their lower floor. This gap shows when the pre-tokenizer constrains fertility and when vocabulary allocation remains available.

  • Predicted against measured: Every one of the ten Figure 2 points lies on or above its arm’s training-free bound.Proposition 1 guarantees the inequality for both arms across all five matched pairs.
  • Predicted against measured: Control arms lie on the bound across the entire range, while treatment arms lie well above their own lower floors.The dashed y = x line represents each arm’s bound.
  • Practical interpretation: The practical difference between the series is not forced by the theorem: letters-only pre-tokenization leaves corpus interventions unable to reach the floor, whereas the fixed setting leaves vocabulary budget as the tradeable variable.The bound constrains the outcome only when the letters-only class is active; otherwise, the floor is slack.
  • Measured fertility: Table 12 reports each tokenizer’s word class alongside measured Nepali fertility, sorted first by class and then by fertility.The table organizes the measured comparison by the pre-tokenizer’s character class.
Loading 2608.26449v1…