Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

1 Introduction

• Cross-lingual token-budget disparity affects usable context, latency, and prompt-processing cost, and the same disparity also matters for pay-per-token pricing.The paper describes fairness operationally as avoiding systematic token-budget differences caused solely by scripts’ per-character byte-encoding costs.Source: §1 · pp. 1–2
• Up to 15× as many tokens as English may be required for parallel Shan text, and a fixed 128K-token context then holds only about 8.5K English tokens’ worth of Shan content.For 12 selected configurations in 3-byte BMP scripts, BBPE averages 200.188 tokens per parallel sentence versus 34.863 for English under a compact 16K vocabulary.Source: §1 · p. 2
• In BBPE, uncovered Unicode scalars fall back to byte-derived symbols, so the encoding floor is their pre-merge scalar-level cost rather than a lower bound on post-BPE token counts.Finite multilingual vocabularies cannot cover every subword pattern, and the UTF-8 floor varies by script and code point.Source: §1 · p. 2
• The encoding floor is a scalar-level analogue of corpus-level byte premium, while persistent token-premium disparities motivate aligned premiums and English-normalized dispersion metrics.The cited work reports that added vocabulary tokens become progressively less frequent and that disparities persist as vocabulary size increases.Source: §1 · p. 2
• UBE keeps characters encoded in 1–2 UTF-8 bytes on that path and routes 3-byte BMP characters through UTF-16, lowering their floor without imposing a uniform 2-byte floor on efficient spans.It leaves BPE training and merge learning unchanged, preserves exact round-trip decoding, and requires neither language identification nor model-architecture changes.Source: §1 · p. 2

2 Related work

• Aligned sentence-level token premiums are used as the main intrinsic measure of cross-lingual token-budget disparity, motivated by prior findings on parallel-text token lengths and API pricing.Follow-up work links tokenization disparity to unequal utility, downstream performance, latency, and infrastructure or access costs.Source: §2 · pp. 2–3
• Related metrics measure disparity at tokenizer, downstream LM-task, and dialectal NLP-task levels, while other work analyzes inequity causes, broader multilingual design choices, and tokenizer–model co-design.The cited metrics include Single Token Retention Rate, Performance Realization Ratio, and Tokenization Parity.Source: §2 · p. 3
• Within-substrate methods change segmentation or vocabulary while retaining the byte encoding, whereas substrate-changing methods alter input symbols through UTF-16, script-aware, morphology-driven, or bit-level representations.UBE routes characters between UTF-8 and UTF-16; the paper compares these alternatives at tokenizer and model levels.Source: §2 · p. 3
• Tokenization-free and byte-level models reduce reliance on fixed tokenizers but change architecture or input granularity, whereas UBE preserves standard BPE training and exact decoding.SentencePiece also supports BPE and unigram segmentation directly from raw text, rather than being restricted to byte-based representations.Source: §2 · p. 3

3 Background and method

• UBE leaves ASCII at a 1-symbol floor, lowers 3-byte BMP scripts from 3 to 2, and leaves 4-byte non-BMP code points at 4; BBPE16 is the UTF-16-only baseline.The table identifies CJK, Devanagari, and Thai as examples of the 3-byte BMP scripts targeted by UBE.Source: Table 1 · p. 3, §3 · p. 3
• The design targets 3-byte BMP scripts near byte fallback; non-BMP code points, including many emoji, remain at a 4-symbol floor because UTF-16 represents them as surrogate pairs.Unless otherwise noted, the paper focuses on BMP 3-byte scripts.Source: §3 · p. 3
• The mixed-script example “GPU性能” uses 9 base symbols with BBPE, 10 with BBPE16, and 7 with UBE before merges.The 512-symbol disjoint union lets the decoder identify the branch after merge-token expansion, even when merges span routing boundaries.Source: Figure 1 · p. 4, §3 · p. 5
• UBE routes each Unicode scalar to UTF-8 when its UTF-8 length is at most 2 bytes and to UTF-16 otherwise, without marker tokens or a language-specific routing rule.Characters are routed scalar by scalar, without additional normalization; disjoint byte alphabets let the decoder infer the active encoding.Source: §3 · p. 4
• UBE changes the initial symbol stream and uses a 512-symbol union alphabet, but retains standard BPE pair counting and merge selection and proves exact round-trip decoding.The Unicode 17 audit recovers every tested input, including all Unicode scalar values and official normalization, grapheme-break, and emoji test-suite inputs.Source: §3 · p. 4
• With 5 shared special tokens, UBE has 256 fewer merge slots than BBPE, an overhead of roughly 256/V that is below 1% at typical vocabulary sizes of V ≥32K.This is the vocabulary-budget trade-off for using the 512-symbol union alphabet rather than BBPE’s 256-symbol alphabet.Source: §3 · p. 5

4.1 Intrinsic tokenizer evaluation

• UBE reduces FLORES-200 token-premium Gini in every core setting by 6%–14%; the 95% confidence interval for the 101-lang/32K difference is below zero.Across the nine core comparisons, the reductions are supported by paired language bootstraps.Source: §4.1 · p. 5
• At 101-lang/32K, UBE lowers Gini from 0.2183 to 0.1979 (-9.4%); it produces fewer tokens in 189/204 languages, with its Gini advantage persisting through 256K.The largest reductions occur in low-coverage 3-byte scripts where UBE reduces fallback cost from 3 bytes to 2 bytes per character.Source: §4.1 · pp. 6–7
• On 400 mixed-script mC4 documents, UBE matches or slightly improves on BBPE in 7 of 8 tested languages, while Bengali shows a +0.4% regression.UBE also improves token-premium Gini on SIB-200 (-8.5%); MGSM is effectively unchanged (+0.1%), while UDHR is treated as a public-text stress test because of prefix overlaps.Source: §4.1 · pp. 6–8, Table 2 · p. 7

4.2 Language model evaluation

• Across 3 seeds at 101-lang/32K, mean BPB Gini is effectively tied: BBPE 0.1819±0.0052 versus UBE 0.1822±0.0074; downstream benchmark deltas are small and mixed.A source-byte-matched comparison finds UBE has the lowest common-text BPB and BPB Gini among UBE, BBPE, and BBPE16, while task accuracy is mixed.Source: §4.2 · pp. 8–9
• At 101-lang/32K, Santali gains 53% more usable context; in a separate 6-lang/32K prompt benchmark, sequence-processing rate improves by +45.0% for Amharic and +3.9% for English.The processing-rate comparison is end-to-end, reflecting sequence-length effects on MLP and memory-bandwidth costs as well as attention.Source: §4.2 · p. 9

4.3 Composition and tokenizer comparisons

• UBE composes with boundary policies and morphology maps: it lowers Gini under three boundary policies and reduces MYTE base-symbol positions by 4.58% across 201/204 configurations.The MYTE decomposition and morphology-merge maps remain fixed, and MYTE-UBE preserves MYTE’s decoded output.Source: §4.3 · p. 9
• In a five-tokenizer comparison, official SCRIPT-BPE has the lowest overall mean and token-premium Gini; UBE uses fewer English tokens and fewer tokens in 169/204 configurations.At the model level, differences among the five tokenizer conditions are small.Source: §4.3 · pp. 9–10, Table 6 · p. 10

5 Conclusion

• UBE reduces token-budget disparity across intrinsic evaluations, with the largest usefulness for compact vocabularies and low-coverage 3-byte BMP scripts near the encoding floor.Downstream differences remain small and mixed; same-content prompt processing improves by up to +45.0%.Source: §5 · p. 10

A.1 Pre-tokenizer: UTF-8-byte-length routing

• Characters encoded in 1–2 UTF-8 bytes stay on UTF-8, while 3–4-byte characters use UTF-16; only 3-byte BMP characters fall from 3 bytes to 2.Routing is per Unicode scalar value, not per language or grapheme cluster.Source: §A.1 · p. 18

A.2 Reference pseudocode

• UBE applies standard BPE independently within each GPT-2 regex chunk, and its BPE merge learning, ranking, and vocabulary construction remain unchanged.The decoder routes symbols through separate UTF-8 and UTF-16 buffers and flushes the active buffer when the alphabet switches.Source: §A.2 · pp. 18–19

A.3 Encoding example

• For 性, BBPE uses three UTF-8 bytes while UBE uses two UTF-16LE bytes, mapped to the PUA alphabet.The UTF-16 path uses a disjoint 256-symbol alphabet, making alphabet membership identify the active byte path.Source: §A.1 · p. 18, §A.3 · p. 19

A.4 Correctness and implementation notes

• For well-formed Unicode scalar strings without isolated surrogates, decoding an encoded string returns the original input unless optional prefix-space preprocessing changes it.The proof relies on disjoint alphabets identifying separate byte buffers, which are flushed when the stream switches paths.Source: §A.4 · p. 20
• The exact round-trip guarantee covers encoder-produced streams, not arbitrary model-generated sequences that may expand to malformed UTF-8 or UTF-16 bytes.Malformed same-path runs decode with replacement characters rather than raising an exception.Source: §A.4 · p. 20
• The Rust core implements routing, dual-buffer decoding, and offset post-processing, with Python wrappers exposing Hugging Face training APIs.Offset trimming is disabled so returned whitespace offsets remain as-is.Source: §A.4 · p. 20
• The decoder’s streaming state is an active alphabet tag plus one pending byte buffer, and constrained decoders need the per-symbol alphabet tag to apply byte constraints.Constrained-decoding latency is not benchmarked.Source: §A.4 · p. 20

B.1 Tokenizer training pipeline

• The canonical training flow processes each shard independently, merges its retained word-count state, then runs the full BPE merge loop to the target vocabulary size.The resulting tokenizer is saved as a Hugging Face-compatible tokenizer.json.Source: §B.1 · pp. 20–21
• Phase 1 filters per-shard singleton patterns with min_frequency=2 before saving states, so the later reduction is not equivalent to exact whole-corpus BPE.Phase 2 sums retained word counts in an order-independent reduce pass before finalizing the BPE merges.Source: §B.1 · p. 21
• UBE’s disjoint alphabet uses 256 more merge slots than BBPE at each vocabulary size, with the same five special tokens reserved.The table reports the resulting merge-slot accounting.Source: Table B.1 · p. 21, §B.1 · p. 21
• In the 6-lang/32K tie-break sensitivity check, differences were at most 0.0005 in Gini and 0.0028 in mean premium, so intrinsic conclusions were not driven by tie order in that stress setting.This was a targeted check, not a full deterministic retrain sweep across vocabulary sizes.Source: §B.1 · p. 22

B.2 Language-model training and evaluation configuration

• The primary LM matrix compares 182M/32K and 1.3B/32K settings, with a separate 329M/128K composition check; within each setting, architecture, optimizer, and token-step budget are matched.The corpus is fixed to the 101-language mC4 release, though raw-text exposure can differ in lower-coverage stress rows.Source: §B.2 · p. 22, Table B.3 · p. 23

B.3 Corpus and token-count accounting

• LM token counts are estimated using the same sampled shard per language for both tokenizers, while the closest tokenizer-isolation setting is 101-lang because BBPE and UBE have nearly identical raw-text coverage there.En-only and 6-lang conditions are fixed-compute stress tests with raw-text exposure differences.Source: §B.3 · p. 24

C.1 Representative language examples

• In the 6-lang/32K stress setting, UBE cuts token counts by 6–35% for the listed non-English languages outside tokenizer training, equivalent to 6–53% more fixed-budget content.The table reports mean tokens per sentence.Source: §C.1 · p. 24, Table C.1 · p. 25

C.2 Naturally occurring mixed-script documents

• Across 400 naturally occurring documents from eight languages, UBE changes total tokens by -0.2% relative to BBPE; Bengali is the only language that regresses, at +0.4%.UBE matches or slightly improves token counts in seven of the eight languages.Source: §C.2 · p. 25

C.3 Contamination checks

• The audit scanned 89.47M mC4 documents across 101 languages using exact document–sentence equality and a 50-character prefix-substring check.The 89.47M documents are the scan corpus, not the full tokenizer-training document estimate.Source: §C.3 · p. 25
• Exact-match scans found no FLORES-200, SIB-200, or MGSM overlaps and one UDHR overlap; prefix overlaps were rare for FLORES-200 and SIB-200 and absent for MGSM.UDHR had many prefix overlaps, so the authors treat it as a robustness stress test rather than a fully held-out corpus result.Source: §C.3 · p. 26, Table C.4 · p. 26
• The prefix-substring scan covered sentences at least 50 characters long: 201,686 FLORES-200, 22,133 UDHR, 40,771 SIB-200, and 2,717 MGSM sentences.Source: §C.3 · p. 26

D.1 Training config ablation

• Across nine core configurations, UBE reduces token-premium variance by 24%–49% and Gini by 6%–14%; all nine Gini intervals are below zero, while token-count tests are significant in eight.The sole token-count exception is En-only/64K; the same eight comparisons remain significant under Holm and Benjamini–Hochberg corrections.Source: §D.1 · pp. 26–27, §E.1 · p. 37

D.2.1 Why Gini rises with vocabulary size

• With BPE, script-dependent merge productivity widens premium distributions as vocabulary grows: 75 of 204 languages see premiums rise; UBE lowers the premium-curve intercept but does not remove this scaling effect.Observed ΔGini at 32K was -11.1% for UDHR, -9.4% for FLORES-200, and +0.1% for MGSM; their ordering is consistent with a first-order sketch that ranks coverage breadth, not quantitative predictions from it. Across 16K–256K, ΔGini approaches zero from -12.1% to -1.9%.Source: §D.2.1 · p. 27, §D.2.2 · p. 29

D.3 SuperBPE composition results

• Across vocabulary sizes, S-UBE has lower Gini than S-BBPE, while plain UBE uses fewer FLORES-200 tokens than S-UBE at 16K–64K.The interaction term changes sign across sizes (+4.4% at 16K to -4.4% at 256K), supporting qualitative complementarity rather than a uniform additive effect; at 64K it is -1.7% (95% CI [-3.2%, -0.1%]).Source: §D.3 · pp. 29–30

D.4 Data-to-vocabulary ratio ablation

• Full- and half-data tokenizer Gini curves nearly overlap across all four families, with median |∆Gini| of 0.0011 across 20 comparisons.The maximum difference is 0.0087, supporting the interpretation that rising Gini is driven mainly by encoding-floor asymmetry and script-dependent merge productivity rather than insufficient training data.Source: §D.4 · pp. 30–31, Figure D.2 · p. 31

D.5 Per-language distribution and mechanism diagnostics

• UBE’s largest token reductions occur in low-resource three-byte scripts, while effects for high-coverage scripts are negligible (< 1 token) at every vocabulary size.For a 24-character Ol Chiki fragment, UBE uses 39 tokens versus 63 for BBPE, a 38% reduction; recurrent regressions are concentrated in a small set, peaking at +7.4% for Tibetan at 128K.Source: §D.5 · p. 32, §D.5.2 · pp. 33–34, §D.5.1 · p. 33

D.5.3 Vocabulary utilization diagnostics

• UBE uses all 256 PUA symbols with 99.8% active utilization and has fewer dead merges than BBPE (9 vs. 13).About 23.6% of UBE merges join two pure PUA tokens, and UBE has a 17.3% depth-1 merge share versus 7.9% for BBPE, consistent with routed BMP characters needing fewer merges to reach character-level units.Source: §D.5.3 · p. 35

D.5.4 Cross-dataset robustness by vocabulary size

• UBE’s Gini advantage is generally larger on UDHR than FLORES-200 from 32K onward, while FLORES-200 shows the marginally larger reduction at 16K.The paper attributes stronger effects in byte-fallback-heavy settings to UDHR’s 387 entries and many scripts without mC4 training data.Source: §D.5.4 · p. 36

E.2 Tokenizer comparisons and composition

• SCRIPT-style has a lower overall mean than standard boundaries, driven by the South Asian cohort despite higher English and selected-BMP means; within each boundary policy, UBE’s effect varies across cohorts.Source: §E.2 · p. 38
• Under SCRIPT-style pre-tokenization, UBE lowers token-premium Gini from 0.2048 to 0.1832 and uses fewer tokens than SCRIPT-BPE in 163/204 configurations across five matched settings.UBE is lower in 169/204 FLORES-200 configurations; on XNLI, XCOPA, and MGSM natural-language fields, it is lower for all 25 Latin/Arabic/Cyrillic pairs, while SCRIPT-BPE is lower for 11/12 other-script pairs. SCRIPT-BPE jointly changes block/index representation, script/category pre-tokenization, and merge constraints; UBE changes only reversible base-symbol representation and uses neither a language identifier nor Unicode script table.Source: §E.2 · p. 38, Table E.3 · p. 39, Table E.4 · p. 39
• With SuperBPE-style boundaries, replacing BBPE with UBE lowers the selected-BMP mean by 17.2%, overall mean by 2.4%, and Gini by 9.4%.The comparison uses the same 101-language mC4 prefix and 32K vocabulary; UBE also lowers every reported quantity under either fixed boundary policy.Source: §E.2 · p. 38, Table E.5 · p. 40
• MYTE-UBE reduces the selected-BMP mean by 28.10% and total positions by 4.58% versus MYTE, using fewer positions in 201/204 configurations and more in none.It routes only residual characters through UBE, applies no learned BPE, exactly reconstructs the post-MYTE byte stream on all 206,448 FLORES-200 sentences, and matches MYTE’s released decoder output.Source: §E.2 · p. 39, Table E.6 · p. 40

E.3 Unicode fidelity

• UBE recovers every input across 1,155,724 Unicode 17 suite-level cases, whereas MYTE recovers 1,138,306.Under raw-identity evaluation, SCRIPT-BPE recovers 153,943/155,063 scalars in its released valid set; NFC changes the remaining 1,120, and its implementation filters Private-Use and unassigned scalars.Source: §E.3 · p. 39, Table E.7 · p. 40

E.4 Source-byte-matched language-model comparison

• UBE’s BPB is 0.91% below BBPE and 0.78% below BBPE16, while BBPE16 needs 18.61% more model-input training tokens for matched source-byte exposure.The three checkpoints use the same frozen corpus and training protocol, architecture, vocabulary, seed, and held-out source; task-level accuracy leaders vary. Sequence files are versioned stress tests, not conformance claims.Source: §E.4 · p. 40, Table E.8 · p. 41

E.5 Five-tokenizer language-model comparison at matched exposure

• UBE has the lowest BPB Gini and the highest multilingual and English benchmark-group means; SCRIPT-BPE has the lowest common-text BPB.The five 181,688,064-parameter models share the seed, training stream, Transformer blocks, optimizer, and evaluation suite, with training steps varied to match UBE’s supervised-byte target. MYTE+BPE and SCRIPT-BPE report representation-conditioned common-text BPB because they can canonicalize or filter inputs; direct MYTE was excluded because matching exposure would require approximately 520,943 steps, or 2.60× UBE’s 200,000-step run.Source: §E.5 · p. 41

E.6 Evaluation coverage and code-heavy inputs

• The downstream benchmarks include languages using 3-byte BMP scripts, but none includes a primary 4-byte non-BMP group; held-out BPB covers all 101 mC4 languages, including 20 primary 3-byte entries.These counts describe evaluation language entries, not the number of LM training languages.Source: §E.6 · p. 41, Table E.9 · p. 41
• The code-heavy comparison uses 100 SWE-bench tasks and ten released MAPS translations, with token counts from the primary 101-language/32K BBPE and UBE tokenizers and a same-corpus 32K BBPE16 tokenizer.The Chinese MAPS translation described in its paper was not released.Source: §E.6 · pp. 41–42
• UBE uses fewer final BPE tokens than BBPE for English and all ten MAPS translations, even when reference-edit files are 99.24% ASCII bytes.Uniform UTF-16 BBPE16 is substantially longer on these inputs.Source: Table E.10 · p. 42, §E.6 · p. 42

F.1 Full LM benchmark results

• The full tables report zero-shot accuracy across multilingual and English-only benchmarks, with variants covering 182M/32K, 1.3B/32K, 230M/64K, and 329M/128K LM settings.The 128K table includes SuperBPE composition rows; the passages do not provide cell-level comparisons.Source: §F.1 · p. 42, Table F.1 · p. 42, Table F.2 · p. 42

F.2 Multi-seed stability

• Across three seeds, the 182M/32K comparison has a mean same-seed ΔBPB of +0.0046 and a largest absolute gap of 0.0063, showing no systematic tokenizer-level quality advantage in this setting.Absolute macro BPB varies across seeds.Source: §F.2 · p. 43, Table F.5 · p. 43

F.3 Context-window and prompt-processing benchmark

• The prompt-processing benchmark reports same-content FLORES-200 bundles, and its protocol uses a single A100, batch size 1, FP32, 30 trials, and trimmed means with 95% confidence intervals.After symmetric trimming, confidence intervals use 28 effective samples.Source: Table F.6 · p. 44, §F.3 · p. 44
• The benchmark measures prompt processing because prefill depends on sequence length; it reports sequences per second to compare ingestion of the same text despite tokenizer-dependent token counts.Eight of ten language pairs are significant after false-discovery-rate correction; Chinese (Simplified) and Burmese are not.Source: §F.3 · pp. 44–45
• Amharic shows the largest observed speedup (+45.0%) and token reduction (-32.1%) in the prompt-processing benchmark.The measurements compare same-content prompts; the reported rate is sequences per second, not tokens per second.Source: §F.3 · p. 44, Table F.6 · p. 44

G Reproducibility and compute disclosure

• The paper plans to release UBE tokenizer artifacts and fidelity/parity code, but the runner, runtime, and configurations for the five-way LM comparison use private infrastructure and are excluded.The artifact card describes standard tokenizer.json files with unchanged BPE merge semantics.Source: G Reproducibility and compute disclosure · p. 45, §G.1 · p. 45

G.2 Compute disclosure

• The primary 28-run LM suite used an estimated 14.8K–27.3K A100 80 GB GPU-hours, plus less than 6 GPU-hours for downstream evaluation and inference.The estimate excludes comparisons in Tables E.8 and 6.Source: §G.2 · pp. 45–46, Table G.1 · p. 45
• The five additional runs underlying Table 6 are reported separately because available records do not establish complete accelerator-time accounting for those comparisons.Table E.8 reuses the primary UBE checkpoint and adds BBPE on H100 and BBPE16 on A100.Source: §G.2 · p. 46, Table G.2 · p. 46

H Positioning, related work, and limitations

• The comparison survey spans studies with differing evaluation scope; UBE combines tokenizer training on 101 languages, intrinsic evaluation across 204 language configurations, and 28 primary LM-training runs.The cited study summary identifies Abagyan et al.’s multilingual benchmarks as Belebele and M-MMLU, with Aya as a separate generation evaluation [2025].Source: H Positioning, related work, and… · p. 46, Table H.1 · p. 46

H.1 Public large language model tokenizer comparison

• The public-tokenizer comparison measures FLORES-200 Gini and total token count across 100K–248K vocabularies, including tiktoken-derived, Qwen-family, and Tekken designs.The Qwen 3.6 multimodal row uses measured scale 248,044 rather than its padded 248,320 vocabulary and is not directly comparable to the Qwen 2.5 text-model row.Source: §H.1 · pp. 46–47, Table H.2 · p. 47

H.2 Representative compact language models and tokenizer scales

• Public compact-model disclosures suggest a split between 32K–50K English-centric vocabularies and much larger multilingual-family tokenizers, but the survey is not a like-for-like tokenizer-training comparison.Exact tokenizer-training language counts are rarely disclosed.Source: §H.2 · p. 47, Table H.3 · p. 48

H.3 Extended related work

• Script-specific methods target particular families, while UBE uses a reversible, language-agnostic routing rule; other byte-level approaches move segmentation into the model and are generally not direct replacements for existing subword pipelines.Examples include MUTANT [2026], CBPE [2025], and grapheme-aware units [2025]; model-integrated methods include CANINE [Clark et al., 2022] and ByT5 [Xue et al., 2022].Source: §H.3 · pp. 47–48

H.4 Additional scope and limitations

• The primary evaluation relies on translated FLORES-200 text, which may contain translationese artifacts; three additional corpora provide checks without sentence alignment.Token premiums are normalized relative to English, while Gini and variance exclude the English reference row.Source: §H.4 · p. 48
• At very narrow tokenizer coverage and small vocabularies, UBE wins in only 37 of 204 languages; users should expect a reduced per-language win rate, not uniform improvement.At En-only/16K, Gini still falls 13.9%; the reported median regression at En-only/32K is 0.016 tokens per sentence.Source: §H.4 · p. 49
• The largest intrinsic reductions include Santali, Central Atlas Tamazight, and Tibetan, but these languages are absent from the 101-language mC4 LM corpus, so downstream experiments do not measure gains for those exact languages.Source: §H.4 · p. 49
• SuperBPE’s interaction is sign-changing, averaging -1.1% across vocabulary sizes, so the tested evidence supports qualitative complementarity rather than uniform additivity; UBE also forgoes some cross-script byte sharing available to BBPE.The lost sharing occasionally helps high-coverage languages such as Georgian and Amharic.Source: §H.4 · p. 49
• The intrinsic matrix uses the GPT-2 regex pre-tokenizer, and results under other boundary policies may differ; comparisons under SuperBPE- and SCRIPT-style policies do not establish invariance to all rules.UBE routes scalars rather than grapheme clusters and does not itself normalize canonically equivalent strings.Source: §H.4 · p. 49
• The matched-exposure model-level comparison with SCRIPT-BPE and derived MYTE+BPE changes multiple tokenizer-design axes and uses one seed; direct model-level comparisons with MYTE and Bit-level BPE remain future work.Source: §H.4 · p. 49
• UBE does not reduce the encoding floor for supplementary-plane characters, and robustness has not been evaluated on symbol-heavy social-media or chat domains.Four-byte UTF-8 code points remain four bytes under UTF-16 surrogate pairs.Source: §H.4 · p. 49

I.1 Existing assets and licenses

• The asset inventories identify datasets and software with their governing licenses or usage constraints, while the complete FLORES-200 table reports mean tokens per sentence for 204 language–script configurations.The table pairs UBE and BBPE rows across vocabulary sizes; displayed means are rounded to three decimals.Source: §I.1 · p. 50, Table I.1 · p. 50, Table I.2 · p. 51

• Section 4.2 summarizes held-out BPB differences by encoding-floor group, and script-family aggregates can be recomputed from the per-language tables.Source: §I.3 · p. 59
• Tables I.4 and I.5 report XNLI accuracy for 15 languages and XCOPA accuracy for 11 under the primary 101-lang/32K 182M configuration.Source: Table I.4 · p. 59, §I.3 · p. 59, Table I.5 · p. 60
• UBE wins 20/122 languages overall on Belebele.Source: §I.3 · p. 60
• Table I.6 reports held-out BPB for 101 languages; its UBE token-level perplexity is included only as a scale reference.Source: Table I.6 · p. 60, §I.3 · p. 60