Latent Core Tokenizer: Compress, but Meaningfully

1 Introduction

• Frequency-driven vocabulary learning can favor high-resource language patterns, while morphology-preserving units need not be the units that optimize compression.Multilingual languages differ in corpus size, script, and morphological structure.Source: §1 · p. 1
• LCT first identifies reusable latent structure with minimum description length, branching entropy, and morphotactic refinement under language-balanced evidence, then uses it to construct the surface vocabulary.The latent pieces guide vocabulary learning but need not correspond to the final tokens consumed by the model.Source: §1 · p. 1
• Across 104 languages, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and Parity-Aware BPE while maintaining comparable cross-lingual tokenization-cost disparity.Under controlled encoder pretraining, aggregate downstream scores improve by 1.48, 1.83, and 2.00 points over BPE, Unigram, and Parity-Aware BPE, respectively.Source: §1 · p. 1
• The paper reports that controlled multilingual encoder pretraining extends LCT’s tokenizer improvements to downstream representations.The contributions describe evaluation across 104 languages and downstream pretraining under controlled conditions.Source: §1 · pp. 1–2

2 Latent Core Tokenizer

• LCT’s latent pieces guide vocabulary learning but are not labels or additional inputs presented to the encoder; structural discovery and surface packing determine building blocks and embedding-slot combinations separately.A productive stem can influence vocabulary construction even when a frequent whole word containing it is emitted as one token.Source: §2 · p. 2
• Language frequencies are rescaled to approximately equal aggregate mass, not equal vocabulary quotas, so pieces reused across languages can still receive more support.The MDL segmenter trades the cost of storing pieces against corpus savings from reuse; it resegments weighted words iteratively, for up to eight epochs or until analyses stop changing.Source: §2 · pp. 2–3
• Branching entropy proposes internal boundaries, and a morphotactic model refines latent pieces into prefix-, stem-, suffix-, or noise-like roles without gold morphological annotations.The method applies bidirectional boundary signals and a PRE/STM/SUF grammar with NON as a noise state; NON fragments are merged into a neighbor.Source: §2 · pp. 2–3
• Surface packing repeatedly merges the adjacent pair with the greatest weighted token savings until the exact vocabulary budget is filled.Supported characters and latent pieces seed the vocabulary; deterministic tie-breaking and recorded merge ranks make the fitted vocabulary reproducible.Source: §2 · pp. 2–3
• At runtime, minimum-token dynamic programming selects known surface pieces, while UTF-8 byte tokens provide reversible coverage for unseen strings.Source: §2 · p. 2

3 Experimental Setup

• Tokenizer and encoder variants are compared using the same multilingual corpus, vocabulary budget, and encoder architecture.Source: §3 · p. 3

3.1 Multilingual Corpus

• The corpus combines English from FineWeb with 103 languages from filtered FineWeb2 subsets and applies temperature-based language sampling to reduce high-resource dominance while preserving the available-data distribution.Documents are selected deterministically, duplicates are removed, and language quotas are drawn from shared training and validation budgets; the corpus is held fixed across tokenizer arms.Source: §3.1 · pp. 3–4

3.2 Evaluation

• Evaluation compares LCT with BPE, Unigram, and Parity-Aware BPE using intrinsic tokenizer metrics and four downstream benchmarks: XNLI, Belebele, XStoryCloze, and PAWS-X.Intrinsic measures include fertility, MorphScore F1, compression rate, type-token ratio, vocabulary utilization, and Gini; each tokenizer’s encoder uses the same architecture and training budget.Source: §3.2 · p. 4

4.1 Intrinsic Evaluation

• At 200k, LCT-MEM has the highest compression rate (4.866 bytes per token), vocabulary utilization (25.26%), and MorphScore F1 (0.5209), plus the lowest fertility (1.999 tokens per word).Its Cost Gini (0.1341) is effectively tied with BPE (0.1342); Parity-Aware BPE is stronger only on Rényi efficiency.Source: §4.1 · p. 4, Table 1 · p. 5

4.1.1 Ablation Study

• At both 200k and 256k vocabulary sizes, LCT-MEM with τ = 0.2 ranks first and τ = 0 ranks second; LCT-MEM has the strongest MorphScore, while LCT-MDL variants yield the shortest sequences.The two criteria are weighted equally as a design choice rather than derived from the data.Source: §4.1.1 · p. 6
• Increasing LCT vocabulary size from 200k to 256k lowers fertility and raises MorphScore, while Gini changes less and BPE ranks best for Gini among shown configurations at 256k.The ablation varies language-frequency weighting, four structural variants, and two vocabulary sizes.Source: §4.1.1 · pp. 5–6

4.2 Extrinsic Evaluation

• Balancing strength improves LCT-MDL by +0.80 points and LCT-MEM by +1.21 points; at τ = 0.2, LCT-MEM improves by +0.65 points over LCT-MDL in the raw paired test.The downstream comparisons use a 200k vocabulary and one pretrained encoder per configuration.Source: §4.2 · p. 7, Table 4 · p. 7
• LCT-MDL gives the shortest sequences but omits branching-entropy search and morphotactic induction; LCT-MEM performs best downstream with a more complex construction pipeline.The selected configurations test whether structural components improve representations beyond shorter sequences alone.Source: §4.1.1 · p. 6, §4.2 · p. 7
• On Belebele, LCT-MEM is effectively tied with BPE in the smallest pretraining-corpus quartile (−0.07 points), but gains +2.45, +2.58, and +1.95 points in the remaining quartiles.The authors identify data availability as a separate bottleneck for languages with the least pretraining data.Source: §4.2 · p. 7

5.1 A controllable tokenizer design space

• LCT makes corpus-size influence and structural preservation separate design controls, turning vocabulary allocation into a tunable decision.Temperature τ controls how strongly corpus size shapes multilingual evidence, while the discovery objective controls structural information preserved before surface packing.Source: §5.1 · p. 7
• Partial balancing at τ = 0.2 improves both LCT-MDL and LCT-MEM, whereas complete equalization at τ = 0 is not consistently better than BPE.With only two pretrained temperatures, the results identify a direction rather than an optimum.Source: §5.1 · p. 7

5.2 Different objectives support different deployment priorities

• LCT-MEM has stronger morphological alignment and downstream performance, while LCT-MDL produces shorter sequences; LCT-MEM’s intrinsic gains are not explained by compression alone.MorphScore stays higher across resource groups, while fertility gains are clearer for low-resource languages and smaller relative to BPE in mid- and high-resource groups.Source: §5.2 · p. 7
• Shorter sequences reduce encoder computation, but serving-latency claims require compiling the lossless packing algorithm into an optimized runtime.The current Hugging Face adapter is a Python slow tokenizer.Source: §5.2 · pp. 7–8

5.3 Allocation parity is not outcome parity

• Downstream differences are relatively small for low-resource languages and clearer for mid- and high-resource groups, where LCT-MEM at τ = 0.2 performs strongest.The authors suggest structural units and vocabulary allocation are most useful when the encoder has sufficient pretraining data to learn from them.Source: §5.3 · p. 8
• Better vocabulary allocation does not guarantee stronger outcomes for languages with very little training text, whose encoders may lack evidence to exploit improved pieces.The authors call for directly optimizing low-resource outcomes rather than treating lower allocation or Cost Gini as sufficient.Source: §5.3 · p. 8

6 Related Work

• Prior multilingual work studies language-specific allocation, cross-language overlap, and tokenization-efficiency disparities; Parity-Aware BPE changes BPE merge selection to reduce compression disparities [Foroutan et al., 2026].LCT also reduces corpus-size influence but separates balancing and latent discovery from surface vocabulary construction.Source: §6 · p. 8
• Morphological approaches use boundary constraints, segmented data, or transformations [Vemula et al., 2025; Gazit et al., 2025], while MorphBPE constrains final BPE segmentation.LCT instead treats structural analysis as an intermediate latent representation and may recombine its pieces for the final vocabulary; MDL has also been used for reusable subword discovery.Source: §6 · p. 8
• Prior work finds that compression, morphological alignment, and downstream performance need not favor the same tokenizer [Vemula et al., 2025].LCT builds on this distinction by separating structural discovery from surface vocabulary construction and evaluating tokenizer quality and learned multilingual representations.Source: §6 · p. 8

7 Conclusion

• LCT-MEM at τ = 0.2 improves aggregate downstream performance by 1.48, 1.83, and 2.00 points over BPE, Unigram, and Parity-Aware BPE, respectively.The conclusion also reports improved intrinsic tokenizer quality and smaller downstream gains for languages with the least pretraining data; the shortest-sequence configuration is not the strongest for downstream representations.Source: §7 · p. 8

Limitations

• All seven encoders were pretrained once, so reported variance reflects fine-tuning variation only and part of the 1.48-point BPE improvement may reflect pretraining variance.Multiple pretraining seeds would provide a stronger estimate of the tokenizer effect.Source: Limitations · pp. 8–9
• Experiments use DeBERTa-v3 XSmall encoders, so observed effects are established at that scale but may not transfer unchanged to larger models.Larger models may better recover from suboptimal segmentation.Source: Limitations · p. 9
• Because 62 of 79 evaluated languages appear only in Belebele, aggregate results are heavily influenced by that benchmark.The other listed benchmarks cover fewer languages, and XStoryCloze has 360 training examples; broader multilingual evaluation is needed.Source: Limitations · p. 9
• Only two temperatures and two structural objectives were taken to encoder pretraining from 48 intrinsically evaluated LCT configurations, so downstream results identify directions rather than an optimum.The intrinsic-to-downstream relationship is assessed on a much smaller subset of the design space.Source: Limitations · p. 9
• Downstream experiments use one DeBERTa-v3 XSmall architecture, leaving generalization to other encoders and decoder-only models unestablished.Holding architecture constant controls it as a source of variation.Source: Limitations · p. 9

Ethical Considerations

• The experiments use existing multilingual corpora and public evaluation benchmarks, without human participants or newly collected personal data.Source: Ethical Considerations · p. 9

A.1 LCT algorithm

• LCT builds an analyzer from language-balanced evidence, then seeds and fills a fixed surface vocabulary through weighted adjacent-pair merges.The algorithm retries packing with support threshold 1 if threshold 10 cannot fill the exact budget.Source: §A.1 · p. 11

A.1.1 Lossless runtime

• Lossless runtime encoding preserves whitespace and punctuation, backoffs pruned pieces to known vocabulary pieces, and optimizes each protected segment for minimum-token exact encoding.Ties prefer fewer byte IDs, learned pieces, and longer pieces; UTF-8 byte fallback ensures a complete path.Source: §A.1.1 · pp. 11–12

A.1.2 Worked example with a dummy corpus

• The dummy-corpus example uses entropy and morphotactics to identify stem–suffix structure, then packs frequent adjacent pieces into shorter surface tokens.Language balancing gives the smaller corpus multiplier three, so walk and play each contribute 12 effective observations; the first additional slot creates walking.Source: §A.1.2 · pp. 12–13

• Figure 2 presents per-language pretraining statistics.Source: Figure 2 · p. 14

C.1 Encoder

• Each tokenizer trains a DeBERTa-v3-style XSmall encoder with 12 discriminator layers, hidden size 384, six attention heads, intermediate size 1,536, and a maximum sequence length of 512.A six-layer generator corrupts 15% of eligible tokens for replaced-token detection.Source: §C.1 · p. 15, Table 5 · p. 15

C.2 Tokenizers

• The tokenizer systems share a 200,024-identifier budget, byte-level pre-tokenizer, encoder corpus, and 41,347,986 lines of tokenizer training text, differing in learned-piece selection.The comparison includes BPE, Unigram, parity-aware BPE, and LCT objectives.Source: Table 6 · p. 15, §C.2 · pp. 15–16

D Results

• Figure 3 presents intrinsic evaluation results for the 200k and 256k vocabulary configurations.Source: Figure 3 · p. 16

E Intrinsic evaluation by language

• Tables 8 and 9 report intrinsic evaluation by language, covering fertility, MorphScore, Gini, compression rate, type-token ratio, vocabulary utilization, and Rényi measures.Source: Table 8 · p. 17, Table 9 · p. 18

F Per-language transfer

• Figure 5 aggregates accuracy across low-, mid-, and high-resource language tertiles for XNLI, Belebele, XStoryCloze, and PAWS-X, averaged over five fine-tuning seeds.Table 10 reports per-language test accuracy across evaluated languages and describes paired differences against BPE over the five shared seeds.Source: Table 10 · p. 21, Figure 4 · p. 22, Figure 5 · p. 23