PaperlayerView exact revision on arXiv

Source-linked AI summary

What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?

Saketh Reddy Vemula, Parameswari Krishnamurthy

arXiv:2608.17325v1cs.CL

TL;DR

Fixed tokenizers are learned independently of language models, leaving it unclear which token structures joint optimization would produce across diverse languages. This paper compares tokenizer-free and fixed-tokenizer approaches across 18 languages and finds that SSLMs learn morphologically aligned, contextually efficient segments, while H-Nets prioritize byte-level efficiency.

  • Problem

    It is unclear whether fixed tokenizers provide the best performance across languages with diverse morphological complexities, typologies, and scripts.

  • Method

    The paper analyzes SSLM and H-Net learned tokens across 18 diverse languages, compares them with BPE and ULM, and evaluates downstream task performance.

  • Results

    Jointly learned tokenization fundamentally changes token structure: SSLMs learn morphologically aligned, contextually efficient segments, while H-Nets prioritize byte-level efficiency and remain below 0.1 morphscore-F1 across languages.

  • Takeaways & Limitations

    Tokenizer-free approaches can learn linguistically meaningful and effective vocabularies despite substantially different structures from standard tokenization approaches.

  • Takeaways & Limitations

    Downstream evaluation covered only English, Hindi, and Telugu, so generalizability across the full 18-language set remains untested.

Abstract

from arXiv · show

Tokenization is a fundamental component of language modeling pipelines. Despite its importance, it is often fixed, even though it significantly impacts model performance across languages. In this work, we analyze what tokens are learned when tokenization is jointly optimized with language modeling. We compare tokenizer-free approaches such as SSLMs and H-Nets with fixed tokenizers across 18 typologically and script-diverse languages. Our results show that joint optimization fundamentally alters token structure. SSLMs recover morphologically aligned and contextually efficient tokens, whereas H-Nets prioritize byte-level efficiency, producing longer tokens with very low overlap with standard subword vocabularies. We further show that tokenization behavior varies across language typologies. Agglutinative languages exhibit more dynamic segmentation patterns while learning. Through downstream evaluation, with pretrained-then-finetuned BERT models, we find that SSLM-based pretokenization consistently reduces language modeling perplexity and achieves competitive downstream performance despite distinct vocabularies. Overall, tokenizer-free approaches optimize for contextual and computational efficiency rather than strict morphological structure, resulting in fundamentally different yet effective vocabularies for downstream NLP.

1 Introduction

Tokenization is typically learned independently of language-model training, leaving its effectiveness across morphologically and typologically diverse languages unclear. This work investigates jointly learned tokenization through tokenizer-free models, compares it with fixed-tokenizer methods, and evaluates learned vocabularies and downstream effects.

  • Motivation: Fixed-tokenizer approaches learn tokenization separately from language-model training and keep it static, making their suitability across diverse languages unclear.The concern spans languages with different morphological complexities, typologies, and scripts.
  • Motivation: No single tokenization approach is universally best because conclusions depend on language properties, model architecture, dataset size, and other hyperparameters.The passage explicitly identifies these factors as sources of variation in comparative conclusions.
  • Approach: Tokenizer-free architectures jointly perform data-dependent boundary prediction and byte-level language modeling in an end-to-end pipeline.Dynamic Token Pooling Transformers and H-Nets are cited as examples, with H-Nets described as stable.
  • Contributions: The study analyzes SSLM- and H-Net-learned tokens across diverse languages, compares their vocabularies with BPE and ULM, and evaluates downstream task performance.These contributions target intrinsic and linguistic properties, vocabulary similarity, and whether jointly learned tokenizers improve downstream results.

2 Related Work

Related work has examined token learning in tokenizer-free language models and the downstream effects of fixed-tokenizer algorithms. However, tokenizer-free analyses remain narrow in language coverage, while cross-language comparisons often omit dataset-size and language-disparity confounds.

  • Tokenizer-free language models: Prior tokenizer-free work tracked SSLM subword learning through morphological alignment, productivity, idiosyncrasy, and fertility, identifying distinct learning stages.The study by Meyer and Buys (2025) analyzed token-learning dynamics from a linguistic perspective.
  • Tokenizer-free language models: That tokenizer-free analysis covered only three languages and did not account for dataset size or disparities across languages in cross-language conclusions.These limitations constrain the interpretation of its findings across languages.
  • Fixed tokenizers: Fixed-tokenizer research has mainly evaluated how subword algorithms affect downstream language-model performance, with ULM reported to perform better and recover more morphologically aligned subwords.The cited comparisons include Bostrom and Durrett (2020) and Vemula et al. (2025).

3 Methodology

The methodology compares tokenizer-free and fixed-tokenizer approaches across 18 typologically and script-diverse languages using controlled monolingual data. It evaluates token properties and downstream BERT performance through standardized training, validation, testing, and parameter settings.

  • Language and Data Selection: The study covers 18 languages spanning diverse morphology, typology, and scripts.The languages are listed in Table 1, with a detailed version in Appendix Table 11.
  • Language and Data Selection: The experiments use nonparallel monolingual corpora with controlled data sources and byte-premium-scaled sizes across languages.The primary source is WMT News Crawl with the News domain, falling back to NLLB where unavailable; English data are anchored at 250,000 sentences.
  • Compared Approaches: Tokenizer-free methods include SSLMs, which marginalize over segmentations, and H-Nets, which jointly predict boundaries and perform byte-level language modeling.Fixed-tokenizer comparisons include BPE, ULM, WPC, SaGe, Morfessor, and inference-modified methods.
  • Evaluation: Evaluation measures contextual exponence, effective vocabulary size, morphological alignment, and downstream performance across multiple NLP tasks.The downstream study pretrains 12M-parameter BERT models and fine-tunes them for Sentiment Analysis, POS Tagging, NER, and Dependency Parsing in English, Hindi, and Telugu.
  • Experimental Protocol: Training uses train, validation, and test subsets to isolate tokenizer effects, with approximately 3M parameters for H-Nets and 2M for SSLMs.All tokenization approaches are trained on the train subset, monitored on validation, and evaluated on test.

4 Experiments & Results

Jointly optimized tokenizers learn language- and script-dependent structures: SSLMs develop morphologically aligned, contextually efficient tokens, while H-Nets favor byte-level efficiency and diverge from fixed subword vocabularies. Across evaluations, tokenizer-free approaches reduce perplexity and remain competitive downstream despite distinct vocabularies.

  • Morphological alignment: SSLM morphological alignment follows four phases—rapid evolution, fluctuation, convergence, and saturation—with larger fluctuations and lower final alignment in agglutinative languages.Templatic and fusional languages achieve higher alignment overall; Tamil converges near F1≈0.3, whereas English stabilizes near ≈0.5.
  • Morphological alignment: Jointly learned tokenizers adapt segmentation to morphology, sometimes sacrificing alignment after initially discovering accurate morpheme boundaries.Tamil reaches perfect alignment at step 450 but later converges to an inaccurate split at step 7150; English moves from “rea ping” at step 2200 to “reap ing” at step 2950.
  • Script and contextual efficiency: Latin-script languages reach effective vocabularies of 14,000–15,500 tokens and low contextual exponence, whereas non-Latin languages use smaller vocabularies with higher contextual exponence.The results suggest larger dedicated units in Latin scripts but smaller, flexible morphemic units in languages such as Tamil and Hindi.
  • Tokenizer comparison: H-Nets prioritize byte-level efficiency, showing morphscore-F1 below 0.1 across languages, while SSLMs trade computational efficiency for higher linguistic fidelity.H-Nets also produce the longest tokens in non-Latin languages at ≈17–18.5 characters, compared with ≈2.5–5.0 characters for SSLMs and most fixed tokenizers.
  • Tokenizer comparison: Tokenizer-free approaches yield higher fertility but lower contextual exponence than fixed-tokenizer baselines, with SSLM and H-Net averaging fewer than 75 neighboring tokens.The fertility effect is strongest for H-Nets in agglutinative languages, reaching ≈5.0–5.5, while analytic and fusional languages remain below ≈2.0.
  • Downstream evaluation: Tokenizer-free approaches achieve the lowest validation perplexity across the three evaluated languages and remain downstream-competitive despite fundamentally distinct vocabularies.SSLM-ULM reaches 90.17% F1 for Telugu POS tagging; only Hindi shows SaGe outperforming tokenizer-free methods, with perplexity near 1.

5 Conclusion

Jointly learned tokenization changes token structure across diverse languages: SSLMs favor morphologically aligned, contextually efficient segments, while H-Nets favor byte-level efficiency and produce substantially different structures. Although morphology-aware pretokenization remains strongest overall, SSLM-based pretokenization remains effective.

  • SSLMs learn segments that are more morphologically aligned and contextually efficient than those of standard frequency-based tokenizers.
  • H-Nets prioritize byte-level efficiency and produce substantially different token structures from other approaches.
  • Morphology-aware pretokenization remains strongest overall, while SSLM-based pretokenization remains effective.

Limitations

The study’s conclusions are limited by relatively small model and dataset scales, restricted downstream language coverage, and incomplete hyperparameter exploration for tokenizer-free language models.

  • Model and dataset sizes: The analysis uses relatively small models and datasets, including 3M-parameter tokenizer-free LMs, 250,000-sentence datasets, and 12M-parameter BERT models with 10M-sentence datasets.The broad comparison across tokenization approaches and languages constrained experimental scale.
  • Generalizability of downstream results: Downstream evaluation covers only English, Hindi, and Telugu because pretraining every tokenization–language combination across all 18 languages was computationally costly.The selected languages provide a controlled evaluation across three typologically distinct cases.
  • Hyperparameters of tokenizer-free LMs: Tokenizer-free LM hyperparameters, such as SSLM maximum token length, were fixed using prior-work heuristics rather than systematically swept because such experiments require substantial computation.This leaves the effects of untested hyperparameter settings unresolved.

Ethics Statement

The study analyzes tokenization across languages using publicly available text corpora, without human subjects or sensitive personal data. Dataset quality and linguistic coverage may nevertheless bias observed tokenization behavior.

  • The work does not involve human subjects or sensitive personal data.
  • All experiments use publicly available text corpora.
  • Differences in dataset quality and linguistic coverage across languages may bias observed tokenization behavior.

A Ablation Study: Model Scale

The scale ablation compares BPE and SSLM-BPE BERT models at approximately 2M, 12M, and 30M parameters using the same data size. SSLM-BPE consistently lowers validation perplexity across scales, although its gains saturate as models grow, while further scaling of slower tokenizer-free models is left for future work.

  • A Ablation Study: Model Scale: The ablation evaluates BPE and SSLM-BPE at approximately 2M, 12M, and 30M parameters while holding the data size fixed.The evaluation uses approximately 10M sentences and limits the analysis to these two tokenization approaches.
  • A Ablation Study: Model Scale: SSLM-BPE consistently achieves lower validation perplexity than BPE across all evaluated model sizes.The results indicate that aligning tokens through SSLM pretokenization provides more efficient language modeling.
  • A Ablation Study: Model Scale: As model size increases, SSLM-BPE’s perplexity improvement over BPE saturates.The saturation suggests diminishing gains from the pretokenization advantage at larger model scales.
  • A Ablation Study: Model Scale: Further scaling analysis is deferred because the tokenizer-free models require approximately 10× more computation than transformers.The authors leave scaling analysis that accounts for this computational difference to future work.

B Hyperparameters & Experimental Setup · B.1 H-Nets · B.2 SSLMs

The experimental setup specifies compact, separately configured pretraining pipelines for H-Nets and SSLMs. H-Nets use hierarchical compression and scheduled optimization, while SSLMs use a lightweight decoder with 2,105,836 parameters; downstream comparisons evaluate BPE against SSLM-BPE pretokenization.

  • B.1 H-Nets: H-Nets use 256-dimensional embeddings and hidden states at both hierarchical levels, with the bottom level omitting its FFN block.
  • B.1 H-Nets: H-Net pretraining uses a cosine learning-rate scheduler with a peak base learning rate of 3 × 10−4.
  • B.1 H-Nets: H-Nets apply 10% linear warmup, early stopping with patience 5, and weight decay 0.01 to regularize optimization.
  • B.1 H-Nets: H-Net structural compression remains active throughout training, with ratio-loss scale 1.0 and zero warmup compression epochs.
  • B.2 SSLMs: SSLM pretraining uses a lightweight Transformer decoder tailored to subword segmental modeling, with 2,105,836 total parameters.
  • B.2 SSLMs: The SSLM architecture reduces layers, embedding dimensions, and feed-forward dimensions to support fast, efficient training with limited parameters.
  • B.2 SSLMs: Downstream BERT comparisons evaluate BPE and SSLM-BPE tokenizers across sentiment analysis, POS tagging, NER, and dependency parsing.
  • B.2 SSLMs: Scores average three finetuning runs with seeds 42, 43, and 44, reporting accuracy, F1, and LAS for the respective tasks.

B.3 Fixed-tokenizer Variants

Fixed-tokenizer variants were standardized across strategies, with vocabulary sizes and implementation settings chosen to support fair comparison. The configurations included standard subword, adaptive BPE, and morphologically constrained tokenizers.

  • Global Settings: All tokenizers used target vocabulary sizes of 5,000, 10,000, or 20,000, with SentencePiece character coverage fixed at 100%.The 10,000-vocabulary configuration was selected for direct comparison with SSLMs initialized with a 10,000-item lexicon.
  • Standard Subword Architectures: WordPiece used NFKC normalization, lowercasing, whitespace pre-tokenization, “##” continuation markers, and five enforced special tokens.The special tokens were [UNK], [CLS], [SEP], [PAD], and [MASK].
  • SuperBPE: SuperBPE set its stage-transition limit to 4,000 for a 10,000-token vocabulary and 10,000 for a 25,000-token vocabulary.These settings follow the stated relationship T = t × 2.5.
  • Morphologically-Informed Tokenizers: MorphBPE, MorphULM, and MorphWP constrained subword training to boundaries predicted by an unsupervised Morfessor Baseline model.Raw texts were pre-tokenized at predicted morphological seams before training the BPE, Unigram, or WordPiece variants.

B.4 Downstream Evaluation

Downstream evaluation pretrains BERT models and then fine-tunes them on each task. The evaluation includes POS tagging on English, Telugu, and Hindi Universal Dependency treebanks, with additional configurations and analyses documented in the accompanying tables and figures.

  • Evaluation Setup: BERT models are pretrained before being fine-tuned separately for each downstream task.Pretraining configurations are listed in Table 8, while fine-tuning hyperparameters are listed in Table 9.
  • Downstream Tasks: POS tagging assigns grammatical categories to words using both word definitions and sentence context.Each word receives the class corresponding to its first subword.
  • Downstream Tasks: POS tagging uses Universal Dependency treebanks for English, Telugu, and Hindi.The datasets are en_ewt, te_mtg, and hi_hdtb, respectively, accessed through the HuggingFace universal dependency dataset.
  • Evaluation Configuration: The section documents model-scale pretraining configurations and task-wide fine-tuning hyperparameters in Tables 8 and 9.Table 8 covers three BERT scales, while Table 9 summarizes fine-tuning settings across downstream tasks.
Loading 2608.17325v1…