Source-linked AI summary
Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
Saketh Reddy Vemula, Sandipan Dandapat, Dipti Misra Sharma, Parameswari Krishnamurthy
TL;DR
The relationship between tokenizer algorithm, morphological alignment, tokenization quality, and downstream performance remains unclear, especially for morphologically complex languages. The authors evaluate subword-, hybrid-, morphemic-, and word-level tokenizers with 8.5-million-parameter BERT models across Telugu, Hindi, and English, using Telugu gold segmentations for approximately 600 derivational and 7000 inflectional words. Unigram-based tokenizers consistently outperform BPE across most Telugu settings, while morphological alignment independently predicts structure-prediction performance but has weaker effects on other tasks.
Problem
The relationship between tokenizer algorithm, morphological alignment, tokenization quality, and downstream performance remains unclear, especially for morphologically complex languages.
Method
The authors evaluate subword-, hybrid-, morphemic-, and word-level tokenizers with 8.5-million-parameter BERT models across Telugu, Hindi, and English, using Telugu gold segmentations for approximately 600 derivational and 7000 inflectional words.
Results
Unigram-based tokenizers consistently outperform BPE across most Telugu settings, while morphological alignment independently predicts structure-prediction performance but has weaker effects on other tasks.
Takeaways & Limitations
Tokenizer algorithm choice is a stronger performance factor than morphological alignment alone, while morphology-aware pre-segmentation substantially benefits BPE but not Unigram.
Takeaways & Limitations
The findings are limited to small encoder-only BERT models, three languages, and NLU tasks, so generalization to larger models, other architectures, languages, and generative tasks is uncertain.
Abstract
from arXiv · showhide
The relationship between tokenizer algorithm (e.g., Byte-Pair Encoding (BPE), Unigram), morphological alignment, tokenization quality (e.g., compression efficiency), and downstream performance remains largely unclear, particularly for languages with complex morphology. In this paper, we conduct a comprehensive evaluation of tokenizers using small-sized BERT models -- from pre-training through fine-tuning -- for Telugu (agglutinative), along with preliminary evaluation in Hindi (primarily fusional with some agglutination) and English (fusional). To evaluate morphological alignment of tokenizers in Telugu, we create a dataset containing gold morpheme segmentations of 600 derivational and 7000 inflectional word forms. Our experiments reveal two key findings for Telugu. First, the choice of tokenizer algorithm is the most significant factor influencing performance, with Unigram-based tokenizers consistently outperforming BPE across most settings. Second, while better morphological alignment shows a moderate, positive correlation with performance on text classification and structure prediction tasks, its impact is secondary to the tokenizer algorithm. Notably, hybrid approaches that use morphological information for pre-segmentation significantly boost the performance of BPE, though not Unigram. Our results further showcase the need for comprehensive intrinsic evaluation metrics for tokenizers that could explain downstream performance trends consistently.
1 Introduction
The paper asks whether morphologically aligned tokenization better handles morphologically complex languages, focusing on Telugu and comparing tokenizer strategies across model evaluation stages.
- Language-specific linguistic cues are presented as increasingly relevant for efficient, compact models in low-resource and morphologically complex settings.The paper frames morphology and syntax as potential resources for improving performance and generalizability.
- Morphologically rich languages generate many surface forms per lemma, increasing type-to-token ratios, data sparsity, and modeling difficulty.Agglutination and fusion attach grammatical markers such as tense, number, case, and person to roots, often producing longer and more unique words.
- The paper investigates whether morphological alignment, tokenization quality, or tokenizer choice better explains downstream performance in complex languages.The motivating question is whether morphologically aligned approaches handle grammatical complexity more effectively.
- The study evaluates character-, subword-, hybrid-, morphemic-, and word-level approaches using small BERT models for Telugu, with preliminary comparisons in Hindi and English.Each tokenizer variant is evaluated through pre-training, fine-tuning, and multiple benchmarks; Telugu is emphasized because of its highly agglutinative word formation.
2 Related Work
Related work studies subword tokenization, morphological boundary alignment, and intrinsic tokenizer metrics, but no single metric reliably explains downstream quality.
- Common subword algorithms include BPE, the Unigram Language Model, and WordPiece, with later methods targeting statistical effectiveness or morpheme-boundary alignment.
- Intrinsic tokenizer evaluations measure properties such as compression efficiency, cognitive plausibility, and morphological alignment.
- Existing intrinsic evaluation methods have not produced a single reliable explanation of tokenizer quality or strong, consistent correlation with downstream performance.
3 Evaluating Tokenization Approaches
The study compares tokenizer variants under controlled small-BERT experiments across languages and downstream tasks, with Telugu results emphasizing Unigram and morphology-aware BPE behavior.
- Evaluating Tokenization Approaches: The framework varies character-, subword-, hybrid-, morphemic-, and word-level tokenization while holding vocabulary limits and model comparisons controlled.Hybrid tokenizers first apply Morfessor or morphological-analyzer segmentation, then train a subword tokenizer on the segmented text.
- Experimental Settings: The experiments pre-train 72 encoder-only BERT models at 8.5 million parameters and evaluate classification, structure prediction, similarity, and natural-language-inference tasks.Models are trained across tokenizer variants, with downstream evaluations drawn from GLUE, IndicGLUE, IndicXTREME, and additional curated datasets.
- Results and Observations: Naive Unigram tokenizers consistently achieve the best overall Telugu performance with significant margins across most variants and downstream tasks.Naive BPE gains are marginal relative to character- or word-level tokenizers, and naive BPE performs worse in text classification at larger vocabularies.
- Results and Observations: Morfessor pre-segmentation substantially improves BPE in text classification and structure prediction, whereas comparable gains for hybrid Unigram appear only at smaller vocabulary sizes.
- Results and Observations: BPE performs best at lower vocabulary sizes, while Unigram peaks at higher vocabulary sizes; linguistically informed improvements are more consistent at smaller sizes.
- Results and Observations: Comparisons across Hindi and English have reduced statistical power because morphological-analyzer pre-tokenizer variants were omitted, so fixed-effects findings are Telugu-only and exploratory.
4 Morphological Alignment
Morphological alignment is moderately associated with downstream performance, but tokenizer algorithm choice has a substantially stronger effect across tasks. Alignment independently predicts structure-prediction performance after controlling for tokenizer factors, motivating further analysis of tokenization quality.
- Evaluation: The Telugu evaluation used MorphScore against gold morpheme boundaries, supported by a dataset of approximately 600 derivational and 7000 inflectional word forms.MorphScore compares character-level boundary positions and computes recall and precision, excluding single-token words and words without gold morpheme boundaries.
- Evaluation: The analysis examined downstream performance against morphological alignment, Corpus Token Count, and normalized Rényi entropy across tokenizer variants and vocabulary sizes.The supplied figures cover alignment by vocabulary size, CTC across Telugu, Hindi, and English, and Rényi entropy across those languages.
- Results & Discussion: Morphological alignment showed a statistically significant but moderate positive relationship with downstream performance, strongest for structure prediction.Spearman correlations were 0.486 (p = 0.041) for recall and 0.474 (p = 0.047) for F1-score; structure prediction reached r = 0.478 (p = 0.045) for recall.
- Results & Discussion: Tokenizer algorithm had a very large, statistically significant effect across almost all tasks, while morphological alignment alone could not explain the broad performance differences.For text classification, the tokenizer factor yielded F = 276.82 (p < 0.001); the pre-tokenizer had no significant effect.
- Results & Discussion: After controlling for tokenizer and pre-tokenizer, MorphScore F1 remained a significant predictor for structure prediction but not for other tasks.ANCOVA reported F = 5.71 (p = 0.033) for structure prediction, while fixed-effects coefficients were 9.182 for precision (p = 0.046) and 13.148 for F1-score (p = 0.033).
5 Tokenization Quality
Neither Corpus Token Count nor Rényi entropy significantly explains downstream performance in the evaluated small-sized BERT models, whereas tokenizer type has a major effect.
- 5.1 Corpus Token Count (CTC): No statistically significant correlation was found between Corpus Token Count and performance across tasks in Telugu, Hindi, or English.Pearson and Spearman correlations between CTC and performance were very weak and statistically insignificant.
- 5.1 Corpus Token Count (CTC): Corpus Token Count measures the number of tokens required to encode text, with lower values commonly assumed to indicate better compression and downstream performance.The analysis does not support that assumption for small-sized BERT models at 8.5M scale.
- 5.2 Rényi Entropy: Rényi entropy was evaluated on five million pre-training sentences using α = 2.5, the setting previously reported as most correlated with performance.Despite this setup, the study found no statistically significant direct correlation with downstream performance.
- Tokenizer type significantly affects performance, while Rényi entropy alone does not explain the observed trends.Pearson and Spearman tests found weak, statistically insignificant relationships between Rényi entropy and performance; ANOVA identified tokenizer type as significant.
6 Discussion
Unigram-based tokenizers consistently outperform BPE in small-scale encoder-only BERT models, while morphological pre-tokenization benefits BPE but not Unigram.
- Unigram-based tokenizers consistently outperform BPE for small-scale encoder-only BERT models.The precise reasons for Unigram’s advantage remain unclear, and intrinsic metrics such as CTC and Rényi entropy do not explain it.
- Morphologically informed pre-tokenization significantly boosts BPE performance, but the same benefit is not observed for Unigram tokenizers.
- Morphological alignment moderately and significantly correlates with text classification and structure prediction performance but does not explain variance across all tasks.Algorithmic design and vocabulary configuration also play significant roles.
7 Conclusion
The study systematically evaluates tokenization strategies across Telugu, Hindi, and English, finding that morphological alignment correlates positively with downstream effectiveness and motivating broader intrinsic evaluation.
- The evaluation covers tokenization strategies for Telugu, Hindi, and English, emphasizing Telugu as an agglutinative language.
- Morphological alignment has a positive correlation with downstream tokenizer effectiveness.
- The findings highlight the need for comprehensive intrinsic tokenizer metrics that account for multiple trade-offs.
Limitations
The experiments are limited to small encoder-only BERT models, three languages, and natural language understanding tasks, constraining generalizability.
- Results may not generalize beyond encoder-only BERT architectures or models larger than 8.5 million parameters.
- Using only Telugu, Hindi, and English limits conclusions about the broader diversity of morphologically complex languages.
- Because evaluations focus on natural language understanding, tokenizer effects on generative tasks may differ.The paper specifically identifies text summarization and machine translation as examples requiring further evaluation.
Ethical considerations
The study uses publicly available corpora while acknowledging potential web-source biases, and aims to support more equitable NLP for morphologically complex, underresourced languages.
- Publicly available corpora were used to train the models, but their web-based sources may contain biases.
- The Telugu gold-standard morphological dataset is intended to support more equitable and effective language models.
- Pre-training 72 models and conducting over 2,160 fine-tuning runs incurred substantial computational and environmental costs.The authors describe this extensive experimentation as a necessary trade-off for robustness and validity.
A.1 Pre-training Corpus Statistics
The study characterizes training corpora and evaluates tokenizer morphology through boundary-based MorphScores across languages and downstream tasks.
- A.1 Pre-training Corpus Statistics: Table 4 summarizes corpus statistics for languages used to train tokenizers and pretrain language models, including TTR, MATTR, and MLW.Metrics marked with † use a 100 million character subset, while ⋆ identifies IndicCorp data.
- A.2 Morphological Alignment: Tables 5–7 report MorphScores for tokenizer variants across vocabulary sizes in Telugu, Hindi, and English.
- A.2 Morphological Alignment: MorphScore compares tokenizer-produced character boundaries with gold morpheme boundaries using precision, recall, and F1-score.The evaluation converts both segmentations into boundary-index sets, then aggregates metrics by micro-averaging over non-excluded words.
- A.2 Morphological Alignment: False positives represent over-segmentation, whereas false negatives represent missed gold boundaries and under-segmentation.
- A.3.1 Tasks Description: The evaluation spans classification, structure prediction, question answering, and natural language inference tasks across languages.Reported task examples include Word and Definition, Word and Morphology, Word and Word, sentiment analysis, NER, and paraphrase classification.