Source-linked AI summary
SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
Iaroslav Chelombitko, Ekaterina Chelombitko, Aleksey Komissarov
TL;DR
Subword tokenizer evaluation for morphologically rich Uralic languages is limited by scarce clean morpheme lexicons and unresolved vocabulary-size trade-offs. The paper introduces SampoNLP’s corpus-free lexicon-refinement pipeline and IPS-based BPE evaluation, finding language-specific optimal ranges and low performance ceilings for Finnish and Estonian.
Problem
Clean morphological resources are scarce, while the vocabulary size that provides robust morphological representation in Uralic-language tokenizers remains under-explored.
Method
SampoNLP uses corpus-free MDL-inspired Self-Referential Atomicity Scoring to refine candidate lists, while IPS balances morpheme coverage against over-splitting in BPE evaluation.
Results
Optimal vocabulary ranges are 80k-128k for Hungarian and Estonian and 80k-150k for Finnish, while standard BPE reaches a strikingly low performance level for Finnish.
Takeaways & Limitations
BPE has language-dependent morphological performance, with Hungarian reaching max ∼0.73 IPS, Finnish ∼0.31, and Estonian ∼0.39, and gains diminishing beyond each language’s sweet spot.
Takeaways & Limitations
IPS abstracts away qualitative segmentation differences, and standardized corpora do not reflect noise in real-world data.
Abstract
from arXiv · showhide
The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons. We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings. Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the "elbow points" of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP
1 Introduction
The paper addresses the need for clean morphological resources and principled vocabulary-size evaluation for BPE in morphologically rich Uralic languages. It introduces SampoNLP and evaluates Finnish, Hungarian, and Estonian tokenizers using IPS.
- BPE tokenization is challenging for Uralic languages because words often concatenate many morphemes, making tokenization important for representing grammatical structure.
- High-purity morphological evaluation resources are scarce because dictionary combinations are noisy, manual curation does not scale, and corpus-based methods suit low-resource settings poorly.
- SampoNLP uses corpus-free, MDL-inspired Self-Referential Atomicity Scoring to distinguish simple from composite forms through internal structural patterns.
- The study evaluates BPE tokenizers for Finnish, Hungarian, and Estonian across vocabulary sizes from 8k to 256k.
- The Integrated Performance Score (IPS) balances Lexical Morpheme Coverage against Over-Split Rate to identify effective vocabulary ranges.
- The contributions comprise an automatic reproducible corpus-free lexicon-refinement toolkit and a quantitative analysis of vocabulary size, coverage, and over-segmentation.
2 Related Work
Related work spans statistical and rule-based morphological analysis, tokenization, and intrinsic evaluation. SampoNLP extends these lines with corpus-free candidate filtering and IPS-based vocabulary-size recommendations.
- Existing tokenization methods such as BPE and Unigram are statistically driven and often fail to align with morpheme boundaries in morphologically complex languages.
- Corpus-based morphological discovery includes branching-entropy methods and MDL-based systems such as Morfessor, which learn from corpus statistics.
- SampoNLP differs by filtering candidate lists through internal structure in a corpus-free, type-only regime suited to resource-scarce scenarios.
- FST resources such as Omorfi and GiellaLT are linguistically comprehensive, whereas the IMDP pipeline distills a minimal reference lexicon from dictionary-based candidate lists without corpus frequencies.
- IPS responds to calls for intrinsic, linguistically informed tokenizer evaluation beyond downstream task performance.
- The work builds on earlier high-quality data sources and provides concrete optimal vocabulary sizes that prior research had indicated but not determined.
3 Methodology. The IMDP Pipeline
The Iterative Morphological Decomposition Pipeline creates high-purity morpheme lexicons automatically from candidate lists without corpus frequencies. Its core method iteratively scores candidate fundamentalness and then applies automated thresholding.
- IMDP operates in a corpus-free, type-only regime and requires only a candidate list as input.
- MDL-inspired Self-Referential Atomicity Scoring evaluates each candidate’s fundamentalness relative to the entire candidate set.
- The pipeline has three stages: pre-filtering and initial scoring, iterative score refinement, and final filtering via automated thresholding.
3.1 Stage 1: Candidate Pre-filtering and Initial Scoring
Stage 1 removes nonlinguistic and weakly supported candidates, then assigns initial atomicity scores. The resulting pool is intended to retain structurally recurrent forms for later processing.
- Initial scoring: Initial processing reduces nonlinguistic noise and establishes a baseline score for each plausible candidate.
- Hard pre-filtering: Hard filters discard candidates with non-target scripts, non-alphabetic characters, proper-noun or acronym patterns, or unsuitable lengths.
- Type-support filtering: Type-support filtering retains a candidate only when it appears as a substring in at least m = 3 other unique candidates.
- Type-support filtering: The retained candidates form the final candidate pool C for subsequent processing.
- Initial scoring: Initial atomicity scoring assigns S0(t) = 1/|t|, making shorter forms more likely to receive higher baseline scores.
3.2 Stage 2: Iterative Score Refinement
Stage 2 iteratively refines each token’s atomicity score by comparing its own score with the strongest explanation from smaller-token decompositions. The process supports multi-part segmentations, prevents degenerate one-character decompositions, and continues until scores stabilize.
- Iterative score refinement: Each token’s score is recalculated from its explainability by other tokens in the candidate set.The refinement is iterative and self-referential.
- Optimal decomposition: The best explanation is the decomposition maximizing the sum of prior-iteration scores for its constituent tokens.The maximum is computed with dynamic programming and is called Best Explanation Power, BEP.
- Decomposition constraints: The search permits segmentations into any number of components rather than restricting decompositions to pairs.
- Decomposition constraints: Length-one segments are considered only when they belong to the whitelist W, preventing degenerate decompositions.
- Score update: A token is penalized only when its decomposition-based evidence exceeds its evidence for being atomic.This update preserves the token’s score when BEP does not exceed its current atomicity score.
- Convergence: Atomic morphemes retain high scores while composite words are iteratively pushed toward zero, and refinement stops when maximum score changes fall below a small threshold.
3.3 Stage 3: Final Filtering via Automated Thresholding
Stage 3 separates atomic from composite candidates after score convergence by applying Otsu’s data-driven thresholding method. Candidates meeting the resulting threshold form the final high-purity morpheme lexicon.
- Score distribution: After convergence, composite candidates typically have very low scores while atomic candidates retain higher scores.
- Automated thresholding: Otsu’s method selects threshold τ by maximizing inter-class variance between the atomic and composite score classes.The approach adapts to each dataset’s score distribution without manual parameter tuning.
- Final filtering: Tokens with final score Sfinal(t) >= τ are classified as atomic and included in the final high-purity morpheme lexicon.
4 Experimental Setup
The experiments create reference morpheme lexicons, train BPE tokenizers, and evaluate vocabulary-size effects for Finnish, Hungarian, and Estonian. IPS combines morpheme coverage with over-segmentation to compare tokenizers and identify elbow points.
- Experimental design: The evaluation analyzes BPE tokenizers for Finnish, Hungarian, and Estonian across vocabulary sizes from 8k to 256k.
- Data: The methodology uses raw morpheme-candidate lists for lexicon cleaning and large text corpora for tokenizer training.The corpora are not used in the morpheme-list refinement pipeline.
- Data: Candidate lists come from Hunspell-based open-source dictionaries, with dedicated Finnish resources for compounds and LibreOffice dictionaries for Hungarian and Estonian.
- Reference lexicons: The IMDP produces high-purity Finnish, Hungarian, and Estonian reference lexicons after iterative filtering and automated Otsu thresholding.
- Evaluation metrics: The IPS addresses the trade-off between representing morphemes and avoiding excessive fragmentation in a single evaluation score.It is based on lexical morpheme coverage and over-split rate, with the ideal tokenizer at Coverage=1 and OverSplit=0.
- Evaluation metrics: Lexical Morpheme Coverage measures reference morphemes represented as single vocabulary tokens, while Over-split Rate measures those always split into multiple pieces.
- Evaluation metrics: Optimal vocabulary sizes are identified from elbow points on the IPS-versus-vocabulary-size curve, where additional vocabulary yields diminishing returns.Higher IPS indicates a better balance between morpheme representation and avoiding fragmentation.
5 Results and Analysis
IPS curves across Finnish, Estonian, and Hungarian show diminishing returns as vocabulary size increases, with distinct performance tiers and language-specific optimal ranges. Hungarian performs best, Estonian occupies a middle tier, and Finnish remains the most challenging for standard BPE.
- General Observation: A Clear Trade-off Profile: IPS curves for all three languages follow logarithmic growth, with rapid early gains and progressively smaller improvements at larger vocabulary sizes.The pattern indicates a trade-off between morphological coverage and the increasing cost of model size.
- Cross-Linguistic Analysis: Three Distinct Performance Tiers: Hungarian achieves the highest IPS, rising from 0.29 to a maximum of 0.73.This suggests BPE is reasonably effective at learning Hungarian morphological regularities.
- Cross-Linguistic Analysis: Three Distinct Performance Tiers: Finnish reaches a maximum IPS of only 0.31, quantitatively demonstrating the difficulty of capturing its morphological complexity with standard BPE.Finnish has the lowest maximum IPS among the three evaluated languages.
- Identifying the Optimal Vocabulary Range (k*): Hungarian and Estonian share a recommended vocabulary range of k=80,000–128,000, bounded by the elbow and 90% quality points.For both languages, expanding beyond this range yields only minimal additional gains for Hungarian, while the range is identified as the best compromise between performance and size for Estonian.
- Identifying the Optimal Vocabulary Range (k*): Finnish requires the larger recommended range k=80,000–150,000 because reaching 90% of its maximum IPS requires k=150,000.The elbow remains at k=80,000, but the maximum IPS is comparatively low.
6 Conclusion
The paper introduces SampoNLP for corpus-free morphological resource creation and uses its lexicons to evaluate BPE vocabulary sizing in Uralic languages. It recommends language-specific vocabulary ranges while showing that standard BPE remains limited for highly agglutinative Finnish.
- SampoNLP uses MDL-inspired Self-Referential Atomicity Scoring to refine noisy candidate lists into clean morpheme lexicons.The toolkit is designed for corpus-free, reproducible resource creation.
- BPE evaluation identifies optimal vocabulary ranges of 80k–128k for Hungarian and Estonian and 80k–150k for Finnish.These ranges represent the reported trade-off between performance and model size.
- Finnish exposes severe limitations of standard BPE because performance plateaus at a strikingly low level.
- SampoNLP and the generated morpheme lists are released to support reproducible research and development of morphologically aware tokenization methods.
Discussion
The results show that BPE effectiveness varies substantially across languages and that each language has a vocabulary-size sweet spot with diminishing returns. Finnish remains especially challenging, and the paper frames its resources as a basis for developing alternatives beyond vocabulary optimization.
- Hungarian reaches a maximum IPS of ∼0.73, compared with ∼0.31 for Finnish and ∼0.39 for Estonian.
- Finnish has the lowest IPS plateau, consistent with consonant gradation and stem alternations that make orthographic segmentation less stable for BPE.
- The paper defines optimality as morphological sufficiency: capturing productive language structure with minimal redundancy through a language-level rather than task-specific criterion.
- The IPS metric abstracts away qualitative segmentation differences, and standardized corpora do not represent noise in real-world data.
- The study suggests that optimizing k may be insufficient for Finnish and underscores the need for morphologically aware tokenization methods.
A Appendix
Table 3 reports detailed BPE results across vocabulary sizes for Finnish, Estonian, and Hungarian using morpheme coverage and over-split rate measures.
- Table 3 compares BPE tokenizers across vocabulary sizes for three Uralic languages using Morpheme Coverage and Over-Split Rate.Morpheme Coverage is LMC; Over-Split Rate measures reference morphemes supported in W but never appearing as single tokens.