Source-linked AI summary
MoirfEolas and CríochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology
Jane Adkins, Abigail Walsh, Brian Davis, Elaine Uí Dhonnchadha
TL;DR
Tokenization can create efficiency and representation challenges for low-resource, morphologically complex languages such as Irish. This paper introduces Irish-specific morphological resources and an alignment metric, evaluates common tokenizers, and finds that Unigram aligns most often with Irish morphology while revealing trade-offs with compression and vocabulary efficiency.
Problem
Low-resource, morphologically complex languages can suffer tokenization disparities and inadequate morphological coverage, motivating better Irish-specific evaluation resources.
Method
The paper constructs MoirfEolas, defines CríochScore, and evaluates six tokenization algorithms across vocabulary sizes using Irish corpus data and intrinsic metrics.
Results
Unigram LM with an 8k vocabulary achieved the highest overall CríochScore at 40.55%, while larger vocabularies generally aligned less often with morphological boundaries.
Takeaways & Limitations
Unigram implementations, particularly with smaller vocabularies, produce the most morphologically aligned Irish tokenizations, with trade-offs involving compression and vocabulary efficiency.
Takeaways & Limitations
SentencePiece-Unigram LM was evaluated only up to a 43k vocabulary because that was the maximum available for the corpus.
Abstract
from arXiv · showhide
This paper presents new tokenization resources for Irish and evaluation measures of alignment with the morphological boundaries of the language. We present MoirfEolas, a dataset of over 35,000 Irish words mapped to their respective eclipses, prefixes and suffixes as well as an evaluation metric CríochScore, that evaluates the alignment of tokenizations with the morphological boundaries present in MoirfEolas. We evaluate common tokenization algorithms using CríochScore as well as intrinsic metrics present in the tokenization literature. We find that the Unigram Language Model aligns with Irish morphology more often than the other algorithms evaluated. We also find trade-offs between morphological-alignment of tokenization with both compression as well as vocabulary efficiency, providing practical insights for Irish natural language processing development. This dataset contributes towards combating the Irish language's low-resource status; moreover, the construction process reported in this paper can be emulated by other languages to create specialised morphological resources.
1 Introduction
Tokenization is a crucial preprocessing choice whose alignment with morphology matters especially for low-resource, morphologically complex languages such as Irish. The paper develops Irish-specific resources to evaluate this alignment.
- Tokenization converts data into discrete units for large language model input, and algorithm choice can affect downstream performance.
- Morphologically complex languages may have lexical variation that tokenizer vocabularies do not adequately capture.
- Low-resource languages face vocabulary bottlenecks that produce longer tokenizations, increasing context-window, cost, processing-time, and storage pressures.
- The paper creates MoirfEolas and CríochScore to evaluate tokenization alignment with Irish morphology.The resources map Irish words to morphological components and assess whether token boundaries align with them.
2 Background
The background contrasts common subword algorithms and intrinsic tokenization metrics, while reviewing Irish tokenization research and the trade-offs between segmentation, compression, and vocabulary efficiency.
- Tokenization algorithms: BPE and WordPiece construct vocabularies through character-merging processes, whereas Unigram LM iteratively prunes subword units using likelihood estimates.
- Tokenization algorithms: SentencePiece provides language-independent tokenization and implements both BPE and Unigram LM.
- Irish tokenization: Irish tokenization research has largely used word- and rule-based tools, with newer transformer work examining subword methods.
- Evaluation: Intrinsic evaluation assesses token quality, but intrinsic metrics are not always indicators of downstream performance.
- Intrinsic metrics: Corpus Token Count measures compression, while fertility measures tokens per word and can be higher when complex morphology requires more segmentation.
- Morphological metrics: MorphScore, Suffix Recall, and Suffix Precision evaluate how tokenizer segments align with morphological units, especially affixes.
3 MoirfEolas
MoirfEolas combines Irish corpus data with handcrafted and UniMorph morphological lists to identify eclipses, prefixes, and suffixes. Cleaning and manual inspection produce a dataset of over 35,000 words.
- MoirfEolas is built from the Irish side of ParaCrawl V9 together with handcrafted and UniMorph lists of eclipses, prefixes, and suffixes.The source corpus contains 57,587 sentences and 1,447,047 space-separated words.
- A greedy-longest-match procedure identifies morphological components within each unique corpus word.
- The resulting dataset contains over 35,000 words, including 216 unique prefixes, 178 suffixes, and 13 eclipses.
- English entries and incorrect mappings were removed using Irish morphological tools and manual inspection.More than 15,000 words were removed, with hyphen variation in eclipses a common source of incorrect mappings.
4 Cr´ıochScore
CríochScore measures whether tokenizer segments align with Irish eclipses, prefixes, and suffixes. It assigns component-level scores, normalizes them within each word, and averages word-level scores.
- CríochScore measures alignment between tokenization boundaries and Irish morphological components identified in MoirfEolas.
- A completely and independently segmented component receives 1, while a segmented but further-split component receives 0.5.
- A component that is not completely separated from the stem receives a score of 0.
- Each word-level score is normalized by the number of evaluated components, and the tokenizer’s overall score averages the word-level CríochScores.
5 Evaluation
The evaluation tests six tokenization algorithms across four vocabulary sizes on Irish data, combining CríochScore with intrinsic metrics for morphology and tokenizer behavior.
- Six tokenization algorithms are evaluated with CríochScore across vocabulary sizes of 8k, 16k, 32k, and 64k.The evaluated systems include Unigram LM, SentencePiece Unigram LM, WordPiece, BPE, SentencePiece BPE, and Byte-level BPE.
- The experiments use the Irish side of the cleaned ParaCrawl V9 corpus, selected as the largest para…
- Intrinsic evaluation includes Corpus Token Count, Rényi Efficiency, Fertility, and prefix-, suffix-, and eclipsis-level recall and precision.These metrics were adapted from previous work on Turkish to evaluate Irish tokenization behavior.
- MorphScore was considered but excluded because available Irish entries were limited and prefix labels were incomplete.
- SentencePiece Unigram LM had a maximum available vocabulary size of 43k for this corpus.
6 Results and Discussion
Unigram-based tokenizers generally align best with Irish morphology, especially at smaller vocabulary sizes, while intrinsic metrics reveal trade-offs between segmentation, compression, and vocabulary efficiency.
- 6.1 CríochScore: 40.55% was the overall CríochScore for 8k Unigram LM, the highest alignment with Irish morphological boundaries.SentencePiece-BPE at 16k followed with 40.39%, while 64k BPE had the lowest score at 9.6%.
- 6.1 CríochScore: Unigram performed relatively well across morphological categories and handled words containing multiple evaluated components best.BPE at 8k outperformed it specifically in prefix handling.
- 6.1 CríochScore: All eclipsis-only cases received 50% CríochScore because 13 words paired eclipses with hyphens or apostrophes that split the forms at punctuation.
- 6.2 Intrinsic Metrics: Larger vocabularies reduce corpus token count, fertility, Rényi Efficiency, and affix- or eclipsis-level precision by storing more whole-word tokens.The resulting higher average token lengths indicate fewer segmentations as vocabulary size increases.
- 6.2 Intrinsic Metrics: Unigram has higher fertility and lower average token length than the other algorithms, but its higher corpus token count creates a compression trade-off.Its lower Rényi Efficiency also indicates a trade-off with vocabulary efficiency relative to BPE varieties.
- 6.3 Absorption Analysis: Eclipses show near-perfect recall but near-zero precision, indicating that they are usually present in vocabularies but rarely segmented independently.Suffixes similarly show high recall but low precision, while prefixes generally have lower recall and higher precision than suffixes.
- 6.3 Absorption Analysis: SentencePiece implementations and 64k WordPiece and BPE absorb 100% of eclipses into tokens with subsequent characters.
- 6.3 Absorption Analysis: Unigram-based tokenizers appear more appropriate for Irish morphological alignment because suffix material is frequently merged with preceding subwords and Irish has extensive suffixation.For example, splitting labhraíonn as ‘labhraí’ ‘onn’ fails to align with the suffix boundary.
7 Conclusion
The paper introduces Irish morphology resources and finds that smaller-vocabulary Unigram implementations align most closely with morphological boundaries, while exposing trade-offs requiring downstream study.
- MoirfEolas and CríochScore provide resources for evaluating Irish tokenization boundaries against morphological structure.The dataset-construction approach is presented as a model for languages with similar morphological forms.
- Unigram implementations, particularly with smaller vocabularies, produce the most morphology-aligned vocabularies and smaller tokens than merge-based methods.
- The evaluation identifies trade-offs between morphological alignment, compression, and vocabulary efficiency for Irish NLP development.
- Downstream evaluation is necessary to verify the practical insights and the role of morphological alignment in improving performance.
A.1 Eclipses Rules in the Irish Language
Irish eclipsis adds a consonant to a word stem according to phonological rules and occurs before specific character sequences in defined grammatical contexts.
- Eclipsis adds a consonant to the stem before specific characters according to Irish phonological rules.
- Eclipsis occurs in contexts including certain prepositions, numbers, and possessive determiners.
A.2 Absorption Analysis Results
The absorption analysis distinguishes complete merging, partial merging, and mapping-based alignment for eclipses, prefixes, and suffixes.
- Absorption: Absorption occurs when a morphological component merges into a token with preceding or subsequent word characters.For example, a prefix is absorbed when tokenization yields fragments such as ‘neamhsp’, ‘leách’ or ‘neamhsple’, ‘ách’.
- Absorption: Split absorption occurs when a component is segmented but part of it remains merged with neighboring characters.For example, suffix ‘áíonn’ may be tokenized as ‘labhra’, ‘áíonn’ or as smaller fragments such as ‘labhráio’, ‘nn’.
- Eclipsis mapping: Eclipsis mapping records whether tokenization follows the language-specific mapping between an eclipsis and its subsequent character.For eclipsis ‘g’ in ‘gcríochnaithe’, mapping is present when tokenization begins with ‘gc’.
A.3 MorphScore Unsuitability
The paper rejects MorphScore for Irish because its data are limited and its lemma–stem procedure and three-subword structure misrepresent Irish morphology.
- Data limitations: MorphScore was not implemented because its Irish data contain only 2,576 entries, compared with over 35,000 in MoirfEolas.The paper identifies the limited Irish coverage as one of three main reasons for excluding MorphScore.
- Stem-based segmentation: MorphScore derives subwords around a stem shared by the word and lemma, but this produces morphologically inaccurate boundaries in many Irish words.The procedure places preceding and following subwords around the calculated stem, which may not correspond to actual prefixes, suffixes, or eclipses.
- Subword capacity: MorphScore allows at most three subwords, whereas the Irish morphology examined sometimes requires four components.The paper gives words containing an eclipsis, prefix, and suffix as examples requiring four subwords.
- Eclipses: In ‘dtugtar’, MorphScore segments ‘dtug’ ‘ta’ ‘r’, while MoirfEolas segments ‘d’ ‘tug’ because the calculated stem incorrectly identifies ‘ta’.The incorrect stem is caused by the irregular verb’s lemma differing substantially from its surface form, causing the eclipsis to be absorbed into the leading characters.
- Prefixes and suffixes: In ‘comhionannais’, MorphScore yields ‘comhionanna’ ‘is’, while MoirfEolas separates the prefix, stem, and suffix as ‘comh’ ‘ionann’ ‘ais’.The lemma–stem procedure misses the compound’s prefix and the suffix’s effect on the word’s morphology.
- Multiple components: For ‘t-idirghabhálaí’, MorphScore captures only ‘t-’ and the remainder, while MoirfEolas represents four morphological subwords.The MoirfEolas segmentation is ‘t-’ ‘idir’ ‘ghabhal’ ‘aí’, reflecting the eclipsis, prefix, stem, and suffix.