PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay

Duygu Altinok

arXiv:2602.06942v1cs.CLcs.AI

TL;DR

Turkish subword research lacks systematic evidence on how vocabulary size, tokenizer-training data, and morphology interact across broad tasks. This paper jointly varies those factors and introduces fine-grained diagnostics, finding that practical mid-sized regimes balance segmentation fidelity and lemma preservation while extreme settings can collapse into near-character models.

  • Problem

    Prior Turkish studies vary tokenizer families or vocabulary sizes but generally lack data-coupled controls, fine-grained intrinsic diagnostics, and broad downstream evaluation.

  • Method

    The study jointly sweeps vocabulary and tokenizer-training corpus sizes, compares tokenizer families under matched parameter budgets, and evaluates semantic, syntactic, and morphology-sensitive behavior with detailed diagnostics.

  • Results

    Mid-sized vocabularies balance segmentation fidelity with interpretability and lemma preservation, while practical 5k–52k settings show stronger common-paradigm alignment than long-tail lemma integrity.

  • Takeaways & Limitations

    Boundary alignment, lemma integrity, and sequence-level divergence together provide an interpretable basis for choosing tokenizer regimes in Turkish.

  • Takeaways & Limitations

    Extreme 2k, 20k, and 128k regimes on Medium and Alldata become near-character models with poor precision and sequence fidelity, and are not competitive.

Abstract

from arXiv · show

Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fidelity. Prior studies have explored tokenizer families and vocabulary sizes but typically (i) vary vocabulary without systematically controlling the tokenizer's training corpus, (ii) provide limited intrinsic diagnostics, and (iii) evaluate a narrow slice of downstream tasks. We present the first comprehensive, principled study of Turkish subword tokenization; a "subwords manifest", that jointly varies vocabulary size and tokenizer training corpus size (data and vocabulary coupling), compares multiple tokenizer families under matched parameter budgets (WordPiece, morphology level, and character baselines), and evaluates across semantic (NLI, STS, sentiment analysis, NER), syntactic (POS, dependency parsing), and morphology-sensitive probes. To explain why tokenizers succeed or fail, we introduce a morphology-aware diagnostic toolkit that goes beyond coarse aggregates to boundary-level micro/macro F1, decoupled lemma atomicity vs. surface boundary hits, over/under-segmentation indices, character/word edit distances (CER/WER), continuation rates, and affix-type coverage and token-level atomicity. Our contributions are fourfold: (i) a systematic investigation of the vocabulary-corpus-success triad; (ii) a unified, morphology-aware evaluation framework linking intrinsic diagnostics to extrinsic outcomes; (iii) controlled comparisons identifying when character-level and morphology-level tokenization pay off; and (iv) an open-source release of evaluation code, tokenizer pipelines, and models. As the first work of its kind, this "subwords manifest" delivers actionable guidance for building effective tokenizers in MRLs and establishes a reproducible foundation for future research.

1. Introduction

Turkish tokenization is a core modeling choice because agglutination creates long, sparse forms and challenges vocabulary efficiency and morphological alignment. The paper addresses gaps in prior work with a controlled, broad, morphology-aware evaluation of tokenizer strategies.

  • Motivation: Turkish agglutination makes tokenization consequential because surface forms are long and sparse, while word-, character-, and standard subword approaches impose different costs.Word-level methods risk vocabulary explosion and OOV failures; character-level methods lengthen sequences; standard subwords can fragment stems or affixes.
  • Research gap: Prior Turkish studies examine tokenizer choice or vocabulary size, but generally do not systematically control tokenizer training-corpus scale.This leaves the interaction between vocabulary size, training data, and downstream success insufficiently characterized.
  • Approach: The study jointly varies vocabulary size and tokenizer training-corpus size, compares WordPiece, morphology-level, and character baselines, and evaluates semantic, syntactic, and morphology-sensitive tasks.Comparisons are conducted under matched parameter budgets.
  • Approach: A morphology-aware toolkit diagnoses boundary-level F1, lemma atomicity, segmentation errors, edit distances, continuation rates, and affix coverage.These measures are designed to explain why tokenizer choices succeed or fail beyond coarse aggregates such as fertility.
  • Contributions: The paper releases evaluation code, tokenizer pipelines, and Transformer checkpoints to support reproducibility and further Turkish NLP research.The study presents its results as a broad, evidence-grounded manifest for tokenizer design in morphologically rich languages.

2. Related Work

Earlier Turkish studies address tokenizer families, vocabulary size, or selected tasks, but do not provide a holistic, data-coupled diagnostic account. This work broadens task coverage, scale sweeps, and morphology-aware analysis to derive more actionable comparisons.

  • Prior Turkish studies: Earlier work compared BPE, WordPiece, and Unigram across Turkish corpus and vocabulary sizes, including a suffix-preserving morphosubword tokenizer.Morphosubwords kept affixes atomic while allowing roots to be learned.
  • Prior Turkish studies: Morphosubwords produced small, consistent gains over WordPiece on POS, NER, QA, and sentiment, alongside reduced pseudo-perplexity and faster convergence.Reported intrinsic trends included increasing fertility and morphology compatibility with data and vocabulary size, plus diminishing returns at larger scales.
  • Prior Turkish studies: RoBERTa-scale comparisons found WordPiece and BPE strongest overall, morphology-level tokenization slightly behind, word-level tokenization harmed by UNKs, and character-level results varying by task.These studies linked vocabulary size to downstream performance across multiple benchmarks.
  • Prior Turkish studies: Other work found larger WordPiece vocabularies improved NER and QA until saturation around 128k–256k, while sentiment did not benefit from larger vocabularies.It also examined normalization and simple morphology-injection schemes.
  • Positioning: This study extends prior comparisons with finer vocabulary sweeps, syntactic evaluations, parameter-allocation analysis, and intrinsic diagnostics linking segmentation behavior to task outcomes.The broader design includes very small vocabularies where segmentation and sequence-length pressure are especially revealing.
  • Positioning: The paper frames its contribution as the first holistic, data-coupled and diagnostic account of Turkish subword tokenization.It aims to transform fragmented observations into prescriptive tokenizer guidance.

3. Datasets

The evaluation spans controlled data and vocabulary regimes, semantic and syntax-sensitive benchmarks, and a dedicated morphology test set. These resources support comparisons of tokenizer behavior, task performance, and morphological fidelity.

  • Evaluation design: The benchmarking design varies tokenizer-training data scale, vocabulary size, and morphology-aware evaluation across controlled tokenizer variants.These three axes organize the datasets and diagnostics used throughout the study.
  • Semantic benchmarks: TrGLUE provides standardized Turkish semantic tasks covering inference, similarity, and classification for consistent representation-quality comparisons.The benchmark supplies standardized splits and evaluation metrics.
  • Benchmark coverage: The task suite includes POS tagging, NER, morphosyntactic acceptability, sentence inference, and other sentence-level semantic evaluations.POS and NER are selected because they are directly sensitive to morpheme segmentation, affix boundaries, and lemma integrity.
  • Named entity recognition: Turkish WikiNER contains approximately 20k sentences and 19 entity tags, with a BERTurk reference F1-score of 0.77 on the official test split.The split contains roughly 18k training and 1k each development and test sentences.
  • Syntax and morphology: The BOUN Treebank contains roughly 7.8k training sentences and about 1k each for development and test, with rich morphological annotation for POS, dependency, and morphology evaluation.The BERTurk baseline achieves 92.63 UPOS accuracy, 81.51 UAS, and 74.59 LAS.
  • Morphology evaluation: The dedicated morphology set pairs surface words with lemmas and suffix chains across five validation-only subsets, enabling boundary, atomicity, and segmentation-error analysis.It has no training split and is intended for test-only tokenizer assessment.

4. Tokenization Metrics

The metrics characterize tokenizer fragmentation, continuation structure, and alignment with Turkish morphemes. Together, they distinguish compression from over- or under-segmentation and assess boundary, lemma, sequence, and affix fidelity.

  • Fertility and continuation rate: Fertility measures average subwords per word, while continuation rate measures the proportion of tokens continuing an intra-word span.Fertility reflects sequence-length inflation; continuation rate captures segmentation shape within words.
  • Fertility and continuation rate: 2.17 fertility and 0.38 continuation rate illustrate a morpheme-aligned mid-granularity segmentation on a Turkish sentence.The example contains 13 subwords across six words and five continuation tokens.
  • Interpretation: Very high fertility with very high continuation indicates over-fragmentation, whereas very low values suggest whole-word memorization and possible loss of compositionality.Moderate fertility and continuation often signal morpheme-aligned segmentation.
  • Boundary and lemma metrics: Boundary micro- and macro-F1 compare predicted subword ends with gold morpheme boundaries, using precision, recall, and per-item averaging.Character offsets exclude word-start and word-end boundaries; continuation markers are removed before length calculations for WordPiece-like tokenizers.
  • Boundary and lemma metrics: Lemma boundary hit rate detects whether the lemma’s surface endpoint is preserved, while lemma single-token rate measures standalone lemma atomicity.These metrics distinguish a correct lemma endpoint from a lemma represented as one contiguous vocabulary item.

5. Pre-Transformer Tokenization Benchmarks

Character-level models provide robust tokenizer-free baselines for morphology and surface-driven tasks, but expose limitations on long-range syntax, grammatical acceptability, and graded semantics. Word-level vocabularies show diminishing returns because increased coverage does not reliably overcome sparse, morphology-fragmented representations.

  • Character-level baselines: 84.3% SST-2 accuracy and 67.1% MNLI accuracy show that character models capture substantial surface and sublexical signal.They nevertheless reach only MCC = 0.08 on CoLA and ρ = 0.12 Pearson on STS-B.
  • Character-level baselines: 0.70 F1 on NER demonstrates that character-level representations can recover entity spans without pre-tokenized inputs.The model learns patterns associated with morphology, affixation, and character shape despite longer sequences.
  • Character-level baselines: 91.56 POS accuracy, 65.19 UAS, and 57.15 LAS accompany 96.19 morphological micro-accuracy on BOUN.Compared with BERTurk, the character baseline sacrifices syntactic attachment quality while substantially improving morphological fidelity.
  • Key findings: Character models are strong on surface-driven classification and morphology, but weaker on long-distance syntax and structure- or meaning-sensitive tasks.These complementary strengths motivate hybrid designs and further subword-vocabulary exploration.
  • Word-level benchmarks: Increasing word-level coverage does not improve CoLA, whose MCC remains negative and drifts downward as more vocabulary is retained.The results attribute the bottleneck to representation limits rather than lexical coverage.
  • Word-level benchmarks: 85% SST-2 accuracy is reached by roughly 75–80% train coverage, after which larger vocabularies provide no gains.For MNLI, MRPC, and STS-B, performance similarly shows early plateaus, while morphology fragments word representations and makes additional coverage inefficient.
  • Word-level benchmarks: After 75% train coverage, BOUN word-level performance stalls at approximately 60/19/12 for POS/LAS/Morph F1, versus 91/65/96 for character-level models.The train–test coverage mismatch and sparse inflected surface forms limit generalization.
  • Explainability: Word-level explainability shows scattered CoLA cues and OOV sensitivity, whereas sentiment relies on compact polarity cues and sequence labeling favors frequent in-vocabulary anchors.These patterns connect early sentiment gains and weak morphosyntactic generalization to the available lexical inventory.

6. WordPiece Tokenization

The WordPiece study examines how vocabulary size, tokenizer training data, and morphological alignment interact across Turkish tasks. It defines research questions spanning tokenizer type, task sensitivity, corpus scale, sequence length, and morphological fidelity, using three BellaTurca corpus regimes.

  • Research questions: The study asks whether morphological alignment improves downstream performance and how tokenizer training corpus size interacts with vocabulary size.Its questions cover syntax-, morphology-, semantics-, and entity-oriented tasks.
  • Research questions: RQ1–RQ4 examine tokenizer type, task-dependent gains, corpus sizes of 5/20/80 GB, sequence length, morphological fidelity, and resulting Pareto frontiers.The agenda explicitly treats vocabulary size and training-data scale as interacting design axes.
  • Pretraining corpora: Tokenizer and Transformer pretraining use BellaTurca Turkish text from high-quality web data, books, and cleaned OSCAR web data.The corpus combines multiple genres and quality-filtered sources.
  • Pretraining corpora: Minimal, Medium, and Alldata contain approximately 5 GB, 20 GB, and 80 GB, respectively.Minimal uses filtered web data, Medium adds books, and Alldata additionally includes cleaned OSCAR.

6.2 Training WordPiece Tokenizers

WordPiece tokenizers are trained across vocabulary sizes and corpus scales, then evaluated through intrinsic coverage and morphology measures alongside downstream performance. This design directly tests data–vocabulary coupling under a controlled pipeline.

  • Training design: Each vocabulary size is trained on Minimal, Medium, and Alldata corpora of approximately 5 GB, 20 GB, and 80 GB.This creates a two-axis sweep over vocabulary size and tokenizer training corpus scale.
  • Training design: Vocabulary sizes range from 2k to 128k, spanning very small to very large WordPiece inventories.The tested sizes are 2k, 5k, 10k, 20k, 32k, 52k, and 128k.
  • Evaluation pipeline: The trained variants are analyzed for coverage, OOV rate, sequence length, and morphology alignment before relating these properties to downstream performance.Hugging Face Tokenizers, Trainer, and downstream libraries support the training and evaluation pipeline.

6.3 Tokenization Behavior Across Corpora and Vocabulary Sizes

Vocabulary size and corpus diversity jointly determine whether Turkish tokenizers fragment words, preserve morphemes, or memorize inflected forms. Mid-sized vocabularies generally provide the strongest balance between sequence efficiency, boundary fidelity, and lemma integrity.

  • 20–32k vocabularies stabilize segmentation around frequent stems and suffixes, reducing fertility to roughly 1.4–1.7 while retaining interpretable morphological units.Common suffixes and inflected forms increasingly appear as reusable subwords or whole tokens.
  • At 52–128k, fertility approaches 1.15–1.18, but memorized inflected forms can fuse morphemes and weaken compositional access.Compression improves efficiency while potentially reducing generalization to rarer inflectional combinations.
  • Very small vocabularies on large, heterogeneous corpora produce near-character segmentation, with fertility around 6.5 and almost entirely continued tokens.This regime inflates sequence length and obscures systematic morphology.
  • The practical Pareto frontier is fertility around 1.4–1.7 with continued-subword rates near 0.30–0.45, reached with larger vocabularies as corpora grow.Typical operating ranges are 20–32k for smaller or medium corpora and 32–52k for very large corpora.
  • Boundary alignment is strongest on frequent simple paradigms and weakest on Çekimli, while suffix coverage remains robust but suffix atomicity varies across vocabularies.Sequence-level exact matches remain low on Çekimli despite reasonable boundary F1.
  • Extreme fragmentation can yield boundary recall and lemma boundary rates near 1.0 while producing very low precision and CER above 0.5.Such regimes are stress tests rather than competitive configurations.
  • For morphology-sensitive applications, roughly 10k–32k settings can achieve F1 µ ≥0.60 on Çekimli and F1 µ ≥0.78 on Common Nouns.Higher lemma_single_rate and lemma_boundary_rate are preferable when lemma preservation is the priority.
  • The evaluation aligns predictions to gold morphemes at character offsets within whitespace-delimited tokens and reports micro-averaged boundary metrics across three fixed-seed runs.Confidence intervals use sentence-level bootstrap with 1,000 resamples.

6.4 Transformer Benchmarking of WordPiece Tokenizers

Transformer performance generally improves from very small vocabularies toward mid-sized or large inventories, while the best range depends on corpus scale and task. Fragmentation harms several benchmarks, whereas gains taper at the highest vocabulary sizes and optimization sensitivity remains task-dependent.

  • CoLA remains difficult, with meaningful gains mainly at larger vocabularies and higher-data regimes, reaching approximately 0.11–0.13 for Minimal and Alldata.Medium performs best around 20k–52k at approximately 0.09–0.12 before declining at 128k.
  • SST-2 improves into the mid–large range, reaching 85.4–85.8 for Minimal and 85.5–85.7 for Alldata at 52k–128k.Tiny vocabularies underfit polarity markers and idioms; 32k–52k is a practical default.
  • MNLI is robust once vocabularies reach the mid-range, with Medium and Alldata typically achieving 0.82–0.85 from 20k upward.Additional data yields limited improvement after coverage becomes adequate.
  • MRPC improves with vocabulary size but remains optimization-sensitive, with Medium peaking near 32k at 0.65 F1 and 0.57 accuracy.The authors recommend modest tuning to reach the upper range consistently.
  • STS-B is especially sensitive at small vocabularies and stabilizes by 20k–52k, where Medium and Alldata provide the strongest consistent similarity scores.A sufficiently expressive tokenizer plus moderate-to-large pretraining data is identified as the key condition.
  • Very small vocabularies of 2k–5k are weak and unstable across tasks, while 32k–52k is the most consistent overall range.Beyond 52k, gains taper and 128k is not uniformly better.
  • Fragmented settings with fertility around 3.6–6.6 and continuation around 0.69–0.98 co-occur with lower CoLA MCC, SST-2 accuracy, and less faithful explanations.Efficient-but-legible configurations instead cluster near fertility 1.4–1.7 and continuation 0.30–0.45.
  • Syntax scores improve into the 32k–52k band, while morphology micro-accuracy plateaus earlier and can soften at vocabularies of at least 128k.Minimal and Alldata peak around 52k, whereas Medium remains stable across 20k–128k.

6.5 Key Findings

For BERT-style Turkish models, mid–large WordPiece vocabularies trained on mixed-domain data provide the strongest balance among sequence length, morphology visibility, and downstream performance. Smaller vocabularies over-fragment inputs, while larger vocabularies obscure morphological cues.

  • Vocabulary size vs. task class: Mid–large WordPiece vocabularies around 32k–52k offer the best overall balance for Turkish Transformer performance.They improve POS/LAS while maintaining strong semantic-task results.
  • Vocabulary size vs. task class: NER favors slightly higher vocabularies because stable segmentation of names and numbers matters more than fine-grained morpheme boundaries.This explains NER’s preference toward the upper part of the effective range.
  • Training data scale × vocabulary size: Scaling tokenizer training data from 5→20→80 GB improves robustness and reduces variance when vocabulary size is 32k–52k.Additional data does not overcome over-fragmentation with tiny vocabularies or lost morphological legibility with huge vocabularies.
  • Pareto region of tokenization: The efficient operating region combines moderate fertility of ≈1.4–1.7, continuation of ≈0.30–0.45, compact sequences, and strong POS/LAS and SST-like results.This trade-off deteriorates beyond the 32k–52k band as morphology erodes.
  • Practical prescription: The practical default is mixed-domain WordPiece in the 32k–52k band, adjusted downward for grammar-sensitive tasks and upward for NER-heavy applications.Morphology-aware constraints or entity-stem protection can further adapt the tokenizer.
  • Explainability alignment: Attribution analyses align with these results: mid–large vocabularies focus saliency on predicate morphology and syntactically diagnostic spans while keeping entities intact.Tiny vocabularies diffuse saliency across long subword chains, whereas huge vocabularies hide inflectional cues.

7. Optimal Ways of Tokenizing Turkish

Turkish tokenization methods trade off coverage, efficiency, morphological fidelity, and downstream accuracy. No single approach dominates, but mid–large WordPiece vocabularies provide the most dependable general-purpose balance.

  • Character-level tokenization: Character tokenization eliminates OOV tokens and preserves inflectional material, but produces severe fragmentation, long sequences, slower training, and diffuse attributions.With ample compute, it can perform well on morphology-sensitive tasks.
  • Word-level tokenization: Word-level tokenization keeps sequences short and entities stable, but high OOV rates and sparse unseen-inflection statistics make Turkish morphology opaque.It can remain competitive on NER with careful OOV handling but underperforms on syntax-oriented tasks.
  • Morphology-aware subwords: Morphology-aware subwords align units with stems and affixes, improving boundary precision/recall, lemma integrity, and morph-sensitive task performance.They depend on linguistic resources and may over-segment frequent forms without frequency signals.
  • WordPiece: WordPiece provides a tunable middle path: roughly 32k–52k vocabularies on mixed-domain data keep sequences compact while supporting strong syntax, morphology, and semantic performance.Very small vocabularies over-fragment, whereas very large vocabularies over-merge and obscure morphology.
  • Overall comparison: No tokenizer dominates every task; morphology-aware subwords maximize linguistic fidelity, while mid–large WordPiece offers the most dependable accuracy–efficiency–portability trade-off.Smaller units suit parsing and acceptability, whereas larger units suit NER.

8. Conclusion

The study finds that mid–large WordPiece vocabularies trained on mixed-domain data best balance Turkish modeling efficiency and linguistic fidelity. It recommends task-specific adjustments and presents the evaluation framework as a reusable template for other morphologically rich languages.

  • Conclusion: Mid–large WordPiece vocabularies trained on mixed-domain data provide the most reliable accuracy–efficiency trade-off, with morphology-visible fertility and continuation.They preserve cues important for POS, dependency, morphology, and acceptability while remaining strong on semantic and NER benchmarks.
  • Conclusion: Morphology-aware subword schemes can improve morphology-sensitive tasks and interpretability, but require linguistic resources.Character and word tokenization instead favor robustness and speed, respectively, while underperforming on other dimensions.
  • Practical implications: The practical recommendation is WordPiece in the 32k–52k band, using smaller vocabularies for grammar-centric pipelines and larger ones for NER-heavy applications.The framework links fragmentation, morphology alignment, attribution, and downstream accuracy.
  • Availability and ethics: The study’s datasets, including pretraining and benchmarking suites, are openly available under permissive licenses suitable for commercial use.The paper also states that its research complied with relevant ethical and legal requirements.

Appendix 2. Comprehensive Morphology Diagnostics by Vocabulary Size

The appendix organizes morphology-aware tokenization diagnostics across vocabulary sizes and corpus scales. Its tables use consistent abbreviations for subword, lemma, and exact-match measures.

  • Minimal corpus: Table 13 reports morphology-aware tokenization diagnostics across vocabulary sizes for the minimal corpus.The table includes C.Noun, C.Verb, Sw/W, LSingle, LBoun, and ExMatch measures.
  • Medium corpus: Table 14 reports the same diagnostic family across vocabulary sizes for the medium corpus.Sw/W denotes Subwords per Word, while LSingle, LBoun, and ExMatch denote lemma and exact-match measures.
  • Alldata corpus: Table 15 reports morphology-aware tokenization diagnostics across vocabulary sizes for the alldata corpus.C.Noun and C.Verb abbreviate Common Noun and Common Verb.
Loading 2602.06942v1…