PaperlayerView exact revision on arXiv

Source-linked AI summary

SuTRA : Structurally-Unified Tokenization with Root Awareness

Vaibhav Rathore, Siddhant Gole, Dadhichi Telwadkar, Rooshil Bhatia, Maulik Ruparel, Siddharth Surekha, Neha Bhargava

arXiv:2608.18087v1cs.CLcs.AI

TL;DR

Subword tokenizers can fragment roots, affixes, and Indic aksharas because they optimize statistical compression without modeling morphological structure. SuTRA incorporates script and morphological constraints, reducing shattering while improving semantic recoverability, including a +34% relative gain in Hindi Linear R2 over Bpe.

  • Problem

    Existing tokenizers overlook morphological structure and can split Indic aksharas and multimorphemic words in ways that misalign roots and affixes.

  • Method

    SuTRA integrates akshara-aware grouping, morphology-aligned merge scoring, and root-preserving boundary constraints into frequency-based subword learning.

  • Results

    SuTRA reduces Morphological Shattering and improves morphological alignment, semantic recoverability, and downstream translation, including a +34% relative gain in Hindi Linear R2 over Bpe.

  • Takeaways & Limitations

    The results support lightweight morphological priors as a practical way to improve tokenizer interpretability and utility for morphologically rich languages.

  • Takeaways & Limitations

    Evaluation fixes vocabulary size at 32k, leaving SuTRA’s behavior at larger vocabulary regimes unexplored.

Abstract

from arXiv · show

Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.

1. Introduction · 2. Related Work · 3. Morphological Dataset Preparation

The paper presents SuTRA, a morphology-aware extension of BPE designed to reduce Morphological Shattering in Indic languages, alongside a gold-standard morphological segmentation dataset. It situates this approach against statistical, character-level, and prior morphology-aware tokenization methods, then describes a hybrid dataset-construction pipeline.

  • 1. Introduction: Statistical tokenizers such as BPE, WordPiece, and Unigram optimize compression but ignore linguistic structure, often overriding morphological boundaries.This mismatch motivates the paper’s focus on morphology-aware tokenization.
  • 1. Introduction: Morphological Shattering occurs when subword segmentation misaligns multimorphemic words with lexical roots and affixes, especially in morphologically rich Indic languages.Indic abugida scripts represent orthographic syllables as aksharas, making structural violations particularly consequential.
  • 1. Introduction: Indic tokenization also produces semantic blindness: root semantics become harder to recover, while embeddings become overly sensitive to orthographic changes that preserve morphemes.Higher token fertility, known as the Indic Tax, further intensifies Morphological Shattering.
  • 1. Introduction: SuTRA extends BPE with script-aware akshara grouping, morphology-aligned merge scoring, and a root-preserving boundary constraint that penalizes splits across valid morphological units.The method combines frequency-based merging with lightweight linguistic priors.
  • 2.1. Frequency-Driven Tokenization: Prior frequency-driven methods maximize statistical coverage but systematically provoke Morphological Shattering, while BPE-Dropout and Flota provide only auxiliary mitigation or post-hoc realignment.These approaches do not directly correct the underlying tokenization process.
  • 2.2. Character-level and Token-free Architectures: Tokenization-free and character-level models such as Canine, ByT5, and Charformer avoid explicit subword shattering by learning representations directly from bytes or characters.Their principal drawback is a Compute Penalty caused by character-level processing.
  • 2.3. Morphological Challenges in Indic LLMs: Indic MRLs face high fertility and poor semantic anchoring; traditional BPE also mishandles Sandhi and can separate matras from their base consonants.Earlier morphology-aware and attention-guided methods have explored these challenges but remain limited.
  • 3. Morphological Dataset Preparation: Existing resources lack verified explicit morphological boundaries, motivating a Gold Standard Morphological Split Dataset for Hindi, Marathi, and Gujarati.Unimorph and MorphyNet are paradigm-centric, while GujMORPH relies on rule-based stemming that misses oblique roots and Sandhi.

4. Methodology · 5. Experiments and Results

SuTRA combines akshara-aware pre-tokenization with morphology-aware, dynamically constrained merging to preserve roots and morpheme boundaries. Experiments across Hindi, Marathi, and Gujarati evaluate structural alignment, semantic recoverability, translation quality, and robustness to surface noise, with reported gains over frequency-based tokenizers.

  • 4. Methodology: SuTRA uses orthographic rules to form akshara-like atomic units and marks forbidden morpheme-boundary crossings before statistical merging.Forbidden boundaries can come from a gold morphological lexicon or a seq2seq model.
  • 4. Methodology: Its merge score combines pair frequency with morphological validity and a dynamic rigidity constraint that is annealed during training.The curriculum favors safe lexical-root merges early, then relaxes constraints to attach functional affixes.
  • 5. Experiments and Results: Experiments evaluate Hindi, Marathi, and Gujarati along four axes: morphological alignment, semantic recoverability, translation quality, and robustness to surface noise.These axes correspond to questions about morpheme boundaries, whole-word meaning, downstream translation, and perturbation resilience.
  • 5.1. Morphological Alignment Evaluation: 0.586 and 0.617 are SuTRA’s highest Boundary F1 scores for Hindi and Marathi, respectively, while Gujarati remains competitive.The evaluation also reports Fertility Ratio, and SuTRA keeps fertility within a moderate range rather than producing arbitrary fragmentation.
  • 5.2. Semantic Recoverability: +34% is SuTRA’s relative gain in Hindi Linear R2 over Bpe, while Marathi and Gujarati show substantial benefits from the MLP.The results suggest SuTRA preserves semantic information more effectively than Bpe, whereas deeper modeling adds little for Bpe in Hindi.
  • 5.3. Machine Translation Performance: SuTRA is evaluated on Hindi↔Marathi translation with a 3-layer Transformer under identical architecture and optimization settings across tokenizers.The controlled setup isolates the impact of segmentation.
  • 5.3. Machine Translation Performance: SuTRA attains the best scores for Marathi→Hindi and remains competitive with the strongest baseline in the reverse direction.The reported results connect improved morphological alignment and semantic recoverability with meaningful neural machine-translation gains.
  • 5.4. Morphological Robustness Evaluation: 10,000 words per language are tested under synthetic orthographic and inflectional perturbations using Jaccard Overlap and Root-Affected Distance.These metrics measure segmentation consistency and whether perturbations alter root-related subwords.

6. Conclusion and Future Work

The results support SuTRA’s core hypothesis: respecting Indic script structure and morpheme boundaries reduces Morphological Shattering and improves semantic recoverability. These structural constraints also yield more robust downstream translation without increasing token fertility or vocabulary size.

  • Conclusion: Constraining tokenization to Indic script structure and morpheme boundaries reduces Morphological Shattering.This supports SuTRA’s structural approach to subword learning.
  • Conclusion: Respecting these structures makes whole-word semantics more linearly recoverable from subword embeddings.The passage links structural tokenization directly to improved semantic recoverability.
  • Conclusion: SuTRA improves robustness and downstream translation without increasing token fertility or vocabulary size.The reported gains are achieved while maintaining both token fertility and vocabulary size.

8. Use of Generative AI Disclosure · SuTRA: Supplementary Material · Summary of Supplementary Contents

The supplementary material documents the paper’s dataset construction, SuTRA methodology, complexity, evaluations, translation setup, robustness analyses, and limitations, while disclosing the authors’ use of generative AI tools. It also describes human review of AI-assisted writing and verification procedures.

  • 8. Use of Generative AI Disclosure: Gemini 3.0 Flash verified dataset ambiguities and corrected over-segmentation; PaperBanana generated Figures 1–2, while LLMs assisted minor writing and formatting.The authors state that human authors reviewed, modified, and validated the AI-assisted manuscript work.
  • SuTRA: Supplementary Material: The Gold Standard dataset uses a three-stage pipeline combining SampoNLP, statistical filtering, and LLM-based verification.The supplementary material also includes the rigidity constraint γ, including dynamic annealing and curriculum-based merging justifications.
  • SuTRA: Supplementary Material: The SuTRA section provides complete pseudocode covering pre-tokenization, semantic-unit tokenization, rigidity annealing, and trilateral enforcement.The supplementary contents separately identify this as the algorithmic description of SuTRA.
  • SuTRA: Supplementary Material: The complexity analysis theoretically compares training and inference time for SuTRA, standard Bpe, and MorphTok.This comparison appears as a dedicated supplementary section.
  • Summary of Supplementary Contents: Extended alignment analysis reports Exact Match, Precision, and Recall across all target languages, while the translation setup specifies architectures, hyperparameters, and preprocessing.These sections expand the main paper’s alignment results and document the neural machine translation experiments.
  • Summary of Supplementary Contents: The causal language modeling section evaluates predictive efficiency, derives perplexity, and presents training loss curves.These materials describe the generative modeling evaluation.
  • Summary of Supplementary Contents: The robustness analysis examines Morphological Shattering and secondary metrics for tokenizer resilience under noise.It provides a deeper analysis of the phenomenon introduced in the main paper.
  • Summary of Supplementary Contents: The limitations discussion addresses vocabulary scaling and the boundaries of semantic constraint saturation.These topics define the stated limits of the method’s semantic constraints.

10. Methodology for Morphological Dataset Preparation

The Gold Standard Morphological Split Dataset was built through a three-stage hybrid pipeline combining information-theoretic candidate segmentation with Gemini-based semantic verification. The process was adapted for Indic scripts and designed to correct statistical over-segmentation while restoring dictionary roots and preserving morphological structure.

  • Pipeline Design: The three-stage pipeline combines unsupervised information-theoretic segmentation with Gemini 3.0 Flash semantic verification.This combines statistical segmentation’s high recall with expert-aligned semantic reasoning for greater precision.
  • Candidate Segmentation: Initial candidate splits use Self-Referential Atomicity Scoring, which iteratively evaluates morpheme atomicity through optimal lexicon compression.The approach was adapted from the SampoNLP pipeline.
  • Indic Language Adaptation: The pipeline was adapted for Hindi, Marathi, and Gujarati by adding Devanagari and Gujarati Unicode filtering and expanding the atomic whitelist to include matras.These changes address the original implementation’s Latin-script constraints.
  • Semantic Verification: Statistical over-segmentation split roots containing suffix-like substrings, exemplified by Hindi shriman becoming shri+man, prompting semantic verification.This error ignored the semantic integrity of the root.
  • Semantic Verification: Gemini 3.0 Flash verifies splits using a unified prompt with language-specific rules that restore oblique roots and separate grammatical affixes.The structured output includes the root, prefixes, suffixes, stem, origin, and error type, ensuring deep rather than surface morphological structure.

11. The Rigidity Constraint (γ)

SuTRA uses a dynamic Rigidity Constraint (γ_t) that exponentially penalizes semantically invalid merges, then linearly anneals this constraint from strict morphological adherence toward standard BPE compression. The curriculum protects foundational roots and affixes early, while removing the penalty by the end of training.

  • Dynamic constraint: SuTRA scores each candidate merge as S(a, b) = f(a, b) · Ψ(a, b)^γt, applying the Rigidity Constraint as an exponential penalty on semantic validity.The constraint is dynamically adjusted during BPE training.
  • Annealing schedule: γstart = 4 aggressively penalizes boundary conflicts: when Ψ = 0.9, Ψ^4 ≈ 0.65, effectively blocking structurally unsafe merges.The high initial rigidity favors structurally safer alternatives.
  • Annealing schedule: As γt approaches γend = 0, Ψ(a, b)^0 approaches 1, removing the penalty and reducing the score to standard BPE frequency for compression.This curriculum prevents early boundary-crossing errors from propagating into Morphological Shattering.
  • Annealing schedule: At t = 0 and γ = 4, the exponential penalty prioritizes structurally safe merges with Ψ ≈ 1.0, even when their raw frequencies are lower.This forces the tokenizer to build valid semantic roots early in training.

12. Algorithm for SuTRA

SuTRA constructs an initial akshara-level vocabulary, identifies forbidden morphological boundaries, and performs morphology-aware curriculum merging until reaching the target vocabulary size. A post-processing bottleneck force-merges root tokens in overly segmented words to enforce a prefix–root–suffix structure.

  • Morphology-Aware Merging: During morphology-aware merging, SuTRA counts adjacent-pair frequencies and forbidden-boundary crossings before scoring candidate merges.The algorithm iterates while |V| < K, where K is the target vocabulary size.
  • Pre-tokenization & Boundary Identification: SuTRA pre-tokenizes the corpus into aksharas, initializes the vocabulary from unique units, and identifies morphological boundaries using a lexicon or Seq2Seq method.These steps form Phase 1: Pre-tokenization & Boundary Identification.
  • Morphology-Aware Merging: SuTRA selects the pair maximizing f(a, b) · Ψ(a, b)^γt, adds the merged unit and rule, and applies the merge to the corpus.The validity factor is defined as Ψ(a, b) ← 1 − χ(a,b) / f(a,b), and γt is annealed exponentially.
  • Structural Bottleneck: For words with more than three segments, post-processing identifies root indices and force-merges root tokens to enforce W = [P] ⊕R ⊕[S].This phase is labeled the Structural Bottleneck.

13. Computational Complexity Analysis for SuTRA · 14. Extended Analysis of Morphological Alignment.

SuTRA has training complexity asymptotically identical to MorphTok while integrating morphological constraints into vocabulary construction. Across Hindi and Marathi, it achieves the highest reported Boundary F1 and mitigates Morphological Shattering through orthographic constraints and morphological priors.

  • 13. Computational Complexity Analysis for SuTRA: ByT5 boundary prediction contributes O(|w|2) inference complexity per word because its byte-level Transformer scales quadratically with sequence length.This applies to MorphTok’s baseline and SuTRA’s pre-segmentation phase.
  • 13.1. Training Complexity: Standard BPE has training complexity O(N + V log M), whereas MorphTok adds ByT5 pre-segmentation at O(Vin · |w|2).MorphTok then performs standard BPE within constrained boundaries.
  • 13.1. Training Complexity: SuTRA’s total training complexity is O(Vin · |w|2 + N + V log M), asymptotically identical to MorphTok.The variables denote unique corpus words, word length, corpus size, vocabulary size, and active candidate pairs.
  • 13.2. Inference Complexity: Standard BPE encodes words in O(|w|), while MorphTok incurs O(|w|2) for out-of-vocabulary words because runtime boundary enforcement requires ByT5.SuTRA incorporates morphological constraints into its final vocabulary merge rules during training.
  • 14. Extended Analysis of Morphological Alignment.: SuTRA’s additional segments are characterized as accurate functional-affix captures rather than noise, despite its slightly higher fertility ratio than standard BPE.This frames the efficiency-accuracy frontier as a tradeoff between subword compression and linguistic integrity.
  • 14. Extended Analysis of Morphological Alignment.: SuTRA achieves the highest Boundary F1 in Hindi (0.586) and Marathi (0.617), establishing a new performance baseline.It maintains high alignment despite a slightly higher fertility ratio than standard BPE.
  • 14. Extended Analysis of Morphological Alignment.: Integrating orthographic constraints (Φ) and morphological priors (Ψ) creates a more robust semantic anchor for Indic LLMs.The paper reports that this integration effectively mitigates Morphological Shattering in purely data-driven baselines.

15. Experimental Setup for Neural Machine Translation · 16. Experiment Causal Language Modeling

The paper evaluates SuTRA in controlled Hindi→Marathi translation and downstream causal language modeling, using fixed data, architectures, training conditions, and task-specific metrics. The CLM setup additionally preserves Indic aksharas during pre-tokenization and measures out-of-distribution predictive efficiency with perplexity.

  • 15.1. Neural Machine Translation: SuTRA’s Hindi→Marathi translation utility is tested while keeping architectural and optimization variables constant across tokenizer comparisons.This design isolates the impact of tokenization.
  • 15.1.1. Data and Preprocessing: The translation experiments use identical BhasaAnuvaad train (581379), validation (32298), and test (32300) sentence pairs for every tokenizer variant.Preprocessing targets 32k units per language with a shared Hindi–Marathi subword dictionary after normalization.
  • 15.1.2. Model Architecture: The translation model is a standard Transformer with 3 encoder layers, 3 decoder layers, dmodel = 100, dff = 400, 4 attention heads, dropout 0.2, and tied decoder embeddings.These settings define the fixed architecture used for the controlled comparisons.
  • 15.1.3. Training and Optimization: Translation models train for 100k updates with Adam, inverse-square-root scheduling, 4,000 warmup updates, gradient clipping at 1.0, and 4,096 tokens per update.Best checkpoints are selected by validation loss.
  • 15.1.4. Evaluation Protocol: Translation evaluation uses detokenized beam-search outputs with beam size 5 and reports BLEU, chrF, validation perplexity, token fertility ratios, and OOV percentages.The metrics combine translation quality with intrinsic tokenizer and language-modeling indicators.
  • 16.1. Metric Derivation: Perplexity: The CLM experiment evaluates SuTRA with perplexity, defined from average negative log-likelihood over non-padding predicted tokens and interpreted as next-token uncertainty.Perplexity is the exponential of the average NLL.
  • 16.2. Implementation of SuTRA Pre-tokenization: SuTRA’s SCBPEPreTokenizer groups Devanagari and Gujarati phonetic clusters before BPE merging, keeping matras, halants, and anusvara attached to base consonants and marking word endings with </w>.The regex targets the relevant Devanagari and Gujarati Unicode base and mark ranges.
  • 16.3. Model Configuration and Training Hyperparameters: The CLM uses a GPT-2-style Transformer trained from scratch with Hugging Face Trainer, evaluates OOD perplexity on held-out parquet data using 256-row batches and a 256-token limit, and reports Avg NLL and PPL for BPE and SuTRA.Evaluation runs in eval() mode with torch.no_grad() for deterministic NLL calculations; the reported table states that lower values indicate better compression and predictive efficiency.

17. Extended Analysis of Morphological Robustness Evaluation

This section evaluates whether semantics-constrained tokenization preserves root–affix integrity under orthographic and morphological variation. SuTRA is tested against six baselines on noisy variants of Indic-language words using metrics that quantify catastrophic re-segmentation and semantic inconsistency.

  • Motivation: Morphological Shattering describes arbitrary root fragmentation caused by standard tokenizers’ lack of linguistic awareness on noisy text.The phenomenon motivates the robustness investigation.
  • Evaluation Setup: 10,000 unique words per language form an adversarial test set with synthetic swaps, deletions, substitutions, and insertions simulating typos and morphological shifts.These perturbations test tokenizer behavior under orthographic variation.
  • Evaluation Setup: SuTRA is compared against six baselines using three metrics to quantify susceptibility to catastrophic re-segmentation.The evaluation targets whether noisy inputs trigger drastic changes in tokenization.
  • Metrics: Jaccard Overlap (Jac. ↑) measures token-level intersection over union to assess segmentation consistency across variants.It is one of the three robustness metrics used in the comparison.
  • Metrics: Eight distinct metrics collectively quantify the Shattering effect by measuring whether tokenizers lose semantic consistency under orthographic variations.The metric relationship provides a broader evaluation of Indic-language morphology.

18. Limitations

The evaluation uses a uniform 32k vocabulary, leaving SuTRA’s behavior at larger vocabulary sizes unexplored and requiring future benchmarking.

  • Vocabulary Scaling: 32k was used as the uniform vocabulary size for controlled comparisons across tokenizers.This provides a consistent baseline for evaluating architectural efficiency.
  • Vocabulary Scaling: SuTRA’s performance with larger vocabularies, such as 64k or 128k, remains unexplored.It is unknown whether increased capacity would further mitigate Morphological Shattering.
  • Vocabulary Scaling: Future benchmarking is needed to determine whether semantic-constraint benefits saturate at higher vocabulary scales.The passage frames saturation as an open question.
Loading 2608.18087v1…