PaperlayerView exact revision on arXiv

Source-linked AI summary

A Grapheme-Aware Indic Tokenizer for Tamil: Large-Scale Training and Intrinsic Evaluation

Hari Krishnan K V, Sudarsun Santhiappan

arXiv:2609.06690v1cs.CL

TL;DR

Tamil grapheme clusters can be split by Unicode-level tokenizers, and intrinsic evaluation provides a way to assess segmentation apart from downstream tasks. This work maps complete graphemes reversibly before WordPiece learning and compares the Tamil tokenizer with multilingual systems; it reports the strongest fragmentation- and sequence-efficiency results and matches the highest observed compression ratio.

  • Problem

    Unicode-level tokenizers can split Tamil grapheme clusters into fragments, while intrinsic metrics assess tokenizer behavior independently of downstream models.

    Source: §3.1 · p. 3

  • Method

    The tokenizer reversibly maps complete Tamil graphemes to atomic representations before WordPiece vocabulary learning.

    Source: §3.1 · p. 3, §9 · p. 10

  • Results

    The tokenizer achieves the lowest Fertility, Word Fragmentation Rate, Percentage of Pretokens Changed, and Normalized Sequence Length, the highest Characters per Token, and matches the highest observed Compression Ratio among evaluated models.

    Source: §9 · p. 10

  • Takeaways & Limitations

    The results provide evidence that grapheme-aware preprocessing improves Tamil segmentation efficiency while preserving complete grapheme units.

    Source: §6.5 · p. 9

    Intrinsic evaluation does not directly measure effects on downstream tasks such as language modeling, machine translation, or question answering.

    Source: §7 · p. 9

Abstract

from arXiv · show

Tokenization forms the foundation of modern Natural Language Processing (NLP) systems by transforming raw text into discrete units that neural language models can process. The effectiveness of this process directly influences vocabulary efficiency, sequence length, computational cost, and downstream model performance. Although multilingual tokenizers such as Byte Pair Encoding (BPE), WordPiece, and SentencePiece have performed well across numerous languages, they often segment morphologically rich Indic languages inefficiently. Tamil, in particular, poses unique challenges because its grapheme-based writing system can represent a single visible character with multiple Unicode code points. In this work, we present a grapheme-aware Indic tokenizer for Tamil that preserves complete grapheme clusters through a reversible Unicode mapping strategy prior to WordPiece vocabulary learning. By operating on grapheme-level representations instead of individual Unicode code points, the tokenizer produces linguistically meaningful token boundaries while remaining fully compatible with transformer-based language models. The tokenizer is trained on a large-scale Tamil corpus and evaluated using a comprehensive intrinsic evaluation framework that measures compression efficiency, token fragmentation, information density, and vocabulary utilization. Experimental evaluation compares the proposed tokenizer against five widely used multilingual tokenizers: GPT-2, mBERT, mT5, mBART, and NLLB. The proposed tokenizer achieves the strongest performance among the evaluated tokenizers on fragmentation- and sequence-efficiency-oriented intrinsic metrics, while matching the highest observed compression ratio. These results demonstrate the effectiveness of grapheme-aware preprocessing for Tamil tokenization.

1 Introduction

Shared multilingual vocabularies can segment Tamil grapheme clusters into fragments, while downstream-task evaluation can obscure tokenizer-specific behavior. This work presents a grapheme-aware Tamil WordPiece tokenizer and evaluates it directly with intrinsic metrics against multilingual tokenizers.

1 Introduction

  • Shared multilingual vocabularies can allocate too little capacity to Indic scripts, while Tamil grapheme clusters may be split into fragments that increase sequence length and reduce compression efficiency.Source: §1 · p. 2
  • Grapheme-aware tokenization preserves complete grapheme clusters before subword learning while retaining compatibility with transformer architectures.Source: §1 · p. 2
  • Intrinsic evaluation directly measures compression, fragmentation, sequence length, information density, and vocabulary utilization without requiring expensive model training.The paper uses these metrics to compare tokenizer behavior independently of downstream language models.Source: §1 · p. 2
  • The tokenizer reduces fragmentation while maintaining competitive compression efficiency and vocabulary utilisation, and is compared with GPT-2, mBERT, mT5, mBART, and NLLB.The evaluation uses a large-scale Tamil corpus and examines vocabulary scaling.Source: §1 · p. 2

2 Related Work

Prior work established subword tokenization as a multilingual standard, but shared vocabularies can be inefficient for Indic scripts and Tamil graphemes. Intrinsic evaluation offers a direct way to compare tokenizer behavior apart from downstream models.

2 Related Work

  • Subword tokenization methods underpin contemporary multilingual models; GPT-2 employs byte-level BPE, while multilingual BERT adopts WordPiece and mT5, mBART, and NLLB use SentencePiece vocabularies learned from substantially larger multilingual datasets.Source: §2 · p. 2
  • Shared multilingual vocabularies allocate relatively few entries to individual Indic scripts, contributing to increased fragmentation, longer sequences, and lower compression efficiency.Source: §2 · pp. 2–3
  • Tamil grapheme clusters can be split by Unicode-level tokenizers, while grapheme-aware segmentation treats complete clusters as atomic units compatible with transformers.Source: §2 · p. 3
  • Intrinsic metrics quantify vocabulary efficiency, fragmentation, information density, and sequence compression independently of downstream language models.The paper adopts this framework to compare its Tamil tokenizer with multilingual tokenizers.Source: §2 · p. 3

3 Grapheme-Aware Indic Tokenizer

The tokenizer preserves Tamil grapheme clusters as atomic units through reversible Unicode mapping before WordPiece vocabulary learning. Its architecture retains transformer compatibility and varies vocabulary size to examine intrinsic tokenization quality.

3.1 Motivation

  • Tamil grapheme clusters can be split by Unicode-level tokenizers, increasing sequence length and reducing the information represented per token; the proposed preprocessing maps complete clusters to reversible atomic symbols before WordPiece learning.Source: §3.1 · p. 3

3.3 Grapheme Mapping

  • Each Tamil grapheme behaves as an indivisible symbol during training, preserving grapheme integrity while allowing frequent grapheme combinations to become subword units.The deterministic, reversible mapping works with existing WordPiece implementations and retains standard transformer-tokenizer compatibility.Source: §3.3 · pp. 3–4

3.2 Tokenizer Architecture

  • The pipeline extracts graphemes from normalized Tamil text, maps each cluster to a unique reversible Unicode representation, and learns a WordPiece vocabulary; inference reverses the mapping to restore Tamil text.The mapping is bijective, so encoding and decoding lose no linguistic information.Source: §3.2 · p. 3, Figure 1 · p. 4

3.4 WordPiece Vocabulary Learning

  • WordPiece learns vocabulary entries from mapped grapheme sequences, and the tokenizer architecture remains fixed while vocabulary size varies across experiments.Source: §3.4 · p. 4

4 Training Methodology

The tokenizer is trained on a cleaned, large-scale Tamil Wikipedia corpus converted to grapheme-level representations. Experiments vary vocabulary size while holding the remaining training and evaluation conditions constant.

4 Training Methodology

  • The tokenizer is trained on a cleaned large-scale Tamil Wikipedia corpus normalized and converted to grapheme-level representations before WordPiece training.Source: §4 · p. 4
  • Only vocabulary size varies during training; the corpus, preprocessing, tokenizer architecture, training configuration, and evaluation dataset remain unchanged.This controlled setup attributes observed performance differences to vocabulary scaling.Source: §4 · p. 5

4.1 Vocabulary Scaling

  • The study trains four tokenizer models with progressively larger vocabularies and evaluates each using the same intrinsic framework to analyze scaling effects on compression, fragmentation, sequence length, and vocabulary utilization.Source: §4.1 · p. 5

5 Experimental Setup

The evaluation uses a shared Tamil test set and five multilingual tokenizer baselines, then assesses tokenization through complementary intrinsic metrics. The setup notes that whitespace-based pretokenization makes WFR and PPC closely related measures.

5.1 Experimental Configuration

  • All tokenizers were evaluated on the same 5,000-sample Tamil subset of FLORES+, with identical metric definitions and procedures.This controlled setup reduces variation from dataset differences and enables direct comparison.Source: §5.1 · p. 6

5.2 Baseline Tokenizers

  • The proposed tokenizer was compared with GPT-2, mBERT, mT5, mBART, and NLLB, selected as widely adopted multilingual baselines.The baselines represent different subword tokenization approaches.Source: §5.2 · p. 6

5.3 Intrinsic Evaluation Metrics

  • The framework measures compression, fertility, characters per token, word fragmentation, pretoken changes, sequence length, and vocabulary coverage.The metrics assess compression, segmentation, information density, vocabulary utilization, and sequence length without downstream language-model training.Source: §5.3 · pp. 6–7
  • With predominantly whitespace-based pretokenization, WFR and PPC measure closely related phenomena, not mathematically equivalent metrics.Pretoken boundaries closely correspond to word boundaries in this evaluation strategy.Source: §5.3 · p. 7

6 Results

Across the multilingual comparison, the 50K Grapheme-Aware IndicTokenizer leads on fragmentation and sequence-efficiency metrics while matching the highest observed compression ratio. Larger vocabularies improve segmentation efficiency, and a representative sentence illustrates the resulting differences in token fragmentation.

6.1 Vocabulary Scaling

  • Table 4 reports results for progressively larger tokenizer vocabularies.The experiments varied vocabulary size while holding the corpus, preprocessing, architecture, training configuration, and evaluation dataset unchanged.Source: Table 4 · p. 7, §6.1 · p. 7

6.2 Analysis of Vocabulary Scaling

  • Compression Ratio remained constant across vocabulary sizes, while WFR and PPC decreased and Vocabulary Coverage stayed approximately 99.96%.The paper reports that vocabulary expansion primarily improved segmentation rather than lexical coverage.Source: §6.2 · p. 7
  • Fertility fell from 1.98 for the 10K tokenizer to 1.55 for the 50K tokenizer, while Characters per Token rose from 4.56 to 5.83.The paper interprets these trends as fewer subword units per word and more linguistic information per token.Source: §6.2 · p. 7
  • NSL decreased from 0.0806 to 0.0631 as vocabulary size increased.The paper notes that shorter sequences may improve computational efficiency during downstream language-model training.Source: §6.2 · p. 7

6.5 Discussion

  • The tokenizer achieved the lowest Fertility, WFR, PPC, and NSL, the highest CPT, and matched the highest observed Compression Ratio among evaluated models.The discussion attributes its longer, more informative subword units and shorter sequences to grapheme-aware preprocessing.Source: Table 5 · p. 8, §6.5 · p. 9

6.4 Qualitative Tokenization Analysis

  • For one Tamil sentence, the 50K tokenizer produced 10 tokens, compared with 19 for mBERT and 157 for GPT-2.GPT-2 required more than fifteen times as many tokens as the proposed tokenizer in this example; the 50K tokenizer preserved several complete Tamil lexical units.Source: §6.4 · pp. 8–9, Table 6 · p. 8
  • The qualitative comparison aligns with the intrinsic results: the 50K tokenizer produced the shortest sequence in the example while preserving larger Tamil subword units.The paper connects this behavior with lower Fertility and NSL and higher Characters per Token.Source: §6.4 · p. 9

7 Limitations

The study's scope is limited to Tamil, intrinsic evaluation, comparisons with multilingual tokenizers, and training on Tamil Wikipedia. These boundaries leave other languages, downstream effects, Indic-specific alternatives, and broader Tamil domains untested.

7 Limitations

  • The tokenizer was evaluated only for Tamil; its effectiveness on Malayalam, Hindi, Telugu, Kannada, and Bengali remains untested.The architecture is designed to support Indic scripts more broadly, but those languages were not evaluated.Source: §7 · p. 9
  • Intrinsic metrics do not directly measure tokenization's influence on downstream tasks such as language modeling, machine translation, or question answering.The study primarily relies on intrinsic evaluation.Source: §7 · p. 9
  • The comparison omits recently proposed Indic-specific tokenizers, limiting assessment within the broader Indic NLP ecosystem.The evaluation considers widely used multilingual tokenizers.Source: §7 · p. 9
  • Training used Tamil Wikipedia, which does not cover all domains, including conversational language, social media, literary text, and domain-specific corpora.The passage notes that larger and more diverse corpora may further improve robustness.Source: §7 · p. 9

8 Future Work

Future work would extend evaluation to additional Indic languages, downstream transformer tasks, and Indic-specific tokenizers. It would also test alternative vocabulary-learning algorithms alongside grapheme-aware preprocessing.

8 Future Work

  • Test whether grapheme-aware tokenization generalizes beyond Tamil by evaluating additional Indic languages with similar orthographic characteristics.The passage names multilingual evaluation as the most immediate extension.Source: §8 · p. 10
  • Measure the tokenizer's impact on downstream tasks by integrating it into encoder-only and encoder-decoder transformer architectures.Suggested tasks include language modeling, machine translation, and summarization.Source: §8 · p. 10
  • Compare against recently developed Indic-specific tokenizers and multilingual models trained specifically for Indian languages.Such comparisons would provide a more complete assessment within the Indic NLP landscape.Source: §8 · p. 10
  • Combine grapheme-aware preprocessing with SentencePiece Unigram or Byte Pair Encoding to test whether gains arise from grapheme preservation or its interaction with WordPiece.Source: §8 · p. 10

9 Conclusion

The paper introduces a grapheme-aware Tamil tokenizer and reports improved intrinsic segmentation as vocabulary size increases. Against the evaluated multilingual tokenizers, it leads on fragmentation and sequence-efficiency metrics while matching the highest compression ratio.

9 Conclusion

  • The approach preserves complete grapheme clusters through reversible Unicode mapping before WordPiece learning, enabling vocabulary construction over linguistically meaningful units.The tokenizer remains compatible with existing transformer tokenization pipelines.Source: §9 · p. 10
  • Across vocabulary sizes from 10K to 50K, increasing vocabulary size reduced fragmentation, raised information per token, and shortened sequences while coverage remained essentially constant.The results indicate that vocabulary expansion primarily enhanced segmentation efficiency rather than lexical coverage.Source: §9 · p. 10
  • The tokenizer achieved the lowest Fertility, Word Fragmentation Rate, Percentage of Pretokens Changed, and Normalized Sequence Length, and matched the highest Compression Ratio among evaluated models.It also achieved the highest Characters per Token; the comparison included GPT-2, mBERT, mT5, mBART, and NLLB.Source: §9 · p. 10
  • The results support grapheme-boundary preservation as an effective way to incorporate script-specific structure into Tamil subword tokenization.The work establishes a foundation for extending the method to other Indic languages and studying downstream language-modeling effects.Source: §9 · p. 10
Loading 2609.06690v1…