PaperlayerView exact revision on arXiv

Source-linked AI summary

When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages

Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos Aletras

arXiv:2608.27658v1cs.CL

TL;DR

Low-resource languages face tokenizer overfragmentation, while byte-level approaches create word-level granularity and backbone-alignment challenges. The paper adapts H-Net by initializing byte embeddings from a frozen subword model, aligning chunks to subword targets, and adding POS supervision. Across Indic-language experiments, the tokenizer-free approach improves word-level morphological tasks, with POS tagging gains of up to 13.3%.

  • Problem

    Subword tokenization overfragments low-resource Indic languages, while existing byte-level architectures require massive training and can misalign with frozen subword backbones.

  • Method

    The adapted H-Net initializes byte embeddings from subword representations, applies chunk alignment loss, and interleaves lightweight POS supervision while adapting only introduced components.

  • Results

    13.3% is the maximum POS-tagging improvement over a subword baseline, alongside improved word-level morphological-task performance across the evaluated Indic languages.

  • Takeaways & Limitations

    The framework extends structural knowledge from high-resource frozen language models to extremely low-resource languages without shared tokenization vocabularies.

  • Takeaways & Limitations

    Transfer is expected to be limited for languages sharing neither Hindi’s script nor its morphological structure, and evaluation covers only Devanagari and Nastaliq.

Abstract

from arXiv · show

Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.

1 Introduction

Low-resource Indic languages expose severe subword fragmentation and byte-level granularity challenges, motivating an adapted hierarchical framework that aligns byte chunks with frozen subword representations. Across experiments, the approach improves zero-shot transfer and word-level morphological processing while addressing limited-data training constraints.

  • Motivation: Qwen3 splits 97.7% of Bhojpuri words into multiple subword tokens, averaging 4.43 tokens per word.Shared Devanagari script with Hindi can cause tokenizers to apply Hindi-derived subwords to low-resource languages.
  • Motivation: Byte-level models avoid tokenizer segmentation but dilute word-level signals when non-Latin characters occupy multiple UTF-8 bytes.Hierarchical architectures group byte representations to address this granularity mismatch.
  • Research gap: Existing hierarchical paradigms require massive pretraining budgets, while randomly initialized byte modules misalign with frozen subword backbones and hinder cross-lingual transfer.These constraints make direct adaptation difficult in data-constrained scenarios.
  • Approach: The adapted H-Net initializes byte embeddings from backbone subword embeddings, uses chunk alignment loss, and interleaves POS supervision for boundary detection.Together, these modifications bridge byte-level chunks and the frozen backbone without large-scale training.
  • Evaluation: The evaluation covers POS tagging, named entity recognition, and sentiment analysis across five Indic languages, using two model families at three parameter scales.The reported experiments examine models at 1.7B, 4B, and 12B parameters.
  • Results: 13.3% is the maximum POS-tagging improvement over a subword baseline, with the architecture consistently improving zero-shot cross-lingual transfer.The results also report competitive target-language word-level morphological processing capabilities.
  • Analysis: Task-guided pretraining improves chunk-level representations by up to 16.4 points, while backbone initialization prevents collapse under limited data.Tokenizer-aware alignment targets are essential for SentencePiece backbones.

2 Related Work

Prior work addresses byte-level modeling, tokenization failures, and cross-lingual transfer, but commonly remains focused on high-resource or subword-based settings. This framework adapts frozen subword language models to raw bytes for low-resource Indic languages using limited training data.

  • Byte-level LMs: Byte-level research uses hierarchical architectures to manage long sequences, group adjacent representations, or align distinct modalities.These approaches operate directly on raw UTF-8 bytes but are described as constrained to high-resource settings.
  • This work: The proposed framework processes raw bytes, learns word-aligned chunks without a tokenizer, and adapts off-the-shelf subword-based language models.It replaces the original embeddings and language-model heads with H-Net byte-level modules.
  • This work: 500K sentences, approximately 1B bytes, are used to train only the newly introduced components while the backbone remains frozen.This setup targets minimal-corpus adaptation rather than full model pretraining.
  • Tokenization: Prior studies report that BPE overfragments low-resource languages and that vocabulary selection disadvantages underrepresented languages under data constraints.Related findings include segmentation failures in African and Indic languages.
  • Cross-lingual transfer: Shared vocabulary and tokenizer coverage are associated with cross-lingual transfer quality and representation alignment.Prior multilingual studies connect vocabulary overlap with transfer performance.

3 Methodology

H-Net processes raw UTF-8 bytes through local encoding, dynamic chunking, a global language-model backbone, and local decoding. The adaptation bridges byte-level chunks with a frozen subword model using initialized embeddings, chunk alignment, and POS supervision.

  • 3.1 Preliminaries: H-Net: H-Net processes raw UTF-8 bytes through four stages: local encoding, dynamic chunking, global backbone processing, and local decoding.The architecture produces contextualized byte representations, variable-length chunks, chunk-level backbone inputs, and byte-level outputs enriched with global context.
  • 3.1 Preliminaries: H-Net: The local encoder embeds each byte into D dimensions and applies sequential Mamba layers with residual updates to capture short-range dependencies.The final contextualized byte representations support boundary detection.
  • 3.1 Preliminaries: H-Net: Dynamic chunking estimates boundary probabilities from adjacent-byte similarity, assigns hard boundaries when p_t ≥ τ, and smooths each chunk with self-attention.The representation at the boundary position initializes each chunk before refinement across its bytes.
  • 3.1 Preliminaries: H-Net: Chunk embeddings enter the global language-model backbone, avoiding tokenizer-induced overfragmentation by operating on chunk-level rather than subword representations.EMA DeChunking then distributes chunk representations back to byte positions, combining local byte information with global context.
  • 3.2 Adapting H-Net for low-resource languages: The adaptation replaces random byte initialization with averages of containing subword embeddings, placing byte representations in the backbone’s pretrained semantic space.This initialization accelerates convergence under limited data.
  • 3.2 Adapting H-Net for low-resource languages: A chunk alignment loss pulls each dynamically formed chunk toward the average embedding of its corresponding backbone subwords, enabling interpretation by the frozen backbone.The alignment is intended to facilitate transfer without extensive fine-tuning.
  • 3.2 Adapting H-Net for low-resource languages: Interleaved POS supervision guides boundary detection, while the full objective combines autoregressive, boundary-ratio, alignment, and POS losses.POS training occurs every k steps, with k = 10 in this study, using a temporary linear POS head.

4 Experimental Setup

The study evaluates adapted H-Net in zero-shot transfer from Hindi to five low-resource Indic languages across POS tagging, NER, and sentiment analysis. It compares H-Net with frozen subword and continued-pretraining baselines using Qwen3 and Gemma3 models.

  • Target languages: The target languages are Bhojpuri, Marathi, Magahi, Sanskrit, and Urdu, with Hindi selected as the source adaptation language.Hindi provides sufficient training data and typological proximity through shared roots, structure, or script.
  • Models: Experiments use Qwen3-1.7B, Qwen3-4B, Gemma3-4B, and Gemma3-12B, contrasting model families pretrained on 119 and 140 languages.Coverage of several target languages is not explicitly reported, while Marathi and Urdu coverage is described as likely.
  • Training: Adaptation uses 500K Hindi sentences totaling 1.02B bytes, plus 13K Hindi Universal Dependencies sentences for task-guided POS pretraining.The setup unfreezes the first and last transformer layers for computationally efficient adaptation.
  • Baselines: The baselines are a frozen subword feature extractor with probes and continued pretraining on the same 500K Hindi sentences.The continued-pretraining baseline controls for additional Hindi exposure while isolating the byte-level architecture’s contribution.
  • Evaluation tasks, metrics, and protocols: Evaluation covers POS tagging and NER as word-level sequence-labeling tasks, plus sentence-level sentiment analysis.Metrics are token-level accuracy for POS, entity-level F1 for NER, and accuracy for sentiment; probes transfer from Hindi to targets without target-language supervision.
  • Evaluation tasks, metrics, and protocols: Table 2 reports zero-shot performance across three downstream tasks, highlighting the best result among Sub, CPT, and H-Net for each task and model family.The table organizes comparisons across the three model configurations and task settings.
  • Evaluation tasks, metrics, and protocols: H-Net is evaluated at chunk level, assigning each chunk the label of the word contributing most of its bytes, while subword baselines use only each word’s first subword.Chunks containing only spaces or punctuation are excluded.

5 Results

H-Net consistently improves zero-shot transfer on word-level POS tagging and NER, especially where subword tokenization is highly fragmented, but subword models remain stronger for sentence-level sentiment analysis.

  • POS tagging: H-Net consistently outperforms corresponding subword baselines on POS tagging across the evaluated low-resource languages.Qwen3-1.7B gains 10.4 points on average, while Qwen3-4B gains 5.5 points.
  • POS tagging: 12.1 points: H-Net (Qwen3-1.7B) improves performance on Magahi, whose words average 4.90 Qwen3 tokens.The largest advantage occurs where tokenizer overfragmentation is most severe.
  • POS tagging: 52.1% zero-shot accuracy (+13.3 points): H-Net (Qwen3-1.7B) transfers to Urdu despite its Perso-Arabic Nastaliq script.The authors hypothesize that linguistic relatedness supports morphological transfer even when byte patterns differ.
  • NER: H-Net broadly outperforms subword baselines on NER, with average gains of 3.1 points for Qwen3-1.7B and 3.6 points for Gemma3-4B.The maximum improvement is 10.6 points on Sanskrit, where tokenization averages 5.72 tokens per word.
  • Sentiment analysis: Subword baselines dominate sentiment analysis, with Gemma3-4B reaching 80.6% and Gemma3-12B reaching 87.0%.All Qwen3 models reach near-random performance on Urdu at approximately 49%.
  • Takeaways: Overall, H-Net performs best on word-level POS and NER tasks, whereas sentiment analysis benefits more from subword models and large-scale pretraining.CPT consistently underperforms H-Net across tasks, indicating that the gains are associated with the H-Net architecture rather than additional Hindi exposure.

6 Analysis

The analysis attributes H-Net's training and boundary improvements to backbone initialization, POS supervision, and chunk alignment, while revealing a tokenizer-specific limitation in alignment targets.

  • Backbone-initialized byte embeddings: Random byte initialization causes BPB to plateau at 4.2 after rising from 3.6, whereas backbone initialization mitigates this training difficulty.The random model begins near 9.5 BPB and shows no further improvement over 20K steps.
  • Backbone-initialized byte embeddings: Backbone initialization reduces the boundary ratio from 0.50 to approximately 0.05, near the target ρ*=0.056.This provides an immediate gradient signal to the alignment loss.
  • Task-guided pretraining: 16.4 points: explicit POS supervision raises chunk-level accuracy from 43.1% to 59.5%.Byte-level accuracy increases by 0.3 points, indicating that raw byte prediction capabilities are maintained.
  • Chunk alignment loss: Without chunk alignment loss, C/W falls to 0.06–0.13 and 1:1% approaches zero across all six languages.Chunks then span 8–17 words on average instead of matching the approximately one-word target.
  • Chunk alignment loss: The Qwen3 tokenizer's dedicated space token creates geometric contrast, while Gemma3's space handling yields a near-zero space target that masks boundaries.The proposed centroid intervention restores contrast between space and text bytes and recovers word-level chunking.

7 Conclusion

The adapted H-Net framework addresses tokenizer overfragmentation in low-resource Indic languages by combining byte-level representations with alignment and POS supervision. Across five languages, it improves word-level morphological task performance using only 500K Hindi sentences.

  • The method addresses tokenizer overfragmentation for low-resource Indic languages without extensive training.
  • The framework combines backbone-initialized byte embeddings, interleaved POS supervision, and tokenizer-aware chunk alignment loss.These components produce word-aligned chunk representations while bypassing shared tokenization vocabularies.
  • 10.4 points average gains on POS tagging and 7.8 points on NER were observed across five Indic languages.
  • The adapted architecture extends structural knowledge from high-resource language models to extremely low-resource languages.

Limitations

The evaluation is limited to Indic languages, two scripts, three task types, and an unmatched compute setting. Several broader transfer and efficiency questions remain open.

  • Transfer relies on Hindi pretraining and is unlikely to benefit languages sharing neither Hindi’s script nor its morphological structure without additional data.
  • The evaluation covers Devanagari and Nastaliq, while effectiveness for Tamil, Chinese, Bengali, and other non-Latin scripts remains unvalidated.
  • Evaluation covers POS tagging, NER, and sentiment analysis, leaving deeper semantic tasks such as machine translation and natural language inference unexplored.
  • Wall-clock inference time is not reported, although dynamic chunking reduces backbone sequence length by ∼18× while byte processing adds overhead.
  • H-Net processes 1.02B byte tokens versus approximately 7.5M subword tokens for CPT, so a fully compute-matched comparison remains open.
  • The selected languages vary in morphological complexity and script, but Sanskrit and Urdu present especially challenging conditions for boundary detection and transfer.

A.2 Language and Dataset

The dataset suite evaluates five Indic languages spanning two scripts, varied morphology, resource availability, and proximity to Hindi. It covers POS, NER, and sentiment under cross-lingual and cross-script transfer.

  • The five-language evaluation spans Devanagari and Nastaliq, with variation in morphological complexity, resource availability, and typological proximity to Hindi.
  • Bhojpuri and Magahi are extremely low-resource, Hindi-related Devanagari languages suited to testing transfer under severe tokenizer overfragmentation.
  • POS tagging uses Hindi as the source language and substantially smaller target-language UD treebanks, including 230 Sanskrit and 357 Bhojpuri training sentences.
  • NER uses WikiANN, while sentiment analysis uses IndicSentiment with 1,000 binary-classification examples each for Hindi, Marathi, and Urdu.
  • Together, the datasets cover morphologically grounded sequence labeling and semantic classification in cross-lingual and cross-script settings.

A.3 Training Details

H-Net pretraining uses AdamW with cosine decay, warmup, bfloat16 precision, and compact Mamba encoder-decoder components. Pretraining takes about 26 hours per backbone on one A100 GPU.

  • H-Net pretraining uses AdamW with lr = 5 × 10−5, weight_decay = 0.01, max_grad_norm = 0.5, cosine decay, and 5% warmup.
  • Training uses batches of 8 sequences containing 2,048 bytes in bfloat16 precision, with gradient accumulation across 4 steps.
  • The local encoder and decoder each use a single Mamba layer with backbone-specific expansion settings.
  • Approximately 26 hours of pretraining are required per backbone on one NVIDIA A100 GPU, while downstream probe training takes about 10–30 minutes.
  • Table 5 organizes hyperparameters for H-Net pretraining, the CPT baseline, and downstream probe training.

A.4 In-Language Hindi Evaluation

In-language Hindi results are mixed: H-Net improves NER but generally trails the subword baseline on POS tagging and sentiment.

  • H-Net outperforms the subword baseline on Hindi NER across all models, with gains of +5.3, +9.4, and +3.2 pp.The result occurs at a fragmentation rate of 2.61 tok/word.
  • H-Net underperforms the subword baseline on Hindi POS tagging for most models.Hindi’s relatively low fragmentation gives the supervised subword baseline an advantage.
  • H-Net consistently underperforms relative to target-language results on sentiment.The passage attributes this pattern to semantic tasks benefiting more from training-data scale than morphological processing.
Loading 2608.27658v1…