PaperlayerView exact revision on arXiv

Source-linked AI summary

NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages

Badal Nyalang

arXiv:2608.18094v1cs.CLcs.AI

TL;DR

Underrepresented Northeast Indian languages remain poorly served by mainstream multilingual NLP models. NE-BERT is a domain-specific multilingual encoder with tailored language coverage and tokenization, outperforming IndicBERT-V2 across all nine evaluated languages by 15.97× on average.

  • Problem

    Mainstream multilingual models poorly support Northeast Indian languages, which face extreme scarcity, morphological complexity, script diversity, and limited standardization.

  • Method

    NE-BERT is a ModernBERT-based multilingual encoder trained for nine Northeast Indian languages with Hindi and English anchors and Unigram tokenization.

  • Results

    15.97× average improvement over IndicBERT-V2 across all nine evaluated languages, with POS-tagging evaluation also showing practical utility.

  • Takeaways & Limitations

    Domain-specific modeling and appropriate tokenization can effectively serve ultra-low-resource Northeast Indian languages with as few as 1,000 training sentences.

  • Takeaways & Limitations

    Downstream validation covers only POS tagging in three languages, and small datasets of 214–519 sentences limit statistical robustness.

Abstract

from arXiv · show

Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource languages remain marginalized. We present NE-BERT, a domain-specific multilingual encoder model trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages (Hindi, English), a linguistically diverse region with minimal representation in existing multilingual models. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE-BERT outperforms IndicBERT-V2 and MuRIL across all 9 Northeast Indian languages, achieving 15.97X and 7.64X lower average perplexity respectively, with 1.50X better tokenization fertility than mBERT. We address critical vocabulary fragmentation issues in extremely low-resource languages such as Pnar (1,002 sentences) and Kokborok (2,463 sentences) through aggressive upsampling strategies. Downstream evaluation on part-of-speech tagging validates practical utility on three Northeast Indian languages. We release NE-BERT, test sets, and training corpus under CC-BY-4.0 to support NLP research and digital inclusion for Northeast Indian communities.

1 Introduction · 2 Related Work · 3 Dataset Construction

NE-BERT addresses severe underrepresentation of Northeast Indian languages with a specialized multilingual corpus, tokenizer, and weighted sampling strategy. The paper situates this approach within multilingual and regional-modeling limitations, then details data selection, preprocessing, and public release.

  • 1 Introduction: NE-BERT is a ModernBERT-based encoder designed specifically for Northeast Indian languages.The model targets languages underserved by existing multilingual and regional models.
  • 1 Introduction: 8.3M sentences cover 9 Northeast Indian languages plus Hindi and English through strategic weighted sampling.The corpus includes Assamese, Garo, Khasi, Meitei, Mizo, Naga, Nyishi, Pnar, and Kokborok.
  • 1 Introduction: 1.50× better average tokenization efficiency than mBERT is achieved by a custom 50,368-token SentencePiece Unigram tokenizer.The tokenizer is optimized for morphologically rich and agglutinative languages.
  • 1 Introduction: 7–15× gains on Pnar, Kokborok, and Nyishi accompany NE-BERT’s superior evaluation across all 9 Northeast Indian languages.The reported baselines are IndicBERT-V2 and MuRIL.
  • 2.1 Multilingual Language Models: mBERT and XLM-RoBERTa provide cross-lingual coverage but remain limited by multilinguality-related degradation and vocabulary fragmentation in low-resource languages.Language-specific adaptation and targeted continued pretraining are identified as promising responses.
  • 2.2 Regional Language Models: IndicBERT covers 12 scheduled Indian languages with 9B tokens, but regional initiatives generally underserve Northeast Indian languages.Region-specific African-language models demonstrate the viability of localized approaches.
  • 3.1 Language Selection and Sources: The dataset spans 3 language families across 9 Northeast Indian languages, with Hindi and English serving as anchors for cross-lingual transfer.Sources include government documents, news archives, educational and cultural materials, WMT 2025 parallel corpora, and public datasets.
  • 3.3 Weighted Sampling Strategy: 100× upsampling is applied to Pnar and Kokborok during tokenizer training, while Hindi and English receive 0.05× and 0.2× weights.MLM training uses raw sentence counts, and preprocessing includes length filtering, NFKC normalization, whitespace condensation, and a 99.5%/0.5% train-validation split.

4 Tokenization

NE-BERT uses SentencePiece Unigram instead of BPE to reduce subword fragmentation in morphologically rich Northeast Indian languages and to allocate vocabulary efficiently across languages. Weighted virtual counts further encourage single-token representations for ultra-low-resource languages, improving inference costs and semantic coherence.

  • Tokenizer choice: SentencePiece Unigram reduces harmful subword fragmentation in morphologically rich and agglutinative languages such as Kokborok, Garo, and Meitei.Its probabilistic approach avoids BPE’s greedy merging strategy, preserving complex grammatical information encoded within single words.
  • Configuration: 50,368 tokens comprise the tokenizer vocabulary, selected as the nearest 128 multiple for efficient Tensor Core execution on modern GPUs.The configuration also uses full Unicode character coverage and a maximum piece length of 16 characters.
  • Low-resource handling: Weighted virtual counts ensure common Pnar and Kokborok words form single tokens rather than fragmented multi-token sequences.This dramatically reduces inference costs and improves semantic coherence for ultra-low-resource languages.

5 Model Architecture

NE-BERT adopts a 149M-parameter ModernBERT-base architecture designed for efficient training and deployment. It combines deep transformer encoding, RoPE positional embeddings, Flash Attention 2, and unpadding while balancing capability against computational cost.

  • Architecture: 22 transformer layers comprise 38.7M embedding parameters and 110.3M encoder-layer parameters.These components define the model’s parameter allocation.
  • Positional Encoding: RoPE uses θglobal = 160,000 and θlocal = 10,000 for positional encoding.The design is intended to support longer-context training and better length extrapolation than learned position embeddings.
  • Training Efficiency: Approximately 30% throughput improvement during training comes from enabling unpadding, alongside Flash Attention 2 for memory efficiency.These mechanisms support efficient training on longer contexts.
  • Model Size Justification: 149M parameters balance model capability and computational efficiency, enabling cost-effective training for $7.31 on a single A40 GPU.The configuration is comparable to mBERT (110M) while smaller than IndicBERT-V2 (237M) and MuRIL (236M).

6 Training

NE-BERT was trained with dynamic masked language modeling, mixed-precision computation, and a 1,024-example effective batch size. Over 10 epochs, loss converged to 1.62 on training and 1.64 on validation without overfitting, using modest single-GPU resources.

  • Training objective: 15% masking probability and dynamic masking exposed different masked positions each epoch, improving generalization compared with static masking.Dynamic masking was used within masked language model training.
  • Implementation: 1,024 examples formed the effective batch size through 32 examples per device and 32 gradient accumulation steps.Training used mixed FP16 precision with TF32 enabled.
  • Compute: 17 hours of training on a single NVIDIA A40 GPU with 48GB VRAM cost $7.31.The implementation used PyTorch 2.4+, Hugging Face Transformers 4.48+, and Flash Attention 2.x.
  • Training dynamics: 10 epochs reduced loss from approximately 10.0 at initialization to 1.62 for training and 1.64 for validation.The close tracking of training and validation loss indicated no overfitting despite the small corpus size for some languages.

7 Evaluation

NE-BERT is evaluated on deduplicated, leakage-controlled test sets covering all nine Northeast Indian languages, using perplexity, tokenization fertility, and bits per character. It achieves the strongest average performance across Northeast Indian languages and maintains effective results across resource levels and anchor languages.

  • Evaluation Setup: Held-out test sets contain 500 sentences per language, deduplicated against training data and drawn from newer sources to prevent data leakage.Sentences were filtered to a minimum length of 15 characters, and the test sets were publicly released.
  • Evaluation Coverage: All 9 Northeast Indian languages in training are evaluated, although Nyishi and Naga use smaller test sets from WMT 2025 parallel corpora.These smaller sets provide initial validation across the complete language coverage.
  • Perplexity Results: 2.21 average perplexity makes NE-BERT the strongest model across Northeast Indian languages, ahead of IndicBERT-V2 (35.29), MuRIL (16.88), and mBERT (2.77).NE-BERT’s average perplexity improvements over IndicBERT-V2 and MuRIL are 15.97× and 7.64×, respectively.
  • Tokenization Efficiency: 1.68 average tokens per word and 0.347 average BPC show that NE-BERT’s tokenizer is more efficient than all evaluated baselines.Its fertility is lower than IndicBERT-V2 (2.08), MuRIL (2.14), and mBERT (2.51), while its BPC is lower than IndicBERT-V2 (1.497), MuRIL (1.271), and mBERT (0.590).
  • Resource-Dependent and Anchor-Language Performance: NE-BERT performs especially well on ultra-low-resource languages, while mBERT remains competitive on Assamese and Meitei, which have extensive Wikipedia coverage.NE-BERT also outperforms IndicBERT-V2 and MuRIL on English (1.55 PPL) and Hindi (1.43 PPL) anchor evaluations.

8 Downstream Evaluation

NE-BERT’s downstream utility was evaluated through POS tagging on Khasi, Mizo, and Nagamese using Universal Dependencies annotations. It achieved the highest accuracy across all three languages, averaging 82.4% and surpassing established multilingual baselines.

  • Evaluation setup: The POS-tagging evaluation covered 519 Khasi sentences, 502 Mizo sentences, and 214 Nagamese sentences.The datasets included train, development, and test splits and used Universal Dependencies annotations.
  • Evaluation setup: All models were fine-tuned for 5 epochs with learning rate 2 × 10−5, batch size 16, and standard cross-entropy loss.The authors characterize the results as illustrative rather than definitive because the annotated datasets are small.
  • Results: 82.4% average POS-tagging accuracy made NE-BERT the top-performing model across Khasi, Mizo, and Nagamese test sets.The evaluation compared NE-BERT with mBERT, IndicBERT-V2, and MuRIL.
  • Results: 9.1 percentage points was NE-BERT’s margin over mBERT, while 23.2 percentage points separated it from IndicBERT-V2.These figures are reported for average POS-tagging accuracy across the three languages.
  • Results: 86.4% versus 41.7–44.8% on Nagamese showed a particularly large gap between NE-BERT and IndicBERT-V2 or MuRIL.The result was presented as evidence that adequate language representation during pretraining matters.

9 Conclusion

NE-BERT demonstrates that domain-specific modeling and tailored tokenization can serve ultra-low-resource Northeast Indian languages, while outperforming established baselines. Its efficiency-focused design and open release provide a practical foundation for further research and community applications.

  • Conclusion: NE-BERT serves 9 Northeast Indian languages, including languages with as few as 1,000 training sentences.The conclusion presents the model as a foundation for future NLP research on Northeast Indian languages.
  • Conclusion: 1.50× better fertility than mBERT and 4.3× better BPC than IndicBERT-V2 demonstrate the benefits of vocabulary optimization.The approach combines weighted Unigram tokenization, aggressive upsampling, and cost-effective training at $7.31 on a single A40 GPU.
  • Conclusion: NE-BERT, its tokenizer, training code, and documentation are released under CC-BY-4.0 to support community-driven improvements and applications.The release is positioned as a foundation for future NLP research on Northeast Indian languages.

Limitations

NE-BERT’s limitations include narrow downstream validation, an encoder-only design that excludes generation tasks, and continued vulnerability in ultra-low-resource languages under distribution shift.

  • Limited Downstream Evaluation: POS tagging was evaluated for only three languages, while broader tasks and all nine trained languages remain future work.The available POS datasets contain just 214–519 sentences, limiting statistical robustness.
  • Encoder-Only Architecture: NE-BERT supports representation tasks but cannot directly perform machine translation, summarization, or dialogue generation.Generation requires decoder or encoder-decoder architectures.
  • Ultra-Low-Resource Vulnerability: Languages with fewer than 3,000 sentences remain vulnerable to distribution shift and unexpected behavior on out-of-distribution inputs.Weighted sampling mitigates vocabulary fragmentation but does not eliminate this vulnerability for Pnar, Kokborok, Garo, and Naga.

Future Work

Future work will broaden NE-BERT’s evaluation, data coverage, model architectures, transfer analysis, and real-world deployment. Planned efforts include stronger benchmarks across Northeast Indian languages, decoder-only models for conversational assistants, expanded ultra-low-resource corpora, and deployment pilots.

  • Comprehensive Downstream Evaluation: Benchmark datasets for named entity recognition, sentiment analysis, and additional part-of-speech tagging will expand evaluation across all 9 Northeast Indian languages.This includes Nyishi and Naga, with larger datasets and POS evaluation beyond the current three languages.
  • Decoder Models: Decoder-only models will apply the same data curation and tokenization strategies to generation tasks, targeting conversational assistants for Northeast Indian languages.
  • Data Expansion: 10K+ sentences are targeted for Pnar, Kokborok, Garo, and Naga through collaboration with native speaker communities and linguistic experts.
  • Cross-Lingual Transfer Studies: Zero-shot and few-shot transfer studies will assess generalization to related but unrepresented languages including Bodo, Karbi, and Dimasa.
  • Deployment Studies: Deployment pilots with government and educational institutions will evaluate performance on authentic tasks and gather community feedback.

Ethical Considerations

NE-BERT may inherit social biases and create risks for ultra-low-resource languages, including dominance, quality, hallucination, standardization, power, and representation concerns. The authors recommend evaluation, community feedback, transparent documentation, collaboration, benefit-sharing, and respect for community decisions before sensitive deployment.

  • Data and model risks: Public government, educational, and media corpora may contain gender, religious, caste, and other social biases reflecting their authors’ and publishers’ perspectives.Language models inherit biases present in training data.
  • Data and model risks: Despite weighted sampling, high-resource Meitei and Assamese may dominate model behavior and marginalize ultra-low-resource languages in multilingual contexts.This is identified as a dominance bias risk.
  • Data and model risks: Limited data for Pnar, Kokborok, Garo, and Naga increases sensitivity to data-quality issues and may amplify biases in small corpora.The passage identifies quality variance as an additional risk for these languages.
  • Data and model risks: Models may generate plausible-sounding but incorrect content for out-of-distribution inputs in ultra-low-resource languages.This constitutes a hallucination risk during use beyond the training distribution.
  • Mitigation and commitments: The authors recommend thorough evaluation and community feedback before sensitive deployment, alongside transparent documentation, collaboration, open-source benefit-sharing, and respect for community decisions.Sensitive applications include education, government services, and content moderation.
  • Linguistic and community impacts: NE-BERT may favor formal or written registers, reinforce extractive researcher–community relationships, and underrepresent oral traditions, youth language, and non-elite perspectives.These risks concern standardization pressure, power dynamics, and representation gaps.

Appendix · A Training Loss Curves · B Language Examples

The appendix documents NE-BERT’s 10-epoch training and validation loss behavior and provides representative sentences illustrating the nine Northeast Indian languages’ script and morphological diversity.

  • A Training Loss Curves: Training and validation loss curves are reported across 10 epochs.Figure 1 presents the complete curves.
  • A Training Loss Curves: Close tracking between training and validation loss indicates effective generalization without overfitting despite the ultralow-resource corpus size.This observation is based on the loss curves throughout training.
  • A Training Loss Curves: Figure 1 shows training and validation loss curves over 10 epochs.
  • B Language Examples: Table 2 presents representative sentences from each of the 9 Northeast Indian languages in the corpus.The examples demonstrate script diversity and morphological variation across language families.
  • B Language Examples: Representative language examples demonstrate script diversity and morphological variation across language families.
  • B Language Examples: For Assamese and Meitei, Figure 2 shows both original-script sentences and italicized Latin transliterations.These languages use the Bengali-Assamese script.
Loading 2608.18094v1…