Source-linked AI summary
Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam
Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani, Sujoy Sarkar
TL;DR
NER remains largely unsolved across most Indic languages despite its importance for transforming text into machine-interpretable knowledge and the high cost of annotated corpora. This paper compares generative and encoder-based architectures across all eleven Naamapadam languages under identical strict CoNLL evaluation, finding that encoders outperform generative architectures in ten of eleven languages and motivating architecture-specific deployment guidance.
Problem
NER remains largely unsolved across most Indic languages, while collecting BIO-tagged corpora requires costly linguistic annotation and introduces systematic noise.
Method
The study jointly evaluates five classic model families, four decoder-only LLMs fine-tuned with LoRA, and nine few-shot configurations across all eleven Naamapadam languages under a single strict CoNLL-style protocol.
Results
Encoder models outperform all generative architectures in ten of eleven languages under strict CoNLL evaluation, with gaps of 7.5–40 pp.
Takeaways & Limitations
The findings support encoder-first deployment, with encoder fallback mandatory for multi-language production systems because Gemma-2 showed negligible performance on Oriya.
Takeaways & Limitations
The Assamese negative gap is uncertain because only 17 entity-bearing validation sentences produce high-variance F1 estimates, implying approximate parity rather than a reliable generative advantage.
Abstract
from arXiv · showhide
Language is humanity's most consequential technology, yet for over a billion speakers across India's twenty-two constitutionally recognised languages, its digital layer remains structurally incomplete. Named Entity Recognition (NER), the foundational step in transforming raw text into machine-interpretable knowledge, has been studied exhaustively for English but remains largely unsolved across most Indic languages. This paper presents a rigorous comparative study of generative and encoder-based neural architectures for NER on all eleven languages of the Naamapadam benchmark. We evaluate five classic model families spanning sequence-to-sequence transformers and multilingual encoders; four decoder-only large language models (LLMs) fine-tuned with LoRA and 4-bit NF4 quantisation; and nine generative models in zero-to-5-shot inference. Under strict CoNLL span-level evaluation, encoder-based models (mBERT and XLM-R, both F1=0.675 on Hindi) substantially outperform every generative architecture in ten of eleven languages, with gaps of 7.5-40 percentage points against the strongest competitor (Gemma-2-2B: avg F1=0.427). The best few-shot result reaches only 28% of the encoder baseline. We identify three language clusters--encoder-dominant, partial-coverage, and failure-zone; and provide actionable deployment guidelines grounded in transfer learning and low-resource NLP principles.
1 Introduction
Indic-language NER remains constrained by severe data and annotation-resource disparities, despite its importance for downstream language technology. This paper addresses the gap with a unified comparison of encoder and generative architectures across Naamapadam’s eleven languages.
- NER underpins information extraction, knowledge graphs, and question answering, so failures degrade downstream systems.
- Hindi, Bengali, and Assamese exemplify severe resource disparities, with Assamese having a training corpus of barely ten thousand sentences.
- Generative LLMs could reduce annotation costs if instruction-tuned models learned NER from a few in-context examples.
- The study compares five classic model families and four LoRA-fine-tuned decoder-only LLMs across all eleven Naamapadam languages.
- The experiments include exploratory dataset analysis, consistent encoder and decoder fine-tuning, entity-aware sampling, and strict CoNLL-2003 span-level evaluation.
- The paper also evaluates nine generative configurations from zero to five shots and groups languages into three performance clusters with deployment recommendations.
2 Related Work
Prior NER research spans sequence labelling with multilingual encoders, generative text-to-text reformulations, and parameter-efficient adaptation. The related work also documents persistent structural-output weaknesses in generative models under exact evaluation.
- Sequence labelling for NER: mBERT established strong zero-shot cross-lingual transfer, while XLM-R scaled multilingual coverage through larger SentencePiece vocabularies.
- Sequence labelling for NER: IndicBERT and MuRIL improved Indic-language transfer by exploiting transliterated and codemixed corpora.
- Generative models for structured prediction: T5 reframed NER as conditional text generation, while prompt-based approaches remained competitive mainly under relaxed rather than exact CoNLL span matching.
- Generative models for structured prediction: Prior work found LLMs weak at few-shot structured prediction, and reranking did not resolve output-format unreliability under strict evaluation.
- Parameter-efficient fine-tuning (PEFT): LoRA adapts frozen model weights through trainable low-rank matrices, while QLoRA adds 4-bit NF4 quantisation for training 2B models in under 16 GB VRAM.
3 Dataset: Naamapadam
Naamapadam is a BIO-annotated benchmark of PER, ORG, and LOC entities from AI4Bharat News Crawl across eleven Indian languages. Its resource imbalance, annotation noise, class imbalance, and long-tail entity distribution create uneven evaluation conditions.
- Naamapadam contains BIO-annotated PER, ORG, and LOC entities from AI4Bharat News Crawl across eleven languages.
- Hindi and Bengali are high-resource languages, while Oriya and Assamese form the low-resource tier, with Assamese containing 10K samples.
- Non-entity O tokens comprise 75–91% of training tokens, while PER dominates and ORG is chronically under-represented.
- Invalid BIO transitions peak in Punjabi and Hindi, reflecting semi-automated annotation problems involving agglutinative text and lexical polysemy.
- Over 90% of distinct entity surface forms appear fewer than five times across the languages, producing a universal long-tail distribution.
- Figure 1 highlights the split-size disparity: Hindi and Bengali dominate, whereas Assamese is critically sparse.
4 Models and Experimental Setup
The experimental setup addresses class imbalance under GPU-limited sampling and compares encoder, sequence-to-sequence, and decoder-only architectures under a common strict evaluation protocol. Experiments use fixed sample sizes and reproducible training settings.
- GPU constraints limit each language to 5,000 training and 500 testing samples.
- Entity-aware hybrid sampling combines half uniform random sampling with half stratified entity enrichment, guaranteeing at least 50 examples per entity type.
- Assamese provides only 2,552 entity-containing sentences within the 5K sample, imposing a corpus-size ceiling rather than a sampling failure.
- Encoders use token classification, sequence-to-sequence models generate labels, and decoder-only LLMs are fine-tuned with LoRA and 4-bit NF4 quantisation.
- Evaluation requires simultaneous agreement on start token, end token, and entity type under the CoNLL-2003 span-level protocol.
- All experiments use SEED = 42, with encoders trained using AdamW and decoder LLMs using paged AdamW 8-bit.
5 Fine-Tuning: Classic Models
Classic-model fine-tuning compares five model families across Naamapadam languages using 5K/500 subsets and strict evaluation. Encoder models dominate, while language-specific script coverage, morphology, transfer, and data scarcity shape outcomes.
- Experimental setup: The comparison covers five classic families across all eleven languages, with results visualised using P/R/F1 metrics.The experiments use 5K training and 500 testing samples where specified.
- Results: F1 = 0.000 for FLAN-T5 on Bengali because one misordered generation token can cascade across an entire span.The generative objective provides no structural guarantee on BIO validity.
- Results: XLM-R leads on Malayalam by 11.3 percentage points and Gujarati by 11.7 percentage points, linked to greater CommonCrawl Indic-script coverage.The passage attributes these language-specific gains to coverage differences.
- Results: mBERT leads on Tamil, where agglutinative morphology and distinctive Brahmic script interact better with character-level casing than SentencePiece segmentation.This is presented as a language-specific model advantage.
- Results: Marathi reaches approximately F1 = 0.75, benefiting from Devanagari sharing with Hindi and strong transfer across closely related languages.Oriya remains below F1 = 0.20 and Assamese reaches F1 = 0.41 amid script-coverage gaps and data scarcity.
6 Fine-Tuning: Decoder-Only LLMs
Decoder-only LLM fine-tuning is compared with encoder and seq2seq families across Hindi and all eleven languages. Encoders remain strongest overall despite having fewer parameters, while Gemma-2 is the leading generative model with important language-specific failures.
- Hindi comparison: On Hindi, encoders exceed F1 = 0.67, decoder LLMs range from 0.14–0.52, and seq2seq models range from 0.03–0.07.Figure 3 presents P/R/F1 for all nine model families.
- Cross-language comparison: 177–278M-parameter encoders outperform 500M–2B-parameter decoder LLMs in 10/11 languages.This parameter-efficiency inversion indicates that architectural suitability, rather than parameter count or pre-training scale, governs strict-evaluation NER performance.
- Cross-language comparison: Gemma-2 achieves average F1 = 0.427 and leads the generative models, attributed to diverse multilingual pre-training with stronger Indic-script coverage.The cross-language decoder results are extended through Table 7 and Figure 4.
- Language-specific outcomes: Assamese is the sole exception, with Gemma-2 at F1 = 0.455 edging mBERT at F1 = 0.413, though the passage recommends interpreting this as approximate parity.The 17-sentence validation set makes the estimate noisy.
- Language-specific outcomes: LLaMA-3.2 and Gemma-2 achieve near-zero F1 on Oriya because outputs emit Roman or Hindi characters instead of Odia glyphs.The passage identifies this as a tokeniser-level barrier requiring vocabulary extension before fine-tuning can help.
7 Few-Shot Inference
Zero-to-5-shot inference evaluates generative models on the full Hindi test set. Instruction-tuned models perform best, but few-shot results remain far below encoder performance and show format and context-related weaknesses.
- Experimental setup: The zero-to-5-shot experiment uses the full Hindi test set of 867 sentences.Table 8 and Figure 5 present the inference results.
- Results: F1 = 0.191 is the best few-shot result, achieved by Gemma3-1b-IT at 5-shot.Navarasa-2.0 reaches F1 = 0.160.
- Instruction tuning: Instruction-tuned models outperform non-instruction-tuned models by 3.8× (Mann-Whitney U, p <0.01).The passage attributes the gain to reproducing the token(LABEL) structure from in-context examples.
- Model behaviour: Seq2seq models produce F1 = 0.000 at every shot count, lacking decoder-only models’ autoregressive in-context learning mechanism.This contrasts their few-shot behaviour with instruction-tuned decoder models.
- Model behaviour: LLaMA-3.2-1B peaks at 3-shot and declines at 4–5 shots, consistent with long-context instability for structured outputs.The result indicates diminishing returns from adding demonstrations for this model.
8 Encoder vs. Generative: Head-to-Head
Across Naamapadam’s eleven languages, encoder models substantially outperform generative architectures under strict span-level evaluation, although the gap varies by language and generative family.
- 7.5–40.1 pp separates encoder and Gemma-2 performance across languages, with the widest gap on Malayalam.Malayalam records 40.1 pp, while Telugu has the narrowest gap at 7.5 pp.
- 40.1 pp is the Malayalam encoder–Gemma-2 gap, with XLM-R at 0.740 and Gemma-2 at 0.339.The passage attributes the disparity partly to agglutinative morphology and left-to-right token prediction.
- Assamese is the sole language where Gemma-2 shows a marginal advantage, whereas Oriya has near-zero generative performance.The Assamese result should be interpreted cautiously because its validation sample is extremely small.
- 0.54+ is the encoder advantage over seq2seq models in every language, while decoder-only LLMs are comparatively stronger but still trail encoders.Figure 7 contrasts uniformly large encoder–seq2seq gaps with smaller encoder–Gemma-2 margins.
9 Language Clusters and Error Analysis
Performance separates the languages into encoder-dominant, partial-coverage, and constrained-resource clusters, while error analysis identifies structural output failures in generative models.
- Encoder-Dominant: Nine languages form the encoder-dominant cluster, where encoders exceed F1 0.50 and decoder LLMs trail by 14–40 pp.The recommended deployment is mBERT or XLM-R with entity-aware hybrid sampling.
- Partial Coverage: Assamese forms the partial-coverage cluster, where Gemma-2 matches or marginally surpasses encoders despite sparse data.Recommended remedies include Bengali transfer, entity substitution augmentation, and adapter-based adaptation.
- Constrained-Resource Zone: Oriya forms the constrained-resource zone: generative models achieve near-zero F1, while encoders remain below F1 0.20.The paper recommends script-level tokeniser extension and targeted data collection before modelling efforts.
- Error Analysis: 55–80% of seq2seq errors and 18–34% of decoder-LLM errors are BIO malformations, compared with 0% for encoders.Decoder-only models also show more boundary off-by-one errors under exact-match evaluation.
10 Conclusion
This study unifies encoder and generative NER evaluation across all eleven Naamapadam languages and finds a strong, deployment-relevant advantage for encoders.
- The study jointly evaluates five classic model families, four decoder-only LLMs, and nine few-shot configurations under one CoNLL-style protocol.The benchmark covers all eleven Naamapadam languages.
- Encoders outperform all generative architectures in ten of eleven languages, with gaps of 7.5–40 pp.The best few-shot result reaches only 28% of the encoder baseline.
- Entity-aware sampling eliminates ORG under-representation at zero extra cost.The paper also reports that 177M-parameter encoders outperform 2B-parameter LLMs.
- Future work targets constrained BIO decoding, retrieval augmentation, cross-lingual transfer, and hybrid encoder–decoder architectures.The paper argues that progress requires targeted data, tokeniser engineering, and architecture choices suited to structural prediction.