PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

Rakib Abdullah, Md. Maruful Islam Maruf

arXiv:2609.29101v1cs.LG

TL;DR

Bangla medical NER is challenged by linguistic variability and scarce domain-specific annotation. The paper benchmarks fine-tuned encoders against GPT-4o mini prompting on BanglaHealthNER, finding XLM-RoBERTa strongest among the reported fine-tuned results and fine-tuned models substantially stronger than prompting.

  • Problem

    Medical NER for low-resource languages remains challenging amid linguistic variability and scarce domain-specific annotated corpora, while prior LLM evaluation covered only 50 test samples and omitted BanglaBERT and per-entity analysis.

    Source: §I · p. 1

  • Method

    The paper benchmarks fine-tuned BanglaBERT, mBERT, and XLM-RoBERTa against GPT-4o mini zero-shot and few-shot prompting on BanglaHealthNER.

    Source: §III · p. 2

  • Results

    3.76×: Fine-tuned XLM-R outperforms the best GPT-4o mini configuration, zero-shot at F1 0.1585; XLM-RoBERTa reaches F1 0.5959 versus the previously reported 0.5848.

    Source: §I · p. 1, §V · p. 6

  • Takeaways & Limitations

    The benchmark finds domain diversity can outweigh language specificity for clinical NER, while Symptom remains the hardest entity type despite the largest training support.

    Source: §V · pp. 5–6

Abstract

from arXiv · show

Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical benchmark evaluating three fine-tuned transformer encoders-BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa-against GPT-4o mini under zero-shot and few-shot prompting configurations for Bangla medical NER. In contrast to prior studies that evaluated large language models on limited subsets of only 50 samples, we conduct a large-scale evaluation across the full test set of 3,179 samples, providing statistically robust and reproducible baselines. Our fine-tuned XLM-RoBERTa model achieves an F1- score of 0.5959, establishing a new state-of-the-art and surpassing the previously reported best result of 0.5848. Crucially, we demonstrate that the language-specific BanglaBERT model consistently underperforms its multilingual counterparts with an F1-score of 0.4937, indicating that pretraining domain diversity can outweigh language specificity in highly specialized clinical settings. Furthermore, we present a detailed per-entity-type analysis for this task, revealing that Medicine and Specialist categories are recognized with high reliability, achieving F1- scores above 0.83, while the Symptom category remains the most challenging with an F1-score of 0.4367 despite being the most frequent training class. Finally, fine-tuned transformer models outperform the optimal prompting configuration by a factor of 3.76, confirming that prompt-only pipelines remain inadequate for structured clinical entity extraction in low-resource language environments.

I. Introduction

Bangla medical NER remains underexplored amid linguistic variability and scarce domain-specific annotation. This benchmark expands prior comparisons and finds multilingual encoders outperform BanglaBERT, with XLM-RoBERTa achieving the strongest reported F1.

I. Introduction

  • Medical NER extracts clinical concepts such as diseases, symptoms, medications, dosages, and procedures from unstructured medical narratives.The task supports downstream healthcare applications, including clinical decision support, EHR analytics, pharmacovigilance, and biomedical knowledge graphs.Source: §I · p. 1
  • Medical NER for low-resource languages remains underexplored, and Bangla clinical text processing has received little systematic attention amid linguistic variability and scarce annotated corpora.Bangla has over 230 million native speakers.Source: §I · p. 1
  • Prior work evaluated LLMs on only 50 test samples, omitted BanglaBERT, and did not report performance by entity type.These limitations left statistical reliability, language-specific model comparison, and category-level difficulty unclear.Source: §I · p. 1
  • The study evaluates fine-tuned BanglaBERT, mBERT, and XLM-RoBERTa alongside GPT-4o mini zero-shot and few-shot prompting across the complete test corpus.This controlled benchmark addresses prior limitations in model coverage, test-set size, and per-entity analysis.Source: §I · p. 1
  • 0.5959 F1: XLM-RoBERTa exceeds the previously reported 0.5848 on the same dataset.The benchmark also finds BanglaBERT underperforms multilingual models, while Medicine and Specialist are the most reliably extracted categories and Symptom is the most challenging.Source: §I · pp. 1–2
  • GPT-4o mini shows non-monotonic few-shot behavior: zero-shot outperforms one-shot configurations.Source: §I · p. 2

II. Related Work

Prior NER research highlights both domain-specific pretraining for biomedical text and multilingual pretraining for cross-lingual transfer.

A. Transformer-Based NER

  • Biomedical encoders such as BioBERT and PubMedBERT show benefits from domain-specific pretraining, and a review reports domain-specific transformers outperforming classical and hybrid health-text NER approaches,,.The passage also notes BERT's advances in sequence labeling.Source: A. Transformer-Based NER · p. 2
  • Biomedical pretraining and multilingual transfer provide distinct motivations for transformer encoder comparisons in medical NER.Source: A. Transformer-Based NER · p. 2
  • XLM-RoBERTa's pretraining on 2.5TB of CommonCrawl text across 100 languages supports strong cross-lingual transfer and its use as a leading baseline for non-English NER.Source: A. Transformer-Based NER · p. 2

B. Bangla NLP and Named Entity Recognition

Bangla NLP research spans general-domain tasks and NER, while Bangla medical NER has a specific dataset and prior-model landscape. Prompt-based LLMs face structural challenges for exact-span clinical NER.

B. Bangla NLP and Named Entity Recognition

  • BanglaBERT is a BERT-style encoder pretrained on 27.5GB of Bangla web text and performs strongly on the Bangla Language Understanding Benchmark.General-domain Bangla research also includes sentiment analysis, question answering, and summarization,, alongside the B-NER dataset.Source: B. Bangla NLP and Named Entity… · p. 2
  • Bangla medical NER includes Zamil et al.’s attention-based BiLSTM-CRF; BanglaMedNER contains 117,000 tokens across three entity classes and achieves F1 0.78 with BiLSTM-CRF, while BanglaHealthNER has seven entity types,,.Source: B. Bangla NLP and Named Entity… · p. 2
  • The work extends the BanglaHealthNER comparison by broadening model coverage, evaluating the full test set, and analyzing performance across entity categories.Source: B. Bangla NLP and Named Entity… · p. 2

C. Prompt-Based LLMs for Clinical NER

  • Generative models can struggle with exact span boundaries required by token-level NER, and reviews identify hallucination, inconsistent outputs, and boundary failures as clinical deployment barriers,.Prior work found few-shot clinical information extraction possible when large annotated datasets are unavailable.Source: C. Prompt-Based LLMs for Clinical NER · p. 2

III. Methodology

The benchmark uses a shared preprocessing pipeline and parallel supervised and prompt-based evaluation pathways. It applies them to BanglaHealthNER, which contains seven clinical entity types and an imbalanced training distribution.

III. Methodology

  • A shared preprocessing stage precedes the supervised and prompting pathways; supervised inputs receive subword BIO-label alignment, masking, and truncation or padding to 128 tokens.Source: §III · p. 2, Figure 1 · p. 3
  • The framework compares three fine-tuned encoders with GPT-4o mini prompting and evaluates both pathways using span-level precision, recall, and F1.Token-level accuracy is additionally reported for fine-tuned models; the fine-tuned models use the complete 3,179-sample test set, while LLM experiments use 200 filtered samples.Source: §III · pp. 2–3

A. Dataset

  • BanglaHealthNER contains 25,426 training, 3,178 validation, and 3,179 test samples, with seven clinical entity types and 15 BIO labels.Source: A. Dataset · p. 3
  • Symptom is the largest training class at 104,870 tokens, while Medical Procedure is the smallest at 7,303 tokens, creating class imbalance.Source: A. Dataset · p. 3
  • Table II presents a representative Bangla medical sentence with BIO tags and code-mixed English medical terminology.Multi-token entities use B- and I- prefixes.Source: A. Dataset · p. 3, Table II · p. 3

B. Data Preprocessing and Tokenization

The preprocessing pathways tokenize supervised inputs into model-specific subwords with BIO labels aligned at the word level, while prompt inputs are reconstructed as natural Bangla sentences and use a fixed token-label output schema.

B. Data Preprocessing and Tokenization

C. Fine-Tuned Transformer Models

The benchmark fine-tunes three encoders spanning Bangla-specific and multilingual pretraining, using identical training settings and selecting checkpoints by validation F1.

C. Fine-Tuned Transformer Models

  • XLM-RoBERTa, mBERT, and BanglaBERT are fine-tuned with token-classification heads, contrasting multilingual baselines with a Bangla-specific encoder.XLM-RoBERTa is also the backbone evaluated by Sami et al., enabling direct comparison.Source: C. Fine-Tuned Transformer Models · p. 4
  • All models use a 2 × 10^-5 learning rate, batch size 16, four epochs, weight decay 0.01, AdamW, sequence length 128, and fp16 training.The checkpoint with the highest validation F1 is evaluated once on the 3,179-sample held-out test set.Source: C. Fine-Tuned Transformer Models · p. 4

D. Prompt-Based LLM Evaluation

The benchmark compares fine-tuned encoders with GPT-4o mini prompting and reports span-level extraction results. XLM-RoBERTa leads the fine-tuned models, while prompting remains substantially lower and validation trajectories differ across encoders.

D. Prompt-Based LLM Evaluation

  • 200 test samples are used for GPT-4o mini zero- and few-shot evaluation, filtered to sequences no longer than 60 tokens to avoid output truncation.The evaluation uses the same model as Sami et al. and a larger sample than their 50-sample evaluation.Source: D. Prompt-Based LLM Evaluation · p. 4
  • Prompt queries use temperature zero, define seven entity categories and BIO tagging, request JSON token-label pairs, and include labeled training examples for few-shot settings.Generated JSON is post-processed and aligned to BIO labels; predictions are padded or truncated when their length differs from the gold sequence.Source: D. Prompt-Based LLM Evaluation · p. 4

E. Evaluation Metric

  • Span-level precision, recall, and F1 require exact entity-boundary matching under BIO; token-level accuracy is additionally reported for fine-tuned models.Fine-tuned models use the full 3,179-sample test set, whereas LLM evaluation uses 200 samples due to API rate and cost constraints.Source: E. Evaluation Metric · p. 4

A. Overall Model Comparison

  • 3.76×: XLM-RoBERTa’s F1 of 0.5959 exceeds the best GPT-4o mini configuration, zero-shot at 0.1585; zero-shot also outperforms one-shot at 0.1063.Three-shot and five-shot scores are 0.1195 and 0.1293, respectively.Source: A. Overall Model Comparison · p. 4
  • 0.5959 F1 for XLM-RoBERTa exceeds the 0.5848 reported by Sami et al., while mBERT scores 0.5738 and BanglaBERT 0.4937.XLM-RoBERTa has token-level accuracy 0.8979 versus 0.9005 for the prior result.Source: Table IV · p. 4, A. Overall Model Comparison · p. 4
  • All three fine-tuned models improve steadily in validation F1; XLM-RoBERTa and mBERT continue through epoch four, while BanglaBERT plateaus earlier.Source: Figure 2 · p. 5, A. Overall Model Comparison · p. 5

B. Per-Entity-Type Analysis

XLM-RoBERTa’s per-entity results span high-, middle-, and low-performing tiers, with Symptom both the lowest-scoring and most-supported category.

B. Per-Entity-Type Analysis

  • The per-type breakdown covers seven entity types and 8,180 spans; its support-weighted aggregate differs slightly from Table IV’s sequence-level F1 of 0.5959.Source: B. Per-Entity-Type Analysis · p. 5
  • 0.8783 F1 for Specialist and 0.8366 for Medicine place them in the high-performing tier; Age scores 0.7759 and Medical Procedure 0.6140.Medicine is the second-most frequent class.Source: B. Per-Entity-Type Analysis · p. 5
  • 0.4367 F1 for Symptom is the lowest category score, despite 3,419 spans representing 41.8% of test entities.Dosage scores 0.4743 and Health Condition 0.4511; the passage notes that data volume alone does not resolve Symptom’s linguistic difficulty.Source: B. Per-Entity-Type Analysis · p. 5

V. Discussion

The discussion finds multilingual encoders outperform BanglaBERT, with mBERT close to XLM-RoBERTa at lower cost. Entity performance varies sharply, and fine-tuned XLM-RoBERTa substantially outperforms GPT-4o mini prompting.

Domain diversity over language specificity

  • 0.4937 F1 for BanglaBERT trails XLM-RoBERTa (0.5959) and mBERT (0.5738), with the result reproduced consistently across repeated runs.The authors suggest multilingual pretraining’s domain diversity may help with code-mixed Bangla-English medical terminology.Source: §V · p. 5

mBERT as a low-cost alternative

  • 0.022 F1 separates mBERT (0.5738) from XLM-R (0.5959), making mBERT a competitive lower-cost option for inference-constrained deployment.The discussion characterizes mBERT as older, smaller, and cheaper to run.Source: §V · p. 5

Full test-set evaluation

  • 0.5959 F1 on the full 3,179-sample test set modestly exceeds Sami et al.’s 0.5848, while token-level accuracy is lower (0.8979 vs. 0.9005).The difference is attributed possibly to run-to-run variance rather than methodological advantage.Source: §V · p. 5

Non-monotonic few-shot behavior

  • 0.1585 F1 in zero-shot is GPT-4o mini’s peak; one-shot drops to 0.1063, and 3-shot and 5-shot only partially recover to 0.1195 and 0.1293.The observed pattern is non-monotonic and does not regain the zero-shot level.Source: §V · p. 5

Entity-level difficulty and clinical implications

  • 0.8783 for Specialist and 0.8366 for Medicine contrast with mid-tier Age (0.7759) and Medical Procedure (0.6140), while Dosage (0.4743), Health Condition (0.4511), and Symptom (0.4367) are difficult.Symptom has the largest support (3,419 spans); the discussion links its difficulty to linguistic variability and recommends human review for symptom and condition extractions.Source: §V · pp. 5–6

Fine-tuning vs. prompting

  • 3.76× is the factor by which fine-tuned XLM-R outperforms the best GPT-4o mini configuration, zero-shot at 0.1585 F1.The authors conclude prompt-only pipelines remain inadequate for structured clinical NER in low-resource settings,.Source: §V · p. 6

VI. Conclusion

The benchmark extends prior Bangla medical NER work with broader model coverage, full test-set evaluation, and per-entity analysis. Its findings favor multilingual encoders, identify mBERT as a lower-cost alternative, and show that symptoms remain difficult despite high support.

VI. Conclusion

  • XLM-RoBERTa consistently outperforms BanglaBERT, suggesting pretraining domain diversity outweighs language specificity for clinical NER.The conclusion also reports that mBERT is nearly comparable at lower computational cost.Source: §VI · p. 6
  • Fine-tuned transformers outperform the best GPT-4o mini configuration by 3.76×, while prompt-only pipelines remain inadequate for structured clinical NER in low-resource languages.Source: §VI · p. 6
  • Symptom is the hardest entity type despite having the largest training support, pointing to linguistic variability rather than data scarcity as the primary bottleneck.Source: §VI · p. 6
  • Future work includes Bangla-specific clinical domain-adaptive pretraining, cross-dataset generalization with Bangla-MedER, and chain-of-thought prompting for LLM-based Bangla medical NER.Source: §VI · p. 6

Related papers

Loading 2609.29101v1…