PaperlayerView exact revision on arXiv

Source-linked AI summary

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed, Mir Sazzat Hossain, Md Fahim, Md Farhad Alam Bhuiyan

arXiv:2609.09964v1cs.CL

TL;DR

Regional Bangla dialects lack multi-annotation benchmarks needed for reproducible dialect-aware LLM evaluation. The paper introduces 5-DIALECTS-BN, a 6,000-entry benchmark aligned across scripts, translations, and subjectivity, and evaluates models across tasks and adaptation regimes. Its central findings are that limited LoRA supervision can outperform prompting baselines, whereas Romanized input consistently degrades current LLM performance.

  • Problem

    Regional Bangla dialects remain critically underresourced, and the absence of multi-annotation benchmarks limits reproducible diagnosis and evaluation of dialect-aware systems.

  • Method

    The paper releases 5-DIALECTS-BN and evaluates seven LLMs across three tasks, native and Romanized scripts, prompting regimes, and LoRA fine-tuning.

  • Results

    LoRA fine-tuning with 160 examples per dialect surpasses closed-source prompting baselines, while Romanized input consistently and severely degrades performance across models and regimes.

  • Takeaways & Limitations

    The benchmark makes transliteration effects and dialect-aligned supervision measurable across Bangla dialect tasks and LLM evaluation settings.

  • Takeaways & Limitations

    The fine-tuning evidence uses one budget of 160 examples per dialect and one hyperparameter configuration, without a data-scaling curve.

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable progress across natural language processing (NLP) tasks, yet their capabilities degrade sharply for low-resource languages and dialectally diverse settings. Bangla, the world's sixth most spoken language, exemplifies this gap: existing resources overwhelmingly target Standard Bangla, leaving its regional dialects without the benchmarks needed to develop or evaluate dialect-aware systems. We address this gap with 5-Dialects-BN, the first multi-annotation Bangla dialect benchmark to align Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional varieties. The dataset comprises 6,000 manually annotated entries spanning five major dialects: Chittagong, Barisal, Noakhali, Sylhet, and Rangpur (Chittagong 1,900; Noakhali 1,500; Sylhet 1,200; Barisal 700; Rangpur 700), reflecting natural online availability. Each entry is enriched with five aligned annotations: the original dialectal text, a Romanized transliteration, an English translation, a Standard Bangla translation, and a subjectivity label (subjective vs. objective). Annotations were produced and cross-validated by native speakers and undergraduate linguistics students to ensure dialectal authenticity and semantic fidelity. The resulting resource supports a diverse suite of tasks, including dialect identification, dialect-to-standard normalization, machine translation, subjectivity classification, and parameter-efficient fine-tuning (e.g., LoRA) of multilingual LLMs. By providing a standardized, multi-annotation benchmark, 5-Dialects-BN enables principled evaluation of LLMs on dialectally diverse Bangla and lays a foundation for further research in low-resource, dialect-aware NLP.

1 Introduction

5-DIALECTS-BN addresses the lack of reproducible resources for Bangla’s underresourced regional dialects by introducing a multi-annotation benchmark and evaluating LLMs across tasks, scripts, and adaptation regimes. The evaluation reports that limited LoRA supervision can outperform prompting baselines, while Romanized input consistently harms current LLM performance.

  • Regional Bangla dialects remain underresourced, and missing multi-annotation benchmarks hinder reproducible diagnosis and evaluation of dialect-aware systems.
  • 6,000 utterances across five dialects are aligned with dialectal text, Romanized transliteration, Standard Bangla, English, and subjectivity labels.
  • The benchmark evaluates seven LLMs on translation, subjectivity classification, and normalization using zero-shot, few-shot, CoT, and LoRA regimes.
  • 73.6 BLEU vs. Gemini 3 Flash’s 46.4 zero-shot: LoRA on 160 examples per dialect surpasses closed-source prompting baselines on translation and subjectivity classification.
  • Romanized input consistently and severely degrades current LLM performance through many-to-one phonemic information loss and subword token fragmentation.
  • Romanized text is treated as a first-class evaluation variable because it is widely used online and permits causal decomposition of transliteration penalties.

2 Related Work

Prior Bangla resources address subsets of dialectal, transliteration, and affective-text challenges, but 5-DIALECTS-BN expands coverage and aligns multiple annotations for dialect-aware evaluation.

  • Existing regional resources cover subsets of Bangla dialects, while DIALTSA-BN contains 600 utterances across four dialects with Standard Bangla and sentiment labels.
  • 5-DIALECTS-BN expands the corpus to 6,000 utterances and adds Rangpur plus Romanized and English fields to its aligned annotations.
  • Standard-Bangla translation systems largely train on formal text and may fail to generalize to colloquial regional dialects.
  • Prior transliteration studies report gains from adapted encoders, while other work finds limited improvement over classical baselines on transliterated text.
  • The benchmark uses dialect-to-English translation as an orthographically stable target and chrF++ and COMET for normalization without surface orthographic bias.

3 5-DIALECTS-BN

5-DIALECTS-BN is built through harvesting, native-speaker verification, multi-annotation, and agreement validation, producing 6,000 naturally sourced entries with five aligned fields.

  • A four-stage pipeline harvests online data, verifies dialect authenticity and boundaries, performs multi-annotation, and validates inter-annotator agreement.
  • Native-speaker screening accepts utterances with at least one dialect-diagnostic feature and rejects ambiguous or heavily code-mixed items.
  • 6,000 verified entries cover Chittagong, Noakhali, Sylhet, Barisal, and Rangpur, with counts reflecting natural online availability.
  • Each entry contains dialectal text, Romanized transliteration, Standard Bangla translation, English translation, and a binary subjectivity label.
  • Binary subjectivity is chosen because its boundaries show consistent agreement across dialects, unlike dialect-sensitive sentiment polarity.
  • Cohen’s κ = 0.78 for subjectivity and κ = 0.74 for Standard Bangla translation acceptance support categorical annotation reliability.

4 Experimental Setup

The experiments compare LLMs across three dialect tasks, prompting and LoRA adaptation, native and Romanized scripts, and closed- and open-source model regimes.

  • The benchmark covers dialect-to-English translation, binary subjectivity classification, and dialect-to-Standard-Bangla normalization.
  • Prompting is evaluated in zero-shot, few-shot, and CoT regimes, with few-shot prompts using 36 resampled demonstrations.
  • Every configuration uses either native Bangla script or Romanized transliteration while keeping target outputs and label spaces identical.
  • LoRA adapters are trained separately for each task-by-script combination on open-source models, excluding closed-source models because their weights are inaccessible.
  • Seven primary LLMs span closed-source and open-source systems, with IndicBART and BanglaT5 added as dedicated Indic sequence-to-sequence baselines.
  • Prompting uses dialect-balanced evaluation subsets, while LoRA uses disjoint folds with 160 training and 40 test instances per dialect.

5 Results and Analysis

Across three tasks and four regimes, the benchmark shows that prompting can create model-specific failures, while LoRA improves open-source performance but does not eliminate transliteration penalties.

  • Experimental setup: Seven LLMs are evaluated across zero-shot, few-shot, chain-of-thought, and LoRA regimes for translation, subjectivity classification, and normalization.Scores are macro-averaged across the five dialects plus Standard Bangla.
  • Prompting strategies vs. model competence: Prompting scores reflect model–strategy interactions: Gemini leads zero-shot translation at 46.4 BLEU and normalization at 71.41 chrF++, while open-source models show strategy-specific failures.Llama suffers demonstration interference, whereas Qwen’s 0.2 BLEU zero-shot score reflects 91.4% unparseable output.
  • Prompting strategies vs. model competence: CoT improves subjectivity classification but degrades translation surface fidelity through paraphrase drift and verbosity.GPT-4o-mini gains +11.2 F1 and Gemma +6.4 F1, while Gemini falls from 46.4 to 38.2 BLEU.
  • Causal decomposition of the transliteration penalty: Native Bangla outperforms Romanized input in all 19 model-regime pairs under both neural COMET and chrF++, with three proposed contributing mechanisms.The analysis attributes the penalty to many-to-one information loss, subword token fragmentation, and distributional sparsity across spelling variants.
  • Supervision and adaptation dynamics: 160 examples per dialect enable LoRA to raise Mistral-7B translation from 16.2 to 73.6 BLEU and place all open models at 74.0–78.4 subjectivity F1.Mistral-7B exceeds Gemini 3 Flash zero-shot translation, while all four open models exceed closed-source zero-shot and few-shot subjectivity baselines.
  • Linguistic divergence and error analysis: LoRA leaves a substantial native-over-transliterated translation gap of ∆=12.0–17.0 BLEU, while linguistic divergence continues to shape errors after adaptation.High-lexical-divergence items retain higher mistranslation rates than low-divergence items, and Chittagong non-cognate verbs remain especially difficult.

6 Conclusion

5-DIALECTS-BN aligns transliteration with dialectal, Standard Bangla, English, and subjectivity annotations across five regional dialects. Experiments show LoRA adaptation with limited dialect-aligned supervision can outperform closed-source prompting, while transliteration degrades current LLM performance.

  • 5-DIALECTS-BN aligns Romanized transliteration with Standard Bangla, English, and subjectivity labels across five regional dialects and 6,000 entries.
  • LoRA fine-tuning with only 160 examples per dialect surpasses closed-source models in the reported experiments.
  • Romanized transliteration consistently degrades current LLMs through many-to-one information loss and token fragmentation.

Limitations

The dataset reflects naturally uneven online availability, transliteration lacks standardized conventions, and the reported adaptation experiments cover limited budgets, models, tasks, and written online varieties.

  • The dataset is imbalanced across dialects, with 1,900 Chittagong entries versus 700 each for Barisal and Rangpur, which may bias future cross-dialect transfer studies.
  • Bangla dialects lack fixed transliteration rules, so orthographic variance should be considered when comparing transliterated resources.
  • The LoRA experiments use 160 instances per dialect and one hyperparameter configuration, without a data-scaling curve.
  • Normalization receives limited supervised evaluation, larger Indic foundation models remain future work, and written online dialectal Bangla may differ from spoken varieties.

Ethical Considerations

The dataset uses publicly accessible, ordinary user-generated content and documents dialectal variation across regional Bangla varieties. Its construction applies filtering, native-speaker verification, deduplication, and agreement-checked release procedures.

  • All data came from publicly accessible online sources, excluding private, personally identifying, and sensitive information.
  • The five dialects differ systematically from Standard Bangla in phonological, lexical, and morphological features.
  • Chittagong shows the strongest divergence from Standard Bangla and the highest residual error after LoRA adaptation.
  • Rangpur is closest to Standard Bangla in lexical overlap and produces the strongest zero-shot baseline.
  • The dataset was filtered through language identification, length screening, native-speaker verification, and exact or near-match deduplication.
  • After filtering, 6,000 utterances remained, with distribution reflecting natural online availability.

E Inter-Annotator Agreement & Quality Controls

Agreement checks support annotation reliability, while targeted analyses identify boundary ambiguity and model-specific prompting failures. Demonstration interference is especially pronounced for Llama-3.1-8B and is not shared by the comparison models.

  • Inter-Annotator Agreement: 1,500 entries, or 300 per dialect, were independently re-annotated by a second annotator from the same dialect pool.
  • Inter-Annotator Agreement: Cohen’s κ was 0.78 for subjectivity, 0.71 for English translation acceptance, and 0.74 for Standard Bangla rendering acceptance.
  • Boundary Cases: Chittagong–Noakhali disagreements concern variety assignment rather than transcription because canonical renderings diverge despite shared innovations.
  • Dialect Difficulty: Chittagong is the hardest dialect across normalization, chain-of-thought transliteration, and Indic seq2seq conditions, with a 61.0% empty-output rate in Appendix M.
  • Inter-Annotator Agreement: English translation agreement reached mean ROUGE-L 0.84 overall, ranging from 0.79 to 0.91 across dialects.
  • Prompting Diagnostics: Llama-3.1-8B shows a 5.9% correct-to-unrelated flip rate under few-shot prompting, compared with 0.2–0.4% for other models.
  • Prompting Diagnostics: Llama-3.1-8B falls from 54.63 to 38.64 macro chrF++ after one demonstration per dialect, while additional demonstrations only partially recover performance.
  • Prompting Diagnostics: Gemma-4-4B and Mistral-7B improve with 30 demonstrations, whereas Llama remains 10.83 chrF++ below its zero-shot score.

H.3 Chain-of-Thought Error Decomposition

The analysis decomposes prompting and script effects, showing that CoT translation drops mainly through surface paraphrase drift, while Romanization damages performance through subword fragmentation rather than context exhaustion. Back-transliteration and scheme comparisons indicate that restoring script or changing convention does not remove the penalty.

  • CoT error decomposition: Median length ratio is 1.0 across regimes, while surface paraphrase drift dominates the CoT translation drop across closed models.Parsing failures affect only GPT-4o-mini at 6.3%.
  • CoT error decomposition: Qwen-3-4B is the sole exception where CoT improves scores by repairing its 91.4% zero-shot format collapse.
  • Tokenizer analysis: Romanized text is token-compact at approximately 0.4 tokens per character, ruling out context exhaustion as the explanation for the script penalty.
  • Tokenizer analysis: The positive, statistically significant token-inflation regression coefficient across open models under LoRA identifies subword fragmentation as a per-item marker of damaged mapping.The reported significance threshold is p ≤0.035.
  • Round-trip control: Native-script recovery after inverse transliteration restores essentially none of the performance penalty, with reconstruction CER of 0.14–0.21 and exact match of only 2–14%.
  • Scheme robustness: The transliteration penalty persists across Avro, ITRANS, and ISO-15919, although Avro exhibits the mildest degradation.

K Feature-Conditioned Linguistic Error Analysis

Feature-conditioned analysis separates lexical divergence from phonological shifts: non-cognate lexical differences remain difficult after LoRA, whereas the reported phonological shift is solved.

  • Romanization robustness: The robustness check finds transliteration degradation across Avro, ITRANS, and ISO-15919, with Avro showing the mildest degradation.
  • Lexical divergence: 16.2% CompleteMistranslation persists for non-cognate lexical divergence after LoRA adaptation.The example is Chittagong goijja-class verbs.
  • Phonological divergence: 0% error is reported for the analyzed Barisal phonological hC-cluster shifts after LoRA adaptation.

M Indic-Tuned Baseline Experiments

Indic-tuned sequence-to-sequence baselines were evaluated on fold-matched dialect translation and normalization tasks, but IndicBART translation was dominated by empty outputs rather than ordinary mistranslation. Its non-empty outputs remained semantically incorrect despite being fluent.

  • Evaluation design: The baseline comparison uses identical 160-per-dialect training folds and a shared held-out test fold of 100 items per dialect.The matched evaluation covers 500 test items across five dialects.
  • Checkpoint selection: The reported baseline checkpoints were selected to avoid a BanglaT5 vocabulary/configuration mismatch that stalls fine-tuning.The excluded base checkpoint has training loss around 9–11, compared with approximately 1.4–2.0 for the reported checkpoints.
  • Translation baseline: IndicBART emits an empty string on 253 of 500 translation test items, yielding a 50.6% empty rate.The empty rate ranges from 26.0% for Barisal to 61.0% for Chittagong and Sylhet.
  • Translation baseline: Among IndicBART’s 247 non-empty outputs, translation reaches 22.66 chrF++ and 14.97 BLEU, with fluent but semantically incorrect hypotheses.Non-empty hypotheses average 22.9 characters versus 27.3-character references.
  • Failure interpretation: IndicBART’s 11.19 chrF++ translation score is dominated by degenerate generation rather than ordinary mistranslation.
Loading 2609.09964v1…