Source-linked AI summary
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Kazi Samin, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, Rifat Shahriyar
TL;DR
Bangla is resource-scarce, and multilingual pretrained models are computationally demanding while often performing worse on low-resource languages. The paper introduces BanglaBERT, BanglishBERT, new Bangla NLI and QA datasets, and the BLUB benchmark; BanglaBERT outperforms multilingual and monolingual baselines in supervised evaluation.
Problem
Bangla NLU relies on large multilingual pretrained models that require substantial fine-tuning resources and tend to perform worse for low-resource languages.
Method
The authors pretrain BanglaBERT and BanglishBERT, introduce Bangla NLI and QA datasets, and evaluate them with multilingual models across four tasks in the BLUB benchmark.
Results
6.8 BLUB scores: BanglaBERT outperforms mBERT in supervised evaluation, while also outperforming multilingual models and monolingual sahajBERT across all tasks.
Takeaways & Limitations
BanglaBERT establishes state-of-the-art Bangla NLU results, and BLUB provides a benchmark for evaluating Bangla language understanding models.
Takeaways & Limitations
The pretraining data may contain objectionable content despite efforts to minimize it, so the model should be used carefully, especially for text generation.
Abstract
from arXiv · showhide
In this work, we introduce BanglaBERT, a BERT-based Natural Language Understanding (NLU) model pretrained in Bangla, a widely spoken yet low-resource language in the NLP literature. To pretrain BanglaBERT, we collect 27.5 GB of Bangla pretraining data (dubbed `Bangla2B+') by crawling 110 popular Bangla sites. We introduce two downstream task datasets on natural language inference and question answering and benchmark on four diverse NLU tasks covering text classification, sequence labeling, and span prediction. In the process, we bring them under the first-ever Bangla Language Understanding Benchmark (BLUB). BanglaBERT achieves state-of-the-art results outperforming multilingual and monolingual models. We are making the models, datasets, and a leaderboard publicly available at https://github.com/csebuetnlp/banglabert to advance Bangla NLP.
1 Introduction
Bangla is widely spoken but resource-scarce, and existing multilingual models are costly and often weaker on low-resource NLU. The paper introduces Bangla-focused models, datasets, and BLUB to address this gap.
- Motivation: Bangla is resource-scarce because labeled-data collection and language-specific pretraining efforts remain limited.Bangla NLU has largely relied on fine-tuning multilingual pretrained language models.
- Motivation: Multilingual pretrained models require substantial computational resources and tend to perform worse for low-resource languages.These models cover many languages and consequently have hundreds of millions of parameters.
- Contributions: BanglaBERT is a BERT-based Bangla NLU model pretrained on 27.5 GB collected from 110 popular Bangla websites.The corpus is named Bangla2B+.
- Contributions: The paper introduces Bangla NLI and QA datasets and evaluates models across sentiment classification, NLI, named entity recognition, and QA.These tasks cover text classification, sequence labeling, and span prediction.
- Contributions: The four tasks are unified into the first-ever Bangla Language Understanding Benchmark, BLUB.The benchmark supports comparisons between Bangla-specific and multilingual models.
- Results: BanglaBERT outperforms mBERT and XLM-R (base) by 6.8 and 4.3 BLUB scores in supervised evaluation.BanglishBERT outperforms them by 15.8 and 10.8 scores in zero-shot cross-lingual transfer.
2 BanglaBERT
BanglaBERT is built from a large, filtered Bangla web corpus and pretrained with ELECTRA’s replaced-token detection objective. A bilingual variant supports zero-shot transfer between Bangla and English.
- 2.1 Pretraining Data: Bangla web resources are constrained: the July 2021 Bangla Wikipedia dump was only 650 MB, two orders of magnitude smaller than English Wikipedia.The shortage motivated extensive web crawling.
- 2.1 Pretraining Data: The authors selected 110 Bangla websites by rankings and inspected text volume and quality across encyclopedias, news, blogs, e-books, stories, and forums.The initial collection totaled around 35 GB.
- 2.1 Pretraining Data: They excluded noisy OSCAR and CCNet dumps because they contained many offensive texts that were infeasible to clean thoroughly.The authors cite concern about potentially harmful impacts.
- 2.1 Pretraining Data: Deduplication, removal of non-textual content, and language filtering reduced the corpus to 27.5 GB and 5.25M documents.The documents average 306.66 words.
- 2.1 Pretraining Data: The processed corpus became 7.18M training samples containing 2.18B tokens and was named Bangla2B+.Samples average 304.14 tokens and do not cross document boundaries.
- 2.3 Pretraining Objective: BanglaBERT uses ELECTRA’s replaced token detection objective, jointly training a generator and discriminator before fine-tuning the discriminator.The generator masks 15% of tokens, while the discriminator predicts whether each token is original or replaced.
- BanglishBERT: BanglishBERT is pretrained bilingually on Bangla and English to enable zero-shot cross-lingual transfer when task labels are available mainly in English.The approach targets settings where labeled Bangla data may be unavailable.
3 The Bangla Language Understanding Benchmark (BLUB)
Bangla NLU research has addressed individual tasks but lacked a comprehensive, unified evaluation. BLUB responds by bringing diverse Bangla NLU tasks into one benchmark.
- Research Gap: Prior Bangla NLU studies examined tasks such as sentiment classification, semantic textual similarity, part-of-speech tagging, and named entity recognition in isolation.The paper identifies the absence of a comprehensive unified study.
1. Single-Sequence Classification
The benchmark includes established sentiment classification and newly curated Bangla natural language inference to cover both single-sequence and sequence-pair classification.
- Single-Sequence Classification: Sentiment classification was selected as the single-sequence task because it is among the most studied Bangla NLU tasks.Only two public datasets were identified, and BYSA contained duplicated examples with inconsistent labels.
- Sequence-Pair Classification: Because Bangla had few publicly available sequence-pair datasets, the authors curated BNLI as a representative natural language inference task.The task predicts entailment, contradiction, or neutrality between a premise and hypothesis.
2. Sequence-pair Classification
The benchmark includes sequence-labeling and extractive question-answering tasks, with QA framed as answerability classification followed by minimal-span prediction.
- Sequence Labeling: Named Entity Recognition uses the Bangla portion of the SemEval 2022 MultiCoNER dataset.
- Span Prediction: Extractive QA uses machine-translated SQuAD 2.0 for training and Bangla TyDiQA validation and test data.
- Span Prediction: Answerable questions require predicting the minimal text span that answers the question, while unanswerable questions must be identified.
4. Span Prediction
The study evaluates pretrained models across four downstream tasks and reports supervised and zero-shot performance, including sample-efficiency comparisons on sentiment classification and NLI.
- Setup: BanglaBERT and BanglishBERT are fine-tuned on four tasks and compared with multilingual models, sahajBERT, and XLM-R variants.
- Setup: The BLUB score is the average performance across all tasks.
- Zero-shot Transfer: In zero-shot transfer, BanglishBERT performed strongly among similarly sized models but fell marginally short of XLM-R large.
- Supervised Fine-tuning: 77.09 BLUB score: BanglaBERT outperformed multilingual models and sahajBERT across all supervised tasks, matching XLM-R large.
- Supervised Fine-tuning: 2-3.5x time and 2.4-3.33x memory: sahajBERT used more time and memory than BanglaBERT despite its smaller size.
- Sample efficiency: 2-9% on SC and 6-10% on NLI with p < 0.05: BanglaBERT outperformed XLM-R large when training used 1k or fewer samples.
5 Conclusion & Future Works
The paper addresses the shortage of strong Bangla pretrained models by introducing BanglaBERT, BanglishBERT, new datasets, and the BLUB benchmark.
- Conclusion: BanglaBERT and BanglishBERT are introduced as NLU models for Bangla, a widely spoken yet low-resource language.
- Conclusion: The work presents new Bangla NLI and QA datasets and establishes the BLUB benchmark.
- Conclusion: BanglaBERT sets new state-of-the-art results according to the conclusion.
- Future Works: Future work will add other Bangla NLU benchmarks, such as dependency parsing, and investigate initializing Bangla NLG models from BanglaBERT.
Ethical Considerations
The paper describes a non-commercial release plan for models and datasets while acknowledging that objectionable content may remain in the pretraining data.
- Dataset and Model Release: BanglaBERT will be released under a non-commercial license, with downstream datasets likewise planned for public non-commercial release.
- Dataset and Model Release: Only pretraining data whose distribution is known not to cause copyright infringement will be released.
- Quality Control in Human Translation: Expert translators produced and reviewed translations, with low-quality samples retranslated or discarded.
- Text Content: The authors cannot guarantee that the pretraining data contains no objectionable content and recommend careful model use, especially for text generation.
Appendix
The appendix documents the construction of the Bangla pretraining corpus by crawling diverse Bangla websites with custom site-specific crawlers.
- The data sources were categorized into six types.
- Sources included encyclopedic, news, blog, literary, informational, and other Bangla-language websites.
- Custom crawlers were written for each listed site, except Wikipedia dumps.
Additional Sample Efficiency Tests
Additional experiments examine BanglaBERT’s sample efficiency on NER and QA, alongside comparisons of model training time and memory usage.
- BanglaBERT is more sample-efficient than the comparison model for NER with ≤1k training samples.
- Both models have identical performance on QA across all tested sample counts.
- Training-time and GPU-memory usage were measured during fine-tuning under matched batch size, accumulation, and sequence-length settings.