Source-linked AI summary
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
Muhammad Abdul-Mageed, AbdelRahim Elmadany, El Moatez Billah Nagoudi
TL;DR
Existing multilingual and Arabic language models provide incomplete coverage of diverse Arabic varieties and realistic evaluation settings. The paper introduces ARBERT, MARBERT, and ARLUE, a 42-dataset benchmark, and reports state-of-the-art performance across most tasks. MARBERT-v2 achieves the highest ARLUE score, while earlier models' short pre-training sequences limit question-answering context.
Problem
Existing language models have limited coverage of Arabic dialects, social-media language, and native non-English evaluation settings.
Method
The paper introduces ARBERT and MARBERT, Arabic-focused Transformer language models, and ARLUE, a standardized benchmark spanning 42 datasets and six task clusters.
Results
The models achieve new state-of-the-art results on 37 of 48 individual datasets, while MARBERT-v2 obtains the highest ARLUE score of 77.40.
Takeaways & Limitations
The models and benchmark support evaluation of Arabic language understanding across domains, communities, dialects, and task clusters.
Takeaways & Limitations
Earlier models' sequence length of 128 may not sufficiently capture both a question and its likely answer during pre-training.
Abstract
from arXiv · showhide
Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (~ 3.4 x larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.
1 Introduction
The paper addresses limitations of multilingual and existing Arabic language models by introducing Arabic-focused models and ARLUE, a broad benchmark for standardized multi-dialectal evaluation.
- Motivation: Multilingual language models can be costly and are often outperformed by monolingual models trained with larger language-specific datasets.Existing models may also inadequately capture social-media language and diverse Arabic dialects.
- Motivation: Existing benchmarks are often English-only or rely on machine translation, limiting evaluation on native non-English and cross-lingual data.The paper identifies meaningful cross-task comparisons under realistic language conditions as an evaluation need.
- Motivation: Arabic offers a challenging setting because it includes MSA and many regionally distinct dialects, while existing Arabic models have limited dialect coverage and narrow evaluation.The paper uses this setting to compare monolingual and multilingual models under standardized conditions.
- Contributions: ARLUE standardizes 42 datasets into six Arabic language-understanding task clusters and reports both dataset-level and aggregate results.The benchmark is intended to facilitate progress measurement for Arabic and multilingual language models.
- Contributions: ARBERT and MARBERT are Arabic-specific Transformer language models trained on very large and diverse datasets for MSA and Arabic dialects.The models target transfer learning across standard and dialectal Arabic varieties.
- Results: The models achieve new state-of-the-art results on 37 of 48 individual datasets and a state-of-the-art ARLUE score.The evaluation compares the models with available alternatives across the benchmark.
2 Arabic LMs
Arabic encompasses multiple varieties and dialects, and prior Arabic language models include AraBERT and GigaBERT with differing training resources and intended coverage.
- Arabic varieties: Arabic comprises a collection of languages, language varieties, and dialects, with Modern Standard Arabic as its standard variety.Many dialects are commonly defined by region or country.
- Existing Arabic LMs: AraBERT is a BERTBase Arabic language model trained on 23GB of Arabic text from Wikipedia, news, and other corpora.Its evaluations cover sentiment analysis, named entity recognition, and Arabic question answering.
- Existing Arabic LMs: GigaBERT is an Arabic-English language model designed with code-switching data in mind, but the paper does not compare against it.The authors note that MARBERT uses more and more diverse pre-training data than GigaBERT.
3 Our Models
ARBERT and MARBERT use large, diverse Arabic pre-training data, while downstream evaluation fine-tunes them across six task clusters under standardized experimental conditions.
- ARBERT training: The pre-training data combines multiple domains and sources, including MSA, Egyptian Arabic, newswire, books, Wikipedia, and large Arabic news corpora.OSCAR contributes MSA and Egyptian Arabic from Common Crawl, while OSIAN contains 3.5 million articles from 31 sources in 24 countries.
- Preprocessing: ARBERT uses light preprocessing that removes diacritics and replaces URLs, user mentions, and hashtags before WordPiece segmentation.The procedure is intended to retain naturally occurring text while normalizing selected tokens.
- Pre-training: ARBERT follows BERT pre-training with whole-word masking, a BERTBase architecture, 12 layers, 768 hidden units, 12 heads, and approximately 163M parameters.Training uses maximum sequences of 128 tokens for approximately 42 epochs over 6.5B tokens.
- MARBERT training: MARBERT is trained from scratch on 1B Arabic tweets sampled from an in-house collection of about 6B tweets, totaling 15.6B tokens.It preserves non-Arabic strings when tweets contain at least three Arabic words.
- MARBERT training: MARBERT uses the BERTBase architecture without next sentence prediction, with a 100K WordPiece vocabulary, approximately 160M parameters, and 128-token sequences.Training lasts approximately 36 epochs and takes about 40 days on an eight-core TPU.
- Fine-tuning and evaluation: The models are compared with mBERT, XLM-R, and AraBERT across sentiment, social meaning, topic classification, dialect identification, NER, and question answering.The evaluation fine-tunes models on individual tasks and task clusters using common experimental settings.
4 Individual Downstream Tasks
The models are evaluated across sentiment analysis, social meaning, topic classification, dialect identification, named entity recognition, and question answering. They perform strongly on most classification tasks, especially dialect identification, but are not competitive on question answering without longer-sequence pre-training.
- Sentiment Analysis: 13 of 17 sentiment-classification tasks achieve the best results with ARBERT or MARBERT, while XLM-R leads the remaining four.XLM-R is substantially larger, and MARBERT outperforms ARBERT, likely reflecting its larger and more diverse pre-training data, including social-media text.
- Social Meaning Tasks: Social meaning covers age and gender, dangerous, hateful, and offensive speech, emotion, irony, and sarcasm detection.The evaluation uses datasets spanning these task types and reports F1 scores.
- Topic Classification: ARBERT achieves the best results on OSAC, Khaleej, and title-only ANT topic-classification settings.AraBERT slightly outperforms the authors’ models on ANT text-only and title+text settings.
- Dialect Identification: The models outperform all SOTA systems and AraBERT across dialect-identification levels, including binary, regional, country, and province distinctions.The results are attributed to MARBERT’s dialectal representation, while ARBERT also outperforms the other models despite being developed mainly for MSA.
- Named Entity Recognition: The models outperform SOTA on two of five named entity recognition datasets.The comparison focuses on person, location, and organization labels, with other labels mapped to the unnamed entity class.
- Question Answering: MARBERT-v2 achieves the best question-answering performance on all but one test set after further pre-training with 512-token sequences.XLM-RLarge marginally outperforms MARBERT-v2 on that remaining test set in F1; the original models were not competitive on QA.
5 ARLUE
ARLUE combines Arabic language-understanding datasets into standardized task clusters and summarizes performance with a macro-averaged score. MARBERT-v2 achieves the highest reported overall score, while MARBERT is constrained by short input sequences on question answering.
- ARLUE construction: ARLUE concatenates corresponding dataset splits into shared TRAIN, DEV, and TEST sets, except social meaning datasets, which remain independent.
- ARLUE construction: ARLUE includes separate dialect, named entity recognition, and question answering clusters formed by concatenating corresponding dataset splits.
- Scoring: The ARLUE score is the macro-average across task-cluster scores, weighting each task equally and averaging multiple metrics within tasks.
- Results: 77.40 is MARBERT-v2’s ARLUE score, exceeding XLM-RL at 76.55 and ARBERT at 76.07.
- Caveat: MARBERT’s overall ranking is limited by weaker question-answering performance associated with its short input sequence length.
6 Related Work
The paper situates Arabic evaluation within a broader progression from general language benchmarks to multilingual transfer benchmarks. It identifies ARLUE as an Arabic-focused evaluation framework with cluster-level metrics and standardized test reporting.
- Benchmark context: GLUE evaluates nine NLP tasks, while SuperGLUE covers seven more challenging tasks.
- ARLUE evaluation: ARLUE reports development and test performance independently for each task cluster using the relevant evaluation metric.
- ARLUE evaluation: ARLUESM summarizes social meaning performance as an average across its component tasks, while ARLUEQA uses Exact Match and F1.
- Benchmark context: XTREME evaluates cross-lingual transfer learning across nine tasks and 40 languages.
7 Conclusion
The conclusion presents ARBERT, MARBERT, and MARBERT-v2 as Arabic Transformer language models trained on broad data spanning domains and genres, including social media. The best models perform better than or on par with the much larger XLM-RLarge.
- ARBERT and MARBERT are Transformer-based Arabic language models trained on large-to-massive datasets covering different domains and text genres.
- MARBERT and MARBERT-v2 are pre-trained on dialectal Arabic to support downstream NLP technologies serving wider and more diverse communities.
- The best models perform better than or on par with XLM-RLarge, which is approximately 3.4× larger.
Appendices
The appendices document the sentiment datasets, baselines, label conventions, and evaluation setups used in the paper. They cover varied Arabic sources, including reviews, tweets, newswire, Wikipedia, and web fora.
- Sentiment datasets: AraNETSent combines 15 MSA and dialect datasets with binary and three-way sentiment labels.
- Sentiment datasets: AraSenTi-Tweet contains 17,573 gold-labeled MSA and Saudi Arabic tweets.
- Sentiment datasets: ArSarcasmSent derives sentiment-labeled sarcasm data from ASTD and SemEval-2017 Twitter datasets.
- Sentiment datasets: The sentiment collection includes datasets from hotel reviews, book reviews, tweets, newswire, Wikipedia, web fora, and Arabic speech-act corpora.
- Baselines and metrics: The paper compares sentiment results using prior studies’ reported setups, including dataset splits and metrics such as accuracy and F1PN.
- ARLUE construction: ARLUESenti merges sentiment datasets by collapsing very negative and very positive labels and removing the mixed class.
A.3 SA Evaluation on DEV
The section identifies the development-set sentiment-analysis evaluation and its model-comparison table. Results are reported using F1.
- Table A.2 reports sentiment-analysis results on development splits for datasets that provide development data.
- The comparison includes mBERT, XLM-RB, XLM-RL, AraBERT, ARBERT, and MARBERT.
- The sentiment-analysis results are measured with F1.
B.1 SM Tasks & Datasets
The section describes social-meaning tasks evaluated on Arabic datasets, including demographic, safety, offensive-language, emotion, irony, and sarcasm tasks. It also identifies prior systems and reports their comparison scores.
- SM Tasks & Datasets: Social-meaning evaluation covers age, gender, dangerous speech, offensive language, hate speech, emotion, irony, and sarcasm.
- SM Tasks & Datasets: The Arap-Tweet dataset covers 17 countries from 11 Arab regions and uses an 80-10-10 split.
- SM Tasks & Datasets: The dangerous-speech dataset contains manually labeled tweets tagged as dangerous or safe.
- SM Tasks & Datasets: The offensive-speech shared task separates offensive-versus-not-offensive detection from hate-speech-versus-not-hate-speech detection.
- Prior Comparisons: Prior comparison scores include 51.42 and 65.30 F1 for age and gender, respectively, and 60.32 F1 for emotion.
- Prior Comparisons: Reported prior baselines include 59.60 F1 for dangerous speech, 82.4 F1 for irony, and 46.00 F1 for sarcasm.
C.1 TC Datasets
The section introduces text-classification datasets spanning Arabic news and web domains, alongside dialect-identification datasets covering regional, country, and province levels. It also lists prior classification results and evaluation tables.
- TC Datasets: The text-classification datasets include Arabic News Text, Khaleej, and OSAC.
- TC Datasets: Arabic News Text is built from transcribed Tunisian radio broadcasts, Khaleej from Gulf Arabic websites, and OSAC from news articles.
- Evaluation: Evaluation results are reported for text classification and dialect identification on development data in Tables C.2 and D.2.
- Dialect Identification: The dialect datasets cover MSA and three regional dialects in AOC, 21 countries in MADAR and NADI country-level data, 100 provinces in NADI, and 18 countries in QADI.
- Dialect Identification: AOC contains 3M Arabic online-news comments labeled as MSA, Egyptian, Gulf, or Levantine.
- Prior Results: Prior systems report 87.23 accuracy for MSA versus dialectal Arabic and 87.81 accuracy for regional classification on AOC.
E.1 NER datasets
The section documents the Arabic named-entity-recognition and question-answering datasets used in evaluation. It specifies prior NER results and the origins or multilingual coverage of the QA benchmarks.
- NER Datasets: Tables E.1 and E.2 provide NER dataset distributions and development-split results for the models.
- NER Baselines: Khalifa and Shaalan report F1 scores of 88.77, 91.47, 94.92, 91.20, and 65.34 on ANERcorp, ACE03NW, ACE03BN, ACE04NW, and Twitter, respectively.
- Question Answering: ARCD was developed through crowdsourcing, and the evaluation uses the same splits as Antoun et al.
- Question Answering: MLQA contains over 5K extractive question-answer instances in SQuAD format across seven languages, including Arabic.
- Question Answering: XQuAD contains 1,190 question-answer pairs and 240 SQuAD v1.1 paragraphs translated into ten languages, including Arabic.
- Question Answering: TyDi QA is manually curated across 11 languages, including Arabic, and the evaluation focuses on its Gold passage task.