Source-linked AI summary
AraBERT: Transformer-based Model for Arabic Language Understanding
Wissam Antoun, Fady Baly, Hazem Hajj
TL;DR
Arabic NLP is constrained by limited resources and linguistic variation, motivating a language-specific contextual model. The paper pretrains BERT on a large Arabic corpus with Arabic-oriented processing and evaluates it across three NLU tasks. ARABERT achieves state-of-the-art performance on most tested datasets and is publicly released.
Problem
Arabic NLP tasks are challenging because Arabic has relatively few resources, rich morphology, and less explored syntax than English.
Method
The paper pretrains BERT for Arabic using a large corpus, Arabic-specific segmentation, and SentencePiece subword modeling, then applies it to three NLU tasks.
Results
ARABERT achieves state-of-the-art performance on most evaluated Arabic NLU datasets compared with multilingual BERT and other baselines.
Takeaways & Limitations
The publicly released ARABERT models provide an Arabic language representation model intended to support Arabic NLP research and applications.
Takeaways & Limitations
Arabic QA evaluation is constrained by inconsistent name spelling, including different spellings between questions and answers.
Abstract
from arXiv · showhide
The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA), have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language-specific BERT based models have proven to be very efficient at language understanding, provided they are pre-trained on a very large corpus. Such models were able to set new standards and achieve state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state-of-the-art approaches. The results showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https://github.com/aub-mind/arabert hoping to encourage research and applications for Arabic NLP.
1. Introduction
The paper introduces ARABERT, a BERT model pretrained specifically for Arabic, and evaluates it across three Arabic NLU tasks. ARABERT achieves state-of-the-art performance on most datasets and is publicly released.
- ARABERT is a BERT model pretrained for Arabic language understanding.
- ARABERT is compared with multilingual BERT and other single-language and task-specific baselines.
- The downstream datasets include both Modern Standard Arabic and Dialectal Arabic.
- The evaluation covers Sentiment Analysis, Named Entity Recognition, and Question Answering.
- The paper publicly releases ARABERT through popular NLP libraries.
2. Related Works
Prior Arabic representation research progressed from static word embeddings to contextualized and language-specific models. Multilingual BERT supports many languages, while monolingual BERT models have often provided stronger performance for individual languages.
- Static embeddings improved over time but lacked contextual information, motivating contextualized representation models.
- Early Arabic representation work included word2vec, FastText, and multidialectal embeddings.
- Multilingual BERT supports more than 100 languages and performs solidly across many languages.
- Monolingual BERT models for non-English languages have shown better performance than multilingual BERT in some cases.
- Arabic-specific contextualized models such as hULMonA preceded ARABERT.
3. ARABERT: Methodology
ARABERT adapts BERT to Arabic through large-scale pretraining, Arabic-specific preprocessing, and task-specific fine-tuning. Its corpus combines diverse Arabic news sources, while segmentation and subword modeling address Arabic morphological variation.
- Pretraining objectives: Pretraining uses masked language modeling with whole-word masking and next sentence prediction.
- Pre-training Dataset: The pretraining dataset contains 70 million deduplicated sentences, corresponding to approximately 24GB of text from Arabic news corpora and scraped websites.
- Sub-Word Units Segmentation: Arabic words are segmented into stems, prefixes, and suffixes with Farasa before SentencePiece produces a subword vocabulary of approximately 60K tokens.
- Sub-Word Units Segmentation: A non-segmented AraBERTv0.1 variant is trained to evaluate the impact of Arabic-specific segmentation.
- Downstream fine-tuning: For sequence classification, the [CLS] representation feeds a softmax classifier trained jointly with the pretrained model.
- Downstream fine-tuning: NER uses IOB2 token labels and inputs only the first sub-token of each word after tokenization.
- Downstream fine-tuning: QA predicts answer-span start and end tokens using classifiers applied to token embeddings.
4. Evaluation
The evaluation benchmarks ARABERT on Arabic sentiment analysis, named entity recognition, and question answering against multilingual BERT and prior approaches. The datasets span multiple genres, dialects, and Arabic QA settings, including ARCD and Arabic-SQuAD.
- ARABERT is evaluated on Sentiment Analysis, Named Entity Recognition, and Question Answering against multilingual BERT and other state-of-the-art results.
- Sentiment Analysis: The sentiment evaluation covers datasets spanning different genres, domains, and dialects.
- Sentiment Analysis: HARD contains 93,700 hotel reviews in Modern Standard Arabic and dialectal Arabic, labeled as positive or negative by rating.
- Sentiment Analysis: ArSenTD-Lev contains 4,000 Levantine tweets covering multiple domains and topics.
- Named Entity Recognition: NER is framed as word-level classification over entity categories and evaluated on ANERcorp, which contains 16.5K entity mentions.
- Question Answering: Arabic QA is challenged by limited massive datasets and linguistic variation, including inconsistent name spelling.
- Question Answering: QA evaluation uses ARCD and Arabic-SQuAD, training on all Arabic-SQuAD and half of ARCD before testing on the remaining half.
- Question Answering: Multilingual BERT had previously achieved state-of-the-art results on ARCD.
5. Experiments
AraBERT was evaluated across Arabic sentiment analysis, NER, and question answering against multilingual BERT and prior systems. It achieved strong results across tasks, while revealing distinct strengths and limitations in NER and QA.
- Experimental Setup: Fine-tuning used the same configuration across tasks without extensive hyperparameter search because of computational and time constraints.
- AraBERT was evaluated on Arabic sentiment analysis, named entity recognition, and question answering against mBERT and previous state-of-the-art systems.
- Sentiment Analysis: Both AraBERT versions outperformed mBERT and other state-of-the-art approaches on most tested sentiment-analysis datasets, including unseen dialects despite MSA training.
- Named Entity Recognition: AraBERTv0.1 improved NER by 2.53 F1 points, reaching 84.2 against the Bi-LSTM-CRF baseline and establishing a new state of the art on ANERcorp.
- Named Entity Recognition: AraBERT’s NER results with tokenized prefixes and suffixes were similar to Bi-LSTM-CRF, while mBERT scored below the baseline.
- Question Answering: QA improved F1 but had significantly lower exact-match scores; sentence match increased by 2% over mBERT, the previous state of the art.Sentence Match counts predictions appearing in the same sentence as the ground-truth answer.
- Discussion: AraBERT’s discussion attributes performance gains to larger and more diverse pretraining data, a larger vocabulary, Arabic-aware preprocessing, and task-dependent segmentation effects.The paper reports approximately 24GB of pretraining data versus 4.3G for multilingual BERT, and a 64k versus 2k vocabulary.
6. Conclusion
AraBERT establishes new state-of-the-art performance for several Arabic downstream tasks and is publicly released to support future Arabic NLP research and applications.
- AraBERT sets a new state of the art for several Arabic language downstream tasks.
- The publicly released models are intended to serve as baselines for Arabic NLP tasks and support future Arabic language-understanding models.
- The authors are developing versions without external tokenizers and models with better understanding of Arabic dialects across countries.