Source-linked AI summary
AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding
Wissam Antoun, Fady Baly, Hazem Hajj
TL;DR
Arabic language representation models have primarily relied on masked language modeling, motivating a more efficient pre-training objective. The paper develops AraELECTRA with replaced token detection and evaluates it across Arabic NLP tasks, reporting improved performance over existing models, including larger ones.
Problem
Arabic language representation models primarily use masked language modeling, while ELECTRA offers a more sample-efficient replaced token detection objective.
Method
AraELECTRA pre-trains an Arabic transformer with a generator that creates corrupted tokens and a discriminator that identifies replacements.
Results
AraELECTRA achieves the highest performance on all tested datasets against base models, with limited exceptions on ARCD compared with larger models.
Takeaways & Limitations
RTD pre-training improves Arabic language representation and performance across question answering, sentiment analysis, and named-entity recognition compared with MLM-based models.
Takeaways & Limitations
Evaluation is constrained by noisy or limited datasets, including translated ARCD examples, HTML artifacts, and a small imbalanced sentiment dataset.
Abstract
from arXiv · showhide
Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
1 Introduction
Arabic language models have largely used masked language modeling, while ELECTRA’s replaced token detection offers a more efficient alternative. AraELECTRA applies this objective to Arabic and evaluates it across multiple NLP tasks.
- Motivation: Masked language modeling learns only from randomly hidden tokens, limiting its sample efficiency.The passage contrasts this with ELECTRA’s use of replaced token detection.
- ELECTRA: ELECTRA distinguishes original tokens from synthetically generated replacements using a generator and discriminator.The generator produces corrupted tokens, while the discriminator classifies each token as original or replaced.
- Arabic Language Modeling: Current state-of-the-art Arabic language representation models use masked language modeling as their pre-training objective.
- AraELECTRA: AraELECTRA pre-trains an Arabic transformer encoder with the replaced token detection objective and evaluates it on multiple Arabic NLP tasks.
- Contributions: The paper reports a new state-of-the-art on multiple Arabic NLP tasks and publicly releases AraELECTRA through popular NLP libraries.
2 Related Works
Arabic language representation research progressed from word embeddings and recurrent language models to transformer models pretrained with masked language modeling. This work proposes an Arabic ELECTRA model pretrained with replaced token detection on large MSA corpora.
- Related Works: Earlier Arabic transfer learning used pretrained word embeddings such as AraVec.
- Related Works: Later approaches introduced recurrent and transformer-based Arabic language models, with transformers pretrained using masked language modeling on large corpora.
- Proposed Approach: The paper proposes an Arabic ELECTRA-based language representation model pretrained with replaced token detection on large MSA corpora.
3 ARAELECTRA: Methodology
AraELECTRA is a bidirectional transformer encoder pretrained with ELECTRA’s replaced token detection objective. Its generator masks and predicts tokens, while its discriminator identifies replacements before the encoder is fine-tuned on Arabic NLP tasks.
- 3 ARAELECTRA: Methodology: ARAELECTRA is a bidirectional transformer encoder with 12 layers, 12 attention heads, 768 hidden units, and 136M parameters.It supports input sequences up to 512 tokens.
- 3.1 Pre-training Setup: The replaced token detection setup trains a generator to recover masked tokens and a discriminator to identify replaced versus original tokens.The generator is a small BERT model, while the discriminator is AraELECTRA.
- 3.1 Pre-training Setup: Unlike adversarial GAN training, ELECTRA trains its generator with maximum likelihood on corrupted token sequences rather than random noise.
- 3.2 Pre-training Dataset: AraELECTRA uses the same pre-training dataset and wordpiece vocabulary as ARABERTv0.2 to make model comparisons fair.
- 3.2 Pre-training Dataset: The training corpus contains 77GB and 8.8 billion words, comprising mostly news articles.Validation uses Wikipedia articles published after the September 2020 dump.
- 3.3 Fine-tuning: AraELECTRA adds a linear classification layer and fine-tunes the full discriminator on question answering, sentiment analysis, and named-entity recognition.
4 Experiments and Evaluation
The experiments evaluate ARAELECTRA on Arabic question answering, sentiment analysis, and named-entity recognition against multiple Arabic pretrained models. ARAELECTRA achieved the highest performance on all tested datasets among base models, with limited exceptions against substantially larger models on ARCD.
- Evaluation setup: ARAELECTRA was evaluated on question answering, sentiment analysis, and named-entity recognition using Arabic downstream datasets.The evaluation covered ARCD and TyDiQA for question answering, ArSenTD-Lev for sentiment analysis, and ANERcorp for named-entity recognition.
- Evaluation setup: The comparison included AraBERT, Arabic-BERT, Arabic ALBERT, and ARBERT model variants.
- Results: ARAELECTRA achieved the highest performance on all tested datasets among the other base models.
- Results: On ARCD, ARAELECTRA fell short of Arabic-ALBERT-xlarge in exact match and ARABERTv0.2-large in F1-score; both models were larger.Arabic-ALBERT-xlarge was four times ARAELECTRA’s size.
- Dataset limitations: ARCD’s translated and multilingual training artifacts reduced performance, while some TyDiQA examples contained HTML artifacts in contexts and answers.
- Dataset limitations: All tested Arabic models struggled with fine-grained ArSenTD-Lev labeling because the dataset has 4K examples, five sentiment classes, six topics, and high class imbalance.
- Interpretation: The results indicate that ELECTRA’s replaced token detection objective achieves higher performance, especially on question answering, than masked language modeling for Arabic text.
5 Conclusion
The paper concludes that RTD-based pretraining produces stronger and more efficient Arabic language representation models than MLM. ARAELECTRA improves state-of-the-art performance across Arabic tasks, including with the same data and smaller models, and its code is publicly released.
- Conclusion: RTD pretraining on Arabic text is more efficient and produces better pretrained language representation models than MLM.
- Conclusion: ARAELECTRA improves state-of-the-art results for Arabic question answering, sentiment analysis, and named-entity recognition.
- Conclusion: ARAELECTRA achieves higher performance than models pretrained on the same dataset and larger model sizes.
- Release: The model, pretraining code, and fine-tuning code are publicly released through the authors’ repository.