Source-linked AI summary
AraGPT2: Pre-Trained Transformer for Arabic Language Generation
Wissam Antoun, Fady Baly, Hazem Hajj
TL;DR
Arabic language generation lacked advanced models trained directly on large Arabic corpora. The paper develops AraGPT2 from scratch with multiple model sizes and evaluates both generation quality and automatic detection, reporting strong Arabic text generation and a high-performing detector.
Problem
Arabic text generation has received limited attention, with prior work often adapting English GPT-2 or focusing on Arabic understanding.
Method
AraGPT2 is a family of stacked transformer-decoder models trained with causal language modeling on large Arabic corpora, alongside an ARAELECTRA-based detector.
Results
AraGPT2 produces high-quality, coherent Arabic text, while its detector achieves 98% accuracy on model-generated synthetic text.
Takeaways & Limitations
Publicly releasing the models and detector is intended to support new research possibilities and applications in Arabic NLP.
Takeaways & Limitations
The models have ethical risks because they can be used for malicious applications such as automatic fake-news generation and may reflect dataset biases.
Abstract
from arXiv · showhide
Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
1 Introduction
Arabic language generation has received less attention than Arabic understanding, while existing generation approaches relied on limited transfer or adaptation. AraGPT2 addresses this gap with advanced Arabic GPT-2 models, a detector, and public releases.
- Arabic language modeling has focused mainly on natural language understanding, leaving Arabic text generation and causal language modeling comparatively underdeveloped.
- AraGPT2 introduces four Arabic GPT-2 variants trained from scratch, ranging from 135M to 1.46B parameters.The variants support different data-availability and computational constraints.
- The project evaluates AraGPT2 with perplexity and human judgments, including whether generated news articles can deceive human evaluators.
- 98% accuracy is reported for an automatic discriminator detecting model-generated synthetic text.
- The four AraGPT2 variants and the discriminator are released through popular NLP libraries for further Arabic NLP research.
2 Related Works
Prior Arabic language-modeling work emphasized masked-language understanding, speech-recognition systems, or adapting English GPT-2, whereas AraGPT2 is presented as an Arabic transformer causal language model trained from scratch.
- Causal language modeling predicts the next word from preceding context and differs from masked language modeling, which predicts a masked word within a sequence.
- Scaling model sizes and datasets improves capability but requires high computational cost and large pretraining corpora.Training GPT-2 and GPT-3 was estimated at $43K and $4.6M respectively without hyper-parameter tuning.
- Arabic causal language modeling had been used mainly in automatic speech recognition, including n-gram and recurrent neural-network language models.
- hULMonA used Arabic causal language modeling for transfer learning and downstream text classification with AWD-LSTM layers trained on 600K Wikipedia articles.
- Earlier Arabic transformer causal models adapted OpenAI GPT-2 using Arabic Wikipedia and, in one case, dialectal Arabic datasets.
- AraGPT2 is presented as the first Arabic transformer causal language model trained from scratch on the largest Arabic corpora available at the time.
3 ARAGPT2: Methodology
AraGPT2 is a stacked transformer decoder trained with causal language modeling on 77GB of Arabic text. Its four variants use different scales and training configurations, with a detector based on fine-tuned ARAELECTRA.
- ARAGPT2 uses a stacked transformer-decoder architecture trained with the causal language modeling objective on 77GB of Arabic text.
- The model is offered in four variants, with larger versions improving performance while increasing fine-tuning difficulty and computational cost.
- The ARAGPT2 detector is based on pretrained ARAELECTRA fine-tuned on synthetically generated data.
- The large and mega variants use Adafactor to reduce optimizer memory requirements, while their architecture changes transformer-block layer-normalization order following GROVER.
- Dataset: The training corpus combines OSCAR, Arabic Wikipedia, the 1.5B words Arabic Corpus, OSIAN, and As-safir news articles.
- Dataset: Preprocessing removes short or repetitive documents, replaces URLs and mentions, removes diacritics and elongations, and applies a 64,000-vocabulary byte-level BPE tokenizer.
4 Experiments and Evaluation
ARAGPT2 was evaluated through perplexity, zero-shot tasks, and human assessment of generated news. The mega model produced coherent text and fooled many human readers, while zero-shot performance varied by task.
- Perplexity evaluation: Perplexity improved with larger ARAGPT2 variants on held-out Arabic Wikipedia articles.The validation set used Arabic Wikipedia articles published after August 2020, and the models were still under-fitting it.
- Zero-shot evaluation: Zero-shot evaluation provided only a natural-language instruction, without back-propagation, and was sensitive to prompt engineering.The evaluation covered question answering and translation.
- Zero-shot evaluation: 25% of trivia questions were answered correctly by ARAGPT2-MEGA, while English-to-Arabic translation failed.The reported zero-shot results were task-dependent.
- Human evaluation: ARAGPT2-Mega fooled approximately 60% of human respondents, with longer passages producing higher human error rates.The survey included 74 Arabic-speaking subjects and both machine-generated and human-written news articles.
- Human evaluation: Humans misclassified human-written text 50% of the time, reaching chance-level performance.Respondents cited factual inconsistencies, grammatical errors, punctuation, coherence, repetition, and unusual writing styles.
5 Automatic Detection of Machine Generated Text
The paper addresses the difficulty of detecting ARAGPT2-generated text by training an automatic detector on synthetic examples. The detector performs strongly, especially on longer passages.
- Motivation: The detector addresses risks from machine-generated misleading news, fake reviews, and phishing messages that humans may not consistently detect.The motivation follows the survey finding that average readers often failed to distinguish generated from human-written text.
- Dataset construction: The detection dataset used short texts of 150 tokens and long texts of 500 tokens generated from 25-word prompts.The two lengths were designed to evaluate the impact of text length.
- Detection results: ARAELECTRA achieved 98.7% F1-score on long text and 94.9% F1-score on short text.The results indicate that longer machine-generated text was easier to detect.
6 Conclusion
ARAGPT2 is presented as an advanced Arabic transformer language-generation model that produces coherent, grammatically correct, and syntactically sound text. The paper also emphasizes both public research access and misuse risks.
- Conclusion: ARAGPT2 is described as the first advanced Arabic language-generation model based on the transformer architecture.It was trained on the largest publicly available collection of filtered Arabic corpora.
- Conclusion: The reported outputs are coherent, grammatically correct, and syntactically sound high-quality Arabic text.The conclusion bases this assessment on perplexity evaluation and generation results.
- Conclusion: The released detector is intended to help detect misuse, while public release is intended to support new Arabic NLP research.The paper identifies fake-news generation and inherited dataset biases as ethical concerns.
A Generated Samples from ARAGPT2
The paper presents ARAGPT2-mega generations from unseen prompts spanning coronavirus vaccines, children’s stories, and miscellaneous topics. Sampling uses nucleus-probability settings such as topp = 0.95 and topp = 1.
- Coronavirus-vaccine samples: Coronavirus-vaccine samples compare two ARAGPT2-mega generations from the same unseen context using topp = 0.95 and topp = 1.The figure places the random context above the two generated samples.
- Children’s stories: A children’s-story sample follows a random unseen context and uses ARAGPT2-mega with topp = 0.95.The example illustrates generation from an unseen story context.
- Miscellaneous topics: Miscellaneous-topic samples use random unseen contexts and ARAGPT2-mega with topp = 0.95.The examples include recipe-related Arabic text among the displayed generations.
B.1 Question Answering
AraGPT2 is evaluated on zero-shot Arabic question answering using prompted generation without providing context passages. Exact-match and F1 scores are low, while manual scoring finds some correct answers, with predefined answer length limiting performance.
- Evaluation setup: The evaluation prompts AraGPT2 with an Arabic question-answer template and optionally adds “in the year” for year-based answers.The model generates answers using a fixed answer length, repetition penalty, and no-repeat tri-gram strategy.
- Evaluation setup: The question-answering tests use 921 Arabic TyDiQA validation examples and 702 ARCD test examples.
- Results: 3.93% exact match and 14.51% F1 were achieved on TyDiQA, compared with 4.07% exact match and 13.88% F1 on ARCD.
- Results: Manual scoring found that AraGPT2 correctly answered 24.6% of 500 TyDiQA questions, including countries, capitals, and birth- or death-year questions.Exact-match and F1 metrics can miss answers that are correct but worded differently.
- Limitation: Predefined answer length negatively affected generated answers in some cases, limiting the current question-answering approach.
B.2 Translation
AraGPT2’s zero-shot translation capability was tested by prompting it to translate English-Arabic sentence pairs and using greedy decoding. Translation performance was low, with vocabulary coverage and prompt design identified as contributing factors.
- Method: The translation experiment prompts the model to translate a source sentence and uses greedy decoding to generate the target sentence.
- Results: 1.32 BLEU was achieved on 5,000 randomly selected English-Arabic Tatoeba pairs.
- Interpretation: The low translation score was attributed to scarce English vocabulary representation and potentially sub-optimal prompt design.Many English words were split into single characters, while prompt engineering was left for future work.
C GLTR Analysis and Visualizations
GLTR visualizations distinguish machine-generated from human-written text through different distributions of highlighted words. AraGPT2-generated text is mostly green and yellow, whereas human-written text shows more red and purple words.
- GLTR visualization: AraGPT2-generated text is mostly highlighted in green and yellow in the GLTR visualization.
- GLTR visualization: Human-written text shows an increase in red and purple highlighted words compared with machine-generated text.
- GLTR visualization: The visualization uses ARAGPT2-base as the GLTR backend model.