Source-linked AI summary
Pre-Training BERT on Arabic Tweets: Practical Considerations
Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, Younes Samih
TL;DR
Arabic tweet-focused BERT pretraining requires choices about data domain, size, segmentation, and training evaluation. The paper trains and compares five QARiB models, finding that mixed formal and informal data and linguistic segmentation help, while more data or steps do not necessarily improve results.
Problem
Arabic tweets combine dialectal and informal language with social-media conventions, motivating domain-focused pretrained models for downstream NLP tasks.
Method
The authors train 5 QARiB transformer models with varied data sources, sizes, and linguistic processing, compare them with three existing BERT models, and evaluate checkpoints across tasks.
Results
Mixed formal and informal Arabic outperformed tweets-only training, linguistic segmentation produced substantial improvements on some tasks, and more data or training steps did not necessarily yield better models.
Takeaways & Limitations
Pretraining BERT for Arabic tweets requires considering genre, data size, linguistic segmentation, and multiple evaluation checkpoints rather than relying on training loss alone.
Abstract
from arXiv · showhide
Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.
1 Introduction
The paper motivates Arabic tweet-focused BERT models by the domain sensitivity of contextual embeddings and the mismatch between formal Arabic and dialectal social media text. It introduces QARiB models, evaluates them against existing transformers, and releases pretrained checkpoints.
- Domain-matched training data can enhance contextual embeddings for downstream tasks such as NER and text classification.
- QARiB targets Arabic tweets, which often contain dialectal, informal, and convention-poor language alongside emojis, hashtags, and user mentions.
- The study trains 5 QARiB transformer models differing in data sources, sizes, and linguistic processing.
- The authors compare QARiB with mBERT, AraBERT, and ArabicBERT using a battery of tests.
- The pretrained models are released for community use in fine-tuning tasks, with 10 QARiB checkpoints planned.
2 Related Work
Related work describes transformer-based contextual representations as broadly successful and notes that specialized monolingual BERT models often outperform multilingual BERT.
- Transformer-based language models have motivated wider deployment in downstream NLP applications.
- Multilingual BERT was trained on Wikipedia articles spanning 104 languages.
- Specialized monolingual BERT models routinely outperform multilingual BERT in several language settings.
3 Data Collection
The corpus combines formal and informal Arabic, including large news, subtitle, and Twitter collections, then applies Arabic-aware normalization and alternative segmentation schemes.
- The formal-data sources include Arabic Gigaword, Abu El-Khair Corpus, and Open Subtitles, while informal data comes from Arabic tweets.
- The Twitter collection contains over 440M unique tweets totaling 2.7B words after deduplication.
- Preprocessing tokenizes text with Farasa, removes diacritics and Kashida, normalizes mentions, URLs, and numbers, splits hashtags, and limits repeated letters.
- The corpus has versions with and without Farasa word segmentation to represent Arabic clitics and morphemes separately.
- Both versions subsequently use Byte-Pair Encoding instead of WordPiece to limit vocabulary size.
4 Pretraining
Pretraining uses TPU-based BERT training across datasets that vary in tweet volume, formal-text mixture, and Farasa segmentation. The setup masks words and tracks checkpoints to evaluate training progression.
- Training a single BERT model was estimated to take more than a year on an in-house V100 GPU but a few days on a Google Cloud TPU.
- QARiB10 and QARiB25 contain 40M and 100M random tweets, while mixed variants combine tweets with MSA data.
- QARiB25 mix far matches QARiB25 mix but uses Farasa segmentation.
- Pretraining uses masked-word prediction with a 15% masking probability and omits next-sentence prediction because tweets are commonly single sentences.
- Multiple checkpoints are retained during training to assess model progression and efficacy at different points.
5 Evaluation
The evaluation compares QARiB and existing Arabic BERT models across downstream tasks and checkpoints. Results indicate that data mixture and linguistic segmentation matter, while additional data or training steps do not uniformly improve performance.
- Evaluation setup: The evaluation uses NER, emotion, offensive-language, and QADI tasks to compare Arabic BERT models.The models are assessed at different checkpoints, with F1-based metrics selected according to task type.
- Evaluation setup: QARiB models are compared with AraBERTv0.1, AraBERTv1, ArabicBERT, and mBERT using base-size configurations.The comparison uses L=12, H=768, A=12, and 110M total parameters.
- Checkpoint evaluation: More training steps usually improve performance, but effectiveness fluctuates rather than increasing monotonically.The evaluation therefore examines multiple checkpoints instead of assuming the latest checkpoint is best.
- Data effects: More training data does not necessarily produce better models, with improvements observed specifically from QARiB10 to QARiB25.The reported comparison distinguishes data quantity from guaranteed downstream gains.
- Data effects: Mixing tweets with formal Arabic performs better than using tweets alone, despite all evaluation tasks involving tweets.The result highlights the value of broader training-data variety for informal-text evaluation.
- Segmentation: Linguistically motivated word segmentation yields substantial improvements on some tasks.The observation is consistent with AraBERTv1.0, which uses Farasa segmentation.
6 Conclusion
The paper pretrains BERT from scratch on Modern Standard Arabic and dialectal Twitter data, then examines how data, segmentation, and checkpoint selection affect downstream performance. Its conclusion emphasizes diminishing returns from more data, benefits from varied text and linguistic segmentation, and the need for multi-task checkpoint evaluation.
- Study scope: The study explores pretraining BERT from scratch on Modern Standard Arabic and dialectal Twitter data.
- Main findings: Adding more data shows diminishing returns, while formal-and-informal training mixtures outperform informal text alone on informal evaluation tasks.
- Main findings: Linguistic segmentation helps, and loss alone is misleading for deciding when to stop pretraining.The paper recommends testing checkpoints across a battery of downstream tasks to select models and stopping points.
- Main findings: Pretraining from scratch may be preferable to fine-tuning an existing model on out-of-domain data because of vocabulary mismatch.