Source-linked AI summary
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models
Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, Nizar Habash
TL;DR
The paper asks which pre-training design choices explain performance differences among Arabic language models. It builds controlled CAMeLBERT variants and compares them with existing models across five tasks and 12 datasets, finding that variant proximity to task data matters more than pre-training data size. The authors use this finding to define an optimized system selection model.
Problem
Performance differences among Arabic pre-trained models are difficult to explain because the models use different design choices, while the contributions of those choices remain unclear.
Method
The authors pre-train Arabic models with controlled variation in data size and language variant, then evaluate them against eight existing models across five tasks and 12 datasets.
Results
Variant proximity between pre-training data and fine-tuning data is more important than pre-training data size.
Takeaways & Limitations
The authors exploit variant proximity to define an optimized system selection model for the studied tasks.
Takeaways & Limitations
The proposed system selection method is not claimed to be foolproof, and comparisons with existing models may reflect differences in data sources, genre, mixture, or pre-training details.
Abstract
from arXiv · showhide
In this paper, we explore the effects of language variants, data sizes, and fine-tuning task types in Arabic pre-trained language models. To do so, we build three pre-trained language models across three variants of Arabic: Modern Standard Arabic (MSA), dialectal Arabic, and classical Arabic, in addition to a fourth language model which is pre-trained on a mix of the three. We also examine the importance of pre-training data size by building additional models that are pre-trained on a scaled-down set of the MSA variant. We compare our different models to each other, as well as to eight publicly available models by fine-tuning them on five NLP tasks spanning 12 datasets. Our results suggest that the variant proximity of pre-training data to fine-tuning data is more important than the pre-training data size. We exploit this insight in defining an optimized system selection model for the studied tasks.
1 Introduction
The paper investigates why Arabic pre-trained models differ in performance by controlling pre-training data size and language variant while evaluating across diverse tasks. It finds that variant proximity matters more than pre-training data size and uses this insight for system selection.
- Existing Arabic pre-trained models are difficult to compare because they differ in data size, language variant, tokenization, vocabulary, and training choices.
- The authors vary pre-training data size and Arabic language variant while keeping other pre-training hyperparameters constant.They scale down MSA data and train models on MSA, dialectal Arabic, classical Arabic, and their mixture.
- The evaluation covers five NLP tasks and 12 datasets, alongside eight publicly available Arabic pre-trained models.The tasks are NER, POS tagging, sentiment analysis, dialect identification, and poetry classification.
- The released CAMeLBERT models include different design choices, including CAMeLBERT-Mix trained on the largest dataset to date.
- Variant proximity between pre-training and task data is more important than pre-training data size.The authors use this finding to define an optimized system selection model.
2 Related Work
Related work established strong Arabic pre-trained models and examined how pre-training data, tokenization, and linguistic knowledge affect downstream performance. This paper extends that line of inquiry by focusing on language variants and data size in Arabic models.
- AraBERT and ArabicBERT were among earlier Arabic monolingual pre-trained models trained on large Arabic corpora.AraBERT used 24GB of news text, while ArabicBERT used 95GB mainly from Arabic OSCAR data.
- Prior studies used probing techniques to examine linguistic knowledge captured by pre-trained models, including morphosyntactic and semantic knowledge.
- Reliable encoding of many syntactic and semantic features can emerge with 10M to 100M words, but typical downstream NLU performance requires substantially more data.
- Comparisons of multilingual and monolingual models found that tokenizer choice can be as important as pre-training data size for downstream performance.
- Across 12 subtasks, AraBERTv02 (X3) wins or ties in six, CAMeLBERT-Star in four, and the two systems together in nine.
3 Pre-training CAMeLBERT
CAMeLBERT pre-training combines multiple Arabic corpora with standardized preprocessing and a common BERT-based training procedure. The models use variant-specific data sources while sharing tokenizer and optimization settings.
- Training data: MSA training data combines Arabic Gigaword, Abu El-Khair, OSIAN, Arabic Wikipedia, and Arabic OSCAR.
- Training data: Dialectal Arabic training data is collected from multiple CALLHOME, Levantine, Gulf, and Iraqi Arabic corpora.
- Training data: Classical Arabic training uses the OpenITI corpus v1.2.
- Preprocessing: Preprocessing removes invalid characters, non-Arabic lines, diacritics, and kashida, then segments each line into sentences.
- Pre-training configuration: A WordPiece tokenizer is trained on the full 167 GB dataset with a 30,000-word vocabulary, while the models retain case and accents and use whole-word masking.
- Pre-training configuration: Pre-training lasts one million steps using TPU training, with sequence lengths of 128 tokens initially and 512 tokens subsequently.The first 900,000 steps use batch size 1,024 and the final 100,000 use batch size 256.
4 Fine-tuning Tasks
The study fine-tunes Arabic pre-trained models on five tasks spanning 12 datasets selected to represent multiple Arabic variants. Task-specific datasets, preprocessing, label structures, and evaluation procedures vary across the benchmark.
- Task coverage: The benchmark covers NER, POS tagging, sentiment analysis, dialect identification, and poetry classification across 12 datasets.
- General setup: Fine-tuning adds a fully connected linear layer to the last hidden state using Hugging Face Transformers.
- Variants: Dataset variants are determined from corpus design and estimated MSA proportions for social-media datasets.The Corpus 6 dialect identification model is used to estimate whether text is MSA or dialectal Arabic.
- Named entity recognition: NER uses ANERcorp, an MSA dataset of approximately 150K words with four IOB entity classes, and reports micro F1.
- Part-of-speech tagging: POS tagging uses PATB, ARZATB, and GUMAR, containing 32, 33, and 35 POS tags respectively, with accuracy as the evaluation metric.
- Sentiment analysis: Sentiment analysis combines four tweet datasets and evaluates with macro F1 over positive and negative classes while excluding neutral.
- Dialect identification: Dialect identification uses four datasets with 6 to 26 labels, preprocesses tweets, groups MADAR Twitter users’ posts into five, and reports macro F1.
- Poetry classification: Poetry classification uses approximately 1.8M poems covering 23 meters, with diacritics removed and verse halves separated by [SEP].
5 Evaluation Results and Discussion
Controlled evaluations across Arabic variants, pre-training sizes, and task types show that variant proximity to fine-tuning data generally matters more than pre-training size, although task-specific patterns vary. Comparisons with existing models further reveal complementary strengths and unresolved effects from differing design choices.
- 5.1 Models with Different Data Sizes: The full MSA and MSA-1/4 models perform best on average, with MSA-1/4 winning or tying in seven of 12 subtasks despite using one-quarter of the full data.The authors conclude that pre-training size has a limited and inconsistent effect on fine-tuning performance.
- 5.1 Models with Different Data Sizes: Performance-size relationships vary by subtask: ArSAS correlates strongly positively at 0.96, MADAR-6 negatively at -0.62, and the 12-experiment average is only 0.25.Some tasks plateau early, while others improve inconsistently as pre-training data increases.
- 5.2.1 MSA, DA, and CA: Variant choice changes performance by 4.6% on average, with NER most sensitive at 14.5% and POS tagging least sensitive at 0.7%.The task-level differences range from 0.2% to 14.5%, supporting task-specific model pairing.
- 5.2.1 MSA, DA, and CA: CAMeLBERT-MSA wins or ties in 10 of 12 subtasks, while CAMeLBERT-DA leads highly dialectal MADAR-6 and CAMeLBERT-CA leads classical-Arabic poetry classification.These exceptions are consistent with proximity between pre-training and fine-tuning variants, though MSA also has substantially more data.
- 5.2.1 MSA, DA, and CA: The best-performing model in every case has the lowest OOV rate, with performance and OOV rate showing a strong average correlation of -0.82.CAMeLBERT-CA reaches 0.9% OOV on poetry with 6GB of data, versus CAMeLBERT-MSA’s 107GB, yet outperforms larger-data models there.
- 5.2.2 Mix of MSA, DA, and CA: CAMeLBERT-Mix improves performance in three dialectal cases, but increased data variety and size do not improve all tasks; proximity remains important for MSA and CA.The proposed selection approach is not foolproof, and differing vocabulary sizes and training steps across existing models remain potential contributors to performance.
- 5.3 Comparison with Existing Models: Among existing models, AraBERTv02 (X3) is best on average with six wins or ties, CAMeLBERT-Star is second with four, and together they cover nine of 12 subtasks.This complementarity can be exploited by system developers to achieve higher overall performance.
- 5.3 Comparison with Existing Models: Across existing models, smaller pre-training datasets can outperform larger ones, while monolingual MSA models favor MSA tasks and mixed models favor DA tasks.Average scores are 78.5% versus 78.4% overall, 82.6% versus 82.1% on MSA, and 73.5% versus 73.9% on DA for monolingual versus mixed models.
6 Conclusion and Future Work
The paper finds that matching pre-training and subtask language variants matters more than pre-training data size, and uses this finding to define an optimized system-selection model for the studied Arabic NLP tasks. It also releases the created models and fine-tuning code while identifying further design decisions for future study.
- Variant proximity between pre-training data and subtask data is more important than pre-training data size.
- The authors use this insight to define an optimized system-selection model for the studied tasks.
- Future work will examine vocabulary size, tokenization techniques, and additional data mixtures as possible contributors to fine-tuning performance.