PaperlayerView exact revision on arXiv

Source-linked AI summary

AraT5: Text-to-Text Transformers for Arabic Language Generation

El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed

arXiv:2109.12068v4cs.CL

TL;DR

The paper asks whether multilingual mT5 adequately handles Arabic and its diverse varieties, given limited comparisons with Arabic-specific sequence-to-sequence models and limited generation benchmarks. It introduces three Arabic T5-style models and the seven-task ARGEN benchmark, finding that the models outperform mT5 on 52 of 59 ARGEN test sets despite using less pre-training data. The models also establish new SOTA results on ARLUE.

  • Problem

    It remains unclear how well mT5 serves Arabic’s diverse varieties, while Arabic lacks sequence-to-sequence models and a systematic language-generation benchmark.

  • Method

    The authors pre-train three Arabic-specific T5-style encoder-decoder models and evaluate them on the seven-task ARGEN benchmark, including translation from four foreign languages.

  • Results

    52 of 59 ARGEN test sets favor the authors’ models over mT5, and the models also establish new SOTA results on ARLUE.

  • Takeaways & Limitations

    Arabic-specific text-to-text models can outperform mT5 across Arabic generation tasks, including translation involving languages with marginal or no pre-training data.

  • Takeaways & Limitations

    The models require significant pre-training time and are not energy efficient.

Abstract

from arXiv · show

Transfer learning with a unified Transformer framework (T5) that converts all language problems into a text-to-text format was recently proposed as a simple and effective transfer learning approach. Although a multilingual version of the T5 model (mT5) was also introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialects--Arabic. For evaluation, we introduce a novel benchmark for ARabic language GENeration (ARGEN), covering seven important tasks. For model comparison, we pre-train three powerful Arabic T5-style models and evaluate them on ARGEN. Although pre-trained with ~49 less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al., 2021). Our new models are publicly available. We also link to ARGEN datasets through our repository: https://github.com/UBC-NLP/araT5.

1 Introduction

The paper investigates whether multilingual mT5 adequately supports Arabic and its diverse varieties, where evidence against language-specific models remains limited. It introduces Arabic-specific text-to-text models and ARGEN to address gaps in sequence-to-sequence modeling and Arabic generation evaluation.

  • Research gap: mT5’s suitability for individual languages and varieties remains unclear, especially relative to language-specific models in non-English contexts.The paper also notes systematic issues in multilingual training corpora.
  • Motivation: Arabic provides a challenging test case because it contains diverse varieties and is widely used on social media.
  • Contributions: The authors introduce three Arabic-focused T5 variants covering Modern Standard Arabic and diverse dialects.Their vocabulary also includes 11 additional languages for zero-shot pre-training evaluation.
  • Contributions: ARGEN is a unified Arabic language-generation benchmark spanning seven tasks and 19 datasets, including nine newly proposed datasets.
  • Contributions: The work evaluates its models under full and zero-shot pre-training conditions and reports new state-of-the-art results across most datasets.It also evaluates Arabic language understanding using a large recently proposed benchmark.

2 Our Models

AraT5 is built from large Modern Standard Arabic and Twitter corpora, including dialectal and naturally code-switched content. Three T5Base encoder-decoder variants share a common architecture and denoising pre-training objective.

  • Pre-training data: 70GB of Modern Standard Arabic text, totaling 7.1B tokens, supplies one major pre-training resource.
  • Pre-training data: 1.5B Arabic tweets totaling 178GB provide social-media data selected to contain at least three Arabic words.
  • Pre-training data: The combined corpus contains 29B tokens, approximately 49% fewer Arabic tokens than mT5’s 57B Arabic tokens.
  • Pre-training data: A 100M-tweet analysis estimates 28.39% dialectal and 71.61% Modern Standard Arabic content.
  • Pre-training data: Natural code-switching accounts for 4.14% of analyzed data and involves many foreign languages, including English, French, and Korean.
  • Model design: AraT5MSA, AraT5TW, and AraT5 use the T5Base encoder-decoder architecture with approximately 220 million parameters.The models are trained on MSA data, Twitter data, or both, respectively.
  • Pre-training objective: Pre-training corrupts inputs by randomly dropping 15% of tokens, replacing consecutive dropped spans with sentinel tokens, and reconstructing the original sequence.

3 ARGEN

ARGEN is a broad Arabic generation benchmark designed to evaluate models across tasks, datasets, linguistic varieties, and translation directions. It combines existing resources with newly created components, including code-switched translation, title generation, question generation, and transliteration datasets.

  • Benchmark scope: ARGEN contains 19 datasets, 59 test splits, and seven tasks spanning translation, summarization, title generation, question generation, transliteration, and paraphrasing.The benchmark covers MSA, Arabic dialects, Arabizi, and code-switching.
  • Machine translation: ARGENMT consolidates seven machine-translation datasets with 46 test splits covering MSA and Arabic dialects.The datasets range from 600 to 138K sentences.
  • Machine translation: OPUS-X-Ara evaluates English, French, German, and Russian to Arabic translation using 1M training sentences per language.Each language also has 5K-sentence development and test splits.
  • Code-switched translation: The code-switched translation component contains natural Algerian Arabic-French and Jordanian Arabic-English datasets, each comprising 300 tweets.The benchmark also includes two synthetic code-switched test sets.
  • Summarization: The summarization component includes EASC with 153 articles and 765 human-generated summaries, plus Arabic WikiLingua data for 29.2K articles.
  • Title generation: ARGEN introduces a 120K-article news-title-generation dataset, retaining titles with at least three words and using 93.3K training examples.
  • Question generation: The question-generation dataset contains 96K passage-answer-question triplets extracted from Arabic and multilingual question-answering resources.
  • Paraphrasing and transliteration: The benchmark also covers paraphrasing and transliteration, including 603 filtered paraphrase pairs, a 1,010-pair Arabic benchmark, and a 58.9K-example transliteration training split.

4 Evaluation on ARGEN

The evaluation compares Arabic T5-style models with mT5 and other baselines across ARGEN’s generation tasks. Dedicated models achieve strong results across translation, summarization, title and question generation, paraphrasing, and transliteration.

  • Code-Switched Translation: The models achieve the best results in one natural and all four synthetic code-switched translation test sets.The gains include +4.36 BLEU on ALG-FR and +4.55 BLEU on MSA-EN→MSA.
  • Text Summarization: AraT5Tw performs best on WikiLingua summarization, while mT5 outperforms the authors’ models on EASC.The authors establish new SOTA on both datasets with their pretrained models and mT5.
  • News Title and Question Generation: All models outperform mT5 on news title generation and question generation, with AraT5MSA reaching 20.61% BLEU and AraT5 reaching 16.99% BLEU.The results are reported for ARGENNTG and ARGENQG, respectively.
  • Paraphrasing and Transliteration: AraT5MSA is best on APB and ASEP paraphrasing, achieving 17.52 and 19.38 BLEU, respectively.For transliteration, AraT5MSA reaches 65.88 BLEU and exceeds previous SOTA by 7.1 points.
  • Arabic Language Understanding: The models establish a new ARLUE SOTA with an ARLUE score of 77.52 versus the previous SOTA of 76.53.ARLUE covers six Arabic language-understanding task clusters.

5 Analysis and Discussion

The analysis attributes the models’ performance to dedicated Arabic pretraining, data quality, and exposure to natural code-switching. Qualitative and length-based evaluations further examine robustness across Arabic varieties and generation tasks.

  • Multilingual vs. Dedicated Models: AraT5 outperforms mT5 despite being pretrained with 49% less data.The authors suggest multilingual capacity and data-quality issues as possible explanations, while confirming systematic problems in Arabic mC4 data.
  • Code-Switching: Natural code-switching occurs in approximately 4% of Twitter sequences but is very rare in Arabic mC4 data.The authors connect this difference with AraT5Tw’s strong performance on natural French code-switched translation.
  • Length Analysis: AraT5MSA outperforms mT5 in all but one sentence-length condition in merged Arabic→English test sets.The remaining condition shows marginally lower performance for AraT5MSA.
  • Qualitative Analysis: Qualitative examples indicate that the models handle Algerian and Jordanian dialects in code-switched contexts better than mT5.The examples cover Algerian-to-French and Jordanian-to-English translation.
  • Qualitative Analysis: Paraphrase, transliteration, and title-generation samples are described as high-quality, informative, and fluent.The paraphrase samples also tightly capture the source sentences’ meanings.

6 Related Work

Related work includes multilingual masked-language and text-to-text models alongside Arabic-specific BERT-based models. These systems differ in architecture, pretraining data, language coverage, and Arabic variety coverage.

  • Multilingual LMs: mT5 is a multilingual T5 encoder-decoder pretrained on approximately 26.76TB of mC4 data covering 101 languages.mBERT and XLM-R are multilingual encoder-based masked-language models covering 104 and 100 languages, respectively.
  • Arabic LMs: Arabic-specific models include AraBERT, ARBERT, MARBERT, QARiB, and CamelBERT, all based on BERT-style architectures.Their pretraining covers combinations of Modern Standard, dialectal, classical, and social-media Arabic.

7 Conclusion

The paper introduces Arabic-specific T5 models and ARGEN, a seven-task benchmark for Arabic natural language generation. The models outperform mT5 across ARGEN and also achieve a new SOTA on ARLUE.

  • The authors introduce three Arabic-specific text-to-text Transformer models trained on large MSA and/or Arabic dialectal data.
  • ARGEN unifies seven Arabic natural language generation tasks across 19 datasets.
  • 52 out of 59 test sets, or 88.14%, show the Arabic-specific models outperforming mT5 across ARGEN tasks.
  • The models establish a new state of the art on the Arabic language understanding benchmark ARLUE.
  • The models are publicly available, and the ARGEN datasets are accessible through the authors’ repository.

Ethics Statement

The ethics statement highlights energy costs, public-domain data coverage, potential misuse, and socially beneficial deployment goals. The authors release the models for non-malicious use while acknowledging misuse risks.

  • The models require significant pre-training time and are not energy efficient.
  • The pre-training datasets come from the public domain and cover diverse communities.
  • The authors acknowledge that the models may still be misused in the real world.
  • The models are released for non-malicious use, with intended applications including education, disaster management, health, recreation, and travel.

A A Study of Arabic mC4 Data Quality

The study examines Arabic content in mC4 and finds substantial non-Arabic material and very little dialectal content. Most sampled text is classified as MSA, while some apparent dialect data is non-linguistic or mislabeled.

  • The mC4 Arabic portion contains 57B Arabic tokens from 53M webpages, representing 1.66% of mT5’s data.
  • 13.59% of the sampled mC4 paragraphs are predicted to be non-Arabic, mostly English, French, code, HTML, or other non-Arabic text.
  • 99.83% of the sampled mC4 data is classified as Modern Standard Arabic, leaving 0.17% predicted as dialectal.
  • Manual inspection finds that much of the predicted dialectal material consists of soap-opera names or advertisements rather than dialectal text.
  • ARLUE is an Arabic understanding benchmark spanning six cluster tasks and 42 publicly available datasets.
  • ARLUE reports individual dataset performance, averages across tasks, and an overall score rather than only merging social-meaning datasets.

B.2 ARLUE Evaluation

ARLUE evaluation compares AraT5 models with mT5 and established Arabic and multilingual baselines under single-task and multitask settings. AraT5 achieves the highest overall ARLUE score, while multitask training helps most dialect experiments but only half of social-meaning experiments.

  • Evaluation setup: The evaluation fine-tunes mT5 and MARBERT on the same training data and reports results for ARLUE’s task clusters, excluding NER.
  • Evaluation setup: ARLUE’s overall score is a macro-average in which each task cluster receives equal weight.
  • Single-task results: MARBERT remains the stated SOTA across most of ARLUE’s six cluster tasks and has the highest prior ARLUE score.
  • Single-task results: AraT5 achieves the highest ARLUE score at 77.52, followed by AraT5MSA at 77.50 and AraT5TW at 75.33.
  • Single-task results: All AraT5 models outperform mT5 and MARBERT by approximately +2.74 and +1 ARLUE score points, respectively.
  • Multitask results: Multitask training outperforms single-task models in 7 of 9 dialect experiments, or 77.78% of tasks.
  • Multitask results: Multitask training outperforms single-task models in 18 of 36 social-meaning experiments, representing 50% of tasks.
  • Related dataset: AraPara is a new multi-domain Arabic paraphrasing dataset created from English-Arabic parallel OPUS data through filtering and manual analysis.

D Qualitative Analysis of Models

The qualitative analysis examines how the models generate MSA and dialectal Arabic under different conditions, using representative samples and development-set examples across several tasks.

  • Analysis scope: The analysis explores MSA and dialectal Arabic generation under various conditions, while noting that its handpicked samples are mostly representative of model outputs.The stated analyses focus on linguistic attributes considered relevant to evaluation.
  • Translation examples: ARGENMT covers Arabic-to-English datasets spanning city-level dialects, MSA, and Moroccan and Tunisian varieties.The listed datasets include MADAR I, MADAR II, Bible I, and Bible II with separate development and test splits.
  • Task coverage: The appendix reports model performance on development splits for translation, document summarization, title generation, question generation, transliteration, and paraphrasing.Separate tables summarize sequence-to-sequence baselines, summarization results, and BLEU-based results for several generation tasks.
  • Translation behavior: Sequence-length results are reported for ARGENMT test datasets, alongside MSA and dialectal translation examples comparing the models with mT5.The examples mark translations as good or problematic and are extracted from development data.
  • Generation examples: Additional development-set examples illustrate paraphrasing, transliteration, and title generation produced by the models.These examples are presented in separate appendix tables for each task.
Loading 2109.12068v4…