PaperlayerView exact revision on arXiv

Source-linked AI summary

NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task

Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, Nizar Habash

arXiv:2010.11334v3cs.CLcs.AI

TL;DR

Arabic dialects vary across geographic regions, but fine-grained identification at the province level has had limited prior investigation. NADI provides Twitter data for country- and province-level classification across 21 countries and 100 provinces; submitted systems achieved the reported best F1 scores for both subtasks.

  • Problem

    Arabic dialects vary geographically, including within countries, while province-level automatic dialect identification had not previously been investigated to the authors’ knowledge.

    Source: §1 · p. 1

  • Method

    NADI collected Twitter data across 21 countries and 100 provinces and organized country- and province-level dialect identification subtasks.

    Source: §1 · p. 1, §4.2 · p. 5

  • Results

    Mawdoo3-AI achieved 26.78% F1 in Subtask 1, and BERT-NGRAMS achieved 6.39% F1 in Subtask 2.

    Source: §5.3 · pp. 7–8

  • Takeaways & Limitations

    The participation and submissions reflect community interest and call for further work in Arabic dialect detection and processing.

    Source: §5.2 · p. 7, §6 · p. 9

    Using consistent posting location as a dialect-label proxy is useful but imperfect: users may post from a country different from their dialect’s country.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

We present the results and findings of the First Nuanced Arabic Dialect Identification Shared Task (NADI). This Shared Task includes two subtasks: country-level dialect identification (Subtask 1) and province-level sub-dialect identification (Subtask 2). The data for the shared task covers a total of 100 provinces from 21 Arab countries and are collected from the Twitter domain. As such, NADI is the first shared task to target naturally-occurring fine-grained dialectal text at the sub-country level. A total of 61 teams from 25 countries registered to participate in the tasks, thus reflecting the interest of the community in this area. We received 47 submissions for Subtask 1 from 18 teams and 9 submissions for Subtask 2 from 9 teams.

1 Introduction

Arabic dialects vary geographically, including within countries, but the everyday dialectal variety comprises many distinct variants. NADI provides Twitter data and two identification tasks spanning countries and provinces to encourage research on this variation.

1 Introduction

  • Arabic’s everyday dialectal variety comprises geographically and politically defined variants, with differences possible between provinces within the same country.Source: §1 · p. 1
  • NADI targets dialect variation across 21 countries and 100 provinces through country-level and province-level identification tasks using a newly collected Twitter dataset.The dataset is described as publicly available for research.Source: §1 · p. 1
  • 52 teams registered, 18 submitted systems, and 14 of the 15 submitted papers were accepted.Source: §1 · p. 2
  • The paper previews related work, the task and dataset, and participating teams and shared-task results.Source: §1 · p. 2

2 Related Work

Prior Arabic dialect identification and resources largely focused on regional or country-level groupings, with finer-grained city work also emerging. NADI situates its province-level task in this landscape using naturally occurring data.

2 Related Work

  • Dialectal Arabic has received relatively recent scholarly attention compared with Classical Arabic and Modern Standard Arabic.Source: §2 · p. 2
  • Earlier multi-dialect identification efforts included regional shared tasks using broadcast transcriptions and acoustic or phonetic features.Source: §2 · p. 2
  • MADAR developed controlled parallel dialect data and supported city-level identification across 25 Arab cities.Source: §2 · p. 2
  • NADI’s province-level task uses naturally occurring data and a larger label set than MADAR’s city-level task, extending identification to provinces.Source: §2 · p. 2

3 Task Description

NADI defines country-level and province-level dialect classification tasks, with the latter covering a more fine-grained set of geographic labels. The task description also specifies split and evaluation arrangements for comparisons.

3 Task Description

  • The task comprises country-level dialect identification and province-level classification.Source: §3 · p. 2

3.1 Subtask 1: Country-level Classification

  • Subtask 1 classifies tweets among 21 country labels, with TRAIN and DEV data for every country and TEST predictions submitted through an online system.Source: §3.1 · pp. 2–3

3.2 Subtask 2: Province-level Classification

  • Subtask 2 predicts a province from 100 labels, a finer geographic target than prior city- or country-level prediction; split users are distinct across TRAIN, DEV, and TEST.Source: §3.2 · p. 3

3.3 Restrictions and Evaluation Metrics

  • Participants received distributed tweets for shared-data comparisons and were asked to use only that data; CodaLab evaluation supported transparency and timely scoring.Source: §3.3 · p. 3

èX@ñ«_ áÓ_ Õ»A‚«ð_ ¼PAJ.Ó_ Õ»YJ «#

The shared task provided participants with additional unlabeled Twitter data and established a common evaluation setup. Researchers could continue using the test set after the competition.

4 Data Sets

4 Shared Task Datasets

The shared task provides one Twitter dataset with country- and province-level labels, alongside an optional unlabeled collection. The data section describes its collection, splits, and issues affecting the labels and tweet content.

4.1 Data Collection

  • Tweets were collected from 100 provinces in 21 Arab countries over 10 months, using users who posted consistently and exclusively from one province.The collection ran from January to October 2019, with up to 3,200 tweets crawled per user.Source: §4.1 · p. 5

4.2 Data Sets

  • 30,957 tweets of at least five words were split into TRAIN (21,000), DEV (4,957), and TEST (5,000), with users kept separate across splits.Each tweet appears in both subtasks with different labels: country for Subtask 1 and province for Subtask 2.Source: §4.2 · p. 5, §4 · p. 5
  • The optional UNLABELED 10M collection contains 10 million Arabic Twitter posts from 2019 and was made available for development in either subtask.Participants received tweet IDs and a script to retrieve the posts.Source: §4.2 · p. 5

4.3 Data Issues

  • Consistent posting location is an imperfect proxy for dialect: users may post from a country different from the one associated with their dialect.The paper reports manual verification of a small random sample of users (n=30).Source: §4.3 · p. 5
  • An in-house model estimated MSA tweets at 49.5% of TRAIN, 46.6% of DEV, and 49.7% of TEST, but those estimates require caution because dialect features can mix with MSA.The authors retained all collected tweets to make classification more challenging, including tweets that were fully or partly MSA.Source: §4.3 · p. 6
  • Despite Arabic-language filtering and an Arabic-script threshold, 2.52% of the dataset was Farsi, which shares Arabic script.The paper attributes the remaining non-Arabic text to errors in the Twitter API language tag and the script overlap.Source: §4.3 · p. 6

5 Shared Task Teams & Results

NADI reports baseline and shared-task results for country- and province-level dialect identification, alongside participation and system approaches. The leading systems used Transformers, unlabeled-data pretraining, and/or self-training.

5.1 Our Baseline Systems

  • The majority-class baseline scored 21.84% accuracy and 1.71% F1 on Subtask 1, and 1.92% accuracy and 0.04% F1 on Subtask 2.The mBERT baseline achieved 32.38% accuracy and 13.32% F1 on Subtask 1, and 3.32% accuracy and 2.13% F1 on Subtask 2.Source: §5.1 · pp. 6–7

5.2 Participating Teams

  • The task received 61 registrations, 47 Subtask 1 submissions from 18 teams, and 9 Subtask 2 submissions from 4 teams; 14 teams had accepted description papers.Table 4 lists the teams with accepted description papers.Source: §5.2 · p. 7, Table 4 · p. 7

5.3 Shared Task Results

  • The cited results report a best Subtask 1 test F1 of 26.78% across 18 submitted teams; for Subtask 2, BERT-NGRAMS had the best test F1, 6.39%, among four submitting teams.Source: §5.3 · pp. 7–8

5.4 General Description of Submitted Systems

  • Most top teams used Transformers, further pre-trained on unlabeled data and/or enhanced through self-training; other teams used non-Transformer neural networks or linear models, often with ensembling.Table 7 summarizes participating teams’ approaches.Source: §5.4 · p. 8, Table 7 · p. 8

6 Conclusion and Future Work

The authors conclude that participation reflects community interest in Arabic dialect detection and processing and motivates further work. They plan a second NADI iteration with new datasets and questions, including MSA regional-use classification.

6 Conclusion and Future Work

  • The authors say the submissions reflect community interest and call for further work in Arabic dialect detection and, more broadly, Arabic dialect processing.They also identify dataset collection as an area for improvement.Source: §6 · p. 9
  • A planned second NADI task will use new datasets and explore MSA regional-use classification and interactions between MSA and dialectal Arabic.The authors hypothesize that MSA regional-use classification will be more challenging than dialectal Arabic classification.Source: §6 · p. 9

A Appendix

The appendix provides province-level data distributions and full results for both subtasks.

A Appendix

  • Table A1 distributes NADI data over provinces by country and across the TRAIN, DEV, and TEST splits.The passage identifies the table’s coverage, not its underlying counts.Source: Table A1 · p. 12
  • Tables A2 and A3 provide full results for Subtasks 1 and 2, respectively, sorted by official macro-F1.Parenthetical numbers indicate ranks.Source: Table A2 · p. 13, Table A3 · p. 13
Loading 2010.11334v3…