PaperlayerView exact revision on arXiv

Source-linked AI summary

Masader: Metadata Sourcing for Arabic Text and Speech Data Resources

Zaid Alyafeai, Maraim Masoud, Mustafa Ghaleb, Maged S. Al-shaibani

arXiv:2110.06744v1cs.CL

TL;DR

Arabic NLP datasets lack a centralized catalogue, and their characteristics are often unclear. The paper creates and analyzes a metadata-annotated catalogue of publicly available Arabic NLP datasets. It identifies 200 datasets and reports challenges including discoverability and resource coverage.

  • Problem

    There is no centralized online catalogue for Arabic NLP and speech datasets, and their characteristics are unclear.

    Source: §1 · p. 1

  • Method

    The authors catalogue publicly available Arabic NLP datasets with metadata annotations and analyze the collection and its issues.

    Source: §8 · p. 9

  • Results

    The catalogue includes 200 datasets, and the review identifies discoverability as an issue for publicly available resources.

    Source: §4.1 · p. 5, §6 · p. 7

  • Takeaways & Limitations

    The catalogue is a snapshot in time, and further efforts are needed to advance Arabic NLP beyond Modern Standard Arabic.

    Source: §8 · p. 9

    Keyword-based retrieval may miss resources, and open or regulated-access resources outside the inclusion criteria were not examined.

    Source: §7 · p. 9

Abstract

from arXiv · show

The NLP pipeline has evolved dramatically in the last few years. The first step in the pipeline is to find suitable annotated datasets to evaluate the tasks we are trying to solve. Unfortunately, most of the published datasets lack metadata annotations that describe their attributes. Not to mention, the absence of a public catalogue that indexes all the publicly available datasets related to specific regions or languages. When we consider low-resource dialectical languages, for example, this issue becomes more prominent. In this paper we create \textit{Masader}, the largest public catalogue for Arabic NLP datasets, which consists of 200 datasets annotated with 25 attributes. Furthermore, We develop a metadata annotation strategy that could be extended to other languages. We also make remarks and highlight some issues about the current status of Arabic NLP datasets and suggest recommendations to address them.

1 Introduction

Arabic NLP datasets lack a centralized catalogue and consistent metadata describing their characteristics. The paper addresses this gap by creating a 200-dataset catalogue, designing a metadata schema, and analyzing dataset issues.

  • The paper creates a catalogue of 200 Arabic NLP and speech datasets annotated with 25 attributes.Source: §1 · p. 1
  • The study designs a metadata schema for annotating Arabic datasets.Source: §1 · p. 1
  • The authors analyze the datasets’ current status, identify issues, and recommend solutions.Source: §1 · p. 1

2 Related Work

Prior work surveys research fields and proposes ways to document datasets, while Arabic resource surveys vary in scope and coverage. This paper’s related-work discussion situates its catalogue among those efforts.

  • Large-scale literature analyses map research fields using collected scholarly records, including work drawing on ACL Anthology papers and Google Scholar data.Ammar et al. (2018), Radev et al. (2016), and Mohammad (2020) are discussed as examples.Source: §2 · pp. 1–2
  • Dataset documentation proposals include datasheets for general dataset attributes and data statements tailored more specifically to linguistics and NLP.Gebru et al. (2018) and Bender and Friedman (2018) are discussed as examples.Source: §2 · p. 2
  • Earlier Arabic dataset surveys include a list of around 80 freely available datasets, including resources outside NLP, while some linked resources no longer worked.Zaghouani (2017) is the cited example.Source: §2 · p. 2
  • Other Arabic resource surveys focus on dialects, classify studies, or provide links to a limited number of datasets.The passage discusses surveys of Maghrebi Arabic, Arabic varieties, and Arabic tools and resources.Source: §2 · p. 2

3 Methodology

The methodology combines keyword-based resource discovery, inclusion filtering, metadata annotation, verification, and analysis. The schema covers multiple aspects of Arabic resources, and the search and annotation work took place in 2021.

3 Methodology

  • The five-step process searches for resources, filters them, annotates metadata, verifies the annotations, and analyzes the resulting collection.The initial search and filtering occurred in July–August 2021, and annotation concluded in September 2021.Source: §3 · p. 2

3.1 Step 1: Searching Resources

  • Searches used Google and selected repositories with Arabic- and NLP-related keywords, producing a preliminary list of around 299 resources.The named repositories include GitHub, Paperswithcode, Huggingface, LREC, Google Scholar, and LDC.Source: §3.1 · pp. 2–3

3.2 Step 2: Filtering with Inclusion Criteria

  • Resources were screened for NLP relevance, an associated publication, post-2010 creation, suitability for language tasks, and exclusion of NLP tools.Source: §3.2 · p. 3
  • After applying the inclusion criteria, the preliminary list was reduced to 207 papers.The criteria initially removed 93 papers.Source: §3.2 · p. 3

3.3 Step 3: The Annotation Process

  • Metadata annotation combined manual inspection of resource links and papers with automated publication lookup; four authors used Google Sheets and recorded missing or conflicting information.The process began after filtering and used pre-agreed metadata.Source: §3.3 · pp. 3–4
  • The schema organizes metadata into five subcategories covering publication, content, accessibility, diversity, and evaluation.Its fields include dataset size and form, collection and ethical-risk details, access information, and Arabic dialect categories.Source: §3.3 · pp. 3–4

3.4 Step 4: Verification of metadata

  • Manual verification resolved annotator notes and removed seven papers: two duplicates and five with incorrect pre-2010 year annotations.Source: §3.4 · p. 5

3.5 Step 5: Analysis

  • After verification, the final annotated collection was analyzed across its metadata.Source: §3.5 · p. 5

4 Findings and Representation

The findings describe the collected Arabic NLP resources and how Masader presents them. The catalogue reports dataset volume and provides a searchable interface connected to its underlying spreadsheet.

4.1 Data Statistics

  • Dataset volume is expressed using task-appropriate units, such as tokens for token-level tasks, sentences for sentence-level tasks, or documents for very large datasets.Source: §4.1 · p. 5

4.2 Masader Interface

  • The website reflects Google Sheets updates immediately and supports filtering and sorting, dataset-paper associations, download links, and dataset cards.Its primary interface displays nine attributes; the remaining metadata appears on dataset cards.Source: §4.2 · p. 5

4.1 Data Statistics

  • 200 datasets are catalogued, with more than 90% in written text format and the remainder in speech data.Reported sizes were validated by downloading datasets when possible; otherwise, paper-reported values were used.Source: Table 1 · p. 5, §4.1 · p. 5

5 Examining the Arabic NLP Landscape

The surveyed Arabic NLP landscape shows growth in dataset publication alongside uneven access, licensing, dialect coverage, and task representation. The section identifies patterns across publication venues and dataset characteristics.

5.1 Publications development

  • Arabic NLP dataset publications generally grew, with increases in even years attributed to LREC; most appeared in conferences and workshops.The paper anticipates further growth in 2020 alongside AraBERT, Multi-dialect BERT, and Araelectra.Source: §5.1 · p. 5, Figure 3 · p. 7

5.2 Data Accessibility

  • More than 80% of data could be accessed freely through data hosters in the last three years, while over half of 93 discarded datasets lacked online presence or access details.Some datasets required registration or email; papers offering private access through author contact were excluded from the final list.Source: §5.2 · p. 6

5.3 Data Providers and Licensing

  • About 50% of datasets lack licences; QCRI, Qatar University, NYU Abu Dhabi, and Nile University are the top four providers.Datasets from these prominent providers are typically accompanied by clear licensing.Source: §5.3 · p. 6

5.4 Dialects Diversity

  • Egyptian dialects are most represented, followed by Algerian, Moroccan, and Saudi; Somali, Djibouti, and Mauritanian dialects have only three resources each.More than 20 of the 200 surveyed entries annotate dialects, primarily for dialect identification.Source: §5.4 · pp. 6–7, Figure 4 · p. 7

5.5 Tasks Coverage

  • Machine translation and sentiment analysis are the most popular tasks; dialect identification, topic classification, named entity recognition, and speech recognition also recur.Some tasks appear only once, while KALIMAT covers multiple tasks and ALUE aggregates multiple datasets.Source: §5.5 · p. 7

6 Arabic NLP Datasets Challenges and Recommendations

The section describes challenges in dataset availability, documentation, sharing, evaluation, and ethical and legal practices. It recommends clearer risk information, documentation, and more sustainable hosting and sharing practices.

Data Availability

  • Many datasets are hard to discover, and invalid URLs can orphan resources; public repositories such as GitHub and GitLab are proposed as possible hosts.The authors identify terminated academic affiliations and researcher webpages as one cause of broken links.Source: §6 · p. 7

Data Documentation

  • Dataset documentation is often inadequate for reuse, with missing provenance, collection details, and clear access or use terms; the authors recommend structured documentation frameworks.The paper identifies governance information, including licensing, as crucial to documentation and says its omission may limit reuse.Source: §6 · pp. 7–8

Data Sharing

  • Some datasets remain unpublished or inaccessible despite stated sharing intentions, and registration requirements and unsustainable links pose further barriers.The paper notes that repositories such as GitHub and GitLab are commonly used for sustainable dataset sharing.Source: §6 · p. 8

Evaluation

  • More than 60% of surveyed datasets lack predefined test splits, making results incomparable when researchers evaluate models on different random splits.Researchers may replicate earlier experiments using a chosen random split.Source: §6 · p. 8

Ethical Concerns and Privacy

  • Crawled datasets may neglect platform and country legal and ethical frameworks, while translated datasets may omit model and verification details; providers are urged to disclose risks.The survey found no explicit risk indications across collection, modelling, evaluation, or deployment, including for social-media and human-produced-literature datasets.Source: §6 · p. 9

7 Limitations and Future Work

The catalogue is limited by its search strategy and inclusion criteria, leaving some datasets outside the study. The authors plan to update it and support community contributions.

7 Limitations and Future Work

  • The search strategy likely retrieves datasets indexed with metadata whose publications contain one of the search key-phrases.Source: §7 · p. 9
  • Some open- or regulated-access data resources, including LDC, were excluded because they did not meet the inclusion criteria.Source: §7 · p. 9
  • The authors plan to keep the catalogue updated and allow dataset authors to submit metadata through community-based contributions.Source: §7 · p. 9

8 Conclusion

The authors conclude that Masader catalogs 200 Arabic NLP datasets with metadata and analyzes the findings, issues, and possible resolutions. They describe this analysis as a snapshot of a rapidly evolving field.

8 Conclusion

  • The study created an online catalogue of 200 Arabic NLP datasets with metadata annotations.Source: §8 · p. 9
  • The authors analyzed the findings, identified issues, and suggested resolutions for Arabic NLP datasets.Source: §8 · p. 9
  • The authors characterize their analysis as a snapshot in a rapidly evolving field and say extra efforts are required to drive Arabic NLP further in that direction.Source: §8 · p. 9

A Extra Analysis

The extra analysis describes dataset hosting, access, licensing, venues, ethical risks, domains, and abstract embeddings. It reports patterns in these areas and outlines how the embedding projections were prepared.

Repositories

Accessibility and licenses

  • Most datasets are free, with more than 80% of data published free in recent years; over 50% of providers do not declare licenses, and around 10% use custom licenses.Source: A Extra Analysis · p. 11, Figure 9 · p. 12

Venues

Abstract Projection

  • The authors extracted dataset abstracts using the Semantic Scholar API, embedded them with sentence-transformers, and projected 200 384-dimensional embeddings into 2D with t-SNE.Source: A Extra Analysis · p. 11
  • The positive and negative x-axis projections are separated for visualization, with 11 manually highlighted clusters grouped by shared task, format, or dataset type.Source: A Extra Analysis · p. 11, Figure 11 · p. 13, Figure 12 · p. 13

Ethical Risks

Domain Representation

Cost

  • Figure 8 concerns dataset costs from 2010 to 2021, but the supplied analysis gives no cost breakdown values.Source: Figure 8 · p. 12
Loading 2110.06744v1…