PaperlayerView exact revision on arXiv

Source-linked AI summary

Bulbul: A Dataset for Dialectal Arabic Speech Recognition

Ahmed Ashraf, Aisha Alansari, Fadel Al Abbas, Nada Almarwani, Samah Aloufi, Saad Ezzini, Maged S. Al-Shaibani, Doaa Dalaq, AbdelRahim A. Elmadany, Muhammad Abdul-Mageed, Mohamed Mehdi Trigui, Dania Refai, Layan Refai, Mohamed Akrout, Mustafa Jarrar, Wasfi G. Al-Khatib, Alaa Dalaq, Darin El-Nakla, Samir Abdaljalil, Abdulrahman Al-Fakih, Nour El Imane Zeghib, Moussa Redah, Salmane Chafik, Mohamed El-Attar, Rima Grati, Sarah Kohail, Malak Alkhorasani, Khadijah Al Safwan, Ismail M. Mudhaffar, Ali Altam, Ahmed Al-Shaikh, Adnan Saeed, Hamzah Luqman

arXiv:2608.21950v1cs.CLcs.AI

TL;DR

Arabic ASR datasets leave many dialects, accented MSA, and classical Arabic comparatively underrepresented. BULBUL is a community-driven corpus from 275 participants across 11 Arab countries; its benchmark reports OmniLLM-7B as best overall for dialectal speech, with a mean WER of 41.7 and CER of 12.7.

  • Problem

    Arabic ASR datasets cover few dialects, leave several regional varieties low-resource, and have largely overlooked accented MSA and classical Arabic.

    Source: §1 · p. 2

  • Method

    BULBUL is a community-driven Arabic speech corpus assembled from 275 participants across 11 Arab countries.

    Source: §3 · p. 2

  • Results

    41.7 mean WER and 12.7 CER were reported for OmniLLM-7B, which achieved the best overall dialectal-speech performance and outperformed baseline multilingual ASR models across regional clusters.

    Source: §6 · p. 7

  • Takeaways & Limitations

    Dialectal variation poses a greater challenge to the evaluated multilingual ASR systems than standardized MSA/CA speech.

    Source: §6 · p. 7

    BULBUL does not comprehensively cover all sub-dialects, since collecting samples from rural and hard-to-access areas presents logistical challenges.

    Source: §8 · p. 9

Abstract

from arXiv · show

Arabic automatic speech recognition (ASR) faces unique challenges due to diglossia, extensive regional dialect variation, and limited speech resources. Existing speech datasets often focus on single dialects or large-scale broadcast/web data, leading to trade-offs between linguistic diversity and annotation quality. We present BULBUL, a multi-dialect Arabic ASR dataset collected from 275 speakers in 11 Arab countries. BULBUL includes structured dialect and sub-dialect coverage, as well as recordings of classical Arabic and modern standard Arabic spoken by participants in their native dialectal accents to support accent-aware modeling. The quality of the recordings was ensured through a two-level human verification process. We further benchmark a range of recent ASR systems, establishing strong baselines for modern dialectal and accented Arabic ASR.

1 Introduction

BULBUL addresses imbalanced Arabic ASR resources and gaps in coverage of underrepresented dialects and accented formal Arabic. It provides speech from 11 countries, including fine-grained dialect coverage, accented MSA and CA, and verified transcriptions.

1 Introduction

  • Arabic ASR resources are limited and imbalanced, while Algerian, Sudanese, Yemeni, Tunisian, and some Palestinian and Jordanian varieties remain comparatively low-resource.Existing datasets largely cover a few dialects, and dedicated systematic evaluation of accented MSA and CA across dialectal backgrounds is lacking, to the authors' knowledge (Sullivan et al., 2026; Talafha et al., 2024).Source: §1 · p. 2
  • BULBUL covers 11 Arab countries to improve dialectal coverage in Arabic ASR.The dataset includes speakers from Algeria, Egypt, Jordan, Morocco, Palestine, Saudi Arabia, Sudan, Syria, Tunisia, the United Arab Emirates, and Yemen.Source: §1 · p. 2
  • Saudi and Yemeni dialects receive fine-grained sub-dialect coverage, alongside accented MSA and CA for accent-aware evaluation.The dataset also draws text from multiple sources and domains for lexical and topical diversity.Source: §1 · p. 2
  • A two-level human verification process checks transcription accuracy, audio quality, and dialect authenticity; BULBUL also provides ASR baselines and demographic metadata.Age and gender metadata support demographic analysis and fairness-aware model evaluation.Source: §1 · p. 2

2 Related Work

Prior Arabic speech resources trade broad linguistic coverage for dialectal depth, or capture multiple dialects with domain and annotation limitations. BULBUL addresses these gaps with naturally diverse, fine-grained dialect speech and accented MSA and CA recordings.

2 Related Work

  • Multilingual Arabic resources such as GlobalPhone and Mozilla Common Voice offer cross-lingual support but limited dialect diversity, while uni-dialect corpora provide depth with narrow geographic reach.Uni-dialect resources also use controlled recording conditions, limiting generalizability across the Arabic-speaking world.Source: §2 · p. 2
  • Multi-dialect datasets including OrienTel, Fisher Levantine, QASR (Mubarak et al., 2021), MGB (Ali et al., 2019a), and MASC (Al-Fetyani et al., 2021) broaden coverage but often have domain bias and inconsistent annotation.Broadcast, telephone, and web-sourced recordings also limit representation of spontaneous community speech.Source: §2 · p. 2
  • BULBUL combines naturally diverse speech across dialect families, some explicit sub-dialect annotations, and accented MSA and CA recordings from 11 dialect speakers.Human-verified transcriptions, demographic metadata, and balanced domain diversity support a more realistic and comprehensive Arabic ASR benchmark.Source: §2 · p. 2

3 BULBUL Dataset

BULBUL is a community-driven Arabic ASR dataset built with 275 participants across 11 Arab countries, covering dialectal speech and classical and modern standard Arabic spoken in participants’ native accents. Its collection combines regionally varied text and guided recordings with two levels of human verification.

3 BULBUL Dataset

  • 275 participants across 11 Arab countries, organized into country teams, contributed to BULBUL; 160 were male and 115 female.Country leaders guided recording and verified both raw text and recorded audio.Source: §3 · pp. 2–3

3.1 Data Collection

  • The dataset spans dialectal Arabic and CA and MSA spoken in participants’ native accents, enabling dialectal and accent-aware speech recognition.Collection expanded from Saudi Arabia and Egypt to 11 countries; it used white dialects in seven countries, Levantine in Syria, and Abu Dhabi dialect in the UAE.Source: §3.1 · p. 3
  • Saudi coverage distinguishes six sub-dialects, combining Qatifi and Hassawi as Eastern; Yemeni coverage includes Sana’ani and Ta’izzi.The sub-dialect selections reflect available data and speaker contributions from those regions.Source: §3.1 · p. 3
  • Dialect texts drew on public datasets, social media, and private messaging; accented MSA and CA used Targamat, with 129 MSA and 200 CA sentences after filtering.GPT-5-mini classified texts into 11 thematic domains, with human review described in the appendix.Source: §3.1 · p. 4

3.2 Recording Process

  • Participants recorded country- and, where applicable, sub-dialect-organized texts using a Gradio tool, written guidelines, and an instructional video.Saudi volunteers were recruited through the national volunteer platform and credited one hour of voluntary work per 10 recorded minutes; country leaders selected participants elsewhere.Source: §3.2 · p. 4

3.3 Human Sanity Check

  • BULBUL used optional speaker self-verification before submission and external review by a native member from the corresponding country.Reviewers received written guidelines and an explanatory video.Source: §3.3 · p. 4

4 BULBUL Analysis

BULBUL combines 36,949 dialectal utterances with 2,325 accented MSA/CA recordings, while its coverage varies across dialects, speaking tempos, demographics, and topics. The analysis also describes unusually long recordings and a speaker-disjoint development/test split of human-verified data.

Cross-dialectal Speech Tempo

  • Dialectal tempo ranges from Egyptian at 1.94 WPS to Yemeni at 1.09 WPS, while accented MSA/CA averages 1.65 WPS versus 1.46 WPS for dialectal speech.Yemeni speakers shift from the slowest dialectal tempo to 1.75 WPS in accented speech; the speech-style comparison is qualified by different recording conditions, including read speech.Source: §4 · p. 5

Overall Dataset Distribution

  • BULBUL contains 36,949 dialectal utterances over 61.46 hours and 2,325 accented MSA/CA recordings over 10.42 hours, with Sudanese speech under one hour.Dialectal recordings are longest for Yemeni (18.97 hours), Saudi (11.89 hours), and Syrian (8.21 hours); limited availability of Sudanese participants contributed to the lower coverage.Source: Table 2 · p. 5, §4 · p. 5

Outlier Analysis

  • Yemeni Arabic has a maximum recording duration of 348,96 seconds and Saudi Arabic 152.340 seconds, both far above average segment durations.The paper suggests that such long recordings likely represent long scripts.Source: Table 2 · p. 5, §4 · p. 5

Sub-dialectal Distribution

  • Yemeni and Egyptian speakers have the longest accented MSA/CA recordings, while UAE and Tunisian have the shortest; sub-dialect coverage also differs within Saudi Arabia and Yemen.Accented durations are 2.82 hours for Yemeni, 2.02 for Egyptian, 0.02 for UAE, and 0.11 for Tunisian. Hijazi has 7,120 utterances and 10,73 hours; Ta’izzi has nearly twice Sana’ani’s utterances and approximately 1.6 times its speech hours.Source: Table 3 · p. 5, Table 2 · p. 5, §4 · p. 5

Gender Distribution

  • Gender representation is uneven across countries: Sudan has only male participants, while the UAE has a strong female majority of ∼89.5%.Algeria, Jordan, Saudi Arabia, and Tunisia have approximately balanced distributions; equal gender representation was a recruitment objective rather than an enforced quota, given voluntary participation and participant availability.Source: Figure 3 · p. 6, §4 · p. 6

Data Splits

  • Development and test sets use human-verified data split 50/50 within each country and are speaker-disjoint.The split avoids imposing a fixed absolute duration because verified-data duration varies by country; Table 9 reports split utterances, durations, and speakers.Source: §4 · p. 6

Domain Distribution

  • The Social domain makes up 65.4% of the corpus, followed by Economy at 15.7%, with each remaining domain more sparsely represented.The analysis attributes this concentration to the prevalence of conversational and socially oriented content in community-contributed speech, while noting broad topic coverage.Source: Figure 4 · p. 6, §4 · p. 6

5 Evaluation

The evaluation benchmarks multilingual ASR models on BULBUL in a zero-shot test-set setting, using default decoding and a shared normalization pipeline for CER and WER.

Evaluated Models

Evaluation Setup

  • Models are evaluated zero-shot on the BULBUL test set, without fine-tuning on BULBUL or any country-specific subset, using default decoding configurations.These settings measure out-of-the-box generalization and assess cross-lingual and cross-accent robustness.Source: §5 · p. 6

Evaluation Protocol

  • The same text normalization pipeline is applied to reference and predicted transcripts so that CER and WER reflect transcription quality consistently.Normalization details are described in Section F.1.Source: §5 · p. 6

6 Results and Discussion

BULBUL results show that OmniLLM-7B performs best overall, with lower error rates on accented MSA/CA than on dialectal speech. The error analysis identifies phonetic confusions and dialect-driven pronunciation variation as recurring sources of transcription errors.

6 Results and Discussion

  • The evaluation groups countries into Northwest African, Arabian Peninsula, Levantine, and Nile Valley regional dialect categories.The groups are based on geographic proximity and established Arabic dialect classifications.Source: §6 · p. 7
  • Palestinian and Saudi dialects have the lowest error rates across most models, while Algeria and Sudan have the highest.Palestinian speech yields the lowest WER for 9 of 12 systems; the paper suggests limited domain diversity contributes to easier recognition. Sudanese difficulty may reflect limited public speech data, while Algerian difficulty is linked to lexical and phonological distance from MSA, non-standard orthography, spontaneous speech, and longer utterances.Source: §6 · p. 7
  • Levantine-accented speech has the lowest error rates overall, including OmniLLM-7B’s Syrian-accented WER/CER of 8.1/1.7; Sudanese-accented speech exceeds 20% WER even for the strongest systems.Figure 5 shows the dialectal-versus-accented WER gap is largest for North African and Yemeni dialects, indicating greater difficulty from spontaneous dialectal variation than accented formal speech.Source: §6 · p. 7, Figure 5 · p. 7
  • Within OmniASR-LLM, performance improves almost monotonically with increasing model size, with larger models yielding lower error rates in both metrics; by contrast, OmniASR-CTC scaling is less stable.Source: §6 · p. 7
  • OmniLLM-7B achieves the best overall dialectal performance, with mean WER 41.7 and CER 12.7, while Omni-based models consistently outperform baseline multilingual ASR systems.The paper attributes this advantage to the LLM decoder’s contextual understanding, which may help select appropriate words and generalize to unseen regional dialects in zero-shot settings.Source: §6 · p. 7, Table 4 · p. 8
  • On accented MSA/CA, OmniLLM-7B achieves mean WER/CER of 13.3/4.8, ahead of OmniLLM-3B (13.7/5.1) and OmniLLM-1B (14.2/5.3).Seamless scores 14.4/5.4, while MMS1B performs worst at 33.8/9.7; performance improves substantially across all models relative to dialectal speech.Source: §6 · p. 7, Table 5 · p. 8

6.1 Error Analysis

  • OmniASR-LLM-7B errors include confusions between acoustically similar phonemes, especially when dialectal pronunciations diverge from standard forms.Examples include emphatic /sˇ/ versus plain /s/, weakened pharyngeal /Q/, and /q/ realized as a glottal stop.Source: §6.1 · p. 8
  • Dialect-driven pronunciation variation causes inconsistent orthographic transcriptions, including variation in imperfective prefixes, reduced-vowel deletions, and variable realizations of h.The analysis also notes that character-level errors often reflect systematic phoneme-to-grapheme mismatches, while word-level errors more often affect proper nouns and low-frequency vocabulary.Source: §6.1 · p. 8

7 Conclusions

BULBUL is a community-driven Arabic speech corpus designed to reflect the Arab world’s linguistic diversity beyond dominant dialects and MSA. Its benchmark findings show uneven ASR support across regional varieties and substantial speech-community heterogeneity.

7 Conclusions

  • BULBUL introduces a large-scale, community-driven Arabic speech corpus designed to reflect linguistic diversity beyond dominant dialects and MSA.Source: §7 · p. 8
  • 11 countries and 16 dialects, with fine-grained sub-dialect annotations, make BULBUL a challenging benchmark under diverse regional and demographic conditions.Source: §7 · pp. 8–9
  • Accented MSA and CA recordings enable analysis of accent transfer and pronunciation variation across standardized speech forms.Source: §7 · p. 9
  • Benchmark evaluations reveal substantial performance disparities across countries and dialects, showing current ASR models unevenly support underrepresented Arabic varieties.Source: §7 · p. 9
  • Lexical-overlap and speaking-rate analyses show heterogeneity across Arabic speech communities, highlighting limits of evaluation based only on coarse dialect categories.Source: §7 · p. 9
  • BULBUL’s release aims to support inclusive Arabic speech research, including dialect-aware ASR, dialect identification, accent adaptation, and speech generation for low-resource varieties.Source: §7 · p. 9

8 Limitations

BULBUL’s controlled, neutral recordings and uneven coverage across populations, sub-dialects, domains, and language varieties limit the scope and generalizability of its evaluations. These constraints should inform interpretation of emotion, noise robustness, and cross-country performance findings.

Emotionally Neutral Recordings

  • Neutral, low-noise recordings limit applicability to emotion recognition and noisy, unconstrained speech, so evaluations primarily reflect dialectal understanding.Participants used a formal neutral tone and avoided excessive background noise and overlapping speech.Source: §8 · p. 9

Country and Gender Imbalance

  • Some countries are underrepresented and gender distributions across dialects are imbalanced, largely because participation was voluntary.The paper provides statistical analysis to characterize these imbalances and support interpretation of findings.Source: §8 · p. 9

Sub-Dialect Coverage Limitation

  • Sub-dialect coverage is restricted, since comprehensive sampling would require collecting speech in rural and hard-to-access areas.Dialectal variation occurs both between and within countries, making comprehensive collection logistically challenging.Source: §8 · p. 9

Domain Coverage Imbalance

  • Limited domain diversity in some country subsets, such as Palestine’s concentration in the economy domain, may make test data more homogeneous and easier for ASR models.This imbalance should be considered when interpreting cross-country performance differences.Source: §8 · p. 9

Imbalance in CA and MSA Coverage

  • CA and MSA subsets include few participants, reducing speaker balance and diversity in speaking styles, accents, and acoustic conditions.Their observed performance may not generalize to broader speaker populations, so findings for CA and MSA warrant caution.Source: §8 · p. 9

9 Ethics and Data Statement … B.3 Domain Classification Schema

BULBUL’s supplementary materials describe ethical safeguards and text-processing procedures alongside prior Arabic speech resources and domain annotation. Domain labels span most countries, though coverage is imbalanced.

9 Ethics and Data Statement

  • Participants consented to research use, and privately contributed chat conversations were included only with their owners’ explicit permission.Conversational texts were anonymized; only age and gender metadata remain, and original chat logs are not distributed. The dataset is for non-commercial academic use, with safeguards against identification and surveillance; demographic and regional imbalances may affect model performance, motivating fairness evaluation.Source: §9 · pp. 9–10

Appendix

  • Appendix B provides further details on BULBUL’s recorded textual samples.The appendices also cover related work, recording and verification procedures, dataset splits, and benchmark setup.Source: Appendix · p. 13

A Related Work

  • Prior Arabic speech datasets vary in coverage and collection conditions, with broadcast bias, web-audio variability, and inconsistent annotation remaining challenges for robust ASR.Multilingual resources often prioritize language interaction over Arabic dialect diversity, while uni-dialect corpora have narrow geographic scope.Source: A Related Work · pp. 13–14

B Transcription Content

  • Section B introduces additional details about the textual samples recorded in BULBUL.These materials provide further context for the dataset’s transcription content.Source: B Transcription Content · p. 14

B.1 Text Pre-processing

  • BULBUL’s text-cleaning pipeline normalizes Arabic text, removes formatting artifacts and emoticons, standardizes repeated characters and whitespace, and trims punctuation.For the Tarjamat MSA/CA subset, it also removes numerical digits to avoid inconsistencies when speakers verbalize numbers differently.Source: §B.1 · p. 14

B.2 Transcription Samples from BULBUL

  • BULBUL’s transcription samples illustrate linguistic diversity through representative examples from participating countries and English translations.Table 6 presents the samples.Source: §B.2 · p. 14, Table 6 · p. 15

B.3 Domain Classification Schema

  • The domain schema defines 11 topical categories and prompts annotators for one primary label, an optional second label, justification for Other, reasoning, and confidence.All cases scored below 0.70 were manually reviewed; the Other category requires explicit justification.Source: §B.3 · pp. 14–15, Figure 6 · p. 15, Table 7 · p. 16
  • Religion, Education, and Entertainment are represented in 10 of 11 countries; Crime and Technology in 8, while Politics and Sports each appear in 7. Together, these results indicate broad topical consistency across regional subsets.Palestine consists exclusively of economically oriented utterances.Source: §B.3 · p. 15

B.4 Lexical and Semantic Analysis … D.2 Verification Guidelines

BULBUL’s analysis finds lexical variation across countries and sub-dialects alongside broad sentence-level semantic similarity; the paper also describes participant recording procedures and a structured review process for verifying speech data.

B.4 Lexical and Semantic Analysis

  • 1,464 shared vocabulary items linked Sana’ani and Ta’izzi, while Saudi sub-dialects also diverged phonologically despite shared orthography.Lexical diversity varied with corpus scale and domain coverage; semantic similarity was generally above 0.75, with a small-sample caveat and comparatively lower Palestinian scores.Source: §B.4 · pp. 16–17

C Speech Recording

  • Recording procedures and guidelines covered speech data for dialectal Arabic and formal Arabic spoken with dialectal accents.Source: C Speech Recording · p. 17

C.1 Recording Tools

  • A Gradio web interface let participants record country-specific text, review, trim, repeat, or skip samples, and track progress and a country leaderboard.Source: §C.1 · pp. 17–18

C.2 Recording Guidelines

  • Recording guidance asked participants to use a quiet environment, speak clearly and naturally without changing words except for local pronunciation, and skip difficult sentences if needed.An external microphone or headset was preferred; participants could edit text before recording and save, rerecord, or skip each sample.Source: §C.2 · p. 18

C.3 Recording Agreement Letter

  • Volunteers had to agree to a consent form presented in Arabic, their native language; the provided English version was for clarity.Source: §C.3 · p. 18

D Speech Verification

  • Speech verification procedures covered the tools and guidelines used to verify recorded dialectal Arabic speech.Source: D Speech Verification · p. 18

D.1 Verification Tool

  • A secondary reviewer could accept or reject each sample, with predefined rejection reasons intended to make evaluation decisions more consistent and less susceptible to bias.Source: §D.1 · p. 18

D.2 Verification Guidelines

  • Reviewers could reject samples for unclear audio, text–audio mismatch, prolonged silence, dialect mismatch, or other reasons, with required sub-reasons for unclear audio and mismatches.Samples rejected for unclear recording were designated for publication as a separate noise set; reviewers could add a free-text comment under Other.Source: §D.2 · p. 18

E Dataset Split Statistics

This section describes the composition of evaluation samples underlying the reported ASR results, including dataset-split statistics and the accented MSA/CA recordings.

E Dataset Split Statistics

  • Table 9 reports utterance counts, total minutes, and speaker counts across dialectal and accented dataset splits.Its statistics cover development and test splits for dialect datasets and the accented MSA/CA dataset.Source: Table 9 · p. 19, E Dataset Split Statistics · p. 19
  • Development splits were used primarily for error analysis, while each country's accented MSA/CA recordings came from a single speaker because speaker availability was limited.Source: E Dataset Split Statistics · p. 19

F Benchmarking Experiment Setup · F.1 Text Normalization

The benchmarking setup used pre-trained ASR models for inference across varied computing environments, with consistent configurations for fair comparisons. CER and WER evaluation applied the same normalization pipeline to references and predictions, removing formatting differences through standardized text processing.

F Benchmarking Experiment Setup

  • Experiments used NVIDIA T4, A100, and H100 GPUs on Google Colab and RTX 3090, RTX A6000, and A100 GPUs on institutional servers, selected by scale and availability.Source: F Benchmarking Experiment Setup · p. 19
  • The systems were implemented in Python 3, with separate execution environments for each model family using isolated Anaconda environments on servers and corresponding Colab notebooks.Source: F Benchmarking Experiment Setup · p. 19
  • Experiments used pre-trained ASR models for inference without additional training or fine-tuning, so hardware differences primarily affected execution time and resource use rather than transcription quality.Source: F Benchmarking Experiment Setup · p. 19
  • Fair, reproducible comparisons used consistent model weights, inference settings, and software configurations across environments.Source: F Benchmarking Experiment Setup · p. 19

F.1 Text Normalization

  • The same normalization pipeline was applied to references and predictions, using Unicode NFKC normalization and removing Tatweel characters and Arabic diacritics.Source: §F.1 · p. 19
  • The pipeline converted Arabic-Indic numerals to ASCII, standardized Alef variants, removed punctuation and special symbols, and collapsed whitespace after trimming leading and trailing spaces.Source: §F.1 · p. 19
Loading 2608.21950v1…