PaperlayerView exact revision on arXiv

Source-linked AI summary

Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition

Omid Ghahroodi, Anas Madkoor, Dima Faris Al Saudi, Fagr Tahir, Malak Annan, Talha shahid javad allah rakha, Omar Al-Busaidi, Zineb El Kahla, Iheb Zouari, Essa Ahmed Abou Jabal, Ahmed Ezzat, Hind AL-Merekhi, Aisha Hamad M A Al-Naimi, Hadi Wazni, Bushra Alnajjar, Omar Amin, Haya Al-Thani, Houssam Eddine-Othman Lachemat, Marwa Elwakedy, Sundus Abdulmalik Al Nahari, Elahe Zahiri, Osamah Sarraj, Raghad Mousa, Mckeen Assi, Ahd Al Jumah, Heyam Salman, Alhanouf Abdulraqib, Sara Benoumhani, Alia Hamwi, Ayaat Al-Yasseri, Rim Ibrahim Ghazal, Lamia Ben hiba, Mohamed Eltabakh, Fatima Al-Raisi, Yassine El Kheir, Mohammed Abdulrahman, Hamdy Mubarak, Ayah Hashem, Lefkir Meriem, Ehsaneddin Asgari

arXiv:2609.35564v1cs.CLcs.AI

TL;DR

Arabic ASR research has largely focused on Modern Standard Arabic, leaving many spoken dialects under-represented. ALMIEYAR builds a culturally grounded benchmark from newly recorded speech and evaluates 12 systems zero-shot. GPT-4O-TRANSCRIBE has the lowest overall error, while results vary across dialects and show substantial remaining errors.

  • Problem

    Arabic ASR has focused largely on Modern Standard Arabic and a few high-resource dialects, leaving many spoken varieties under-represented.

    Source: §1 · p. 1

  • Method

    ALMIEYAR covers 17 dialects across six families using newly recorded community-guided image descriptions, and evaluates 12 ASR systems zero-shot.

    Source: §1 · p. 1, §3 · p. 2, §4 · p. 2

  • Results

    Across dialect groups, system performance varies; GPT-4O-TRANSCRIBE achieves the lowest overall error at 34.9% WER.

    Source: §5 · pp. 2–3

  • Takeaways & Limitations

    The results support evaluating Arabic ASR across diverse dialect communities rather than relying on a single aggregate score.

    Source: §5 · p. 3, §7 · p. 4

    The benchmark uses only image-description speech and approximately 50 minutes per dialect, limiting register coverage and making some lower-resource dialect comparisons indicative rather than definitive.

    Source: §8 · p. 4

Abstract

from arXiv · show

Arabic speech technology has largely focused on Modern Standard Arabic, leaving the living dialects spoken by hundreds of millions under-served. We introduce ALMIEYAR, a culturally grounded ASR benchmark covering 17 Arabic dialects across six families, built entirely from newly recorded speech unseen by existing models. Dialect-community coordinators selected culturally relevant images across 10 topics, and native speakers described them through five structured scenarios, yielding approximately 50 minutes per dialect (13.7 hours total). We benchmark 12 state-of-the-art ASR systems zero-shot, including GPT-4o-transcribe, Voxtral-Mini-4B, Fanar-STT-LF, Whisper, SeamlessM4T-v2, and wav2vec2-based models. GPT-4o-transcribe achieves the lowest overall WER at 35.0%, followed by Voxtral-Mini-4B, Fanar-STT-LF, and Whisper-Large-v3 at 41.1%, 45.9%, and 49.5%, respectively, indicating substantial remaining errors across Arabic dialect communities. Performance varies considerably across dialect groups, with no model performing uniformly best across all groups. WER alone also obscures dialectal ASR behaviour: wav2vec2-based models show large WER/CER gaps, where character-level agreement remains much higher than word-level accuracy, motivating joint WER/CER reporting. ALMIEYAR provides a unified benchmark for culturally grounded Arabic ASR evaluation, including the first published benchmark for Ahwazi Arabic.

1. INTRODUCTION

Arabic ASR research has centered on MSA and a few high-resource dialects, leaving many spoken varieties under-represented. ALMIEYAR addresses gaps in dialect coverage and data provenance with newly recorded, culturally grounded speech.

1. INTRODUCTION

  • Arabic ASR research has focused on MSA and a small number of high-resource dialects, while many spoken varieties remain under-represented.The passage notes that some varieties, including Ahwazi Arabic, have essentially no prior ASR research.Source: §1 · p. 1
  • ALMIEYAR covers 17 dialects across six families, including the first published ASR benchmark for Ahwazi Arabic.Its newly elicited recordings have not appeared in prior corpora, reducing evaluation-leakage risk.Source: §1 · p. 1
  • Community-curated images across 10 topics ground utterances in contexts and vocabulary that broadcast-derived corpora may miss.The benchmark evaluates 12 ASR systems zero-shot using WER, CER, and MER.Source: §1 · p. 2

2. RELATED WORK

Earlier Arabic ASR benchmarks cover limited dialect ranges and often draw on broadcast or online speech that may overlap with model training data. ALMIEYAR uses community expertise to construct a broader benchmark from culturally grounded image descriptions.

2. RELATED WORK

  • Prior benchmarks cover at most eight dialect varieties, omit dialect families such as Ahwazi, and often use broadcast or YouTube speech that may overlap with training data.ALMIEYAR combines broad dialect coverage with newly generated, culturally grounded speech for contamination-free evaluation.Source: §2 · p. 2

3. THE ALMIEYAR BENCHMARK

  • Dialect-community coordinators contribute cultural expertise and select representative images across 10 topics.The target set covers 17 dialects across six families.Source: §3 · p. 2
  • Speakers describe images through five structured scenarios, yielding approximately 50 minutes per dialect and 13.7 hours overall.Automatic transcription assists speaker correction; coordinators review references for dialect authenticity and code-switching.Source: §3 · p. 2

4. EVALUATION FRAMEWORK

The evaluation compares 12 zero-shot ASR systems on manually reviewed, normalized references using word- and character-level metrics. Results show substantial overall errors, variation across dialect groups, and a gap between character and word recognition for some systems.

4. EVALUATION FRAMEWORK

  • 12 ASR systems are evaluated zero-shot against manually reviewed references normalized consistently for Arabic spelling and punctuation.The evaluation reports WER and CER, with MER also used to account for hypothesis length.Source: §4 · p. 2

5. RESULTS AND DISCUSSION

  • 34.9% WER and 16.5% CER: GPT-4O-TRANSCRIBE has the lowest overall error, followed by VOXTRAL-MINI-4B at 41.1%/20.6%, FANAR-STT-LF at 45.9%/21.0%, and WHISPER-LARGE-V3 at 49.5%/29.0%.Wav2Vec2-based systems and SINAI-STT remain substantially behind at 79–86% WER.Source: §5 · p. 2
  • Large WER/CER gaps for Wav2Vec2-based systems indicate partial character-level agreement despite incorrect word recognition, often involving weak letters and short function morphemes.Dialect-to-MSA substitutions occur but are not the dominant failure pattern.Source: §5 · p. 3
  • Performance varies across dialect groups, and no system is uniformly best; dialect-level gaps reach 16.8 points in Maghrebi and 12.8 points in Gulf.The authors caution that family averages can hide dialect-specific behaviour.Source: §5 · p. 3
  • 30.6% WER: GPT-4O-TRANSCRIBE leads on Sudanese Arabic, while lower-performing systems exceed 74% WER.The supplied results also show variation in Egyptian performance, where SEAMLESSM4T-V2 achieves 43.9% WER.Source: §5 · p. 3

6. MATCH ERROR RATE RESULTS

MER results broadly agree with the WER/CER comparison, while adding a word-event normalization that is more conservative than WER when hypotheses contain more words than references.

6. MATCH ERROR RATE RESULTS

  • MER broadly reproduces the Table 2 pattern, with Wav2Vec2 systems and SINAI-STT among the highest-error systems despite substantial character-level overlap.MER is more conservative than WER when hypotheses contain more words than references.Source: §6 · p. 3

WER (%) CER (%)

The tables report model errors by dialect family, using WER/CER in Table 2 and MER in Table 3.

WER (%) CER (%)

  • Table 2 reports family-level WER (%) and CER (%) per model, with the overall average included.Its cells encode lower-to-higher values from green to red, and bold marks the best value per column.Source: Table 2 · p. 4
  • Table 2 groups Bahraini, Omani, Qatari, Saudi, and Yemeni under Gulf, and groups Iraqi and Ahwazi together.Its other family abbreviations cover Levantine, Maghrebi, Egyptian, Sudanese, and the overall average.Source: Table 2 · p. 4
  • Table 3 reports MER (%) per model and dialect family, with cells colour-coded from low to high.The best value in each column is bolded.Source: Table 3 · p. 4

7. CONCLUSIONS

ALMIEYAR introduces a culturally grounded, contamination-free benchmark spanning 17 dialects across six families, including Ahwazi Arabic. Zero-shot evaluation finds substantial errors and variation across systems and dialect groups, supporting evaluation across diverse communities and multiple metrics.

7. CONCLUSIONS

  • GPT-4o-transcribe achieves the strongest overall performance at 34.9% WER across the 12 zero-shot systems.The benchmark covers 17 dialects across six families and includes the first published Ahwazi benchmark.Source: §7 · p. 4
  • ALMIEYAR uses entirely newly recorded speech and community-guided image descriptions to benchmark Arabic ASR across 17 dialects and six families.It includes the first published Ahwazi benchmark.Source: §7 · p. 4
  • Performance varies across systems and dialect groups, and the WER/CER analysis shows architecture-specific failure patterns.The authors emphasize evaluation across diverse communities rather than relying on a single aggregate score, and multi-metric evaluation.Source: §7 · p. 4

8. LIMITATIONS

The authors identify limits in elicitation scope, per-dialect scale, dialect categorization, evaluation setting, and reference transcription quality.

8. LIMITATIONS

  • The benchmark uses image-description speech alone, leaving conversational, read, broadcast, and domain-specific registers outside its coverage.Examples of omitted domains include medical, legal, and technical discourse.Source: §8 · p. 4
  • At approximately 50 minutes per dialect, estimates for lower-resource families carry non-trivial variance, making single-dialect comparisons between similar models indicative rather than definitive.The authors name Ahwazi, Sudanese, and Iraqi as examples.Source: §8 · p. 4
  • The 17-way dialect labelling is a working approximation because Arabic dialects form a continuum.Speaker metadata is recorded but not used as a stratification variable in headline results.Source: §8 · p. 4
  • Evaluation is zero-shot only; performance after dialect-specific fine-tuning remains future work.Source: §8 · p. 4
  • Coordinator review reduces but does not eliminate reference ambiguity, especially at code-switching boundaries and where dialect-internal spelling lacks a single norm.Source: §8 · p. 4

9. ETHICAL CONSIDERATIONS

ALMIEYAR distinguishes community coordinators’ substantive intellectual contributions from participants’ bounded recording contributions, while addressing consent, privacy, representation, and intended use.

9. ETHICAL CONSIDERATIONS

  • Dialect-community coordinators curate images, recruit and supervise speakers, calibrate prompts, review transcripts, and adjudicate orthographic conventions; they are co-authors.Their authorship follows the paper’s stated CRediT/ICMJE criteria.Source: §9 · pp. 4–5
  • Native-speaker participants provide one-time recordings of approximately 50 minutes and receive monetary compensation and, with consent, named acknowledgement.Compensation follows local minimum-wage benchmarks and approved IRB rates.Source: §9 · p. 5
  • Speakers consent in writing to research use and public release, with forms in their dialect and a right to withdraw before publication.Oral readback is used where literacy is uneven.Source: §9 · p. 5
  • Released audio retains dialect labels and study-internal speaker IDs, while names, contact details, granular demographics, and incidental personal information are redacted or pseudonymised.Source: §9 · p. 5
  • Community coordinators guide cultural authenticity, and ALMIEYAR is intended for ASR evaluation rather than training or individual-level claims about speakers.The dataset, scripts, outputs, and per-utterance reports are to be released with a datasheet documenting methods and known biases.Source: §9 · p. 5

A. FIVE-SCENARIO ELICITATION INSTRUCTIONS

Coordinators received instructions in English and dialectal written Arabic, and speakers described each image through five structured scenarios covering its cultural context, subjects, surroundings, sensory qualities, and meaning.

B. TELEGRAM BOT COLLECTION INTERFACE

The Telegram bot guided participants through recording and correcting descriptions for each scenario, while storing metadata for each submission and enabling separate coordinator review.

Loading 2609.35564v1…