PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Recognition Challenge in the Wild: Arabic MGB-3

Ahmed Ali, Stephan Vogel, Steve Renals

arXiv:1709.07276v1cs.CL

TL;DR

MGB-3 addresses speech recognition and dialect identification for dialectal Arabic, where orthographic variation, genre diversity, and limited in-domain data shape the evaluation. It uses a multi-genre Egyptian YouTube corpus alongside MGB-2 training data and reports challenge results across the two tasks, including an 80% ADI result for the best system.

  • Problem

    Dialectal Egyptian speech presents orthographic variation, multi-genre conditions, and a low-resource setting with only 16 in-domain hours.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The challenge used 16 hours of Egyptian speech from multi-genre YouTube videos for transcription and reused MGB-2 training data, while evaluating dialect identification with acoustic and lexical information.

    Source: §1 · p. 1, §5 · p. 6

  • Results

    The best ADI system achieved 80% overall accuracy across five classes using GANs and combined acoustic and lexical information.

    Source: §5 · p. 6

  • Takeaways & Limitations

    MGB-3 extended speech-transcription evaluation to non-orthographic dialectal Arabic in a multi-genre YouTube collection.

    Source: §5 · p. 6

    The authors assumed MGB-3 data alone was insufficient to build robust speech-recognition systems, so they reused MGB-2 training data and treated in-domain data as supervised adaptation data.

    Source: §1 · p. 1

Abstract

from arXiv · show

This paper describes the Arabic MGB-3 Challenge - Arabic Speech Recognition in the Wild. Unlike last year's Arabic MGB-2 Challenge, for which the recognition task was based on more than 1,200 hours broadcast TV news recordings from Aljazeera Arabic TV programs, MGB-3 emphasises dialectal Arabic using a multi-genre collection of Egyptian YouTube videos. Seven genres were used for the data collection: comedy, cooking, family/kids, fashion, drama, sports, and science (TEDx). A total of 16 hours of videos, split evenly across the different genres, were divided into adaptation, development and evaluation data sets. The Arabic MGB-Challenge comprised two tasks: A) Speech transcription, evaluated on the MGB-3 test set, along with the 10 hour MGB-2 test set to report progress on the MGB-2 evaluation; B) Arabic dialect identification, introduced this year in order to distinguish between four major Arabic dialects - Egyptian, Levantine, North African, Gulf, as well as Modern Standard Arabic. Two hours of audio per dialect were released for development and a further two hours were used for evaluation. For dialect identification, both lexical features and i-vector bottleneck features were shared with participants in addition to the raw audio recordings. Overall, thirteen teams submitted ten systems to the challenge. We outline the approaches adopted in each system, and summarise the evaluation results.

1. INTRODUCTION

The MGB-3 Arabic Challenge evaluated speech transcription and dialect identification using Egyptian YouTube speech. It broadened the data beyond mainstream broadcast media to address dialect, genre diversity, and limited in-domain resources.

1. INTRODUCTION

  • The challenge covered speech-to-text transcription and Arabic dialect identification across Egyptian, Levantine, Gulf, North African, and Modern Standard Arabic.ADI participants received more than 50 hours of labeled data across the five dialects, with overall accuracy, precision, and recall as evaluation criteria.Source: §1 · pp. 1–2
  • YouTube recordings enabled easier collection of dialectal speech and multiple genres, extending data diversity beyond the BBC and Al Jazeera broadcast media used in earlier MGB challenges.The paper also reports a relatively high MGB-3 baseline word error rate compared with MGB-2.Source: §1 · p. 1
  • The challenge targeted Egyptian Arabic's poorly defined orthography, seven-genre scenarios, and low-resource conditions with only 16 in-domain hours.Those hours were split into adaptation, development, and testing data.Source: §1 · p. 1
  • The MGB-3 speech data comprised 16 hours from 80 Egyptian YouTube videos spanning comedy, cooking, family/kids, fashion, drama, sports, and TEDx science talks.The chosen TEDx talks were in Egyptian dialect.Source: §1 · p. 1
  • The challenge reused MGB-2 training data because the authors assumed MGB-3 data alone was insufficient for robust systems but useful for adaptation and hyper-parameter tuning.The in-domain data was treated as supervised adaptation data.Source: §1 · p. 1

2. MGB-3 DATA

MGB-3 assembled Egyptian speech-recognition data with multiple acceptable transcript spellings and separately prepared acoustic and lexical resources for dialect identification. The data design reflects dialectal orthographic variation and differences in the sources and recording conditions.

2.1. Data for Speech Recognition

  • 80 videos supplied the speech-recognition corpus; clips used each video's first 12 minutes, removed manually identified non-speech, and were split into adaptation, development, and testing sets.The test set was slightly larger than the other two sets.Source: §2.1 · p. 2
  • Because Egyptian Arabic lacks specific orthographic rules, transcribers were allowed spelling variation and multiple transcripts rather than strict standardized spelling guidelines.The authors report about 13% inter-annotator disagreement and report MGB-3 results after surface normalization only.Source: §2.1 · p. 2, Table 2 · p. 2
  • The cited passage says an unspecified evaluation issue can be addressed by using multi-reference WER estimation (MR-WER), but provides no definition of its word-matching rule.Source: §2.1 · p. 2

2.2. Data for Dialect Identification

  • The ADI task classified utterances using acoustic and lexical information, including speech-recognition transcripts and acoustic features supplied for MGB-2 training data.The challenge built on earlier VarDial releases that successively supplied transcripts, lexical features, acoustic features, and audio.Source: §2.2 · p. 2
  • ADI acoustic features were 400-dimensional BNF i-vectors, while the task data drew on broadcast-news training and high-quality Al Jazeera test and development recordings.The test and development recording setup differed from that of the training data.Source: §2.2 · pp. 2–3
  • The LDA projection shows clear separation of MSA from the other four dialects, with similar acoustic-space patterns for development and test data.The projection is of the shared 400-dimensional BNF i-vector features for the test set.Source: §2.2 · p. 3, Figure 1 · p. 3

3. BASELINE RESULTS

The reported baselines provide reference performance for speech recognition and dialect identification. The speech-recognition baseline was trained on MGB-2 data without MGB-3 adaptation, while the ADI baseline used an SVM with lexical and bottleneck features.

3. BASELINE RESULTS

  • The ASR baseline used multi-reference WER, which compares recognized text against multiple manual transcriptions and accepts a word found in any reference.The metric was investigated in response to the reported inter-annotator disagreement on dialectal Arabic.Source: §3 · p. 3

3.2. Dialect Identification Baseline

  • 57.2% overall accuracy, 60.8% average precision, and 58% overall recall were reported for the five-class ADI baseline.The baseline used a multi-class SVM with lexical features from speech recognition and bottleneck features from the i-vector framework.Source: Table 4 · p. 3, §3.2 · pp. 3–4

3.1. ASR Baseline

  • The MGB-2-trained ASR baseline performed poorly on MGB-3 development data, as expected because it was not adapted to the changed data characteristics.The system used the full 1,200 hours of MGB-2 audio without MGB-3 adaptation.Source: §3.1 · p. 3

4. SUBMITTED SYSTEMS AND RESULTS

Thirteen teams submitted ten systems for speech transcription and dialect identification. The challenge reports leading results alongside system strategies for multi-genre Egyptian Arabic and five-dialect classification.

4.1. Speech-to-Text Transcription

  • 13.2% WER on MGB-2 and 37.5% AV-WER / 29.25% MR-WER on MGB-3 were Aalto's best challenge results.Aalto's final system combined more than 30 systems using two acoustic models and character-, subword-, and word-based language models.Source: §4.1 · p. 4, Table 7 · p. 5
  • Leading ASR systems benefited from MGB-2 transfer learning and MGB-3 adaptation, while language-model methods and multi-reference confusion networks also improved systems.Only Aalto used subword language modeling; BUT and JHU explored aligning four transcriptions in confusion networks.Source: §4.1 · p. 4

4.2. Arabic Dialect Identification

  • 79.8% overall accuracy was UTD's later post-release best, while MIT-QCRI led primary ADI submissions with 75.0% overall accuracy.MIT-QCRI's primary result also reported 75.1% precision and 75.5% recall; UTD's later result reported 80.3% precision and 79.9% recall.Source: §4.2 · p. 6, Table 8 · p. 6
  • Leading ADI systems combined acoustic and linguistic features and explored GANs, Siamese networks, and recursive whitening to reduce domain-mismatch effects.MIT-QCRI used acoustic and linguistic features, including character and phonetic features with an SVM backend.Source: §4.2 · p. 6

5. CONCLUSIONS

The challenge evaluates speech recognition on diverse Egyptian dialectal YouTube speech and adds Arabic dialect identification. Its reported outcomes include the best results on both tasks and approaches for reducing transcription error.

5. CONCLUSIONS

  • 13.2% WER on MGB-2, 37.5% AV-WER, and 29.3% MR-WER on MGB-3 were the best speech-transcription results reported.The MGB-2 result was a 10% relative improvement over the best system in the 2016 challenge.Source: §5 · p. 6
  • 80% overall accuracy across five classes was the best ADI result, achieved using GANs and combined acoustic and lexical information.Source: §5 · p. 6
  • Combining multiple reference transcripts with confusion networks and using sub-word language models were among the techniques that reduced error rates.Source: §5 · p. 6
  • The authors plan to add dialects and potentially collect more YouTube recordings to explore transfer learning with in-domain untranscribed speech.Source: §5 · p. 6

Related papers

Loading 1709.07276v1…