Source-linked AI summary
Multilingual and code-switching ASR challenges for low resource Indian languages
Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, Ashish Mittal, Prasanta Kumar Ghosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri, Sunayana Sitaram, Samarth Bharadwaj, Jai Nanavati, Raoul Nanavati, Karthik Sankaranarayanan, Tejaswi Seeram, Basil Abraham
TL;DR
Low-resource multilingual and code-switching ASR in India faces limited corpora and language variation. The challenge provides data and baseline systems for multilingual and code-switching tasks. Reported test and blind-test WERs vary by subtask and baseline system.
Problem
Low-resource Indian languages and scarce large code-switched corpora make multilingual and code-switching ASR challenging.
Source: §1 · p. 1Method
The challenge releases multilingual speech data, two code-switched language-pair datasets, and baseline ASR systems for two subtasks.
Source: §1 · pp. 1–2Results
30.73%/32.73% averaged test/blind WERs for multilingual ASR; code-switching test/blind WERs were 33.35%/28.52% (GMM-HMM), 29.37%/32.09% (TDNN), and 28.45%/34.08% (end-to-end).
Source: §1 · p. 2Takeaways & Limitations
The challenge provides a large transcribed corpus and baselines for research on multilingual and code-switching ASR in the Indian context.
Source: §4 · p. 5Training and development tutorial segments have transcription-timing misalignments, while blind-test transcriptions were manually edited to match the speech.
Source: §2.2.2 · p. 3
Abstract
from arXiv · showhide
Recently, there is increasing interest in multilingual automatic speech recognition (ASR) where a speech recognition system caters to multiple low resource languages by taking advantage of low amounts of labeled corpora in multiple languages. With multilingualism becoming common in today's world, there has been increasing interest in code-switching ASR as well. In code-switching, multiple languages are freely interchanged within a single sentence or between sentences. The success of low-resource multilingual and code-switching ASR often depends on the variety of languages in terms of their acoustics, linguistic characteristics as well as the amount of data available and how these are carefully considered in building the ASR system. In this challenge, we would like to focus on building multilingual and code-switching ASR systems through two different subtasks related to a total of seven Indian languages, namely Hindi, Marathi, Odia, Tamil, Telugu, Gujarati and Bengali. For this purpose, we provide a total of ~600 hours of transcribed speech data, comprising train and test sets, in these languages including two code-switched language pairs, Hindi-English and Bengali-English. We also provide a baseline recipe for both the tasks with a WER of 30.73% and 32.45% on the test sets of multilingual and code-switching subtasks, respectively.
1. Introduction
The challenge targets low-resource multilingual and code-switching ASR in India, where language variation, limited corpora, and code-switching complicate system development. It contributes datasets, baseline systems, and evaluation sets for both tasks.
1. Introduction
- Low-resource languages, many without written scripts, could benefit from speech technology, while code-switching ASR remains challenging because large code-switched corpora are lacking.The paper notes that code-switching commonly occurs between an Indian language and Indian English.Source: §1 · p. 1
- The challenge selects six Indian languages for multilingual ASR and Hindi-English and Bengali-English pairs for code-switching, spanning three major language families, four regions, and a language continuum.The authors expect the selected languages could help extend multilingual ASR to other low-resource languages.Source: §1 · p. 1
- The release provides approximately 600 hours for six Indian languages, an additional 150 hours of code-switched speech, baseline systems, and held-out blind test sets.The code-switched recordings come from technical lectures, while the multilingual recordings span different domains.Source: §1 · p. 2
- 30.73%/32.73% averaged test/blind WERs for multilingual ASR; code-switching test/blind WERs were 33.35%/28.52% (GMM-HMM), 29.37%/32.09% (TDNN), and 28.45%/34.08% (end-to-end).The paper reports results for two subtasks and three code-switching baseline systems.Source: §1 · p. 2
2. Data Details of Two Subtasks
The two subtasks draw on multilingual recordings and code-switched tutorials, with dataset design shaped by language variation and transcription artifacts. The paper describes the sources, validation and normalization procedures, and dataset scope.
2.1. Multilingual ASR
- Shared language properties across families motivate the multilingual subtask, although variation in graphemes, phonemes, and acoustics makes a six-language system challenging.The authors present the publicly released data as an opportunity for future multilingual ASR research.Source: §2.1 · p. 2
2.1.2. Characteristics of the dataset
- Multilingual data combine native-speaker reading recordings with different literacy and domain profiles, and validation procedures retain speech variation while filtering noisy or erroneous audio.Hindi and Marathi selection used WER and lattice-likelihood thresholds; Telugu, Tamil, and Gujarati data came from an earlier challenge, with blind-test audio modified on 29.0%, 23.8%, and 34.1% of the respective data.Source: §2.1.2 · pp. 2–3
2.1.1. Dataset Description
- The multilingual dataset table describes train, test, and blind-test size, channel compression, unique sentences, speakers, and vocabulary; Odia recordings represent four dialect regions.The table caption notes single-channel, 16-bit audio, sampled at 8 kHz except for Telugu, Tamil, and Gujarati train and test data, which use 16 kHz.Source: §2.1.1 · p. 2, Table 1 · p. 3
2.2. Code-switching ASR
- The code-switching subtask addresses the scarcity of public corpora with tutorial speech whose sentence-level transcripts and timestamps include label noise.The tutorials cover computer science topics; the authors identify understanding these data challenges as important for scaling to real-life data and more Indian languages.Source: §2.2 · p. 3
2.2.1. Dataset Description
- Hindi-English and Bengali-English data come from technical tutorials; speaker identities are unavailable, and train/test overlap and out-of-vocabulary rates differ across the two pairs.Test/train overlap is 33.9% for Hindi-English and 10.8% for Bengali-English; test and blind-test OOV rates are 12.5%/19.6% and 22.9%/27.3%, respectively.Source: §2.2.1 · p. 3, Table 2 · p. 3
2.2.2. Characteristics and Artefacts in the Dataset
- Subtitle timestamps can misalign training and development transcripts with speech, whereas blind-test transcripts were manually edited; inconsistent English scripts prompted the transliterated WER metric.T-WER counts an English reference word as correct when predicted in Latin script or its native-script transliteration; blind-test references and mappings were manually prepared.Source: §2.2.2 · p. 3
- Other tutorial-transcription artifacts include enunciated or non-enunciated punctuation, Bengali-English mixed words, incomplete segment audio, and merged English words.Blind-test transcripts were normalized for punctuation, mixed words were transliterated into Bengali, unclear incomplete-audio words were omitted, and merged English words were split.Source: §2.2.2 · p. 4
3. Experiments and Results
The experiments compare multilingual and code-switching ASR baselines across test and blind-test sets, including the effects of language-model mixing and realigned training audio. Results vary by task and evaluation split.
3.1.1. Multilingual ASR
- The multilingual baseline uses a shared six-language lexicon and 3-gram language model with a 6-block TDNN; evaluation reports averaged WER across languages, with and without Marathi’s channel mismatch.Monolingual systems use language-specific training data, lexicons, and language models.Source: §3.1.1 · p. 4
3.1.2. Code-switching ASR
- Code-switching baselines use a shared hybrid DNN-HMM architecture across pairs, alongside an end-to-end CTC-attention Transformer; separate lexicons and trigram language models are built for each pair.The experiments report standard WER and T-WER for code-switched speech.Source: §3.1.2 · p. 4
3.2.1. Multilingual ASR
- Blind-test averaged WER across five languages was 33.47 for Multi versus 29.98 for Mono; multilingual ASR had lower WER only for Tamil among the six languages.The paper notes that multilingual ASR avoids an explicit language-identification system, while its language model can introduce noise through code-mixing.Source: §3.2.1 · p. 4, Table 3 · p. 5
3.2.2. Code-switching ASR
- ReA averaged WERs were consistently better than UnA; TDNN performed better on the test set, while speaker-adapted triphone GMM-HMM performed best on the blind test set.The comparison reports results for Hindi-English and Bengali-English code-switching systems.Source: §3.2.2 · p. 5, Table 4 · p. 5
- For aligned blind-test sets, T-WER is always better than WER; Hindi-English WER gains are attributed to fewer OOVs and more training data.Transliterated scores further improve the blind-set results.Source: §3.2.2 · p. 5, Table 5 · p. 5
4. Conclusion
The challenge presents multilingual and code-switching ASR tasks for low-resource Indian languages, with corpus resources, baseline systems, and held-out blind test sets for evaluation.
4. Conclusion
- The challenge targets data scarcity and code-switching in the Indian context through two ASR subtasks.The paper presents dataset details, baseline recipes, and results for the challenge.Source: §4 · p. 5
- The release provides approximately 600 hours of transcribed speech data, described as a reasonably large corpus for six Indian languages.The paper compares its size with existing publicly available Indian-language datasets.Source: §4 · p. 5
- Baseline systems use hybrid DNN-HMM and end-to-end models, and carefully curated held-out blind test sets are released for evaluation.Source: §4 · p. 5