Source-linked AI summary
Multilingual Speech Recognition With A Single End-To-End Model
Shubham Toshniwal, Tara N. Sainath, Ron J. Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, Kanishka Rao
TL;DR
Conventional multilingual ASR relies on language-specific units and models, creating challenges when resources are limited. This paper trains a single grapheme-based sequence-to-sequence model jointly on nine Indian languages, then adds language identity as an input. The language-agnostic model improves weighted average WER by more than 21% relative to monolingual models, while language conditioning provides further gains and removes cross-language confusion.
Problem
Conventional multilingual ASR uses language-specific sub-word units, lexicons, pronunciation models, and language models, while training resources are limited for many languages.
Method
The paper jointly trains a grapheme-based LAS sequence-to-sequence model on the union of datasets and character sets from nine Indian languages, with optional language conditioning.
Results
More than 21% relative reduction in weighted average WER is achieved versus monolingual models, and language-conditioned joint models outperform the unconditioned joint model.
Takeaways & Limitations
A single sequence-to-sequence model can recognize the nine languages without runtime language specification, while encoder language conditioning yields the largest language-dependent improvement.
Takeaways & Limitations
The joint model cannot handle code-switching, suggesting that its language model dominates the acoustic model.
Abstract
from arXiv · showhide
Training a conventional automatic speech recognition (ASR) system to support multiple languages is challenging because the sub-word unit, lexicon and word inventories are typically language specific. In contrast, sequence-to-sequence models are well suited for multilingual ASR because they encapsulate an acoustic, pronunciation and language model jointly in a single network. In this work we present a single sequence-to-sequence ASR model trained on 9 different Indian languages, which have very little overlap in their scripts. Specifically, we take a union of language-specific grapheme sets and train a grapheme-based sequence-to-sequence model jointly on data from all languages. We find that this model, which is not explicitly given any information about language identity, improves recognition performance by 21% relative compared to analogous sequence-to-sequence models trained on each language individually. By modifying the model to accept a language identifier as an additional input feature, we further improve performance by an additional 7% relative and eliminate confusion between different languages.
1. INTRODUCTION
Multilingual ASR is difficult because conventional systems use language-specific components, while sequence-to-sequence models can combine these components in one jointly trained network. The paper applies this approach to nine Indian languages and reports gains over independently trained monolingual models, with further improvement from language conditioning.
- Motivation: Conventional multilingual ASR systems often retain language-specific pronunciation and language models, requiring language identity during inference.Their independently optimized components can also propagate errors between stages.
- Approach: Sequence-to-sequence models combine acoustic, pronunciation, and language modeling in one network using the union of language-specific grapheme sets.The model is trained jointly on data from all languages.
- Findings: A jointly trained LAS model across 9 Indian languages consistently outperforms independently trained monolingual LAS models without explicit language specification.The model rarely confuses languages despite lacking runtime language information.
- Findings: Conditioning the encoder on speech language identity produces the largest improvement among the tested language-dependent variants.The paper also studies synthesized data, code-switching, and transliteration behavior.
2. MODEL
The paper uses an attention-based LAS encoder-decoder that predicts grapheme sequences from acoustic features, then extends it to multilingual training and language-aware variants. Joint training uses combined language datasets, while multitask and conditional models incorporate language identity in different ways.
- 2.1. LAS Model: The LAS model contains jointly trained encoder, decoder, and attention modules that predict graphemes from acoustic feature frames.The encoder processes acoustic inputs into hidden states, while the decoder generates character sequences.
- 2.1. LAS Model: The decoder models previous character context and uses attention-derived context from encoder states to compute output probabilities.Its recurrent state is updated from the previous character embedding, prior decoder state, and context.
- 2.2. Multilingual Models: The multilingual model combines language-specific training datasets and grapheme sets, then trains one LAS model without explicit language indication.This joint model recognizes multiple languages without runtime language specification.
- 2.2.2. Multitask: The multitask variant jointly performs speech recognition and language prediction during training, combining the ASR and language-identification losses with weight λ.Language identity is used as a training annotation but not passed as an inference input.
- 2.2.3. Conditional Models: Conditional variants learn language embeddings and feed them into the encoder, decoder, or both during inference.The encoder-conditioned, decoder-conditioned, and encoder-plus-decoder variants differ in where language identity enters the model.
3. EXPERIMENTAL SETUP
Experiments use nine Indian languages with substantial multilingual training and test data, mostly distinct scripts, and a large union grapheme vocabulary. Monolingual and multilingual baselines use recurrent LAS models with configurations tuned initially on Marathi.
- 3.1. Data: The dataset contains about 1500 hours of training data and 90 hours of test data across nine Indian languages.The utterances are dictated queries collected using desktop and mobile devices.
- 3.1. Data: The languages have little character-set overlap except Hindi and Marathi, producing a multilingual output vocabulary of 964 characters.Separate validation sets of around 10k utterances per language support hyperparameter tuning.
- 3.2. Models: Nine monolingual baseline models are trained independently, with hyperparameters tuned on Marathi and reused for the other languages.The selected baseline uses a four-layer bidirectional encoder and two-layer decoder.
- 3.2. Models: The multilingual model uses a larger five-layer encoder with 700 bidirectional LSTM cells and a two-layer decoder with 1024 LSTM cells per layer.Its larger capacity is enabled by the substantially larger multilingual training corpus.
- 3.2. Models: All models are implemented in TensorFlow and trained with asynchronous stochastic gradient descent using 16 workers.Monolingual and multilingual models use initial learning rates of 1e-3 and 1e-4, respectively.
4. RESULTS
The joint multilingual LAS model outperforms language-specific models across all nine languages, while language conditioning provides further gains and nearly eliminates cross-language confusion.
- More than 21% relative reduction in weighted average WER was achieved by the joint model versus nine independently trained monolingual models.The comparison covers all nine languages and uses weighting by number of words.
- The joint model’s performance gain is attributed partly to the Indian languages’ phonetic similarity despite different grapheme sets.
- All joint models conditioned on language ID outperform the unconditioned joint model.The evaluated variants condition the encoder, decoder, or both on language identity.
- Encoder conditioning performs better than decoder conditioning, indicating acoustic adaptation to different languages and accents.Conditioning both encoder and decoder adds little beyond encoder conditioning because encoder outputs feed the decoder through attention.
- All models perform worst on Malayalam and Kannada, plausibly because their agglutinative words are longer on average.An average Malayalam training-set word has 9 characters versus 5 in Hindi; Hindi and Malayalam have relatively close CER despite differing WER.
- The joint model is rarely confused between languages, while encoder conditioning removes those remaining cases almost completely.The confusion matrices are truncated to precision of 10^-3.
5. ANALYSIS
The analysis examines language confusion, code-switching, and mismatched language identifiers in the multilingual LAS models. The models rarely confuse languages, but the joint model cannot code-switch and the conditioned model follows the supplied language ID.
- Language confusion: The joint and encoder-conditioned models are rarely confused between languages, while conditioning on language ID eliminates confusion.The joint model’s behavior suggests it implicitly learns language identity; the conditioned model uses a single language’s character set.
- Code-switching: The joint model can theoretically code-switch because English words occur in training data, but this capability was not established for unseen Indian-language pairs.The experiment tested code-switching between Tamil and Hindi, a pair not seen together during training.
- Code-switching: On approximately 1,000 concatenated Tamil-Hindi utterances, the model failed to code-switch and instead selected one script for the utterance.When it selected Hindi or Tamil, it generally transcribed only that segment; rare transliteration of the other segment also occurred.
- Code-switching: The code-switching failure suggests that the language model dominates the acoustic model and indicates overfitting.This interpretation is presented as a known issue with attention-based sequence-to-sequence models.
- Mismatched language ID: With approximately 1,000 Urdu utterances labeled as Hindi, the conditioned model remained faithful to the Hindi language ID and used Hindi’s character set.The experiment tests whether the model follows acoustics or the supplied language identifier.
6. CONCLUSION
The paper presents multilingual sequence-to-sequence ASR that recognizes speech without explicit language specification, along with language-conditioned variants. The models outperform monolingual baselines and rarely select an incorrect grapheme set, but cannot handle code-switching.
- Conclusion: The proposed sequence-to-sequence model recognizes multilingual speech without explicit language specification.The paper also proposes simple variants conditioned on language identity.
- Conclusion: The proposed models substantially outperform baseline monolingual sequence-to-sequence models for all languages and rarely choose an incorrect grapheme set.
- Limitations: The model cannot handle code-switching, and this suggests that its language model dominates the acoustic model.The paper identifies this behavior as a direction for future investigation.