Source-linked AI summary
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model
Anjuli Kannan, Arindrima Datta, Tara N. Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, Seungji Lee
TL;DR
Multilingual E2E ASR must support low-latency interaction while handling uneven training data and competing with conventional systems. The paper evaluates a streaming RNN-T system using language vectors and adapter modules across nine Indian languages, finding that the combined design achieves the strongest reported comparisons. It outperforms monolingual E2E models and conventional recognizers, with at least a 10% relative WER reduction against monolingual conventional systems in each language.
Problem
E2E multilingual systems had not been conclusively shown to match state-of-the-art conventional models while satisfying real-time interactive latency constraints and handling imbalanced data.
Method
The paper evaluates a streaming RNN-T multilingual model and compares language vectors, sampling ratios, and language-specific adapter modules on nine Indian languages.
Results
The combined language-vector and adapter design outperforms monolingual E2E and conventional systems, reducing WER by at least 10% relatively versus monolingual conventional systems in all nine languages.
Takeaways & Limitations
A single streaming multilingual E2E system can address interactive latency and imbalanced-data challenges while outperforming the reported monolingual baselines.
Takeaways & Limitations
RNN-T assumes each output symbol is independent of future acoustic frames, an assumption used to enable streaming operation.
Abstract
from arXiv · showhide
Multilingual end-to-end (E2E) models have shown great promise in expansion of automatic speech recognition (ASR) coverage of the world's languages. They have shown improvement over monolingual systems, and have simplified training and serving by eliminating language-specific acoustic, pronunciation, and language models. This work presents an E2E multilingual system which is equipped to operate in low-latency interactive applications, as well as handle a key challenge of real world data: the imbalance in training data across languages. Using nine Indic languages, we compare a variety of techniques, and find that a combination of conditioning on a language vector and training language-specific adapter layers produces the best model. The resulting E2E multilingual model achieves a lower word error rate (WER) than both monolingual E2E models (eight of nine languages) and monolingual conventional systems (all nine languages).
1. Introduction
The paper develops a streaming E2E multilingual ASR system designed for interactive latency, imbalanced real-world data, and competitiveness with conventional systems. Using nine Indian languages, it combines language conditioning with adapter modules and reports improvements over monolingual baselines.
- The work targets E2E multilingual ASR that meets interactive latency constraints, handles imbalanced training data, and remains competitive with conventional models.
- The proposed streaming system uses an RNN-T architecture, unlike prior attention-based multilingual E2E models without straightforward streaming implementations.
- Experiments on 37K hours across nine Indian languages compare language vectors, sampling ratios, and language-specific adapter modules for data imbalance.
- Combining a language vector with adapter modules yields the best multilingual E2E system among the investigated techniques.
- The resulting system surpasses monolingual E2E and conventional systems, achieving at least a 10% relative WER reduction against monolingual conventional systems in every language.
2. Streaming E2E multilingual model
The streaming model uses RNN-T to process acoustic features and previously emitted symbols while supporting interactive latency. Its prediction structure avoids future-frame dependence, enabling streaming operation.
- RNN-T comprises encoder, prediction, and joint networks that together produce the next output-symbol distribution.
- The encoder is an LSTM-based recurrent network that transforms a sequence of d-dimensional acoustic feature vectors into higher-order representations.
- The prediction network processes previously emitted non-blank symbols and produces a dense representation analogous to a language-model representation.
- The joint network combines encoder and prediction representations to predict the distribution over the next output symbol.
- RNN-T conditions each prediction on acoustics and prior labels but assumes independence from future acoustic frames, enabling streaming operation.
3. Imbalanced multilingual data
The paper addresses multilingual training-data imbalance through sampling and architectural strategies, including language vectors and language-specific adapters. Adapters preserve a shared model while enabling per-language specialization with limited additional parameters.
- Motivation: Data imbalance makes end-to-end multilingual models more sensitive to skew because every model component is trained on multilingual transcribed speech.Languages with more speakers tend to contribute more transcribed data and influence training more strongly.
- Data sampling: Uniform sampling and tunable interpolation between natural and uniform frequencies are investigated to increase representation of under-represented languages.The interpolation uses α, with α = 0 for natural frequencies and α = 1 for uniform sampling.
- Conditioning on language vector: A language vector lets one universal model become an expert on each language, with the language specified by user preferences or inferred by language identification.The study extends language-vector conditioning to a streaming RNN-T and evaluates its role in addressing data imbalance.
- Adapter modules: Language-specific adapter modules adjust activations after encoder layers while retaining a shared global model.Adapters use a two-stage process: train a global RNN-T, then freeze it and add separate per-language adapter parameters.
- Adapter modules: Adapters typically add less than 10% of the original model size, supporting parameter sharing alongside per-language specialization.At inference, the adapter corresponding to the input language is applied after each encoder layer.
4. Experimental details
Experiments use anonymized, human-transcribed utterances from nine Indian languages with language-specific data availability and mixed writing systems. Evaluation uses transliteration-optimized WER, alongside specified RNN-T and conventional baselines.
- Data: Training and test data comprise anonymized, human-transcribed utterances representative of Google traffic across nine Indian languages.Training data are augmented by corrupting clean utterances with a room simulator.
- Data: Transcriptions use different scripts across languages, except Hindi and Marathi, which both use Devanagari, and may mix Latin-alphabet text.Loanwords and proper nouns are among the reasons Latin text appears in transcripts.
- Evaluation: The reported metric is transliteration-optimized WER, which does not count Latin-versus-native-script differences as errors when the word is otherwise correct.This accommodates equivalent renderings across the mixed-script transcriptions.
- Model configuration: RNN-T models use 80-dimensional log-mel features with 25ms windows, 10ms shifts, stacking, and downsampling to a 30ms frame rate.The encoder has eight 2,048-dimensional LSTM layers with 640-dimensional projections, and the unified grapheme softmax contains 988 graphemes.
- Baselines: Monolingual RNN-T baselines use the multilingual architecture with L2 regularization, while conventional baselines use five 768-dimensional LSTM acoustic-model layers and context-dependent phone-state softmaxes.The conventional systems provide a separate baseline family for comparison with the end-to-end models.
5. Results
The results show that language conditioning and language-specific adapters address multilingual data imbalance more effectively than upsampling alone. The best multilingual RNN-T model outperforms monolingual E2E and conventional systems while retaining compact, streaming-oriented deployment.
- Conditioning on language vector: More than 50% relative WER reductions occur for Urdu, Kannada, and Malayalam when the RNN-T encoder receives a language vector.The comparison is between models A0 and A1.
- Conditioning on language vector: Bengali WER is nearly 50% lower with language conditioning because the vector helps select the correct script amid overlapping vocabulary.Model A0 often defaults to Hindi, the dominant language in the training data; model A1 uses the language vector to avoid this error.
- Data sampling: Upsampling smaller languages lowers WER for all but Hindi and Marathi, but causes 10-20% relative regression on those two largest languages.Upsampling changes the model’s prior for ambiguous utterances, making dominant-language defaults less likely.
- Data sampling: With a language vector, upsampling provides no additional WER benefit and causes small languages to overfit while larger languages continue improving.At intermediate training, large languages underfit; after full training, small languages heavily overfit.
- Adapters: Adapters add WER reductions on all languages, with relative gains of 9% for Kannada, 8% for Urdu, and 6% for Bengali.Each language-specific adapter adds 2.5M parameters, about 2% of the original 120M-parameter model.
- Comparison with Baseline Models: The best multilingual RNN-T has lower WER than monolingual RNN-T models on eight of nine languages and than conventional models on all nine.Relative gains over conventional systems are about 10% overall, reaching 34% on Kannada and 25% on Urdu.
- Comparison with Baseline Models: A single compact multilingual recognizer replaces nine separate recognizers containing acoustic, pronunciation, and language models.The system is presented as suitable for streaming applications while outperforming monolingual conventional models.
6. Conclusions
The paper extends multilingual E2E ASR to streaming operation and imbalanced training data. Using nine Indian languages, its best RNN-T with adapter modules outperforms monolingual RNN-T and state-of-the-art conventional recognizers.
- Conclusions: The work addresses streaming ASR and imbalanced training data in large-scale practical multilingual applications.It presents a system and compares techniques for handling data imbalance.
- Conclusions: The best system combines an RNN-T model with adapter modules and significantly outperforms both monolingual RNN-T models and state-of-the-art monolingual conventional recognizers.The evaluation uses nine Indian languages.