Source-linked AI summary
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, Ronan Collobert
TL;DR
The paper asks whether one acoustic model can improve ASR for low-resource languages while simplifying support for many languages. It benchmarks joint, language-conditioned, and multi-head multilingual models across 51 languages and more than 16,000 hours of audio, finding relative WER reductions versus monolingual baselines, especially for low-resource languages. The study concludes that a single massive architecture is feasible and less time-consuming to tune than 51 separate monolingual baselines.
Problem
Can one ASR model support many languages while improving recognition for low-resource languages and simplifying deployment compared with separate language models?
Method
The study trains and compares joint, language-embedded joint, and multi-head multilingual ASR models across 51 languages using shared multilingual training data.
Results
28.76% average WER improvement is achieved for low-resource languages by the largest multi-headed model versus the monolingual baseline.
Takeaways & Limitations
Multilingual representations can improve recognition for new languages not seen during initial training, supporting multilingual transfer through fine-tuning.
Abstract
from arXiv · showhide
We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all simplifying deployment of ASR systems that support diverse languages. We perform an extensive benchmark on 51 languages, with varying amount of training data by language(from 100 hours to 1100 hours). We compare three variants of multilingual training from a single joint model without knowing the input language, to using this information, to multiple heads (one per language cluster). We show that multilingual training of ASR models on several languages can improve recognition performance, in particular, on low resource languages. We see 20.9%, 23% and 28.8% average WER relative reduction compared to monolingual baselines on joint model, joint model with language input and multi head model respectively. To our knowledge, this is the first work studying multilingual ASR at massive scale, with more than 50 languages and more than 16,000 hours of audio across them.
1. Introduction
The paper studies massive multilingual ASR to improve recognition for low-resource languages while simplifying deployment. It trains models across 51 languages and develops joint and multi-head approaches that outperform monolingual baselines.
- A single model can simplify training, deployment, and maintenance compared with separate models for many languages.
- 51 languages are used to study multilingual ASR at massive scale across several language families.
- Joint multilingual training with a shared vocabulary can surpass strong monolingual baselines on low-resource languages.
- The refined multi-head approach improves on the monolithic joint model and remains competitive with monolingual baselines on higher-resource languages.
- Multilingual representations improve baseline WER on languages not seen during initial training.
2. Related Work
Prior multilingual ASR studies explored cross-lingual transfer and shared representations but generally used fewer languages or limited datasets. This work extends the evaluation to 51 languages and more than 16,000 hours of audio.
- Multilingual ASR has been studied for decades in HMM-GMM, DNN-HMM, and end-to-end systems.
- Cross-lingual sequence-to-sequence systems pre-train on multiple languages and fine-tune on a target language, while related work has used up to 100 languages.
- Prior work on Indian languages used shared encoders and decoders with language identification or adaptive per-language layers.
- Other work predicts Unicode bytes directly to obtain multilingual gains and avoid large softmax layers.
- 51 languages and more than 16,000 hours of audio make this, to the authors’ knowledge, the first massive-scale multilingual ASR study.
3. Multilingual models
The paper builds multilingual sequence-to-sequence ASR models with shared encoders, decoders, and token sets, then contrasts a joint model with language-conditioned and multi-headed designs. Sampling and curriculum strategies address imbalance and convergence across languages.
- Seq2Seq model: The encoder maps audio to hidden representations, while an autoregressive decoder maps those representations to output transcriptions.
- Multilingual training: Multilingual training combines independent datasets from N languages, whose grapheme sets may overlap.
- Shared sub-word tokens: A shared SentencePiece token set simplifies training and maintenance across languages with distinct character sets and tokenization rules.
- Joint model: The joint model shares encoder, decoder, and token-set parameters across languages and can receive language information through a jointly trained embedding.
- Joint model: Curriculum training incrementally adds languages and enables convergence for models trained on up to 51 languages.
- Multi-headed model: Multi-headed models share one encoder but use separate decoders for language groups because unrelated languages may not benefit from shared decoder parameters.
- Multi-headed model: The language groups are manually combined into six groups to avoid tuning decoder capacity separately for highly uneven group sizes.
4. Experimental details
Experiments use anonymized public videos spanning 51 languages with high-, mid-, and low-resource categories, standardized text processing, and a common ASR training framework. Language sampling is tuned to balance low-resource gains against degradation for higher-resource languages.
- Dataset: The dataset spans 51 languages, with high-resource languages above 600 hours, mid-resource languages at 300–500 hours, and low-resource languages at 100–150 hours.Each language has about 20 hours of test data, with development data used for hyperparameter tuning.
- Data preparation: Text is normalized with NFKC, punctuation removal, and language-specific Unicode filtering before token and lexicon generation.
- Model and training: All experiments use wav2letter++, 80-dimensional log-mel filter banks, SpecAugment, and BMUF for scalable training.
- Model and training: Baseline models use TDS convolutional encoders and a two-layer GRU decoder, totaling about 150 million parameters.
- Model and training: High- and mid-resource languages use SentencePiece sub-word tokens, whereas low-resource languages use graphemes because they perform better.
- Training data sampling: Sampling uses a multinomial distribution over languages, with β controlling the transition from natural-frequency sampling at β = 1 to uniform sampling at β = 0.
5. Results and analysis
The experiments compare multilingual ASR model variants, sampling strategies, model sizes, and transfer to unseen languages. Performance generally improves with balanced sampling, larger models, language embeddings, multi-head designs, and multilingual fine-tuning.
- Sampling strategy: Balanced sampling with α = 0.5 and β = 0.5 performs best on low-resource languages while limiting degradation on higher-resource languages.Natural-frequency sampling suffers from data imbalance, whereas uniform sampling can overfit low-resource languages.
- Joint models: Increasing model size improves WER for both joint models and joint models with language embeddings.The comparison uses relative WER change across model sizes, with training data decreasing along each plot’s x-axis.
- Joint models: 20.87% average WER improvement occurs for low-resource languages with the 1 billion parameter joint model, while high-resource languages degrade by 3.15%.Mid-resource languages improve by 2.5% in the same setting.
- Joint models: 23.03% average WER improvement occurs for low-resource languages with language embeddings, alongside improvements of 7.48% and 12.11% on high- and mid-resource languages.The language-embedded 1 billion parameter joint model improves WER on all languages.
- Multi-head models: 28.76% average WER improvement occurs for low-resource languages with the largest multi-headed model, which also improves high- and mid-resource languages by 9.1% and 12.44%.The multi-headed model outperforms joint models even when those models receive language embeddings and does not require curriculum training.
- Transfer learning: Fine-tuning the 1 billion parameter multilingual joint model improves WER over monolingual baselines on three unseen low-resource languages.The languages have 100–150 hours of training data, and both encoder and decoder are trained during fine-tuning.
- Language embedding analysis: Learned language embeddings form noticeable clusters corresponding to language families.The paper notes that these learned clusters could support future multi-head experiments instead of manual cluster selection.
6. Conclusion
The study demonstrates that one massive ASR architecture can serve 51 languages. In practice, tuning this single multilingual system was considerably less time-consuming than tuning 51 separate monolingual baselines.
- Conclusion: A single ASR architecture can be trained for 51 languages and tuned considerably faster than 51 separate monolingual baselines.The conclusion frames this as a practical result of the massive multilingual approach.