Source-linked AI summary
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath, Pedro Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Françoise Beaufays, Yonghui Wu
TL;DR
USM addresses the challenge of expanding ASR beyond mainstream languages toward a model covering all spoken languages. It combines scalable multilingual self-supervised and multimodal pre-training with labeled data, achieving state-of-the-art results across multilingual ASR and speech-to-text translation benchmarks while showing that in-domain fine-tuning remains most effective for a given domain.
Problem
The work investigates how to expand universal ASR beyond mainstream languages toward coverage of all spoken languages.
Method
USM combines scalable BEST-RQ self-supervised learning, multi-modal text-injection pre-training, multi-softmax loss, and supervised training with transcribed data and unlabeled text.
Results
USM achieves state-of-the-art performance across multilingual ASR and AST datasets, including FLEURS ASR across 102 languages and CoVoST 2 AST across 21 languages.
Takeaways & Limitations
Diverse unlabeled data can support usable ASR for tail languages, while in-domain data remains the most effective way to optimize performance for a given domain.
Takeaways & Limitations
Public data is used only for in-domain pre-training and excluded from generic USM-LAS/CTC training, so public-task results serve as out-of-domain benchmarks for those models.
Abstract
from arXiv · showhide
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.
1 Introduction
USM is a scalable multilingual speech framework designed to address limited transcribed data by combining large unlabeled multilingual speech and text corpora with smaller labeled datasets. Its models support multilingual ASR and AST, including long-form recognition and adaptation to new domains.
- Motivation: Limited transcribed speech, especially for tail languages, motivates using practically abundant untranscribed speech and text for multilingual ASR pre-training.Conventional transcription is lengthy and expensive, while semi-supervised learning can leverage unlabeled data with limited transcribed speech.
- Approach: The training pipeline combines BEST-RQ unsupervised encoder pre-training, optional MOST multi-objective pre-training, and supervised ASR training for downstream tasks.MOST jointly uses masked speech modeling, text-injection, supervised ASR, modality matching, and related objectives across speech, text, and paired data.
- Results: USM models achieve state-of-the-art performance across multilingual ASR and AST benchmarks spanning speech recognition, translation, domains, and language groups.Reported evaluations include SpeechStew, CORAAL, FLEURS, YouTube long-form ASR, and CoVoST.
- Results: The YouTube captioning model achieves < 30% WER on 73 languages and performs better than Whisper on selected languages despite using less supervised data.The comparison covers 18 languages that Whisper successfully decodes below 40% WER; Whisper was trained on more than 400k hours of transcribed data.
- Extensions: Chunk-wise attention extends models trained on short utterances to very long speech inputs, while lightweight residual adapters support rapid domain adaptation.Adapters add 2% of additional parameters while the rest of the model remains frozen.
2 Methods
The model uses a Conformer encoder whose representations feed CTC, RNN-T, or LAS units for downstream speech tasks. Experiments consider 600M- and 2B-parameter models, with the 2B model used for the main results.
- Conformer encoder features feed CTC, RNN-T, or LAS units for downstream ASR and AST tasks.BEST-RQ pre-training is applied exclusively to the encoder.
- 600M- and 2B-parameter Conformer models are considered, with the 2B model used for the main results.
- Table 2 lists the Conformer model parameters.
2.2 Pre-training: BEST-RQ
BEST-RQ pre-trains speech encoders with a BERT-style masked-feature prediction task using quantized speech targets. The paper uses multiple independent codebooks to improve training stability and convergence without the extra quantization module required by w2v-BERT.
- BEST-RQ pre-trains networks on large-scale unlabeled audio with a BERT-style task that predicts quantized labels for masked speech features.
- Figure 3 depicts BEST-RQ pre-training with a Conformer encoder.
- BEST-RQ avoids the additional quantization module required by w2v-BERT, making it more scalable as model size and language coverage increase.
- Multiple codebooks produce independent quantization-target predictions, and equal weighting of their softmax losses improves training stability and convergence.
2.3 Self-training: Noisy Student Training
Noisy Student Training augments supervised ASR training by generating filtered pseudo-labels for unlabeled audio and mixing them with supervised data.
- Noisy Student Training uses a supervised teacher to generate transcripts for unlabeled audio, filters them heuristically, and trains a student on mixed data.Filtering uses the ratio between word count and audio length.
2.4 Chunk-wise Attention for Long-form ASR
Long-form ASR is difficult because models trained on short segments can require impractically large receptive fields during inference. Chunk-wise attention restricts attention to audio chunks, making receptive-field width independent of encoder depth and supporting long utterances.
- ASR systems trained on segments shorter than 30 seconds face long-form degradation when processing minutes- or hours-long audio.
- 327 seconds is the receptive-field length described for a large local-attention encoder, despite training on at most 30-second segments.
- Figure 4 compares receptive fields for four-layer networks using local self attention and chunk-wise attention.
- Chunk-wise attention divides audio into chunks and prevents context leaking, so receptive-field width is independent of the number of layers.An 8-second chunk gave the best recognition-quality versus computational-cost trade-off in the experiments.
- Compared with local self attention, chunk-wise attention is presented as a more flexible alternative to block processing because other encoder layers can access broader context.
2.5 Multi-Objective Supervised Pre-training: BEST-RQ + text-injection
MOST combines unlabeled speech, unlabeled text, and paired speech-text data in a multi-objective pre-training pipeline. It matches speech and text representations while retaining BEST-RQ speech pre-training and supervised ASR objectives.
- Multi-objective training: MOST jointly trains on unlabeled speech, unlabeled text, and paired speech-text data using BEST-RQ and text-injection losses.The objective includes a weighted BEST-RQ loss alongside supervised ASR and modality-matching losses.
- Text-injection: Speech and text encoders produce co-aligned embeddings in a shared space, allowing unpaired text to contribute to the speech task.The representations are matched in embedding space and aligned along the time dimension.
- Paired data: Paired speech-text data supplies the standard ASR loss, trains duration upsampling from decoder alignments, and contributes a consistency loss.The consistency loss compares encoded text and speech sequences, while the text encoder is trained separately for that loss.
- Text reconstruction: Unlabeled text is encoded, masked, and passed through the shared encoder and decoder to reconstruct the original text with an ASR loss.This reconstruction loss trains the model using text features that share an embedding space with speech features.
- Training schedule: MOST first trains on paired data for 20k steps, then activates unlabeled-text losses after stable decoder alignments are learned.The two-stage schedule is introduced for training stability.
- Fine-tuning: For ASR fine-tuning, the speech feature encoder and shared Conformer are initialized from pre-trained encoders, while the task-specific transducer is randomized.MOST-prepared models can also use joint fine-tuning for speech-to-text translation.
2.6 Residual Adaptation with a Frozen Encoder
Residual adaptation offers a lightweight alternative to individually fine-tuning the full USM for multiple domains and tasks. It adds small language-specific adapters while freezing the pre-trained USM.
- Frozen-encoder adaptation: Residual adapters add a small number of language-specific parameters while the pre-trained USM remains entirely frozen during fine-tuning.This approach is explored to reduce the cost of adapting the model across downstream languages, domains, and tasks.
2.7 Training Details
Training uses standardized audio and text preprocessing, large Conformer-based BEST-RQ and MOST configurations, staged data mixing, separate optimizers, and TPU-based distributed training.
- Preprocessing: Audio is resampled to 16 kHz and represented with 128-dimensional log-mel filterbank coefficients.FLEURS uses grapheme tokenization, while other tasks use word-piece models.
- BEST-RQ: BEST-RQ uses default masking and quantization settings with a 16-codebook multi-softmax loss and no EMA.The multi-softmax loss is used to stabilize training and improve performance.
- MOST configuration: MOST uses a 1536-dimensional speech Conformer layer, a Conformer-2B shared encoder, 4k sentencepiece models, and mixed batches of three data types.The model is initialized with the BEST-RQ pre-trained encoder.
- Curriculum: Unlabeled text enters MOST only after 20k steps, followed by 100k steps of joint training with all three data types.This curriculum begins with unlabeled speech and paired speech-text data.
- Supervised training: USM-CTC and USM-LAS are trained for 100k steps with batch size 2048 using separate encoder and decoder optimizers.In-domain checkpoints are selected using development-set performance.
- Infrastructure: Large models are trained with GShard and the GSPMD backend on TPUs.
3 Datasets
The paper assembles multilingual audio, text, supervised, and evaluation resources spanning mainstream and long-tail languages. The datasets support pre-training, domain-specific evaluation, ASR, speech translation, and comparisons with existing systems.
- YouTube audio: YT-NTL-U contains 12.1M hours of segmented unlabeled audio, combining 12M hours across 55 languages with 100k hours across 513 tail languages.The tail-language component is vendor-identified and complements YouTube production language identification coverage.
- Supervised audio: YT-SUP+ provides 90k hours of labeled audio across 75 languages plus 100k hours of pseudo-labeled en-US audio.The pseudo-labels come from a 600M-parameter CTC teacher trained on YT-SUP.
- YouTube audio: YT-513-U contributes 102k hours of unlabeled speech collected from videos in 513 languages, with varied domains, voices, and content.Its distribution includes 88 languages above 500 hours, 237 between 100–500 hours, and 188 languages below that range.
- Public data: Public unsupervised speech contributes approximately 429k hours in 51 languages, but it is excluded from generic USM-LAS/CTC training.This exclusion allows public-task performance to function as an out-of-domain benchmark for those models.
- Text data: Web-NTL supplies over 28B monolingual text sentences spanning 1140 languages, with temperature-based upsampling for lower-resource languages.The sampling temperature is T = 3.0.
- Evaluation datasets: Evaluation covers SpeechStew, FLEURS, YouTube speech recognition, and CoVoST 2 translation from 21 source languages into English.FLEURS contains 10 hours of read speech in 102 languages, while the YouTube test set covers 73 languages.
4 Key Results
USM models achieve strong multilingual and cross-domain ASR and speech translation results, often surpassing Whisper and prior state of the art with less supervised data. Adapter-based pseudo-labeling further extends recognition to unseen languages.
- Robust Speech Recognition: USM-CTC outperforms Whisper by a wide margin on YouTube en-US despite using 3.5k versus 400k supervised hours.USM-CTC also avoids the long-form degradation observed for USM-LAS and achieves the best YouTube performance.
- Robust Speech Recognition: More than 10% relative WER improvement over Whisper is achieved by both USM-CTC and USM-LAS on out-of-domain CORAAL.Both models also outperform Whisper on SpeechStew, whose training data they did not access.
- Multilingual Results: 66% relative WER improvement over Whisper is achieved on FLEURS despite using 90k versus 117k multilingual supervised hours.USM-LAS consistently outperforms USM-CTC on short-form ASR tasks.
- Multilingual Results: 30% relative WER improvement over the previous state of the art is achieved across 102 languages with in-domain fine-tuning.The results indicate that in-domain fine-tuning maximizes performance for a target domain.
- Representation Adaptation: Adding only 2% to total parameters, USM-M-adapter performs only slightly worse than the full MOST representation model.The experiment freezes the MOST encoder and trains residual adapters for domain adaptation.
- Unseen Languages: Adapter teachers generate pseudo-labels for unseen-language YouTube data, yielding consistent FLEURS gains and improvements larger than 30% for some languages.The approach uses only 10 hours of out-of-domain FLEURS data to train the adapter teacher.
- Speech Translation: Fine-tuned speech-only USMs are comparable to CoVoST 2 state of the art using 859 hours of data versus 125k supervised speech-translation hours.MOST additionally enables speech and text inputs during fine-tuning.
5 Analysis and Ablations
The analyses show that larger multilingual pre-training, multi-softmax objectives, and chunk-wise attention improve performance and robustness. Chunk-wise attention specifically addresses long-form degradation caused by receptive-field mismatch while retaining practical serving capacity.
- Multi-Softmax Ablation: More than 5% relative improvement occurs across ASR and AST benchmarks when multi-softmax groups increase from 1 to 16.Multiple groups also reduce variation across pre-training runs and improve convergence speed.
- Scaling Data and Coverage: YT-NTL pre-training yields a 10% relative ASR and AST improvement over YT-55 despite only about 500 hours per newly added language.YT-NTL covers 27 additional languages according to Table 5.
- Pre-training Comparison: BEST-RQ is evaluated against W2v-BERT on CoVoST 2 X →En using BLEU scores, with higher scores preferred.The comparison tests the relative performance of two speech pre-training methods.
- Long-Form Degradation: A 48-layer model develops higher long-form WER as training progresses because its larger receptive field mismatches the local attention context.The 17-layer model with 65-length local context instead gradually improves on the long-form test set.
- Chunk-Wise Attention: Chunk-wise attention is evaluated with 8-second chunks against local self-attention using 128 context frames across en-US, ru-RU, ko-KR, and uk-UA.The comparison targets robust performance across four languages on YouTube long-form speech.
- Serving Capacity: The 2B-parameter USM-CTC model is only 3.9× slower than a 100M-parameter streaming model under fully packed batch processing.The measurement uses real-time factor in an ideal TPU serving setup.
6 Discussion
The discussion emphasizes practical scaling through unlabeled multilingual data, while confirming that in-domain fine-tuning remains best for optimizing a specific domain. Transducer choice should follow the downstream task.
- Unlabeled Data: Diverse unlabeled data is considered more practical than weakly labeled data for building usable ASR in tail languages.The authors demonstrate that native-speaker collaboration can identify unsupervised data across hundreds of tail languages.
- Unlabeled Data: Native-speaker collaboration improves recognition performance on low-resource languages by identifying unsupervised data in hundreds of tail languages.This supports expanding usable speech data beyond languages with paired transcriptions.
- In-Domain Data: In-domain fine-tuning is the most effective way to optimize performance for a given domain after broad unsupervised and limited labeled training.The broader system remains robust across many domains without requiring extensive labeled data for each one.
- Transducer Choice: The best transducer depends on the downstream task, so a frozen encoder enables rapid comparison of CTC, RNN-T, and LAS.The large pre-trained model supports testing task-specific transducers without retraining the encoder.