Source-linked AI summary
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, Mohammad Norouzi
TL;DR
Speech recognition still struggles on noisy, low-resource data despite strong end-to-end results elsewhere. SpeechStew simply mixes publicly available speech datasets to train a large model and fine-tunes it on new tasks, achieving SoTA or near-SoTA results across varied benchmarks and near-SoTA performance on CHiME-6.
Problem
End-to-end speech recognition models struggle on noisy, low-resource datasets such as CHiME-6, where overfitting is significant.
Method
SpeechStew combines publicly available speech recognition datasets without domain-dependent re-balancing or re-weighting, trains one large neural network, and fine-tunes it on new labelled tasks.
Results
SpeechStew achieves SoTA or near-SoTA results across varied tasks without an external language model, including 4.7 WER on Switchboard, 8.3 WER on CallHome, and 1.3 WER on WSJ.
Takeaways & Limitations
A general-purpose SpeechStew model can be fine-tuned for strong results on new low-resource speech tasks, making transfer learning a practical use of broadly available data.
Abstract
from arXiv · showhide
We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.
1. Introduction
SpeechStew addresses the difficulty of end-to-end speech recognition on noisy, low-resource data by combining multi-domain training with a large model and transfer learning. The approach mixes speech datasets without domain-specific balancing and achieves strong results across tasks, including near-SoTA performance after fine-tuning on CHiME-6.
- Motivation: End-to-end models perform strongly on high-resource tasks but struggle on noisy, low-resource CHiME-6, where overfitting is significant.The best prior end-to-end model achieved 49.0% WER on CHiME-6 dev, compared with 36.9% for the best HMM model.
- Approach: SpeechStew combines multi-domain training and transfer learning in a simple end-to-end speech recognition approach.The method is designed to improve generalization by leveraging data from multiple domains and adapting to new tasks.
- Approach: SpeechStew mixes all available speech recognition data and trains one large neural network without domain-dependent re-balancing or re-weighting.The model configurations contain either 100M or 1B parameters.
- Results: The method uses no domain labels, adds no data-combination hyperparameters, and does not use an external language model during inference.It achieves SoTA or near-SoTA results across AMI, Common Voice, LibriSpeech, Switchboard, Tedlium, and WSJ.
- Transfer Learning: SpeechStew supports transfer learning by fine-tuning the pretrained model on a new labelled dataset.The authors report near-SoTA results on CHiME-6 and compatibility with unsupervised wav2vec pre-training.
2. SpeechStew
SpeechStew is trained by combining diverse speech datasets and scaling a Conformer RNN-T model, then adapted to unseen tasks through fine-tuning. Its transfer-learning setup targets noisy, low-resource CHiME-6, where direct end-to-end training is difficult because of overfitting.
- Model: The implementation uses a Conformer RNN-T architecture with 100M- and 1B-parameter configurations and no external language model.Wav2vec pre-training is needed to train the 1B-parameter model.
- Training Data: SpeechStew combines AMI, Common Voice, Broadcast News, LibriSpeech, Switchboard/Fisher, TED-LIUM, and WSJ without re-weighting or resampling.The component datasets range from approximately 50 to 2000 hours.
- Transfer Learning: CHiME-6 contains approximately 40 hours of distant-microphone conversational speech in everyday home environments.The dataset is difficult for direct end-to-end training because of overfitting issues.
- Transfer Learning: SpeechStew is fine-tuned on CHiME-6 to demonstrate transfer learning to a new low-resource task.The authors describe this as adapting a general-purpose model trained on the combined datasets.
- Transfer Learning: Fine-tuning typically requires only a few thousand steps, compared with approximately 100k steps to train a model from scratch.The authors characterize this transfer-learning workflow as inexpensive and practical.
3. Experiments
Experiments evaluate SpeechStew across multiple speech-recognition tasks and test transfer to noisy, low-resource CHiME-6. The model achieves strong results without an external language model, while larger-model benefits vary by domain.
- Evaluation setup: SpeechStew is evaluated on AMI, Common Voice, LibriSpeech, Switchboard, TED-LIUM, WSJ, and CHiME-6.The evaluation includes multiple AMI, LibriSpeech, and Switchboard subsets.
- Main results: SpeechStew achieves SoTA or near-SoTA results across many tasks without an external language model.Table 1 reports word error rates across multiple benchmarks under this evaluation condition.
- Main results: 9.0 WER on AMI-IHM, 21.7 WER on AMI-SDM1, 12.1 WER on Common Voice, and 2.0 and 4.0 WER on LibriSpeech clean and other are reported.Common Voice reaches 9.7 WER with punctuation normalization.
- Main results: 4.7 WER on Switchboard, 8.3 WER on CallHome, 5.3 WER on TED-LIUM v3, and 1.3 WER on WSJ are reported.TED-LIUM is slightly behind a reported 5.2 WER result using a language model.
- Model scaling: SpeechStew 1B with LibriLight wav2vec pre-training improves Common Voice and LibriSpeech over the 100M model, while other tasks show small relative degradation.The authors suggest in-domain LibriSpeech pre-training may bias performance toward that domain.
- Transfer learning and CHiME-6: Fine-tuning SpeechStew on CHiME-6 yields 31.9 WER on the dev set and 38.9 WER on the eval set without an external language model.The eval result compares with 38.6 WER from a strong HMM model using a strong language model.
4. Related Work
Related work combines multi-dataset training, model scaling, and transfer learning in speech and other fields. SpeechStew follows these directions while emphasizing larger models and supervised pretraining followed by fine-tuning.
- Multi-dataset training: Prior speech-recognition systems have mixed multiple datasets, including proprietary corpora exceeding 160k hours and several end-to-end datasets.Kaldi’s multien recipe is described as having a setup very similar to SpeechStew.
- Model scaling: SpeechStew differs from cited prior multi-dataset work by scaling to substantially larger models.The paper uses 100M- and 1B-parameter models.
- Transfer learning: SpeechStew adopts pretrain-and-finetune transfer learning using large labeled-corpus pretraining, while also leveraging unsupervised wav2vec pretraining in its 1B-parameter models.The paper distinguishes its supervised pretraining and supervised fine-tuning focus from prior unsupervised-audio work.
5. Discussion
The discussion frames SpeechStew around two scaling principles: more data and larger models. It argues that mixing available speech data and fine-tuning a pretrained model can make speech recognition training more practical.
- Scaling principles: Deep learning progress is associated with training on more data and training larger, deeper neural networks.These principles are presented as the discussion’s framing for SpeechStew.
- Data use: SpeechStew addresses the expense of supervised data acquisition by mixing publicly available speech-recognition data and leveraging labeled and unlabeled resources.The authors contrast this with training only on task-specific datasets.
- Transfer learning: Because training large models is expensive and impractical to repeat for new tasks, SpeechStew demonstrates fine-tuning a pretrained model for only a few thousand gradient steps.The authors describe this transfer-learning approach as inexpensive and practical.