Source-linked AI summary
Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
Jason Li, Ravi Gadde, Boris Ginsburg, Vitaly Lavrukhin
TL;DR
Large public speech datasets remain limited for training accurate ASR systems. The paper augments LibriSpeech with synthetic speech and trains large end-to-end models, achieving state-of-the-art character-level WER without an external language model.
Problem
The lack of large, diverse, open speech datasets limits training large neural ASR models.
Method
The authors generate synthetic speech with a Tacotron-2-like model and combine it with LibriSpeech to train deep end-to-end Wave2Letter+ models.
Results
The 54-layer model achieves 4.32% WER on test-clean and 14.08% on test-other with greedy decoding, reported as state of the art for character-level greedy decoding.
Takeaways & Limitations
Synthetic speech is an effective way to build large neural speech recognition systems when combined with natural data at an appropriate ratio.
Takeaways & Limitations
The synthetic dataset has less speaker variety than LibriSpeech because it currently uses a three-speaker speech synthesis model.
Abstract
from arXiv · showhide
Building an accurate automatic speech recognition (ASR) system requires a large dataset that contains many hours of labeled speech samples produced by a diverse set of speakers. The lack of such open free datasets is one of the main issues preventing advancements in ASR research. To address this problem, we propose to augment a natural speech dataset with synthetic speech. We train very large end-to-end neural speech recognition models using the LibriSpeech dataset augmented with synthetic speech. These new models achieve state of the art Word Error Rate (WER) for character-level based models without an external language model.
1 Introduction
The paper addresses the shortage of large public speech datasets by augmenting LibriSpeech with synthetic speech, enabling larger neural ASR models. This approach improves ASR performance and reaches state-of-the-art WER with greedy decoding.
- 1 Introduction: The work builds on end-to-end neural ASR, which replaces engineered acoustic-modeling and HMM pipelines with direct spectrogram-to-text prediction.
- 1 Introduction: Synthetic speech augmentation addresses the lack of large public speech datasets needed to train large neural ASR models.The authors pursue increasing model size rather than complexity, using synthetic data to supply additional training speech.
- 1 Introduction: The augmented models improve ASR performance over models trained only on LibriSpeech.
- 1 Introduction: The approach achieves state-of-the-art WER with a greedy decoder by increasing model depth.
2 Synthetic Speech Dataset
The authors generate a synthetic LibriSpeech-scale dataset with a Tacotron-2-like model augmented by Global Style Tokens. Varying decoder dropout creates additional speech, producing a synthetic dataset three times the original training-set size.
- 2 Synthetic Speech Dataset: A Tacotron-2-like model with Global Style Tokens learns three speakers’ styles and accents from approximately 100 hours of MAILABS English-US speech.
- 2 Synthetic Speech Dataset: Synthetic LibriSpeech audio is created by pairing LibriSpeech transcripts with MAILABS style spectrograms.The resulting dataset matches the original LibriSpeech training portion in size and uses MAILABS speaker tones.
- 2 Synthetic Speech Dataset: Using 46%, 48%, and 50% decoder-prenet dropout produces speech with varied speaking rates and expands the synthetic dataset to three times the LibriSpeech training-set size.
3 Training Speech Recognition with Synthetic Data
The study trains deep Wave2Letter+ character-level ASR models on mixtures of natural and synthetic speech and evaluates greedy and language-model-assisted decoding. Synthetic augmentation improves WER, with the strongest results from deep models and a balanced natural–synthetic mixture.
- 3.1 Neural Speech Recognition Models: The Wave2Letter+ recognizer maps logarithmic mel-scale spectrograms directly to characters using an end-to-end deep convolutional network.
- 3.2 Word Error Rate Improvement using Synthetic Augmentation: The 54-layer model reaches 4.32% WER on test-clean and 14.08% on test-other using greedy character decoding without language-model rescoring.These results are compared against prior best no-language-model WERs of 4.87% and 15.39%.
- 3.2 Word Error Rate Improvement using Synthetic Augmentation: Synthetic augmentation improves test-clean WER by 0.15 and 0.44 for 24- and 34-layer models, and test-other WER by 0.08 and 0.74.
- 3.2 Word Error Rate Improvement using Synthetic Augmentation: Beam search with width 128 and a 4-gram OpenSLR language model reduces 54-layer test-other WER to 12.21%.The result is reported as better than previous public 4-gram language models and comparable to LSTM language models.
- 3.3 How To Mix Natural and Synthetic Speech: Natural and synthetic speech are most effective when sampled at a 50/50 rate, because synthetic speech does not capture LibriSpeech’s broader speaker variety.
- 3.4 Traditional Speech Augmentation vs Synthetic Speech: Synthetic data significantly outperforms dropout and conventional speech augmentation as a regularization method.Slightly larger dropout yields only minor WER improvement, while speech augmentation effects are negligible or can worsen WER.
4 Conclusion and Future Plans
Synthetic speech enables large neural ASR systems, but it must be combined with natural data in the correct ratio. The approach reaches state-of-the-art character-level greedy-decoding WER and improves further with language-model beam search.
- 4.32% WER on test-clean and 14.08% on test-other establish state-of-the-art character-level greedy decoding.With a language model and beam search width 128, test-other WER reaches 12.21%.
- Synthetic and natural speech must be combined in the correct ratio for best results.
- The current synthetic dataset uses only LibriSpeech training transcripts, while future work would add noise and phrases from other text sources.