Speech Recognition with Augmented Synthesized Speech

Andrew Rosenberg; Yu Zhang; Bhuvana Ramabhadran; Ye Jia; Pedro Moreno; Yonghui Wu; Zelin Wu

Source-linked AI summary

Speech Recognition with Augmented Synthesized Speech

Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro Moreno, Yonghui Wu, Zelin Wu

arXiv:1909.11699v1Sep 25, 2019cs.CLcs.SDeess.AS

TL;DR

The paper asks whether synthesized speech can replace or supplement expensive transcribed speech for ASR training. It varies speaker and lexical diversity across two domains and finds that augmentation improves ASR, while synthetic speech remains substantially less valuable than real speech.

Problem
The paper examines whether speech synthesis can provide the additional diverse training data needed to improve ASR and support domain transfer.
Method
The study augments human ASR training data with multi-speaker TTS, varying acoustic speaker representations and generating lexically diverse utterances across two corpora.
Results
Synthesized speech improves ASR through expanded acoustic and lexical diversity, including when a LIBRISPEECH-trained synthesizer generates material for a target domain.
Takeaways & Limitations
Speech-synthesis augmentation can provide ASR gains and support cross-domain training, but it does not yet match the value of real speech.
Takeaways & Limitations
The value of synthesized speech as ASR training data remains dramatically less than that of real speech.

Abstract

from arXiv · show

Recent success of the Tacotron speech synthesis architecture and its variants in producing natural sounding multi-speaker synthesized speech has raised the exciting possibility of replacing expensive, manually transcribed, domain-specific, human speech that is used to train speech recognizers. The multi-speaker speech synthesis architecture can learn latent embedding spaces of prosody, speaker and style variations derived from input acoustic representations thereby allowing for manipulation of the synthesized speech. In this paper, we evaluate the feasibility of enhancing speech recognition performance using speech synthesis using two corpora from different domains. We explore algorithms to provide the necessary acoustic and lexical diversity needed for robust speech recognition. Finally, we demonstrate the feasibility of this approach as a data augmentation strategy for domain-transfer. We find that improvements to speech recognition performance is achievable by augmenting training data with synthesized material. However, there remains a substantial gap in performance between recognizers trained on human speech those trained on synthesized speech.

1. INTRODUCTION

The paper evaluates synthesized speech as ASR training-data augmentation, targeting acoustic and lexical diversity across domains. It reports gains from synthesized material while finding that synthetic speech remains less valuable than real speech.

1. INTRODUCTION: The paper tests whether combining human training speech with synthesized material can provide an alternative to additional manually transcribed data.This hypothesis is motivated by improvements in natural-sounding multi-speaker speech synthesis.
1. INTRODUCTION: The experiments use LIBRISPEECH and ISOLATED-SENTENCES, which differ substantially in corpus size, utterance type, and recording conditions.LIBRISPEECH contains 960 hours, while ISOLATED-SENTENCES contains 76 hours across 201k utterances and 1,988 speakers.
1. INTRODUCTION: The study examines whether a synthesizer trained in one domain can improve recognition in another domain.This tests whether domain-transfer augmentation can use a general-purpose synthesizer or requires domain-specific synthesis.
1. INTRODUCTION: Synthesized speech augmentation improves ASR by expanding both acoustic and lexical diversity in training data.The study investigates speaker variation, newly generated utterances, and domain transfer across two corpora.
1. INTRODUCTION: The contributions include studying diverse speaker representations, lexically diverse synthesized utterances, and a hierarchical VAE for limited multi-speaker data.The hierarchical VAE is described as aiding TTS training with a small amount of multi-speaker training data.

2. RELATED WORK

Prior work established speech-synthesis augmentation and related joint ASR-TTS approaches. This paper compares its acoustic-diversity results with earlier work and reports stronger performance on the more challenging test-other partition.

2. RELATED WORK: Li et al. expanded prosodic variation with Global Style Tokens, while this work uses a distinct approach to acoustic diversity.The paper also confirms their reported benefit from increasing recognition-network depth.
2. RELATED WORK: Figure 1 depicts local and global VAE encoders whose summed style vector is broadcast to every decoder RNN step.This architecture is part of the hierarchical style representation used for synthesis.
2. RELATED WORK: The paper reports better performance on LIBRISPEECH test-other, whereas Li et al. report better results on test-clean.Both studies obtain gains through speech-synthesis-based augmentation, but their strongest partition differs.
2. RELATED WORK: Related work also includes speech-chain systems that jointly train recognition and synthesis models in a closed-loop architecture.DeepChain is presented as an example of simultaneous ASR and TTS training.

3. SPEECH SYNTHESIS MODEL

The TTS system builds on Tacotron 2 with speaker conditioning and WaveRNN vocoding, then adds GMM attention and a hierarchical VAE for challenging non-studio data.

3. SPEECH SYNTHESIS MODEL: Tacotron 2 maps text sequences to mel-spectrogram frames using convolutional encoding, bidirectional LSTM processing, and location-sensitive attention.The decoder accesses encoded text through attention that incorporates attention history.
3. SPEECH SYNTHESIS MODEL: A separately trained 256-dimensional speaker d-vector conditions the autoregressive decoder, while WaveRNN converts mel spectrograms into waveforms.The vocoder is trained separately from the Tacotron-based synthesizer.
3. SPEECH SYNTHESIS MODEL: For non-studio ISOLATED-SENTENCES data, the model uses GMM attention and a variational autoencoder to improve synthesis fidelity.These changes address the difficulty of training TTS on noisy and highly biased material.
3. SPEECH SYNTHESIS MODEL: The hierarchical VAE adds local and global encoders to represent style information at different temporal scales.The local encoder processes fixed two-second chunks with one-second overlap, while the global encoder captures global style.

4. SPEECH RECOGNITION MODEL

ASR uses a fixed LAS encoder-decoder configuration across experiments, with 80-mel acoustic features and graphemic word-piece targets. Synthesized utterances are filtered for severe fidelity failures in LIBRISPEECH.

4. SPEECH RECOGNITION MODEL: The ASR system is a listen-attend-spell encoder-decoder with additive attention, using 80-mel features plus deltas and double deltas.The encoder has convolutional layers followed by four bidirectional LSTM layers, and the decoder has two unidirectional LSTM layers.
4. SPEECH RECOGNITION MODEL: The recognizer predicts targets from a 16k graphemic word-piece vocabulary learned from LIBRISPEECH.The same target inventory is used for both LIBRISPEECH and ISOLATED-SENTENCES experiments.
4. SPEECH RECOGNITION MODEL: The architecture and training hyperparameters were tuned without augmentation and then held fixed across all experiments.The authors note that these settings may not be optimal in every augmented condition.
4. SPEECH RECOGNITION MODEL: LIBRISPEECH synthesized utterances with recognition WER above 20% are removed to exclude premature stops and babbling-like synthesis failures.This filtering removes 10–20% of utterances depending on speaker representation; ISOLATED-SENTENCES receives no such filtering.

5. ACOUSTIC DIVERSITY

Augmenting human training utterances with multi-speaker TTS can improve ASR, but gains depend on speaker-conditioning diversity and the amount of human source speech.

5.2. ASR augmentation: The augmentation experiments duplicate human training text with synthesized copies and vary speaker d-vectors to control acoustic diversity.The same end-to-end recognizer is evaluated across augmentation conditions and reduced-source settings.
5.2. ASR augmentation: Approximately 4% relative WER gains occur on both test-clean and test-other when speaker representations are sampled from observed training speakers.Random d-vectors are less effective, while original d-vectors provide no reliable gains and can degrade performance.
5.2. ASR augmentation: Multi-speaker TTS augmentation outperforms higher-quality single-speaker TTS, which yields WERs of 13.55 and 24.84 on test-clean and test-other.The comparison indicates that synthesis quality alone is insufficient for effective ASR augmentation.
5.3. Reduced Source Material: With 460 hours of human speech, augmentation preserves clean-set gains but reduces performance on noisy test data; with 100 hours, improvement is larger, while 10 hours fails to converge.These experiments add 960 hours of synthesized material without changing the model for different source-data amounts.

6. LEXICAL DIVERSITY

Speech synthesis is used to expand lexical diversity by generating unseen utterances, and increasing this material improves ASR up to a point.

6. LEXICAL DIVERSITY: Unseen utterance synthesis can expand both the lexical and acoustic diversity of ASR training material.The approach is intended to supplement resynthesis-based speaker variability with novel linguistic content.
6.1. Topline experiments: A topline experiment augments training data with synthesized versions of test utterances, conditioning speakers on sampled or test-audio-derived d-vectors.The sampled condition avoids test audio, whereas the original condition uses extracted test-speaker representations.
6.2. Language Model Sampling: Lexical diversity is expanded by generating new utterances with a MaxEnt language model, synthesizing them with sampled speaker representations, and filtering high-WER outputs.Generated sequences are limited to fewer than 20 words and perplexity under 500, then utterances with WER above 20% are removed.
6.2. Language Model Sampling: A 5% relative error reduction results from adding up to 600k lexically diverse synthesized utterances, approximately matching the source training-set size.Adding 1.1M utterances provides no clear further improvement, with worse test-clean but better test-other WER.

7. DOMAIN ADAPTATION

Domain-transfer experiments show that TTS augmentation can improve recognition in ISOLATED-SENTENCES, with the strongest gains from in-domain synthesis and lexical diversity when training uses only target-domain data.

7.1. Isolated-Sentences data augmentation: TTS augmentation improves performance beyond combining ISOLATED-SENTENCES with LIBRISPEECH, although the synthesizer provides similar results when both corpora are used.When training only on ISOLATED-SENTENCES, LIBRISPEECH TTS degrades performance while ISOLATED-SENTENCES TTS does not show a difference through augmentation.
7.2. Isolated-Sentences lexical diversity: Lexical-diversity augmentation produces significant gains when training uses ISOLATED-SENTENCES alone, but not when LIBRISPEECH is included.The test material is 5.5% of ISOLATED-SENTENCES training data but only 0.36% of the combined training data.
7. DOMAIN ADAPTATION: 29.51 WER is achieved by combining out-of-domain human speech with in-domain TTS, improving on 30.74 with in-domain TTS alone and 30.94 with out-of-domain human speech alone.The baseline before these additions is 32.9 WER.
7.2. Isolated-Sentences lexical diversity: Lexically diverse synthesized material from either synthesizer helps performance, while an in-domain synthesizer is more helpful than an out-of-domain synthesizer.The experiments constrain generated utterances to fewer than 5 words, perplexity under 200, and up to twice the source-training amount.

8. CONCLUSIONS

The paper shows that speech-synthesis augmentation can improve ASR through acoustic and lexical diversity, including domain-transfer use of a synthesizer trained on another domain. However, synthesized speech remains substantially less valuable as ASR training data than real speech.

8. CONCLUSIONS: Speech-synthesis augmentation improves ASR by expanding acoustic diversity through speaker characteristics and lexical diversity through new utterances generated from a language model.Both approaches are reported to improve ASR performance.
8. CONCLUSIONS: A multi-speaker TTS model trained on LIBRISPEECH can improve performance on the distinct ISOLATED-SENTENCES target domain.This demonstrates the reported domain-transfer use of speech-synthesis augmentation.
8. CONCLUSIONS: Synthesized speech remains dramatically less valuable as ASR training data than real speech despite improvements in several directions.This is the paper's stated limitation on the value of synthesized training data.