Source-linked AI summary
A spelling correction model for end-to-end speech recognition
Jinxi Guo, Tara N. Sainath, Ron J. Weiss
TL;DR
End-to-end ASR has limited text exposure and therefore struggles particularly with rare words, while standard external-LM methods do not model the recognizer’s characteristic errors. The paper trains a spelling correction model from text-only-derived synthetic error pairs and achieves 18.6% relative WER improvement by correcting top hypotheses and 29.0% when rescoring expanded n-best lists with an external LM.
Problem
End-to-end ASR language-model components train only on parallel audio-text data, limiting performance on rare words, while existing text-only LM methods do not explicitly account for the recognizer’s characteristic errors.
Method
The paper synthesizes speech from text-only data, decodes it with LAS to create error hypotheses paired with ground truth, and trains a spelling correction model on these pairs.
Results
18.6% relative WER improvement comes from directly correcting the top ASR hypothesis, rising to 29.0% when an external LM rescored an expanded n-best list.
Takeaways & Limitations
The spelling corrector outperforms simple LM rescoring and direct LAS training on synthetic speech when used with expanded n-best rescoring.
Takeaways & Limitations
Synthetic TTS errors differ substantially from real-audio errors, and the spelling corrector performs better on TTS test data than on real-audio test data.
Abstract
from arXiv · showhide
Attention-based sequence-to-sequence models for speech recognition jointly train an acoustic model, language model (LM), and alignment mechanism using a single neural network and require only parallel audio-text pairs. Thus, the language model component of the end-to-end model is only trained on transcribed audio-text pairs, which leads to performance degradation especially on rare words. While there have been a variety of work that look at incorporating an external LM trained on text-only data into the end-to-end framework, none of them have taken into account the characteristic error distribution made by the model. In this paper, we propose a novel approach to utilizing text-only data, by training a spelling correction (SC) model to explicitly correct those errors. On the LibriSpeech dataset, we demonstrate that the proposed model results in an 18.6% relative improvement in WER over the baseline model when directly correcting top ASR hypothesis, and a 29.0% relative improvement when further rescoring an expanded n-best list using an external LM.
1. INTRODUCTION
End-to-end ASR models use limited audio-text data, leaving rare-word errors that conventional external-LM methods do not explicitly target. The paper trains a spelling correction model on text-only-derived error pairs and reports substantial LibriSpeech WER improvements.
- End-to-end ASR models use far less language-model training data than conventional recognizers, harming performance on rare words.External-LM approaches use text-only data through n-best rescoring or fusion, but do not explicitly optimize for correcting ASR errors.
- Existing LM fusion methods leave numerous rare-word and proper-noun errors because their objectives do not target the end-to-end model’s characteristic errors.The paper contrasts training-loss-based fusion with decoding-only rescoring.
- The proposed method trains a spelling corrector using text-only data converted into synthetic audio, then decoded to create error hypotheses paired with ground truth.This procedure resembles backtranslation and supplies text-to-text training pairs for correction.
- 29.0% relative WER improvement results when the spelling corrector rescored an expanded n-best list with an external LM.Directly correcting the top ASR hypothesis yields an 18.6% relative improvement over the baseline.
2. BASELINE RECOGNITION MODEL
The baseline recognizer is a LAS-inspired attention-based encoder-decoder that maps mel-spectrogram features to wordpiece transcript probabilities.
- The LAS-inspired baseline encodes audio with convolutional and LSTM layers, aligns frames through attention, and decodes wordpiece units autoregressively.The decoder conditions on the attention context and previous prediction embedding to generate the next-token distribution.
- The model trains by minimizing cross-entropy over the transcript token sequence.
- The decoder functions similarly to a language model while jointly using acoustic attention context.
3. UTILIZING TEXT-ONLY DATA
The paper uses text-only data to synthesize recognizer-specific errors, trains an attention-based spelling corrector on those errors, and expands n-best hypotheses for rescoring.
- 3.1. External LM: External-LM rescoring combines LAS, spelling-correction, and LM scores using held-out-set interpolation weights.
- 3.3. Spelling correction model: The spelling corrector explicitly targets the recognizer’s characteristic errors rather than modeling text likelihood alone.Because the recognizer already supplies language-model capacity, correction often consists of copying the input while changing likely errors.
- 3.3.1. Training data: Text-only transcripts are converted to synthetic speech, decoded by LAS, and paired with ground truth to create correction training examples.Using all n-best hypotheses increases diversity and captures more variance in the recognizer’s error distribution.
- 3.3.2. Spelling correction model: The spelling corrector is an attention-based encoder-decoder that maps input wordpieces to corrected wordpiece sequences.Its architecture uses bidirectional encoder LSTMs, unidirectional decoder LSTMs, residual connections, layer normalization, and multi-head attention.
- 3.3.3. Inference: Correcting each of N LAS hypotheses produces N × M candidates that can be rescored with an external LM.The system can instead correct only the top LAS hypothesis, reducing the inference configuration to N = 1.
4. EXPERIMENTAL SETUP
Experiments use LibriSpeech speech and a large text-only corpus to train baseline, external-LM, synthetic-speech, and spelling-correction components.
- The experiments use 960 hours of LibriSpeech speech and an 800M-word text-only language-model corpus.The text corpus is filtered to 40M sequences for language-model training, TTS generation, and spelling-correction data creation.
- The baseline LAS and spelling-correction systems use attention-based LSTM architectures with a shared 16K wordpiece vocabulary.The baseline uses convolutional and bidirectional encoder layers, while the corrector uses bidirectional encoder and unidirectional decoder layers.
- A two-layer LSTM external LM rescored LAS n-best lists, with its interpolation weight tuned on a held-out development set.
- Parallel WaveNet generated 40M synthetic audio utterances from the text-only corpus for TTS-augmented LAS and spelling-correction training.The combined LAS training setup mixed real and synthetic speech.
- Each of the 40M clean TTS utterances produced eight LAS hypotheses, yielding about 320M spelling-correction training pairs.
- A multi-style configuration additionally corrupted synthetic speech with noise and reverberation at 20–40 dB SNR.
5. RESULTS
The spelling-correction approach improves ASR performance by correcting top hypotheses and rescoring expanded n-best lists, while realistic TTS training and external LM scores contribute complementary gains. Results also indicate that SC generates richer hypotheses and corrects rare-word, proper-noun, tense, and grammar errors, although TTS-to-real-audio mismatch remains a limitation.
- 5. RESULTS: 21.7% relative improvement over the LAS baseline comes from external LM rescoring, while TTS augmentation adds smaller gains but improves the rescored system further.The paper describes TTS-augmented training and LM rescoring as complementary methods.
- 5.1. SC model: 15.8% relative improvement over baseline is obtained when SC corrects only the recognizer’s top hypothesis.The SC attention is generally monotonic and uses adjacent context around errors to choose more suitable outputs.
- 5.1.1. Generating richer n-best lists: Correcting all eight LAS n-best entries expands the list to 64 candidates and nearly halves oracle WER, unlike correcting only the top hypothesis.The richer list is more likely to contain the correct transcript.
- 5.1.2. Train on more realistic TTS dataset: TTS-synthesized errors differ substantially from real-audio errors, and the SC model performs better on TTS test data even after LM rescoring.Noise is added to TTS data to make the generated n-best list less clean and more realistic.
- 5.1.2. Train on more realistic TTS dataset: 29.0% relative WER improvement over the LAS baseline results when LM rescoring is applied to the MTR-trained SC model.This is the reported overall result after applying LM rescoring to the expanded list.
6. CONCLUSIONS
The paper proposes a spelling correction model trained on recognizer-specific errors generated from TTS data, improving LAS recognition and n-best rescoring.
- The spelling correction model explicitly corrects errors made by a speech recognizer using synthetic error hypotheses from text-only data.The hypotheses are generated by decoding TTS audio synthesized from a large text-only corpus.
- Correcting all LAS n-best entries produces an expanded list with significantly lower oracle WER.
- The spelling correction model outperforms simple LM rescoring and direct LAS training on TTS data after external-LM rescoring.
- Training on MTR TTS data improves spelling-correction performance over clean TTS data.