Source-linked AI summary
Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models
Rohit Prabhavalkar, Tara N. Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Cheng Chiu, Anjuli Kannan
TL;DR
Attention-based sequence-to-sequence ASR models are commonly optimized for log-likelihood even though performance is measured by WER. The paper trains them to minimize expected WER using sampling and N-best hypothesis approximations, finding stronger results with N-best lists and up to 8.2% relative improvement. The resulting grapheme-based uni-directional models match a traditional state-of-the-art discriminative ASR system on mobile voice search.
Problem
Sequence-to-sequence ASR models are typically trained for log-likelihood, while system performance is measured by WER.
Method
The paper minimizes expected WER using sampling-based and N-best-list approximations over hypotheses from attention-based models.
Results
8.2% relative WER improvement is achieved over the baseline, and grapheme-based uni-directional models match a traditional state-of-the-art discriminative ASR system.
Takeaways & Limitations
N-best-based minimum WER training supports grapheme-based attention models that match traditional discriminative ASR performance on a mobile voice-search task.
Takeaways & Limitations
Sampling-based approximations are less effective than N-best decoded hypotheses, and sampling-based training can degrade beam-search top-hypothesis WER.
Abstract
from arXiv · showhide
Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the log-likelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likelihood. Traditional ASR systems benefit from discriminative sequence training which optimizes criteria such as the state-level minimum Bayes risk (sMBR) which are more closely related to WER. In the present work, we explore techniques to train attention-based models to directly minimize expected word error rate. We consider two loss functions which approximate the expected number of word errors: either by sampling from the model, or by using N-best lists of decoded hypotheses, which we find to be more effective than the sampling-based method. In experimental evaluations, we find that the proposed training procedure improves performance by up to 8.2% relative to the baseline system. This allows us to train grapheme-based, uni-directional attention-based models which match the performance of a traditional, state-of-the-art, discriminative sequence-trained system on a mobile voice-search task.
1. INTRODUCTION
Attention-based sequence-to-sequence ASR models are usually trained with cross-entropy even though performance is measured by WER. This work directly minimizes expected WER using sampling- and N-best-based approximations, with N-best training producing strong gains.
- Motivation: Cross-entropy training improves log-likelihood, whereas ASR performance is commonly evaluated using word error rate.Traditional ASR addresses this mismatch with discriminative sequence training criteria such as sMBR.
- Approach: The work trains attention-based sequence-to-sequence models to directly minimize word error rate using approximations based on sampled or decoded hypotheses.The proposed methods approximate expected WER with either samples from the model or N-best recognition hypotheses.
- Approach: N-best hypotheses decoded with beam search are more effective for approximating expected WER than sampling from the model.The authors explicitly contrast the two approximation strategies and report greater effectiveness for N-best hypotheses.
- Results: 8.2% relative WER improvement is achieved over a cross-entropy-trained baseline on an English mobile voice-search task.The evaluation covers grapheme-based models with uni- and bi-directional encoders.
- Results: The resulting grapheme-based sequence-to-sequence models are comparable in performance to a state-of-the-art context-dependent phoneme-based ASR system.This comparison is reported for the mobile voice-search task.
2. ATTENTION-BASED MODELS
The attention-based model encodes acoustic features, uses attention to summarize encoder outputs, and decodes labels sequentially from previous predictions. Its output distribution is produced by a softmax layer, while the model uses multi-headed additive attention and beam-search inference.
- Model architecture: The encoder maps acoustic feature vectors to a sequence of encoded representations used by the decoder.The encoder is analogous to an acoustic model in a traditional ASR system.
- Model architecture: The attention mechanism summarizes encoded features according to the decoder state at each prediction step.The resulting context vector supplies information from the input utterance to the decoder.
- Model architecture: The decoder predicts one label at a time conditioned on previous predictions, with a softmax layer producing the next-label distribution.Training uses the ground-truth label sequence as input rather than scheduled sampling.
- Multi-headed attention: The model uses additive attention instead of dot-product attention and multiple independent attention heads that can attend to multiple input locations simultaneously.The authors report additive attention as more stable and multi-headed attention as significantly improving model performance.
- Training and inference: Beam search feeds predicted labels back into the model until the end-of-sentence symbol is produced.This procedure is used for inference after the model has been trained.
3. MINIMUM WORD ERROR RATE TRAINING OF ATTENTION-BASED MODELS
The paper trains attention-based models toward expected word error rate using tractable sample-based or N-best approximations. The two losses are combined with cross-entropy and initialized from a cross-entropy-trained model to stabilize optimization.
- Exact expected word-error loss is intractable because it sums over all possible label sequences.
- 3.1. Approximation By Sampling: The sampling approximation estimates expected word errors with an empirical average over hypotheses drawn from the model.Its gradient can likewise be estimated from samples, using the average sampled word-error count as a variance-reduction baseline.
- 3.2. Approximation Using N-best Lists: The N-best approximation restricts the loss to beam-search hypotheses, assuming probability mass is concentrated on the top N candidates.The hypotheses are re-normalized over the N-best list, and their average word-error count provides variance reduction.
- The two approximations define separate loss functions for expected word-error training.
- Both losses are interpolated with cross-entropy, and direct optimization is initialized from a cross-entropy-trained model to stabilize training.The interpolation coefficient λ is reported as important for stabilization.
4. EXPERIMENTAL SETUP
The experiments evaluate attention-based models on a large mobile voice-search task using a shared acoustic frontend. The models differ in whether their five-layer LSTM encoders are unidirectional or bidirectional.
- The evaluation uses approximately 15M hand-transcribed anonymized utterances, totaling approximately 12,500 hours of mobile voice-search traffic.Multi-style training data add reverberation and environmental noise to improve robustness.
- Uni-LAS uses five layers of 1,400 unidirectional LSTM cells, whereas Bidi-LAS uses five layers of 1,024 bidirectional LSTM cells.Both models use two decoder layers with 1,024 LSTM cells per layer and multi-headed attention.
5. RESULTS
Experiments compare sampling-based and N-best-list approximations for minimum WER training, showing that N-best optimization improves held-out WER and that MWER training yields substantial test-set gains for LAS models.
- 5.1. Comparison of loss functions: LSample and LN-best: ∼50% fewer expected word errors result from optimizing LSample, but beam-search top-hypothesis WER degrades after training.Performance appears to improve as the number of samples increases, while sampling and beam-search decoding emphasize different parts of the model distribution.
- 5.1. Comparison of loss functions: LSample and LN-best: About 10.4% lower held-out WER results from optimizing LN-best, with similar performance when only the top four hypotheses are used.The N-best-list approximation is more effective than the sampling-based approximation for the beam-search top hypothesis.
- 5.1. Comparison of loss functions: LSample and LN-best: CE-loss interpolation is important when optimizing LN-best, illustrated with N = 4 hypotheses and interpolation weight λ.The reported experiments set N = 4 and λ = 0.01 for minimum WER training.
- 5.2. Improvements from Minimum WER Training for LAS Models: 7.4% and 4.2% relative improvements occur for uni-directional and bi-directional LAS models without second-pass rescoring.The models are evaluated by directly decoding grapheme sequences with beam-search decoding using 8 beams.
- 5.2. Improvements from Minimum WER Training for LAS Models: 8.2% and 6.1% relative improvements occur after second-pass rescoring, and the MWER-trained grapheme-based uni-directional LAS system matches a traditional state-of-the-art system.The comparison system is a traditional CD-phone-based ASR system.
6. CONCLUSIONS
The paper presents expected minimum WER training for attention-based sequence-to-sequence systems. N-best decoded hypotheses are more effective than sampling-based approximations, improving WER by up to 8.2% relative and matching a traditional state-of-the-art system on voice search.
- The proposed technique trains attention-based sequence-to-sequence systems to optimize expected test error rate.
- N-best decoded-hypothesis approximations are more effective than sampling-based approximations for minimum WER training.
- Up to 8.2% relative WER improvement is achieved, with grapheme-based sequence-to-sequence models matching a traditional state-of-the-art system on a voice-search task.