Source-linked AI summary
Towards better decoding and language model integration in sequence to sequence models
Jan Chorowski, Navdeep Jaitly
TL;DR
The paper examines overconfidence and incomplete transcriptions in attention-based seq2seq speech recognition, particularly when language models are used. It analyzes an encoder-decoder system and proposes smoothing and decoding strategies, with Temporal LS + Cov. reporting 9.7 and 6.7 in the experiments.
Problem
Seq2seq speech recognition models can produce sharply overconfident predictions that reduce beam-search transcript diversity, while wide beams with language models can yield incomplete transcripts.
Method
The system uses an attention-based encoder-decoder architecture with listener, speller, and attention modules, alongside neighborhood smoothing and coverage-related decoding strategies.
Results
Temporal LS + Cov. reports 9.7 and 6.7 in the experiments, while smoothing regularizes the model and raises prediction entropy for beam search.
Takeaways & Limitations
Efficient regularization and careful decoding make seq2seq speech recognition competitive with other non-HMM techniques such as CTC.
Abstract
from arXiv · showhide
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion. In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters. We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used. We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.
1. Introduction
Seq2seq models replace multi-stage processing pipelines with neural networks for complex transduction tasks, but can become overconfident and produce incomplete transcriptions, especially with language models.
- Seq2seq networks decompose processing into encoding and decoding modules for complex tasks including translation, speech recognition, and lip-reading.
- Overconfident predictions concentrate probability on few hypotheses, reducing transcript diversity during beam search.
- One-step-ahead training does not ensure that every relevant fragment of an input utterance is transcribed.
- Decoding mistakes can skip words and jump within the recording, making incomplete transcripts especially apparent with external language models.
2. Model Description
The system is an attention-based encoder-decoder that transcribes acoustic recordings into character sequences. It uses recurrent character modeling, attention over listener activations, cross-entropy training, beam-search decoding, and optional language-model integration.
- The attention-based system directly maps an audio recording x to a space-delimited character sequence y using listener, speller, and attention modules.
- The Listener: The listener transforms acoustic feature frames into a possibly shorter sequence of hidden activations for downstream decoding.
- The Speller and the Attention Mechanism: The speller assigns sequence probabilities one character at a time using the chain rule and conditions on listener activations.
- The Speller and the Attention Mechanism: Attention selects relevant listener activations into a context vector, while recurrent state stores the previously emitted character history.
- Training Criterion: Cross-entropy training scores character predictions conditioned on the partial transcript and the whole utterance, with label smoothing replacing indicator targets by distributions.
- Decoding: Beam Search: Beam search approximates most-probable transcript decoding by extending hypotheses character by character and retaining top-scoring candidates.
- Language Model Integration: Language-model integration extends beam-search cost with a language-model term, while coverage promotes longer transcripts to address incomplete outputs.
3. Solutions to Seq2Seq Failure Modes
The paper addresses seq2seq overconfidence and incomplete transcripts by combining neighborhood label smoothing with decoding strategies that improve input coverage and transcript completeness. These methods improve beam-search behavior, with temperature tuning helping baseline models while smoothing reduces its necessity.
- 3. Solutions to Seq2Seq Failure Modes: The proposed analysis separates confidence effects on model accuracy from beam-search effectiveness and introduces a practical solution based on input-utterance coverage.The decoding strategies include EOS constraints, transcript-length promotion, and coverage criteria.
- 3.1. Impact of Model Overconfidence: As temperature increases, baseline beam search finds better solutions without changing greedy accuracy, but truncated transcripts must be prevented.Temperature changes preserve token ordering and therefore do not affect next-step character selection, while smoother distributions improve beam-search exploration.
- 3.2. Label Smoothing Prevents Overconfidence: Neighborhood smoothing assigns probability mass to transcript-neighboring tokens, helping the model recover from beam-search errors that skip characters.Unlike uniform smoothing, it uses the temporal structure of transcripts and makes simple character omissions easier to recover from.
- 3.2. Label Smoothing Prevents Overconfidence: Nearly 3 percentage smaller error rate is achieved with greedy decoding after neighborhood smoothing, while higher prediction entropy lets beam search work without temperature control.The smoothed model is trained and evaluated with T = 1, avoiding separate control of EOS emission.
- 3.3. Solutions to Partial Transcripts Problem: Wide beam searches with a trigram language model can produce incomplete transcripts, including omissions at the beginning or severe truncations at wider beam sizes.A width 100 search may miss the utterance beginning, while even wider beams can discover severely incomplete transcriptions.
- 3.3. Solutions to Partial Transcripts Problem: Coverage promotion and transcript-length promotion improve WER as beam widths increase, whereas maximizing transcript length alone causes more word insertion errors and worse overall WER.Constraining EOS emissions is insufficient at large beam widths, while coverage and length-based methods improve decoding in that regime.
4. Experiments
Experiments on WSJ evaluate label smoothing and decoding techniques for attention-based seq2seq speech recognition, both without a language model and with an extended trigram language model. The reported results show lower error rates with smoothing, coverage, and careful decoding.
- Experimental setup: Experiments used WSJ si284 for training, dev93 for validation, and eval92 for evaluation.Results were averaged across two runs for the reported experiments.
- Decoding strategies: At large beam widths, coverage promotion and transcript-length promotion improve decoding, whereas constraining EOS emissions alone is insufficient.The comparison was made for networks using label smoothing and a trigram language model.
- Results without a language model: 12.6% and 8.9% are reported for the baseline with coverage without a separate language model.The two values are reported together in the results table.
- Results without a language model: 9.9% and 7.0% are reported with unigram label smoothing and coverage without a separate language model.This configuration improves on the baseline values shown in the same table.
- Results with a language model: With an extended trigram language model, beam-search parameters were tuned on validation data and applied to the test set.A typical setup used beam width 200, language-model weight λ = 0.5, coverage weight γ = 1.5, and coverage threshold τ = 0.5.
5. Related Work
Related work situates the paper’s approach among label-smoothing methods, normalization strategies, and coverage mechanisms developed for neural sequence models. These methods address prediction entropy, training normalization, and source-coverage representation or penalties.
- Label smoothing: Earlier label-smoothing schemes distribute probability mass uniformly, according to marginal class probabilities, or through model-generated erroneous labels.The paper positions neighborhood smoothing as a temporal-structure-based alternative.
- Label smoothing: Smoothing techniques increase prediction entropy and can prevent SoftMax saturation while improving gradient flow to lower network layers.The cited discussion connects smoothing to regularization and exploration in reinforcement learning.
- Normalization: Seq2seq networks use local normalization at every speller step, whereas global normalization scores whole transcripts and has benefited several NLP tasks.Classical ASR discriminative training normalizes over lattices, while recurrent-network alternatives use beam-search results.
- Coverage: Coverage has been represented through past attention vectors, cumulative coverage information, or coverage embeddings in neural machine translation.The paper’s decode-time coverage penalty is described as most similar to the Google Translation system’s approach.
6. Conclusions
The paper concludes that efficient regularization and careful decoding make sequence-to-sequence speech recognition competitive with other non-HMM techniques. It specifically identifies competitiveness with CTC as the supported conclusion.
- Conclusion: Efficient regularization and careful decoding make seq2seq speech recognition competitive with other non-HMM techniques such as CTC.This is the paper’s stated conclusion about the practical competitiveness of its approach.