Source-linked AI summary
Attention-Based Models for Speech Recognition
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, Yoshua Bengio
TL;DR
Attention-based recurrent networks must handle speech inputs that are much longer and noisier than those in related sequence tasks. The paper develops hybrid attention using both content and location information, then smooths attention to avoid single-frame concentration. The resulting models achieve robust long-utterance recognition and reduce TIMIT phoneme error rate to 17.6%.
Problem
Speech recognition requires attention mechanisms to process long, noisy inputs containing similar speech fragments, while content-based attention can score similar representations equally regardless of position.
Method
The paper combines previous alignment locations with input features through convolutional attention features, and modifies score normalization to produce smoother attention.
Results
17.6% PER was achieved after smoothing, while convolutional features yielded 18.0% PER and remained below 20% PER on much longer utterances.
Takeaways & Limitations
The proposed end-to-end architecture can recognize utterances much longer than those used for training and its attention ideas may apply beyond speech recognition.
Takeaways & Limitations
Content-based attention remains limited by the capacity of contextual representations, and considering all input frames at every decoding step has O(LT) complexity.
Abstract
from arXiv · showhide
Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks in- cluding machine translation, handwriting synthesis and image caption gen- eration. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the at- tention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
1 Introduction
The paper adapts attention-based recurrent networks for speech recognition, where inputs are longer and noisier than in related tasks. A location-aware attention mechanism improves robustness to long utterances and, with smoother attention, lowers phoneme error rate.
- Motivation: Speech recognition poses a harder attention problem because inputs contain thousands of noisy frames and similar speech fragments, unlike machine translation’s shorter word sequences.This makes speech recognition a testbed for attention architectures that process long, noisy inputs.
- Baseline: 18.7% PER was achieved by the machine-translation attention baseline on TIMIT, but its performance degraded quickly on longer concatenated utterances.The baseline tracked absolute input location, which worked for short utterances but was inherently unscalable.
- Method: The proposed hybrid attention adds convolutional features from previous attention weights, combining prior focus location with input-sequence features.This provides generic location awareness while retaining content-based selection.
- Results: 18.0% PER was obtained with convolutional features, and the model stayed below 20% PER on utterances many times longer than the training examples.The result addresses the baseline’s rapid degradation on long concatenated inputs.
- Results: 17.6% PER was reached after modifying attention to avoid concentrating on a single frame and thereby reduce less effective training examples.The paper presents this smoothing change as a further contribution beyond location awareness.
- Contributions: The work contributes a purely neural attention-based speech recognizer, a generic location-awareness method, and a modification that smooths attention.Its performance is described as comparable to conventional approaches on TIMIT.
2 Attention-Based Model for Speech Recognition
The model generates phoneme sequences by attending to encoded speech representations, extending attention with location information and score-normalization strategies suited to long acoustic inputs.
- General Framework: An ARSG generates each output phoneme by attending to a sequential representation h produced by an encoder.The encoder is a deep bidirectional recurrent network, and the attention-weighted representation conditions the generator.
- General Framework: Hybrid attention combines the previous alignment with input features so similar speech fragments can be disambiguated by location and content.The previous alignment selects a local candidate region, after which content-based attention identifies relevant elements.
- Proposed Model: The proposed location-aware extension extracts convolutional features from the previous alignment and adds them to the attention scoring mechanism.These auxiliary features are computed by convolving the previous attention weights with trainable filters.
- Score Normalization: Positive normalized attention weights can spread a glimpse across many irrelevant frames in long inputs, creating noisy conditioning information.The issue arises because all weights are positive and sum to 1.
- Score Normalization: Softmax normalization can instead concentrate attention on one feature vector, preventing aggregation of multiple highly scored frames.The paper investigates smoothing to encourage selection from multiple top-scored frames.
- Score Normalization: Windowing sharpens attention while reducing decoding complexity from O(LT) to O(L + T) by scoring only a subsequence around the previous alignment.The window is centered using the median of the previous alignment and has predefined width w ≪ L.
3 Related Work
The paper situates its attention-based recurrent sequence generator alongside CTC, RNN Transducer, and prior hybrid attention mechanisms, emphasizing deterministic alignment and improved handling of long sequences.
- CTC and RNN Transducer are described as closely related speech-recognition models that perform well on phoneme recognition.They extend earlier end-to-end sequence-learning work with gradient signals flowing through alignment processes.
- CTC and RNN Transducer treat alignment as a latent random variable, whereas the ARSG deterministically aligns input and output sequences.The deterministic alignment also allows a simpler beam-search procedure.
- The proposed hybrid attention mechanism combines content and location information through a learned deep parametric function.This differs from the neural Turing machine approach, which convolves content-based weights with a predicted shifting distribution.
- A related location-embedding approach cannot handle input sequences longer than those seen during training, unlike the paper’s approach.The paper reports that its method works on sequences many times longer than the training sequences.
4 Experimental Setup
Experiments use TIMIT with recurrent neural encoders and generators, compare three attention configurations, and decode phoneme sequences using left-to-right beam search.
- All experiments use the TIMIT corpus with the Kaldi train-dev-test split, training on 462 speakers and testing on the 24-speaker core test set.The 50-speaker development set is used for early stopping, with SA utterances removed from training.
- The models use 40 mel-scale filterbank features plus energy and first- and second-order temporal differences for each frame.
- Adaptive learning rates, weight initialization, orthogonal recurrent weights, and adaptive weight noise address parameter-scale differences and regularization on the small dataset.Training begins with a column-norm constraint before adaptive weight noise is introduced.
- Figure 3 compares attention-selected frames with ground-truth phone locations and shows a left-to-right alignment that looks slightly ahead without confusing repeated phonemes.
- The encoder is a 3-layer bidirectional recurrent network with 256 GRU units per direction, while the generator has one recurrent layer of 256 GRU units.The top encoder layer provides a 512-unit representation, and the generator’s Generate network uses 64 maxout units.
- The baseline uses purely content-based attention, while the other two models use convolutional features with k = 10 and r = 201; one additionally applies smoothing.
- Decoding uses left-to-right beam search, starting at width 10 and increasing to 40 when the end-of-sequence token is not produced.The paper reports little-to-no benefit from wider beams.
5 Results
The baseline achieved competitive PER on the original task but degraded on longer utterances, while location-aware attention and sharpening improved alignment robustness and long-sequence decoding.
- 5.1 Forced Alignment of Long Utterances: The hybrid attention mechanism combined with sharpening achieved the highest robustness on concatenated utterances.This comparison is shown in the bottom-right plot of Figure 4.
- 5.1 Forced Alignment of Long Utterances: The baseline aligned sequences up to about 120 repeated phones or 150 concatenated phones, whereas convolutional features aligned sequences up to 200 phones.After failure, the convolutional-feature model aligned almost all phones incorrectly, unlike the baseline’s partial alignment.
- 5.1 Forced Alignment of Long Utterances: The baseline properly aligned about 40 first phones before jumping to the recording’s end and cycling over the last 10 phones.This behavior suggests that its approximate location tracking saturated on longer inputs.
- 5.1 Forced Alignment of Long Utterances: Sharpening helped the location-aware network find proper alignments, while windowing helped both baseline and location-aware networks.With windowing, the location-aware network properly aligned nearly all sequences.
- 5.2 Decoding Long Utterances: The baseline failed to decode long utterances, while the two location-aware networks decoded concatenations of up to 11 test utterances.With a wide window, both location-aware networks scored about 20% PER on long utterances.
6 Conclusions
The paper proposes an end-to-end speech recognition architecture that combines content and location information in attention. It also introduces smoother normalization and reusable features from previous alignments.
- The proposed architecture combines content and location information to select the next input position during decoding.The architecture is intended to recognize utterances much longer than those used for training.
- The paper contributes a generic principle for extracting and using features from previous alignments.The authors suggest this principle can apply beyond speech recognition, including neural Turing machines and image caption generation.
- A better normalization approach yields smoother alignments.
A Additional Figures
The additional figures examine attention alignments across utterances, repetitions, and models. They show baseline and location-aware alignment behavior under increasingly long or repeated inputs.
- Alignment visualizations: Figure 6 visualizes evaluated-model alignments for the FDHC0 SX209 test utterance against TIMIT phone locations.Rows show frames selected by attention to emit each phone symbol.
- Alignment visualizations: Figure 7 provides the same alignment visualization for the FAEM0 SI2022 training utterance.The figure compares selected attention frames with TIMIT ground-truth phone locations.
- Repeated-utterance alignment: The baseline network fails to align more than 3 repetitions of FDHC0 SX209.The repeated phrase is “Michael colored the bedroom wall with crayons.”
- Concatenated-utterance alignment: The baseline network aligns 3 concatenated utterances but fails with 5, while the location-aware network shows the same boundary in the illustrated concatenation test.The figures compare alignment robustness as concatenated utterance count increases.
- Failure analysis: The baseline’s attention window is ±75 frames around the expected generator position, yet similar content can still be confused.Confusion begins at the third repetition after the first two are aligned without confusion.
- Repeated-utterance alignment: The location-aware network correctly aligns 7 and 11 repetitions of FDHC0 SX209 but fails to align 15 repetitions.The baseline’s longer-input behavior is illustrated separately in the repeated-utterance figures.
B Detailed results of experiments
The detailed-results appendix includes a table of phoneme error rates for decoding with various modifications. It directs readers to compare these results with Figure 5 from the main paper.
- Detailed decoding results: Table 2 reports phoneme error rates for decoding with various model modifications.The table is intended as a detailed experimental-results reference.
- Detailed decoding results: The table’s results should be compared with Figure 5 from the main paper.This comparison is explicitly specified in the table caption.
- Detailed decoding results: Table 2 focuses on decoding outcomes rather than introducing a new evaluation task.Its caption frames the content as phoneme error rates under modified decoding conditions.