Attention-Based Models for Speech Recognition

• Speech recognition presents a challenge for attention because inputs contain thousands of noisy frames, making long sequences and similar speech fragments difficult to handle.The paper frames this as a testbed for attention architectures that can process long, noisy inputs.Source: §1 · p. 1
• Attention-based speech recognition is positioned as a step toward end-to-end trainable systems, unlike hybrid pipelines requiring hand-crafted pronunciation and phoneme lexicons and multistage training.The introduction also notes that a reported HMM-less CTC system added its language model only at the final stage, leaving its benefit during training open.Source: §1 · pp. 1–2
• On TIMIT, the paper evaluates phoneme recognition and tests longer-input performance using utterances created by concatenating existing utterances.Attention selects or weighs input-frame features at each step to help generate the next phoneme.Source: §1 · p. 2
• 18.7% PER on the original test set was competitive, but performance degraded quickly on concatenated utterances as the model tracked absolute input location.That strategy worked for short utterances but did not scale to longer ones.Source: §1 · p. 2
• 18.0% PER and below 20% PER on much longer utterances show that convolutional attention features enable robust recognition beyond training-length inputs.The auxiliary features combine the previous step’s attention location with input features, using trainable filters convolved with the previous attention weights.Source: §1 · p. 2
• 17.6% PER is reached by modifying attention to prevent it from concentrating on a single frame.The paper presents this as avoiding less effective training examples.Source: §1 · p. 2

2.1 General Framework

• The speech model generates phoneme sequences from audio-frame features encoded as a sequential representation by a deep bidirectional recurrent network.The encoder representation has the same length as the input feature sequence.Source: §2.1 · p. 2
• Content-based attention can confuse similar input elements, while location-only attention must predict variable distances between phonemes; hybrid attention uses previous alignment to narrow content selection.The encoder’s contextual information can help disambiguate similar fragments, but its capacity limits that benefit.Source: §2.1 · p. 3

2.2 Proposed Model: ARSG with Convolutional Features

• The proposed model makes content-based attention location-aware by convolving the previous alignment into vectors used in the attention-scoring mechanism.The convolution uses a matrix F to produce k location-feature vectors for each alignment position.Source: §2.2 · p. 4

2.3 Score Normalization: Sharpening and Smoothing

• For long inputs, normalized attention can spread weight across irrelevant frames, while softmax can also focus too narrowly on a single frame.The attention mechanism scores all L frames at each of T output steps, giving O(LT) complexity.Source: §2.3 · p. 4
• Windowing sharpens attention by scoring only frames near the previous alignment’s median, reducing complexity to O(L + T).Temperature sharpening and top-k selection also sharpen attention, but still score every frame and worsen overly narrow focus.Source: §2.3 · p. 4
• Replacing softmax’s unbounded exponential with a bounded logistic sigmoid smooths attention focus, encouraging aggregation across multiple high-scoring frames.Sharpening helped on long utterances but harmed performance on the short-utterance development set, motivating smoothing.Source: §2.3 · p. 4

3 Related Work

• CTC and RNN Transducer [14] are closest to the ARSG; CTC-based recognizers perform well on phoneme recognition [16] and can transcribe speech directly to text.These models extend earlier end-to-end sequence learning with gradients through alignment.Source: §3 · p. 5
• Unlike CTC and RNN Transducer, ARSG uses deterministic rather than latent alignments, enabling simpler beam search, faster decoding with smaller beams, and non-monotonic alignment.CTC and RNN Transducer constrain alignments to be monotonic to keep marginalization tractable; ARSG’s non-monotonic attention may suit a wider range of tasks.Source: §3 · p. 5
• The NTM hybrid [6] convolves content-based weights with a predicted shift distribution, whereas this paper learns how content- and location-based addressing combine through a deep parametric function.The learned combination is described in Eq. (9).Source: §3 · p. 5
• Unlike Sukhbaatar et al.’s [18] hybrid attention, which cannot handle inputs longer than those seen in training, this approach works on sequences many times longer.Source: §3 · p. 5

4 Experimental Setup

• The TIMIT experiments used 462 training speakers, 50 development speakers, and 24 core-test speakers, with 123 normalized features per frame and 61+1 decoding phones scored on 39 phones.Targets included an end-of-sequence token, and an all-zero input frame marked utterance end.Source: §4 · p. 5

4.1 Training Procedure

• Training used AdaDelta, orthogonalized recurrent weights, staged column-norm and adaptive-weight-noise regularization, and PER-based fine-tuning with batch size 1.The final fine-tuning stage used ϵ = 10−10 and continued until PER stopped improving for 100K weight updates.Source: §4.1 · p. 6

4.2 Details of Evaluated Models

• The shared model used a three-layer bidirectional GRU encoder, a single-layer GRU generator, and either content-based attention or convolutional location-aware attention, with smoothing in one variant.The encoder had 256 GRU units per direction; the generator had 256 units, and the attention convolution used k = 10 and r = 201.Source: §4.2 · p. 6
• Decoding used left-to-right beam search with width 10, increased to 40 if ⟨eos⟩ was not emitted; wider beams gave little-to-no benefit.Search stopped when the end-of-sequence token was emitted.Source: §4.2 · p. 6

5 Results

• Convolutional attention features improved PER by 3.7% relative to the baseline, and smoothing provided a further 5.9% relative improvement.Table 1 reports PERs, with its bold-faced value identifying the best rate for an ARSG with convolutional attention features and smooth focus.Source: §5 · p. 6, Table 1 · p. 7
• The baseline aligned repeated phonemes properly but favored frames near the start of the recording, sometimes slightly before the dataset’s phoneme locations; other models’ alignments looked similar.The alignment shown in Figure 3 demonstrates that the baseline was not confused by short-range repetitions.Source: §5 · pp. 6–7

5.1 Forced Alignment of Long Utterances

• The baseline aligned sequences up to about 120 phones when repeating one utterance and 150 when concatenating different utterances, while convolutional location-aware attention aligned up to 200 phones before failing abruptly.The long-utterance datasets used repeated or randomly concatenated test utterances, cross-faded with 0.05 seconds of silence, and included up to 15 utterances.Source: §5.1 · p. 7, Figure 4 · p. 7
• The baseline tracked approximate source location only up to training lengths before jumping to the recording’s end, whereas the location-aware network stopped aligning amid irrelevant frames in long concatenations.The location-aware network worked slightly better on repeated copies of one utterance, where frames were more relevant, than on randomly concatenated utterances.Source: §5.1 · pp. 7–8
• Alignment sharpening helped the location-aware network, and windowing let it align nearly all sequences; sharpening had little effect on the baseline.Windowing helped both networks, and the strongest robustness in Figure 4 came from combining hybrid attention with sharpening.Source: Figure 4 · p. 7, §5.1 · p. 8

5.2 Decoding Long Utterances

• About 20% PER was achieved by both location-aware networks on sequences formed from up to 11 test utterances using a wider decoding window.The baseline failed on long utterances, even with a narrow window; the wider window more closely resembled training conditions.Source: §5.2 · p. 8, Figure 5 · p. 8

6 Conclusions

• The hybrid attention architecture combines content and location information to select input positions and can recognize utterances much longer than its training examples.The authors envision direct speech-to-text recognition [10], potentially with a monolingual language model in the ARSG architecture [26].Source: §6 · p. 8
• The work contributes smoother alignment normalization and a general principle for extracting and using features from previous alignments.These attention ideas may extend beyond speech recognition, including to neural Turing machines and image caption generation using 2-D convolution [3].Source: §6 · p. 8

Additional Figures

• Figures 6 and 7 show evaluated-model alignments against ground-truth phone locations for one test utterance and one train utterance.Each upper-image row indicates frames selected by attention to emit a phone symbol.Source: Figure 6 · p. 11, Figure 7 · p. 12
• On long sequences, the content-only baseline aligns about 50 phones, while the convolutional hybrid fails to align any phone.Both models fail to properly align long sequences, but in different ways.Source: Figure 8 · p. 13
• The baseline aligns no more than 3 repetitions of FDHC0 SX209 and aligns 3 different concatenated utterances but fails on 5.These figures show the baseline's limits on repeated and concatenated inputs.Source: Figure 9 · p. 14, Figure 10 · p. 15
• With windowing enabled, the baseline aligns 7 repetitions of “Michael colored” but begins confusing phoneme locations on the third repetition.The window was ±75 frames from the expected generator position, wider than the pattern, and the network confuses similar content.Source: Figure 11 · p. 16
• 7 and 11 repetitions of FDHC0 SX209 are correctly aligned by the location-aware network, but 15 are not.The same network aligns a concatenation of 3 different utterances but fails on 5.Source: Figure 12 · p. 17, Figure 13 · p. 18

• Table 2 reports phoneme error rates under various decoding modifications for comparison with Figure 5 in the main paper.Source: Table 2 · p. 19