Source-linked AI summary
Listen, Attend and Spell
William Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals
TL;DR
Speech recognition traditionally separates acoustic, pronunciation, and language modeling, while prior end-to-end approaches impose limitations on label dependencies. LAS addresses this with a jointly trained pyramidal listener and attention-based character speller, achieving 14.1% WER without a dictionary or language model and 10.3% with rescoring, though generalization to long utterances remains difficult.
Problem
Traditional speech recognizers train acoustic, pronunciation, and language components separately, while previous end-to-end models make independence assumptions in label sequences.
Method
LAS jointly trains a pyramidal recurrent listener and an attention-based recurrent speller to encode speech and emit characters sequentially.
Results
14.1% WER was achieved without a dictionary or language model, improving to 10.3% with language-model rescoring over the top 32 beams.
Takeaways & Limitations
LAS directly transcribes acoustic signals to characters without phonemes, pronunciation dictionaries, HMMs, or label-sequence independence assumptions.
Takeaways & Limitations
The model struggles to generalize to long utterances when trained on a distribution of shorter utterances.
Abstract
from arXiv · showhide
We present Listen, Attend and Spell (LAS), a neural network that learns to transcribe speech utterances to characters. Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly. Our system has two components: a listener and a speller. The listener is a pyramidal recurrent network encoder that accepts filter bank spectra as inputs. The speller is an attention-based recurrent network decoder that emits characters as outputs. The network produces character sequences without making any independence assumptions between the characters. This is the key improvement of LAS over previous end-to-end CTC models. On a subset of the Google voice search task, LAS achieves a word error rate (WER) of 14.1% without a dictionary or a language model, and 10.3% with language model rescoring over the top 32 beams. By comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0%.
1 Introduction
LAS jointly learns speech recognition components end-to-end, transcribing audio into character sequences without label-sequence independence assumptions. Its pyramidal listener and attention-based speller improve handling of long acoustic inputs, while achieving competitive WER without a dictionary or language model.
- Model overview: LAS jointly trains a listener and speller to transcribe speech directly into character sequences.The listener is a pyramidal recurrent encoder, while the speller is an attention-based recurrent decoder.
- Model architecture: The pyramidal listener reduces the attention model’s time steps, helping process acoustic signals that may contain hundreds to thousands of frames.Without the pyramid structure, training converged too slowly and produced higher error rates.
- Character modeling: Character outputs automatically handle rare and out-of-vocabulary words and can generate multiple spelling variants.For “triple a,” the model produced both “triple a” and “aaa” among the top beams.
- Ablations: The attention mechanism and pyramid structure are necessary: without attention the model overfits, while without pyramidal encoding it converges too slowly.The reported failures occurred despite training on three million utterances.
- Results: 14.1% WER was achieved on a Google voice search subset without a dictionary or language model, versus 10.3% with language-model rescoring and 8.0% for CLDNN-HMM.The 10.3% result used language-model rescoring, while the CLDNN-HMM comparison was on the same dataset.
2 Related Work
Earlier speech recognizers combined sequential models with neural networks because variable-length sequence mapping was difficult to train directly. Sequence-to-sequence learning and attention provide a framework for mapping variable-length inputs to outputs while giving the decoder repeated access to encoder information.
- Prior approaches: Traditional structured speech recognizers combine neural networks with HMMs or CRFs, limiting straightforward end-to-end training.These systems also make simplifying assumptions about sequential structure.
- Sequence-to-sequence learning: Sequence-to-sequence learning maps a variable-length input through an encoder to a decoder that produces a variable-length output one token at a time.Training uses ground-truth decoder labels, while inference uses beam search for next-step predictions.
- Attention: Attention lets the decoder use an input-sequence attention vector at every output step instead of receiving encoder information only once.The mechanism provides repeated information flow from the encoder to the decoder.
- Applications: The sequence-to-sequence framework has been applied to translation, image captioning, parsing, and conversational modeling, motivating its use for speech recognition.Its broad application suggests speech recognition as another direct sequence-to-sequence task.
3 Model
LAS is an end-to-end sequence-to-sequence speech recognizer with a pyramidal listener and attention-based speller. The listener compresses acoustic inputs into high-level features, while the speller attends to them to generate character sequences.
- Model formulation: LAS accepts filter bank spectra and emits English character sequences, including start- and end-of-sentence tokens.The output vocabulary includes letters, digits, punctuation, space, and an unknown token.
- Model formulation: The listener transforms the acoustic sequence x into a shorter high-level representation h, and the speller produces character-sequence probabilities from h.This defines the model's two jointly trained operations: Listen and AttendAndSpell.
- Listener: The listener uses a pyramidal BLSTM that reduces time resolution by a factor of 2 in each successive stacked layer.Three pBLSTM layers above the bottom BLSTM reduce the time resolution 8 times, giving attention fewer steps to process.
- Listener: The pyramid structure reduces attention's computational complexity from dependence on the original sequence length, with attention over U features costing O(US).Reducing U speeds learning and inference while the deep architecture learns nonlinear feature representations.
- Speller: At each output step, the attention-based LSTM decoder conditions the next-character distribution on previous characters and a context vector derived from listener features.Content-based attention matches decoder state with listener features, then linearly blends them into the context vector.
- Learning and decoding: LAS is trained jointly end-to-end, using sampled previous-character predictions with a constant 10% rate to reduce the training–inference mismatch.This sampling exposes the decoder during training to its own possible prediction errors rather than only ground-truth histories.
4 Experiments
LAS was evaluated on Google voice search using large-scale augmented data, beam search, language-model rescoring, and comparisons with CLDNN-HMM. Experiments also examined beam width, utterance length, word frequency, attention alignments, and alternative decoding behavior.
- Data and setup: Three million utterances and 2,000 hours of Google voice search data were used, with augmentation increasing the audio amount twentyfold.Approximately 10 hours were held out for validation; augmentation added noise and reverberation using a room simulator.
- Baseline: The CLDNN-HMM baseline achieved 8.0% WER on clean and 8.9% on noisy test data.
- Main results: 14.1% clean and 16.5% noisy WER were obtained without language-model rescoring after sampling previous character predictions during training.
- Beam search: Beam-width increases improved WER through width 16, after which no significant benefit was observed; oracle rescoring of 32 beams reached 4.3% clean and 5.5% noisy WER.
- Error analysis: Long utterances produced more deletions and generalized poorly from shorter-utterance training, while short utterances showed mainly insertion and substitution errors.
5 Conclusions
LAS directly transcribes acoustic signals into characters using an end-to-end listener–speller architecture with attention and a pyramidal encoder. It avoids phonemes, pronunciation dictionaries, HMMs, and CTC-style conditional-independence assumptions while supporting multiple spelling variants.
- LAS directly transcribes acoustic signals to characters with an end-to-end attention-based neural network.Its listener encodes acoustics, while its speller attends to the resulting features and emits one character at a time.
- The listener is a pyramidal acoustic RNN encoder that reduces the number of timesteps attended to by the decoder.The speller is an RNN decoder that attends to high-level listener features to produce the transcript.
- LAS avoids phonemes, pronunciation dictionaries, and HMMs while bypassing CTC's conditional-independence assumptions.This character-level design can learn an implicit language model and generate multiple spelling variants from the same acoustics.
- Language-model rescoring further improves LAS hypotheses produced by the decoder.The conclusion states that additional text can support reranking of the model's top hypotheses.
A Alignment Examples
The alignment examples visualize LAS attention distributions and decoding behavior across spelling variants and repeated phrases. They show how the model can retain alternative spellings and recover the correct repeated phrase despite some attention confusion.
- Alignment visualizations: The alignment visualizations provide additional examples of LAS attention distributions.
- Spelling variants: Both “aaa” and “triple a” appear in the top beams with different attention distributions.The ground-truth phrase is “aaa emergency roadside service”.
- Spelling variants: Both “st” and “saint” appear in the top beams with different attention distributions.The ground-truth phrase is “st mary’s animal clinic”.
- Repeated phrases: For “cancel cancel cancel”, parallel attention diagonals show slight confusion, but the model still emits the correct hypothesis.