Source-linked AI summary
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions
Awni Hannun, Ann Lee, Qiantong Xu, Ronan Collobert
TL;DR
Sequence-to-sequence speech recognition can generalize well, but its large bidirectional-LSTM encoders and sequential decoders can be inefficient. This paper combines a fully convolutional encoder with time-depth separable blocks and efficient decoding; it reports state-of-the-art end-to-end WER on both LibriSpeech test sets and substantially faster training than an RNN baseline.
Problem
Sequence-to-sequence speech recognition can generalize well, but large bidirectional-LSTM encoders and sequential decoding techniques often impose an efficiency cost.
Source: §1 · p. 1Method
The approach uses a fully convolutional encoder with time-depth separable blocks, efficient decoding, and stable beam search for language-model integration.
Source: §1 · p. 1, §2.1 · p. 1Results
The model achieves state-of-the-art end-to-end results of 3.28 WER on test clean and 9.84 WER on test other, with 14.1% and 22.9% lower WER than the RNN-based encoder, respectively.
Source: §4.1 · p. 3Takeaways & Limitations
The fully convolutional encoder and simple decoder outperform a strong RNN baseline while being an order of magnitude more efficient, and TDS blocks retain a large receptive field.
Source: §6 · p. 4
Abstract
from arXiv · showhide
We propose a fully convolutional sequence-to-sequence encoder architecture with a simple and efficient decoder. Our model improves WER on LibriSpeech while being an order of magnitude more efficient than a strong RNN baseline. Key to our approach is a time-depth separable convolution block which dramatically reduces the number of parameters in the model while keeping the receptive field large. We also give a stable and efficient beam search inference procedure which allows us to effectively integrate a language model. Coupled with a convolutional language model, our time-depth separable convolution architecture improves by more than 22% relative WER over the best previously reported sequence-to-sequence results on the noisy LibriSpeech test set.
1. Introduction
Attention-based sequence-to-sequence speech recognition can produce strong results, but its recurrent encoders and sequential decoders often reduce efficiency. The paper presents a fully convolutional TDS encoder and stable beam search to improve efficiency while maintaining strong recognition performance and using a convolutional language model.
1. Introduction
- Sequence-to-sequence speech recognition often uses large bidirectional LSTM encoders and sequential decoder techniques, which reduce efficiency.Efficiency supports faster training and evaluation and matters for large-scale semi-supervised and weakly supervised regimes.Source: §1 · p. 1
- The TDS model achieves state-of-the-art results for non-speaker-adapted sequence-to-sequence systems on both LibriSpeech test sets.Its TDS convolution improves WER over an RNN baseline, with parallel computation for greater efficiency.Source: §1 · p. 1
- A stable, efficient beam search maintains accuracy at very large beam sizes and helps leverage a convolutional language model, which improves WER over an n-gram baseline.The model also discards neural content attention, location-based attention, and scheduled sampling in favor of more efficient alternatives.Source: §1 · p. 1
2. Model
The sequence-to-sequence model encodes an input utterance into a hidden representation, then decodes it into output-token predictions. An RNN updates decoder queries, attention produces summary vectors, and a separate function computes token distributions.
2. Model
- The model encodes an input utterance into a hidden representation and decodes that representation into predictions for each output token.Source: §2 · p. 1
- The decoder’s RNN uses the previous token and query vector to produce the next query vector.Attention produces a summary vector, and another function computes the output-token distribution.Source: §2 · p. 1
2.1. Time-Depth Separable Convolutions
The TDS block separates temporal aggregation from channel mixing, enabling a larger receptive field with a negligible increase in parameters. It implements the temporal convolution followed by a fully connected block, with residual connections and layer normalization.
2.1. Time-Depth Separable Convolutions
- The TDS block partially separates aggregation over time from channel mixing, allowing a larger receptive field with a negligible parameter increase.The block can be implemented efficiently with a standard 2D convolution.Source: §2.1 · p. 1
- A 2D convolution with kernels of size k × 1 maps an input of shape T × w × c to the same shape and has kc^2 parameters.Keeping c small keeps this layer’s parameter count small.Source: §2.1 · p. 2
- The convolution output is reshaped to T × 1 × wc and passed through two 1 × 1 convolutions with an intervening ReLU.Residual connections and layer normalization follow both the convolution and fully connected layer.Source: §2.1 · p. 2
2.2. Efficient Decoder
The decoder is sequential at inference, but teacher forcing allows output frames to be computed in parallel during training; the authors use efficient attention and avoid other sequential techniques. Random sampling replaces scheduled sampling and improves WER.
2.2. Efficient Decoder
- Teacher forcing lets the decoder compute output frames simultaneously during training, and optimized RNN computation leaves it under 10% of iteration time.The decoder remains sequential because each next output requires the previous prediction.Source: §2.2 · p. 2
- The decoder avoids input feeding and location-based attention, uses efficient inner-product key-value attention, and scales inner products by the inverse square root of hidden dimension.Scaling improves convergence and alignment learning, but shows no consistent generalization improvement.Source: §2.2 · p. 2
2.2.1. Random Sampling
- Random sampling replaces the previous prediction with a uniformly sampled token, allowing vectorized implementation and improving WER.The method samples a token with probability Prs; its uniform distribution excludes EOS.Source: §2.2.1 · p. 2
2.3. Soft Window Pre-training
The authors introduce a soft attention-window pre-training scheme for deep convolutional encoders, penalizing attention far from uniformly spaced input-output alignments. They apply it only during early epochs, with σ adjusted if encoder subsampling changes.
2.3. Soft Window Pre-training
- The pre-training scheme is simple, adds negligible computational expense, and needs little tuning compared with prior work.It enables training of very deep convolutional encoders.Source: §2.3 · p. 2
- Soft-window pre-training encourages uniform input-output alignment by penalizing attention far from the expected locations.The expected locations space outputs uniformly along the input, with distances scaled by T/U.Source: §2.3 · p. 2
- The attention penalty acts like a Gaussian-shaped mask after softmax, with σ serving as its standard deviation.The parameter σ dampens the effect of the distance matrix.Source: §2.3 · p. 2
- The authors switch off window pre-training after the first few epochs; σ generally need not be retuned except when encoder subsampling changes.This schedule suffices for the model to learn an alignment and converge.Source: §2.3 · p. 2
2.4. Regularization
The section presents regularization and word-piece sampling choices for controlling overfitting and improving generalization. These include encoder dropout, label smoothing, and sampling word-piece representations at the word level.
2.4.1. Dropout
- Dropout follows each encoder block’s nonlinearity and precedes layer normalization; the decoder uses no dropout.The section introduces three forms of regularization to control overfitting and improve generalization.Source: §2.4.1 · p. 2, §2.4 · p. 2
2.4.2. Label Smoothing
- Label smoothing hurts development-set loss but improves WER.The authors use it to reduce over-confidence in predictions.Source: §2.4.2 · p. 2
2.4.3. Word Piece Sampling
- Word-piece representations are sampled per word rather than per sentence, using the most likely representation or a uniformly sampled top-ten alternative.The choice between these options is controlled by Pwp.Source: §2.4.3 · p. 2
3. Beam Search Decoding
The decoder searches hypotheses using sequence-to-sequence, language-model, and token-count scores. Stabilization techniques support larger beams, while pruning and batched computation improve decoding efficiency.
3. Beam Search Decoding
- The open-vocabulary beam search scores hypotheses using sequence-to-sequence probability, language-model probability, and a token-insertion term.The token-count term is |Y|; α weights the language model and β weights token insertion.Source: §3 · pp. 2–3
3.1. Stabilizing Beam Search
- Two stabilization techniques address beam-search instability, enabling larger-beam search to leverage language-model integration.Prior decoders can perform worse as beam size increases [14].Source: §3.1 · p. 3
3.1.1. Hard Attention Limit
- A hard attention limit rejects hypotheses that move more than tmax frames from the previous attention peak.The authors report that tmax needs tuning only once per dataset.Source: §3.1.1 · p. 3
3.1.2. End-of-sentence Threshold
- End-of-sentence proposals are considered only when their score exceeds a specified factor of the best candidate score.The parameter γ needs tuning only once per dataset.Source: §3.1.2 · p. 3
3.2. Efficiency
- Beam thresholds prune low-scoring hypotheses and candidate tokens, while batched probability updates require only one forward pass per step.Together, these heuristics yield fast decoding with a deep convolutional language model and a large beam.Source: §3.2 · p. 3
5. Related Work
Prior work used fully convolutional encoders in machine translation and speech recognition with structured losses, but the authors knew of no competitive fully convolutional sequence-to-sequence speech recognizer. Their encoder relates to Transformer and lightweight convolution designs.
5. Related Work
- Earlier fully convolutional encoders succeeded in machine translation [15] and speech recognition with structured losses [6] [31], but competitive sequence-to-sequence speech results were not known.The authors distinguish these results from their target setting: sequence-to-sequence speech recognition with a fully convolutional encoder.Source: §5 · p. 4
- Like lightweight convolution [34], the encoder separates temporal integration from channel mixing; unlike Transformer, it uses convolutions instead of self-attention.The time-depth separable convolution can be implemented with a simple 2D convolution and does not normalize over the time dimension of the kernels.Source: §5 · p. 4
- The TDS block's first layer resembles a grouped 1D convolution with tied weights between groups, connecting it to efficiency-oriented grouped-convolution work.Depth-wise separable convolutions had also been used to improve computer-vision model efficiency and accuracy [36].Source: §5 · p. 4
6. Conclusion
A fully convolutional encoder with a simple decoder outperforms a strong RNN baseline while running an order of magnitude more efficiently. The TDS block preserves a large receptive field, and stable scalable beam search supports integration of a strong convolutional language model.
6. Conclusion
- The fully convolutional encoder and simple decoder outperform a strong RNN baseline while being an order of magnitude more efficient.The conclusion attributes the convolutional encoder's success to a TDS block structure that retains a large receptive field.Source: §6 · p. 4
- The TDS block structure lets the convolutional encoder retain a large receptive field.The conclusion identifies this structure as key to the encoder's success.Source: §6 · p. 4
- A stable, scalable beam search procedure integrates a strong convolutional language model.Source: §6 · p. 4