Source-linked AI summary
Very Deep Convolutional Networks for End-to-End Speech Recognition
Yu Zhang, William Chan, Navdeep Jaitly
TL;DR
End-to-end seq2seq ASR had relied on shallow acoustic encoders, motivating deeper models with more expressive processing. The paper builds very deep convolutional and recurrent encoders using Network-in-Network, Batch Normalization, residual connections, and Convolutional LSTMs. On WSJ, the best model achieves 10.53% WER without a language model, while the conclusion reports 10.5% WER without one.
Problem
End-to-end seq2seq ASR models had relied on shallow acoustic encoder networks, limiting the explored depth of their acoustic representations.
Method
The paper builds very deep hybrid convolutional and recurrent acoustic encoders using Network-in-Network, Batch Normalization, residual connections, and Convolutional LSTMs.
Results
10.5% WER was obtained on the WSJ ASR task without a language model.
Takeaways & Limitations
The techniques add expressive capacity and computational depth without substantially increasing the number of parameters.
Takeaways & Limitations
The convolutional models used only 32 output channels, leaving room for improvement by increasing channel size.
Abstract
from arXiv · showhide
Sequence-to-sequence models have shown success in end-to-end speech recognition. However these models have only used shallow acoustic encoder networks. In our work, we successively train very deep convolutional networks to add more expressive power and better generalization for end-to-end ASR models. We apply network-in-network principles, batch normalization, residual connections and convolutional LSTMs to build very deep recurrent and convolutional structures. Our models exploit the spectral structure in the feature space and add computational depth without overfitting issues. We experiment with the WSJ ASR task and achieve 10.5\% word error rate without any dictionary or language using a 15 layer deep network.
1. INTRODUCTION
The paper extends end-to-end ASR beyond shallow acoustic encoders by applying very deep convolutional and recurrent techniques. These methods add expressive depth while managing parameters and training challenges, improving the WSJ result over the baseline.
- Motivation: Very deep hybrid convolutional and recurrent encoders extend shallow seq2seq ASR models with greater expressive power.The approach adapts recent computer-vision developments to speech recognition.
- Motivation: CNNs exploit locality in spectral features through shared filters and pooling, supporting more robust and generalized models.Very deep CNNs add nonlinearities while using fewer parameters.
- Methods: Network-in-Network modules use 1x1 convolutions to increase depth and expressive power while reducing the parameters required for deeper models.The paper applies these principles within hierarchical recurrent neural networks.
- Methods: Batch Normalization reduces internal covariate shift, speeds training, regularizes the model, and was necessary for training the deeper seq2seq models.The attention mechanism’s gradients had high variance, especially from random initialization.
- Methods: Residual connections enable deeper acoustic encoders by addressing optimization and generalization problems associated with very deep networks.The paper also uses Convolutional LSTMs, replacing LSTM inner products with convolutions to preserve structural representations.
- Results: 10.53% WER was achieved by the best model, compared with 14.76% WER for the baseline.The model adds depth through nonlinear processing while keeping the number of parameters manageable.
2. MODEL
The model replaces a shallow LAS listener with very deep CNN, ConvLSTM, and residual structures, using normalization, attention, and subsampling to process speech representations.
- 2.1. Listen, Attend and Spell: LAS combines a listener acoustic encoder with an attention-based character speller that generates one character at a time.The listener transforms audio frames into a high-level representation, while the decoder consumes that representation to produce character probabilities.
- 2.1. Listen, Attend and Spell: The listener is replaced with very deep CNN and BLSTM networks, while the attention-based transducer remains the character decoder.The study adds NiN modules in hierarchical subsampling connections and between LSTM layers to increase depth.
- 2.2. Convolutional Neural Networks: CNN layers exploit spectral and temporal structure through convolution, weight sharing, and striding while reducing temporal resolution.The model investigates convolutional layers for sequence-to-sequence speech recognition and uses strided convolutions as a natural subsampling mechanism.
- 2.4. Batch Normalization: Batch normalization normalizes layer inputs and is applied sequencewise across output-channel statistics from all minibatch timesteps.The paper uses BN to accelerate training and improve generalization in the sequence-to-sequence acoustic encoder.
- 2.5. Convolutional LSTM: ConvLSTM replaces LSTM inner products with convolutions, preserving structural locality in cell states and outputs while reducing parameters.Its input-to-state and state-to-state transitions use convolutional operations; deeper ConvLSTM layers precede fully connected LSTM layers for attention compatibility.
- 2.6. Residual Network: Residual CNN/LSTM blocks use identity skip connections to support training of deeper acoustic networks.Convolutional residual blocks always use skip connections, whereas the experiments found them unnecessary for LSTM layers.
3. EXPERIMENTS
Experiments on WSJ evaluate progressively deeper acoustic encoders, combining subsampling, NiN modules, convolutions, residual blocks, and convolutional LSTMs. These changes improve WER from the 14.76% baseline to 10.53% for the best model.
- Setup: The WSJ experiments use si284 for training, dev93 for validation, and eval92 for testing, with 80-dimensional filterbank features.Features include delta and deltadelta accelerations normalized by speaker.
- Hierarchical Connections: 13.61% WER, a 7.8% relative gain over baseline, follows use of a projection subsampling layer in the deeper encoder.Simply increasing encoder depth to eight layers did not converge well.
- Network in Network: 12.88% WER, a 12.7% relative improvement over baseline, results from adding NiN 1 × 1 convolution modules between LSTM layers.Batch-normalization layers were critical for convergence, and Table 1 summarizes these experiments.
- Convolutions and Residual Connections: 11.80% WER follows two additional 3 × 3 convolutional layers, while 23 encoder layers with eight residual blocks reach 11.11% WER.The latter corresponds to a 24.7% relative improvement over baseline; eight residual blocks slightly outperform four.
- Convolutional LSTM: 10.53% WER is achieved by the best model, which combines two bottom convolutional layers, four residual blocks, and an LSTM NiN block.Each residual block contains one convolutional LSTM layer and one convolutional layer; Table 3 evaluates convolutional LSTM variants.
4. CONCLUSION
The work builds very deep convolutional and recurrent end-to-end ASR models, achieving 10.5% WER on WSJ without a language model while increasing expressive capacity without substantially increasing parameters.
- 10.5% WER was achieved on the WSJ ASR task without a language model, an 8.5% absolute improvement over the published best result.
- Network-in-Network principles added depth and non-linearities to hierarchical RNNs.
- Batch normalization and residual connections enabled very deep convolutional towers for processing acoustic features.
- Convolutional LSTMs replaced inner-product transitions with convolutions to preserve spectral structure in representations.
- The architecture increased expressive capacity without substantially increasing the number of parameters.
- The models used 32 output channels, leaving potential for improvement through larger channel sizes.