Source-linked AI summary
Very Deep Self-Attention Networks for End-to-End Speech Recognition
Ngoc-Quan Pham, Thai-Son Nguyen, Jan Niehues, Markus Müller, Sebastian Stüker, Alexander Waibel
TL;DR
End-to-end speech recognition needs alternatives to recurrent and time-delay acoustic models, particularly for challenging acoustics-to-text mapping. This paper uses very deep stochastic Transformer models and achieves state-of-the-art results among end-to-end ASR systems on two standard benchmarks.
Problem
End-to-end speech recognition requires effective acoustic modeling beyond prior LSTM and TDNN approaches, while mapping acoustics directly to text remains challenging.
Method
The paper replaces recurrence with self-attention in very deep Transformer encoder-decoders and uses stochastic residual layers to facilitate training and regularization.
Results
The models achieve state-of-the-art performance among end-to-end ASR systems on two standard benchmarks and outperform previous end-to-end approaches on Switchboard.
Takeaways & Limitations
Very deep stochastic Transformers bring end-to-end ASR to competitive levels with hybrid systems and can outperform some of them.
Takeaways & Limitations
In one reported model, dropout values were unchanged, causing each layer’s hidden layers to be dropped more severely.
Abstract
from arXiv · showhide
Recently, end-to-end sequence-to-sequence models for speech recognition have gained significant interest in the research community. While previous architecture choices revolve around time-delay neural networks (TDNN) and long short-term memory (LSTM) recurrent neural networks, we propose to use self-attention via the Transformer architecture as an alternative. Our analysis shows that deep Transformer networks with high learning capacity are able to exceed performance from previous end-to-end approaches and even match the conventional hybrid systems. Moreover, we trained very deep models with up to 48 Transformer layers for both encoder and decoders combined with stochastic residual connections, which greatly improve generalizability and training efficiency. The resulting models outperform all previous end-to-end ASR approaches on the Switchboard benchmark. An ensemble of these models achieve 9.9% and 17.7% WER on Switchboard and CallHome test sets respectively. This finding brings our end-to-end models to competitive levels with previous hybrid systems. Further, with model ensembling the Transformers can outperform certain hybrid systems, which are more complicated in terms of both structure and training procedure.
1. Introduction
The paper revisits acoustic modeling for end-to-end ASR by replacing prior LSTM- and TDNN-based sequence modeling with Transformer self-attention. It emphasizes depth and stochastic depth as central to training competitive models, achieving a state-of-the-art end-to-end result on Switchboard with 48 Transformer layers.
- End-to-end sequence-to-sequence ASR jointly trains components toward a common goal, reducing complexity and error propagation relative to traditional hybrid systems.
- Previous end-to-end approaches generally used LSTM or time-delay neural networks on frame-level features to learn sequence-level representations.
- Transformer self-attention establishes direct connections between sequence elements and has achieved state-of-the-art performance in mainstream NLP tasks.
- The paper identifies depth as important for competitive Transformer-based end-to-end ASR and proposes stochastic depth to facilitate training very deep configurations.
- 48 Transformer layers across the encoder and decoder achieved a state-of-the-art end-to-end result on the standard 300h Switchboard benchmark.
2. Model Description
The model is a Transformer encoder–decoder in which multi-head self-attention replaces recurrence. Its residual, normalized, and stochastic connections support efficient computation and training of deep networks.
- Transformer architecture: Multi-head attention is the Transformer’s core mechanism, replacing recurrence while modeling relationships across input and output time steps.The decoder conditions on encoder representations through attention.
- Transformer architecture: Each encoder and decoder layer combines self-attention with a feed-forward neural network, while residual connections and layer normalization stabilize deep representations.The encoder feed-forward network uses one hidden layer with ReLU activation.
- Input adaptation: The encoder groups consecutive speech frames, adds sinusoidal positional encoding, and projects concatenated acoustic and positional features before addition.Directly adding acoustic features to positional encodings can cause divergence during training.
- Transformer architecture: The decoder masks self-attention to preserve autoregressive access and adds encoder–decoder attention between self-attention and feed-forward processing.Target hidden states provide queries, while encoder outputs provide keys and values.
- Stochastic residual connections: Stochastic residual connections randomly skip entire layers during training and expose the full network during inference, creating varied subnetworks and an ensembling effect.The mask is Bernoulli-distributed, with M = 1 activating the inner function and M = 0 skipping it.
3. Experimental Setup
Experiments used the Switchboard-1 corpus and Hub5’00 test set, with down-sampled log mel filter-bank inputs. Models followed a Transformer Base-style configuration with specified optimization and regularization settings.
- Data and features: Over 300 hours of Switchboard-1 Release 2 speech were used for training, with Hub5’00 as the test set.The corpus is identified as LDC97S62 and the evaluation data as LDC2002S09.
- Data and features: 40 normalized log mel filter-bank features were extracted per conversation, and stacking 4 consecutive vectors reduced input-sequence length by a factor of 4.The features were normalized per conversation before down-sampling.
- Stochastic residual connections: Dropping lower-level representations was less tolerable than dropping higher-level representations when using a constant p across connections.This finding came from early experiments with stochastic residual connections.
- Model and optimization: The Transformer configuration used embedding dimension d = 512 and feed-forward hidden-state size 1024.The hyperparameter search was based on the original Transformer paper’s Base machine-translation configuration.
- Model and optimization: Optimization used Adam with an initial learning rate of 2, 8000 warm-up steps, and updates every 25000 characters.Mini-batches were sized to fit the model in GPU memory, with gradients accumulated between updates.
- Regularization: Dropout was set to 0.2, while character dropout and label smoothing used p = 0.1 and ϵ = 0.1, respectively.Dropout was applied before residual connections and on attention weights.
4. Results
Results show that increasing Transformer depth improves speech-recognition performance, while stochastic residual connections improve generalization, especially in deeper models. Deep encoders, combined regularization, and data augmentation produce competitive or state-of-the-art results across SWB and TED-LIUM benchmarks.
- Depth and stochastic layers: WER fell from 20.8% to 12.1% on SWB as depth increased from 4 to 24 layers, although gains between 12 and 24 layers were smaller.The smaller gain was only 5% relative WER improvement, suggesting overfitting.
- Depth and stochastic layers: Stochastic connections improved CH performance substantially more at 24 layers than at 12, while the stochastic 48-layer model continued improving.The 48-layer result indicates better generalization on the CH test set.
- Architecture configuration: Deeper, narrower networks outperformed a wider 8-layer model with more parameters, showing that depth was more beneficial than simply increasing model width.The contrastive model was compared against the deep 24-layer model.
- Architecture configuration: With 48 total layers, the encoder required greater depth than the decoder, reflecting their different roles in representation learning and character-sequence generation.The encoder processes audio features, whereas the decoder conditionally generates character sequences.
- Regularization and benchmarks: 18.1% WER was achieved with the 36 −12 setup after combining dropout, label smoothing, stochastic networks, and data augmentation, establishing a state-of-the-art SWB end-to-end ASR result.The models also outperformed the best hybrid models on the CH test set under similar data constraints.
- Regularization and benchmarks: 21.8% relative WER improvement was obtained on TED-LIUM using 36 encoder and 12 decoder layers with stochastic residual connections, outperforming a strong baseline.The baseline used an external language model trained on larger data and speed perturbation.
5. Related Work
Prior ASR research explored self-attention alongside LSTMs, within CTC models, and in Transformer variants augmented with TDNN downsampling. Although self-attention offered benefits such as faster training and interpretability, these works did not identify an enhancement.
- 5. Related Work: Prior ASR studies combined self-attention with LSTMs, used it as an alternative in CTC models, or augmented Transformers with TDNN downsampling.These approaches investigated self-attention as a main ASR component in several architectural forms.
- 5. Related Work: Self-attention offered benefits including training speed and model interpretability, but previous work did not identify an enhancement.
6. Conclusion
The paper shows that Transformer self-attention can replace TDNNs or LSTMs for acoustic modeling in end-to-end speech recognition. Very deep stochastic Transformer models achieve state-of-the-art results among end-to-end systems on two standard benchmarks.
- End-to-end mapping from acoustics to text transcriptions remains challenging for sequence-to-sequence models.
- The work demonstrates that Transformer self-attention can serve as an alternative to TDNNs or LSTMs for acoustic modeling.The authors identify this as the first demonstration of Transformer effectiveness for ASR.
- Very deep stochastic Transformer models achieve state-of-the-art results among end-to-end models on two standard benchmarks.The networks are also among the deepest configurations reported for ASR.