Source-linked AI summary
Transformers with convolutional context for ASR
Abdelrahman Mohamed, Dmytro Okhonko, Luke Zettlemoyer
TL;DR
Transformer ASR must represent speech position while modeling long-range dependencies and maintaining stable optimization. This paper replaces sinusoidal positional embeddings with convolutionally learned context for local structure and relative position. The resulting system reports 12% and 16% relative WER reductions on Librispeech “dev other” and “test other” without extra LM text.
Problem
Transformer ASR research needed to determine how positional information should be introduced into speech features and how to optimize these models stably.
Method
The model uses convolutional layers to learn local relationships and implicit relative positional encodings before transformer layers model global sequential structure.
Results
12% and 16% relative WER reductions were achieved on Librispeech “dev other” and “test other” versus previously published systems without extra LM text.
Takeaways & Limitations
Convolutional context provides sufficient positional signal for transformer layers to recover broader word order and learn long-range relationships.
Takeaways & Limitations
The study concentrates on Librispeech because findings from smaller datasets were considered unable to generalize reliably to new scenarios.
Abstract
from arXiv · showhide
The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.
1. Introduction
Transformers offer long-range modeling for speech recognition, but positional information and optimization remain central design questions. The paper replaces sinusoidal positional embeddings with learned convolutional context and reports relative WER reductions on challenging Librispeech subsets.
- 1. Introduction: Transformers model distant relationships through self-attention but require positional embeddings to represent word order.Sinusoidal embeddings are the standard approach described for transformer models.
- 1. Introduction: Prior end-to-end ASR transformer studies reported modest performance compared with baseline systems.Research explored both CTC and encoder-decoder transformer configurations.
- 1. Introduction: The paper investigates how positional information should be introduced into speech features, including omission, addition, or concatenation strategies.These alternatives leave different roles for the neural network in combining position and speech content.
- 1. Introduction: The proposed model replaces sinusoidal embeddings with 2-D encoder and 1-D decoder convolutions that learn local relationships and implicit relative positional encodings.Subsequent transformer layers model longer-range sequential structure.
- 1. Introduction: 12% and 16% relative WER reductions were achieved on Librispeech “dev other” and “test other” versus previously published results without extra LM text.The best configuration used a fixed learning rate of 1.0 and no hyperparameter or decoder optimization.
2. Transformers with convolutional context
The system divides speech-sequence modeling between convolutional layers for local relationships and transformer layers for global structure. Convolutions replace positional encodings, while masked decoder attention and full-utterance encoder attention support the end-to-end architecture.
- 2. Transformers with convolutional context: The model assigns local relationship learning to convolutional layers and global sequential modeling to transformer layers.The authors describe this division as simplifying transformer optimization and improving training stability.
- 2.1. Transformer layer: Multihead self-attention projects inputs into queries, keys, and values to combine information across time steps.Key-query similarities determine value-combination weights, with multiple attention heads operating in parallel.
- 2.1. Transformer layer: Transformer blocks apply attention alongside dropout, residual connections, layer normalization, and feed-forward layers.These operations are applied at each time step within the transformer layer.
- 2. Transformers with convolutional context: Encoder convolutional blocks use 2-D convolutions, layer normalization, ReLU activations, and max pooling, while decoder convolutions process prior outputs.The architecture omits separate positional encodings and repeats blocks throughout the model.
- 2. Transformers with convolutional context: Decoder self-attention is masked to current and previous targets, whereas encoder self-attention can access the entire input utterance.The setup does not investigate online or streaming decoding.
3. Experimental results
Experiments on Librispeech evaluate convolutional-context transformer ASR across positional encodings, convolutional context sizes, transformer depths, and published baselines. The strongest configuration improves WER substantially on the harder “other” subsets without external language-model text.
- Experimental setup: The evaluation uses 1000 hours of Librispeech speech, with clean and harder “other” development and test subsets.Inputs are 80-D log mel-filterbank coefficients plus three fundamental-frequency features; targets use 5k unigram subword units.
- Experimental setup: Experiments use fixed learning rate=1.0, gradient clipping at 10.0, 80 epochs, and checkpoint averaging without hyperparameter or decoder optimization.The regularization setup uses a single dropout rate of 0.15, with no scheduled sampling or label smoothing.
- Architectural comparisons: Convolutional positional context outperforms sinusoidal positional embedding, while combining both provides no additional gain.The result supports using relative convolutional positional information as sufficient signal for recovering more global word order.
- Architectural comparisons: The best decoder performance comes from wider context distributed across multiple convolutional layers under a fixed parameter budget.The decoder still achieves reasonable WER with a context of just 3 words, while deeper encoders capture longer-range structure.
- Final configuration: 16 encoder transformer blocks and 6 decoder transformer blocks are used in the best combined configuration, decoded with beam size 20.The combined model also uses 4k ReLU layers in all transformer blocks.
- Final results: 12% to 16% relative WER reduction is achieved on Librispeech “dev other” and “test other” versus previously published systems without external language-model text.The model shows little improvement on the clean subsets, while the reported gains remain compatible with further language-model improvements.
4. Conclusion and future work
The paper presents a sequence-to-sequence ASR transformer with learned convolutional context in both encoder and decoder. The best system reduces WER on Librispeech “dev other” and “test other,” and future work may combine these gains with improved training procedures.
- Conclusion: The system uses learned convolutional context in both the encoder and decoder.Convolutional layers capture local relationships and relative positional information before transformer layers model longer-range structure.
- Conclusion: 12% and 16% relative WER reduction is achieved on Librispeech “dev other” and “test other” without extra language-model text.A deep transformer encoder was important for reaching the best performance.
- Future work: Combining the system with a better training procedure such as Optimal Completion Distillation is identified as future work.