Source-linked AI summary
Conformer: Convolution-augmented Transformer for Speech Recognition
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang
TL;DR
ASR models must capture both long-range global context and fine-grained local patterns efficiently. Conformer combines self-attention with convolution and achieves state-of-the-art LibriSpeech performance at 1.9%/3.9% WER on test/testother with an external language model.
Problem
ASR models based on self-attention or convolution have complementary limitations in capturing global context and fine-grained local patterns.
Method
Conformer combines self-attention and convolution within encoder blocks, with feed-forward modules surrounding them, to model global and local audio dependencies.
Results
1.9%/3.9% WER on LibriSpeech test/testother with an external language model establishes state-of-the-art performance.
Takeaways & Limitations
Conformer achieves better LibriSpeech accuracy with fewer parameters than previous work, with convolution modules critical to its performance.
Abstract
from arXiv · showhide
Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.
1. Introduction
The paper introduces Conformer, a parameter-efficient ASR architecture that combines self-attention for global interactions with convolutions for local correlations. It reports state-of-the-art LibriSpeech results across 10M, 30M, and 118M-parameter models while analyzing key architectural factors.
- Architecture: Conformer uses macaron-like feed-forward layers with half-step residual connections around multi-headed self-attention and convolution modules.The encoder is followed by a post layernorm.
- Motivation and contribution: Conformer combines self-attention and convolutions to model global interactions and relative-offset-based local correlations efficiently.The approach is motivated by the complementary strengths of Transformers and CNNs.
- Results: 15% relative improvement over the published Transformer Transducer is achieved on LibriSpeech testother with an external language model.The comparison uses the previous best published Transformer Transducer.
- Results: 2.7%/6.3% WER is achieved by the 10M-parameter model on LibriSpeech test/testother.This improves over similar-sized contemporary work.
- Results: 2.1%/4.3% WER is achieved by the 118M-parameter model without a language model on LibriSpeech test/testother.The paper also reports that its 30M model outperforms a 139M-parameter Transformer Transducer.
- Ablation studies: The study evaluates attention heads, convolution kernel sizes, activation functions, feed-forward placement, and convolution-module insertion strategies.These analyses examine how each factor contributes to accuracy improvements.
2. Conformer Encoder
The Conformer encoder begins with convolutional subsampling followed by Conformer blocks that replace Transformer blocks. Each block combines two feed-forward modules with self-attention and convolution, with ablations favoring this sandwich structure and post-attention convolution.
- Encoder architecture: The encoder applies convolution subsampling before a sequence of Conformer blocks, replacing Transformer blocks in the architecture.This design combines convolutional and Transformer-style processing in the encoder.
- Block composition: Each Conformer block stacks a feed-forward module, self-attention module, convolution module, and a second feed-forward module.The block is organized into four modules, with the second feed-forward module at the end.
- Self-attention module: Relative sinusoidal positional encoding in multi-headed self-attention improves generalization across input lengths and robustness to utterance-length variance.The attention module uses pre-norm residual units with dropout to support training and regularization of deeper models.
- Convolution module: The convolution module uses pointwise convolution with a GLU gate, followed by a single 1-D depthwise convolution and batch normalization.Batch normalization is placed after the convolution to aid training deep models.
- Block composition: Ablations find that two Macaron-style half-step feed-forward layers sandwiching attention and convolution significantly outperform a single feed-forward module.The second feed-forward module is followed by a final layer-normalization layer.
- Module ordering: Convolution stacked after self-attention works best for speech recognition among the studied convolution-and-attention arrangements.The ordering of these modules is evaluated in the ablation study described in Section 3.4.2.
3. Experiments · 3.1. Data
The model is evaluated on LibriSpeech using labeled speech, text-only data for language-model construction, 80-channel filterbanks, and SpecAugment.
- 3.1. Data: 970 hours of labeled speech and an additional 800M-word text-only corpus comprise the LibriSpeech evaluation resources.The text-only corpus is used to build a language model.
- 3.1. Data: 80-channel filterbank features are computed from 25ms windows with a 10ms stride.
- 3.1. Data: SpecAugment uses a frequency-mask parameter of F = 27 and ten time masks.The maximum time-mask ratio is pS = 0.05, with maximum mask size set to pS times utterance length.
3.2. Conformer Transducer
The Conformer Transducer uses parameter-scaled models with a single-LSTM-layer decoder, trained with dropout, variational noise, ℓ2 regularization, and Adam. An external language model supports shallow fusion and is tuned on development data.
- Model configurations: 10M, 30M, and 118M parameters define the small, medium, and large models, respectively, selected under model-size constraints.The architectures vary network depth, model dimensions, and attention heads; every model uses a single-LSTM-layer decoder.
- Regularization: 0.1 dropout is applied to each Conformer residual unit before its module output is added to the module input.The models also use variational noise and ℓ2 regularization with 1e−6 weight.
- Optimization: 10k warm-up steps and Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10−9 are used for optimization.Training follows a transformer learning-rate schedule with a peak learning rate of 0.05/√d, where d is the Conformer encoder model dimension.
- Language-model fusion: 4096 width characterizes the 3-layer LSTM language model used for shallow fusion, whose weight λ is tuned on development data by grid search.The language model has word-level perplexity 63.9 on development-set transcripts and is trained using LibriSpeech resources with 1k WPM tokenization.
3.3. Results on LibriSpeech
Conformer achieves competitive and consistently improved WER on LibriSpeech across model sizes, outperforming prior Transformer, LSTM, and convolutional models. Adding a language model yields the lowest word error rate among the compared models, demonstrating the benefit of combining convolution and Transformer layers.
- Without a language model: 2.3/5.0 WER on test/testother is achieved by the medium Conformer without a language model, outperforming the best known Transformer, LSTM, and similarly sized convolutional models.The metric is reported on LibriSpeech test/testother.
- Results across model sizes: 0.7% better on testother is achieved by the 10.3M-parameter Conformer than contemporary ContextNet(S).Table 2 compares Conformer with recent published models under different parameter-size constraints.
- Results across model sizes: 30.7M-parameter Conformer significantly outperforms Transformer Transducer, which uses 139M parameters.The comparison is against previously published state-of-the-art results.
- With a language model: The language-model-enhanced Conformer achieves the lowest word error rate among existing models, demonstrating the effectiveness of combining Transformer and convolution.The evaluation compares WER on LibriSpeech test-clean/test-other with ContextNet, Transformer transducer, and QuartzNet.
3.4. Ablation Studies
The ablations identify convolution as the most important Conformer feature, while Macaron-style FFN pairs and Swish activations also improve the design. Additional studies favor placing convolution after self-attention, using up to 16 attention heads, and selecting depthwise-convolution kernel size 32.
- Disentangling Conformer: The convolution sub-block is the most important difference between Conformer and Transformer when parameter counts are held constant.The ablation mutates Conformer toward Transformer while keeping the total number of parameters unchanged.
- Macaron-net Feed Forward modules: Macaron-style FFN pairs outperform a single FFN with the same parameter count, and Swish activations accelerate convergence.Conformer uses two FFNs around the attention and convolution blocks, with half-step residuals.
- Attention Convolution Blocks: A 0.1 degradation on dev-other occurs when convolution precedes MHSA, while lightweight convolution causes a significant performance drop, especially on devother.Table 4 compares Conformer, lightweight convolution, convolution before MHSA, and parallel convolution and MHSA.
- Attention heads: Increasing attention heads from 4 to 32 improves accuracy up to 16 heads, particularly on devother.The large model uses the same number of heads in every layer.
- Depthwise convolution kernel sizes: Kernel sizes 17 and 32 improve performance over smaller sizes, but size 65 worsens it; size 32 performs best by the second decimal in dev WER.The sweep evaluates kernel sizes {3, 7, 17, 32, 65} using the same size in all layers.
4. Conclusion
Conformer integrates CNN and Transformer components for end-to-end speech recognition, with convolution modules critical to performance. It achieves better accuracy with fewer parameters and new state-of-the-art LibriSpeech results.
- Conclusion: Conformer integrates CNN and Transformer components for end-to-end speech recognition.The architecture combines components from both model families.
- Conclusion: Convolution modules are critical to Conformer’s performance.The study examined the importance of each component and found that including convolution modules was essential.
- Conclusion: 1.9%/3.9% WER for test/testother establishes new state-of-the-art LibriSpeech performance.The model also achieves better accuracy with fewer parameters than previous work.