Source-linked AI summary
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, Yonghui Wu
TL;DR
CNN-based ASR models are parameter-efficient but lag behind RNN and Transformer models, motivating methods that provide convolutional encoders with broader context. The paper proposes ContextNet, a CNN-RNN-Transducer architecture with squeeze-and-excitation global context and width scaling. ContextNet improves LibriSpeech results and accuracy–size trade-offs, with findings also confirmed on a larger, more challenging dataset.
Problem
CNN ASR models are parameter-efficient, but their WER remains behind RNN- and Transformer-based models because limited convolutional windows omit global sequence information.
Method
ContextNet combines a CNN-based audio encoder with squeeze-and-excitation global context, progressive downsampling, and global channel-width scaling in an RNN-Transducer architecture.
Results
1.9%/4.1% WER is reported on LibriSpeech test-clean/test-other, while the medium model reaches similar WER to much larger systems with 31M parameters.
Takeaways & Limitations
ContextNet provides a CNN-based ASR architecture with improved accuracy and model-size trade-offs, and its findings are also confirmed on a larger, more challenging dataset.
Takeaways & Limitations
Reported numbers can differ when the train and evaluation sets change, as noted for reproduced results in Table 6.
Abstract
from arXiv · showhide
Convolutional neural networks (CNN) have shown promising results for end-to-end speech recognition, albeit still behind other state-of-the-art methods in performance. In this paper, we study how to bridge this gap and go beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. In addition, we propose a simple scaling method that scales the widths of ContextNet that achieves good trade-off between computation and accuracy. We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2.1%/4.6% without external language model (LM), 1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy LibriSpeech test sets. This compares to the previous best published system of 2.0%/4.6% with LM and 3.9%/11.3% with 20M parameters. The superiority of the proposed ContextNet model is also verified on a much larger internal dataset.
1. Introduction
ContextNet targets the WER gap between CNN and RNN/Transformer ASR models by adding global context to a convolutional encoder. Its architecture and scaling scheme improve accuracy–model-size trade-offs on LibriSpeech.
- Motivation: CNN ASR models offer parameter efficiency, but their best WER remains behind RNN- and Transformer-based models.The paper attributes this gap to CNNs’ limited temporal context.
- ContextNet: ContextNet adds squeeze-and-excitation layers that summarize local features into a global context vector and broadcast it back to each feature.The authors report the largest WER reduction on LibriSpeech test-other from adding these layers.
- Results: 1.9%/4.1% WER is achieved on LibriSpeech test-clean/test-other with ContextNet using an RNN-T decoder.The result is reported as an improvement over previous CNN architectures and as outperforming transformer and LSTM models.
- Efficiency: Progressive downsampling reduces encoded-sequence length eight times, lowering computation while maintaining representation power and overall accuracy.It also permits reducing all convolution kernel sizes to five without significantly reducing the effective receptive field.
- Scaling: Global channel-width scaling produces the best model-size–WER trade-offs among the compared methods.Figure 1 plots LibriSpeech test-other WER against model size for models without external language models; α controls ContextNet scaling.
- Contributions: The paper contributes an improved CNN architecture with global context together with progressive downsampling and model scaling.These design choices target computation, accuracy, and model-size trade-offs.
2. Model
The model section introduces ContextNet’s architecture and its use of progressive temporal downsampling to reduce computation while maintaining accuracy.
- 2. Model: ContextNet uses a convolutional encoder whose temporal input length is progressively reduced to lower computation.The section presents high-level design first, then the convolutional encoder and its downsampling strategy.
- 2. Model: The downsampling strategy is designed to maintain the encoder’s representation power while reducing computation.
- 2. Model: The architecture details are organized around the high-level design, convolutional encoder, and temporal downsampling procedure.
2.1. End-to-end Network: CNN-RNN-Transducer
ContextNet uses the RNN-Transducer framework with a new CNN-based audio encoder alongside existing label-encoding and joint-network components.
- 2.1. End-to-end Network: CNN-RNN-Transducer: The end-to-end network contains an audio encoder, a label encoder, and a joint network.The joint network combines the audio and label representations for decoding.
- 2.1. End-to-end Network: CNN-RNN-Transducer: ContextNet replaces the usual audio encoder with a new CNN-based encoder within the RNN-Transducer framework.
- 2.1. End-to-end Network: CNN-RNN-Transducer: The label encoder and joint network are directly taken from the referenced LSTM-based design.
2.2. Encoder Design
ContextNet’s encoder stacks convolution blocks that combine depthwise separable convolutions, squeeze-and-excitation global context, residual projections, and progressive temporal downsampling. The design also uses Swish activations and configurable channel scaling to balance representation power, computation, and accuracy.
- Encoder architecture: The encoder maps an input sequence to a high-level representation by composing convolution blocks C1 through CK.Each block includes convolutional layers, normalization, activation, squeeze-and-excitation, and skip connections.
- Global context: Squeeze-and-excitation pools local features into a global channelwise weight, then multiplies that weight element-wise with each frame.The module uses a bottleneck of two fully connected layers and a Sigmoid output mapped to (0, 1).
- Convolutional layers: Depthwise separable convolution is used for parameter efficiency without impacting accuracy.The same kernel size is used across depthwise convolution layers.
- Activation and scaling: Swish consistently outperforms ReLU in the authors’ experiments, while α scales convolution channels and model representation power.Increasing α above 1 increases the number of convolution channels and model size.
- Convolution blocks: Each convolution block applies an SE module to the final convolution output and adds a projected skip connection.The block can contain multiple convolution layers, with strides configured for optional temporal downsampling.
- Temporal downsampling: Progressive 8× downsampling reduces computation while maintaining a good speed–accuracy trade-off.Strided convolution performs temporal downsampling, but excessive downsampling may negatively affect the decoder.
3. Experiments
Experiments evaluate ContextNet on LibriSpeech through model comparisons, ablations, scaling, computation reductions, and a larger YouTube test set. Results show improved accuracy–efficiency trade-offs and benefits from global context and progressive downsampling.
- Results on LibriSpeech: The LibriSpeech evaluation uses 970 hours of labeled speech, with small, medium, and large ContextNet configurations selected by width α ∈ {0.5, 1, 2}.An LSTM baseline is also built for reference, and language-model comparisons are included.
- Results on LibriSpeech: ContextNet(M) achieves similar WER to much larger systems with only 31M parameters, while ContextNet(L) improves over the previous SOTA by 13% relatively on test-clean and 18% on test-other.ContextNet(S) also improves on similarly sized systems with or without a language model.
- Effect of Context Size: Adding squeeze-and-excitation modules substantially improves WER over the zero-context baseline, with larger context windows providing greater benefit.The ablation uses α = 1.25 and compares context windows of 256, 512, and 1024.
- Depth, Width, Kernel Size and Downsampling: Globally scaling network width produces a good trade-off between model size and WER.The experiments sweep α and report both model size and LibriSpeech WER.
- Depth, Width, Kernel Size and Downsampling: Progressive downsampling significantly reduces FLOPS, slightly improves accuracy, and makes larger convolution kernels reduce WER.The comparison varies temporal reduction and kernel sizes of 3, 5, 11, and 21.
- Large Scale Experiments: ContextNet outperforms the previous best convolutional–bidirectional-LSTM architecture by 12% relatively on a 24.12-hour YouTube test set, using fewer parameters and FLOPS.The test set contains 117 videos with diverse and challenging acoustic environments.
4. Conclusion
The paper proposes and evaluates a CNN-based architecture for end-to-end speech recognition. It reports better LibriSpeech accuracy with fewer parameters than previously published CNN models and supportive results on a larger, more challenging dataset.
- Conclusion: ContextNet achieves better LibriSpeech accuracy with fewer parameters than previously published CNN models.The conclusion presents this as the paper’s main benchmark finding.
- Conclusion: Limiting ContextNet’s network width provides a way to search for small ASR models.The conclusion connects width scaling with model-size exploration.
- Conclusion: An initial study on a much larger and more challenging dataset also confirms the reported findings.The conclusion characterizes this evidence as an initial study.
- Conclusion: Encoder FLOPS are reported as the average computation required to process one second of audio.This defines the computation metric used in the paper’s comparisons.
- Conclusion: The reproduced large-scale results are subject to changed training and evaluation sets, so their values differ from those previously reported.This scope boundary applies specifically to the Table 6 comparison.