PaperlayerView exact revision on arXiv

Source-linked AI summary

ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context

Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, Yonghui Wu

arXiv:2005.03191v3eess.AScs.CLcs.LGcs.SD

TL;DR

CNN-based ASR models are parameter-efficient but lag behind RNN and Transformer models, motivating methods that provide convolutional encoders with broader context. The paper proposes ContextNet, a CNN-RNN-Transducer architecture with squeeze-and-excitation global context and width scaling. ContextNet improves LibriSpeech results and accuracy–size trade-offs, with findings also confirmed on a larger, more challenging dataset.

  • Problem

    CNN ASR models are parameter-efficient, but their WER remains behind RNN- and Transformer-based models because limited convolutional windows omit global sequence information.

    Source: §1 · p. 1

  • Method

    ContextNet combines a CNN-based audio encoder with squeeze-and-excitation global context, progressive downsampling, and global channel-width scaling in an RNN-Transducer architecture.

    Source: §1 · p. 1, §2.2.4 · p. 2

  • Results

    1.9%/4.1% WER is reported on LibriSpeech test-clean/test-other, while the medium model reaches similar WER to much larger systems with 31M parameters.

    Source: §1 · p. 1, §3.1 · p. 3

  • Takeaways & Limitations

    ContextNet provides a CNN-based ASR architecture with improved accuracy and model-size trade-offs, and its findings are also confirmed on a larger, more challenging dataset.

    Source: §1 · p. 2, §4 · p. 4

    Reported numbers can differ when the train and evaluation sets change, as noted for reproduced results in Table 6.

    Source: §4 · p. 4

Abstract

from arXiv · show

Convolutional neural networks (CNN) have shown promising results for end-to-end speech recognition, albeit still behind other state-of-the-art methods in performance. In this paper, we study how to bridge this gap and go beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. In addition, we propose a simple scaling method that scales the widths of ContextNet that achieves good trade-off between computation and accuracy. We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2.1%/4.6% without external language model (LM), 1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy LibriSpeech test sets. This compares to the previous best published system of 2.0%/4.6% with LM and 3.9%/11.3% with 20M parameters. The superiority of the proposed ContextNet model is also verified on a much larger internal dataset.

1. Introduction

ContextNet targets the WER gap between CNN and RNN/Transformer ASR models by adding global context to a convolutional encoder. Its architecture and scaling scheme improve accuracy–model-size trade-offs on LibriSpeech.

Motivation

  • CNN ASR models offer parameter efficiency, but their best WER remains behind RNN- and Transformer-based models.The paper attributes this gap to CNNs’ limited temporal context.Source: §1 · p. 1

ContextNet

  • ContextNet adds squeeze-and-excitation layers that summarize local features into a global context vector and broadcast it back to each feature.The authors report the largest WER reduction on LibriSpeech test-other from adding these layers.Source: §1 · p. 1

Results

  • 1.9%/4.1% WER is achieved on LibriSpeech test-clean/test-other with ContextNet using an RNN-T decoder.The result is reported as an improvement over previous CNN architectures and as outperforming transformer and LSTM models.Source: §1 · p. 1

Efficiency

  • Progressive downsampling reduces encoded-sequence length eight times, lowering computation while maintaining representation power and overall accuracy.It also permits reducing all convolution kernel sizes to five without significantly reducing the effective receptive field.Source: §1 · p. 1

Scaling

  • Global channel-width scaling produces the best model-size–WER trade-offs among the compared methods.Figure 1 plots LibriSpeech test-other WER against model size for models without external language models; α controls ContextNet scaling.Source: §1 · p. 1, Figure 1 · p. 2

Contributions

  • The paper contributes an improved CNN architecture with global context together with progressive downsampling and model scaling.These design choices target computation, accuracy, and model-size trade-offs.Source: §1 · pp. 1–2

2. Model

The model section introduces ContextNet’s architecture and its use of progressive temporal downsampling to reduce computation while maintaining accuracy.

2. Model

  • ContextNet uses a convolutional encoder whose temporal input length is progressively reduced to lower computation.The section presents high-level design first, then the convolutional encoder and its downsampling strategy.Source: §2 · p. 2
  • The downsampling strategy is designed to maintain the encoder’s representation power while reducing computation.Source: §2 · p. 2
  • The architecture details are organized around the high-level design, convolutional encoder, and temporal downsampling procedure.Source: §2 · p. 2

2.1. End-to-end Network: CNN-RNN-Transducer

ContextNet uses the RNN-Transducer framework with a new CNN-based audio encoder alongside existing label-encoding and joint-network components.

2.1. End-to-end Network: CNN-RNN-Transducer

  • The end-to-end network contains an audio encoder, a label encoder, and a joint network.The joint network combines the audio and label representations for decoding.Source: §2.1 · p. 2
  • ContextNet replaces the usual audio encoder with a new CNN-based encoder within the RNN-Transducer framework.Source: §2.1 · p. 2
  • The label encoder and joint network are directly taken from the referenced LSTM-based design.Source: §2.1 · p. 2

2.2. Encoder Design

ContextNet’s encoder stacks convolution blocks that combine depthwise separable convolutions, squeeze-and-excitation global context, residual projections, and progressive temporal downsampling. The design also uses Swish activations and configurable channel scaling to balance representation power, computation, and accuracy.

Encoder architecture

  • The encoder maps an input sequence to a high-level representation by composing convolution blocks C1 through CK.Each block includes convolutional layers, normalization, activation, squeeze-and-excitation, and skip connections.Source: §2.2 · p. 2

Global context

  • Squeeze-and-excitation pools local features into a global channelwise weight, then multiplies that weight element-wise with each frame.The module uses a bottleneck of two fully connected layers and a Sigmoid output mapped to (0, 1).Source: §2.2.1 · p. 2, Figure 2 · p. 3

Convolutional layers

  • Depthwise separable convolution is used for parameter efficiency without impacting accuracy.The same kernel size is used across depthwise convolution layers.Source: §2.2.2 · p. 2

Activation and scaling

  • Swish consistently outperforms ReLU in the authors’ experiments, while α scales convolution channels and model representation power.Increasing α above 1 increases the number of convolution channels and model size.Source: §2.2.3 · p. 2, §2.2.6 · p. 3, Table 1 · p. 3

Convolution blocks

  • Each convolution block applies an SE module to the final convolution output and adds a projected skip connection.The block can contain multiple convolution layers, with strides configured for optional temporal downsampling.Source: §2.2.4 · p. 2, Figure 3 · p. 3

Temporal downsampling

  • Progressive 8× downsampling reduces computation while maintaining a good speed–accuracy trade-off.Strided convolution performs temporal downsampling, but excessive downsampling may negatively affect the decoder.Source: §2.2.5 · p. 2

3. Experiments

Experiments evaluate ContextNet on LibriSpeech through model comparisons, ablations, scaling, computation reductions, and a larger YouTube test set. Results show improved accuracy–efficiency trade-offs and benefits from global context and progressive downsampling.

Results on LibriSpeech

  • The LibriSpeech evaluation uses 970 hours of labeled speech, with small, medium, and large ContextNet configurations selected by width α ∈ {0.5, 1, 2}.An LSTM baseline is also built for reference, and language-model comparisons are included.Source: §3 · p. 3, §3.1 · p. 3, Table 2 · p. 4
  • ContextNet(M) achieves similar WER to much larger systems with only 31M parameters, while ContextNet(L) improves over the previous SOTA by 13% relatively on test-clean and 18% on test-other.ContextNet(S) also improves on similarly sized systems with or without a language model.Source: §3.1 · p. 3

Effect of Context Size

  • Adding squeeze-and-excitation modules substantially improves WER over the zero-context baseline, with larger context windows providing greater benefit.The ablation uses α = 1.25 and compares context windows of 256, 512, and 1024.Source: §3.2 · pp. 3–4

Depth, Width, Kernel Size and Downsampling

  • Globally scaling network width produces a good trade-off between model size and WER.The experiments sweep α and report both model size and LibriSpeech WER.Source: §3.3 · p. 4, Table 5 · p. 4
  • Progressive downsampling significantly reduces FLOPS, slightly improves accuracy, and makes larger convolution kernels reduce WER.The comparison varies temporal reduction and kernel sizes of 3, 5, 11, and 21.Source: §3.3 · p. 4, Table 4 · p. 4

Large Scale Experiments

  • ContextNet outperforms the previous best convolutional–bidirectional-LSTM architecture by 12% relatively on a 24.12-hour YouTube test set, using fewer parameters and FLOPS.The test set contains 117 videos with diverse and challenging acoustic environments.Source: §3.4 · p. 4, Table 6 · p. 4

4. Conclusion

The paper proposes and evaluates a CNN-based architecture for end-to-end speech recognition. It reports better LibriSpeech accuracy with fewer parameters than previously published CNN models and supportive results on a larger, more challenging dataset.

Conclusion

  • ContextNet achieves better LibriSpeech accuracy with fewer parameters than previously published CNN models.The conclusion presents this as the paper’s main benchmark finding.Source: §4 · p. 4
  • Limiting ContextNet’s network width provides a way to search for small ASR models.The conclusion connects width scaling with model-size exploration.Source: §4 · p. 4
  • An initial study on a much larger and more challenging dataset also confirms the reported findings.The conclusion characterizes this evidence as an initial study.Source: §4 · p. 4
  • Encoder FLOPS are reported as the average computation required to process one second of audio.This defines the computation metric used in the paper’s comparisons.Source: §4 · p. 4
  • The reproduced large-scale results are subject to changed training and evaluation sets, so their values differ from those previously reported.This scope boundary applies specifically to the Table 6 comparison.Source: §4 · p. 4

Related papers

Loading 2005.03191v3…