PaperlayerView exact revision on arXiv

Source-linked AI summary

Squeezeformer: An Efficient Transformer for Automatic Speech Recognition

Sehoon Kim, Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W. Mahoney, Kurt Keutzer

arXiv:2206.00888v2eess.AScs.CLcs.SD

TL;DR

Conformer captures local and global speech features, but its quadratic attention cost and complex design choices motivate a systematic architectural re-examination. Squeezeformer redesigns its macro- and micro-architecture and consistently outperforms other ASR models under the same training settings.

  • Problem

    Conformer’s attention has quadratic complexity with input sequence length, while its multiple modules and design choices raise questions about their necessity and optimality.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Method

    Squeezeformer combines a Temporal U-Net and simpler Transformer-style blocks with simplified activations, scaled post-LayerNorm, and depthwise separable input subsampling.

    Source: §1 · pp. 2–3, §3.1.1 · p. 5, §3.1.2 · p. 6

  • Results

    Squeezeformer scales across model sizes and consistently outperforms other state-of-the-art ASR models under the same training settings.

    Source: §1 · p. 3, §5 · pp. 9–10

  • Takeaways & Limitations

    The architecture’s reported advantage spans smaller and larger models and different FLOPs regimes.

    Source: §1 · p. 3, §5 · p. 10

    Conformer WER results differ from the original paper because the original uses an RNN-T decoder, which generally yields better WER than CTC.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that captures both local and global features. However, through a series of systematic studies, we find that the Conformer architecture's design choices are not optimal. After re-examining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the-art ASR models under the same training schemes. In particular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal U-Net structure which reduces the cost of the multi-head attention modules on long sequences, and (ii) a simpler block structure of multi-head attention or convolution modules followed up by feed-forward module instead of the Macaron structure proposed in Conformer. Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the activations in the convolutional block, (ii) removes redundant Layer Normalization operations, and (iii) incorporates an efficient depthwise down-sampling layer to efficiently sub-sample the input signal. Squeezeformer achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on LibriSpeech test-other without external language models, which are 3.1%, 1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our code is open-sourced and available online.

1 Introduction

Conformer captures local and global speech features, but its quadratic attention cost and architectural complexity raise questions about whether its design choices are necessary and optimal. Squeezeformer systematically redesigns its macro- and micro-architecture to improve WER for a given computational budget.

1 Introduction

  • Squeezeformer improves Conformer by 1.40% WER at the same FLOPs on LibriSpeech test-other.Figure 1’s left plot tracks WER and FLOPs through the redesign.Source: Figure 1 · p. 2
  • Conformer’s quadratic attention cost on long sequences and complex block design motivate examining its architectural choices.The paper links that complexity to difficulty deploying Conformer efficiently on dedicated inference hardware.Source: §1 · p. 2
  • Its Temporal U-Net halves the sampling rate mid-network and restores temporal resolution with a light upsampling layer for training stability.The architecture illustration also shows the Temporal U-Net downsampling and upsampling structure.Source: §1 · p. 2, Figure 2 · p. 3
  • The block redesign replaces Conformer’s Macaron structure with MHA or convolution modules each followed by a single feed-forward module.Source: §1 · p. 2
  • Micro-architecture changes unify activations with Swish, replace redundant pre-LayerNorm with scaled post-LayerNorm, and use depthwise separable convolution for initial subsampling.The scaled post-LayerNorm’s learnable residual scaling can be merged with other layers for zero-cost inference.Source: §1 · pp. 2–3

2 Related Work

End-to-end ASR research spans model architectures and training methods, with encoder design central to acoustic representation and overall performance. Squeezeformer builds on Conformer while addressing attention cost and redesigning its architecture rather than its training strategy.

2 Related Work

  • ASR models typically use an encoder to extract acoustic features and a decoder to convert them into text; encoder architecture shapes representational power and overall performance.Source: §2 · p. 3
  • CNNs can miss global context, Transformers model long-range dependencies, and Conformer combines convolution with attention to capture local and global interactions.Source: §2 · p. 3
  • Conformer’s attention remains costly at long sequence lengths; Efficient-Conformer uses progressive downsampling and grouped attention, while Squeezeformer adds U-Net-inspired upsampling with skip connections.The paper identifies the upsampling mechanism as critical for training stability and overall performance.Source: §2 · pp. 3–4
  • Squeezeformer’s architecture changes are evaluated through successive redesigns reporting test-clean and test-other WER, parameters, and FLOPs for a 30s input.The caption says WER improves consistently with each design change.Source: Table 1 · p. 4
  • Self-supervised training methods can be applied independently of the underlying architecture, so training strategies are not this work’s focus.Source: §2 · p. 4

3 Architecture Design

Squeezeformer reexamines Conformer’s macro- and micro-architecture to reduce computational cost while improving or preserving recognition performance. Its design combines temporal resampling, simpler block ordering, unified activations, streamlined normalization, and depthwise subsampling.

3 Architecture Design

  • The design study compares test-other WER with Conformer-CTC-M and measures FLOPs for 30-second audio as a hardware-agnostic efficiency proxy.The authors also measure end-to-end throughput because FLOPs may not linearly indicate runtime efficiency.Source: §3 · p. 4

3.1.1 Temporal U-Net Architecture

  • 20% lower total FLOPs and 0.62% better test-other WER (7.90% to 7.28%) accompany Temporal U-Net downsampling after block 7 and skip-connected restoration to 40ms.Attention FLOPs fall 2.31× for 30s and 2.53× for 60s audio; downsampling alone causes unstable, diverging training.Source: §3.1.1 · p. 5

3.1.2 Transformer-Style Block

  • Replacing the Macaron FMCF sequence with MHA-then-feed-forward or convolution-then-feed-forward blocks improves test-other WER by 0.16%, from 7.28% to 7.12%, without changing FLOPs.The authors treat large-kernel convolution as local attention and avoid placing it back-to-back with MHA.Source: §3.1.2 · p. 6

3.2.1 Unified Activations

  • Replacing the convolutional GLU with Swish unifies activations across the model, simplifying the architecture without noticeable WER or FLOPs changes.The convolution-module expansion rate is retained.Source: §3.2.1 · p. 6

3.2.2 Simplified Layer Normalizations

  • A learnable scaling layer replaces preLN, yielding a postLN-only model and 0.20% better test-other WER, from 7.09% to 6.89%.The learned parameters can be merged into the subsequent linear layer, giving zero inference cost; simply removing preLN or postLN destabilizes training.Source: §3.2.2 · pp. 6–7

3.2.3 Depthwise Separable Subsampling

  • Replacing the second subsampling convolution with depthwise separable convolution saves 22% of baseline FLOPs without a test-other WER drop and improves inference throughput up to 1.34×.The change also improves test-clean WER by 0.06%; the module accounts for up to 28% of Conformer-CTC-M FLOPs on 30-second input.Source: §3.2.3 · p. 7

3 Architecture Design

  • 1.01% lower test-other WER (7.90% to 6.89%) with 40% less FLOPs; scaling to similar FLOPs yields another 0.39% gain, to 6.50%.The initial baseline is Conformer-CTC-M; the scaled architecture is Squeezeformer-M.Source: §3.2.3 · p. 7

4 Results

Across LibriSpeech CTC comparisons, Squeezeformer improves on Conformer-CTC and other models. Ablations show that Temporal U-Net, LayerNorm, and convolution-activation choices affect performance and training stability.

4.2 Main Results

  • Squeezeformer-L improves WER by 0.33/0.58 over Conformer-CTC-L at the same FLOPs count, achieving 2.47/5.97 and the state-of-the-art result.Source: §4.2 · p. 9

4.1 Experiment Setup

  • Comparisons exclude external language models and use LibriSpeech-960hr training for 500 epochs.The paper states that omitting external LMs isolates the representation power of the architectures.Source: §4.1 · p. 8

4.3 Ablation Studies

  • Removing the Temporal U-Net skip connection worsens WER by 0.35/0.87, while removing upsampling prevents convergence.The authors attribute the skip-connection result to the importance of high-resolution information from early layers.Source: Table 4 · p. 9, §4.3 · p. 9
  • Without learned scaling, PostLN-only and PreLN-only schemes worsen WER by 3.17/7.49 and 0.59/1.76, respectively, and both fail to converge.The authors report the best WER before divergence and say learned scaling supports training stability and better WER.Source: Table 4 · p. 9, §4.3 · p. 9
  • Dropping GLU without replacing it with Swish worsens WER by 0.10/0.22.This is the reported convolution-module activation ablation.Source: Table 4 · p. 9, §4.3 · p. 9

4.2 Main Results

  • Across FLOPs regimes, Squeezeformer outperforms other ASR models, including Efficient Conformer by 0.79/1.99 WER at the same FLOPs (2.79/6.89 vs. 3.58/8.88).The paper summarizes the overall comparisons in Fig. 1 (Right).Source: §4.2 · p. 9

5 Conclusions

Squeezeformer combines a Temporal U-Net and simpler MF/CF blocks with micro-architecture changes to reduce compute and improve accuracy. Across model sizes and FLOPs regimes, it consistently surpasses prior architectures under the same training settings.

5 Conclusions

  • Its Temporal U-Net downsamples audio in the network’s second half to reduce neighboring-feature redundancy and save compute.The authors identify high temporal redundancy in learned representations, especially deeper in the network.Source: §5 · p. 9
  • Squeezeformer’s MF/CF block structure simplifies the architecture and improves performance relative to the Conformer design.The conclusion describes the blocks as similar to the standard Transformer style.Source: §5 · p. 9
  • Squeezeformer scales across model sizes and FLOPs regimes, surpassing prior architectures when trained under the same settings.The authors report extensive testing across different model sizes and FLOPs regimes.Source: §5 · p. 10
  • The micro-architecture simplifies activations, replaces redundant LayerNorms with scaled postLN, and reduces initial subsampling cost with depthwise separable convolution.The scaled postLN is described as more efficient and more accurate.Source: §5 · p. 10

A.1 Training Details

Training uses an extended Noam Annealing schedule with warmup, a peak-rate hold, and decay. The experiments also specify model-size-dependent peak rates and augmentation settings.

A.1 Training Details

  • The learning-rate schedule extends Noam Annealing with a peak-rate hold and decay rate d.Noam annealing is the special case d = 0.5 and Tpeak = 0.Source: §A.1 · p. 14
  • Experiments use 20 warmup epochs, 160 epochs at peak rate, and decay with d = 1.These schedule values are fixed throughout the experiments; peak rates are 2e-3, 1.5e-3, and {1, 0.5}e-3 for small, medium, and large variants, respectively.Source: §A.1 · p. 14
  • Training uses SentencePiece with a 128-token vocabulary and SpecAugment with frequency and model-dependent time masks.Two frequency masks use [0, 27]; time-mask counts are 5 for the specified small variants, 7 for Squeezeformer-M, or 10 for large variants, with masking ratio [0, 0.05].Source: §A.1 · p. 14

A.2 Evaluation Details

Final models are evaluated on clean and other datasets with CTC greedy decoding. Throughput is measured on a single A100 GPU using 30-second audio inputs.

A.2 Evaluation Details

  • Final models are evaluated on both clean and other datasets using CTC greedy decoding.Source: §A.2 · p. 14
  • Throughput is measured on a single NVIDIA Tesla A100 using 30-second audio inputs as an indicator of hardware performance.Source: §A.2 · p. 14
  • The throughput test uses CUDA 11.5 and TensorFlow 2.5 with the largest batch size that saturates the machine.Source: §A.2 · p. 14

A.3 Transferrability to TIMIT

The paper tests transfer from LibriSpeech to unseen TIMIT with and without fine-tuning. It reports that Squeezeformer generally outperforms Conformer-CTC at smaller or equal FLOPs and parameter counts in both settings.

A.3 Transferrability to TIMIT

  • On unseen TIMIT, Squeezeformer generally outperforms Conformer-CTC at smaller or equal FLOPs and parameter counts, with and without fine-tuning.Source: §A.3 · p. 14
  • The transfer comparison reports test-split WER and includes parameter counts and FLOPs for models trained on LibriSpeech, with and without fine-tuning.Source: Table A.1 · p. 14, §A.3 · p. 14
Loading 2206.00888v2…