Source-linked AI summary
Zipformer: A faster and better encoder for automatic speech recognition
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, Daniel Povey
TL;DR
Zipformer addresses the need for faster and more memory-efficient ASR encoders while preserving recognition quality. It combines multirate architecture, redesigned attention reuse, BiasNorm, new activations, and ScaledAdam, and achieves state-of-the-art results across three datasets with over 50% faster inference and less GPU memory.
Problem
ASR needs encoder architectures that improve recognition performance while reducing computational cost and memory use.
Method
Zipformer combines a U-Net-like multirate encoder, attention-weight reuse in redesigned blocks, BiasNorm, Swoosh activations, and the parameter-scale-aware ScaledAdam optimizer.
Results
Zipformer achieves state-of-the-art results on LibriSpeech, Aishell-1, and WenetSpeech, with over 50% faster inference and less GPU memory than previous studies.
Takeaways & Limitations
The experiments demonstrate the effectiveness of Zipformer’s modeling and optimization innovations for efficient ASR encoding.
Abstract
from arXiv · showhide
The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster convergence and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https://github.com/k2-fsa/icefall.
1 INTRODUCTION
Zipformer is introduced as a faster, more memory-efficient, and better-performing ASR encoder, combining architectural and optimization changes. Experiments across three datasets report state-of-the-art results, faster convergence, and over 50% faster inference with less GPU memory.
- Contributions: Zipformer combines a U-Net-like multirate encoder, redesigned blocks with reused attention weights, BiasNorm, Swoosh activations, and ScaledAdam.The encoder downsamples sequences at different frame rates, while ScaledAdam scales updates by parameter scale and explicitly learns that scale.
- Experimental evaluation: Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech demonstrate the effectiveness of the proposed modeling and optimization innovations.
- Experimental evaluation: Zipformer achieves state-of-the-art results on LibriSpeech, Aishell-1, and WenetSpeech.The paper also reports results comparable to those in the Conformer paper on LibriSpeech, which the authors describe as difficult for others to reproduce.
- Efficiency: 50% faster inference than previous studies is achieved while requiring less GPU memory.The authors also report faster convergence during training.
2 RELATED WORK
Prior ASR research combines convolutional and Transformer components and explores efficiency-oriented Conformer variants and alternative end-to-end frameworks. Zipformer extends temporal downsampling with stack-specific ratios while adding optimization changes centered on ScaledAdam.
- Architecture research: Conformer combines CNN and Transformer advantages, while subsequent work modifies Conformer to reduce computational cost and improve recognition performance.
- Architecture research: Squeezeformer uses a temporal U-Net with middle modules at half frame rates, while Branchformer uses parallel branches for different context ranges.
- Zipformer’s distinction: Zipformer uses different downsampling ratios across encoder stacks and more aggressive ratios in middle stacks than Squeezeformer’s fixed ratio.
- Zipformer’s distinction: Unlike architecture-only approaches, Zipformer also introduces ScaledAdam, which is shown to improve convergence in experiments.
- End-to-end frameworks: End-to-end ASR frameworks include CTC, CTC/AED, and neural transducers, with pruning and batch-splitting methods proposed to reduce training cost and memory use.
3 METHOD
Zipformer uses a U-Net-like encoder that represents speech at multiple temporal resolutions rather than processing every layer at a fixed frame rate. Its architecture begins by reducing 100Hz acoustic features to a 50Hz embedding sequence before cascaded encoder stacks.
- Downsampled encoder structure: Zipformer replaces Conformer’s fixed 25Hz processing with a U-Net-like structure that learns temporal representations at different resolutions.
- Downsampled encoder structure: Conv-Embed reduces 100Hz acoustic features by a factor of 2, producing a 50Hz embedding sequence.
- Downsampled encoder structure: The resulting sequence is passed through six cascaded stacks that learn temporal representations at progressively varied frame rates.
+BiasNorm
Zipformer combines multi-resolution temporal modeling with redesigned blocks, BiasNorm, new activations, and ScaledAdam to improve efficiency, convergence, and representation learning.
- Multi-resolution encoder: Zipformer uses downsampled and upsampled stacks to model temporal representations at lower frame rates before producing the encoder output.Downsampled stacks apply temporal modeling after sequence-length reduction and restore the sequence through upsampling and bypass modules.
- Zipformer block: The Zipformer block reuses attention weights across Non-Linear Attention and Self-Attention modules to reduce time and memory costs.An MHAW module computes shared attention weights for the NLA module and two SA modules.
- BiasNorm: BiasNorm replaces LayerNorm with bias subtraction, RMS normalization, and a positive scalar scale to retain length information and avoid gradient-sign oscillation.The learnable bias can preserve vector-length information, while exp(γ) remains positive.
- Activation functions: SwooshR and SwooshL are proposed as replacements for Swish, with offsets selected to control their zero-crossing behavior.SwooshL is described as roughly a right-shifted version of SwooshR and remains non-monotonic with a lower bound.
- ScaledAdam: ScaledAdam scales updates by parameter magnitude and explicitly learns parameter scale to keep relative changes similar across tensors.The method addresses Adam’s failure to account for parameter scale and its difficulty directly learning tensor norms.
4 EXPERIMENTS
Experiments compare Zipformer variants with state-of-the-art ASR models across LibriSpeech, Aishell-1, and WenetSpeech, then isolate the contributions of its architectural and optimization choices. Zipformer generally combines strong recognition accuracy with lower computational and memory costs, while ablations support the proposed downsampling, block, normalization, activation, and optimizer designs.
- Experimental setup: Experiments evaluate Zipformer at small, medium, and large scales on LibriSpeech, Aishell-1, and WenetSpeech.The variants are Zipformer-S, Zipformer-M, and Zipformer-L.
- LibriSpeech: Zipformer-L achieves LibriSpeech WERs of 2.00%/4.38% with sufficient computing resources, approaching the original Conformer-L result.The model was trained on 8 80G NVIDIA Tesla A100 GPUs for 170 epochs.
- Efficiency: Zipformer provides a better performance-efficiency trade-off than competing models, with Zipformer-L requiring much less inference time and memory.Inference comparisons use batches of 30-second audios with batch size 30 on a single NVIDIA Tesla V100 GPU.
- WenetSpeech: On WenetSpeech, Zipformer-M and Zipformer-L outperform all other models on Test Net and Test Meeting, while Zipformer-S uses one third as many parameters as Conformer models.Zipformer-S also yields lower WERs than the Conformer models.
- Encoder structure: Removing temporal downsampling increases WER on both test sets, supporting more efficient modeling with fewer parameters rather than information loss.The ablation replaces the downsampled structure with constant-rate processing and produces a larger model.
- Block structure: Replacing Zipformer blocks with stacked Conformer blocks reduces test-other WER by 0.16%, while removing NLA or Bypass degrades performance.Sharing attention weights after removing NLA adds parameters and slows inference without improving WER.
- Normalization layer: Replacing BiasNorm with LayerNorm causes WER drops of 0.08% on test-clean and 0.18% on test-other.The result supports BiasNorm’s ability to retain some length information during normalization.
- Activation function: Using only SwooshR causes WER drops of 0.11% on test-clean and 0.42% on test-other, while Swish degrades performance further.These results support using SwooshL for normally-off modules and SwooshR over Swish.
5 CONCLUSION
Zipformer is an efficient ASR encoder combining multi-resolution temporal modeling, redesigned blocks, BiasNorm, Swoosh activations, and ScaledAdam. Experiments on three datasets demonstrate its effectiveness.
- Zipformer uses a U-Net-like encoder with lower frame rates, redesigned blocks that reuse attention weights, BiasNorm, and SwooshR/SwooshL activations.
- ScaledAdam enables faster convergence and better performance.
- Experiments on LibriSpeech, Aishell-1, and WenetSpeech demonstrate the effectiveness of Zipformer.
A.1.1 SCALEDADAM ALGORITHM.
ScaledAdam updates parameters using gradient moments while also tracking parameter-scale gradients and the RMS scale of the current parameters.
- Recommended defaults are β1 = 0.9, β2 = 0.98, η = 0.1, and ϵ = 10^-8.
- ScaledAdam initializes gradient moments, parameter-scale moments, and the RMS parameter scale before iteratively updating parameters.
- At each step, ScaledAdam computes the parameter gradient and parameter-scale gradient, then updates their first and second moments.
- The algorithm computes parameter changes from bias-corrected gradient moments and separately updates parameter scale using its own moments.
A.1.2 COMPARISON BETWEEN SCALEDADAM AND ADAM.
Figure A.2 compares ScaledAdam and Adam using averaged LibriSpeech WER across epochs and learning rate across steps.
- Figure A.2 compares averaged WER on LibriSpeech test-clean and test-other across epochs for ScaledAdam and Adam.
- The figure also compares learning rates across training steps for ScaledAdam and Adam.
A.2 ACTIVATION FUNCTIONS
This section presents activation functions and training-related components used in Zipformer, including Swoosh variants, Balancer, Whitener, and their experimental configurations.
- Activation functions: Figure A.3 presents the Swish, SwooshR, and SwooshL activation functions.
- Activation constraints: Balancer and Whitener regularize activations through additional backward-pass losses while leaving the forward pass unchanged.
- Activation constraints: Balancer addresses excessively large or small activations and dead neurons among observed channel-wise failure modes.
- Activation constraints: Whitener computes a covariance matrix from mean-centered features and uses an auxiliary loss to measure feature whitening.
- Activation constraints: Removing Whitener changes LibriSpeech test-clean and test-other WER by 0.04% and 0.24%, respectively, in the reported ablation.
- Experimental setup: The reported experiments include LibriSpeech training configurations, CTC and CTC/AED comparisons, and encoder parameter and FLOP measurements.
- Experimental results: Zipformer-M outperforms Squeezeformer-ML on both LibriSpeech test sets with about half the parameters, while Zipformer-L surpasses Squeezeformer-L by 0.27% on test-other.