Source-linked AI summary
Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, Boris Ginsburg
TL;DR
Conformer’s quadratic self-attention limits efficiency and long-sequence speech processing. The paper redesigns the architecture as Fast Conformer with deeper downsampling and optional local attention plus a global token. Fast Conformer reduces compute, accelerates inference, scales to 1B parameters, and supports long-form transcription while retaining competitive or strong accuracy across speech tasks.
Problem
Conformer’s self-attention has quadratic time and memory complexity with sequence length, limiting compute efficiency and maximum audio duration.
Method
Fast Conformer redesigns downsampling to 8x and optionally replaces global self-attention post-training with limited-context attention plus a single global context token.
Results
Fast Conformer uses 2.9x less compute, is 2.8× faster than an equivalent Conformer, and enables single-pass transcription of 11-hour audio with limited-context attention.
Takeaways & Limitations
The architecture supports efficient ASR, speech translation, and spoken language understanding, with scaling to 1B parameters without changing the core architecture.
Takeaways & Limitations
For Conformer-CTC, 8x subsampling requires replacing character tokenization with Sentencepiece BPE because character tokenization may violate the CTC length condition.
Abstract
from arXiv · showhide
Conformer-based models have become the dominant end-to-end architecture for speech processing tasks. With the objective of enhancing the conformer architecture for efficient training and inference, we carefully redesigned Conformer with a novel downsampling schema. The proposed model, named Fast Conformer(FC), is 2.8x faster than the original Conformer, supports scaling to Billion parameters without any changes to the core architecture and also achieves state-of-the-art accuracy on Automatic Speech Recognition benchmarks. To enable transcription of long-form speech up to 11 hours, we replaced global attention with limited context attention post-training, while also improving accuracy through fine-tuning with the addition of a global token. Fast Conformer, when combined with a Transformer decoder also outperforms the original Conformer in accuracy and in speed for Speech Translation and Spoken Language Understanding.
1. INTRODUCTION
Fast Conformer redesigns Conformer to reduce computation and memory costs while addressing self-attention’s sequence-length limitations. It combines deeper downsampling with optional limited-context attention and reports speed, scaling, and long-form speech benefits.
- Motivation: Conformer’s self-attention provides global context but has quadratic time and memory complexity with input length.This limits computation, memory, and maximum processable audio duration.
- Architecture: Fast Conformer increases initial downsampling to 8x by redesigning the downsampling schema and subsampling block.The redesign is presented as a core architectural change.
- Long-form processing: Post-training, Fast Conformer can replace self-attention with local attention and a global context token to scale linearly with sequence length.The approach is evaluated using a Longformer-based attention mechanism.
- Results: The encoder uses 2.9x fewer multiply-add operations with global attention and is 2.8× faster than an equivalent Conformer during inference.The model also maintains highly competitive WER on ASR benchmarks.
- Results: Limited-context attention achieves good long-form ASR results while making inference more than 3× faster.Fast Conformer is also evaluated on speech translation and spoken language understanding, with strong or state-of-the-art task results and speedups.
2. FAST CONFORMER ARCHITECTURE
Fast Conformer combines 8x early downsampling with efficient convolutional subsampling and limited-context attention. These changes target encoder efficiency and long-form transcription while preserving accuracy across the described evaluations.
- Downsampling schema: The original Conformer subsamples from 10 ms to 40 ms, reducing later attention costs, while its subsampling module exceeds 20% of Large-model computation time.The Large Conformer contains 120 M parameters.
- Downsampling schema: Progressive 8x downsampling reduces sequence length but creates a 16x attention-cost imbalance between early and final layers.EfficientConformer, Squeezeformer, and Uconv-Conformer use related progressive or temporal encoder strategies.
- Downsampling schema: RNNT avoids the CTC length constraint, allowing 8x downsampling, while large sub-word vocabularies reduce autoregressive decoder and joint-network calls.For Conformer-CTC, the paper switches from character tokenization to Sentencepiece BPE with vocabularies of 128 to 1024 tokens.
- Downsampling schema: Fast Conformer applies 8x downsampling at the encoder start, reducing subsequent attention-layer compute by 4x.The redesign also uses depthwise separable convolutions, 256 filters, and kernel size 9.
- Long-form audio transcription: Limited-context attention with a global token extends single-GPU processing from 15 minutes to 675 minutes, a 45x increase.The method uses a fixed-size local window, bidirectional interaction with one global token, and overlapping chunks.
3. EXPERIMENTS
Experiments evaluate Fast Conformer on ASR, speech translation, spoken language understanding, and long-form audio. Across these tasks, the model combines competitive accuracy with faster or more scalable inference, while limited-context attention improves long-form performance.
- Automatic Speech Recognition: Fast Conformer is evaluated on LibriSpeech, Multilingual LibriSpeech, Mozilla Common Voice, and Wall Street Journal ASR benchmarks using Large CTC and RNNT configurations.The experiments include models trained on LibriSpeech alone and on a 25K-hour English speech set.
- Spoken Language Understanding: Fast Conformer attains very high accuracy on SLURP, significantly surpassing ESPNet-SLU and SpeechBrain while decoding 10(%) faster than Conformer.The speedup is modest because the autoregressive Transformer decoder dominates execution cost after encoder downsampling.
- Long-form audio transcription: A single global attention token is combined with limited-context attention to support efficient long-form transcription after fine-tuning.The training procedure fine-tunes a full-context checkpoint with limited-context attention for 10k steps.
- Long-form audio transcription: Fast Conformer with limited-context attention significantly outperforms full-context Fast Conformer and Conformer on TED-LIUM v3 and Earnings-21 long-form ASR benchmarks.The limited-context model processes full audio in one forward pass, whereas the comparison models use 20-second buffers.
4. SCALING FAST CONFORMER MODEL
Fast Conformer scales from Large to Extra Extra Large without changing its core architecture, using larger model dimensions and additional encoder layers. Larger models benefit from self-supervised pretraining and more training data, improving benchmark performance and noise robustness.
- Model scaling: Fast Conformer models are constructed at Large, Extra Large, and Extra Extra Large sizes while retaining the same core architecture from L to XXL.Scaling increases the hidden dimension and layer counts rather than changing the conformer blocks or relative attention.
- Model scaling: XL and XXL models achieve superior results within fewer training steps than L models on the evaluated ASR benchmark sets.The comparison is reported for models trained on 25k hours of ASR Set.
- Scaling Dataset: Adding 40,000 hours to the training data improves accuracy and noise robustness for Fast Conformer XL and XXL models.The augmented set is called ASR Set ++ and is evaluated on HF-audio leaderboard test sets.
- Scaling Dataset: The FC-XXL RNNT model trained on 25k hours achieves similar state-of-the-art performance to the cited Conformer model on LS-test other and best performance on other benchmark datasets.The paper reports successful training of a 1B-parameter model and improved results with supplementary data.
5. CONCLUSIONS
Fast Conformer redesigns Conformer with novel downsampling to reduce computation while preserving roughly comparable WER. Limited-context attention and a global token extend long-form transcription, while scaling and larger datasets support stronger accuracy and noise robustness.
- Conclusions: Fast Conformer uses 2.9x less compute while delivering roughly the same WER as the original Conformer.The efficiency comes from the redesigned architecture and downsampling schema.
- Conclusions: Replacing global attention with local attention enables transcription of an 11-hour audio segment in a single forward pass.Adding a single global attention token further improves long-form audio results.
- Conclusions: Fast Conformer evaluations on speech translation and spoken language understanding show strong accuracy with significant speedups in encoder computation.The conclusion summarizes gains across additional speech-processing tasks.
- Conclusions: The architecture scales to 1B parameters without changes to its core architecture, and larger training datasets further improve accuracy and noise robustness.The paper links this scaling result to training on larger datasets.