Source-linked AI summary
Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
Zhifu Gao, Shiliang Zhang, Ian McLoughlin, Zhijie Yan
TL;DR
AR decoders are accurate but inefficient because they generate tokens sequentially, while single-step NAR models can lose accuracy through output-token independence. Paraformer combines a CIF-based predictor, GLM sampler, and MWER-oriented negative sampling, attaining comparable accuracy to a state-of-the-art AR transformer with a 10x speedup on the large corpus.
Problem
AR decoding is computationally inefficient, while single-step NAR models have inferior performance because output tokens are modeled independently, creating challenges in token-length prediction and context modeling.
Method
Paraformer uses a CIF-based predictor for target length and hidden variables, a GLM sampler for semantic embeddings and token interdependence, and negative samples for MWER training.
Results
Comparable recognition accuracy to a state-of-the-art AR transformer was achieved across AISHELL-1, AISHELL-2, and a 20,000-hour task, with a 10x speedup on the large corpus.
Takeaways & Limitations
Paraformer demonstrates that single-step NAR ASR can approach AR recognition accuracy while retaining substantially faster inference on large-scale speech recognition.
Abstract
from arXiv · showhide
Transformers have recently dominated the ASR field. Although able to yield good performance, they involve an autoregressive (AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive (NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step NAR is inferior to that of AR models, especially with a large-scale corpus. There are two challenges to improving single-step NAR: Firstly to accurately predict the number of output tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both challenges, we propose a fast and accurate parallel transformer, termed Paraformer. This utilizes a continuous integrate-and-fire based predictor to predict the number of tokens and generate hidden variables. A glancing language model (GLM) sampler then generates semantic embeddings to enhance the NAR decoder's ability to model context interdependence. Finally, we design a strategy to generate negative samples for minimum word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark, and an industrial-level 20,000 hour task demonstrate that the proposed Paraformer can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
1. Introduction
Single-step NAR ASR enables parallel decoding but struggles with token-length prediction and context interdependence, especially on large corpora. Paraformer addresses both issues with a CIF-based predictor, a GLM sampler, and MWER-oriented negative sampling.
- AR decoders generate tokens sequentially, making decoding time increase with output length despite strong ASR performance.
- Single-step NAR models improve inference speed through parallel generation but make more substitution errors than AR models because of limited context interdependence.
- Paraformer uses a continuous integrate-and-fire predictor to estimate output length and generate hidden variables for single-step decoding.
- A glancing language model sampler strengthens the NAR decoder’s modeling of token interdependence, while negative samples support minimum word error rate training.
- 5.2% and 6.19% CERs on AISHELL-1 and AISHELL-2 are comparable to a state-of-the-art AR transformer, with a 10x speedup on the large corpus.
2. Methods
Paraformer combines a predictor, sampler, bidirectional decoder, and jointly trained losses to enable parallel ASR decoding. Its CIF predictor estimates output length and acoustic embeddings, while GLM sampling supplies semantic context and MWER uses negative candidates.
- 2.1. Overview: The framework contains an encoder, predictor, sampler, bidirectional decoder, and CE, MAE, and MWER loss functions trained jointly.The encoder uses SAN-M/FFN or conformer blocks, while the decoder includes self-attention, feed-forward, and cross-attention modules.
- 2.1. Overview: During inference, the sampler is inactive and the bidirectional decoder produces final predictions in one parallel pass without increasing inference complexity.Training uses two decoder forward passes, but inference uses single-step decoding.
- 2.2. Predictor: CIF accumulates predictor weights over hidden representations to produce acoustic embeddings and estimate the target token number.A dynamic threshold β is used to reduce training–inference mismatch caused by scaling weights during training.
- 2.3. Sampler: GLM training exposes the decoder to semantic context so it can learn interdependency between output tokens despite non-autoregressive decoding.The conditional independence assumption in vanilla single-step NAR otherwise produces inferior performance compared with AR models.
- 2.3. Sampler: The sampler randomly substitutes selected target embeddings into acoustic embeddings to generate semantic embeddings for the parallel decoder.The sampling amount is controlled by λ and the distance between predicted and target sequences.
- 2.4. Loss Function: Negative sampling creates multiple candidate paths for MWER training by randomly masking top-1 tokens, addressing greedy NAR decoding's single-path limitation.MWER and MAE are jointly optimized with CE loss.
3. Experiments
Experiments on public benchmarks and a 20,000-hour industrial task show that Paraformer substantially improves single-step NAR performance while retaining fast inference. Its gains are linked to GLM-based context modeling and improved CIF prediction.
- Industrial task: 13.5% and 14.6% relative improvements over vanilla NAR were achieved on the Far-field and Common tasks with the 41M model.Adding MWER training produced a further slight accuracy improvement.
- Industrial task: 13.0% and 11.1% relative improvements over vanilla NAR were reported on Far-field and Common tasks with the 63M model, alongside 10x speedup.The 63M Paraformer remained within 2.8% relative difference of the AR model.
- Error analysis: Substitution errors increased dramatically in vanilla NAR but decreased significantly in Paraformer, accounting for most of its improvement.The analysis attributes the reduction to GLM-enhanced modeling of output-token interdependence.
4. Conclusion
Paraformer is a single-step NAR ASR model that combines CIF-based token prediction, GLM-based semantic sampling, and MWER training. Across public and industrial tasks, it achieves AR-comparable performance with 10x speedup.
- 4. Conclusion: Paraformer uses a CIF-based predictor with a dynamic threshold to predict token numbers and generate hidden variables.The dynamic threshold reduces mismatch between inference and training.
- 4. Conclusion: A GLM-based sampler generates semantic embeddings that enhance the NAR decoder’s modeling of context interdependence.This targets the conditional-independence limitation of single-step NAR decoding.
- 4. Conclusion: Negative samples enable MWER training to further improve Paraformer performance.
- 4. Conclusion: Paraformer achieves performance comparable to the state-of-the-art AR transformer with 10x speedup across public and industrial ASR evaluations.The evaluations cover AISHELL-1, AISHELL-2, and a 20,000-hour corpus.