Source-linked AI summary
FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration
Kai-Tuo Xu, Feng-Long Xie, Xu Tang, Yao Hu
TL;DR
Mandarin ASR needs open, high-performing models that also accommodate different efficiency requirements and practical scenarios. FireRedASR addresses this with AED and Encoder-Adapter-LLM architectures trained on diverse speech data. The models achieve strong public-benchmark, real-world, dialect, English, and singing-recognition results while being released openly for research.
Problem
Existing large ASR models can underperform on Mandarin or remain closed-source, motivating open-source, high-performance Mandarin ASR solutions.
Method
FireRedASR provides an Encoder-Adapter-LLM variant and a compact Attention-based Encoder-Decoder variant, trained with large-scale, predominantly professionally transcribed speech data.
Results
3.05% average CER and 8.4% relative CER reduction over Seed-ASR establish FireRedASR-LLM’s public Mandarin benchmark result, alongside strong multi-source and singing recognition performance.
Takeaways & Limitations
The two variants cover accuracy-focused and resource-constrained applications, while both extend competitively beyond standard Mandarin ASR to dialects, English, and singing lyrics.
Abstract
from arXiv · showhide
We present FireRedASR, a family of large-scale automatic speech recognition (ASR) models for Mandarin, designed to meet diverse requirements in superior performance and optimal efficiency across various applications. FireRedASR comprises two variants: FireRedASR-LLM: Designed to achieve state-of-the-art (SOTA) performance and to enable seamless end-to-end speech interaction. It adopts an Encoder-Adapter-LLM framework leveraging large language model (LLM) capabilities. On public Mandarin benchmarks, FireRedASR-LLM (8.3B parameters) achieves an average Character Error Rate (CER) of 3.05%, surpassing the latest SOTA of 3.33% with an 8.4% relative CER reduction (CERR). It demonstrates superior generalization capability over industrial-grade baselines, achieving 24%-40% CERR in multi-source Mandarin ASR scenarios such as video, live, and intelligent assistant. FireRedASR-AED: Designed to balance high performance and computational efficiency and to serve as an effective speech representation module in LLM-based speech models. It utilizes an Attention-based Encoder-Decoder (AED) architecture. On public Mandarin benchmarks, FireRedASR-AED (1.1B parameters) achieves an average CER of 3.18%, slightly worse than FireRedASR-LLM but still outperforming the latest SOTA model with over 12B parameters. It offers a more compact size, making it suitable for resource-constrained applications. Moreover, both models exhibit competitive results on Chinese dialects and English speech benchmarks and excel in singing lyrics recognition. To advance research in speech processing, we release our models and inference code at https://github.com/FireRedTeam/FireRedASR.
1 Introduction
FireRedASR addresses the need for open-source, high-performance Mandarin ASR by offering two variants tailored to accuracy and efficiency. The models report strong benchmark, real-world, multilingual, dialect, and singing-recognition results.
- FireRedASR introduces two Mandarin ASR variants to address practical needs in performance, efficiency, and openness.FireRedASR-LLM targets precision when computational resources are available, while FireRedASR-AED balances performance and efficiency.
- 3.05% average CER makes FireRedASR-LLM outperform Seed-ASR’s 3.33% by an 8.4% relative reduction on public Mandarin benchmarks.
- 24%-40% relative CER reduction demonstrates strong performance across short videos, live streaming, auto-captioning, voice input, and intelligent assistants.
- Both variants show competitive results on Chinese dialects and English speech, while achieving 50%-67% CERR in singing lyrics recognition.
- The report is organized around architecture and training, comprehensive evaluations, performance factors, and conclusions.
2 FireRedASR
FireRedASR combines an Attention-based Encoder-Decoder variant with an Encoder-Adapter-LLM variant, sharing speech processing foundations but differing in token sequence modeling. The architectures use Conformer-based encoding, Transformer decoding or LLM integration, and large-scale multilingual training data.
- FireRedASR-AED: FireRedASR-AED uses an end-to-end Conformer Encoder with a Transformer Decoder for speech-feature modeling and sequence transduction.
- Training Data: Approximately 70,000 hours of predominantly professionally transcribed Mandarin speech form the main training corpus, supplemented by approximately 11,000 hours of English speech.
- Model Variants: The study evaluates multiple AED and LLM model sizes, using the largest configurations as the default variants unless otherwise specified.
- FireRedASR-LLM: FireRedASR-LLM integrates a Conformer-based audio Encoder, lightweight audio-text Adapter, and pretrained text LLM in an Encoder-Adapter-LLM architecture.
- FireRedASR-LLM: The Adapter projects encoder outputs into the LLM’s semantic space, enabling the LLM to recognize transcript content from speech.
- FireRedASR-LLM: FireRedASR-LLM initializes the LLM from Qwen2-7B-Instruct and trains the encoder and adapter fully while efficiently adapting most fixed LLM parameters with LoRA.
3 Evaluation
FireRedASR is evaluated across public Mandarin benchmarks, multi-source speech and singing scenarios, and Chinese dialect and English tests. The results show strong accuracy, scaling behavior, real-world robustness, and cross-lingual generalization for both model variants.
- 3.1 Evaluation on Public Mandarin ASR Benchmarks: FireRedASR-LLM achieves an 8.4% relative CER reduction over Seed-ASR across four public Mandarin test sets, while FireRedASR-AED also outperforms Seed-ASR with fewer parameters.FireRedASR-AED contains 1.1B parameters and FireRedASR-LLM 8.3B, compared with Seed-ASR’s nearly 2B-parameter encoder and over 10B-parameter LLM.
- 3.1 Evaluation on Public Mandarin ASR Benchmarks: Increasing model size consistently improves performance, producing 6.1%-5.6% CERR gains across FireRedASR-AED scales and a 7.3% CERR gain for FireRedASR-LLM from XS to L.FireRedASR-AED scales from 140M to 1.1B parameters, while FireRedASR-LLM scales its encoder from 86M to 710M with the LLM backbone unchanged.
- 3.2 Evaluation on Multi-source Mandarin Speech and Singing Benchmarks: FireRedASR-LLM records 3.48% CER on multi-source Mandarin speech, improving 23.7% over ProviderA-Large and 38.6% over Paraformer-Large.The evaluation covers short videos, live streaming, auto-captioning, voice input, and intelligent assistant scenarios.
- 3.2 Evaluation on Multi-source Mandarin Speech and Singing Benchmarks: FireRedASR-LLM achieves 7.05% CER in singing lyrics recognition, corresponding to CERRs of 50.2% against ProviderA-Large and 66.7% against Paraformer-Large.The baseline CERs are 14.16% for ProviderA-Large and 21.19% for Paraformer-Large.
- 3.3 Evaluation on Public Chinese Dialect and English ASR Benchmarks: Both models generalize beyond standard Mandarin, achieving 3.56% and 4.48% CER on KeSpeech and competitive WERs on LibriSpeech English tests.FireRedASR-LLM reaches 1.73% and 3.67% WER on LibriSpeech test-clean and test-other, while FireRedASR-AED reaches 1.93% and 4.44%.
4 Discussion
FireRedASR’s performance is attributed to diverse real-world training data, progressive regularization during scaling, and an efficient architecture with adapter-based adaptation.
- Three factors are credited for FireRedASR’s superior performance: high-quality diverse data, optimized training, and an efficient ASR framework.
- The training corpus uses professionally transcribed real-world audio spanning acoustic conditions, speakers, accents, and domains across tens of thousands of hours.
- Progressive Regularization Training begins without dropout or SpecAugment, then gradually strengthens regularization as overfitting emerges.
- Scaling FireRedASR-AED from 140M to 1.1B parameters identified regularization and learning rate as critical convergence factors.
- The prior Two-pass Transducer model had scaling and hyperparameter-sensitivity limitations, including prediction-network overfitting and substantial memory demands.
- A simple adapter design facilitates efficient model adaptation and research iteration.
5 Conclusion
FireRedASR introduces two Mandarin-optimized ASR models whose architectures, training strategies, and datasets support strong performance across benchmarks and practical speech tasks. The authors release model weights and inference code, while identifying broader language and task support as future work.
- FireRedASR-LLM and FireRedASR-AED achieve state-of-the-art public Mandarin benchmark results and strong performance across real-world scenarios.
- FireRedASR-AED shows that attention-based encoder-decoder architectures remain competitive, while FireRedASR-LLM demonstrates the potential of integrating LLM capabilities into ASR.
- Both models perform strongly in singing lyrics recognition and generalize robustly to Chinese dialects and English speech recognition.
- The released model weights and inference code are intended to contribute to speech processing research.
- Future work will improve performance and expand support for more languages and varied tasks.