Source-linked AI summary
FunASR: A Fundamental End-to-End Speech Recognition Toolkit
Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, Shiliang Zhang
TL;DR
Speech-recognition research needs tools that connect academic experimentation with industrial deployment. FunASR addresses this gap with industrially trained models, configurable Paraformer extensions, and deployment components; its reported evaluations show strong accuracy and efficiency, including a 12x Paraformer inference speedup.
Problem
Existing speech-recognition research and industrial deployment require an open-source toolkit that provides both industrially trained models and practical application support.
Method
FunASR combines industrially trained models with training, fine-tuning, and deployment workflows, while extending Paraformer with timestamp prediction and hotword customization and releasing FSMN-VAD and CT-Transformer.
Results
Paraformer achieves a 12x GPU inference speedup over autoregressive models and Paraformer-large reports CERs of 1.95%, 2.85%, and 6.97% on three Mandarin evaluation tasks.
Takeaways & Limitations
FunASR makes industrial speech-recognition models and supporting modules openly available for researchers and real-world application deployment.
Abstract
from arXiv · showhide
This paper introduces FunASR, an open-source speech recognition toolkit designed to bridge the gap between academic research and industrial applications. FunASR offers models trained on large-scale industrial corpora and the ability to deploy them in applications. The toolkit's flagship model, Paraformer, is a non-autoregressive end-to-end speech recognition model that has been trained on a manually annotated Mandarin speech recognition dataset that contains 60,000 hours of speech. To improve the performance of Paraformer, we have added timestamp prediction and hotword customization capabilities to the standard Paraformer backbone. In addition, to facilitate model deployment, we have open-sourced a voice activity detection model based on the Feedforward Sequential Memory Network (FSMN-VAD) and a text post-processing punctuation model based on the controllable time-delay Transformer (CT-Transformer), both of which were trained on industrial corpora. These functional modules provide a solid foundation for building high-precision long audio speech recognition services. Compared to other models trained on open datasets, Paraformer demonstrates superior performance.
1. Introduction
FunASR is an open-source toolkit designed to connect academic speech-recognition research with industrial applications. It combines industrially trained models with deployment-oriented components and extends Paraformer with timestamp and hotword capabilities.
- FunASR is an open-source speech recognition toolkit designed to bridge academic research and industrial applications.
- Paraformer is a non-autoregressive end-to-end model trained on 60,000 hours of manually annotated Mandarin speech.
- The toolkit provides training and fine-tuning workflows for Transformer, Conformer, and Paraformer models across several speech datasets.
- FunASR adds timestamp prediction and hotword customization to Paraformer while releasing FSMN-VAD and CT-Transformer for speech-recognition services.
- The paper reports Mandarin experiments, while FunASR also supports English, French, German, Spanish, Russian, Japanese, and Korean.
2. Overview of FunASR
FunASR organizes model management, experimentation, and deployment into connected components. Its academic and industrial pipelines support training, inference, fine-tuning, and runtime deployment across multiple hardware platforms.
- ModelScope manages FunASR models, including Paraformer, FSMN-VAD, and CT-Transformer.
- Academic pipeline: The academic run.sh pipeline covers data preparation, feature extraction, dictionary generation, training, inference, and scoring.
- Industrial pipeline: The industrial pipeline separates inference through infer.sh from fine-tuning through finetune.sh.
- Deployment runtime: FunASR supports deployment on CPU, GPU, Android, and iOS through Libtorch, ONNX, and TensorRT backends.
- Deployment runtime: AMP quantization accelerates the inference runtime for application deployment.
3. Main Modules of FunASR
FunASR extends Paraformer with timestamp prediction and hotword customization, and provides VAD and real-time text postprocessing modules for speech recognition applications. Experiments report competitive timestamp, customization, VAD, text postprocessing, and recognition performance across supported datasets.
- Paraformer: Paraformer is a single-step non-autoregressive model whose predictor generates acoustic embeddings and whose training-time sampler incorporates target embeddings to capture token interdependence.The sampler is inactive during inference, when acoustic embeddings are used directly by the decoder.
- Timestamp Predictor: Timestamp prediction redesigns the Paraformer predictor with transposed convolution and LSTM upsampling, then derives token durations and silence from CIF weights.FunASR also releases TP-Aligner, a smaller encoder-based force-alignment-like model that takes speech and transcription as input.
- Timestamp Predictor: Paraformer-TP outperforms the Kaldi force-alignment system on AISHELL and remains within 10ms of the hybrid system on industrial data.The one-pass approach is reported to reduce computation and time overhead for commercial usage.
- Hotword Customization: Contextual Paraformer adds a hotword embedder and decoder multi-head attention to customize recognition using named entities.The hotword embedding is combined with the decoder representation through attention and dimensionality reduction before subsequent processing.
- Hotword Customization: Approximately 58% F1 improvement on the AISHELL-1 named entity subtest and an average 10% improvement on industrial customization tasks are reported for hotwords.The evaluation uses CER and F1-score, with test sets spanning AISHELL entity utterances and industrial AI and common domains.
- VAD and Text Postprocessing: FSMN-VAD filters invalid voice to improve CER, while CT-Transformer performs real-time punctuation and disfluency removal with competitive F1 and faster inference.CT-Transformer uses controllable time delay, fast decoding, and history pruning; VAD evaluation covers meeting and video data.
4. Experiments
Experiments show that Paraformer matches or slightly exceeds Conformer recognition accuracy while providing substantially faster inference. Fine-tuning further improves performance on general and industrial-domain speech, and AMP quantization accelerates runtime.
- ASR evaluation: Paraformer slightly outperforms Conformer without language-model decoding in recognition accuracy.The comparison removes Conformer’s language model and joint-CTC decoding for fairness.
- Runtime performance: 12x faster inference gives Paraformer a substantial speed advantage over autoregressive models.The speedup is measured by GPU RTF with batch size 1.
- ASR evaluation: 1.95%, 2.85%, and 6.97% CERs are reported on AISHELL, AISHELL-2, and WenetSpeech, respectively.Paraformer-large was trained on a 60,000-hour Mandarin industrial corpus.
- Fine-tuning: 7.4% and 8.7% relative improvements are achieved on the AISHELL development and test tasks after fine-tuning.Fine-tuning on AISHELL and 200 hours of industrial-domain data improves both general and domain-specific recognition.
- Fine-tuning: Domain-keyword recall increases from 76.7% to 96.8%, while CER decreases from 11.25% to 10.10% after logistics-domain fine-tuning.The results cover both short and long logistics test sets.
- Runtime performance: 40% faster inference is achieved with AMP quantization without significant recognition degradation.The runtime evaluation uses Paraformer-large on an Intel Xeon CPU with one thread.
5. Conclusions
FunASR bridges academic and industrial speech recognition by openly providing industrially trained models and deployment support. Its released components include Paraformer-large, FSMN-VAD, and CT-Transformer.
- Conclusions: FunASR openly provides industrially trained speech recognition models and supports deployment in real-world applications.The toolkit includes Paraformer-large, FSMN-VAD, and CT-Transformer.