Source-linked AI summary
Fun-ASR Technical Report
Keyu An, Yanni Chen, Zhigao Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Bo Gong, Xiangang Li, Yabin Li, Ying Liu, Xiang Lv, Yunjie Ji, Yiheng Jiang, Bin Ma, Haoneng Luo, Chongjia Ni, Zexu Pan, Yiping Peng, Zhendong Peng, Peiyao Wang, Hao Wang, Haoxu Wang, Wen Wang, Wupeng Wang, Yuzhong Wu, Biao Tian, Zhentao Tan, Nan Yang, Bin Yuan, Jieping Ye, Jixing Yu, Qinglin Zhang, Kun Zou, Han Zhao, Shengkui Zhao, Jingren Zhou, Yanqiao Zhu
TL;DR
LLM-based ASR can hallucinate and may perform poorly on real industry data despite strong open-source benchmark results. Fun-ASR combines large-scale data, model capacity, LLM integration, reinforcement learning, and deployment-focused optimizations, achieving state-of-the-art results across open-source and industry evaluations while retaining limitations in language coverage and long or multi-channel audio.
Problem
LLM-based ASR hallucination can degrade user experience, and benchmark performance may not reflect real-world industrial performance.
Method
Fun-ASR combines massive data, large model capacity, LLM integration, reinforcement learning, and optimizations for streaming, noise, code-switching, and hotword customization.
Results
Fun-ASR achieves state-of-the-art performance on open-source benchmarks and industry datasets, including over 30% relative improvement from noise-robust training in dinner and supermarket settings.
Takeaways & Limitations
Production-oriented design helps Fun-ASR bridge academic benchmark performance and practical speech-recognition deployment across diverse scenarios.
Takeaways & Limitations
Fun-ASR remains primarily optimized for Chinese and English, has constrained effective context without external VAD, and does not support far-field or multi-channel audio.
Abstract
from arXiv · showhide
In recent years, automatic speech recognition (ASR) has witnessed transformative advancements driven by three complementary paradigms: data scaling, model size scaling, and deep integration with large language models (LLMs). However, LLMs are prone to hallucination, which can significantly degrade user experience in real-world ASR applications. In this paper, we present Fun-ASR, a large-scale, LLM-based ASR system that synergistically combines massive data, large model capacity, LLM integration, and reinforcement learning to achieve state-of-the-art performance across diverse and complex speech recognition scenarios. Moreover, Fun-ASR is specifically optimized for practical deployment, with enhancements in streaming capability, noise robustness, code-switching, hotword customization, and satisfying other real-world application requirements. Experimental results show that while most LLM-based ASR systems achieve strong performance on open-source benchmarks, they often underperform on real industry evaluation sets. Thanks to production-oriented optimizations, Fun-ASR achieves state-of-the-art performance on real application datasets, demonstrating its effectiveness and robustness in practical settings. The code and models are accessible at https://github.com/FunAudioLLM/Fun-ASR .
1 Introduction
Fun-ASR builds on data scaling, model scaling, and LLM integration to target accurate, practical speech recognition across diverse conditions. It combines these advances with production-oriented optimizations for real-world deployment.
- Fun-ASR combines data scaling, model size scaling, and LLM integration in a large-scale ASR system.
- The system targets state-of-the-art recognition across diverse linguistic and acoustic domains, outperforming earlier small-scale and leading industry ASR systems on multiple benchmarks.
- Production optimizations address streaming, noise robustness, Chinese-English code-switching, and customizable domain hotwords.
- The evaluation combines standardized benchmarks with real-world usage scenarios to assess practical performance and user experience.
- The report covers architecture, data, training, production optimizations, experiments, limitations, and future plans.
2 Model Architecture
Fun-ASR uses an audio encoder, adaptor, CTC decoder, and LLM-based decoder to convert speech into context-aware transcriptions. Two model sizes trade recognition accuracy against inference efficiency.
- Fun-ASR comprises an audio encoder, audio adaptor, CTC decoder, and LLM-based decoder.The CTC decoder supplies an initial recognition hypothesis for hotword customization, while the LLM-based decoder uses audio and CTC outputs.
- Fun-ASR uses a 0.7B-parameter audio encoder and 7B-parameter LLM decoder for high recognition accuracy.
- Fun-ASR-nano uses a 0.2B-parameter audio encoder and 0.6B-parameter LLM decoder to balance accuracy and efficiency in low-resource settings.
3 Data
Fun-ASR is trained on massive, diverse audio data spanning unlabeled and labeled speech from many real-world domains. Supervised fine-tuning adds curated, noisy, synthetic, streaming, and hotword-customized examples.
- Pre-training uses tens of millions of audio hours, combining unlabeled and labeled audio-text data across diverse real-world domains.
- The labeled-data pipeline uses voice activity detection, pseudo-labels from multiple ASR systems, and follow-up processing.
- Supervised fine-tuning uses millions of hours spanning human and pseudo-labeled data, environmental noise, TTS, simulated streaming, noise augmentation, and hotword customization.
4 Training
Fun-ASR trains its audio-language model through staged encoder pre-training, supervised alignment and adaptation, contextual fine-tuning, and reinforcement learning. FunRL supports efficient large audio-language-model training, while rule-based rewards target accuracy, keywords, noise robustness, hallucination, and language consistency.
- 4.1 Pre-training of Audio Encoder: The audio encoder is pretrained with self-supervised Best-RQ learning and supervised attention-based encoder-decoder training before downstream LLM-ASR integration.AED initialization provides a strong starting point for joint audio-language training and accelerates convergence.
- 4.2 Supervised Fine-tuning: Supervised fine-tuning progressively aligns the adaptor, trains the encoder, adapts the LLM with LoRA, performs high-quality joint tuning, and adds a CTC decoder for retrieval-augmented context.The stages use approximately 200K, 10M, 20K, and 3M hours in successive training phases.
- 4.3 Contextual Supervised Fine-tuning: Contextual fine-tuning uses up to five-minute segments, previous-segment transcripts, extracted keywords, synthesized relevant context, and sampled irrelevant context.The construction process creates over 50K hours of contextual supervised fine-tuning data while reducing dependence on context.
- 4.4.1 The RL Framework for Large Audio-Language Models: FunRL coordinates the audio encoder, rollout, and policy modules through alternating GPU use for large audio-language-model reinforcement learning.On 8 A100 GPUs, one step for about one hour of audio takes 54.6 seconds, with RTF approximately 0.015 and device switching under 6% of total computation.
- 4.4.2 GRPO-based RL for ASR: Enhanced GRPO uses group-level rewards, normalized advantages, a clipped objective, and KL regularization, adding rewards for ASR accuracy, keywords, noise robustness, hallucination suppression, and language match.The added rules improve user experience and reduce WER on hard cases.
5 Production-oriented Optimization
Fun-ASR is optimized for deployment through streaming-aware training, multilingual and code-switching support, hotword customization, and hallucination mitigation. These adaptations target practical conditions such as limited language resources, noisy environments, and domain-specific terminology.
- Streaming ASR: Streaming-style training exposes the model to incremental chunked inputs with only past context, reducing training–inference mismatch during streaming decoding.The streaming data is combined with offline training data during fine-tuning.
- Multilingual ASR: The multilingual Fun-ASR-ML model supports 31 languages, streaming recognition for some languages, prompt-based context biasing, and approximately 500k hours of multilingual audio.Chinese and English data are downsampled while other languages are upsampled to balance the training distribution.
- Code-switching: Code-switching training synthesizes Chinese-English data from more than 40K English keywords or phrases spanning domains including technology, education, finance, and sports.
- Hotword customization: RAG-based hotword customization retrieves phoneme- or word-piece-matched candidates from a hotword vocabulary and supplies them to the LLM with the audio and CTC prediction.
- Hallucination mitigation: To reduce hallucinations in silence, interruptions, and noise, Fun-ASR adds zero-padding before noise augmentation to create pure-noise segments for training.
6.1 Evaluation Setting
Fun-ASR is evaluated on established open-source benchmarks and newly collected industry-oriented evaluation sets. The setting also accounts for potential data leakage in long-public benchmark datasets.
- Evaluation datasets: Evaluation covers AIShell-1, AIShell-2, Librispeech, Fleurs, WeNetSpeech, and Gigaspeech2 test sets, alongside newly collected evaluation data.The open-source datasets have been publicly available for a long time, increasing the risk of training-data leakage.
6.2 Evaluation results
Fun-ASR achieves strong results across open-source, industry, streaming, multilingual, noisy, code-switched, and hotword-focused evaluations. The reported findings also show measurable benefits from noise-robust training and reinforcement learning.
- Overall results: Fun-ASR achieves state-of-the-art performance on both open-source benchmarks and industry datasets, while Fun-ASR-nano closely approaches Seed-ASR with 0.8B parameters.The evaluation uses training data collected before June 30th 2025 to avoid leakage.
- Streaming results: Fun-ASR outperforms Seed-ASR across different test sets and scenarios under streaming decoding.
- Noise robustness: Noise-robust training brings over 30% relative improvement in dinner and supermarket environments, while reinforcement learning further enhances noise robustness.LLM-based ASR systems tend to generate hallucinated outputs under complex acoustic conditions.
- Hotword customization: Hotword customization raises recall above 0.97 on most topics and increases name-topic recall from 0.75 to 1.0.The results indicate that customization can actively promote target keywords rather than merely provide contextual information.
- Multilingual results: Fun-ASR-ML achieves better or comparable results to Kimi-Audio and state-of-the-art performance against Whisper large v3, dolphin-small, and seamless-m4t large v2.Fun-ASR-ML-Nano degrades somewhat relative to Fun-ASR-ML but still outperforms the other compared models.
- Reinforcement learning: Reinforcement learning yields approximately 4.1% relative improvement offline and 9.2% in streaming, with larger offline gains on noisy and complex audio.The reported error reductions include both insertion and deletion errors.
- Hotword reinforcement learning: Reinforcement learning improves hotword accuracy and recall across most test sets, although some philosophy and religion domains show slightly lower values while overall WER decreases.The training selects keywords from actual transcriptions rather than input prompts, supporting recognition of terms absent from the hotword list.
7 Limitations and Future Plans
Fun-ASR remains constrained in language coverage, long-duration recording robustness, and audio-channel support despite strong evaluation results.
- Support for languages beyond Chinese and English remains limited because the model is primarily optimized for Chinese and English streaming and hotword customization.
- Without an external voice activity detection module, Fun-ASR struggles to handle long-duration recordings robustly because its effective context window is constrained.
- The current release does not support far-field or multi-channel audio, which the authors identify as a target for future work.
8 Conclusion
Fun-ASR combines large-scale data, model capacity, LLM integration, and refinement learning for challenging ASR scenarios. Production-oriented optimizations yield superior accuracy on practical application datasets, where many LLM-based systems underperform.
- Fun-ASR achieves state-of-the-art performance across diverse and challenging speech recognition scenarios by combining massive data, extensive model capacity, LLM integration, and refinement learning.
- Fun-ASR improves practical deployment through enhanced streaming, noise robustness, code-switching, and customizable hotword support.
- Many LLM-based ASR systems perform well on open-source benchmarks but underperform on real-world industrial evaluation sets, whereas Fun-ASR demonstrates superior accuracy on practical application datasets.
9 Authors (in alphabetical order of last name)
The author list is presented in alphabetical order of last name and spans three consecutive groups of contributors.
- The first listed author group runs from Biao Tian through Yanqiao Zhu.
- The second listed author group runs from Yiheng Jiang through Yuzhong Wu.
- The third listed author group runs from Keyu An through Yunjie Ji.