Source-linked AI summary
Fun-ASR Technical Report
Keyu An, Yanni Chen, Zhigao Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Bo Gong, Xiangang Li, Yabin Li, Ying Liu, Xiang Lv, Yunjie Ji, Yiheng Jiang, Bin Ma, Haoneng Luo, Chongjia Ni, Zexu Pan, Yiping Peng, Zhendong Peng, Peiyao Wang, Hao Wang, Haoxu Wang, Wen Wang, Wupeng Wang, Yuzhong Wu, Biao Tian, Zhentao Tan, Nan Yang, Bin Yuan, Jieping Ye, Jixing Yu, Qinglin Zhang, Kun Zou, Han Zhao, Shengkui Zhao, Jingren Zhou, Yanqiao Zhu
TL;DR
Open-source benchmark results may not reliably reflect real-world ASR performance. Fun-ASR combines data and model scaling with LLM integration and production-oriented optimizations. It achieves state-of-the-art performance on both open-source benchmarks and industry datasets.
Problem
Open-source test performance may not reliably reflect real-world ASR capabilities, leaving a gap in how well benchmark results indicate industry performance.
Source: §6.2.1 · p. 9Method
Fun-ASR combines data and model scaling with LLM integration in a large-scale ASR system optimized for practical deployment.
Source: §1 · p. 2, §8 · p. 13Results
Fun-ASR achieves state-of-the-art performance on both open-source benchmarks and industry datasets.
Source: §6.2.1 · p. 9Takeaways & Limitations
ASR evaluation should include real-world industry datasets because open-source test performance may not reliably reflect real-world capabilities.
Source: §6.2.1 · p. 9Fun-ASR is primarily optimized for Chinese and English, so support for other languages remains limited.
Source: §7 · pp. 12–13
Abstract
from arXiv · showhide
In recent years, automatic speech recognition (ASR) has witnessed transformative advancements driven by three complementary paradigms: data scaling, model size scaling, and deep integration with large language models (LLMs). However, LLMs are prone to hallucination, which can significantly degrade user experience in real-world ASR applications. In this paper, we present Fun-ASR, a large-scale, LLM-based ASR system that synergistically combines massive data, large model capacity, LLM integration, and reinforcement learning to achieve state-of-the-art performance across diverse and complex speech recognition scenarios. Moreover, Fun-ASR is specifically optimized for practical deployment, with enhancements in streaming capability, noise robustness, code-switching, hotword customization, and satisfying other real-world application requirements. Experimental results show that while most LLM-based ASR systems achieve strong performance on open-source benchmarks, they often underperform on real industry evaluation sets. Thanks to production-oriented optimizations, Fun-ASR achieves state-of-the-art performance on real application datasets, demonstrating its effectiveness and robustness in practical settings. The code and models are accessible at https://github.com/FunAudioLLM/Fun-ASR .
1 Introduction
Fun-ASR is a large-scale LLM-based ASR system that combines data and model scaling with LLM integration, targeting high recognition accuracy and practical deployment across diverse speech scenarios.
1 Introduction
- Fun-ASR is designed to harness the combined benefits of data scaling, model-size scaling, and innovative integration with large language models.The report presents it as a large-scale LLM-based system trained on large-scale data.Source: §1 · p. 2
- Fun-ASR is reported to achieve state-of-the-art recognition accuracy across diverse linguistic and acoustic domains, substantially outperforming prior small-scale models and leading industry systems on challenging benchmarks.The report also claims superior practical performance on real-world usage scenarios.Source: §1 · pp. 2–3
- Production-oriented features include low-latency real-time streaming, multi-stage noise-robustness improvements, Chinese-English code-switching, and customizable hotword recognition.Hotwords can be defined for domain-specific terms, including in healthcare, enterprise, and automotive settings.Source: §1 · pp. 2–3
- Fun-ASR is evaluated with standardized benchmarks and distinct real-world test sets designed to simulate different application contexts.The report says the results show that the system bridges academic research and commercial production readiness.Source: §1 · p. 3
2 Model Architecture
Fun-ASR combines a four-part speech-to-LLM recognition architecture with two model sizes designed for different accuracy and inference-efficiency needs.
2 Model Architecture
- The architecture comprises a transformer audio encoder, a two-layer transformer audio adaptor, a CTC decoder for initial hypotheses and hotword customization, and an LLM-based decoder conditioned on audio and CTC predictions.The CTC decoder is built on the audio encoder; the adaptor connects the encoder output to the LLM.Source: §2 · p. 3
- Fun-ASR uses a 0.7B-parameter audio encoder and 7B-parameter LLM decoder for accuracy, while Fun-ASR-nano uses 0.2B and 0.6B parameters, respectively, for low-resource efficiency.The two model sizes address differing computational-resource constraints and inference-efficiency requirements.Source: §2 · p. 3
3 Data
Fun-ASR is pretrained on tens of millions of hours of broad-domain audio and then fine-tuned on millions of hours of varied supervised and augmented data.
3.1 Pre-taining Data
- Pretraining combines tens of millions of hours of unlabeled and labeled audio, with labeled data processed through VAD, multi-ASR pseudo-labeling, and ITN.Unlabeled audio spans real-world domains; Chinese and English are the primary labeled languages. Pseudo-label systems include Paraformer-V2, Whisper, and SenseVoice.Source: §3.1 · p. 4
3.2 Supervised Fine-tuning Data
- SFT uses millions of hours of human-transcribed, pseudo-labeled, noise-related, TTS-generated, simulated-streaming, and hotword-customized data.TTS data are generated using CosyVoice3.Source: §3.2 · p. 4
4 Training
Fun-ASR is trained through staged audio-encoder pre-training, supervised and contextual fine-tuning, and reinforcement learning. The training combines scalable speech data with mechanisms for contextual recognition, efficient RL, and reducing recognition errors and hallucinations.
4.1 Pre-training of Audio Encoder
- Audio-encoder pre-training combines unlabeled Best-RQ learning initialized from Qwen3 text-LLM layers with supervised AED training on labeled ASR data.The Best-RQ initialization accelerates convergence and improves speech representations; the trained AED encoder initializes the downstream LLM-ASR encoder.Source: §4.1 · pp. 4–5
4.2 Supervised Fine-tuning
- Supervised fine-tuning progresses from adaptor alignment with frozen encoder and LLM to encoder/adaptor training, LoRA adaptation, and joint fine-tuning on high-quality data.A final stage freezes the encoder and trains a CTC decoder to produce a greedy-search hypothesis for retrieval-augmented generation.Source: §4.2 · p. 5
4.3 Contextual Supervised Fine-tuning
- Contextual fine-tuning uses samples up to five minutes long, previous-segment transcripts as prompts, and over 50K hours of constructed contextual SFT data.The construction pipeline extracts keywords, synthesizes and keyword-filters relevant contexts, then mixes in five randomly sampled irrelevant contexts to discourage overreliance.Source: §4.3 · p. 5
4.4.1 The RL Framework for Large Audio-Language Models
- FunRL alternates GPU use among parallel audio encoding, SGLang hypothesis rollout, and FSDP policy optimization, synchronizing the updated policy to keep training on-policy.On 8 A100 GPUs, a step for approximately one hour of input audio takes about 54.6 seconds (RTF approximately 0.015); device-switching overhead is less than 6%.Source: §4.4.1 · p. 6, Figure 4 · p. 6
4.4.2 GRPO-based RL for ASR
- Adding reward rules for keyword accuracy, hallucination suppression, and language matching improves user experience and reduces WER on hard cases.The reward design also includes ASR accuracy; it penalizes hallucinated spans and outputs that do not match the source language.Source: §4.4.2 · p. 7
4.4.3 Constructing the RL Training Data
- RL training uses 100K samples across five subsets covering hard cases, long-duration speech, hallucinations, keywords or hotwords, and regular ASR data.The hard-case subset selects disagreements between the contextual-SFT base model and three external ASR systems; the full RL training completes within one day on 8 A100 GPUs.Source: §4.4.3 · p. 7
5 Production-oriented Optimization
Fun-ASR adapts training and inference for production needs, improving streaming and noisy-condition performance while adding multilingual, code-switching, hotword-customization, and hallucination-mitigation capabilities.
5.1 Streaming Ability
- Simulated streaming training uses incremental, chunked inputs exposing only past context, and combining this data with offline training improves streaming-decoding performance.The data is created by transforming a subset of the offline training corpus to reduce the mismatch between training and inference.Source: §5.1 · p. 8
5.2 Noise Robust Training
- Approximately 13% average relative performance improvement was achieved on the complex-noise evaluation set through large-scale noisy-data augmentation.The approach combines approximately 110K hours of simulated noisy speech with online noise mixing for 30% of training speech.Source: §5.2 · p. 8
5.3 Multilingual ASR
- The latest multilingual Fun-ASR release supports 31 languages and streaming for some; its approximately 500K-hour dataset balances languages by upsampling non-Chinese-English data.The system also supports prompt-based context bias for industrial speech recognition applications.Source: §5.3 · p. 8
5.4 Code-switching
- For Chinese-English code-switching, Fun-ASR synthesizes training speech from over 40K English keywords, Qwen3-generated mixed-language text, and multi-voice text-to-speech (Yang et al., 2025a).The keywords cover domains including technology, education, finance, and sports.Source: §5.4 · p. 8
5.5 Hotword Customization
- At inference, Fun-ASR retrieves hotword candidates by phoneme- or word-piece-level edit distance from CTC hypotheses, then feeds them with audio and CTC predictions to the LLM.Hotwords are represented as phoneme sequences for Chinese or word-piece sequences for other languages.Source: §5.5 · pp. 8–9
5.6 Hallucination Mitigation
- Zero-padding audio before noise addition creates pure-noise segments, teaching Fun-ASR to recognize noise-only inputs and reducing hallucinated text during silence, interruptions, or noisy conditions.The paper reports improved robustness, accuracy, and stability across diverse acoustic conditions.Source: §5.6 · p. 9
6 Evaluation · 6.1 Evaluation Setting
Fun-ASR and Fun-ASR-ML are evaluated on established open-source benchmarks and real-world industry sets, alongside an independently collected video test set and recordings spanning diverse noise conditions.
6.1 Evaluation Setting
- The evaluation covers both Fun-ASR and Fun-ASR-ML on open-source ASR benchmarks and real-world industry evaluation sets.Source: §6.1 · p. 9
- Open-source testing uses AIShell-1, AIShell-2 (Du et al., 2018), Librispeech, Fleurs (Conneau et al., 2022), WeNetSpeech (Zhang et al., 2022), and Gigaspeech2.Source: §6.1 · p. 9
- Because established public datasets risk training-data leakage, the authors build an independent test set from newly uploaded YouTube and Bilibili videos posted after June 30th, manually transcribing them.Source: §6.1 · p. 9
- Noise-robustness evaluation uses real-world audio recordings captured in canteens, dinners, meetings, offices, outdoors, parks, shops, streets, subways, supermarkets, and walk-streets.Source: §6.1 · p. 9
- The real-world recordings are categorized by acoustic conditions and topic to assess performance across diverse and challenging scenarios.Source: §6.1 · p. 9
- The setting combines long-publicly-available benchmarks with newly collected video speech and varied environmental recordings for evaluation.Source: §6.1 · p. 9
6.2 Evaluation results
Fun-ASR achieves state-of-the-art performance on both open-source benchmarks and industry datasets. The evaluation also examines streaming, noise robustness, code-switching, hotword customization, multilingual recognition, and reinforcement learning.
6.2.2 Streaming ASR Performance
- In streaming evaluation, Fun-ASR performs better than Seed-ASR across different test sets and scenarios.The evaluation uses the same test set as the offline speech-recognition evaluation.Source: §6.2.2 · p. 9, Table 3 · p. 10
6.2.3 Evaluation on Noise Robustness
- Noise-robust training brings over 30% relative improvement in dinner and supermarket settings, while reinforcement learning further enhances robustness.Under complex acoustic conditions, LLM-based systems tend to hallucinate outputs.Source: §6.2.3 · pp. 9, 11, Table 4 · p. 11
6.2.1 Overall Results
- Fun-ASR achieves state-of-the-art performance on open-source benchmarks and industry datasets.Fun-ASR-nano, with 0.8B parameters, also outperforms open-source models and performs closely to Seed-ASR.Source: §6.2.1 · p. 9, Table 1 · p. 10, Table 2 · p. 10
6.2.4 Code-switching Evaluation
- Code-switching evaluation uses two test sets drawn from a public Chinese-English speech dataset and in-house recordings, with results reported in Table 5.The passage provides the evaluation setup but does not state the results.Source: §6.2.4 · p. 11, Table 5 · p. 11
6.2.5 Evaluation on Hotword Customization
- Hotword customization raises recall above 0.97 on most topics and from 0.75 to 1.0 for names, indicating keyword activation beyond contextual prompting.The evaluation covers specialized topics including biology, math, religion, food, astronomy, chemistry, philosophy, and physics.Source: §6.2.5 · pp. 11–12, Table 6 · p. 11
6.2.6 Multilingual ASR Results
- Fun-ASR-ML performs better or comparably to Kimi-Audio on Chinese and English open-source and in-house tests, and achieves state-of-the-art results against other listed multilingual models.Fun-ASR-ML-Nano performs slightly worse than Fun-ASR-ML but better than the other compared models.Source: §6.2.6 · p. 12, Table 7 · p. 12
6.2.7 Effect of Reinforcement Learning
- Reinforcement learning brings approximately 4.1% relative improvement offline and 9.2% streaming, with larger offline gains on noisy or complex audio and greater gains in streaming.It reduces insertion and deletion errors, possibly by reducing early termination or premature prediction before full pronunciation.Source: §6.2.7 · p. 12
- For hotword integration, reinforcement learning improves accuracy and recall across most test sets and lowers overall WER even where philosophy or religion accuracy or recall dips.Selecting keywords from actual transcriptions rather than prompts helps Fun-ASR recognize domain-specific terms not explicitly included in the hotword list.Source: Table 6 · p. 11, §6.2.7 · p. 12
7 Limitations and Future Plans
Fun-ASR remains optimized chiefly for Chinese and English, has limited robustness on long recordings without external VAD, and does not support far-field or multi-channel audio; future work will address these limitations.
7 Limitations and Future Plans
- Fun-ASR is primarily optimized for Chinese and English, particularly streaming and hotword customization, so support for other languages remains limited.Source: §7 · p. 12
- Without external VAD, the constrained context window makes long recordings difficult to handle robustly, and the current release lacks far-field and multi-channel support.The authors plan to address these limitations in future work.Source: §7 · p. 13
8 Conclusion
Fun-ASR combines large-scale data, model capacity, LLM integration, and refinement learning to achieve state-of-the-art performance across diverse, challenging scenarios. Its production-oriented optimizations support practical deployment, where it demonstrates superior accuracy on application datasets.
8 Conclusion
- Fun-ASR demonstrates superior accuracy on practical application datasets, where many LLM-based ASR systems underperform despite strong results on open-source benchmarks.The conclusion attributes this performance to production-oriented design and optimization.Source: §8 · p. 13
- Fun-ASR combines massive data, extensive model capacity, LLM integration, and refinement learning to achieve state-of-the-art performance across diverse, challenging scenarios.Source: §8 · p. 13
- For practical deployment, Fun-ASR incorporates enhanced streaming, noise robustness, code-switching support, and customizable hotwords.Source: §8 · p. 13