PaperlayerView exact revision on arXiv

Source-linked AI summary

FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System

Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu, Feng-Long Xie, Xu Tang, Yao Hu

arXiv:2603.10420v1eess.AScs.SD

TL;DR

Practical ASR deployment requires more than a standalone recognizer because real-world audio includes segmentation, language and dialect routing, and readable-output challenges. FireRedASR2S integrates ASR, VAD, LID, and punctuation prediction in a modular unified pipeline, reporting improved ASR accuracy and broader Chinese dialect coverage with strong evaluated performance across its modules.

  • Problem

    Real-world audio requires VAD, LID, and punctuation alongside ASR, while heterogeneous assemblies can have inconsistent interfaces, limited reproducibility, and complex error propagation.

  • Method

    FireRedASR2S integrates ASR, VAD, multilingual and dialect LID, and punctuation prediction into a unified pipeline whose modules can also be deployed independently.

  • Results

    FireRedASR2S reports improved ASR accuracy and broader Chinese dialect coverage, with strong VAD, LID, and punctuation performance on evaluated benchmarks.

  • Takeaways & Limitations

    The system provides an open-source, modular pathway from waveform to structured transcription for diverse acoustic, multilingual, and dialect scenarios.

  • Takeaways & Limitations

    ASR training excludes Chinese dialect or accented speech from MagicData, and AUC-ROC is unavailable for two VAD baselines lacking continuous posterior probabilities.

Abstract

from arXiv · show

We present FireRedASR2S, a state-of-the-art industrial-grade all-in-one automatic speech recognition (ASR) system. It integrates four modules in a unified pipeline: ASR, Voice Activity Detection (VAD), Spoken Language Identification (LID), and Punctuation Prediction (Punc). All modules achieve SOTA performance on the evaluated benchmarks: FireRedASR2: An ASR module with two variants, FireRedASR2-LLM (8B+ parameters) and FireRedASR2-AED (1B+ parameters), supporting speech and singing transcription for Mandarin, Chinese dialects and accents, English, and code-switching. Compared to FireRedASR, FireRedASR2 delivers improved recognition accuracy and broader dialect and accent coverage. FireRedASR2-LLM achieves 2.89% average CER on 4 public Mandarin benchmarks and 11.55% on 19 public Chinese dialects and accents benchmarks, outperforming competitive baselines including Doubao-ASR, Qwen3-ASR, and Fun-ASR. FireRedVAD: An ultra-lightweight module (0.6M parameters) based on the Deep Feedforward Sequential Memory Network (DFSMN), supporting streaming VAD, non-streaming VAD, and multi-label VAD (mVAD). On the FLEURS-VAD-102 benchmark, it achieves 97.57% frame-level F1 and 99.60% AUC-ROC, outperforming Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD. FireRedLID: An Encoder-Decoder LID module supporting 100+ languages and 20+ Chinese dialects and accents. On FLEURS (82 languages), it achieves 97.18% utterance-level accuracy, outperforming Whisper and SpeechBrain. FireRedPunc: A BERT-style punctuation prediction module for Chinese and English. On multi-domain benchmarks, it achieves 78.90% average F1, outperforming FunASR-Punc (62.77%). To advance research in speech processing, we release model weights and code at https://github.com/FireRedTeam/FireRedASR2S.

1 Introduction

FireRedASR2S addresses deployment gaps in standalone and heterogeneous ASR pipelines by integrating ASR, VAD, LID, and punctuation prediction in an open-source modular system. Its contributions target broader recognition coverage, robust segmentation, multilingual and dialect identification, and readable outputs.

  • Motivation: Real-world audio requires VAD, LID, and punctuation prediction alongside ASR to handle silence, music, singing, multilingual speech, and Chinese dialects.Heterogeneous module assemblies can also suffer from inconsistent interfaces, limited reproducibility, and complex error propagation.
  • System contribution: FireRedASR2S integrates FireRedASR2, FireRedVAD, FireRedLID, and FireRedPunc into a unified pipeline while permitting independent module deployment.The system is released as an open-source all-in-one ASR pipeline with unified interfaces and modular deployment.
  • ASR: FireRedASR2 improves recognition accuracy and expands Chinese dialect coverage, achieving strong results on 24 public test sets.The update builds on FireRedASR and scales supervised training data to approximately 200k hours with broader domain, language, and dialect diversity.
  • VAD: FireRedVAD provides multilingual segmentation trained on high-quality human-annotated acoustic event data rather than forced-alignment-derived supervision.This design targets reliable segmentation under diverse acoustic conditions.
  • Punctuation: FireRedPunc achieves strong results on multi-domain Chinese and English punctuation benchmarks.It is included to improve the readability of recognized outputs.

2 FireRedASR2S: System Overview

FireRedASR2S processes waveforms through VAD, LID, ASR, and punctuation stages to produce structured transcription outputs. Its modular design supports both end-to-end transcription and standalone deployment of individual components.

  • Pipeline: The unified pipeline sequentially applies FireRedVAD, FireRedLID, FireRedASR2, and FireRedPunc to diverse acoustic and multilingual inputs.The default configuration handles speech, singing, music, non-speech, and Chinese dialect scenarios.
  • Pipeline: FireRedVAD detects voice segments and filters non-voice regions, FireRedLID predicts language or dialect labels, and FireRedASR2 transcribes segments with confidence scores.The AED variant can additionally provide token- and word-level timestamps.
  • Pipeline: FireRedPunc restores punctuation for ASR output to improve readability and downstream usability.Punctuation is the final processing stage in the waveform-to-transcription pipeline.
  • Structured outputs: FireRedASR2S produces structured outputs including punctuated text, timestamps, confidence scores, language labels, and VAD segmentation results.Sentence-level segments contain start/end timestamps, recognized text, ASR confidence, and optional language-label confidence.
  • Modularity: Each module can operate independently for tasks such as VAD-only segmentation, LID-only routing, ASR on pre-segmented audio, or punctuation on plain text.Independent deployment enables flexible deployment and iteration of individual components.

3 FireRedASR2: Automatic Speech Recognition

FireRedASR2 retains the prior FireRedASR designs while expanding supervised training data and adding timestamp generation to the AED variant. It provides AED and LLM-based ASR architectures for multilingual, dialect, code-switching, speech, and singing transcription.

  • ASR variants: FireRedASR2 comprises AED and Encoder-Adapter-LLM variants with different token-sequence modeling approaches.The AED variant additionally supports token- and word-level timestamps and utterance-level confidence scores.
  • Training data: Approximately 200k hours of supervised data expand coverage to Mandarin, English, Chinese dialects, code-switching, speech, singing, and non-speech audio.This replaces the approximately 70k-hour corpus used for FireRedASR.
  • Timestamp generation: FireRedASR2-AED adds a post-hoc CTC branch that enables alignment-based timestamp prediction while preserving the base AED model’s recognition accuracy.The encoder and decoder are frozen while only the CTC branch is optimized.
  • Timestamp generation: CTC forced alignment converts encoder-frame logits and AED-decoded tokens into token-level timestamps, which are post-processed into word-level timestamps.BPE tokens are merged into words, while each Chinese character token is treated as a word unit.
  • Summary of differences: FireRedASR2 retains FireRedASR’s model designs while improving generalization through a larger, more diverse corpus and enabling AED timestamp generation.These are the principal updates summarized for FireRedASR2.

4 FireRedVAD: Voice Activity Detection

FireRedVAD is a modular DFSMN-based family for offline, streaming, and multi-label acoustic-event segmentation. It uses direct human annotations and a compact footprint to handle speech, singing, music, silence, and noise.

  • Tasks: FireRedVAD includes non-streaming VAD, streaming VAD, and non-streaming multi-label VAD models.The models support offline segmentation, low-latency online segmentation, and acoustic event recognition.
  • Tasks and label definitions: mVAD independently predicts speech, singing, and music posteriors and converts each event stream into timestamped segments.Event-wise post-processing produces per-event timestamps.
  • Tasks and label definitions: VAD classifies frames as voice or non-voice, defining voice as speech or singing and non-voice as music, silence, or noise.This label definition matches typical ASR use in user-generated-content scenarios.
  • Supervision: FireRedVAD is trained on thousands of hours of human-annotated acoustic-event data with direct time boundaries for speech, singing, and music.VAD and mVAD use task-specific labels, objectives, and post-processing criteria.
  • Inference and post-processing: DFSMN models produce frame posteriors that are smoothed, thresholded, and stabilized with duration constraints before segment output.Streaming inference incrementally updates cached history without reprocessing past audio.
  • Model architecture: ∼0.6M parameters per model provide an approximately 2.2 MB float32 footprint for low-overhead cloud and edge deployment.The compact design targets massive concurrent processing and low-resource deployment.

5 FireRedLID: Hierarchical Spoken Language and Dialect Identification

FireRedLID uses hierarchical sequence generation to identify languages and, for Chinese utterances, dialects. Its compact two-token formulation supports broad multilingual and dialect coverage while conditioning dialect prediction on the language decision.

  • Motivation: LID errors can propagate to downstream ASR decoding and punctuation prediction, making robust language and dialect routing important in multilingual processing.The module is designed for diverse acoustic conditions and fine-grained dialect identification.
  • Architecture: FireRedLID uses a Conformer Encoder and Transformer Decoder to generate a short token sequence representing each utterance’s LID result.The encoder produces acoustic representations and the decoder generates the label sequence.
  • Hierarchical labels: The model predicts a language token first and then a Chinese dialect token when the language is Chinese.Non-Chinese utterances usually terminate after the language token.
  • Hierarchical labels: A maximum decoding length of 2 keeps hierarchical LID labels compact and reduces ambiguity relative to a flat label space.Beam search decoding has negligible overhead because the output is at most two tokens.
  • Label coverage: FireRedLID supports 100+ languages and 20+ Chinese dialects grouped into 8 geographical or linguistic clusters.Compact language codes represent supported languages and dialect clusters.

6 FireRedPunc: Punctuation Prediction

FireRedPunc is a BERT-style token classifier that restores punctuation in Chinese and English ASR transcripts. It is trained on large, diverse text corpora and inserts predicted marks into the original token sequence.

  • Architecture: FireRedPunc predicts a punctuation tag after each input token using a BERT-style encoder and token-level classification head.The encoder is initialized from a pre-trained LERT checkpoint and fine-tuned for punctuation prediction.
  • Punctuation set: The model uses a compact 5-way punctuation set consisting of no punctuation and four marks: , . ? !.Chinese text uses corresponding full-width punctuation marks.
  • Training data: Training data contains approximately 18.57B Chinese characters and 2.20B English words across diverse domains and writing styles.The data is intended to improve generalization to ASR-like inputs.
  • Inference: At inference, predicted punctuation tags are inserted into the original ASR text sequence to generate punctuated output.The same tokenizer as the pre-trained LERT encoder is used.

7 Evaluation

The evaluation reports module-level performance for ASR, VAD, LID, and punctuation prediction on public or internal benchmarks, with each module evaluated independently. FireRedASR2S achieves strong results across these tasks, including high ASR, VAD, LID, and punctuation performance.

  • Evaluation protocol: Each module is evaluated independently, and the ASR evaluation covers 24 public test sets spanning Mandarin, Chinese dialects, and singing lyrics recognition.The ASR comparison includes Doubao-ASR, Qwen3-ASR, Fun-ASR, and Fun-ASR-Nano, while API results may change over time.
  • ASR: 9.67% average CER on Avg-All-24 is achieved by FireRedASR2-LLM, alongside 2.89% on Avg-Mandarin-4 and 11.55% on Avg-Dialect-19.FireRedASR2-LLM also achieves 1.12% CER on the opencpop singing lyrics test set.
  • VAD: 99.60% AUC-ROC and 97.57% F1 are achieved by FireRedVAD on FLEURS-VAD-102, outperforming all compared baselines.The module also reports 2.69% FAR and 3.62% MR at the evaluated operating point.
  • LID: 97.18% utterance-level accuracy is achieved by FireRedLID on FLEURS, while the combined Chinese dialect benchmark reaches 88.47% accuracy.FireRedLID achieves the best accuracy among compared systems on CommonVoice and outperforms Whisper and SpeechBrain on FLEURS.
  • Punctuation prediction: 78.90% average F1 is achieved by FireRedPunc across Chinese and English punctuation benchmarks, with 82.96% on Chinese and 74.83% on English.FireRedPunc consistently outperforms FunASR-Punc on both benchmarks.

8 Discussion

The discussion emphasizes modular system design, broader supervised-data coverage for ASR, human-annotated event supervision for VAD, and hierarchical labels for LID. These choices are presented as supporting deployment, robustness, reproducibility, and dialect generalization.

  • System design: Modular interfaces let FireRedASR2S simplify deployment and maintenance while allowing independent iteration of VAD, LID, ASR, and punctuation prediction.The design also improves reproducibility compared with ad-hoc integration of heterogeneous modules.
  • ASR data scaling: Approximately 200k hours of broader supervised training data accompany FireRedASR2's largely preserved architectures and consistent improvements on Mandarin benchmarks.The discussion links expanded data diversity with recognition accuracy and generalization to diverse Chinese dialect scenarios.
  • VAD supervision: Thousands of hours of human-annotated acoustic event data provide FireRedVAD supervision instead of ASR forced-alignment-derived labels.The paper states that this explicit event supervision improves robustness under diverse acoustic conditions and supports VAD and mVAD.
  • LID design: Hierarchical LID predicts language first and dialect conditioned on Chinese, matching the label structure while reducing ambiguity compared with a flat label space.The formulation keeps inference efficient and supports multilingual and fine-grained Chinese dialect identification.

9 Conclusion

FireRedASR2S is presented as an industrial-grade, all-in-one speech recognition system integrating ASR, VAD, LID, and punctuation prediction. Its modules provide improved ASR coverage, robust segmentation, hierarchical identification, and punctuation restoration, with released weights and code supporting research and deployment.

  • Conclusion: FireRedASR2S integrates ASR, VAD, LID, and punctuation prediction in an industrial-grade all-in-one speech recognition system.FireRedASR2 offers LLM-based and AED-based variants for maximum accuracy or a balanced accuracy-efficiency trade-off.
  • Conclusion: FireRedASR2 improves recognition accuracy and expands coverage to broader Chinese dialects, while FireRedVAD, FireRedLID, and FireRedPunc achieve strong task-specific performance.The system releases model weights and code for research and practical deployment.

A Detailed ASR Results on Public Test Sets

The appendix provides per-test-set CER results for all 24 public ASR test sets and includes an open-source Fun-ASR-Nano checkpoint for completeness. Comparisons should account for changing API baselines and the absence of MagicData dialect data from ASR training.

  • Detailed ASR results: Per-test-set CER(%) is reported for all 24 public ASR test sets used in the main evaluation.The appendix includes Fun-ASR-Nano, the open-source checkpoint released by FunAudioLLM.
  • Comparison caveat: API baseline results may change over time because of server-side updates and proprietary components.The reported comparisons disabled ITN and punctuation when available and used each API's default VAD configuration.
  • Data overlap: MagicData Chinese dialect or accented speech is excluded from ASR training and used only for evaluation.The Fun-ASR API may benefit from proprietary training data on some MagicData dialect subsets.

B.2 Full list of Chinese dialect codes

This section provides the full language-code list and the full Chinese-dialect-code list supported by FireRedLID.

  • B.2 Full list of Chinese dialect codes: Table 7 lists the language codes supported by FireRedLID.
  • B.2 Full list of Chinese dialect codes: Together, the tables document FireRedLID’s supported language and Chinese dialect codes.
  • B.2 Full list of Chinese dialect codes: Table 8 lists the Chinese dialect codes supported by FireRedLID.
Loading 2603.10420v1…