PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3-ASR Technical Report

Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, Junyang Lin

arXiv:2601.21337v2cs.CLcs.SDeess.AS

TL;DR

Real-world ASR systems need accurate, flexible timestamp alignment across languages, while open benchmarks may not capture deployment quality. Qwen3-ASR introduces multilingual ASR models and a non-autoregressive forced aligner, achieving strong recognition performance and a 67%–77% relative reduction in accumulated timestamp shift versus other alignment methods.

  • Problem

    Real-world ASR deployments need accurate timestamp alignment across diverse languages and granularities, beyond standard post-processing approaches.

  • Method

    The Qwen3-ASR family combines two Qwen3-Omni-based multilingual ASR models with a lightweight LLM-based non-autoregressive forced aligner.

  • Results

    67%–77% relative reduction in accumulated average timestamp shift was achieved versus other forced alignment methods, alongside strong performance across diverse ASR benchmarks.

  • Takeaways & Limitations

    The released models provide a unified, multilingual option for speech recognition and flexible timestamp alignment across varied real-world speech conditions.

Abstract

from arXiv · show

In this report, we introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model. Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects. Both of them leverage large-scale speech training data and the strong audio understanding ability of their foundation model Qwen3-Omni. We conduct comprehensive internal evaluation besides the open-sourced benchmarks as ASR models might differ little on open-sourced benchmark scores but exhibit significant quality differences in real-world scenarios. The experiments reveal that the 1.7B version achieves SOTA performance among open-sourced ASR models and is competitive with the strongest proprietary APIs while the 0.6B version offers the best accuracy-efficiency trade-off. Qwen3-ASR-0.6B can achieve an average TTFT as low as 92ms and transcribe 2000 seconds speech in 1 second at a concurrency of 128. Qwen3-ForcedAligner-0.6B is an LLM based NAR timestamp predictor that is able to align text-speech pairs in 11 languages. Timestamp accuracy experiments show that the proposed model outperforms the three strongest force alignment models and takes more advantages in efficiency and versatility. To further accelerate the community research of ASR and audio understanding, we release these models under the Apache 2.0 license.

1 Introduction

The report presents Qwen3-ASR, a family of multilingual all-in-one ASR models and a lightweight multilingual forced aligner built from Qwen3-Omni. It targets challenges in LALM-based speech recognition and timestamp prediction, including long-form audio, complex environments, and flexible temporal granularity.

  • Motivation: Large Audio-Language Models replace bottom-up acoustic matching with high-level audio understanding followed by language-model-conditioned transcription, leveraging language modeling and world knowledge.This paradigm addresses challenges that are relatively difficult for conventional ASR models, including long-form audio.
  • Motivation: Because real-world ASR deployments require transcript timestamps, the report argues that LALM-based timestamp prediction can be more accurate, faster, and flexible than conventional post-processing approaches.Prior systems commonly use CTC or CIF post-processing, whereas the proposed approach targets arbitrary temporal granularities and multilingual timestamping.
  • Contributions: Qwen3-ASR-1.7B and Qwen3-ASR-0.6B provide all-in-one ASR and language identification for 52 languages and dialects, including robust recognition in complex environments.The models support singing voice, noise environments, and complex text patterns, while finely supporting 30 languages, 22 Chinese dialects, and English worldwide.
  • Contributions: Qwen3-ForcedAligner-0.6B introduces a lightweight LALM-based multilingual forced aligner supporting 11 languages and flexible timestamp prediction granularities.It is designed to predict timestamps for words or sentences in ASR results and is presented as a novel speech forced-alignment architecture.

2 Qwen3-ASR

Qwen3-ASR combines a Qwen3-Omni foundation model with an AuT speech encoder and staged training, including supervised finetuning and reinforcement learning. The family includes compact ASR models supporting 52 languages and dialects, while Qwen3-ForcedAligner-0.6B supports 11 languages.

  • Architecture: AuT is an AED speech encoder that downsamples 128-dimensional Fbank features by 8× to produce audio tokens at 12.5 Hz.It is pretrained separately from Qwen3-Omni and Qwen3-ASR.
  • Architecture: Qwen3-ASR-0.6B combines Qwen3-0.6B, a projector, and a 180M-parameter AuT encoder with hidden size 896 to balance accuracy and inference efficiency.The model is designed to remain competitive among sub-1B-parameter ASR models.
  • Training: Training proceeds through AuT pretraining, Omni pretraining, ASR supervised finetuning, and ASR reinforcement learning.AuT pretraining uses approximately 40 million hours of pseudo-labeled ASR data, while Omni pretraining trains both model sizes with 3 trillion tokens on multimodal data.
  • Training: ASR finetuning uses disjoint multilingual data and non-speech, streaming-enhancement, and context-biasing data to produce an ASR-only model that resists instruction-following failures.The model also uses context tokens in the system prompt as background knowledge for customized ASR results.
  • Training: GSPO reinforcement learning improves noise robustness, transcription stability, and difficult-case analysis using about 50k utterances spanning Chinese and English, multilingual, and functional data.GSPO is attributed to Zheng et al. (2025).
  • Model family: The ASR models support 52 languages and dialects, comprising 30 languages and 22 Chinese dialects, while Qwen3-ForcedAligner-0.6B supports 11 languages.Table 1 defines NAR as non-autoregressive inference and Seq. Len. as the maximum single-inference audio length in seconds.

Qwen3-ASR-0.6B

Qwen3-ASR-0.6B delivers leading Chinese and English recognition while supporting broad multilingual, dialectal, long-form, streaming, and singing-song transcription capabilities. It recognizes complete songs with background music, demonstrating robustness to accompaniment and complex musical mixtures.

  • The model achieves leading Chinese and English recognition performance across multiple benchmarks against many competing systems.Chinese and English account for most training data across all stages.
  • Qwen3-ASR-0.6B supports 30 languages and 22 dialects, as detailed in Table 1.
  • Qwen3-ASR-0.6B supports single speech up to 20 minutes with unified streaming and offline inference.
  • The model accurately recognizes singing voice and complete songs with background music, remaining robust to accompaniment and complex musical mixtures.

3 Qwen3-ForcedAligner

Qwen3-ForcedAligner-0.6B formulates forced alignment as slot filling, directly predicting discrete timestamps for transcript words or characters. Its non-autoregressive design delivers accurate, fast, and multilingual alignment across flexible word- or character-level scenarios.

  • The model reframes alignment as slot filling, directly predicting discrete start and end timestamp indices for word- or character-level [time] slots.Timestamp indices are converted to actual times by multiplying them by the 80ms frame duration.
  • A 67%~77% relative reduction in accumulated average timestamp shift is achieved on human-labeled test datasets versus other forced alignment methods.
  • Qwen3-ForcedAligner-0.6B supports 11 languages, cross-lingual speech, durations up to 300 seconds, and customizable alignment for any word or character.An AuT encoder and multilingual Qwen3-0.6B LLM provide multilingual and cross-lingual speech and semantic representations.
  • Non-autoregressive decoding predicts all inserted timestamp slots simultaneously, replacing the next-token prediction paradigm for faster inference.Training retains non-shifted output and label sequences, while dynamic slot insertion improves generalization.
  • An RTF close to 0.001 under high concurrency enables processing 1,000 seconds of audio per second.Benchmarks use FlashAttention, bfloat16, and Transformers inference.

4 Experiments · 4.1 Evaluation Details

The evaluation uses four complementary protocols to assess Qwen3-ASR across public benchmarks, difficult real-world conditions, multilingual speech, and singing voice. Accuracy is measured with language-appropriate error rates, while language identification and timestamp alignment use dedicated metrics.

  • 4.1 Evaluation Details: The study compares Qwen3-ASR against leading proprietary APIs—GPT-4o-Transcribe, Gemini-2.5-Pro (Comanici et al., 2025), and Doubao-ASR (Bai et al., 2024)—and multilingual open-source models.The listed open-source baselines include Whisper-large-v3 (Radford et al., 2023) and FunASR-MLT-Nano (An et al.).
  • 4.1 Evaluation Details: Public evaluation covers English, standard Mandarin, and Chinese dialect subsets across broad benchmarks, including two recently released benchmarks.The public benchmark suite draws on datasets cited in the paper, including Conneau et al. (2023), Ardila et al. (2020), Zhang et al. (2022), Panayotov et al. (2015), Dai et al. (2025), and Li et al. (2025).
  • 4.1 Evaluation Details: The internal robustness suite tests 16 English accent groups, 22 Chinese dialect varieties, and challenging speech involving age, noise, fluency, repetition, and multiple speakers.Conditions include elderly and children’s speech, extremely low SNR, nonfluent or tongue-twister-like repetitive speech, and multi-speaker Chinese conversation.
  • 4.1 Evaluation Details: Multilingual evaluation uses Common Voice, Fleurs, MLS, MLC-SLM (Mu et al., 2026a), and an internal test set spanning 15 languages, with progressively expanded Fleurs subsets.The protocol evaluates the model’s 30-language ASR support and groups Fleurs languages by popularity and practical usage for finer-grained analysis.
  • 4.1 Evaluation Details: Singing voice recognition is evaluated on public benchmarks and an internal set, emphasizing long-form songs supplied as single inputs.This tests robustness to long-duration audio and the distinctive acoustic and rhythmic properties of singing.
  • 4.1 Evaluation Details: Recognition accuracy uses CER for character-based languages and WER for word-delimited languages, with macro-averages for aggregated language or dialect results.CER applies to languages such as Mandarin Chinese, Cantonese, and Korean, while WER applies to languages such as English, German, and French.
  • 4.1 Evaluation Details: Language identification is measured by accuracy, while timestamp alignment uses AAS, whose lower values indicate more accurate predictions.AAS is the mean absolute difference between predicted and reference timestamps across slots; references come from MFA or manual annotations.

4.2 English & Chinese ASR Performance

Qwen3-ASR performs strongly across English, Mandarin Chinese, and Chinese dialect benchmarks, outperforming open-source baselines and remaining competitive with commercial APIs. The 1.7B model improves consistently over 0.6B and remains robust under accents, difficult acoustics, and speaking conditions.

  • English & Chinese ASR Performance: Qwen3-ASR-1.7B achieves the strongest overall English results on several real-world datasets and remains close to leading systems on LibriSpeech.Its strongest gains appear on crowdsourced or web-collected speech with greater distribution shift than read-speech benchmarks.
  • English & Chinese ASR Performance: Qwen3-ASR achieves the best overall Mandarin performance on most benchmarks, including a large margin over baselines on diverse, meeting-style WenetSpeech.The model remains reliable on challenging large-scale evaluations with diverse acoustic environments.
  • English & Chinese ASR Performance: Qwen3-ASR remains highly competitive across Chinese dialects, especially Cantonese and challenging long-utterance settings despite pronunciation and lexical variation.Specialized commercial APIs lead in a small number of dialect-specific cases, but Qwen3-ASR ranks among the top systems overall.
  • English & Chinese ASR Performance: Qwen3-ASR delivers strong cross-domain English, Mandarin, and Chinese dialect performance, outperforming open-source baselines and competing with leading commercial APIs.Table 3 covers English, Mandarin Chinese, Cantonese, and other Chinese dialect benchmarks; commercial and open-source results were independently evaluated where published numbers were unavailable.
  • Robustness Evaluation: On internal robustness tests, Qwen3-ASR-1.7B performs best across Mandarin subsets and both conversational Cantonese and aggregated 22-dialect evaluations, while achieving the lowest accented-English WER.Scaling from 0.6B to 1.7B yields stable gains across accents, dialects, difficult acoustic conditions, and challenging speaking scenarios.

4.3 Multilingual ASR and Language Identification

Qwen3-ASR-1.7B achieves the strongest multilingual ASR performance across most public and internal evaluations, while Qwen3-ASR-0.6B remains a competitive lightweight alternative. The models also perform language identification through natural-language prompting before ASR decoding across four multilingual benchmarks.

  • Multilingual ASR: Qwen3-ASR-1.7B achieves the best average performance on most multilingual ASR settings, including the internal News-Multilingual benchmark, while Qwen3-ASR-0.6B remains competitive.The evaluation spans broad public benchmarks and internal multilingual news speech, indicating strong cross-language and cross-domain generalization.
  • Multilingual ASR: Qwen3-ASR-1.7B outperforms evaluated open-source baselines, including Whisper-large-v3, on MLS, Common Voice, and MLC-SLM, and leads Fleurs on its 12- and 20-language subsets.On the full 30-language Fleurs setting, its performance degrades relative to Whisper-large-v3, leaving room for improvement.
  • Multilingual ASR: Qwen3-ASR-1.7B remains markedly better than Qwen3-ASR-0.6B, suggesting that scaling improves robustness in challenging multilingual regimes.This scaling advantage is consistent with the stronger results observed on broader and more difficult multilingual evaluations.
  • Language Identification: Qwen3-ASR performs language identification through natural-language prompting before ASR decoding, evaluated on Fleurs, MLS, CommonVoice, and MLC-SLM.The benchmarks cover 30, 9, 13, and 11 languages, respectively, and compare the 0.6B and 1.7B models with Whisper-large-v3 in Table 6.

4.4 Singing Voice & Songs Recognition Performance · 4.5 Streaming Speech Recognition · 4.6 Precision of Timestamps

Across singing, streaming, and forced-alignment evaluations, Qwen3 models show robust recognition quality under musical and streaming conditions while supporting multilingual timestamp alignment. Qwen3-ForcedAligner-0.6B additionally generalizes well to human-labeled data and cross-lingual, code-switched scenarios.

  • 4.4 Singing Voice & Songs Recognition Performance: Qwen3-ASR-1.7B is robust to melody-induced pronunciation variation and musical accompaniment, outperforming most commercial APIs and open-source baselines across singing and long-form song evaluations.Table 7 covers singing-only benchmarks and long-form songs with background music; unsupported long-form recognition is marked N/A.
  • 4.4 Singing Voice & Songs Recognition Performance: Qwen3-ASR-1.7B achieves the best performance on M4Singer, MIR-1k-vocal, and Popcs, while ranking second on Opencpop by a small margin.FunASR-MLT-Nano leads Opencpop.
  • 4.5 Streaming Speech Recognition: Qwen3-ASR supports unified offline and streaming inference, with streaming preserving strong recognition accuracy on three open-source test sets.The streaming evaluation uses 2-second chunks, a 5-token fallback, and keeps the last four chunks unfixed.
  • 4.5 Streaming Speech Recognition: The dynamic attention-window mechanism enables Qwen3-ASR-1.7B and Qwen3-ASR-0.6B to support streaming inference naturally.Both model sizes are evaluated in offline and streaming modes.
  • 4.6 Precision of Timestamps: Qwen3-ForcedAligner-0.6B supports multilingual, cross-lingual, and code-switched alignment with one model, whereas competing methods require language-specific models and support fewer languages.Table 9 compares accumulated average shift on MFA-labeled and human-labeled test sets.
  • 4.6 Precision of Timestamps: Qwen3-ForcedAligner-0.6B remains consistent on short and long utterances and achieves low AAS on human-labeled test sets, indicating strong real-world generalization.The comparison includes MFA-labeled and human-labeled test sets.

5 Conclusion

Qwen3-ASR is a speech-model family comprising two ASR systems and a forced-alignment model trained on large-scale speech corpora. Leveraging Qwen3-Omni’s audio understanding and a four-stage training pipeline, the ASR models outperform comparable or larger competing models and commercial APIs in speech coverage and recognition accuracy.

  • The Qwen3-ASR family comprises two ASR systems and one forced-alignment model trained on large-scale speech corpora.
  • Leveraging Qwen3-Omni’s audio understanding capability and a four-stage training pipeline, Qwen3-ASR-1.7B and Qwen3-ASR-0.6B outperform competing models of comparable or larger size.
  • The two ASR models also outperform commercial APIs in speech coverage and recognition accuracy.

6 Authors

The report credits core contributors and additional contributors for the Qwen3-ASR work.

  • Core contributors include Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu†, Jingren Zhou, and Junyang Lin†.
  • Additional contributors include Yunfei Chu, Daren Chen, Ting He, Hangrui Hu, Jiayi Leng, Zheng Li, Yuanjun Lv, Bingshen Mu, Hao Su, Xian Yang, Xuechun Wang, Yuezhang Wang, Zhenglin Wang, Lei Xie, Jianwei Zhang, Xinfa Zhu, and Guangdong Zhou.

Appendix

The appendix reports supplementary evaluations of Qwen3-ASR across English, Chinese, Chinese dialect, and open-source multilingual benchmarks, including reference results from the Qwen3-ASR-Flash-1208 API.

  • Appendix: Table A.1 evaluates Qwen3-ASR on English, Chinese, and a range of Chinese dialect benchmarks.
  • Appendix: Qwen3-ASR-Flash-1208 is included as a Qwen3-ASR-family API, with results provided for reference.
  • Appendix: Table A.2 evaluates Qwen3-ASR on open-source multilingual benchmarks.
Loading 2601.21337v2…