PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3-TTS Technical Report

Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, Xinyu Zhang, Pei Zhang, Baosong Yang, Jin Xu, Jingren Zhou, Junyang Lin

arXiv:2601.15621v1cs.SDcs.CLeess.AS

TL;DR

Stable, controllable, human-like speech synthesis remains a central challenge. Qwen3-TTS addresses it with a dual-track architecture and two speech tokenizers, achieving state-of-the-art results across zero-shot cloning, cross-lingual synthesis, and instruction-controlled voice design.

  • Problem

    Stable, controllable, and human-like speech synthesis remains a central research goal for modern text-to-speech systems.

  • Method

    Qwen3-TTS combines a dual-track language-model architecture with semantic-rich and low-latency speech tokenizers for multilingual, controllable, streaming synthesis.

  • Results

    Qwen3-TTS achieves state-of-the-art performance across zero-shot voice cloning, cross-lingual synthesis, and voice design, including an approximately 66% error-rate reduction for zh-to-ko generation.

  • Takeaways & Limitations

    Qwen3-TTS unifies diverse speech-generation tasks within one framework and its open-sourced models and tokenizers support community research and development.

Abstract

from arXiv · show

In this report, we present the Qwen3-TTS series, a family of advanced multilingual, controllable, robust, and streaming text-to-speech models. Qwen3-TTS supports state-of-the-art 3-second voice cloning and description-based control, allowing both the creation of entirely novel voices and fine-grained manipulation over the output speech. Trained on over 5 million hours of speech data spanning 10 languages, Qwen3-TTS adopts a dual-track LM architecture for real-time synthesis, coupled with two speech tokenizers: 1) Qwen-TTS-Tokenizer-25Hz is a single-codebook codec emphasizing semantic content, which offers seamlessly integration with Qwen-Audio and enables streaming waveform reconstruction via a block-wise DiT. 2) Qwen-TTS-Tokenizer-12Hz achieves extreme bitrate reduction and ultra-low-latency streaming, enabling immediate first-packet emission ($97\,\mathrm{ms}$) through its 12.5 Hz, 16-layer multi-codebook design and a lightweight causal ConvNet. Extensive experiments indicate state-of-the-art performance across diverse objective and subjective benchmark (e.g., TTS multilingual test set, InstructTTSEval, and our long speech test set). To facilitate community research and development, we release both tokenizers and models under the Apache 2.0 license.

1 Introduction

Qwen3-TTS is introduced as a stable, controllable, human-like multilingual TTS model supporting voice creation, fine-grained speech control, and practical LLM integration. Trained on over 5 million hours of speech, it achieves state-of-the-art zero-shot voice-cloning performance across 10 languages.

  • Capabilities: Qwen3-TTS is designed as a multilingual, controllable, robust, and streaming TTS model supporting voice cloning, voice creation and control, and complex-text handling.These capabilities are summarized in Figure 1 as motivating a broad range of applications.
  • Background: Large-scale neural TTS advances have enabled high-quality speech generation from a few seconds of reference audio and are supporting applications such as virtual assistants and automated content creation.The background passage cites Wang et al. (2023), Shen et al. (2023), Ju et al. (2024), Yang et al. (2023), Eskimez et al. (2024), Chen et al. (2024), Du et al. (2024a), Wang et al. (2025; 2024), and Ye et al. (2025b).
  • Controllability: Qwen3-TTS enables users to create new voices and manipulate fine-grained generated-speech attributes through natural-language descriptions while stably synthesizing arbitrary content.The model is presented as the first TTS model in the Qwen series.
  • Results: Qwen3-TTS achieves the lowest Word Error Rate on the Seed-TTS zero-shot voice-cloning benchmark and superior speaker similarity across all 10 evaluated languages versus MiniMax and ElevenLabs.The model is trained on over 5 million hours of speech data.

2 Qwen-TTS-Tokenizer

Qwen-TTS introduces two tokenizer designs: a 25 Hz single-codebook tokenizer with streaming waveform reconstruction and a 12.5 Hz multi-codebook tokenizer with jointly optimized semantic and acoustic streams. Both support streaming, using restricted-context diffusion decoding for the former and causal encoding and decoding for the latter.

  • Qwen-TTS-Tokenizer-25Hz: Qwen-TTS-Tokenizer-25Hz is a 25 Hz single-codebook tokenizer built on Qwen2-Audio through two-stage training.Stage 1 continues Qwen2-Audio pretraining on ASR with added resampling and vector-quantization layers; Stage 2 fine-tunes the model with convolution-based processing.
  • Qwen-TTS-Tokenizer-25Hz: The 25 Hz tokenizer enables streaming waveform reconstruction by mapping code sequences to mel-spectrograms with Flow Matching and reconstructing waveforms using modified BigVGAN.A sliding-window block-attention mechanism restricts each token to a limited context through a Diffusion Transformer.
  • Qwen-TTS-Tokenizer-12Hz: Qwen-TTS-Tokenizer-12Hz is a 12.5 Hz multi-codebook tokenizer with jointly optimized semantic and acoustic streams.Its semantic codebook captures high-level content, while its acoustic codebook models acoustic detail and prosody, following Mimi’s semantic–acoustic disentanglement strategy.
  • Qwen-TTS-Tokenizer-12Hz: The 12 Hz tokenizer supports low-latency streaming through fully causal feature encoders and decoders that emit tokens and reconstruct audio incrementally without look-ahead.The encoder produces semantic and acoustic tokens at 12.5 Hz, while the decoder reconstructs audio sequentially for real-time online services.

3 Method

Qwen3-TTS combines a dual-track language-model architecture with specialized 25Hz and 12Hz tokenization paths for controllable, low-latency speech synthesis. Its staged training, instruction conditioning, and streaming codec designs support multilingual, expressive, and real-time generation.

  • Core architecture: The dual-track architecture concatenates text and acoustic tokens, jointly trains a speaker encoder, and predicts speech incrementally for low-latency synthesis.Text uses the standard Qwen tokenizer, while speech uses Qwen-TTS-Tokenizer; incoming textual tokens trigger immediate acoustic prediction.
  • Tokenizer variants: Qwen3-TTS-12Hz hierarchically predicts RVQ tokens, using the backbone for the zeroth codebook and MTP for residual codebooks to improve expressivity while reducing latency.The 25Hz variant instead predicts single-level speech tokens before chunk-wise DiT waveform reconstruction.
  • Training: The long-context stage improves processing of extended, complex inputs and contextually appropriate speech responses by increasing maximum length and upsampling long speech.The reported maximum token length increases from 8,192 to 32,768.
  • Controllable generation: Instruction-prefixed control enables streaming voice cloning, voice design, and fine-grained manipulation through speaker embeddings or text–speech in-context examples.Reference speech supports real-time cloning, while text–speech pairs better preserve prosody; probabilistic thinking improves instruction following for complex descriptions.
  • Streaming efficiency: The 12Hz tokenizer enables immediate left-context waveform decoding, whereas the 25Hz DiT path requires future-token look-ahead before producing the first chunk.At 25Hz, chunk size 8 requires 16 generated tokens and 320 ms of mel content per packet; at 12.5Hz, packets use 4 tokens representing 320 ms of speech.

4 Experiments

The experiments comprehensively evaluate Qwen3-TTS across two categories: speech tokenization and speech generation.

  • Evaluation scope: The evaluation comprehensively assesses Qwen3-TTS.
  • Speech tokenizer: One evaluation category focuses on speech tokenizers.
  • Speech generation: The other evaluation category focuses on speech generation.

4.1 Evaluation of Speech Tokenizer

The evaluation finds Qwen-TTS-Tokenizer-25Hz competitive with supervised semantic tokenizers on English and Chinese ASR benchmarks, while assessing reconstruction quality, speaker similarity, and efficiency on LibriSpeech.

  • ASR Evaluation: Qwen-TTS-Tokenizer-25Hz S1 achieves ASR performance comparable to or better than the S3 Tokenizer series on English and Chinese CommonVoice and Fleurs subsets.The S1 variant is trained with ASR supervision and attains the lowest or near-lowest scores among the compared systems.
  • Speech Reconstruction: Speech reconstruction is evaluated on 2,620 LibriSpeech test-clean utterances using STOI, PESQ, UTMOS, and speaker-similarity metrics alongside tokenizer efficiency parameters.Reported configuration factors include quantizer count, codebook size, and frames per second.
  • Speech Reconstruction: The reported results emphasize a dual improvement in reconstruction quality and efficiency, supporting the tokenizer’s capabilities in speech representation learning and semantic information fusion.The passage characterizes this as a breakthrough across both quality and efficiency dimensions.

4.2 Speech Generation

Qwen3-TTS is evaluated across zero-shot, multilingual, cross-lingual, controllable, target-speaker, and long-form speech generation settings. It achieves strong reported performance, including state-of-the-art cross-lingual results in English- and Korean-targeting scenarios and leading open-source voice design performance.

  • Zero-Shot Speech Generation: In zero-shot speech generation, Qwen3-TTS delivers robust multilingual performance, and its 12Hz variants consistently outperform 25Hz variants in content accuracy.The comparison uses WER on the Seed-TTS test set, where lower values indicate better content consistency.
  • Multilingual Speech Generation: Qwen3-TTS achieves the lowest WER in 6 of 10 evaluated languages, including Chinese, English, Italian, French, Korean, and Russian, against MiniMax-Speech and ElevenLabs Multilingual v2.The remaining evaluated languages are German, Portuguese, Spanish, and Japanese.
  • Cross-Lingual Speech Generation: Qwen3-TTS establishes state-of-the-art cross-lingual results for English- and Korean-targeting scenarios, reducing zh-to-ko error from 14.4 to 4.82 versus CosyVoice3.This corresponds to an approximately 66% reduction in error rate.
  • Controllable Speech Generation: Qwen3-TTS-12Hz-1.7B-VD establishes a new state-of-the-art among open-source models for voice design on InstructTTSEval.Using ChatML, the model treats voice control as a language-modeling task for manipulating speech attributes from text descriptions.
  • Target-Speaker Speech Generation: Speaker fine-tuning transfers Aiden Voice’s timbre and prosody across all 10 evaluated languages despite training exclusively on monolingual data.The target-speaker model is benchmarked against GPT-4o-Audio-Preview’s Ballad Voice on the multilingual test set.
  • Long Speech Generation: The evaluation covers long-form robustness on Chinese and English texts ranging from 200 to 2000 words, comparing fine-tuned Aiden Voice synthesis against open-source systems.The internal dataset contains 100 texts, and Qwen3-ASR supplies transcription for WER evaluation following Seed-TTS Eval methodology.

5 Conclusion

Qwen3-TTS combines a dual-track architecture with semantic-rich and low-latency tokenizers to deliver robust, multilingual, real-time speech synthesis. Evaluations report state-of-the-art results across multiple speech-generation tasks, while the unified framework and open-source release support broader audio research.

  • 5 Conclusion: Qwen3-TTS combines a novel dual-track design with semantic-rich Qwen-TTS-Tokenizer-25Hz and low-latency Qwen-TTS-Tokenizer-12Hz for high-fidelity, streaming speech synthesis.The architecture is designed for real-time speech synthesis and streaming efficiency.
  • 5 Conclusion: Qwen3-TTS achieves state-of-the-art performance across diverse tasks, including zero-shot voice cloning, cross-lingual synthesis, and voice design.It sets new benchmarks and significantly outperforms existing baselines in challenging Chinese-to-Korean generation.
  • 5 Conclusion: A single autoregressive framework unifies zero-shot cloning, cross-lingual transfer, and fine-grained instruction control for diverse speech generation.The authors position this unification as a path toward next-generation omni-capable audio systems.
  • 5 Conclusion: Future work will extend versatile audio generation, expand multilingual coverage beyond the current 10 languages, and explore more granular stylistic control.The planned directions target broader capabilities and finer-grained generation control.
  • 5 Conclusion: The open-source release of Qwen3-TTS models and tokenizers is intended to accelerate community research and more natural, expressive, accessible human-computer interfaces.The release supports development around the paper’s speech-generation technology.

6 Authors

The report identifies a core contributor group led by Hangrui Hu, Xinfa Zhu, Ting He, and Dake Guo, alongside a broader contributor group. Together, the listed contributors comprise the paper’s named author team.

  • Core Contributors: The report lists 16 core contributors, including Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, and Junyang Lin†.The full core-contributor list is Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, Xinyu Zhang, Pei Zhang, Baosong Yang, Jin Xu†, Jingren Zhou, and Junyang Lin†.
  • Author Organization: The author section distinguishes the named core-contributor group from the broader list of additional contributors.The two rosters are presented separately as “Core Contributors” and “Contributors.”
  • Contributors: The report additionally acknowledges 21 contributors, including Yunfei Chu, Daren Chen, Jiayi Leng, Zheng Li, and Yuanjun Lv.The additional contributor list comprises Yunfei Chu, Daren Chen, Jiayi Leng, Zheng Li, Yuanjun Lv, Linhan Ma, Ziyang Ma, Xian Shi, Hao Su, Xuechun Wang, Yongqi Wang, Yuezhang Wang, Yuxuan Wang, Zhenglin Wang, Lei Xie, Kangxiang Xia, Qize Yang, Xian Yang, Jianwei Zhang, Guangdong Zhou, and Jialong Zuo.
Loading 2601.15621v1…