PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3-Omni Technical Report

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, Junyang Lin

arXiv:2509.17765v1cs.CLcs.AIcs.CVeess.AS

TL;DR

Qwen3-Omni addresses the modality trade-offs of multimodal models by integrating text, image, audio, and video processing in one system. Its Thinker-Talker MoE architecture and multimodal training preserve same-size unimodal performance while enabling strong cross-modal capabilities, including 32 open-source SOTA results across audio and audio-visual benchmarks. The model further supports multilingual interaction and low-latency streaming speech, with publicly released 30B-A3B, Thinking, and Captioner variants.

  • Problem

    Contemporary LLM-centric multimodal models often trade gains in one modality against degradation in others, motivating integrated multimodal training.

  • Method

    Qwen3-Omni uses integrated multimodal training with a Thinker-Talker MoE architecture, an AuT audio encoder, multi-codebook speech generation, and lightweight streaming modules.

  • Results

    Qwen3-Omni maintains text and visual performance without degradation relative to same-size Qwen models and achieves open-source SOTA on 32 of 36 audio and audio-visual benchmarks.

  • Takeaways & Limitations

    The released models provide multimodal reasoning, multilingual speech interaction, audio understanding up to 40 minutes, and 234 ms end-to-end first-packet latency.

  • Takeaways & Limitations

    The authors could not conduct a comprehensive sweep across all model scales because of prohibitive experimental cost.

Abstract

from arXiv · show

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet latency in streaming synthesis, Talker autoregressively predicts discrete speech codecs using a multi-codebook scheme. Leveraging the representational capacity of these codebooks, we replace computationally intensive block-wise diffusion with a lightweight causal ConvNet, enabling streaming from the first codec frame. In cold-start settings, Qwen3-Omni achieves a theoretical end-to-end first-packet latency of 234 ms. To further strengthen multimodal reasoning, we introduce a Thinking model that explicitly reasons over inputs from any modality. Since the research community currently lacks a general-purpose audio captioning model, we fine-tuned Qwen3-Omni-30B-A3B to obtain Qwen3-Omni-30B-A3B-Captioner, which produces detailed, low-hallucination captions for arbitrary audio inputs. Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under the Apache 2.0 license.

1 Introduction

Qwen3-Omni addresses modality trade-offs by integrating multimodal training into a unified model that preserves specialized text and vision performance while enabling cross-modal reasoning and interaction. Its Thinker–Talker design and expanded capabilities support broad multimodal tasks, multilingual use, and low-latency streaming speech.

  • Motivation and contribution: Qwen3-Omni targets modality trade-offs by jointly training across modalities while preserving performance across text, vision, audio, and audiovisual capabilities.The training strategy mixes unimodal and cross-modal data during early text pretraining.
  • Applications: Qwen3-Omni supports voice dialogue, video dialogue, and video reasoning through unified processing of text, audio, image, and video inputs.The model generates real-time text or speech responses.
  • Key upgrades: Qwen3-Omni introduces MoE upgrades, the AuT audio encoder trained on 20 million hours of supervised audio, multi-codebook speech generation, and expanded multimodal capabilities.The model also supports audio inputs exceeding 40 minutes, 119 written languages, 19 spoken languages for understanding, and 10 for generation.
  • Results: 32 audio and audio-visual benchmarks achieve open-source SOTA, while 22 achieve overall SOTA against systems including Gemini 2.5 Pro, Seed-ASR, and GPT-4o-Transcribe.Qwen3-Omni also maintains state-of-the-art text and visual performance without degradation relative to same-size Qwen models.
  • Architecture: Qwen3-Omni uses a Thinker–Talker architecture in which Thinker generates text and Talker generates streaming speech tokens from multimodal representations.Talker predicts multi-codebook speech sequences, while MTP and Code2Wav enable frame-by-frame waveform synthesis.

2 Architecture

Qwen3-Omni uses a scalable Thinker–Talker MoE architecture that integrates text, image, audio, and video processing with streaming speech generation. Its AuT encoder and multi-codebook Talker support general-purpose audio understanding and low-latency synthesis.

  • Both Thinker and Talker adopt Mixture-of-Experts architectures to support high concurrency and fast inference.
  • Talker: Talker generates one codec frame per step, while the MTP module predicts the remaining residual codebooks.This multi-codebook autoregressive design represents acoustic detail before waveform reconstruction.
  • Talker: A lightweight causal ConvNet implements Code2Wav, simplifying the final stage of audio synthesis.
  • Audio Transformer (AuT): AuT is a 0.6B-parameter attention-encoder-decoder audio encoder trained from scratch on 20 million hours of supervised audio.It produces representations at a 12.5 Hz token rate and uses dynamic attention windows from 1 to 8 seconds.
  • Multimodal encoding: Thinker processes text, audio, images, and video without audio, while the vision encoder supports both image understanding and video comprehension.Audio is represented at approximately 80 ms per frame, and video frames are sampled at a dynamic rate.
  • Streaming and concurrency: Qwen3-Omni asynchronously prefills Thinker and Talker chunks, reducing Time-To-First-Token while MoE designs improve service throughput.The streaming codec decoder can output waveform immediately after each Talker token, reducing first-packet latency.

3 Pretraining

Qwen3-Omni is pretrained on diverse multilingual and multimodal data using staged encoder alignment, general multimodal training, and long-context adaptation. The final stage increases context length and long-sequence data to improve audio and video understanding.

  • Qwen3-Omni combines unimodal and cross-modal data from early pretraining across image-text, video-text, audio-text, video-audio, video-audio-text, and pure-text corpora.
  • Pretraining comprises encoder alignment, general multimodal training, and long-context stages.The first stage trains encoders with the LLM fixed, the second unfreezes all parameters, and the third uses longer sequences.
  • General Stage: Approximately 2 trillion tokens are used in the general stage, distributed across text, audio, image, video, and video-audio data.
  • Long Context Stage: 32,768 maximum tokens replace 8,192 in the long-context stage, alongside increased proportions of long audio and long video.Experimental results indicate improved understanding of long-sequence data.

4 Post-training

Post-training separately develops Thinker instruction following and Talker speech generation through staged supervised fine-tuning, distillation, reinforcement optimization, and multimodal speech training. A dedicated Captioner extends the system to detailed audio description.

  • Thinker: Thinker post-training uses three stages: lightweight supervised fine-tuning, Strong-to-Weak Distillation, and GSPO-based multimodal enhancement.
  • Thinker: Strong-to-Weak Distillation combines teacher-generated off-policy responses with on-policy student sequences aligned to teacher logits by minimizing KL divergence.
  • Thinker: GSPO uses rule-based rewards for verifiable tasks and model-based evaluation for multimodal tasks without objective predefined metrics.
  • Talker: Talker training has four stages and uses hundreds of millions of multimodally contextualized speech examples to map multimodal representations to speech.Continual pretraining with high-quality data alleviates hallucinations caused by noisy first-stage data and improves speech quality.
  • Captioner: Qwen3-Omni-30B-A3B-Captioner is fine-tuned on detailed audio descriptions to generate detailed, low-hallucination captions for arbitrary audio inputs.

5 Evaluation

Evaluation covers both multimodal understanding and speech generation for several Qwen3-Omni variants. The reported evaluation is divided into X→Text understanding and X→Speech generation.

  • The evaluation includes Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, Qwen3-Omni-Flash-Instruct, and Qwen3-Omni-Flash-Thinking.The Flash variants target computational efficiency and performance efficacy and add support for various dialects.
  • Evaluation results are organized into understanding (X→Text) and speech generation (X→Speech) categories.

5.1 Evaluation of X→Text

Qwen3-Omni is evaluated across text, audio, vision, and audiovisual video inputs, with comparisons spanning general understanding, reasoning, speech, music, and multimodal integration. The reported results show strong performance across these modalities, including state-of-the-art results on several audio and audiovisual benchmarks.

  • The evaluation covers text, audio, vision, and audiovisual video inputs that produce textual responses.Text evaluation includes general, reasoning, coding, alignment, agent, and multilingual tasks; audio evaluation includes ASR, S2TT, music understanding, voice chatting, and audio reasoning.
  • Text→Text: Qwen3-Omni-30B-A3B-Instruct matches the textual capabilities of same-sized text-only Qwen counterparts.
  • Audio→Text: Qwen3-Omni achieves state-of-the-art En & Zh ASR and lyric ASR performance across Librispeech, Wenetspeech, Fleurs, CommonVoice, Opencpop-test, and MIR-1K (vocal).It also delivers better or comparable performance with Voxtral-Small and Gemini-2.5-Pro on Multilingual ASR and S2TT.
  • Audio→Text: 89.5 is Qwen3-Omni-Thinking’s average VoiceBench score, below only Gemini-2.5-Pro’s 89.6 among compared audio language models.Qwen3-Omni also outperforms several closed-source models on MMAU and MMSU audio reasoning benchmarks.
  • Vision→Text: Qwen3-Omni-Instruct performs comparably to Qwen2.5-VL-72B and better than GPT-4o and Gemini-2.0-Flash on several Math and STEM vision tasks.
  • Vision→Text: Qwen3-Omni-Thinking outperforms the Qwen3-Omni-Instruct baseline by 4.4 points on Math and STEM benchmarks.The model is reported as comparable to substantially larger reasoning baselines, while showing suboptimal performance on long video benchmarks.
  • AudioVisual Video→Text: Qwen3-Omni-Instruct achieves state-of-the-art performance on the WorldSense audiovisual benchmark.WorldSense measures integration of visual and auditory signals, while DailyOmni and VideoHolmes assess audiovisual reasoning.

5.2 Evaluation of X→Speech

Qwen3-Omni’s speech-generation evaluation examines zero-shot, multilingual, and cross-lingual speech synthesis. The model is reported as competitive with leading systems, with reinforcement learning improving stability and multilingual and cross-lingual voice cloning showing strong results.

  • Speech-generation evaluation covers zero-shot, multilingual, and cross-lingual speech generation using content consistency and speaker similarity measures.
  • Zero-Shot Speech Generation: Qwen3-Omni demonstrates highly competitive zero-shot speech-generation performance against state-of-the-art TTS systems.Reinforcement learning improves generation stability and produces the best performance on the test-en set.
  • Multilingual Speech Generation: Qwen3-Omni surpasses MiniMax-Speech and ElevenLabs Multilingual v2 in Chinese, English, and French multilingual speech generation.It delivers competitive results in the remaining evaluated languages and supports speech generation across 10 languages.
  • Cross-Lingual Speech Generation: Qwen3-Omni outperforms CosyVoice3 in any-to-en and any-to-ko cross-lingual voice cloning.On any-to-ja tasks, it achieves comparable performance to CosyVoice3 even without text normalization.

6 Evaluating Non-Degradation Across Modalities

The paper evaluates modality non-degradation through controlled comparisons between matched text-only, vision-only, and multimodal models. Early mixing of unimodal and cross-modal data supports performance across modalities, although the study does not cover all model scales.

  • The controlled study compares text-only, vision-only, and multimodal Omni models with matched parameter counts and aligned training conditions.The Omni model uses the same text and vision corpora as the unimodal baselines, with aligned schedules, batch sizes, epochs, and training compute.
  • Early mixing of unimodal and cross-modal data during text pretraining achieves better performance across text, vision, and video modalities.
  • Early multimodal integration allows co-training with vision or audio without degradation in language capability.
  • Adding text substantially improves vision and audio performance, while visual or audio signals produce no measurable language gains.
  • The study could not conduct a comprehensive sweep across all model scales because of prohibitive experimental cost.

7 Conclusion

The paper introduces Qwen3-Omni model variants designed to combine strong multimodal understanding, reasoning, and speech interaction. Its reported results include same-size text and vision parity, broad audio strength, multilingual support, and low streaming latency.

  • The paper introduces Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, Qwen3-Omni-Flash-Instruct, and Qwen3-Omni-Flash-Thinking.
  • Qwen3-Omni-30B-A3B matches or surpasses same-size unimodal Qwen models on text and vision benchmarks.
  • Qwen3-Omni achieves open-source state-of-the-art performance on 32 audio and dialogue benchmarks and is comparable to or better than Gemini-2.5-Pro.
  • Qwen3-Omni supports 119 text languages, 19 speech-recognition languages, 10 speech-synthesis languages, and audio interaction up to 40 minutes.
  • 234 ms is the reported end-to-end first-packet latency at the 30B-A3B scale.The paper attributes this latency to the model’s streaming architecture and multi-codebook design.
  • The authors characterize fully integrated end-to-end multimodal training without degrading core language capability as a milestone.
  • Qwen3-Omni offers strong text and vision capabilities, robust ASR, interactive speech, low latency, and stable naturalistic speech synthesis for practical usage.The paper also reports advantages over cascaded pipelines in cross-modal reasoning, latency, system complexity, and cost.

8 Authors

The authors section lists the paper’s core contributors and additional contributors.

  • Core contributors include Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, and Xiong Wang.
  • The contributor lists comprise many researchers spanning the Qwen3-Omni project.
  • Additional contributors include An Yang, Anfeng Li, Bei Chen, Beichen Zhang, and Bin Lin.

9 Appendix

The appendix reports additional speech and music evaluations alongside qualitative audio-captioning examples across cinematic, conversational, and Mandarin monologue clips.

  • More Evaluation on Speech and Music Understanding: Qwen3-Omni-Thinking is outperformed by its Instruct counterpart on ASR/S2TT and Music understanding tasks.The passage attributes this to sophisticated reasoning failing to improve predominantly perception-based tasks and potentially increasing hallucinations.
  • More Evaluation on Speech and Music Understanding: Table 17 presents Audio→Text transcription performance for ASR and S2TT, comparing Qwen3-Omni-Thinking with baselines.The table caption states that the highest scores are bolded.
  • Qualitative Results from Qwen3-Omni-30B-A3B-Captioner: A Mandarin monologue portrays Taiyi Zhenren through exaggerated bravado and self-deprecating humor in clean studio production.The performance uses theatrical inflection, expressive pacing, subtle electronic hiss and hum, and an intentional cut to silence.
  • Qualitative Results from Qwen3-Omni-30B-A3B-Captioner: A 25-second cinematic soundscape builds tension with orchestral and electronic elements before a colossal impact and abrupt musical cessation.The impact occurs at 15 seconds, while the orchestral music stops at 19 seconds.
  • Qualitative Results from Qwen3-Omni-30B-A3B-Captioner: One caption describes a non-verbal catastrophic event followed by a single survivor’s breathing, fabric movement, and groan.The absence of spoken words and the surrounding orchestration position the clip as a dramatic action-oriented scene cue.
  • Qualitative Results from Qwen3-Omni-30B-A3B-Captioner: Another caption identifies a high-fidelity futuristic or industrial setting with mechanical sounds, family dialogue, and a synthesized musical sting.The scene includes spatially separated voices and playful familial banter during travel.
Loading 2509.17765v1…