Qwen3-Omni Technical Report

1 Introduction

• The introduction motivates integrated multimodal learning by noting that LLM-centric multimodal models can trade off performance across modalities, unlike coordinated human perception and processing.It frames complementary specialization and cross-modal synergy as benefits of learning across modalities.Source: §1 · p. 1
• Qwen3-Omni targets parity with specialized unimodal models while adding cross-modal reasoning and interaction, with early text pretraining mixing unimodal and cross-modal data as a key ingredient.Qwen3-Omni-30B-A3B-Base is reported on par with same-sized text and vision base models across extensive benchmarks, alongside strong audio competence, audiovisual understanding, cross-modal thinking, and real-time audiovisual interaction.Source: §1 · pp. 1–2
• Qwen3-Omni upgrades both Thinker and Talker to MoE designs and redesigns audio encoding and speech generation for real-time, low-latency interaction.The upgrades include an AuT encoder trained on 20 million hours of supervised audio, multi-codebook and multi-track codec prediction, a ConvNet waveform stage instead of block-wise DiT, and 12.5 Hz input and output audio code rates enabling immediate single-frame synthesis. These changes support high-concurrency industrial-scale deployment.Source: §1 · p. 2
• Compared with Qwen2.5-Omni, Qwen3-Omni supports audio inputs exceeding 40 minutes, broader language coverage, full-modality Thinking, and streaming latency as low as 234 ms.It supports 119 written languages, 19 spoken languages for understanding, and 10 for generation; the Thinking model handles audio-video and audio-only reasoning.Source: §1 · p. 2
• Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 and overall SOTA on 22, outperforming strong closed-source systems.It also maintains state-of-the-art text and visual performance without degradation relative to same-size Qwen single-modal counterparts.Source: §1 · p. 2

2.1 Overview

• Both Thinker and Talker use MoE, while Talker conditions on multimodal features rather than Thinker’s high-level text representations.This decoupling permits external modules to intervene on Thinker text and optionally supply text to Talker through controlled preprocessing for streaming synthesis.Source: §2.1 · p. 3

2.2 Audio Transformer (AuT)

• AuT is a roughly 0.6B-parameter attention-encoder-decoder trained from scratch on 20 million hours of supervised audio for speech recognition and audio understanding.Its training uses 80% Chinese and English pseudo-labeled ASR, 10% other-language ASR, and 10% audio-understanding data; dynamic attention windows span 1–8 seconds, and its encoder serves as Qwen3-Omni’s audio encoder.Source: §2.2 · p. 4, Figure 3 · p. 4

2.3 Perceivation

• TM-RoPE interleaves temporal, height, and width dimensions and anchors audiovisual representations to absolute time, supporting streaming inputs of arbitrary duration.Audio temporal IDs correspond to 80 ms, video IDs follow frame timestamps, and unlike Qwen2.5-Omni’s fixed 2-second chunks, Qwen3-Omni aligns representations directly by temporal IDs.Source: §2.3 · p. 5

2.4 Speech Generation

• Talker predicts the zeroth RVQ codebook from aggregated frame features, then its MTP module predicts residual codebooks for detailed, expressive speech.A lightweight causal ConvNet reconstructs the waveform, reducing latency and FLOPs while achieving superior audio fidelity to more complex DiT-based vocoders.Source: §2.4 · p. 5

2.5 Designs for Streaming and Concurrency

• Below 1, Generation RTF remains across concurrency levels, enabling continuous streaming audio responses.The left-context-only codec decoder can synthesize waveform after each Talker token, while asynchronous chunk prefilling and lightweight MTP and ConvNet modules support low-latency, high-concurrency inference.Source: §2.5 · pp. 6–7

Pretraining data and prompts

• Training spans image-text, video-text, audio-text, video-audio, video-audio-text, and pure-text data, using varied natural-language prompts and unimodal plus cross-modal data from early pretraining.The wider prompt range is intended to improve generalization and instruction following compared with Qwen2.5-Omni’s single prompt per task.Source: §3 · p. 7

Encoder Alignment Stage (S1)

• In Encoder Alignment Stage (S1), the LLM is fixed while vision and audio encoders are trained, with adapters trained before the encoders.The LLM starts from Qwen3, the vision encoder from Qwen3-VL, and the audio encoder from AuT; joint encoder-adapter training with a frozen LLM was avoided because it may degrade perception capabilities.Source: §3 · p. 7

General Stage (S2)

• General Stage (S2) uses approximately 2 trillion tokens: 0.57 trillion text, 0.77 trillion audio, 0.82 trillion image, 0.05 trillion video, and 0.05 trillion video-audio tokens.The diverse multimodal data and tasks are intended to strengthen understanding and interaction across auditory, visual, textual, and audiovisual information.Source: §3 · p. 7

Long Context Stage (S3)

• Increasing the maximum token length from 8,192 to 32,768, while raising the proportions of long audio and long video in training data, significantly improved the model’s understanding of long-sequence data.Source: §3 · p. 7

4.1 Thinker

• Thinker uses three post-training stages over ChatML text, visual, audio, and mixed-modality dialogues, beginning with lightweight SFT that adapts pretrained representations while preserving architectural consistency.The SFT schema diverges from pretraining while preserving the completeness of pretrained features for efficient knowledge transfer.Source: §4.1 · pp. 7–8
• Strong-to-Weak Distillation further improves Thinker through off-policy teacher-response distillation followed by on-policy student training aligned to teacher logits using KL divergence (Yang et al., 2025a).On-policy sequences come from student-generated responses to sampled prompts; Qwen3-32B or Qwen3-235B-A22B provides the teacher logits.Source: §4.1 · p. 8
• GSPO enhances Thinker’s capabilities and stability across text, image, video, and audio using rule-based or model-based rewards (Zheng et al., 2025).Rule-based rewards assess verifiable tasks; Qwen3 judges general tasks and Qwen2.5-VL judges visually grounded tasks, with reference answers supplied where applicable.Source: §4.1 · p. 8

4.2 Talker

• Talker’s four-stage training enables simultaneous speech and text responses, progressing from multimodal speech training through refinement, preference optimization, and speaker fine-tuning.Hundreds of millions of contextual speech examples establish a mapping to speech; continual pretraining reduces hallucinations and improves speech quality, while long-context training supports complex inputs. Multilingual preference pairs guide DPO (Rafailov et al., 2023), and speaker fine-tuning improves voice specificity, naturalness, expressiveness, and controllability.Source: §4.2 · p. 8

4.3 Captioner

• Fine-tuning Qwen3-Omni-30B-A3B on large-scale detailed audio descriptions produces the Captioner, which generates detailed, low-hallucination captions for arbitrary audio inputs.The model addresses the relative neglect of audio captioning compared with visual captioning; qualitative examples span diverse acoustic scenarios.Source: §4.3 · p. 8

5 Evaluation

• The evaluation includes Instruct and Thinking models plus Flash variants designed for computational efficiency, performance efficacy, and dialect support.Results are organized into understanding (X→Text) and speech generation (X→Speech).Source: §5 · p. 9

5.1.1 Performance of Text→Text

• Despite fewer parameters, Qwen3-Omni-30B-A3B-Instruct surpasses Qwen3-235B-A22B Non-Thinking and GPT-4o-0327 on five named benchmarks; Thinking is comparable to Gemini-2.5-Flash-Thinking and Qwen3-235B-A22B Non-Thinking.The report also states that Qwen3-Omni’s textual capabilities are on par with Qwen3-30B-A3B-Instruct-2507 and Qwen3-30B-A3B-Thinking-2507; the evaluation covers general, reasoning, coding, alignment, agent, and multilingual tasks.Source: §5.1.1 · p. 9, §5.1 · p. 9, Table 4 · p. 10

5.1.2 Performance of Audio→Text

• Qwen3-Omni-Instruct achieves state-of-the-art En & Zh ASR and lyric ASR, with better or comparable multilingual ASR and S2TT performance versus specialist and generalist models.The evaluation covers ASR and S2TT; Thinking-model ASR and S2TT results are deferred to Appendix 9.1.Source: Table 6 · p. 10, §5.1.2 · pp. 10–11
• 89.5 is Qwen3-Omni-Thinking’s VoiceBench average, below only Gemini-2.5-Pro’s 89.6; Qwen3-Omni also outperforms named closed-source models on MMAU and MMSU.On MMAU it surpasses Gemini-2.5-Pro and Gemini-2.5-Flash; on MMSU it surpasses Gemini-2.5-Flash and GPT-4o-Audio.Source: Table 7 · p. 11, §5.1.2 · p. 11
• Qwen3-Omni-Instruct achieves state-of-the-art RUL-MuchoMusic performance and surpasses audio-language and specialist models on GTZAN, MTG-Jamendo, and MagnaTagATune.For multi-label classification on MTG-Jamendo and MagnaTagATune, the comparison uses micro F1 rather than AP/AUROC because language models output discrete labels without calibrated scores.Source: Table 8 · p. 11, §5.1.2 · p. 11

5.1.3 Performance of Vision →Text

• Qwen3-Omni-Instruct performs comparably to Qwen2.5-VL-72B and surpasses GPT-4o and Gemini-2.0-Flash on named Math and STEM tasks.The cited tasks are MMMU-Pro overall, MathVista mini, and Math-Vision full; the evaluation also covers visual QA, document understanding, counting, and long-video understanding.Source: §5.1 · p. 9, §5.1.3 · pp. 11–12, Table 9 · p. 12
• Qwen3-Omni-Thinking outperforms the Instruct baseline by 4.4 points on Math and STEM benchmarks, while matching substantially larger baselines.The report identifies suboptimal long-video performance, attributing it to limited positional extrapolation and context length.Source: Table 10 · p. 12, §5.1.3 · p. 12

5.1.4 Performance of AudioVisual Video→Text

• Qwen3-Omni-Instruct achieves state-of-the-art WorldSense performance, while the Thinking model shows enhanced performance on audiovisual reasoning benchmarks.The evaluation uses WorldSense for audiovisual integration and DailyOmni and VideoHolmes for higher-order audiovisual reasoning.Source: §5.1 · p. 9, Table 11 · p. 13, Table 12 · p. 13

5.2 Evaluation of X→Speech

• The speech-generation evaluation focuses on text-conditioned generation, assessing zero-shot content consistency and speaker similarity, multilingual consistency and similarity, and cross-lingual content consistency.The cited evaluations use SEED, the MiniMax multilingual test set, and CV3-Eval, respectively.Source: §5.2 · p. 13

5.2.1 Evaluation of Zero-Shot Speech Generation

• With RL optimization, Qwen3-Omni achieves the best performance on test-en and improves generation stability while remaining highly competitive against state-of-the-art zero-shot TTS systems.The zero-shot evaluation measures content consistency (WER) and speaker similarity (SIM) on the Seed-TTS test set.Source: §5.2.1 · p. 13, Table 13 · p. 13

5.2.2 Evaluation of Multilingual Speech Generation

• Qwen3-Omni surpasses MiniMax-Speech and ElevenLabs Multilingual v2 in Chinese, English, and French, with competitive results in the remaining evaluated languages.The model supports speech generation across 10 languages; the evaluation uses the MiniMax multilingual test set.Source: §5.2.2 · p. 14, Table 14 · p. 14

5.2.3 Evaluation of Cross-Lingual Speech Generation

• Qwen3-Omni outperforms CosyVoice3 in any-to-en and any-to-ko voice cloning, and achieves comparable any-to-ja performance without text normalization.CosyVoice3 converts all Japanese characters into phonetic kana; Qwen3-Omni's any-to-ja comparison is made without that text normalization.Source: §5.2.3 · p. 14, Table 15 · p. 14

6 Evaluating Non-Degradation Across Modalities

• The controlled comparison uses matched-parameter text-only, vision-only, and Omni models, with the Omni model additionally trained on audio and audio-visual data.The Omni model shares the unimodal baselines’ text and vision corpora; learning-rate schedules, batch sizes, and effective modality training epochs were aligned.Source: §6 · pp. 14–15
• Text substantially improves vision and audio performance, while adding vision or audio yields no measurable language gains; audio consistently improves MMMU and OCR performance.The authors note that prohibitive experimental cost prevented a comprehensive sweep across model scales.Source: §6 · p. 15
• Early multimodal training improves performance across text, vision, and video benchmarks, with joint multimodal training also enhancing single-modality performance.Table 16 covers text tasks including general, math and STEM, coding, and multilingual tasks; vision tasks including college-level problems and OCR; and video understanding.Source: Table 16 · p. 15, §6 · p. 15
• The models were trained under the same schedule with identical modality-specific datasets and exactly matched training compute in FLOPs.This experimental design was intended to ensure a fair comparison despite the heterogeneous objectives and optimization needs of different modalities.Source: §6 · p. 14, Table 16 · p. 15

• The paper introduces Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, Qwen3-Omni-Flash-Instruct, and Qwen3-Omni-Flash-Thinking.Source: §7 · p. 15
• 32 audio-processing and dialogue benchmarks achieve open-source state-of-the-art performance, with Qwen3-Omni-30B-A3B comparable to or better than Gemini-2.5-Pro.It also matches or surpasses latest same-size unimodal Qwen models on text and vision benchmarks.Source: §7 · pp. 15–16
• 234 ms is Qwen3-Omni's end-to-end first-packet latency at the 30B-A3B scale, supporting interactive use.It supports 119 text languages, speech recognition in 19 languages, speech synthesis in 10 languages, and audio understanding and interactive sessions up to 40 minutes.Source: §7 · p. 16
• The Qwen3-Omni-30B-A3B Thinking variant gains further performance on complex tasks spanning text, vision, and audio-visual reasoning.Source: §7 · p. 16
• The authors present Qwen3-Omni as first evidence, to their knowledge, that fully integrated end-to-end multimodal training can avoid degrading core language capability and other modalities.Source: §7 · p. 16
• Qwen3-Omni offers stronger cross-modal reasoning, lower end-to-end latency, and lower system complexity and cost than cascaded pipelines.Source: §7 · p. 16
• Future work targets multi-speaker ASR, video OCR, audiovisual proactive learning, and stronger support for agent-based workflows and function calling.Source: §7 · p. 16

9.1 More Evaluation on Speech and Music Understanding

• Qwen3-Omni-Thinking is outperformed by Instruct on ASR/S2TT and music understanding, suggesting reasoning does not improve these perception-heavy tasks and may increase hallucinations.The prose identifies Tables 17 and 18 as the comparisons but supplies no score values.Source: §9.1 · p. 17, Table 17 · p. 17, Table 18 · p. 17

9.2 Qualitative Results from Qwen3-Omni-30B-A3B-Captioner

• The expressive-speech case describes a theatrical Mandarin monologue mixing comic bravado and self-deprecation, with clear studio recording, subtle hiss and hum, and a deliberate cut to silence.The example portrays Taiyi Zhenren through dramatic inflection, expressive pacing, and clean editing.Source: §9.2 · p. 18
• The complex-sound case describes a cinematic, nonverbal mechanical disaster that culminates in an impact and leaves a single survivor breathing and groaning amid a vast industrial or futuristic setting.Orchestral and electronic tension builds around machinery, an explosion-like impact, and its aftermath.Source: §9.2 · p. 19
• The mixed-audio case combines a vast metallic machinery soundscape with expressive dialogue and playful familial banter, ending in a synthesized musical sting that suggests a science-fiction or fantasy scene.The example includes distant and close voices, spatial cues, mechanical sounds, and a brief musical transition.Source: §9.2 · p. 20