Qwen3.5-Omni Technical Report

1 Introduction

• The introduction motivates omnimodal agents by noting that existing models remain largely passive and are limited in scalable agency, real-time interaction, autonomous tool use, and cross-modal reasoning.Human interaction instead integrates visual, auditory, and linguistic information and produces text, speech, and goal-directed actions.Source: §1 · p. 1
• Qwen3.5-Omni is natively pretrained on text, visual data, and more than 100 million hours of audio-visual data, and supports text, images, audio, and audio-visual content.Its Plus and Flash variants are instruct models with 256k-token input context; the model can autonomously invoke WebSearch, execute complex FunctionCall, generate speech, and interact in real time.Source: §1 · p. 1
• Thinker and Talker use Hybrid-Attention MoE, while 256k-token context supports over 10 hours of audio and 400 seconds of 720P audio-visual input at 1 FPS.A multi-codebook codec enables single-frame immediate synthesis; ARIA dynamically aligns text and speech units during streaming to improve naturalness and robustness. Speech recognition covers 113 languages and dialects, and synthesis covers 36.Source: §1 · pp. 1–2
• The model adds controllable structured audio-visual captions, comprehensive real-time interaction, and native omnimodal agentic behavior, including Audio-Visual Vibe Coding.Captions can include screenplay-level descriptions, segmentation, timestamps, and character relationships to audio; interaction supports semantic interruption, voice controls, and cloning from user samples. Vibe Coding generates executable code directly from audio-visual instructions.Source: §1 · p. 2
• Across 215 audio and audio-visual tasks, Qwen3.5-Omni-Plus achieves SOTA results, surpasses Gemini-3.1 Pro across audio tasks, and reaches its level in overall audio-visual understanding.It also maintains text and visual performance without degradation relative to same-size single-model Qwen counterparts.Source: §1 · p. 2

2.1 Overview

• The Hybrid MoE Thinker-Talker processes inputs up to 256k tokens, 10 hours of audio, or 400 seconds of 720P video at 1 FPS.Thinker handles text and multimodal representations; Talker generates streaming speech tokens, with an MTP module predicting residual codebooks and a renderer synthesizing waveforms frame by frame.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Audio Transformer (AuT)

• AuT encodes audio at 6.25 Hz after 16× downsampling, using a from-scratch transformer trained on 40 million hours of audio-text pairs.Training incorporates more than 20 languages, with Chinese, English, and multilingual data in a 3.5:3.5:3 proportion; dynamic attention windows support both real-time prefill caching and offline understanding.Source: §2.2 · p. 4, Figure 3 · p. 4

2.3 Perceivation

• Explicit timestamps on video and audio-video patches, plus randomly inserted audio timestamps, improve temporal alignment and robust long-context multimodal perception.The model aligns representations to absolute time using temporal IDs and contiguous position numbering, supporting streaming inputs of arbitrary duration.Source: §2.3 · pp. 4–5

2.4 Speech Generation

• ARIA aligns text and speech in a unified interleaved stream, while a dedicated Talker prompt enables zero-shot voice cloning and controllable speech generation.Its adaptive rate constraint bounds the cumulative speech-to-text token ratio for every generated prefix, supporting flexible alignment across languages and coherent speech continuation from arbitrary text-token prefixes.Source: §2.4 · p. 5, §2.5 · p. 6

2.5 Designs for Streaming and Concurrency

• Chunked prefilling reduces TTFT, while Gated Delta Net reduces KV-cache I/O for long audio-video contexts and supports higher serving concurrency.Table 2 reports theoretical first-packet latency across concurrency levels; the report says latency and decoding efficiency remain stable as concurrency increases, but the variants’ figures are not intended for strict horizontal comparison.Source: §2.5 · p. 6, Table 2 · p. 6

• Pretraining spans multiple languages and dialects and image-text, video-text, audio-text, video-audio, video-audio-text, and text data, with unimodal and cross-modal training from the outset.Following Qwen3-Omni (Xu et al., 2025b), the model uses a wider range of natural-language prompts to support generalization and instruction following.Source: §3 · p. 6
• To avoid sparse, excessively large absolute temporal IDs and costly frame-rate sampling, video and audiovisual patches receive timestamps as text, while audio gets timestamps at random intervals.This modestly increases context length while helping the model learn and interpret timecodes more effectively and precisely.Source: §3 · pp. 6–7

Encoder Alignment Stage (S1)

• Pretraining began with the LLM frozen while vision and audio encoders learned from image-text and audio-text pairs, then unfroze all parameters for broader multimodal training at sequence length 32,768.The LLM and vision encoder were initialized from Qwen3.5, while the audio encoder was initialized with AuT; the encoders first trained their adapters, then the encoders.Source: §3 · p. 7

General Stage (S2)

• The general stage used approximately 4 trillion tokens: 0.92 trillion text, 1.99 trillion audio, 0.95 trillion image, 0.14 trillion video, and 0.29 trillion video-audio.More diverse multimodal data and tasks were intended to enhance understanding and interaction across auditory, visual, textual, and audiovisual information.Source: §3 · p. 7

Long Context Stage (S3)

• Increasing the maximum token length from 32,768 to 262,144 and raising the share of long audio and video data significantly improved long-sequence understanding.These changes were made in the final pretraining stage.Source: §3 · pp. 7–8

4.1 Thinker

• Specialist distillation combines domain-specific data from independently SFT- and RL-trained teachers, fine-tuned from the Qwen-3.5 base checkpoint, into one unified model.Teacher domains include text tasks such as coding, reasoning, and agentic tasks, as well as vision and audio.Source: §4.1 · p. 8
• On-policy distillation uses higher-quality text-conditioned responses as targets for paired audio queries, improving audio response quality and modality-consistent generation.The method targets a remaining quality gap, especially in speech dialogue, by transferring text-conditioned fluency, reasoning, and task completion to audio-conditioned outputs.Source: §4.1 · p. 8
• Interaction-Aligned RL trains on multi-turn trajectories with user-experience rewards to address language code-switching, persona inconsistency, and weakened instruction-following in extended conversations.The stage aims to make prolonged interactions more stable, consistent, and aligned.Source: §4.1 · p. 8

4.2 Talker

• The Talker’s general stage trains on more than 20 million hours of multilingual speech paired with multimodal context, including instruction-following speech generation.Diverse tasks support contextual reasoning and paralinguistic alignment beyond a monotonic mapping from multimodal representations to speech.Source: §4.2 · p. 8
• The Talker’s long-context stage curates high-quality data for continual pre-training, reducing noisy-data hallucinations and improving speech naturalness and quality while extending context to 64k tokens.Qwen3-Omni-Captioner augments this stage, which also supports more contextually grounded responses to long, complex inputs.Source: §4.2 · p. 8
• The Talker’s reinforcement-learning stage uses multilingual human-annotated preference pairs with DPO, plus rule-based rewards and GSPO for capability and training stability (Rafailov et al., 2023; Zheng et al., 2025).The objective is to align behavior with human preferences across diverse tasks.Source: §4.2 · p. 9
• Lightweight speaker fine-tuning enables the Talker to capture target-speaker characteristics while improving speech naturalness, expressiveness, and controllability.The fine-tuning is performed on top of the base model.Source: §4.2 · p. 9

5 Evaluation

• The evaluation compares two model variants: Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus.The results are organized into two categories: understanding (X→Text) and speech generation (X→Speech).Source: §5 · p. 9
• Results are divided into understanding (X→Text) and speech generation (X→Speech).These are the two main evaluation categories.Source: §5 · p. 9

5.1.1 Performance of Text→Text

• Qwen3.5-Omni-Plus performs on par with Qwen3.5-Plus-Instruct across text dimensions and is slightly better at instruction following.The comparison covers knowledge, instruction following, long context, STEM, reasoning, and general agent tasks; the report credits OPD and interaction-aligned RL as possible contributors to the instruction-following improvement.Source: §5.1.1 · p. 10, Table 4 · p. 10

5.1.2 Performance of Audio→Text

• Qwen3.5-Omni outperforms Gemini-3.1 Pro on MMAU, MMSU, RUL-MuchoMusic, SongFormBench, VoiceBench, S2TT, and ASR, with comparable MMAR and other dialogue results.The evaluation spans audio understanding, end-to-end speech dialogue, translation, and recognition; Table 5 covers the audio benchmark comparison.Source: §5.1.2 · p. 10, Table 5 · p. 11

5.1.3 Performance of Vision →Text

• Qwen3.5-Omni-Plus matches Qwen3.5-Plus-Instruct overall on vision tasks and performs better on short- and long-video understanding.The report attributes the dynamic-visual results to real-world perception and suggests joint video-audio training is effective, with audio and visual streams intrinsically coupled.Source: §5.1.3 · pp. 10–11, Table 6 · p. 12

5.1.4 Performance of Audio-Visual Video→Text

• Qwen3.5-Omni leads DailyOmni, is comparable on AVUT, surpasses Gemini-3.1 Pro on Qualcomm IVD, and scores 57.2% on OmniGAIA.It also provides detailed audio, visual, and audio-visual captions; the report describes stronger tool-use capability in this version.Source: §5.1.4 · p. 11

5.2.1 Evaluation of Zero-Shot Speech Generation

• After RLHF optimization, Qwen3.5-Omni achieves the best test-en WER on SEED-TTS, with a WER of 1.26.The evaluation compares zero-shot TTS systems on SEED-TTS; lower WER indicates better content consistency.Source: Table 8 · p. 13, §5.2.1 · p. 13

5.2.2 Evaluation of Multilingual Speech Generation

• Qwen3.5-Omni achieves the lowest WER in 22 of 29 evaluated languages, the highest speaker similarity in most evaluated languages, and outperforms MiniMax-Speech and ElevenLabs overall.On the internal test set covering an additional 9 languages, it also performs strongly across all evaluated languages.Source: §5.2.2 · pp. 13–14, Table 9 · p. 14, Table 10 · p. 14

5.2.3 Evaluation of Cross-Lingual Speech Generation

• Qwen3.5-Omni achieves the best performance in 10 of 12 cross-lingual directions; for zh-to-ko, its error rate falls from 14.4 to 4.03 versus CosyVoice3, an approximately 72% relative reduction.It sets a new state of the art on most English-, Japanese-, and Korean-targeted pairs.Source: §5.2.3 · pp. 14–15, Table 11 · p. 15

5.2.4 Evaluation of Custom-Voice Speech Generation

• Despite fine-tuning only on monolingual data, Qwen3.5-Omni achieves the best WER in 10 languages and remains competitive in many others for custom-voice generation.It transfers target-speaker characteristics to all 29 evaluated languages, with WERs of 3.306 for Japanese and 1.309 for Korean.Source: Table 12 · p. 16, §5.2.4 · p. 16

6 Conclusion

• Qwen3.5-Omni achieves state-of-the-art or highly competitive performance across audio and audio-visual benchmarks while maintaining strong text and vision capabilities of same-scale Qwen models.The report presents these results as evidence that scaling native omnimodal training can support systems that perceive, reason, interact, and act in real time.Source: §6 · p. 16
• Qwen3.5-Omni combines efficient Hybrid-Attention MoE architectures, 256k long-context modeling, multi-codebook codec prediction with ARIA, and expanded multilingual speech support.The model is built on the Thinker–Talker framework.Source: §6 · p. 16
• The model enables controllable audio-visual captioning, comprehensive real-time interaction, and native omnimodal agentic behavior through autonomous tool use and audio-visual code generation.The conclusion presents Qwen3.5-Omni as a foundation for future research on general-purpose omnimodal agents.Source: §6 · p. 16

8.1 Detailed Multilingual Evaluation Results

• 6.6% average WER: Qwen3.5-Omni-Plus beats Gemini-3.1-Pro (7.3%) and GPT-4o-Transcribe (10.4%) on FLEURS, leading in most languages.It also posts 2.2% WER in Cantonese versus 6.3% for Gemini-3.1-Pro.Source: §8.1 · p. 25, Table 13 · p. 26
• Flash averages 10.8% WER versus Gemini-3-Flash’s 10.5%, but reaches 3.1% in Cantonese versus 10.8% and remains strong in Japanese and Korean.In translation, Flash scores 37.5 vs. 22.4 BLEU in Cantonese xx2zh and 37.3 vs. 26.7 in en2xx, with further advantages in Japanese and Korean xx2zh.Source: §8.1 · p. 25, Table 13 · p. 26, Table 15 · p. 28
• 33.8 vs. 31.8 English-to-XX and 21.4 vs. 19.6 Chinese-to-XX average BLEU: Plus surpasses Gemini-3.1-Pro in many-to-many FLEURS translation.Gemini-3.1-Pro retains a slight edge in overall xx2zh averages, while Plus outperforms it in Cantonese, Korean, and Japanese.Source: §8.1 · p. 25, Table 14 · p. 27
• Plus leads xx2en Portuguese (49.4 vs. 47.7) and Indonesian (45.7 vs. 45.1), and exceeds Gemini-3.1-Pro in Cantonese by +15.6 BLEU.The passage also reports Plus advantages in Korean and Japanese.Source: §8.1 · p. 25, Table 15 · p. 28