PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3.5-Omni Technical Report

Qwen Team

arXiv:2604.15804v2cs.CLeess.AS

TL;DR

Qwen3.5-Omni addresses the limited agency, real-time interaction, and cross-modal reasoning of existing models by unifying multimodal understanding, generation, and action. It combines a Thinker–Talker architecture with Hybrid-Attention MoE, long-context modeling, streaming speech improvements, and expanded multilingual support. Across 215 audio and audio-visual subtasks and benchmarks, Qwen3.5-Omni-Plus achieves SOTA results, surpasses Gemini-3.1 Pro across general audio tasks, and reaches its level in overall audio-visual understanding.

  • Problem

    Existing models have limited scalable agentic behavior, real-time interaction, autonomous tool use, and cross-modal reasoning for practical deployment.

  • Method

    Qwen3.5-Omni uses a Thinker–Talker omnimodal architecture with Hybrid-Attention MoE, 256k-context modeling, ARIA-aligned streaming speech, and expanded multilingual training.

  • Results

    Across 215 audio and audio-visual understanding, reasoning, and interaction subtasks and benchmarks, Qwen3.5-Omni-Plus achieves SOTA results, surpasses Gemini-3.1 Pro on general audio tasks, and reaches its level in overall audio-visual understanding.

  • Takeaways & Limitations

    The model supports structured audio-visual captioning, comprehensive real-time interaction, autonomous tool use, and Audio-Visual Vibe Coding from audio-visual instructions.

Abstract

from arXiv · show

In this work, we present Qwen3.5-Omni, the latest advancement in the Qwen-Omni model family. Representing a significant evolution over its predecessor, Qwen3.5-Omni scales to hundreds of billions of parameters and supports a 256k context length. By leveraging a massive dataset comprising heterogeneous text-vision pairs and over 100 million hours of audio-visual content, the model demonstrates robust omni-modality capabilities. Qwen3.5-Omni-plus achieves SOTA results across 215 audio and audio-visual understanding, reasoning, and interaction subtasks and benchmarks, surpassing Gemini-3.1 Pro in key audio tasks and matching it in comprehensive audio-visual understanding. Architecturally, Qwen3.5-Omni employs a Hybrid Attention Mixture-of-Experts (MoE) framework for both Thinker and Talker, enabling efficient long-sequence inference. The model facilitates sophisticated interaction, supporting over 10 hours of audio understanding and 400 seconds of 720P video (at 1 FPS). To address the inherent instability and unnaturalness in streaming speech synthesis, often caused by encoding efficiency discrepancies between text and speech tokenizers, we introduce ARIA. ARIA dynamically aligns text and speech units, significantly enhancing the stability and prosody of conversational speech with minimal latency impact. Furthermore, Qwen3.5-Omni expands linguistic boundaries, supporting multilingual understanding and speech generation across 10 languages with human-like emotional nuance. Finally, Qwen3.5-Omni exhibits superior audio-visual grounding capabilities, generating script-level structured captions with precise temporal synchronization and automated scene segmentation. Remarkably, we observed the emergence of a new capability in omnimodal models: directly performing coding based on audio-visual instructions, which we call Audio-Visual Vibe Coding.

1 Introduction

Qwen3.5-Omni targets practical omnimodal agency by unifying multimodal understanding, reasoning, generation, and action. It combines scalable architecture and long-context support with new interaction capabilities and broad benchmark performance.

  • Existing models are limited in scalable agency, real-time interaction, autonomous tool use, and cross-modal reasoning.
  • Qwen3.5-Omni unifies text, image, audio, and audio-visual understanding with autonomous WebSearch, FunctionCall, speech output, and streaming interaction.The Plus and Flash instruct variants support 256k-token long-context input.
  • Hybrid-Attention MoE designs, 256k context, multi-codebook speech synthesis, ARIA, and expanded multilingual training are the report’s key upgrades.The model supports more than 10 hours of audio and over 400 seconds of 720P audio-visual content at 1 FPS.
  • New capabilities include structured audio-visual captioning, real-time voice interaction, autonomous tool use, and Audio-Visual Vibe Coding.Audio-Visual Vibe Coding generates executable code directly from audio-visual instructions.
  • Across 215 audio and audio-visual subtasks and benchmarks, Qwen3.5-Omni-Plus achieves SOTA results, surpasses Gemini-3.1 Pro on general audio tasks, and reaches its level in overall audio-visual understanding.

2 Architecture

Qwen3.5-Omni uses a Thinker–Talker multimodal architecture designed for scalable, long-context, low-latency interaction. Its encoders, temporal representation, streaming speech pipeline, ARIA alignment, and deployment optimizations support extended multimodal inputs and real-time output.

  • Overview: The Thinker handles multimodal understanding and text generation, while the Talker generates streaming speech from Thinker representations.Talker predicts multi-codebook speech sequences, and Code2Wav incrementally synthesizes waveforms frame by frame.
  • Overview: Hybrid MoE improves scalability and balances capacity and efficiency across multimodal understanding and generation.
  • Perceiving multimodal inputs: The Thinker supports 256k tokens, 10 hours of audio, or 400 seconds of 720P video at 1 FPS through unified multimodal processing and temporal timestamps.Audio and video inputs are interleaved, with explicit timestamps supporting temporal perception.
  • Streaming and concurrency: Chunk-wise Thinker processing and streaming Talker generation enable low-latency end-to-end multimodal conversation.The architecture and associated first-packet latency are summarized under audio and video settings.
  • Speech generation: ARIA dynamically aligns text and speech units, mitigating tokenization-rate mismatch and reducing skipped words, incorrect pronunciations, and ambiguous number rendering.It uses an adaptive rate constraint within a unified single-channel formulation.
  • Audio encoding: AuT is a transformer audio encoder trained on 40 million hours of audio-text pairs and produces audio tokens at a 6.25Hz rate.Its training uses more multilingual data spanning more than 20 languages.
  • Speech generation: Talker’s RVQ and multi-token prediction modules model acoustic detail, while a causal ConvNet reconstructs high-fidelity waveforms with low latency.Context from Thinker enables modulation of prosody, loudness, and emotion during dialogue.

3 Pretraining

Qwen3.5-Omni is pretrained on diverse multilingual, multimodal data through staged training that aligns encoders, expands general multimodal learning, and extends context length. Timestamp-based temporal encoding addresses limitations of prior temporal position IDs, while longer-sequence training improves long-sequence understanding.

  • Multilingual pretraining combines image-text, video-text, audio-text, video-audio, video-audio-text, and pure-text corpora from the early training stages.
  • Timestamp strings in seconds replace absolute temporal position IDs to improve long-range temporal modeling and reduce frame-rate-specific data-construction costs.
  • Three pretraining stages progressively train encoders, all model parameters, and longer-context multimodal representations.
  • Approximately 4 trillion tokens are distributed across text, audio, image, video, and video-audio data during general pretraining.
  • 262,144 tokens is the expanded maximum sequence length, with more long-audio and long-video data; experiments report improved long-sequence understanding.

4 Post-training

Post-training uses staged procedures for Thinker and Talker to preserve multimodal capability, improve response quality, and produce natural, contextually appropriate speech. The Talker pipeline combines multilingual pretraining, long-context curation, preference optimization, and speaker fine-tuning.

  • Thinker post-training uses three stages to preserve cross-modal capabilities, improve audio-query response quality, and optimize interaction.
  • Specialist teacher models trained with supervised fine-tuning and reinforcement learning provide domain-specific distillation for text, vision, and audio tasks.
  • Talker training uses four stages and ChatML-organized data to generate natural, contextually appropriate spoken responses jointly with text.
  • More than 20 million hours of multilingual speech paired with multimodal context support instruction-following speech generation and paralinguistic alignment.
  • 64k tokens is the Talker long-context extension, while curation and Qwen3-Omni-Captioner reduce noisy-data hallucinations and improve speech naturalness.
  • DPO, rule-based rewards, GSPO, and speaker fine-tuning align behavior and improve training stability, speaker fidelity, naturalness, expressiveness, and controllability.

5 Evaluation

The evaluation covers two Qwen3.5-Omni variants and separates multimodal understanding from speech generation. Results are organized as X→Text and X→Speech evaluations.

  • Two model variants, Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus, are evaluated.
  • The evaluation is divided into understanding (X→Text) and speech generation (X→Speech).

5.1 Evaluation of X→Text

X→Text evaluation spans text, audio, vision, and audio-visual video inputs, including understanding, reasoning, dialogue, captioning, and tool use. Across modalities, Qwen3.5-Omni is competitive with or stronger than relevant baselines, with particularly strong audio and audio-visual results.

  • Text→Text: Text→Text evaluation covers knowledge, instruction following, long-context understanding, STEM, reasoning, and general agent capabilities.
  • Audio→Text: Audio→Text evaluation covers audio understanding, end-to-end speech dialogue, speech-to-text translation, and automatic speech recognition.
  • Vision→Text: Vision→Text evaluation includes mathematical, STEM, general visual question answering, and video-understanding benchmarks.
  • Audio-Visual Video→Text: Audio-visual evaluation includes text-query understanding, audio-query interaction, captioning, and tool use across dedicated benchmarks.
  • Text→Text: Qwen3.5-Omni-Plus matches Qwen3.5-Plus-Instruct across multiple text dimensions and is slightly better on instruction following.
  • Audio→Text: Qwen3.5-Omni surpasses or matches Gemini-3.1 Pro across audio understanding, dialogue, translation, and recognition benchmarks.
  • Vision→Text: Qwen3.5-Omni-Plus is comparable to Qwen3.5-Plus-Instruct overall and stronger on short- and long-video understanding tasks.
  • Audio-Visual Video→Text: 57.2% on OmniGAIA reflects enhanced audio-visual tool-use capability, alongside detailed audio, visual, and audio-visual captioning.

5.2 Evaluation of X→Speech

The evaluation covers zero-shot, multilingual, cross-lingual, and custom-voice speech generation, with results indicating strong content fidelity, speaker similarity, and cross-lingual performance.

  • Zero-Shot Speech Generation: Zero-shot speech generation achieves highly competitive SEED-TTS performance, with RLHF producing a test-en WER of 1.26.The evaluation measures content consistency using WER.
  • Multilingual Speech Generation: Qwen3.5-Omni achieves the lowest WER in 22 of 29 evaluated multilingual languages and the highest speaker similarity in most evaluated languages.It outperforms MiniMax-Speech and ElevenLabs overall while remaining competitive on the remaining languages.
  • Multilingual Speech Generation: The model generalizes multilingual speech generation to nine additional internal-test languages, maintaining strong performance across all evaluated languages.The internal evaluation extends beyond the public benchmark languages.
  • Cross-Lingual Speech Generation: Cross-lingual voice cloning achieves the best performance in 10 of 12 directions, including a zh-to-ko error-rate reduction from 14.4 to 4.03 versus CosyVoice3.The reported reduction corresponds to approximately 72% relative reduction.
  • Custom-Voice Speech Generation: Custom-voice generation transfers target-speaker characteristics to all 29 evaluated languages, achieving the best WER in 10 languages.The model was fine-tuned only on monolingual data but showed cross-lingual generalization.

6 Conclusion

Qwen3.5-Omni unifies multimodal understanding, reasoning, generation, and action across text, images, audio, and audio-visual inputs. Its reported results support a unified omnimodal system for real-time interaction and action.

  • Conclusion: Qwen3.5-Omni unifies understanding, reasoning, generation, and action across text, images, audio, and audio-visual inputs.The model is built on the Thinker–Talker framework.
  • Conclusion: Hybrid-Attention MoE architectures, 256k long-context modeling, multi-codebook prediction, ARIA, and expanded multilingual speech support underpin the system.These are presented as the report’s principal technical advances.
  • Conclusion: The model enables controllable audio-visual captioning, comprehensive real-time interaction, autonomous tool use, and audio-visual code generation.These capabilities are identified as three key capability areas, with tool use and code generation included under agentic behavior.
  • Conclusion: Qwen3.5-Omni achieves state-of-the-art or highly competitive performance across broad audio and audio-visual benchmarks while maintaining same-scale Qwen text and vision capabilities.The conclusion frames these results as evidence that native omnimodal training can produce unified systems that perceive, reason, interact, and act in real time.

7 Authors

The author list is presented in alphabetical order, with an asterisk marking the corresponding author.

  • Authors: Authors are listed in alphabetical order.The passage also states that an asterisk denotes the corresponding author.

8 Appendix

The appendix reports multilingual ASR and translation results, showing strong Qwen3.5-Omni performance across languages and especially challenging Asian-language settings.

  • Multilingual ASR: Qwen3.5-Omni-Plus achieves the lowest average FLEURS WER at 6.6%, versus 7.3% for Gemini-3.1-Pro and 10.4% for GPT-4o-Transcribe.It obtains the best performance in the majority of languages, with notable margins in Cantonese, Thai, and Vietnamese.
  • Multilingual ASR: Qwen3.5-Omni-Flash records an average FLEURS WER of 10.8%, close to Gemini-3-Flash at 10.5%.Its Cantonese WER is 3.1% versus 10.8% for Gemini-3-Flash.
  • Multilingual Translation: Qwen3.5-Omni-Plus exceeds Gemini-3.1-Pro in average BLEU for English-to-XX and Chinese-to-XX translation, scoring 33.8 versus 31.8 and 21.4 versus 19.6.It also leads in selected xx2en pairs and substantially outperforms Gemini in Cantonese, Korean, and Japanese.
  • Multilingual Translation: Qwen3.5-Omni-Flash substantially surpasses Gemini-3-Flash in Cantonese translation, including 37.5 versus 22.4 for xx2zh and 37.3 versus 26.7 for en2xx.It also performs better in Japanese and Korean xx2zh tasks.
Loading 2604.15804v2…