Source-linked AI summary
Qwen2.5-Omni Technical Report
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, Junyang Lin
TL;DR
The paper addresses the challenge of unifying multimodal understanding with simultaneous text and speech responses in an end-to-end streaming system. It introduces temporal alignment, separated Thinker-Talker generation, and streaming-oriented components, and reports strong multimodal, speech-following, and speech-generation performance. The authors identify overlooked issues such as video OCR and audio-video collaborative understanding as requiring broader benchmarks and datasets.
Problem
Unifying text, image, video, and audio understanding while producing synchronized text and speech streams end to end remains challenging.
Method
The model combines block-wise multimodal encoders, TMRoPE for audio-video temporal alignment, and a jointly trained Thinker-Talker architecture for streaming text and speech generation.
Results
Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks, speech instruction following comparable to text input, and 1.42% WER on seed-tts-eval test-zh.
Takeaways & Limitations
The results position Qwen2.5-Omni as a unified model with strong multimodal understanding, voice-command following, and streaming speech generation.
Takeaways & Limitations
The authors identify video OCR and audio-video collaborative understanding as critical issues requiring more comprehensive benchmarks and research datasets.
Abstract
from arXiv · showhide
In this report, we present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. To synchronize the timestamps of video inputs with audio, we organize the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose \textbf{Thinker-Talker} architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness.
1 Introduction
Qwen2.5-Omni is introduced as a unified end-to-end model that processes multiple modalities and generates text and natural speech simultaneously in streaming format. Its design addresses multimodal synchronization, cross-output interference, and real-time streaming, while achieving strong benchmark performance.
- Model and challenges: Qwen2.5-Omni unifies multimodal perception with simultaneous streaming generation of text and natural speech.The model is designed to process text, images, videos, and audio within one system.
- Model and challenges: TMRoPE synchronizes audio and video by incorporating temporal information into their interleaved representation.Audio and video frames are organized sequentially in time order.
- Model and challenges: Thinker-Talker separates text and speech generation while allowing Talker to use Thinker’s high-level representations.Thinker generates text, whereas Talker generates streaming speech tokens, and the components are jointly trained end to end.
- Model and challenges: Block-wise multimodal encoding and architectural changes target real-time comprehension and reduced streaming latency.The report identifies efficient streaming and initial latency as central design requirements.
- Evaluation: Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks and strong speech-instruction-following and speech-generation results.The report also describes performance comparable to similarly sized Qwen2.5-VL and superior to Qwen2-Audio in the stated capabilities.
2 Architecture
Qwen2.5-Omni uses a unified Thinker-Talker architecture for multimodal understanding and simultaneous streaming text and speech generation. Its streaming design combines time-aligned multimodal encoding, block-wise input processing, and restricted-context audio decoding.
- 2.1 Overview: Thinker processes multimodal inputs and generates text representations, while Talker uses Thinker’s representations and text to produce streaming speech tokens.Talker directly receives Thinker’s high-dimensional representations and historical context, enabling end-to-end training and inference.
- 2.2 Perceivation: TMRoPE encodes temporal, height, and width positions, assigning absolute temporal positions to synchronize audio and video representations.Audio temporal IDs correspond to 40 ms intervals, while video temporal IDs advance frame by frame.
- 2.2 Perceivation: The time-interleaving method segments audiovisual representations into 2-second chunks, placing visual representations before audio representations within each chunk.This ordering lets the model receive visual and auditory information simultaneously in actual time order.
- 2.3 Generation: Talker combines Thinker representations with sampled text embeddings to anticipate speech tone and attitude during streaming generation.The speech codec and causal audio decoder support streaming audio-token generation without word-level or timestamp-level text alignment.
- 2.4 Designs for Streaming: Audio and visual encoders support block-wise temporal processing for multimodal prefilling, including 2-second audio attention blocks and merged visual tokens.The vision encoder uses flash attention and merges adjacent 2×2 tokens through an MLP.
- 2.4 Designs for Streaming: A sliding-window block-attention DiT restricts decoding context while converting audio codes into mel-spectrogram chunks and waveforms.The method uses Flow Matching followed by a modified BigVGAN, with contextual blocks retained for streaming quality.
3 Pre-training
Qwen2.5-Omni is pretrained through staged expansion from encoder alignment to broad multimodal learning and long-sequence training. The process combines diverse modality-paired data with progressively larger context lengths.
- 3 Pre-training: Qwen2.5-Omni uses three pretraining stages: encoder-focused alignment, full-parameter multimodal training, and 32,768-token sequence training.The first stage freezes the LLM, the second unfreezes all parameters, and the final stage uses longer sequences.
- 3 Pre-training: The pretraining corpus includes image-text, video-text, video-audio, audio-text, and text data, with natural-language prompts used to improve generalization and instruction following.Pure text data also helps maintain and improve language proficiency.
- 3 Pre-training: The initial stage initializes the LLM from Qwen2.5, uses the Qwen2.5-VL vision encoder and Whisper-large-v3 audio encoder, then trains adapters and encoders with the LLM fixed.This stage targets visual-textual and audio-textual correlations and alignments.
- 3 Pre-training: The second stage adds 800 billion image/video tokens, 300 billion audio tokens, and 100 billion video-audio tokens alongside broader multimodal tasks.Mixed multimodal data is used to deepen interaction among auditory, visual, and textual information.
- 3 Pre-training: 32,768-token training extends earlier 8,192-token limits with long audio and video data, and experiments report improved support for long-sequence data.The expanded training covers text, audio, image, and video inputs.
4 Post-training
Post-training adapts Qwen2.5-Omni with mixed-modality instruction data and a three-stage Talker procedure. The Talker stages target continuation, speech stability, and controllable naturalness.
- 4 Post-training: Instruction fine-tuning uses ChatML-formatted pure text, visual, audio, and mixed-modality conversations.The dataset combines modality-specific and multimodal dialogue examples.
- 4.3 Talker: Talker training proceeds through context continuation, DPO-based speech-stability optimization, and multi-speaker instruction fine-tuning.The final stage improves speech naturalness and controllability.
- 4.3 Talker: Talker’s speech-continuation training maps semantic representations monotonically to speech while learning context-appropriate prosody, emotion, and accent.Training uses multimodal dialogue contexts paired with spoken responses.
- 4.3 Talker: Reinforcement learning ranks good and bad generated speech using word error rate and punctuation pause error rate rewards to improve generation stability.The procedure addresses hallucinations associated with label noise and pronunciation errors in pretraining data.
- 4.3 Talker: Speaker fine-tuning lets Talker adopt specific voices and improve speech naturalness.This is performed on the previously trained base model.
5 Evaluation
The evaluation separates Qwen2.5-Omni into understanding and speech-generation categories. This framing covers text-producing multimodal understanding and speech-producing capabilities.
- 5 Evaluation: Evaluation divides Qwen2.5-Omni into understanding (X→Text) and speech generation (X→Speech).The two categories organize assessment of the model’s multimodal outputs.
5.1 Evaluation of X→Text
Qwen2.5-Omni is evaluated across text, audio, image, video, and mixed-modality inputs, with strong results against similarly sized and specialist models. It is especially competitive in multimodal understanding, speech interaction, and visual grounding.
- Evaluation scope: The evaluation covers text, audio, image, video, and mixed-modality-to-text tasks using broad benchmark suites.Benchmarks include MMLU-Pro, MMAU, MMMU, Video-MME, and OmniBench.
- Text→Text: Qwen2.5-Omni outperforms Qwen2-7B on most reported text-to-text benchmarks.Reported benchmarks include MMLU-Pro, MMLU-redux, MATH, GSM8K, MBPP, MultiPL-E, and LiveCodeBench.
- Audio→Text: Qwen2.5-Omni achieves better or comparable audio understanding and state-of-the-art audio reasoning across reported tasks.It reports superior results on several ASR and S2TT test sets and on sound, music, and speech subsets of MMAU.
- Audio→Text: 74.12 average score on VoiceBench surpasses other similarly sized audio language and omni models.On an in-house voice-chat benchmark, Qwen2.5-Omni narrows the gap with Qwen2-7B, which receives text instructions.
- Image→Text: Qwen2.5-Omni performs comparably to Qwen2.5-VL-7B on image understanding and outperforms other open-sourced omni models on several benchmarks.It also surpasses GPT-4o-mini on most reported image benchmarks and achieves 42.2mAP on open-vocabulary object detection.
- Video and multimodality: Qwen2.5-Omni outperforms other open-sourced omni models and GPT-4o-Mini on video understanding, while achieving state-of-the-art performance on OmniBench.Its video results are better or competitive compared to Qwen2.5-VL-7B.
5.2 Evaluation of X→Speech
The speech-generation evaluation measures zero-shot consistency and speaker similarity alongside single-speaker stability and naturalness. Qwen2.5-Omni is highly competitive, improves after reinforcement learning, and approaches human-level quality after speaker fine-tuning.
- Evaluation scope: Speech generation is evaluated for zero-shot and single-speaker settings using objective and subjective measures.Metrics include WER, SIM, and NMOS on SEED and a self-created dataset.
- Zero-Shot Speech Generation: Qwen2.5-Omni demonstrates highly competitive zero-shot TTS performance.The report attributes this result to robust speech understanding and generation capabilities developed through in-context learning.
- Zero-Shot Speech Generation: Reinforcement learning significantly improves generation stability, reducing attention misalignment, pronunciation errors, and inappropriate pauses.
- Single-Speaker Speech Generation: Speaker-finetuned Qwen2.5-Omni more precisely captures target-speaker prosody while preserving the base model’s stability.Its performance approaches human-level quality across subjective and objective metrics.
6 Conclusion
Qwen2.5-Omni unifies multimodal understanding with real-time speech generation through TMRoPE, Thinker-Talker, and streaming-oriented processing mechanisms. Comprehensive evaluations report strong voice-command and multimodal performance, while the authors identify underdeveloped evaluation areas.
- Conclusion: Qwen2.5-Omni understands and generates multiple modalities, including text and real-time speech.
- Conclusion: TMRoPE aligns audio and video timing, while Thinker-Talker supports real-time speech generation with reduced cross-modal interference.The model also uses block-wise audio/vision encoding and a sliding-window mechanism for code-to-wav generation.
- Conclusion: Comprehensive evaluations show stronger performance than similarly sized single-modality models, especially for voice commands, and state-of-the-art multimodal results.
- Conclusion: The authors identify video OCR and audio-video collaborative understanding as critical issues overlooked in prior academic studies.They call for broader evaluation benchmarks and research datasets addressing these areas.
7 Authors
The report lists core contributors and a broader contributor group.
- Authors: Core contributors include Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, and Shuai Bai.
- Authors: Additional core contributors include Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin.
- Authors: The broader contributor list includes An Yang, Anfeng Li, Baosong Yang, Bei Chen, Bin Lin, and Binyuan Hui, among many others.