Qwen2.5-Omni Technical Report

Motivation

• The report identifies joint training across text, images, video, and audio, temporal synchronization, cross-output interference, and low-latency streaming as key challenges for a unified omni-model.It emphasizes real-time multimodal understanding and efficient audio-output streaming.Source: §1 · p. 2

Contributions

• TMRoPE interleaves audio and video frames in time order to encode temporal information and synchronize the modalities.The report introduces TMRoPE as a novel position-embedding approach for audio-video synchronization.Source: §1 · p. 2
• Thinker-Talker separates text generation in Thinker from streaming speech-token generation in Talker, which uses Thinker representations, and trains both components end to end.The architecture is designed to prevent interference between text and speech outputs while generating distinct signals.Source: §1 · p. 2
• Block-wise processing in multimodal encoders supports streaming input and real-time pre-filling, while a dual-track autoregressive model and DiT enable streaming speech generation with reduced initial latency.The dual-track model generates speech tokens, and the DiT converts them into waveforms.Source: §1 · p. 2

Results

• Qwen2.5-Omni matches similarly sized Qwen2.5-VL, outperforms Qwen2-Audio, leads OmniBench and AV-Odyssey Bench, and matches text-input speech instruction following on MMLU and GSM8K.The report describes these comparisons as evidence of strong performance across modalities and state-of-the-art multimodal benchmark performance.Source: §1 · pp. 2–3
• WERs of 1.42% on test-zh, 2.33% on test-en, and 6.54% on test-hard on seed-tts-eval outperform MaskGCT and CosyVoice 2.The report presents these as speech-generation results on the three named evaluation sets.Source: §1 · p. 2

2.1 Overview

• Thinker processes text, audio, images, and video into representations and text; Talker uses Thinker representations and shared context to generate streaming speech in an end-to-end model.Thinker is a Transformer decoder with audio and image encoders; Talker is a dual-track autoregressive Transformer decoder.Source: Figure 2 · p. 3, §2.1 · p. 3

2.2 Perceivation

• TMRoPE combines temporal, height, and width positions, assigning absolute 40-ms temporal IDs and dynamically timestamping video frames; audio-video streams are interleaved in 2-second chunks.Text uses identical position IDs across dimensions; images use fixed temporal IDs and spatial height and width IDs. Text uses Qwen’s 151,643-token byte-level BPE vocabulary; audio is resampled to 16 kHz and represented as a 128-channel mel-spectrogram, while the approximately 675-million-parameter vision encoder handles images and video sampled at a dynamic frame rate.Source: §2.2 · p. 4

2.3 Generation

• Talker combines Thinker’s high-level representations with sampled text-token embeddings to autoregressively emit speech and text tokens, while qwen-tts-tokenizer and a causal decoder enable streaming without word- or timestamp-level alignment.Thinker’s representations convey tone and attitude but may not distinguish phonetically different words, so sampled discrete text tokens resolve that uncertainty.Source: §2.3 · p. 5

2.4 Designs for Streaming

• To support chunked prefills, the audio encoder uses 2-second temporal-attention blocks, while the vision encoder uses flash attention and merges adjacent 2×2 tokens with an MLP.With patch size 14, images of different resolutions can be packed into a sequence; reducing initial packet latency is a key streaming-performance concern.Source: §2.4 · p. 5
• For streaming waveform generation, a sliding-window Flow-Matching DiT limits context before modified BigVGAN reconstructs the waveform from mel-spectrograms.The DiT uses a four-block receptive field with a two-block lookback and one-block lookahead; mel-spectrograms are generated in chunks, as is BigVGAN’s waveform output.Source: §2.4 · p. 5

3 Pre-training

• Pre-training has three stages: encoder-focused training with the LLM locked, multimodal training with all parameters unfrozen, then 32k-sequence training.The first stage trains the vision and audio encoders on audio-text and image-text pairs; the second uses broader multimodal data.Source: §3 · p. 6
• Initially, Qwen2.5-Omni uses Qwen2.5 parameters for its LLM, the Qwen2.5-VL vision encoder, and Whisper-large-v3 for audio; encoders first train adapters with the LLM fixed.This initial training targets visual-textual and audio-textual correlations and alignments.Source: §3 · p. 6
• The second phase adds 800 billion image/video tokens, 300 billion audio tokens, and 100 billion video-with-audio tokens, while natural-language prompts replace hierarchical tags following Qwen2-Audio (Chu et al., 2024a).The expanded multimodal, multitask data deepens cross-modal interaction; pure text data helps maintain and improve language proficiency. The prompt change is intended to improve generalization and instruction following.Source: §3 · p. 6
• Extending training from an 8,192-token limit to 32,768 tokens with long audio and video significantly improved support for long-sequence data.The earlier stages capped token length at 8,192; the later training extended text, audio, image, and video data to 32,768 tokens.Source: §3 · p. 6

4.2 Thinker

• Thinker instruction fine-tuning uses ChatML data spanning text, visual, audio, and mixed-modality conversations.The report describes this as the post-training dataset for instruction-following.Source: §4.2 · p. 6

4.3 Talker

• Talker training proceeds through context continuation, DPO for speech-generation stability, and multi-speaker instruction fine-tuning for naturalness and controllability.The three stages are designed to enable simultaneous text and speech responses.Source: §4.3 · p. 7
• Talker ICL training combines text supervision with next-token speech continuation on multimodal dialogues, learning semantic-to-speech mapping and context-appropriate prosody, emotion, and accent.Timbre disentanglement helps prevent associations between specific voices and infrequent textual patterns.Source: §4.3 · p. 7
• To address hallucinations linked to noisy pretraining labels and pronunciation errors, Talker reinforcement learning ranks generated speech using word error rate and punctuation pause error rate rewards.Training examples pair each request and response text with reference speech and good and bad generated speech sequences.Source: §4.3 · p. 7

• Qwen2.5-Omni is comprehensively evaluated in two categories: understanding (X→Text) and speech generation (X→Speech).Source: §5 · p. 7

5.1.1 Performance of Text→Text

• On text benchmarks, Qwen2.5-Omni generally falls between Qwen2-7B and Qwen2.5-7B, outperforming Qwen2-7B on most benchmarks.The evaluation covers general knowledge, mathematics and science, and coding.Source: §5.1 · p. 7, Table 1 · p. 8, §5.1.1 · p. 8

5.1.2 Performance of Audio→Text

• Audio results are state-of-the-art across music, VSC, and MMAU reasoning; VoiceBench averages 74.12, and speech-instruction testing narrows the gap to text-instruction Qwen2-7B versus Qwen2-Audio.About 90% of text instructions were used in the in-house voice-chat benchmark.Source: §5.1.2 · p. 8, Table 2 · p. 9, Table 3 · p. 10

5.1.3 Performance of Image →Text

• Qwen2.5-Omni is comparable to Qwen2.5-VL-7B on image tasks, beats open-sourced omni models on several benchmarks, and reaches 42.2mAP on open-vocabulary object detection.For visual grounding, it outperforms other models across most benchmarks from box-grounding to point-grounding.Source: §5.1.3 · pp. 10–11, Table 5 · p. 11, Table 6 · p. 11

5.1.4 Performance of Video →Text

• On video understanding, Qwen2.5-Omni outperforms open-sourced omni models and GPT-4o-Mini, with better or competitive results versus Qwen2.5-VL-7B.The evaluation uses Video-MME, MVBench, and EgoSchema.Source: §5.1 · p. 8, §5.1.4 · p. 11, Table 7 · p. 12

5.1.5 Performance of Multimodality→Text

• Qwen2.5-Omni achieves state-of-the-art OmniBench performance, surpassing other Omni models by a large margin.This result demonstrates the model’s multimodality understanding.Source: §5.1.5 · p. 11, Table 8 · p. 12

5.2 Evaluation of X→Speech

• The evaluation focused on text-conditioned zero-shot and single-speaker speech generation, reflecting the lack of relevant speech-generation assessments.Zero-shot evaluation measured content consistency (WER) and speaker similarity (SIM); single-speaker evaluation assessed stability and subjective naturalness (NMOS).Source: §5.2 · p. 12

5.2.1 Evaluation of Zero-Shot Speech Generation.

• Qwen2.5-Omni showed highly competitive zero-shot TTS performance, and reinforcement learning improved generation stability on the challenging test-hard dataset.The reported reductions concerned attention misalignment, pronunciation errors, and inappropriate pauses.Source: §5.2.1 · p. 12

5.2.2 Evaluation of Single-Speaker Speech Generation.

• Speaker fine-tuning captured target speakers’ nuanced prosody while preserving base-model stability, with quality approaching human recordings across subjective and objective metrics.Source: §5.2.2 · p. 12

6 Conclusion

• The model combines TMRoPE for audio-video timing alignment with Thinker-Talker speech generation and block-wise audio and vision encoding.The conclusion also describes a sliding-window mechanism for code-to-wav generation and says the model excels in complex audio-visual interactions and emotional contexts in speech dialogues.Source: §6 · pp. 12–13
• Qwen2.5-Omni outperforms similarly sized single-modality models, particularly on voice-command following, and achieves state-of-the-art performance on multimodal tasks.Source: §6 · p. 13
• The conclusion identifies video OCR and audio-video collaborative understanding as overlooked challenges requiring comprehensive benchmarks, datasets, and academic-industry collaboration.Source: §6 · p. 13
• Future goals include making the model more robust and faster, with expanded output capabilities across images, videos, and music.Source: §6 · p. 13