MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech

• Existing systems struggle with extended audio coherence, one-to-many text-speech alignment, and error accumulation that degrades timbre consistency and audio quality.Source: §1 · p. 2
• Its dual-track architecture separates multimodal reasoning from speech synthesis while unifying vision, language, and audio in an end-to-end framework.Source: §1 · p. 2
• Chunk-Based Parallel Decoding segments text and predicts multiple speech tokens in parallel, narrowing the text-speech token-rate gap and improving alignment and inference speed by up to 3x.Source: §1 · p. 2
• The dual audio encoder fuses acoustic and semantic features, and unified training supports inference across short and long audio.Source: §1 · p. 2
• MGM-Omni significantly outperforms existing methods in long audio understanding and achieves leading performance in zero-shot voice cloning and natural, context-aware long-form speech generation.Source: §1 · p. 2

Multi-modal Large Language Models

• Vision-language models adapted frozen visual encoders to instruction-tuned LLMs, then advanced high-resolution image comprehension, video understanding, and visual grounding.Examples include Flamingo, LLaVA, MiniGPT-4, Mini-Gemini, and Qwen-VL.Source: §2 · p. 3
• Recent multimodal models incorporate audio, but remain limited in long-form audio understanding and generation and in controlling generated-speech timbre.Source: §2 · p. 3

Speech Generation

• Zero-shot TTS systems such as CosyVoice2 and Qwen2.5-Omni support streaming or end-to-end multimodal generation, yet long-form timbre consistency, cross-modal fidelity, and latency remain challenges.Source: §2 · p. 3
• MGM-Omni uses chunk-based parallel decoding to target expressive long-form speech with consistent timbre and low latency.Source: §2 · p. 3

3 MGM-OMNI

• The MLLM handles multimodal inputs and produces text for the SpeechLM, which generates speech in real time.Modality-specific encoders feed the MLLM; SpeechLM speech tokens are converted to Mel-spectrograms by a flow-matching model and synthesized by a vocoder.Source: §3 · p. 3, Figure 2 · p. 4

3.1 OMNI UNDERSTANDING

• A dual audio encoder combines acoustic and semantic features into enhanced audio tokens for subsequent LLM processing.The primary Qwen2-Audio encoder is continually trained on Whisper-large-v3, while Belle-Whisper-large-v3 supplies complementary semantic features, especially for Chinese speech; information mining retrieves cues from the auxiliary features under guidance of the main features.Source: §3.1 · pp. 3–4
• Two-stage audio-to-text and unified omni-modal training integrates audio understanding with transcription, audio QA, audio-instruct VQA, and text-instruction data.For variable-length inputs, the pipeline groups similar audio lengths and adjusts batch size dynamically, using smaller batches for long contexts and larger batches for short ones.Source: §3.1 · p. 4

3.2 OMNI GENERATION

• SpeechLM conditions speech generation on MLLM-produced text and reference audio, with speech tokens decoded autoregressively and rendered through flow matching and a vocoder.The SpeechLM uses a TTS-Adapter to transform text representations into speech representations; the flow-matching model supports chunk-aware streaming decoding.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
• Chunk-based parallel decoding addresses text-speech token-rate mismatch, with FSQ parallel decoding reported to improve efficiency while maintaining synthesis performance.Chunking aligns text and speech segments, delays speech-token generation until after the first four text tokens, and preserves prior text and speech as context for fluency and coherence.Source: §3.2 · pp. 5–6

3.3 OMNI VOICE

• Personalized voice generation uses about 300k hours of raw speech and approximately 100k hours of synthesized Chinese and English speech, followed by adapter-only pre-training and joint post-training.Pre-training freezes the Qwen3 LLM while updating the TTS-Adapter; post-training optimizes both with the adapter learning rate set five times higher.Source: §3.3 · pp. 6–7

4.1.1 AUDIO UNDERSTANDING

• MGM-Omni records 4.0 CER on CommonVoice (ZH) and 1.8 CER on AISHELL, and outperforms open-source Omni LLMs on AIR-Bench.AIR-Bench covers speech, sound, and music inputs.Source: §4.1.1 · pp. 7–8
• MGM-Omni handles audio up to 4,500 seconds and significantly outperforms Qwen2.5-Omni in the needle-in-a-haystack evaluation.Figure 5 reports average success rate across five diverse long-form audio materials.Source: §4.1.1 · p. 8, Figure 5 · p. 8

4.1.2 OMNI-MODALITY UNDERSTANDING

• MGM-Omni shows a strong ability to follow speech instructions on vision-speech VQA benchmarks.Table 4 evaluates multimodal understanding using synthesized speech versions of textual questions.Source: §4.1.2 · p. 8, Table 4 · p. 8

4.1.3 SPEECH GENERATION

• On Seed-TTS-Eval, MGM-Omni achieves lower error rates and higher speaker similarity than open-source TTS models and Omni LLMs.The authors report strong text-to-speech performance and robust zero-shot voice cloning.Source: §4.1.3 · p. 8, Table 5 · p. 9
• MGM-Omni achieves lower error rates across most long-form speech scenarios and the lowest RTF in Long-TTS-Eval.The comparison covers native long-TTS systems and a chunking-based system; the authors report training with less than 400k hours of audio.Source: §4.1.3 · p. 9, Table 5 · p. 9

4.2 ABLATION STUDY

• Combining Qwen2-Audio and Belle-Whisper-large-v3 with information mining yields the best CommonVoice ASR performance among the tested audio encoder designs.This ablation does not use the final model’s long audio QA data.Source: §4.2 · p. 9
• Removing chunk-based decoding substantially increases long-form speech error rates, exceeding those of concurrent works.The authors attribute MGM-Omni’s data efficiency primarily to chunk-based decoding.Source: §4.2 · p. 9
• A parallel size of 4 balances quality and speed; increasing parallel size slightly raises error rates while accelerating inference by 3x.Inference speed was measured on a single H800 GPU.Source: §4.2 · pp. 9–10

5 CONCLUSION

• MGM-Omni separates multimodal reasoning from real-time speech synthesis and uses dual audio encoders and Chunk-Based Parallel Decoding for understanding and generation.SpeechLM is conditioned on reference audio for zero-shot voice cloning with consistent timbre.Source: §5 · p. 10

A.1 DATA FORMAT

• Figure 6 presents the data formats for MLLM and SpeechLM, with chunk-based decoding used by SpeechLM to generate long-form speech.The figure caption identifies MLLM at the top and SpeechLM at the bottom.Source: §A.1 · p. 14, Figure 6 · p. 14

A.2 LONG-TTS-EVAL BENCHMARK

• The appendix section provides details on Long-TTS-Eval’s data composition and evaluation protocol.Source: §A.2 · p. 14

A.2.1 DATA COMPOSITION

• Long-TTS-Eval assesses long-form speech generation and complex-case handling.The benchmark is designed to evaluate both capabilities.Source: §A.2.1 · p. 14
• The benchmark’s six text types—literature, news, knowledge, speeches, reviews, and academic papers—comprise 341 Chinese and 353 English samples.The samples draw on news outlets, Wikipedia, YouTube transcripts, and arXiv papers.Source: §A.2.1 · p. 14, Table 7 · p. 15
• The longest samples contain 1,899 Chinese tokens and 3,277 English tokens, estimated at about 10 and 12 minutes of speech, respectively.The estimates assume 200 Chinese characters per minute and 215 English words per minute, with 1.3–1.5 tokens per word.Source: §A.2.1 · pp. 14–15
• The complex-case set covers web URLs, emails, math formulas, phone numbers, and large numbers, with 265 Chinese and 260 English samples.Formulas come from S1 Long-CoT Instruct; the other categories were generated by Gemini 2.5 Pro.Source: §A.2.1 · p. 15, Table 8 · p. 15

A.2.2 EVALUATION PIPELINE

• English and Chinese speech are transcribed with Whisper-large-v3 and Paraformer-zh, respectively.Each waveform is split into 28-second chunks, transcribed independently, and concatenated before scoring.Source: §A.2.2 · p. 15
• The concatenated English transcripts are scored with word error rate, while Chinese transcripts use character error rate.Source: §A.2.2 · p. 15
• The evaluation pipeline follows Seed-TTS-Eval.The passage identifies the ASR engines and chunking procedure used in that pipeline.Source: §A.2.2 · p. 15

A.2.3 EVALUATION WITH NORMALIZED TEXT

• For expressions with multiple valid readings, such as “5%” spoken as “five percent,” written-text comparison can wrongly count correct speech as an error.Source: §A.2.3 · p. 15
• The evaluation uses GPT-5 to create a natural spoken-form reference and takes the lower WER against written or normalized text for each sample.This method lowers the risk of falsely flagging correct TTS and improves reported error-rate reliability.Source: §A.2.3 · p. 15

A.3.1 LONG AUDIO UNDERSTANDING

• MGM-Omni gives more complete, detailed long-audio summaries than Qwen2.5-Omni and extracts fine-grained information where Qwen2.5-Omni refuses to respond.Source: §A.3.1 · p. 16

A.3.2 LONG SPEECH GENERATION

• On two long-form speech examples, MGM-Omni produces accurate speech with appropriate pauses, while MOSS-TTSD-v0.5 and Higgs-Audio-v2 show later-audio errors and audible noise.The examples are “Preface to the Pavilion of Prince Teng” and a Chinese-English excerpt from “Stray Birds.”Source: §A.3.2 · p. 16