Ming-Omni: A Unified Multimodal Model for Perception and Generation

• To address modality representation disparities and divergent convergence, Ming-Omni uses Ling's MoE with modality-specific routers, stepwise pretraining balance, and adaptive instruction tuning.Source: §1 · pp. 1–2
• Ming-Omni processes image, text, video, and audio inputs and supports unified understanding plus text, real-time speech, and image generation.Source: Figure 1 · p. 2, §1 · p. 3
• For speech generation, BPE reduces token frame rate by 35%, while two-stage training separates understanding from generation to balance efficiency and naturalness.Source: §1 · p. 2
• For image generation, Ming-Omni keeps the MLLM frozen and uses aligned multi-scale learnable tokens to generate images coarse-to-fine through a connector to a diffusion decoder.Source: §1 · p. 2
• 0.64 GenEval accompanies native-resolution image generation, editing, and style transfer, outperforming SDXL (Podell et al., 2023); 4.85 FID sets a new SOTA across existing methods.Source: §1 · p. 3
• Ming-Omni matches Qwen2.5-VL-7B (Bai et al., 2025a) on image perception with 2.8B activated parameters and surpasses Qwen2.5-Omni (Xu et al., 2025) and Kimi-Audio (Ding et al., 2025) on speech understanding and instruction following.Source: §1 · p. 3

2.1 Unified Understanding Across Modalities

• Dedicated Qwen2.5 vision and Whisper audio encoders project and concatenate modality embeddings with text for Ling, whose modality-specific routers direct tokens to specialized experts.Step-wise balancing during pre-training and dynamic loss-weight adjustment during instruction tuning address differences in modality convergence rates.Source: §2.1 · pp. 3–4, Figure 2 · p. 4

2.2 Unified Speech Understanding and Generation

• Optional metadata prompts provide contextual cues for speech understanding, and predicting the audio language identifier before downstream tasks improves performance, especially for dialectal speech recognition.The audio encoder output is transformed to Ling’s latent space using a linear layer and convolutional downsampling.Source: §2.2 · pp. 4–5
• For speech generation, reversible BPE compresses discrete audio tokens by 36% (50Hz to approximately 32Hz), while a context-fed audio decoder is trained with the multimodal LLM frozen.The decoder uses input hidden states to capture paralinguistic context; BPE preserves audio quality and can enhance prosody, while decoder-only training preserves multimodal understanding.Source: §2.2 · p. 5

2.3 Unified Image Understanding and Generation

• Shared visual representations can trade image-generation quality against understanding accuracy; the released work uses lightweight bridging, while fuller shared-feature unification remains future work.The paper notes that tokens optimized for understanding can degrade generation and tokens tuned for generation can reduce understanding accuracy.Source: §2.3 · pp. 5–6
• The image path pairs Qwen2.5-VL understanding with dedicated diffusion generation and multiscale learnable queries, aligning intermediate DiT states with semantic representations.Queries at different spatial scales capture global layout, objects and structures, and fine detail; feature alignment uses mean squared error between DiT hidden states and semantic representations.Source: §2.3 · pp. 6–7
• Ming-Omni supports text-to-image generation and instruction-based image editing.The illustrated tasks include style transfer, text-to-image generation, and image editing.Source: Figure 4 · p. 6, Figure 3 · p. 6, §2.3 · p. 7

2.4 Overall Training Procedure

• Perception training comprises pre-training, instruction tuning, and alignment tuning; generation training then freezes the perception LLM while separately training image-generation components and the context-aware speech decoder.Image generation trains the added connector, multiscale queries, and DiT blocks; text-to-speech training trains the audio decoder, with the two generation tasks proceeding in parallel.Source: §2.4 · p. 7

3.1 Data Overall

• Pre-training data is designed to align modalities and support world-knowledge learning, while instruction-tuning data spans multimodal tasks and generation data is tailored to its task.Stage-specific configurations combine open-source and in-house sources; Figure 6 summarizes pre-training and instruction-tuning configurations.Source: §3.1 · p. 8, Figure 6 · p. 8

3.2 Image Data

• Image data combines understanding and generation datasets, using iterative self-evolving refinement to improve quality and reduce volume while adding structured and encyclopedia data for fine-grained expertise.Source: §3.2 · p. 9

3.2.1 Image Understanding Data

• Caption refinement iteratively scores and verifies candidate pairs, retaining high-quality samples; structured QA uses cross-verified entities and knowledge triplets, while encyclopedia images undergo progressive filtering.Source: §3.2.1 · pp. 9–10
• GUI data from public datasets is improved with MLLM-refined, reviewed interaction reasoning; reasoning data draws on Ling and R1-Onevision, while preference data uses assessed QA pairs across 41 subcategories and 9 domains.Source: §3.2.1 · p. 10

3.2.2 Image Generation Data

• Image-generation data includes high-quality images from public generation datasets and style-transfer examples sampled from StyleBooth and WikiArt.Source: §3.2.2 · p. 10

3.3 Audio Data

• A larger share of English audio clips improves English understanding without degrading Chinese understanding; dialect data is 2% of the corpus, and gains plateau as its share increases.The authors derive labeled audio clips from web data using keyword-based crawling, VAD segmentation, and an iteratively trained audio labeler.Source: §3.3 · pp. 10–11

3.4 Video Data

• Video data from open datasets and public English- and Chinese-language websites is annotated hierarchically into dense captions, open-ended QA, and multiple-choice QA.Source: §3.4 · p. 11

3.5 Text Data

• The text corpus is derived from Ling and M2-omni.Source: §3.5 · p. 11

4.1 Public Benchmarks

• The evaluation spans more than 50 public benchmarks across image understanding and generation, audio understanding and generation, and video understanding.Tasks include image grounding, OCR, GUI, and information seeking; audio QA and speech benchmarks; and four video-understanding benchmarks.Source: §4.1 · pp. 11, 14

4.2 In-house Benchmarks

• Three in-house benchmarks cover expert-level image comprehension, human-centric interaction, and audio understanding across dialects and domains.They include 29,924 encyclopedia samples, 1,025 human-preference samples, and audio sets of 25,000 multi-dialect and 2,252 multi-domain samples.Source: §4.2 · pp. 14, 16

4.3 Quantitative Results

• Across 14 multimodal benchmarks, Ming-Lite-Omni achieves performance comparable to leading MLLMs.The evaluation reports comparisons with state-of-the-art models across Tables 1–12.Source: Table 1 · p. 12, Table 5 · p. 12, Table 10 · p. 15
• Ming-Lite-Omni exceeds InternVL3-8B by +2.6% on ScreenSpot and +2.7% on ScreenSpot-V2, and reaches 66.6% on AITZ(EM), +9.0% over reproduced Qwen2.5VL-7B-Insruct.The paper also reports more relevant, fluent, better-formatted, and less hallucination-prone human-centric responses than Qwen2.5VL-7B-Insruct.Source: Table 3 · p. 12, Table 9 · p. 14, §4.3 · pp. 16–17
• Ming-Lite-Omni gains +8.3%/+9.8%/+7.1% across InfoSeek’s three dimensions and averages +5.20% over Qwen2.5VL-7B-Insruct on the in-house encyclopedia benchmark.The authors attribute these results to richer expert-level knowledge and improved fine-grained image querying.Source: Table 4 · p. 12, Table 7 · p. 14, §4.3 · p. 17
• Ming-Lite-Omni’s text-to-image quality is on par with state-of-the-art diffusion models and it outperforms all baselines on FID.The paper notes a slight GenEval drop, attributed to a trade-off between instruction following and artifact sensitivity; it also highlights image editing and style variation capabilities.Source: Table 5 · p. 12, §4.3 · p. 17
• Ming-Lite-Omni sets new state-of-the-art results on 6/13 audio-understanding benchmarks, is competitive on audio QA, and attains competitive text-to-speech results.It beats Qwen2.5-Omni on average across public audio-understanding benchmarks and in-house multi-dialect and multi-domain tests.Source: Table 10 · p. 15, Table 11 · p. 15, Table 12 · p. 15
• With 128 uniformly sampled frames per video, Ming-Lite-Omni achieves a new four-benchmark average state of the art, beating Qwen2.5VL-7B-Instruct by +0.2% and LLaVA-OneVision-7B by +9.6%.It also improves LongVideoBench by +1.9% over Qwen2.5VL-7B-Instruct.Source: Table 8 · p. 14, §4.3 · pp. 17–18

5 Conclusion

• With only 2.8B active parameters, Ming-Omni performs on par with Qwen2.5-VL-7B, achieves SOTA end-to-end speech understanding and speech instruction following, and surpasses mainstream image-generation models such as SDXL.The conclusion also reports native-resolution image generation, editing, and style transfer.Source: §5 · p. 18
• Ming-Omni perceives text, images, videos, and audio, and can generate text, natural speech in real time, and images simultaneously.It also supports native-resolution image generation, editing, and style transfer.Source: §5 · p. 18
• Built on Ling MoE, Ming-Omni uses modality-specific routers to mitigate modality conflicts and supports multimodal interaction and generation.The conclusion describes its adaptability and efficiency across multimodal perception and generation tasks.Source: §5 · p. 18

A Open-source image data

• Newly added open-source image data used for training are listed in Table 13, supplementing the data used in Guo et al. (2025).Source: A Open-source image data · p. 27, Table 13 · p. 27

B Open-source audio data

• Table 14 provides the complete list of open-source audio data used during training.Source: B Open-source audio data · p. 27