Source-linked AI summary
Ming-Omni: A Unified Multimodal Model for Perception and Generation
Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, Furong Xu, GuangMing Yao, Jun Zhou, Jingdong Chen, Jianxin Sun, Jiajia Liu, Jianjiang Zhu, Jun Peng, Kaixiang Ji, Kaiyou Song, Kaimeng Ren, Libin Wang, Lixiang Ru, Lele Xie, Longhua Tan, Lyuxin Xue, Lan Wang, Mochen Bai, Ning Gao, Pei Chen, Qingpei Guo, Qinglong Zhang, Qiang Xu, Rui Liu, Ruijie Xiong, Sirui Gao, Tinghao Liu, Taisong Li, Weilong Chai, Xinyu Xiao, Xiaomei Wang, Xiaoxue Chen, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Yi Yuan, Yuting Gao, Yunxiao Sun, Yipeng Chen, Yifei Wu, Yongjie Lyu, Ziping Ma, Zipeng Feng, Zhijiang Fang, Zhihao Qiu, Ziyuan Huang, Zhengyu He
TL;DR
Ming-Omni addresses the difficulty of unifying multimodal perception and generation across disparate representations and training dynamics. It combines Ling’s MoE architecture with modality-specific routers, dedicated generation modules, and two-phase training. The model matches the stated modality support of GPT-4o as an open-source system and reports strong results across perception, speech, and image generation tasks.
Problem
Unified Omni-MLLMs face representational disparities across modalities, divergent convergence rates, and a trade-off between image understanding and generation quality.
Method
Ming-Omni uses Ling’s MoE language model with modality-specific routers, dedicated encoders and decoders, and separate perception and generation training phases.
Results
Ming-Omni reports strong cross-modal performance, including image perception comparable to Qwen2.5-VL-7B with 2.8B activated parameters, GenEval 0.64, and FID 4.85.
Takeaways & Limitations
Ming-Omni provides an open-source unified system supporting multimodal perception together with real-time speech and image generation, editing, and style transfer.
Takeaways & Limitations
Balancing image understanding and generation remains an unresolved direction, with generation-tuned tokens potentially reducing understanding accuracy.
Abstract
from arXiv · showhide
We propose Ming-Omni, a unified multimodal model capable of processing images, text, audio, and video, while demonstrating strong proficiency in both speech and image generation. Ming-Omni employs dedicated encoders to extract tokens from different modalities, which are then processed by Ling, an MoE architecture equipped with newly proposed modality-specific routers. This design enables a single model to efficiently process and fuse multimodal inputs within a unified framework, thereby facilitating diverse tasks without requiring separate models, task-specific fine-tuning, or structural redesign. Importantly, Ming-Omni extends beyond conventional multimodal models by supporting audio and image generation. This is achieved through the integration of an advanced audio decoder for natural-sounding speech and Ming-Lite-Uni for high-quality image generation, which also allow the model to engage in context-aware chatting, perform text-to-speech conversion, and conduct versatile image editing. Our experimental results showcase Ming-Omni offers a powerful solution for unified perception and generation across all modalities. Notably, our proposed Ming-Omni is the first open-source model we are aware of to match GPT-4o in modality support, and we release all code and model weights to encourage further research and development in the community.
1 Introduction
Ming-Omni is presented as a unified multimodal model that integrates perception and generation across images, text, video, and audio. Its modality-specific routing and generation modules support diverse cross-modal tasks, with strong reported performance across perception, interaction, and image generation.
- Unified capabilities: Ming-Omni processes images, text, video, and audio while generating text, speech, and images within one unified model.The lite version, Ming-Lite-Omni, supports visual perception, audio-visual interaction, and image generation.
- Training strategy: A stepwise pre-training balance and dynamically adaptive instruction tuning strategy address cross-modal data imbalance and differing training progress.These strategies are reported to improve convergence and model performance across modalities.
- Audio generation: 35% lower token frame rate from BPE supports faster speech generation, while two-stage training separates audio understanding from generation quality optimization.The audio decoder targets prosodic naturalness, real-time response, context awareness, and complex acoustic environments.
- Reported performance: 0.64 GenEval and 4.85 FID accompany reported cross-modal results, with image perception comparable to Qwen2.5-VL-7B using 2.8B activated parameters.The model also supports native-resolution image generation, editing, and style transfer, and is reported to outperform SDXL.
- Unified perception: Modality-specific routers in Ling’s MoE architecture resolve task conflicts and integrate tokens from different modalities coherently.The routing design assigns tailored routing distributions to different token types.
- Unified perception and generation: Ming-Omni combines unified multimodal perception with text, real-time speech, and image generation to interpret instructions and user intent during generation.The paper identifies this integration as improving generation quality and usability across multiple tasks.
2 Approach
Ming-Omni unifies multimodal perception and generation through dedicated modality encoders, Ling with modality-specific routing, and separate audio and image generation components. Its design addresses cross-modal training and representation conflicts while supporting speech, text-to-image, style transfer, and image editing.
- Overall Framework: Ming-Omni processes image, audio, video, and text inputs, then supports speech and image generation in a two-phase perception-generation framework.Perception training develops Ling for multimodal understanding; generation training develops the audio decoder and DiT module.
- Unified Understanding Across Modalities: Dedicated visual and audio encoders project modality representations for processing by Ling, an MoE language model with modality-specific routers.The routers direct tokens from each modality to specialized experts for more precise and efficient routing.
- Unified Speech Understanding and Generation: Ming-Omni connects an autoregressive audio decoder to the language model to generate discrete audio tokens for speech synthesis.BPE compresses discrete audio-token length by 36%, from 50Hz to approximately 32Hz, improving training and inference efficiency.
- Unified Image Understanding and Generation: The image-generation design uses multi-scale learnable tokens, representation alignment, and a dedicated diffusion model to preserve understanding while enabling generation.The multi-scale tokens capture global layout, objects and structures, and fine textures; feature alignment minimizes mean squared error between DiT and semantic representations.
- Unified Image Understanding and Generation: Ming-Omni supports text-to-image generation, instruction-based image editing, and instruction-based image style transfer.The reported image-generation applications include native-resolution generation and editing.
3 Data Construction
Ming-Omni’s data construction combines diverse multimodal sources with specialized pipelines for caption, knowledge, interaction, reasoning, generation, audio, video, and text data. Iterative filtering and annotation improve data quality while reducing caption-data volume.
- 3.1 Data Overall: The training corpus uses different data configurations across pre-training and instruction tuning to align modalities and facilitate world-knowledge learning.The configurations cover different modalities, objectives, and tasks across training stages.
- 3.2.1 Image Understanding Data: An iterative caption-data pipeline evaluates split samples, retains high-quality captions through MLLM verification, and replaces the original corpus with the refined pool.The procedure enhances caption quality and diversity while significantly reducing data volume.
- 3.2.1 Image Understanding Data: Structured and encyclopedia data target fine-grained visual knowledge, information-seeking queries, and expert-level comprehension across specialized domains.The encyclopedia corpus includes biological and cultural categories and uses entities collected from academic databases and institutional websites.
- 3.2.2 Image Generation Data: Image-generation data combines public high-quality generation datasets with style-transfer data sampled from StyleBooth and WikiArt.These sources include instruction-based editing datasets such as InstructPix2Pix-clip-filtered, SEED-Data-Edit, and Ultraedit.
- Audio Data: Audio data combines open-source, web, synthetic, and in-house datasets, with labels and metadata covering varied domains and contexts.Ablations found that more English audio improved English understanding without degrading Chinese understanding; dialect data comprised 2% of the corpus.
- Other Modalities and Tasks: Video data is curated from open-source datasets and websites using hierarchical annotation to generate dense captions and question-answer data, while text data comes from Ling and M2-omni.GUI, reasoning, and preference corpora further support interaction, chain-of-thought learning, and human-centric alignment.
4 Evaluation
Ming-Lite-Omni is evaluated across more than 50 public benchmarks and additional in-house tests spanning image, audio, video, and generation capabilities. Results show comparable or superior performance to leading multimodal models across several tasks.
- Evaluation Scope: The evaluation covers more than 50 public benchmarks across five multimodal dimensions, supplemented by three in-house benchmarks.The dimensions include image-to-text, text-to-image, audio-to-text, text-to-audio, and video-to-text evaluation.
- Overall Results: Ming-Lite-Omni achieves comparable performance with leading MLLMs across different multimodal benchmarks.This overall comparison is reported across Tables 1–12.
- Image Understanding: Ming-Lite-Omni outperforms leading MLLMs on GUI, knowledge-intensive, encyclopedia, and human-preference image-understanding evaluations.It is comparable on grounding and OCR while showing stronger results on GUI, information-seeking, expert-level encyclopedia, and human-centric interaction tasks.
- Image Understanding: 66.6% accuracy on AITZ(EM) exceeds reproduced Qwen2.5VL-7B-Instruct results by +9.0%, while ScreenSpot and ScreenSpot-V2 improve over InternVL3-8B by +2.6% and +2.7%.These results are reported for GUI grounding and action reasoning in an Android environment.
- Image Understanding: +8.3%/+9.8%/+7.1% gains across three InfoSeek dimensions and a +5.20% average improvement on the in-house encyclopedia benchmark are reported over leading comparators.The results are attributed to stronger expert-level knowledge and fine-grained information querying from images.
- Image Generation: Ming-Lite-Omni’s image-generation quality is reported as on par with state-of-the-art diffusion models, with significantly better FID than all baselines.The paper notes a trade-off involving GenEval and highlights support for richer editing and style variation.
- Audio Understanding and Generation: Ming-Lite-Omni attains 6/13 state-of-the-art results on public audio-understanding benchmarks and outperforms Qwen2.5-Omni on average across public and in-house audio tests.It remains competitive on audio question-answering tasks and text-to-audio generation.
- Video Understanding: Ming-Lite-Omni reaches a new average-performance state of the art across four video benchmarks, exceeding Qwen2.5VL-7B-Instruct by +0.2% and LLaVA-OneVision-7B by +9.6%.It also improves LongVideoBench by +1.9% over Qwen2.5VL-7B-Instruct.
5 Conclusion
The conclusion presents Ming-Omni as an open-source unified model that supports multimodal perception and generation. It combines broad modality coverage with efficient activation and strong reported performance across speech and image tasks.
- Conclusion: Ming-Omni is presented as the first open-source model the authors are aware of to match GPT-4o in modality support.It processes text, images, videos, and audio while generating text, real-time natural speech, and images.
- Conclusion: Modality-specific routers in the Ling MoE architecture are used to mitigate modality conflicts.The model activates only 2.8B parameters while achieving performance on par with Qwen2.5-VL-7B.
- Conclusion: Ming-Omni demonstrates state-of-the-art end-to-end speech understanding and speech instruction following, alongside native-resolution image generation, editing, and style transfer.The conclusion reports that it surpasses mainstream generation models such as SDXL.
6 Contributors
The contributor section lists the authors alphabetically by first name. The supplied author list includes contributors from Ant Inclusion AI and spans the names shown in the paper.
- Contributor Listing: Authors are listed alphabetically by their first names.
- Contributor Listing: The listed contributors include Ant Inclusion AI and the authors beginning with Biao Gong through Kaiyou Song.
- Contributor Listing: The author list continues from Kaimeng Ren through Xiaoyu Li, Xingning Dong, and Xuzheng Yu.
- Contributor Listing: The listed contributors conclude with Yi Yuan through Ziyuan Huang and Zhengyu He.
A Open-source image data
The training data section notes that newly added open-source image data supplement previously used open-source image data. These additions are documented in Table 13.
- Open-source Image Data: Newly added open-source image data are presented in Table 13 alongside the open-source image data used in prior work.
- Open-source Image Data: Table 13 lists the newly added open-source image data used during training.
B Open-source audio data
The paper provides a complete list of the open-source audio data used during training in Table 14.
- The training used open-source audio data.
- The paper includes the complete list of open-source audio data used during training.
- Table 14 contains the complete audio-data list.