Source-linked AI summary
Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
Inclusion AI, :, Bowen Ma, Cheng Zou, ChengKun Du, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Chengxiang Fan, Dandan Zheng, Fudong Wang, Furong Xu, Guangming Yao, Haohao Liu, Han Peng, Jun Zhou, Junluan Xia, Jingdong Chen, Jianing Li, Jianxin Sun, Jianjiang Zhu, Jianping Jiang, Jinpeng Ou, Jun Peng, Jin Peng, Kaixiang Ji, Li Tang, Libin Wang, Lixiang Ru, Longhua Tan, Lu Ma, Lan Wang, Mochen Bai, Minghong Cai, Mingxue Yang, Ning Gao, Qingpei Guo, Qinglong Zhang, Qiang Xu, Qin Zhao, Rui Liu, Ruijie Xiong, Ruobing Zheng, Sirui Gao, Shaoxiong Lin, Tao Zhang, Tianqi Li, Tinghao Liu, Tongli Wang, Taoye Huang, Weilong Chai, Xiaomei Wang, Xiaolong Wang, Xiaojian Liu, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Xuezhi Wang, Yi Yuan, Yuting Gao, Yuting Xiao, Yunxiao Sun, Yipeng Chen, Yifan Mao, Yifei Wu, Yongjie Lyu, Yingying Zhang, YuQian Li, Ziping Ma, Zhiqiang Fang, Zhihao Qiu, Ziyuan Huang, Zizheng Yang, Zhengyu He
TL;DR
Unified multimodal models must better integrate perception and generation while scaling efficiently across vision, speech, and language. Ming-Flash-Omni addresses this with a sparse MoE architecture and targeted advances in multimodal understanding and generation, achieving strong cross-modal performance and supporting a practical foundation for general-purpose multimodal intelligence.
Problem
Unified multimodal models face a bottleneck in transferring fine-grained visual knowledge between understanding and high-precision generation and editing.
Method
Ming-Flash-Omni combines a sparse 100B-parameter MoE foundation with targeted advances in multimodal perception, contextual ASR, cross-task understanding, speech generation, segmentation, and controllable image synthesis.
Results
Ming-Flash-Omni achieves SOTA performance across a broad range of multimodal perception and generation tasks, with image perception comparable to Gemini 2.5 Pro and strong speech capabilities.
Takeaways & Limitations
A single unified model can provide an efficient foundation for general-purpose multimodal intelligence across vision, speech, and language.
Abstract
from arXiv · showhide
We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. Notably, it achieves strong performance on vision-language understanding benchmarks, with overall scores on par with Gemini 2.5 Pro, and enables seamless switching among multimodal tasks in multi-turn interactions. In speech, it achieves strong performance in contextual and dialect-aware ASR while enabling joint, continuous-generation of speech, sound, and music. In vision, it introduces generative semantic segmentation that achieves competitive standalone performance and enhances spatial control and editing consistency, alongside marked improvements in identity preservation, and high-fidelity in-image text rendering. Together, these capabilities demonstrate that a single unified model can serve as a practical foundation for general-purpose multimodal intelligence.
1 Introduction
Ming-Flash-Omni is a unified multimodal model designed to approximate human-like integration of visual, auditory, and linguistic intelligence. Built on sparse Ling-Flash-2.0 (lin, 2025), it combines efficient scaling with advances in multimodal understanding and generation, including performance comparable to Gemini 2.5 Pro (Team, 2025).
- Architecture: Ming-Flash-Omni builds on a highly sparse Mixture-of-Experts Ling-Flash-2.0 foundation, increasing model capacity while maintaining bounded inference latency.The architecture targets a favorable performance–efficiency trade-off through increased sparsity.
- Multimodal understanding: Its understanding advances include time-interleaved VideoRoPE for temporal alignment and improved context-aware speech recognition.VideoRoPE interleaves timestamps with video frames so visual tokens reflect precise temporal positions.
- Multimodal generation: Its generation advances include joint single-channel speech, sound, and music synthesis, generative semantic segmentation, and fine-grained controllable image generation.Continuous representations replace discrete acoustic tokens to reduce quantization artifacts and produce more natural, expressive TTS outputs.
- Results: Ming-Flash-Omni reaches image-perception performance comparable to Gemini 2.5 Pro (Team, 2025), while providing strong image-generation control and state-of-the-art speech understanding and generation.The model also supports strong identity preservation and high-fidelity in-image text rendering.
2 Ming-Flash-Omni
Ming-Flash-Omni is a sparse, unified multimodal architecture built on Ling-Flash-2.0’s 100B-parameter MoE, with 6.1B active parameters per token and integrated perception, speech, image generation, and temporal modeling. It addresses understanding–generation alignment through unified representations, generative segmentation, reinforcement learning, and continuous audio generation.
- Architecture: The model uses Ling-Flash-2.0’s sparse MoE language model with 100B total parameters and 6.1B active per token, retaining Ming-Omni AI et al. (2025)’s unified two-stage pipeline.A dual balancing scheme stabilizes training and improves efficiency; the pipeline supports multimodal understanding plus speech and image synthesis.
- Speech and Audio: Ming-Flash-Omni jointly generates speech, sound effects, and music using continuous acoustic latents and a low-frame-rate 12.5 Hz audio tokenizer.The tokenizer compresses long raw-audio sequences to mitigate autoregressive error propagation, while consistent 44.1 kHz training data and super-resolution support high-fidelity audio.
- Speech and Audio: On Instruct-TTS-Eval-zh, Ming-Flash-Omni matches Qwen3-TTS Hu et al. (2026), while its text normalization renders complex mathematical and chemical notation.The system also disentangles timbre from instruction-controlled style attributes without additional modules.
- Vision Generation: Image understanding and generation are aligned through a single-stream diffusion transformer, generative segmentation pretraining, and multi-reward reinforcement learning.Generative segmentation reframes segmentation as semantic-preserving editing, while the single-stream design maps text and images into a unified token space.
- Training Efficiency: 28% MFU delivers more than 4x higher training throughput than baseline Megatron-LM.
3 Data Construction
Ming-Flash-Omni is trained on a diverse multimodal corpus spanning text, images, audio, and video, with processing pipelines for quality, diversity, deduplication, and cross-modal alignment. The construction adds specialized knowledge, reasoning, OCR, safety, reinforcement-learning, image-generation, and contextual ASR data to strengthen unified perception and generation.
- Multimodal Corpus: The training corpus spans text, images, audio, and video, drawing primarily from Ming-Omni while adding pipelines for quality, diversity, deduplication, and multimodal alignment.These strategies support joint multimodal training and seamless knowledge alignment across modalities.
- Image Data: Image data anchors the corpus by combining image-understanding and image-generation datasets with synthesized high-quality data designed to improve capabilities and user interaction.Additional image-generation training covers segmentation, text rendering, and portrait preservation, using embedding-based clustering for semantic balance.
- Knowledge Data: Knowledge data combines expert-level entity recognition and encyclopedia-based visual QA to improve fine-grained concept recognition and knowledge-grounded multimodal reasoning.Entity data uses progressive CLIP, MLLM, and manual filtering, while visual knowledge triplets are validated by multi-VLM consensus before conversion into VQA pairs.
- Reasoning and Alignment Data: The construction also includes filtered chain-of-thought STEM data, millions of heterogeneous OCR samples, safety data expanded nearly 30×, and broad multi-task GRPO data.These datasets target complex reasoning, text recognition, context-aware safety, and general-purpose reinforcement-learning scenarios including QA, instruction following, and OCR.
- Generation and Audio Data: Specialized generation and audio data improve controllable vision and speech capabilities through segmentation, portrait preservation, bilingual text rendering, and contextual ASR datasets.Portrait data includes 200k paired lifestyle-ID photos and 20K portraits from 225 landmarks, while ContextASR contains 3 million Chinese and English samples.
4 Evaluation
Ming-Flash-Omni is evaluated across more than 50 public benchmarks spanning seven multimodal dimensions, plus in-house tests of factual knowledge, streaming dialogue, and dialect- and domain-diverse audio understanding. Across these evaluations, it achieves comparable performance to leading MLLMs, with strong results in vision, generation, editing, segmentation, audio, and video streaming conversation.
- Vision →Text (Understanding): On Vision →Text understanding, Ming-Flash-Omni scores 66.1 on HallusionBench, 85.6 on MMVet, 74.9 on MMStar, and 77.5 on MVBench.It leads on HallusionBench, MMVet, and MVBench among the comparisons described, while MMStar trails GLM-4.6-V at 75.9 but exceeds Qwen3-Omni at 68.5 and Gemini 2.5 Pro at 73.6.
- Evaluation scope: Across more than 50 public benchmarks spanning seven multimodal dimensions, Ming-Flash-Omni achieves comparable performance with leading MLLMs.The evaluation also includes three in-house benchmarks covering Wiki Knowledge, streaming multi-turn dialogue, and multi-dialect and multi-domain audio understanding.
- Image generation, editing, and segmentation: Ming-Flash-Omni matches or exceeds strong baselines across generation, editing, and segmentation, including 86.98 on DPG-Bench versus Qwen-Image’s 88.32 and SD3-Medium’s 84.08.It surpasses all other models on Geneval, supports Chinese editing instructions comparably to English, and performs comparably to specialized segmentation models while exceeding other unified MLLMs.
- Audio →Text (Understanding): Ming-Flash-Omni achieves the best overall average on ContextASR-Bench and remains highly competitive across ASR benchmarks, particularly for dialect recognition.Qwen3-ASR (Shi et al., 2026) and Qwen3-Omni-30B-A3B-Instruct (Xu et al., 2025b) outperform it on some subsets and metrics, indicating complementary strengths.
- Text →Audio (Generation): For Text →Audio generation, Ming-Flash-Omni achieves state-of-the-art performance among similarly sized open-source models on SEED-TTS-Eval and 83.25% accuracy on Sichuanese controllable generation.The Sichuanese result surpasses CosyVoice3, and the model also sets a new state-of-the-art result on the podcast task.
- Video + Audio →Text (Video Streaming Conversation): In video streaming conversation, Ming-Flash-Omni improves over Ming-Lite-Omni across all dimensions and surpasses Qwen3-Omni and Gemini 2.5-Pro in accuracy, completeness, and relevance.The qualitative examples additionally cover image generation, editing, segmentation, multi-image editing, ID-photo tasks, and background replacement, with higher quality and controllability than previous versions.
5 Conclusion
Ming-Flash-Omni is a 100-billion-parameter multimodal model that activates only 6.1B parameters per token, improving computational efficiency while scaling capacity and achieving advanced perception and generation capabilities.
- Architecture: Ming-Flash-Omni is built upon Ling-Flash-2.0 with 100 billion parameters, while activating only 6.1B parameters per token.The sparse activation supports scaling model capacity with fewer active parameters per token.
- Multimodal capabilities: The model demonstrates advanced multimodal perception and generation capabilities with improved computational efficiency.The conclusion links these capabilities to more efficient scaling of model capacity.
- Performance: Ming-Flash-Omni achieves state-of-the-art performance across tasks including multi-image and video processing, image generation, and generative segmentation.The cited passage also identifies Contextual Automatic Speech Recognition among its broad task spectrum.
6 Contributors
The paper lists its contributors alphabetically by first name, spanning the authors from Ant Inclusion AI through Zhengyu He.
- Contributors: Authors are listed alphabetically by first name, beginning with Ant Inclusion AI and continuing through Jun Peng Jin Peng.
- Contributors: The contributor list continues from Kaixiang Ji through Weilong Chai.
- Contributors: The list concludes with Xiaomei Wang through YuQian Li, followed by Ziping Ma through Zhengyu He.
Appendix · A Public Benchmarks
The appendix defines a broad public-benchmark suite spanning multimodal understanding, generation, editing, and segmentation. It covers image, video, audio, text, and cross-modal capabilities, including controllable dialect and podcast generation.
- A Public Benchmarks: Image-to-text understanding is evaluated across general image understanding, STEM and reasoning, and related public benchmark tasks.The listed suite includes MMStar, AI2D, HallusionBench, MMBench, MMVet, MMMUval, MathVistamini, and MathVerseVision.
- A Public Benchmarks: Video-to-text understanding is assessed on MVBench, VideoMME, LongVideoBench, MLVU, PerceptionTest, CharadesSTA, and TOMATO.
- A Public Benchmarks: Text-to-image generation is evaluated on GenEval and DPG-Bench to support unified perception-generation abilities.
- A Public Benchmarks: Image-to-image editing is evaluated using GEdit-Bench.
- A Public Benchmarks: Image-to-image segmentation is evaluated on referring expression segmentation benchmarks RefCOCO/+, and RefCOCOg.
- A Public Benchmarks: Audio-to-text understanding covers fundamental audio benchmarks in Chinese and English plus audio question-answering tasks.Examples include Aishell1, Wenetspeech, Librispeech, and Voxpopuli.
- A Public Benchmarks: Text-to-audio generation is evaluated for speech synthesis, dialect generation, and podcast tasks.Seed-TTS-Eval measures generation, Wenetspeech-chuan evaluates controllable dialect generation, and ZipVoice evaluates podcast tasks.
B The prompt used in data generation
This section specifies prompts for generating portrait-preservation, text-rendering, and video multi-turn conversation data. The video prompt enforces structured JSON outputs with categorical, descriptive, and evaluative fields.
- Portrait preservation data: The portrait-preservation prompt generates realistic life-scene descriptions conditioned on a person’s age and gender, requiring clothing, location, action, and style within 50 words.It varies scenes and costumes to ensure similar images receive distinct descriptions.
- Text generation data: The text-generation prompt randomly selects a theme and produces Chinese, English, and numeric text of 3–5 characters, plus an image description under 100 characters.It also specifies fonts, colors, and layouts according to the requested text style.
- Video multi-turn conversation data: The video-analysis prompt requires an expert analyst to return only one structured JSON object, with no additional explanation or introductory text.The required output is enclosed in a single JSON code block.
- Video multi-turn conversation data: The video schema assigns one category from 34 options, an English caption of at most 30 words, and a 1–10 content score measuring conversational potential.The content score distinguishes sparse or repetitive videos from richer scenes that support questions and interaction.
- Video multi-turn conversation data: The video schema also records field of view as FPV or non-FPV and rates primary task complexity from 1 to 10.Field of view uses 1 for first-person perspective and 0 for third-person or static perspective.