PaperlayerView exact revision on arXiv

Source-linked AI summary

Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

Inclusion AI, :, Bowen Ma, Cheng Zou, ChengKun Du, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Chengxiang Fan, Dandan Zheng, Fudong Wang, Furong Xu, Guangming Yao, Haohao Liu, Han Peng, Jun Zhou, Junluan Xia, Jingdong Chen, Jianing Li, Jianxin Sun, Jianjiang Zhu, Jianping Jiang, Jinpeng Ou, Jun Peng, Jin Peng, Kaixiang Ji, Li Tang, Libin Wang, Lixiang Ru, Longhua Tan, Lu Ma, Lan Wang, Mochen Bai, Minghong Cai, Mingxue Yang, Ning Gao, Qingpei Guo, Qinglong Zhang, Qiang Xu, Qin Zhao, Rui Liu, Ruijie Xiong, Ruobing Zheng, Sirui Gao, Shaoxiong Lin, Tao Zhang, Tianqi Li, Tinghao Liu, Tongli Wang, Taoye Huang, Weilong Chai, Xiaomei Wang, Xiaolong Wang, Xiaojian Liu, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Xuezhi Wang, Yi Yuan, Yuting Gao, Yuting Xiao, Yunxiao Sun, Yipeng Chen, Yifan Mao, Yifei Wu, Yongjie Lyu, Yingying Zhang, YuQian Li, Ziping Ma, Zhiqiang Fang, Zhihao Qiu, Ziyuan Huang, Zizheng Yang, Zhengyu He

arXiv:2510.24821v3cs.CVcs.AI

TL;DR

Unified multimodal models face the challenge of fusing image understanding and generation, where misaligned objectives can limit transfer of fine-grained visual knowledge. Ming-Flash-Omni builds on a sparse MoE foundation with targeted multimodal upgrades and reports state-of-the-art performance across a broad range of perception and generation tasks.

  • Problem

    Misalignment between image-understanding and generation objectives can hinder transfer of fine-grained visual knowledge to high-precision generation and editing.

    Source: §2.3 · p. 5

  • Method

    Ming-Flash-Omni builds on a sparse Ling-Flash-2.0 MoE foundation with 100B parameters, 6.1B active per token, and targeted multimodal upgrades.

    Source: §1 · p. 2, §5 · p. 20

  • Results

    Ming-Flash-Omni reports state-of-the-art performance across multi-image and video processing, image generation, generative segmentation, contextual ASR, and multi-dialect recognition, outperforming comparable-scale omni models.

    Source: §5 · p. 20

  • Takeaways & Limitations

    The authors conclude that releasing the model and code can advance multimodal-intelligence research and enable broader real-world applications.

    Source: §5 · p. 20

Abstract

from arXiv · show

We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. Notably, it achieves strong performance on vision-language understanding benchmarks, with overall scores on par with Gemini 2.5 Pro, and enables seamless switching among multimodal tasks in multi-turn interactions. In speech, it achieves strong performance in contextual and dialect-aware ASR while enabling joint, continuous-generation of speech, sound, and music. In vision, it introduces generative semantic segmentation that achieves competitive standalone performance and enhances spatial control and editing consistency, alongside marked improvements in identity preservation, and high-fidelity in-image text rendering. Together, these capabilities demonstrate that a single unified model can serve as a practical foundation for general-purpose multimodal intelligence.

1 Introduction

Ming-Flash-Omni targets human-like multimodal intelligence by addressing the challenge of unifying comprehension and generation across modalities. It combines a sparse MoE foundation with targeted advances in understanding and generation.

Introduction

  • Although multimodal models show perceptual and generative capabilities in vision and audio, unifying comprehension and generation remains difficult because of representational disparities and modality imbalances.Humans naturally combine modalities and their complementary strengths, motivating models that integrate multimodal intelligence.Source: §1 · p. 1
  • Ming-Flash-Omni builds on Ling-Flash-2.0 lin (2025), a highly sparse Mixture-of-Experts architecture that expands capacity while maintaining bounded inference latency.Its increased sparsity ratio aims to balance performance and efficiency.Source: §1 · p. 2
  • Understanding advances include time-interleaved VideoRoPE Wei et al. (2025), stronger context-aware ASR, and multi-turn switching among heterogeneous comprehension tasks.VideoRoPE interleaves timestamps with frames to align visual tokens with temporal positions; cross-task dialogue can include visual QA, audio captioning, and document analysis.Source: §1 · p. 2
  • Generation advances include joint single-channel speech, sound, and music generation with fine-grained vocal control, generative semantic segmentation, and controllable image synthesis.Continuous representations replace discrete acoustic tokens to avoid quantization artifacts and produce more natural, expressive TTS; image synthesis improves identity preservation and text rendering, with robustness enhanced through vision-specific reinforcement learning.Source: §1 · p. 2
  • Image perception is comparable to Gemini 2.5 Pro, while speech achieves state-of-the-art end-to-end understanding and generation.Image generation supports generative semantic segmentation, fine-grained controllable synthesis, strong identity preservation, and high-fidelity in-image text rendering.Source: §1 · pp. 2–3

2 Ming-Flash-Omni

Ming-Flash-Omni combines a sparse 100B-parameter Ling-Flash-2.0 MoE with a unified perception-and-generation pipeline for multimodal understanding and controllable speech and image synthesis. Its design adds temporal alignment, continuous audio generation, and tighter coupling between image understanding and generation.

2.1 Unified Understanding Across Modalities

  • Freezing the language pathway stabilizes training but misaligns image understanding and generation, limiting transfer of fine-grained visual knowledge to high-precision generation and editing.The resulting bottleneck affects object attributes, spatial relationships, quality, and controllability despite injected hierarchical semantics.Source: §2.1 · p. 3

2.2 Unified Speech Understanding and Generation

  • Speech generation replaces discrete tokens with continuous acoustic latents and uses a low-frame-rate audio tokenizer to shorten sequences and mitigate autoregressive error propagation.A two-stage training strategy maps reference voiceprints to timbre and textual instructions to style without additional modules; training data is unified at 44.1 kHz with built-in super-resolution for lower-rate inputs.Source: §2.2 · pp. 3–4
  • On Instruct-TTS-Eval-zh, Ming-Flash-Omni matches Qwen3-TTS and accurately renders complex mathematical equations and chemical formulas.It also supports joint single-channel generation of speech, sound effects, and music, with fine-grained vocal-attribute control.Source: §2.2 · pp. 3, 5

2.3 Unified Image Understanding and Generation

  • Image generation combines a single-stream diffusion transformer, generative segmentation pretraining, and multi-reward reinforcement learning to couple perception and generation while refining controllability.Segmentation is reframed as semantic-preserving editing, making accurate perception of object boundaries necessary for generation and providing fine-grained spatio-semantic control.Source: §2.3 · p. 5
  • Reference-image features, self-attention, and learnable semantic queries preserve subject identity, while Glyph-byT5 conditioning enables high-fidelity in-image text placement.Motion-aware action labels and balanced sampling improve realism for dynamic scenes; text rendering requires extracting the requested text from the prompt before inference.Source: §2.3 · p. 6

2.4 Overall Training Procedure

  • Perception training progresses from 8K-context pretraining to 24K, then 64K instruction tuning, followed by GRPO and multimodal multi-turn modality-switching optimization.The reward system combines rule-based, model-based, and checklist-based rewards across heterogeneous tasks; generation training freezes the perception model while optimizing the image generator.Source: §2.4 · pp. 6–7

2.5 Infrastructure

  • 28% MFU and more than 4x training throughput over baseline Megatron-LM result from multimodal infrastructure optimizations.Flexible component-specific parallelization and automatically optimized pipeline layouts yield 58.1%–69.7% end-to-end training speedups over uniform manual setups across multiple model scales; custom kernels reduce forward and backward latency by up to 25.3% versus Transformer Engine and deliver 5× speedup over native PyTorch.Source: §2.5 · p. 7

3 Data Construction

Ming-Flash-Omni’s training corpus spans text, images, audio, and video, drawing mainly on Ming-Omni and adding data pipelines for quality, diversity, and deduplication. The section describes modality-specific datasets and construction methods to support multimodal alignment, understanding, and generation.

3.1 Image Data

  • The image corpus combines understanding and generation data, including synthesized datasets; the broader multimodal collection mainly builds on Ming-Omni and uses quality, diversity, and deduplication pipelines.Image-text embeddings and k-means cluster labels support semantically balanced sampling.Source: §3 · p. 8, §3.1 · p. 8, §3.1.2 · p. 10

3.1.1 Image Understanding Data

  • Knowledge data covers biological, cultural, and daily-life entities; recognition uses progressively filtered image matches, while encyclopedia triplets become VQA pairs after multimodal-consensus checks.Recognition filtering combines CLIP consistency, MLLM verification, and manual refinement.Source: §3.1.1 · p. 8
  • STEM training yields 1.6 million multimodal long-CoT samples of up to 8K tokens; OCR training combines millions of samples and uses executable Python as an intermediate reasoning representation.STEM CoT generation includes difficulty-aware filtering and quality validation; OCR data combines open-source, expert-pseudo-labeled, and human-annotated sources.Source: §3.1.1 · pp. 8–9
  • Safety data expands unimodal seeds nearly 30× into multimodal examples, while GRPO training uses task-balanced, difficulty-stratified data and cleaning to build a general-purpose curriculum.Safety responses distinguish prohibited requests, which receive refusals and corrective guidance, from sensitive but safe inquiries, which receive bounded informative responses.Source: §3.1.1 · p. 9

3.1.2 Image Generation Data

  • Image-generation data adds segmentation, text rendering, and portrait preservation, using cluster-aware sampling to balance image-text semantics.Segmentation examples draw on referring, semantic, and panoptic datasets; portrait data includes 200k paired ID photos and 20K landmark portraits, while text-rendering data spans three difficulty levels.Source: §3.1.2 · pp. 9–10

3.2 Audio Data

  • Audio additions include 3 million context-linked ASR samples and TTS clips labeled across speaker and delivery attributes to support controllable synthesis.The TTS pipeline crawls and segments audio before iteratively labeling dimensions including age, style, pitch, emotion, and dialect.Source: §3.2 · pp. 10–11

3.3 Video Data

  • Video data selection scores presentation, content, and reasoning difficulty, while interaction synthesis grounds multi-turn dialogue in long-video events and simulated user cognitive states.Hierarchical event extraction and a look-back mechanism maintain cross-turn consistency for videos up to 30 minutes and 256k tokens.Source: §3.3 · p. 11

3.4 Text Data

4 Evaluation

Ming-Flash-Omni is evaluated across more than 50 public multimodal benchmarks and three in-house benchmarks, with results spanning perception, generation, and interactive multimodal tasks. The section also presents qualitative examples of its understanding and generation capabilities.

4.1 Public Benchmarks

  • The evaluation spans seven public multimodal dimensions and three in-house benchmarks covering factual knowledge, streaming video dialogue, and multi-dialect and multi-domain audio understanding.Wiki Knowledge draws on Wikipedia; StreamingMultiturnBench covers multiple video domains and evaluates accuracy, completeness, relevance, naturalness, and proactivity. The audio datasets contain 51,986 multi-dialect samples from 15 regions and 10,397 multi-domain samples from six domains.Source: §4.1 · p. 11, §4.2 · pp. 11–12

4.3 Quantitative Results

  • Across 50+ multimodal benchmarks, Ming-Flash-Omni performs comparably to leading MLLMs, with strong results in visual reasoning, factual grounding, OCR, and video understanding.It achieves 66.1 on HallusionBench, 85.6 on MMVet, and 77.5 on MVBench, and leads open models on MMMU with 77.1.Source: Table 1 · p. 12, §4.3 · p. 13
  • Ming-Flash-Omni matches state-of-the-art diffusion-model generation quality, surpasses prior methods on Geneval, and scores 86.98 on DPG-Bench.Its DPG-Bench result approaches Qwen-Image’s 88.32 and exceeds SD3-Medium’s 84.08; controllability is especially strong for Position and Color.Source: §4.3 · pp. 13, 15, Table 2 · p. 13
  • For image editing, Ming-Flash-Omni supports Chinese instructions with performance comparable to English and outperforms other models overall; its segmentation is competitive with specialized models.Compared with Z-Image-Edit, it offers comparable semantic consistency with better perceptual quality; it also shows more robust spatial and semantic instruction understanding than other unified MLLMs.Source: Table 3 · p. 14, Table 4 · p. 14, §4.3 · p. 15
  • Ming-Flash-Omni achieves the best ContextASR-Bench average and strong dialect recognition, while surpassing open-source audio-centric and omni models on audio QA except Qwen3-Omni-Flash-Instruct.Qwen3-ASR and Qwen3-Omni-30B-A3B-Instruct perform better on some individual subsets or metrics, indicating complementary strengths.Source: §4.3 · pp. 15, 17, Table 8 · p. 15, Table 9 · p. 16
  • Among comparable-size open-source models, Ming-Flash-Omni achieves state-of-the-art performance on SEED-TTS-Eval test-zh, Sichuanese dialect control, and ZipVoice-Dia(zh) podcast CER and UTMOS.Its Sichuanese control accuracy is 83.25%, outperforming CosyVoice3.Source: Table 5 · p. 14, Table 6 · p. 14, Table 7 · p. 15
  • On StreamingMultiturnBench, Ming-Flash-Omni improves over Ming-Lite-Omni across all dimensions and outperforms Qwen3-Omni and Gemini 2.5-Pro in accuracy, completeness, and relevance.The benchmark uses 380 manually selected videos and 2,200 human-checked question-answer pairs, scored on five dimensions and scaled to 100 points.Source: §4.2 · p. 12, §4.3 · p. 17, Table 11 · p. 17

4.4 Visualization Results

  • Qualitative examples illustrate visual knowledge grounding, mathematical and multi-image reasoning, contextual and dialect-aware ASR, multimodal task switching, and controllable image generation, editing, and segmentation.The examples also include ID-photo generation and editing, background replacement, and multi-image editing.Source: §4.4 · pp. 17, 20, Figure 3 · p. 18, Figure 4 · p. 19

5 Conclusion

Ming-Flash-Omni combines a 100-billion-parameter model with only 6.1B activated per token to improve computational efficiency while scaling capacity. It reports state-of-the-art performance across multimodal tasks and the authors say open-sourcing the model and code will support multimodal research and real-world applications.

5 Conclusion

  • Ming-Flash-Omni achieves SOTA performance across multi-image and video processing, image generation, generative segmentation, ContextASR, and multi-dialect recognition, outperforming omni models of comparable scale.The conclusion attributes these capabilities to a model built on Ling-Flash-2.0 with 100 billion parameters and only 6.1B activated per token.Source: §5 · p. 20
  • With 100 billion parameters and only 6.1B activated per token, Ming-Flash-Omni improves computational efficiency while scaling model capacity.The model is built upon Ling-Flash-2.0.Source: §5 · p. 20
  • The authors state that open-sourcing the models and code will advance multimodal intelligence research, facilitate AGI development, and enable broader real-world applications.Source: §5 · p. 20

Appendix · A Public Benchmarks

The appendix defines public benchmark suites for multimodal understanding and generation across image, video, and audio tasks. Together, they cover perception, editing, segmentation, and controllable media generation.

A Public Benchmarks · Image →Text (Understanding)

  • Image-to-text evaluation spans general image understanding, STEM and reasoning, OCR, and multi-image capabilities.Its benchmark suite includes MMStar, AI2D, HallusionBench, MMBench, MMVet, MMMUval, MathVista, ChartQA, DocVQA, MMTBench, MuirBench, and LLaVA-interleave Bench.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Video →Text (Understanding)

  • Video-to-text understanding is evaluated across seven public benchmarks, including MVBench, VideoMME, LongVideoBench, MLVU, and PerceptionTest.The suite also includes CharadesSTA and TOMATO.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Text →Image (Generation)

  • Text-to-image generation is evaluated on GenEval and DPG-Bench to assess unified perception-generation capabilities.The benchmarks are used for the model’s text-to-image generation capability.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Image →Image (Editing)

  • Image editing is evaluated on GEdit-Bench.This benchmark is used to assess image-to-image editing capability.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Image →Image (Segmentation)

  • Image segmentation is evaluated on the referring expression segmentation benchmarks RefCOCO/+, and RefCOCOg.These standard RES benchmarks assess segmentation capability.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Audio →Text (Understanding)

  • Audio-to-text evaluation covers fundamental audio understanding, audio question answering, and contextual ASR.It uses Chinese and English speech benchmarks, VoiceBench tasks such as AlpacaEval and CommonEval, SD-QA, and ContextASR-Bench.Source: A Public Benchmarks · p. 27

A Public Benchmarks · Text →Audio (Generation)

  • Text-to-audio generation is evaluated with Seed-TTS-Eval, Wenetspeech-chuan for dialect generation, and ZipVoice for podcast tasks.The suite covers general generation, controllable dialect generation, and podcast generation.Source: A Public Benchmarks · p. 27

B The prompt used in data generation

The section specifies prompts for generating portrait-preservation, text-and-image, and video-conversation data, with task-specific content and output requirements.

Prompt for portrait preservation data

  • The portrait-preservation prompt requests a realistic life-scene description conditioned on the person’s age and gender, with varied settings and costumes.Descriptions must include clothing, location, action, and style; stay within 50 words; include at least three realistic details; and be output in English only.Source: B The prompt used in data generation · p. 28

Prompt for text generation data

  • The text-generation prompt asks for short multilingual text and a matching image description styled with suitable font, color, and layout.It specifies Chinese, English, and numbers, 3–5 characters, and a description under 100 characters; the output includes the description, text, font style, font color, and image layout.Source: B The prompt used in data generation · p. 28

Prompt for video multi-turn conversation data

  • The video-analysis prompt requires structured JSON containing a category, caption, content score, field of view, and task-complexity score.It selects one category from a 34-category list, limits the English caption to 30 words, and rates content and task complexity from 1 to 10; field of view distinguishes first-person from third-person or static video.Source: B The prompt used in data generation · pp. 29–30

Related papers

Loading 2510.24821v3…