PaperlayerView exact revision on arXiv

Source-linked AI summary

M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance

Qingpei Guo, Kaiyou Song, Zipeng Feng, Ziping Ma, Qinglong Zhang, Sirui Gao, Xuzheng Yu, Yunxiao Sun, Tai-Wei Chang, Jingdong Chen, Ming Yang, Jun Zhou

arXiv:2502.18778v3cs.LGcs.AIcs.CL

TL;DR

Open-source omni-MLLMs lag GPT-4o in modality support and task versatility. M2-omni combines progressive multimodal alignment with balancing strategies for training; it reports competitive performance across diverse tasks, including a 75.1 average on OpenCompass vision-and-text tasks for its 72B model.

  • Problem

    Existing open-source omni-MLLMs fall short of GPT-4o’s comprehensive modality support and task versatility, including audio or visual-generation capabilities.

    Source: §1 · p. 2

  • Method

    M2-omni uses a unified multimodal framework with progressive modality alignment, step balance during pre-training, and dynamically adaptive balance during instruction tuning.

    Source: §1 · pp. 2–3

  • Results

    M2-omni reports competitive performance across image, video, interleaved image-text, audio understanding and generation, and free-form image generation; M2-omni-72B scores 75.1 average on OpenCompass vision-and-text tasks.

    Source: §1 · p. 3, §6 · p. 20

  • Takeaways & Limitations

    The authors release M2-omni and comprehensive training details to facilitate future research in this domain.

    Source: §1 · p. 3, §6 · p. 21

Abstract

from arXiv · show

We present M2-omni, a cutting-edge, open-source omni-MLLM that achieves competitive performance to GPT-4o. M2-omni employs a unified multimodal sequence modeling framework, which empowers Large Language Models(LLMs) to acquire comprehensive cross-modal understanding and generation capabilities. Specifically, M2-omni can process arbitrary combinations of audio, video, image, and text modalities as input, generating multimodal sequences interleaving with audio, image, or text outputs, thereby enabling an advanced and interactive real-time experience. The training of such an omni-MLLM is challenged by significant disparities in data quantity and convergence rates across modalities. To address these challenges, we propose a step balance strategy during pre-training to handle the quantity disparities in modality-specific data. Additionally, a dynamically adaptive balance strategy is introduced during the instruction tuning stage to synchronize the modality-wise training progress, ensuring optimal convergence. Notably, we prioritize preserving strong performance on pure text tasks to maintain the robustness of M2-omni's language understanding capability throughout the training process. To our best knowledge, M2-omni is currently a very competitive open-source model to GPT-4o, characterized by its comprehensive modality and task support, as well as its exceptional performance. We expect M2-omni will advance the development of omni-MLLMs, thus facilitating future research in this domain.

1 Introduction

Existing open omni-MLLMs fall short of GPT-4o’s modality support and task versatility. M2-omni combines a unified multimodal framework with training strategies for balancing modalities, and reports competitive performance among publicly available omni-MLLMs.

1 Introduction

  • Existing omni-MLLMs lack GPT-4o’s comprehensive modality support and task versatility, with limitations in audio support or visual-generation tasks.Source: §1 · p. 2
  • Text-based image generation and discrete-token speech generation are used to reduce cross-modal interference, with speech tokens supporting real-time streaming synthesis.Source: §1 · pp. 2–3
  • M2-omni-72B scores 75.1 average on the OpenCompass benchmark for vision-and-text tasks; the paper releases the model and comprehensive training details.Source: §1 · p. 3
  • M2-omni unifies multimodal understanding and generation through modality-specific pathways and progressive alignment, accepting arbitrary combinations of text, image, video, and audio inputs.Its outputs can interleave text, image, or audio.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Step balance during pre-training and dynamically adaptive balance during instruction tuning address disparities in modality data volume and convergence rates.The tuning strategy assigns smaller gradient weights to slower-converging modalities and larger weights to faster-converging ones.Source: §1 · pp. 2–3

2 Unified Framework for MultiModal Understanding and Generation

M2-omni combines modality-specific processing pathways for multimodal understanding and generation with a staged training framework that progressively incorporates modalities. Its architecture uses text descriptions for image generation and discrete tokens for speech generation.

2.1 Overall Architecture

  • M2-omni uses modality-specific pathways, text descriptions to condition image generation, and discrete audio-token prediction for speech generation.A pretrained Stable Diffusion model generates images from the textual condition, while predicted audio tokens feed a pretrained CosyVoice model.Source: §2.1 · p. 4

2.2 Multi-Stage Training with Progressively Modality Alignment

  • Training proceeds through pre-training, instruction tuning, and alignment tuning; pre-training and instruction tuning each have three progressive sub-stages.The framework incrementally incorporates knowledge from multiple modalities.Source: §2.2 · p. 5

Pre-training

  • Pre-training aligns encoders and the LLM, strengthens fine-grained image-text and OCR understanding, then jointly trains on image, video, audio, interleaved, and text data.Stage 1 concatenates image-text pairs into interleaved sequences, which the paper reports accelerates training efficiency by 1.5×.Source: Pre-training · pp. 5–6

Instruction Tuning

  • Instruction tuning advances from specialized image tasks to broader visual understanding, then adds audio and generation; dynamic balancing addresses inconsistent task volumes and convergence speeds.The final stage supports following instructions on mixed multimodal sequences.Source: Instruction Tuning · pp. 6–7

Alignment Tuning

  • Alignment tuning combines DPO with instruction tuning to refine chat quality and style; LoRA updates 5.0% of the LLM backbone weights.The paper says this limited update is used to prevent catastrophic forgetting.Source: Alignment Tuning · p. 7

3 MultiModal Multi-Task Balanced Strategy

The section balances multimodal training through modality sampling and loss weighting during pre-training, then adapts instruction-tuning weights using validation progress. It also preserves language ability by including pure-text data.

3.1 Step Balance Strategy

  • Step Balance addresses unequal modality data volumes and differently scaled losses by balancing samples and loss weights during multimodal joint pre-training.The section describes random sampling, round-robin, and accumulation as alternatives for sample balance; normalization weights are applied to parameter-gradient updates.Source: §3.1 · pp. 7–8
  • Accumulation consistently outperforms random sampling and round-robin, with the authors attributing the advantage to more stable gradients and adequate training for each modality.It forwards one batch per modality before making a single consolidated update.Source: §3.1 · p. 8

3.2 Dynamic Adaptive Balance Strategy

  • During instruction tuning, validation-loss convergence slopes guide modality-specific weights, while an exponential moving average smooths updates to reduce abrupt fluctuations.Weights start at 1 for the first H validation segments; adjusted weights are used until the next validation segment.Source: §3.2 · pp. 8–10
  • With dynamic adaptive balancing, all modalities achieve performance superior to their corresponding single-modality model counterparts.Source: §3.2 · p. 10

3.3 Language Proficiency Maintenance Strategies

  • Including pure-text data during training when the LLM is unfrozen helps maintain linguistic ability; around 25% text data is reported to preserve it without deteriorating multimodal capabilities.Source: §3.3 · p. 10

4 Data configurations

The data configurations stage multimodal pre-training and instruction tuning, add targeted data for underrepresented instruction categories, and combine instruction data with preference data for alignment tuning.

4.1 Pre-training Data

  • Pre-training progresses from image-text, OCR, and audio-text alignment data to broader multimodal data, including video-text and interleaved image-text; its third stage includes 25.4M, 27.3M, and 36.9M samples of those added types, respectively.The pre-training criteria are modality alignment and world-knowledge concept learning.Source: §4.1 · pp. 10, 12, Figure 6 · p. 11

4.2 Instruction Tuning Data

  • For instruction tuning, a pipeline identifies long-tail academic categories and uses GPT-4 topic generation and GPT-4V image-question generation to construct supplementary data.The collected open-source data were severely imbalanced, with science and mathematics given as examples of scarce categories.Source: §4.2 · p. 12
  • Instruction-tuning data expand in stages: Stage 1 combines image-text and text, Stage 2 adds video-text and interleaved image-text, and Stage 3 adds audio-text, audio-QA, and free-form image generation.Stage 3 also selectively incorporates high-quality video-text data and reuses text and interleaved image-text data.Source: Figure 6 · p. 11, §4.2 · p. 12

4.3 Alignment Tuning Data

  • Alignment tuning uses both preference and instruction-tuning data, with the latter sampled across modalities at a 1:1 ratio to preference data.Preference pairs are built by human selection between responses generated by the SFT-tuned M2-omni model and GPT-4o for user-generated multimodal prompts.Source: §4.3 · p. 13

5 Experiments

The experiments section introduces quantitative and qualitative performance analyses of M2-omni, alongside ablations of selected design components.

5 Experiments

  • The evaluation includes quantitative and qualitative analyses of M2-omni’s performance.Source: §5 · p. 13
  • The section also reports ablation studies of several key design components.Source: §5 · p. 13
  • The stated aim of the ablations is to examine the components’ distinct impacts on model performance.Source: §5 · p. 13

5.1 Quantitative Results

M2-omni is evaluated across image-text, video, audio, speech generation, text-only, user-experience, and free-form dialogue tasks. Results report competitive performance across modalities, with specific strengths and scope limits varying by task.

5.1.1 Image-Text Understanding

  • M2-omni-72B achieves top-tier results on most of eight OpenCompass image-text benchmarks, surpassing GPT-4o and Gemini-1.5-Pro.M2-omni-9B is competitive with similarly sized models.Source: §5.1.1 · p. 13, Table 3 · p. 14

5.1.2 Video & Interleaved Image-Text Understanding

  • M2-omni-9B ranks second on VideoMME and MVBench, while both model sizes surpass other baselines on in- and out-of-domain multi-image benchmarks.The video gains do not scale to 72B, which the authors attribute to limited instruction-tuned video data; the 9B model uses fewer frames than the leading Qwen2-VL-7B.Source: §5.1.2 · pp. 13–14, Table 4 · p. 15

5.1.3 Audio Understanding

  • M2-omni-9B is comparable to GPT-4o-Realtime and ranks second on four speech-recognition benchmarks, while outperforming other baselines on AudioCaps.Source: §5.1.3 · p. 15, Table 5 · p. 16

5.1.5 Text-only Performance

  • M2-omni-9B outperforms its Llama3.1-8B counterpart across most language-benchmark datasets.The authors attribute the result to multi-stage language preservation and high-quality instruction-tuning data.Source: Table 6 · p. 16, §5.1.5 · p. 16

5.1.7 Free-Form Dialogue Generation

  • On 50 realistic multi-turn conversations, M2-omni-9B shows better overall understanding and generation than TextBind.GPT-4o rates relevance, fluency, and informativeness; M2-omni generates interleaved text and images in conversation contexts.Source: Table 7 · p. 17, §5.1.7 · p. 17

5.1.4 Audio Generation

  • M2-omni-9B outperforms MiniCPM-o-2.6 on both SEED-TTS test-zh metrics but still trails traditional speech-generation models.Source: §5.1.4 · p. 15, Table 8 · p. 17

5.1.6 User Experience Evaluation

  • Alignment tuning raises user-experience performance by 5.7%-23.4%, with selected cases validated by human annotations and other-modality OpenCompass performance remaining relatively consistent.The evaluation uses relevance, fluency, informativeness, and format rationality; its dataset contains nearly 300 samples.Source: §5.1.6 · pp. 16–17

5.2 Qualitative Results

Qualitative examples illustrate M2-omni's cross-modal understanding and free-form dialogue generation. The examples include image creation within a conversation without an explicit image-generation request.

  • Examples show image, video, interleaved image-text, and image-audio understanding.Source: §5.2 · p. 17
  • In free-form dialogue, M2-omni can create images from conversation context without explicit user input, for explaining ideas.Source: §5.2 · p. 17

5.3 Ablation Study

The ablations examine step balancing during pre-training and dynamic adaptive balancing during instruction tuning. They report benefits from balancing modality data and adjusting training weights as modalities progress at different rates.

5.3.1 Step Balance Strategy

  • The step balance strategy is reported to improve pretrained-model performance on few-shot VQA and zero-shot image captioning tasks.The evaluation uses OK-VQA, VQAv2, and Flickr30k.Source: §5.3.1 · p. 18, Table 11 · p. 19

5.3.2 Dynamic Adaptive Balance Strategy

  • Under mixed training without balancing, image-text peaks at epoch two, video at epoch one, and audio at epoch three; ASR remains unconverged after three epochs.The comparison starts from pretrained models and contrasts single-modality baselines with a unified three-modality model.Source: §5.3.2 · pp. 18–19
  • Dynamic weighting accelerates audio training while reducing image-text and video weights to prevent overfitting, improving performance across modalities despite residual imbalance.Source: §5.3.2 · p. 19
  • With balancing, the model reaches single-modality-optimal performance on seven of nine benchmarks; its epoch-two model exceeds single-modality baselines on six.At epoch three, the audio modality exceeds those baselines on five benchmarks.Source: §5.3.2 · pp. 19–20

6 Conclusion

The conclusion presents M2-omni as a competitive omni-MLLM with broad modality and task support. It combines multi-stage training and balancing strategies, and the authors release the model and training details for further research.

  • M2-omni supports image, video, interleaved image-text, audio understanding and generation, and free-form image generation, with competitive performance across tasks.Source: §6 · p. 20
  • Multi-stage training enables progressive modality alignment, while step-wise pretraining balance and dynamically adaptive instruction-tuning balance address differences in data volume and convergence rates.Source: §6 · p. 20
  • The authors publicly release M2-omni with data configurations and training procedures to facilitate future research.Source: §6 · p. 21

A More Qualitative Results

The qualitative examples span image understanding, interleaved image-text, video understanding, and audio QA, illustrating M2-omni’s reported capabilities across these tasks.

Image Understanding

Interleave Image-Text Understanding

Video Understanding

Audio QA

  • In audio QA, the model answers questions about audio, including when combined with image or video content.Figure S5 presents audio-QA cases.Source: A More Qualitative Results · p. 38

B Training Data Details

The training-data details cover pre-training and three instruction-tuning stages, with tables documenting data sources, sampling ratios, volumes, and languages.

Pre-training

  • The pre-training data table details open-source data used in the pre-training stage.The section says the training-data listings include sources, sampling ratios, data volumes, and language information.Source: B Training Data Details · p. 38, Table S1 · p. 39

IT-Stage-1-Data

  • Instruction-tuning stage 1 data are detailed in separate tables for image-text and text data.The image-text data are split across two tables.Source: Table S3 · p. 45, Table S4 · p. 46, Table S2

IT-Stage-2-Data

IT-Stage-3-Data

Loading 2502.18778v3…