PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training

Peng Sun, Jun Xie, Tao Lin

arXiv:2603.16139v1cs.CV

TL;DR

UMMs are limited by scarce paired data and inefficient visual-generation pre-training. IOMM addresses this with image-only pre-training followed by mixed-data fine-tuning, achieving strong performance and efficiency while improving instruction following and generation quality.

  • Problem

    UMM visual-generation training depends heavily on scarce, curated image-text pairs and inefficient procedures, limiting data- and compute-efficient development.

  • Method

    IOMM pre-trains visual generation on unlabeled images, then fine-tunes with a mixture of image-only and high-quality paired data.

  • Results

    IOMM achieves SOTA or comparable performance across diverse benchmarks while operating with substantially greater compute efficiency.

  • Takeaways & Limitations

    Mixed-data fine-tuning consistently enhances instruction-following fidelity and image-generation quality across existing powerful UMMs.

  • Takeaways & Limitations

    The image-only pre-training approach assumes that image content can provide sufficient semantic conditioning and may risk learning a trivial identity mapping without masking.

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) are often constrained by the pre-training of their $\textbf{visual generation components}$, which typically relies on inefficient paradigms and scarce, high-quality text-image paired data. In this paper, we systematically analyze pre-training recipes for $\textbf{UMM visual generation}$ and identify these two issues as the major bottlenecks. To address them, we propose $\textbf{Image-Only Training for UMMs (IOMM)}$, a data-efficient two-stage training framework. The first stage pre-trains the visual generative component $\textbf{exclusively}$ using abundant unlabeled image-only data, thereby removing the dependency on paired data $\textbf{for this costly phase}$. The second stage fine-tunes the model using a mixture of unlabeled images and a small curated set of text-image pairs, leading to improved instruction alignment and generative quality. Extensive experiments show that IOMM not only improves training efficiency but also achieves state-of-the-art (SOTA) performance. For example, our IOMM-B (3.6B) model was trained from scratch using only $\sim \textbf{1050}$ H800 GPU hours (with the vast majority, $\textbf{1000}$ hours, dedicated to the efficient $\textbf{image-only pre-training stage}$). It achieves $\textbf{0.89}$ on GenEval and $\textbf{0.55}$ on WISE--surpassing strong baselines such as BAGEL-7B (0.82 & 0.55) and BLIP3-o-4B (0.84 & 0.50). Code is available $\href{https://github.com/LINs-lab/IOMM}{https://github.com/LINs-lab/IOMM}$.

1. Introduction

UMM training is constrained by scarce paired data and inefficient procedures, motivating IOMM’s two-stage image-only pre-training and mixed-data fine-tuning paradigm. Experiments report improved efficiency, strong benchmark performance, and enhanced instruction-following and generation quality.

  • Motivation: Current UMM training relies on costly proprietary text-image datasets and inefficient procedures, creating a need for more data- and compute-efficient training.The paper frames reducing paired-data reliance while improving performance as a central research question.
  • IOMM framework: IOMM uses unlabeled image-only data for pre-training, then fine-tunes with a mixture of image-only and high-quality paired data.This two-stage design targets the paired-data bottleneck while supporting instruction following and image generation.
  • Technical contributions: IOMM combines a residual query adapter with masked image modeling to adapt frozen MLLMs and learn a robust visual prior with minimal parameter overhead.The masked objective frames pre-training as sparse-to-dense reconstruction.
  • Training analysis: The authors systematically analyze six training recipes combining image-only, text-image, and mixed data across pre-training and fine-tuning.The two-stage image-only pre-training followed by mixed-data fine-tuning recipe yields the best performance in their analysis.
  • Results: IOMM models achieve SOTA or comparable performance across diverse benchmarks while using substantially greater compute efficiency.The introduction attributes the efficiency claim to results discussed in Section 4.
  • Results: IOMM’s mixed-data fine-tuning strategy consistently enhances instruction-following fidelity and image-generation quality across existing UMMs, including Qwen-Image.The paper presents this strategy as generalizable across diverse models.

2. Related Work

Related work spans specialized text-to-image systems and unified models that combine multimodal understanding with generation. Despite architectural diversity, existing unified models remain heavily dependent on curated image-text pairs, while masked signal modeling motivates self-supervised alternatives.

  • Text-to-image diffusion models: Text-to-image research has advanced through diffusion architectures and training refinements, including LDM, MM-DiT, data curation, and preference optimization.These systems primarily target text-to-image synthesis rather than unified multimodal understanding and generation.
  • Text-to-image diffusion models: Specialized text-to-image models lack inherent multimodal understanding, limiting direct use in interactive tasks such as dialogue-based image editing.Such tasks require comprehension and generation to operate together.
  • Unified understanding and generation models: Unified understanding-and-generation models are trained either end-to-end from scratch or by integrating pre-trained components.The from-scratch group includes autoregressive and masked autoregressive systems.
  • Unified understanding and generation models: Component-based unified models commonly connect frozen MLLMs with diffusion backbones through learnable queries or multi-stage training protocols.This integration supports both comprehension and generation across a broad task range.
  • Unified understanding and generation models: UAE and ViLex model UMMs as auto-encoding tasks that reconstruct input images to improve understanding and generation.
  • Masked signal modeling: Existing unified models still depend heavily on large, carefully curated image-text datasets, creating a barrier to open and community-driven research.The limitation concerns the generative modules of these models.
  • Masked signal modeling: Masked signal modeling trains models to reconstruct randomly masked portions of an input, providing a self-supervised basis for generative tasks.The paper situates this approach as a foundation for predicting masked visual tokens and related objectives.

3. Methodology

IOMM pre-trains UMM visual generation from image-only data by combining frozen-MLLM self-conditioning, a lightweight adapter, and masked image modeling. Its flow-matching generator learns from masked visual conditions through sparse-to-dense reconstruction while preserving the MLLM’s original weights.

  • Image-Only Pre-training: Image-only pre-training uses a frozen MLLM and ViT encoder to derive conditioning directly from unlabeled images, avoiding paired text during this phase.The image features are combined with a fixed auxiliary prompt before MLLM processing.
  • Flow Matching: The flow-matching generator interpolates data images with Gaussian noise and learns the constant-velocity field needed to reverse this path.Generation begins from Gaussian noise and integrates the learned field backward to obtain the final image.
  • Residual Query Adapter: 29M-parameter Residual Query Adapter uses 256 learned query tokens and cross-attention to refine conditioning without modifying the frozen MLLM.The adapter appends a learned residual query to the original condition, addressing the mismatch between understanding-oriented representations and generative control.
  • Training Procedure: The training procedure samples images, noise, and time, masks image embeddings, refines the condition with the adapter, computes the loss, and updates trainable parameters until convergence.The frozen MLLM and ViT provide conditioning while the diffusion network and residual query adapter are trainable.
  • Masked Image Modeling: Masked Image Modeling randomly masks image patch tokens, forcing the model to infer missing content and learn robust, context-aware visual representations.This changes dense reconstruction into sparse-to-dense reconstruction and is reported to improve generation quality.

4. Experiment

Experiments show that IOMM’s image-only pre-training and mixed-data fine-tuning improve efficiency, generation quality, and generalization across UMMs and benchmarks.

  • Performance: IOMM-B achieves 0.89 on GenEval, surpassing BAGEL (0.88) and BLIP3-o-8B* (0.84) with 1050 H800 GPU hours.The BLIP3-o-8B* baseline used an additional 30M proprietary image-text pairs.
  • Scaling: With equal five-epoch training, IOMM-L outperforms IOMM-B on GenEval, 0.87 versus 0.86, indicating a positive scaling trend.The lower unconditioned IOMM-L result reflects training for half as many epochs.
  • Data Composition: Image-only pre-training consistently matches or outperforms text-image-pair pre-training across fine-tuning data compositions.This comparison is evaluated across distinct datasets and focuses on text-to-image generation.
  • Data Composition: Mixed-data fine-tuning yields the highest GenEval performance for models pretrained with either data paradigm, while image-only fine-tuning performs worst.The mixed strategy also consistently outperforms alternatives on OpenUni-L-3.6B and Qwen-Image-20B.
  • Generalization: Text-image-pair or mixed-data fine-tuning improves WISE performance, whereas image-only fine-tuning harms prompt-following ability across nearly all open-source UMM scenarios.The uplift reaches 0.10 for OpenUni-L and 0.01 for Qwen-Image.
  • Image Editing: Image-only pre-training enables competitive zero-shot ImgEdit-Bench performance without task-specific editing fine-tuning, outperforming the paired-data counterpart and several editing-trained baselines.The comparison includes UltraEdit, which was explicitly trained on editing datasets.
  • Ablations: A fine-tuning mix ratio of approximately 0.5 gives the best results and greater training stability, whereas lower ratios can become volatile late in training.Here, 1.0 denotes pure image-only data and 0.0 denotes pure text-image pairs.

5. Conclusion

IOMM trains UMM visual generation components primarily with image-only data, then uses mixed-data fine-tuning to achieve strong performance efficiently and improve existing UMMs.

  • Conclusion: IOMM addresses the paired-data bottleneck with image-only pre-training followed by mixed-data fine-tuning.The framework is presented as an efficient approach for training UMM visual generation components.
  • Conclusion: Mixed-data fine-tuning consistently enhances the performance of existing powerful UMMs, supporting the generalizability of the strategy.The conclusion points to detailed settings and results in Appendix B.

B.1. Pre-training Settings

The pre-training and fine-tuning settings specify datasets, image processing, model adaptation, and training configurations for the evaluated UMMs.

  • Pre-training Settings: Approximately 11 million images from Megalith-10M and text-to-image-2M were used for pre-training, except for IOMM-XL.Images were resized to a 512-pixel shorter edge while preserving the original aspect ratio.
  • Fine-tuning Settings: Approximately 210,000 high-resolution images from three datasets were used for fine-tuning, except for IOMM-XL.The datasets were BLIP3o-60K, Echo-4o-Image, and ShareGPT-4o-Image; images were 1024 × 1024 before central cropping.
  • UMM Fine-tuning Settings: OpenUni-L used full fine-tuning of its connector and generative model, whereas Qwen-Image-20B used LoRA with a frozen understanding module.EMA decay was not implemented for Qwen-Image-20B because of computational constraints.

C.1. DPGBench Evaluation Results

The DPGBench results in Table 1 are accompanied by detailed results in Table 7.

  • Table 7 provides detailed DPGBench evaluation results corresponding to the evaluation reported in Table 1.
  • The section points readers to Table 7 for the detailed breakdown of the DPGBench evaluation.

C.2. WISE Evaluation Results

The WISE results in Table 1 are accompanied by detailed results in Table 8, while Table 7 documents DPGBench results.

  • Table 8 provides detailed WISE evaluation results corresponding to the evaluation reported in Table 1.
  • Table 7 is identified as containing DPGBench evaluation results, including a separately noted BLIP3-o-8B* training variant.The caption states that BLIP3-o-8B* was trained with 30 million proprietary data.

C.3. Different training recipe

The compared training recipes used the same broad epoch schedule and image counts, while differing in pre-training data type; image-only pre-training with mixed-data fine-tuning performed better across most GenEval metrics.

  • Training Schedule: All models underwent approximately 5 epochs of pre-training on 11 million images, followed by 10 fine-tuning epochs on approximately 210,000 images.
  • Different Training Recipe: The model pre-trained exclusively on image-only data and fine-tuned on mixed data achieved superior performance across most GenEval metrics.

C.4. Image Editing Results

Image-only pre-training improves zero-shot image-editing consistency relative to image-text-pair pre-training under otherwise consistent settings. The section also reports detailed WISE results for UMM fine-tuning.

  • Image-only pre-training produces superior zero-shot consistency with the original input image than image-text-pair pre-training.The compared models differ only in pre-training data type, while hyperparameters and fine-tuning settings remain consistent.
  • The image-only pre-trained model preserves gestures from the original input more closely in multiple editing examples.The first-row output closely resembles the raw input, while second- and third-row outputs maintain nearly identical gestures.
  • Detailed WISE scores are reported for the UMM fine-tuning results.

C.6. Generation results comparison of UMM finetuning

Fine-tuning improves qualitative generation outcomes, including reasoning-task performance, visual detail, and textual alignment. The section presents comparisons across UMM models and training recipes, alongside the associated WISE evaluation and prompt examples.

  • Fine-tuning enhances performance on tasks requiring reasoning while the understanding module remains frozen.Improved image-text alignment enables more accurate generation of desired details.
  • WISE evaluation results are presented for UMM fine-tuning, with BLIP3-o-8B* denoting a model trained with 30 million proprietary data.Table 10 uses A⊕B to denote results obtained by combining methods A and B.
  • Training recipes are compared using GenEval scores, with bold marking the best performance and underlining marking the second-best performance.
  • Generation results are compared before and after fine-tuning for OpenUni-L.The figure contrasts outputs from the original model with outputs from the fine-tuned model.
  • Fine-tuning gives Qwen-Image richer visual detail and improved alignment to textual prompts.The comparison is between the original Qwen-Image model and its fine-tuned version.
Loading 2603.16139v1…