Source-linked AI summary
Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation
Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, Weilin Huang
TL;DR
Existing unified models generally support multimodal conditioning but remain limited in generating arbitrarily interleaved text and images. Mogao addresses this with a causal hybrid of autoregressive text generation and diffusion image synthesis, plus interleaved-data training; it reports strong multimodal understanding, image generation, and coherent interleaved outputs, alongside zero-shot editing and composition. The paper also notes training-efficiency limitations from communication costs and sequence-length scaling.
Problem
Most existing unified models remain limited to single-modal generation conditioned on multiple modalities, despite the need for arbitrary interleaved multimodal interaction.
Method
Mogao combines autoregressive text generation and diffusion-based image generation through a unified architecture, interleaved multimodal data, efficient training, and multimodal classifier-free guidance.
Results
Mogao achieves state-of-the-art performance in multimodal understanding and text-to-image generation while producing high-quality, coherent interleaved outputs with emergent zero-shot editing and element composition.
Takeaways & Limitations
Mogao demonstrates a practical omni-foundation model that extends unified systems from single-modal generation toward causal interleaved multimodal generation.
Takeaways & Limitations
Training alternatives that compute losses only for final images can scale quadratically with sequence length, making them computationally prohibitive for long multimodal sequences.
Abstract
from arXiv · showhide
Recent progress in unified models for image understanding and generation has been impressive, yet most approaches remain limited to single-modal generation conditioned on multiple modalities. In this paper, we present Mogao, a unified framework that advances this paradigm by enabling interleaved multi-modal generation through a causal approach. Mogao integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance, which allow it to harness the strengths of both autoregressive models for text generation and diffusion models for high-quality image synthesis. These practical improvements also make Mogao particularly effective to process interleaved sequences of text and images arbitrarily. To further unlock the potential of unified models, we introduce an efficient training strategy on a large-scale, in-house dataset specifically curated for joint text and image generation. Extensive experiments show that Mogao not only achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs. Its emergent capabilities in zero-shot image editing and compositional generation highlight Mogao as a practical omni-modal foundation model, paving the way for future development and scaling the unified multi-modal systems.
1 Introduction
Mogao targets the gap between single-modal generation systems and human-like interaction with arbitrarily interleaved text-and-image inputs and outputs. It combines autoregressive text generation with diffusion-based image generation, supported by efficient training and technical improvements for interleaved modeling.
- Current LLMs and text-to-image models typically process single-modal inputs and produce single-modal outputs, limiting human-like multimodal interaction.
- Mogao combines autoregressive text generation with diffusion-based image generation in a unified architecture.Text and image representations are processed in a shared sequence with modality-specific transformer components and separate text and image objectives.
- Distinct modality-specific position identifiers and frequency patterns, together with separate supervision, are designed to reduce interference between text and image tasks.The text objective uses next-token prediction, while image generation uses diffusion loss.
- Mogao supports native interleaved multimodal modeling through a tenmillion-scale dataset and compute-efficient training of textual and corrupted visual tokens.
- A dual classifier-free guidance mechanism combines empty and visual-only conditions to improve image-generation precision and address repetition during interleaved generation.
- Experiments report state-of-the-art performance across multiple open benchmarks, with human evaluations supporting multimodal generation quality.
2 Related Work
Related work unifies visual understanding and generation by adapting autoregressive language models or embedding diffusion modules into shared backbones. These approaches face modality misalignment, representation trade-offs, and challenges in producing high-quality interleaved outputs.
- Unified foundation models aim to combine vision-language understanding with diffusion-based visual generation, which were traditionally handled by separate models.
- Autoregressive approaches represent images as continuous features or discrete visual tokens, but often generate lower-quality images than diffusion-based methods.
- Embedding diffusion modules into shared language-model backbones can improve visual generation but may sacrifice understanding because the tasks conflict through shared parameters.
- Visual representations optimized for understanding may be suboptimal for generation, making tokenizer and latent-space design important.
- Interleaved-data models support mixed-modality conditioning for understanding, text-to-image generation, editing, and composition, while diffusion-based denoising improves visual fidelity.
3 Mogao
Mogao unifies autoregressive text modeling and diffusion-based image generation in a causal backbone that supports arbitrary interleaved multimodal sequences. Its design combines modality-specific processing, dual visual representations, interleaved positional encoding, multimodal guidance, and efficient teacher-forced training.
- Interleaved Multi-Modal Generation: Mogao models visual and textual modalities causally, adding each generated modality to the history so any modality can condition on an arbitrary mixture of modalities.The model represents x_t as a modality rather than an individual token and models sequential modality-level distributions.
- Unified Framework: The unified backbone combines next-token prediction for text with flow matching for noised images in a weighted training objective.Text uses NTP loss, while images use diffusion-style flow matching loss within interleaved samples.
- Improved Dual Visual Encoder: Mogao uses dual visual encoders: ViT features route to the text branch for semantic understanding, while VAE features support visual generation.The framework uses both VAE and ViT when images are conditions, but VAE alone during image generation; timestep conditioning modulates visual features through AdaLN.
- Interleaved Rotary Position Embedding: IL-RoPE captures image height, width, and temporal positions by interleaving frequency assignments across T, H, and W dimensions.Image spatial IDs restart from zero for each image, while temporal IDs advance across images and text; alternating assignments balance local and long-range semantics.
- Multi-Modal Classifier-Free Guidance: Multimodal classifier-free guidance separately controls text and image conditions because conditioned images can shortcut subsequent image generation and cause temporal stagnation.The condition is decomposed into c_txt and c_img, with distinct guidance coefficients for their influence.
- Efficient Complete Teacher Forcing: Efficient Complete Teacher Forcing addresses training–inference domain shift by decoupling clean images from noisy images through a dynamic causal attention mask.The method prevents redundant computation while allowing VAE tokens to attend broadly and restricting text and ViT tokens from attending to VAE tokens.
4 Experiments
Experiments evaluate Mogao's interleaved generation, text-to-image generation, multimodal understanding, and architectural components using public benchmarks, human ratings, visualizations, and controlled ablations. Results show strong benchmark and human-evaluation performance, emergent editing abilities, and benefits from the proposed position embeddings, dual visual encoder, and deep-fusion architecture.
- 4.1 Experimental Setup: Mogao is evaluated on GenEval, DPG-Bench, and GenAI-Bench for text-to-image generation, with GenEval prompts rewritten to match the training format.GenEval contains 553 prompts with four generated images per prompt; DPG-Bench contains 1,065 dense prompts, while GenAI-Bench includes basic and advanced prompts.
- 4.3 Text-to-Image Generation: Mogao achieves comparable GenEval performance to Janus-Pro-7B while activating only 3B parameters per token from its 7B-parameter model.The reported activated-parameter count indicates lower inference cost than Janus-Pro-7B.
- 4.3 Text-to-Image Generation: Mogao achieves strong generation ability consistently across the reported text-to-image benchmarks.
- 4.3 Text-to-Image Generation: Expert human evaluations rate Mogao on text-image alignment and structural correction using 1-to-5 scores averaged across reviewers.The evaluation uses Bench-240 and follows the strategy of Seedream 2.0.
- 4.3 Text-to-Image Generation: Mogao ranks first in human evaluations across three dimensions and significantly outperforms Emu3-Gen and Janus-Pro.The authors describe this result as indicating potential for practical real-world use.
- 4.3 Text-to-Image Generation: Mogao produces aesthetically strong images with composition, color manipulation, reality-virtual interplay, emotional conveyance, and poster-style text rendering.These results are obtained despite fine-tuning with a limited dataset of high-quality 512 × 512 images.
- 4.4 Multi-modal Understanding: Mogao achieves state-of-the-art results on POPE, MME-P, SEED, and MMMU and the highest average score across six multimodal-understanding benchmarks.The comparison covers multimodal understanding and unified models.
- 4.5 Ablation Studies: Ablation experiments use a 1.5B Qwen2 backbone, 256 × 256 maximum resolution, and 35,000 training steps to test key designs.The experiments report multimodal-understanding metrics and image-generation loss curves.
5 Conclusion
Mogao is introduced as an omni-foundation model that combines autoregressive and diffusion modeling for interleaved multimodal generation. Its causal modal-level generation and large-scale interleaved training support multimodal generation, zero-shot editing, and element composition.
- 5 Conclusion: Mogao integrates autoregressive and diffusion models through key architecture designs to support multimodal understanding and image generation.
- 5 Conclusion: Causal interleaved generation at the modal level extends Mogao beyond prior unified models' single-modal generation capabilities.
- 5 Conclusion: Large-scale training on interleaved multimodal data supports Mogao's interleaved generation, zero-shot image editing, and element composition capabilities.The authors report strong consistency for the emergent editing and composition abilities.