PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities

Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang

arXiv:2505.02567v6cs.CV

TL;DR

Unified multimodal models aim to combine multimodal understanding and image generation, but their autoregressive and diffusion foundations create architectural challenges. This survey organizes existing approaches, assembles evaluation resources, and identifies challenges for advancing the field.

  • Problem

    Autoregressive models dominate multimodal understanding while diffusion models dominate image generation, and existing evaluations often assess these capabilities separately rather than their interaction.

  • Method

    The survey reviews foundational models and unified systems across diffusion-based, autoregressive-based, and hybrid paradigms, while compiling datasets and benchmarks.

  • Results

    The survey provides a structured landscape of unified multimodal models, categorizes their designs, and curates resources for training and evaluation.

  • Takeaways & Limitations

    Future progress depends on improving tokenization, cross-modal attention, data quality, and evaluation protocols for unified models.

  • Takeaways & Limitations

    Unified models remain constrained by long multimodal token sequences, cross-modal attention bottlenecks, noisy or biased image–text data, and isolated-task evaluation protocols.

Abstract

from arXiv · show

Recent years have seen remarkable progress in both multimodal understanding models and image generation models. Despite their respective successes, these two domains have evolved independently, leading to distinct architectural paradigms: While autoregressive-based architectures have dominated multimodal understanding, diffusion-based models have become the cornerstone of image generation. Recently, there has been growing interest in developing unified frameworks that integrate these tasks. The emergence of GPT-4o's new capabilities exemplifies this trend, highlighting the potential for unification. However, the architectural differences between the two domains pose significant challenges. To provide a clear overview of current efforts toward unification, we present a comprehensive survey aimed at guiding future research. First, we introduce the foundational concepts and recent advancements in multimodal understanding and text-to-image generation models. Next, we review existing unified models, categorizing them into three main architectural paradigms: diffusion-based, autoregressive-based, and hybrid approaches that fuse autoregressive and diffusion mechanisms. For each category, we analyze the structural designs and innovations introduced by related works. Additionally, we compile datasets and benchmarks tailored for unified models, offering resources for future exploration. Finally, we discuss the key challenges facing this nascent field, including tokenization strategy, cross-modal attention, and data. As this area is still in its early stages, we anticipate rapid advancements and will regularly update this survey. Our goal is to inspire further research and provide a valuable reference for the community. The references associated with this survey are available on GitHub (https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).

1 INTRODUCTION

Unified multimodal models seek to combine autoregressive multimodal understanding with diffusion-based image generation, but their architectural differences create unresolved design challenges. This survey reviews the field’s models, tokenization strategies, datasets, benchmarks, and open problems.

  • Autoregressive architectures dominate multimodal understanding, whereas diffusion-based models have become central to text-to-image generation.
  • Unified models could reason about visual data, follow complex instructions, and visualize multimodal analyses through generated outputs.
  • A central unresolved issue is how to tokenize images for autoregressive generation using discrete or continuous representations and varied visual encoders.
  • The survey categorizes unified models into diffusion-based, autoregressive-based, and hybrid paradigms, further grouping autoregressive and hybrid models by image tokenization strategy.
  • It assembles datasets and benchmarks spanning multimodal understanding, text-to-image generation, image editing, and related tasks.
  • The survey identifies efficient tokenization, data construction, and model evaluation among the key challenges for unified multimodal models.

2 PRELIMINARY

Multimodal understanding models connect multimodal encoders to LLMs, while image generation uses diffusion or autoregressive sequence modeling. These paradigms provide distinct foundations and motivate unified architectures.

  • Multimodal Understanding Models: Multimodal understanding models encode images, audio, or videos into features, then use connectors to provide LLM inputs.Connectors are projection-based, query-based, or fusion-based.
  • Autoregressive Models: Autoregressive vision models factorize image generation into sequential predictions over pixels, visual tokens, or multiple tokens per step.These models map images to one-dimensional discrete sequences and condition each prediction on preceding elements.
  • Multimodal Understanding Models: Decoder-only multimodal models transform image embeddings into LLM token space through connectors, including projection and querying mechanisms.Examples include MiniGPT-4 and BLIP-2, which connect visual encoders with language models.
  • Diffusion Models: Diffusion generation combines a forward chain that adds Gaussian noise with a learned reverse chain that progressively denoises data.The reverse network receives noised data and a timestep and predicts normal-distribution parameters for denoising.
  • Diffusion Models: Diffusion models evolved from U-Net-based pixel or latent methods toward transformer-based DiT architectures conditioned on diffusion timesteps and additional signals.Later methods incorporate visual representations, separate modality weights, and multimodal language-model priors.

STANDING AND GENERATION

Unified multimodal models combine modality-specific encoders, a fusion backbone, and modality-specific decoders to support understanding and generation. Existing approaches span diffusion, autoregressive, and hybrid backbones, with encoding choices and efficiency–quality trade-offs shaping their designs.

  • Unified multimodal frameworks: Unified frameworks encode multiple input modalities, fuse them for cross-modal reasoning, and decode outputs into one or more target modalities.Typical systems use modality-specific encoders, a modality-fusion backbone, and modality-specific decoders.
  • Architectural taxonomy: Existing vision-language unified models are grouped into diffusion, autoregressive, and fused AR + diffusion categories.The survey focuses on models taking image and text inputs and producing text or image outputs.
  • Encoding strategies: Autoregressive visual tokenization is classified as pixel-based, semantic-based, learnable query-based, or hybrid encoding.These strategies represent different ways of handling visual and textual data within the autoregressive framework.
  • Diffusion-based models: Diffusion models support high-quality, controllable image generation and can condition denoising on multimodal contexts for synchronized text-image outputs.Conditioning signals include textual descriptions, images, or joint embeddings; guidance and accelerated sampling improve controllability and efficiency.
  • Challenges: Unified discrete diffusion models remain constrained by inference speed, sparse supervision, output-length bias, limited infrastructure, and deployment complexity.Parallel decoding can degrade quality, while immature pipelines complicate fair comparisons and real-world use.
  • Fused autoregressive and diffusion models: Hybrid models combine autoregressive text generation with diffusion image synthesis, using joint supervision to balance language-based control and visual fidelity.Pixel-based hybrid approaches use VAE-derived visual representations, while hybrid designs generally incur greater complexity and alignment costs.

4 DATASETS ON UNIFIED MODELS

The survey organizes datasets for unified multimodal models by their primary use in understanding, generation, editing, and related multimodal tasks. These resources vary from large-scale image-text collections to specialized datasets for instruction tuning, aesthetic quality, text rendering, artistic generation, and human-image synthesis.

  • Multimodal Understanding Datasets: Multimodal understanding datasets primarily pair images with text to train capabilities such as captioning, visual question answering, retrieval, and grounding.They include broad web-scale collections and specialized instruction-tuning resources.
  • Multimodal Understanding Datasets: 100 million Chinese image-text pairs make Wukong a large-scale pre-training resource designed for Chinese multimodal scenarios.The dataset addresses a shortage of large-scale, high-quality Chinese multimodal pre-training data.
  • Multimodal Understanding Datasets: LAION-5B contains nearly 6 billion web-crawled image-text pairs, while Laion-COCO contains 600 million samples with captions closer in style to MS COCO.Both are described as resources for large-scale multimodal pre-training.
  • Multimodal Understanding Datasets: Instruction-tuning resources include ShareGPT4V with approximately 100K conversational image-text points and ALLaVA with 1.4 million synthetically generated samples.ShareGPT4V targets instruction following and dialogue, whereas ALLaVA emphasizes detailed captions and reasoning-oriented visual question answering.
  • Text-to-Image Generation Datasets: Text-to-image datasets emphasize image-text pairs with aesthetic quality, rich content, or stylistic attributes, including CC-12M and LAION-Aesthetics.LAION-Aesthetics selects approximately 120 million images and texts using an aesthetic scoring model.
  • Specialized Generation Datasets: Specialized generation datasets target text rendering, artistic image-prompt mappings, and human-image synthesis through resources such as Mario-10M, JourneyDB, and CosmicMan-HQ 1.0.Mario-10M supports text placement and legibility, JourneyDB contains 4 million Midjourney image-prompt pairs, and CosmicMan-HQ 1.0 contains 6 million annotated human images.

5 BENCHMARKS

The survey reviews benchmarks for multimodal understanding, image generation, editing, and unified interaction. It emphasizes that isolated task evaluations cannot determine whether understanding and generation complement one another, motivating dedicated unification benchmarks.

  • Multimodal Understanding: Understanding benchmarks span perception, grounding, retrieval, visual question answering, commonsense reasoning, and domain-specific mathematical or chart-based tasks.Examples include MMBench, MMMU, VQA extensions, ChartQA, MathVista, and General-Bench.
  • Image Generation: Image-generation benchmarks increasingly evaluate compositional reasoning, prompt alignment, spatial understanding, commonsense grounding, quality, robustness, fairness, and efficiency.The survey lists GenEval, GenAI-Bench, HRS-Bench, T2I-CompBench++, VI-SOR, Commonsense-T2I, and HEIM among these evaluations.
  • Image Editing: Image-editing benchmarks assess multi-step instructions, fine-grained edits, inpainting, causal reasoning, masks, and human-aligned quality across diverse edit types.Examples include I2EBench, EditVal, Emu-Edit, Reason-Edit, EditBench, HumanEdit, and IE-Bench.
  • Evaluation on Unification: Existing evaluation paradigms mainly assess understanding and generation separately, so they cannot determine whether unified models achieve complementary interaction between the two capabilities.The survey identifies understanding-enhanced generation and generation-enhanced understanding as examples of this interaction.
  • Evaluation on Unification: RealUnify evaluates unification through Understanding-Enhanced Generation and Generation-Enhanced Understanding using 1,000 human-annotated instances across 10 categories and 32 subtasks.It combines end-to-end and diagnostic step-by-step evaluation to distinguish core capability deficiencies from integration failures.

6 CHALLENGES AND OPPORTUNITIES ON UNIFIED MODELS

Unified multimodal models remain at a rudimentary stage, with challenges spanning efficiency, attention, data quality, evaluation, and broader capabilities. Image editing and several controlled or interleaved generation functions remain largely unexplored or dependent on postfinetuning.

  • Tokenization and efficiency: Visual and textual data create extremely long token sequences, making efficient tokenization and compression necessary to reduce memory and computation costs while preserving fidelity.The challenge arises from the high dimensionality of both modalities.
  • Cross-modal attention: Cross-modal attention becomes a bottleneck as image resolution and context length increase, motivating sparse or hierarchical alternatives.
  • Data: Noisy or biased image–text pairs, especially in complex compositions and interleaved data, require filtering, debiasing, and synthesis for fairness and robustness.
  • Evaluation: Existing evaluation protocols typically isolate tasks, creating a need for integrated benchmarks covering understanding and generation, including image editing and interleaved image-text generation.
  • Capability gaps: Image editing is often achieved only through postfinetuning, while spatially controlled, subject-driven, and interleaved image-text generation remain largely unexplored.

7 CONCLUSION

The survey presents a comprehensive view of unified multimodal models combining vision–language understanding and image generation. It organizes the field, curates evaluation resources, and identifies challenges and opportunities for a still-infant research area.

  • Survey scope: The survey covers foundational concepts and recent advances in multimodal understanding and text-to-image generation.
  • Taxonomy: Existing unified models are categorized into diffusion-based, autoregressive-based, and hybrid-based paradigms with further subdivisions.
  • Resources: The survey curates datasets and benchmarks to facilitate practical implementation and evaluation.
  • Outlook: Unified multimodal model research is still in its infancy, leaving opportunities for further research and innovation.
Loading 2505.02567v6…