PaperlayerView exact revision on arXiv

Source-linked AI summary

BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, Ran Xu

arXiv:2505.09568v1cs.CVcs.AI

TL;DR

The paper addresses the underexplored architecture and training recipe for unified image understanding and generation. It evaluates representations, objectives, and training strategies, then introduces BLIP3-o with diffusion-generated CLIP features, sequential training, and BLIP3o-60k. BLIP3-o performs strongly across image understanding and generation benchmarks, while the authors note limitations in generation diversity and ongoing coverage of downstream tasks.

  • Problem

    The optimal architecture and training strategy for unified models that support both image understanding and generation remain underexplored.

  • Method

    BLIP3-o uses a diffusion transformer with flow matching to generate CLIP image features, sequentially trains understanding before generation, and adds the BLIP3o-60k instruction-tuning dataset.

  • Results

    BLIP3-o achieves superior performance across most popular image understanding and generation benchmarks, including 1682.6 on MME-P, 50.6 on MMMU and 0.84 on GenEval for its 8B model.

  • Takeaways & Limitations

    CLIP embeddings with flow matching provide faster training and higher-quality outputs, while BLIP3o-60k improves prompt alignment and visual aesthetics.

  • Takeaways & Limitations

    MSE-based continuous-feature modeling produces nearly deterministic outputs for a prompt, limiting generation diversity; downstream applications such as editing and visual dialogue remain future work.

Abstract

from arXiv · show

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.

1 Introduction

BLIP3-o addresses underexplored design and training choices for unified image understanding and generation. It combines a diffusion-based generation approach, sequential training, curated instruction tuning, and open-source release.

  • Unified multimodal models support both image understanding and generation, but optimal architecture and training strategy for generation remain underexplored.
  • CLIP features and sequential training are identified as key design choices for efficient generation while preserving image understanding capability.The model first trains on image understanding, then trains image generation while freezing the autoregressive backbone.
  • BLIP3o-60k is a 60k-example instruction-tuning dataset created with diverse GPT-4o prompts to improve human-preference alignment and aesthetic quality.
  • 1682.6 on MME-P, 50.6 on MMMU and 0.84 on GenEval are reported for BLIP3-o 8B across image understanding and generation benchmarks.
  • The authors fully open-source model weights, code, datasets, and evaluation pipelines to support further unified multimodal research.

2 Unified Multimodal for Image Generation and Understanding

The paper motivates unified models as a way to combine multimodal understanding, reasoning, instruction following, and generation in one system. It studies an autoregressive-plus-diffusion design because the optimal representation and alignment strategy remain unclear.

  • Unified models aim to combine image understanding and generation within a single multimodal architecture.
  • Adding image generation to autoregressive multimodal models can leverage pretrained knowledge, reasoning, and instruction-following abilities.
  • Joint understanding and generation naturally support in-context learning for iterative editing, visual dialogue, and step-by-step visual reasoning.
  • Unified multimodal architectures are presented as a route toward systems that perceive, interpret, and generate content across modalities.
  • The study adopts an autoregressive-plus-diffusion framework because the optimal architecture and continuous-feature modeling strategy remain unclear.The open choices include the ground-truth image representation and whether to use MSE or diffusion-based modeling.

3 Image Generation in Unified Multimodal

The paper compares continuous image representations and training objectives for unified image generation, using autoregressive models with diffusion-based generation. It finds that CLIP features paired with flow matching provide the most effective overall design, balancing semantic representation, efficiency, diversity, and quality.

  • Autoregressive conditioning: The autoregressive transformer encodes a prompt, appends a learnable query vector Q, and predicts continuous visual features aligned with ground-truth image embeddings X.The predicted features are then used by the image-generation module.
  • Training objectives: MSE aligns predicted visual features with the target mean, while flow matching models the target distribution by transporting Gaussian noise toward the ground-truth image features.Flow matching uses a diffusion transformer to predict the velocity of this process conditioned on Q and the timestep.
  • Training objectives: MSE produces nearly deterministic visual features for a prompt, causing repeated outputs to remain almost identical and limiting generation diversity.This limitation persists with both VAE-based and CLIP + Diffusion visual decoders.
  • Design comparison: CLIP + Flow Matching achieves the best prompt alignment on GenEval and DPG-Bench, while VAE + Flow Matching obtains the lowest FID.Because FID can misrepresent generative quality and prompt alignment, the experiments identify CLIP + Flow Matching as the most effective overall design.
  • Image representations: CLIP encoders provide compact, semantically rich image representations, whereas VAEs encode lower-level pixel features and generally offer stronger reconstruction quality.CLIP-Diffusion requires additional adaptation for reconstruction, while VAEs can be integrated directly into generation pipelines.
  • Design comparison: Flow matching better captures the image distribution, producing greater sample diversity and enhanced visual quality, while CLIP features improve the efficiency of learning semantic-level representations.The figure compares autoregressive-plus-diffusion designs that vary their image-generation components; for flow matching, the autoregressive model remains frozen during module fine-tuning.

4 Training Strategies for Unified Multimodal

BLIP3-o unifies image understanding and generation by aligning both tasks in CLIP’s embedding space and comparing joint with sequential training. Sequential training freezes the autoregressive backbone during image-generation training, preserving understanding capability while dedicating capacity to generation.

  • The unified model uses CLIP + Flow Matching for generation because image understanding also operates in CLIP’s embedding space.This aligns both tasks within a shared semantic space.
  • Joint Training: Joint training mixes image-understanding and image-generation data while updating the autoregressive backbone and generation module simultaneously.This is the multitask-learning approach used by several prior unified models.
  • Sequential Training: Sequential training first trains image understanding, then freezes the MLLM backbone and trains only the image-generation module.The two-stage procedure separates learning of the two capabilities.
  • Training Trade-offs: Sequential training offers greater flexibility by preserving image understanding capability while allocating subsequent training capacity to image generation.Joint-training outcomes depend on total data size and the ratio between understanding and generation data.

5 BLIP3-o: Our State-of-the-Art Unified Multimodal

BLIP3-o combines CLIP representations, Flow Matching, a diffusion transformer, and sequential training into unified models with 4B and 8B variants. The 8B model performs strongly across image-understanding and image-generation evaluations, while BLIP3o-60k improves generation quality and human evaluation favors BLIP3-o over Janus Pro.

  • Model Design: BLIP3-o adopts CLIP + Flow Matching and sequential training for its unified multimodal architecture.The design follows the paper’s findings on representation, objective, and training strategy.
  • Model Variants: The 8B model freezes the Qwen2.5-VL-7B-Instruct backbone and trains diffusion transformers with 1.4B trainable parameters, while the 4B model uses Qwen2.5-VL-3B-Instruct.The 8B model uses proprietary data; the 4B model uses only open-source data.
  • Model Architecture: The diffusion transformer uses Lumina-Next’s scalable Next-DiT architecture with 3D rotary position embeddings, sandwich normalization, and grouped-query attention.These components encode spatial-temporal structure, improve stability, and reduce computation.
  • Instruction Tuning: Instruction tuning targets complex gestures, common objects, landmarks, and simple text because limited pretraining data did not adequately cover these categories.GPT-4o generates roughly 10k prompt–image pairs per category, alongside additional aesthetic prompts.
  • Benchmark Results: BLIP3-o 8B achieves the best performance in most image-understanding benchmarks and scores 0.84 on GenEval and 0.62 on WISE.The image-generation evaluation also reports a lower DPG-Bench score, motivating a complementary human study.
  • Instruction-Tuning Results: Using only 60k prompt–image pairs, BLIP3o-60k markedly improves prompt alignment and visual aesthetics while reducing many generation artifacts.The dataset does not fully resolve difficult cases such as complex human-gesture generation.
  • Human Study: Human evaluation on about 1,000 DPG-Bench prompts finds BLIP3-o superior to Janus Pro on visual quality and prompt alignment despite Janus Pro’s higher DPG score.The reported p-values are 5.05e-06 for visual quality and 1.16e-05 for prompt alignment.

6 Future Work

Future work extends BLIP3-o toward image editing, multiturn visual dialogue, and interleaved generation, beginning with image reconstruction. The planned reconstruction pipeline feeds images through the understanding encoder and reconstructs them with the generation model.

  • The authors plan to extend the unified model to image editing, multiturn visual dialogue, and interleaved generation.
  • Their first step is image reconstruction, feeding images into the understanding vision encoder and reconstructing them through the image-generation model.They also plan instruction-tuning datasets for downstream applications.

7 Related Work

Related unified multimodal models use regression, discrete autoregressive token prediction, or diffusion objectives for image generation. BLIP3-o differs from LMFusion by using a relatively lightweight image-generation design while also freezing the MLLM backbone.

  • Prior unified models learn visual features through regression losses, discrete token prediction, or diffusion objectives.Regression-based examples include SEED-X, Emu-2, and MetaMorph; discrete-token examples include Chameleon, Show-o, EMU3, and Janus.
  • BLIP3-o and LMFusion both freeze the MLLM backbone and train image-specific components, but LMFusion uses parallel diffusion transformers that significantly expand model size.BLIP3-o instead introduces a relatively lightweight image-generation approach.

8 Conclusion

The paper systematically evaluates representation, objective, and training-strategy choices for unified multimodal modeling, then builds BLIP3-o from the resulting findings. Its models pair CLIP embeddings with flow matching and use a 60k instruction-tuning dataset to improve generation quality and prompt alignment.

  • The study compares CLIP versus VAE representations, Flow Matching versus MSE objectives, and joint versus sequential training strategies.
  • CLIP embeddings paired with flow matching provide faster training and higher-quality outputs.
  • Sequential training first develops image understanding and then trains image generation while preserving the unified model design.
  • BLIP3-o is a family of unified models enhanced with the 60k BLIP3o-60k instruction-tuning dataset.
  • The dataset substantially improves prompt alignment and visual aesthetics, while the authors identify iterative editing, visual dialogue, and step-by-step visual reasoning as applications in progress.

A Prompt used in Figure 2

Figure 2 uses prompts spanning diverse scenes, objects, animals, people, actions, text, and imaginative settings. The examples illustrate the breadth of image-generation prompts used for evaluation or demonstration.

  • Other prompts combine unusual objects or compositions, such as a broccoli-shaped vessel, a glass bottle containing a miniature forest, and a blue BMW before a yellow brick wall.
  • The prompts cover diverse scenes, including ice caverns, frozen Manhattan streets, stormy seas, tropical waterfalls, and alpine lakes.
  • The set tests rendering of text embedded in images, including “Salesforce,” “Transformer,” “Diffusion,” “Deep Learning,” and “BLIP3o.”
  • They include animals and fantastical creatures such as wolves, glowing deer, raccoons, sea turtles, blue jays, and cyberpunk figures.
  • Several prompts specify people, gestures, or activities, including twirling, walking hand in hand, sipping coffee, talking on the phone, and having a picnic.
  • The collection also includes ordinary natural and domestic subjects, such as a golden retriever on a porch, a woman in a wheat field, and a snowy mountain reflection.
Loading 2505.09568v1…