Source-linked AI summary
BLIP3o-NEXT: Next Frontier of Native Image Generation
Jiuhai Chen, Le Xue, Zhiyang Xu, Xichen Pan, Shusheng Yang, Can Qin, An Yan, Honglu Zhou, Zeyuan Chen, Lifu Huang, Tianyi Zhou, Junnan Li, Silvio Savarese, Caiming Xiong, Ran Xu
TL;DR
Native image generation still requires a unified approach to text-to-image synthesis and image editing, while balancing quality, instruction following, and consistency. BLIP3o-NEXT addresses this with a fully open-source Autoregressive + Diffusion model that generates discrete image tokens before diffusion rendering. Across diverse benchmarks, it reports superior performance in both generation and editing, while identifying scalability, reinforcement learning, post-training, and data quality as central factors.
Problem
Native image generation must support both text-to-image synthesis and image editing while image editing remains challenged by consistency between generated and reference images.
Method
BLIP3o-NEXT uses an Autoregressive + Diffusion architecture in which multimodal inputs produce discrete image tokens whose hidden states condition a diffusion model.
Results
BLIP3o-NEXT achieves superior performance across diverse text-to-image generation and image-editing benchmarks.
Takeaways & Limitations
Simple scalable architectures, reinforcement learning, post-training, and high-quality large-scale data are identified as key factors for advancing native image generation.
Takeaways & Limitations
Reward design remains an open challenge because rewards must balance image quality, instruction following, and human preference alignment.
Abstract
from arXiv · showhide
We present BLIP3o-NEXT, a fully open-source foundation model in the BLIP3 series that advances the next frontier of native image generation. BLIP3o-NEXT unifies text-to-image generation and image editing within a single architecture, demonstrating strong image generation and image editing capabilities. In developing the state-of-the-art native image generation model, we identify four key insights: (1) Most architectural choices yield comparable performance; an architecture can be deemed effective provided it scales efficiently and supports fast inference; (2) The successful application of reinforcement learning can further push the frontier of native image generation; (3) Image editing still remains a challenging task, yet instruction following and the consistency between generated and reference images can be significantly enhanced through post-training and data engine; (4) Data quality and scale continue to be decisive factors that determine the upper bound of model performance. Building upon these insights, BLIP3o-NEXT leverages an Autoregressive + Diffusion architecture in which an autoregressive model first generates discrete image tokens conditioned on multimodal inputs, whose hidden states are then used as conditioning signals for a diffusion model to generate high-fidelity images. This architecture integrates the reasoning strength and instruction following of autoregressive models with the fine-detail rendering ability of diffusion models, achieving a new level of coherence and realism. Extensive evaluations of various text-to-image and image-editing benchmarks show that BLIP3o-NEXT achieves superior performance over existing models.
1 Introduction
BLIP3o-NEXT is a fully open-source foundation model unifying text-to-image generation and image editing through an Autoregressive + Diffusion architecture. Its design, training, reinforcement learning, data engineering, and evaluation target coherent, detailed outputs and strong instruction following.
- Architecture: BLIP3o-NEXT combines an autoregressive model that generates discrete image tokens with diffusion-based high-fidelity image synthesis.The autoregressive model conditions on multimodal inputs, while the diffusion model uses the generated tokens’ hidden states as conditioning signals.
- Training: The model is trained for text-to-image generation, input image reconstruction, and image editing, then enhanced through curated data and reinforcement learning.The reinforcement-learning framework leverages discrete image tokens and targets text rendering and instruction following.
- Image Editing: VAE-feature integration improves reference-image consistency and visual fidelity when features condition both autoregressive hidden states and diffusion noise.The reported best strategy concatenates VAE features to both conditioning locations.
- Results: Evaluations across text-to-image and image-editing benchmarks report that BLIP3o-NEXT consistently outperforms existing models.The paper identifies strong performance across diverse benchmarks as a central contribution.
- Open Source: The authors fully release model weights, datasets, training and inference code, and evaluation pipelines to support reproducibility and future research.The release is framed as part of the BLIP3 family’s open-source philosophy.
2 Overview of Architectures for Native Image Generation
Native image-generation systems commonly combine autoregressive models with diffusion models, but differ in how they form diffusion conditioning signals. BLIP3o-NEXT uses discrete visual tokens and argues that simple, scalable, fast-inference architectures are generally effective.
- Existing Architectures: Existing approaches derive diffusion conditions from continuous embeddings, fixed learnable queries, or multimodal transformer interactions.Continuous embeddings and learnable queries can support reasoning-oriented generation but limit representational capacity through fixed-token compression.
- Existing Architectures: LMFusion and BAGEL use two transformer experts with shared multimodal self-attention, but face flexibility, scalability, and inference-latency challenges.Their experts separately process understanding and generation information while exchanging information within transformer blocks.
- BLIP3o-NEXT Architecture: BLIP3o-NEXT autoregressively predicts discrete image tokens from multimodal inputs and uses their hidden states to condition diffusion synthesis.This differs from approaches that generate continuous image embeddings or compress conditioning into fixed query tokens.
- Training Objective: BLIP3o-NEXT optimizes cross-entropy and diffusion losses, using approximately 3B parameters initialized from Qwen3 and SANA1.5.The cross-entropy loss covers text and discrete image tokens, while λ balances it against diffusion loss.
- Architecture Discussion: Most Autoregressive + Diffusion architectures deliver comparable performance, so effective designs should prioritize simplicity, scalability, and fast inference.The paper characterizes design variations as producing only marginal performance differences.
3 Image Generation with Reinforcement Learning
This section examines reinforcement learning for native image generation, contrasting autoregressive and diffusion targets and reporting GRPO experiments on object composition and visual text rendering.
- 3.1 RL for Autoregressive Model: BLIP3o-NEXT applies GRPO to autoregressive trajectories, decodes them with a frozen diffusion model, assigns rewards, and optimizes normalized group advantages.Each sampled trajectory contains 729 discrete image tokens, and the diffusion model remains frozen during policy optimization.
- 3.2 RL for Diffusion Model: Diffusion-model RL is an alternative exemplified by Qwen-Image and Flow-GRPO, which uses stochastic differential equations to support exploration during trajectory sampling.Flow-GRPO addresses the lack of stochasticity in deterministic sampling by injecting randomness into the process.
- 3.1 RL for Autoregressive Model: GRPO training increases rewards for multiple-object composition and visual text rendering, with qualitative improvements reported for both tasks.GenEval evaluates object composition, while PaddleOCR evaluates visual text rendering.
- 3.1 RL for Autoregressive Model: The autoregressive model is the natural RL target in BLIP3o-NEXT because it produces image tokens and plays a central role in image generation.The diffusion model primarily functions as an image decoder in this framework.
- 3.5 Discussion: Reward design remains the central challenge because rewards must balance image quality, instruction following, and human preference alignment.The experiments focus on verifiable rewards for object composition and visual text rendering rather than model-based rewards.
4 Image Editing
BLIP3o-NEXT addresses image-editing consistency by combining reconstruction training with low-level VAE conditioning and task-oriented post-training strategies. The model’s VAE integration experiments identify combined cross-attention and noise-space injection as the most consistent approach, while benchmark results show its 3B model remains competitive with some existing systems.
- 4.1 Improving Consistency in Image Editing: Image reconstruction trains the model to preserve reference-image details when prompted to keep the image unchanged.This objective aligns the generative process with the conditioning image.
- 4.1 Improving Consistency in Image Editing: BLIP3o-NEXT’s 3B model lags behind GPT-Image and Qwen-Image on ImgEdit but achieves comparable performance to BAGEL and OmniGen2.Table 2 reports GPT-4.1-evaluated metrics, with Overall averaging task categories.
- 4.1 Improving Consistency in Image Editing: Combining VAE features as cross-attention inputs and noise-space injections yields the best visual consistency with reference images.The two methods respectively condition DiT cross-attention and augment the diffusion noise input with reference-image VAE features.
- 4.1 Improving Consistency in Image Editing: The editing setup jointly trains reconstruction and editing objectives on an approximately 10-million-sample corpus assembled from multiple open-source datasets.Selected subsets are repeated to enlarge the dataset ensemble and stabilize training.
- 4.2 Future Exploration: Image editing consistency remains challenging even with VAE features, motivating improved data engineering and reinforcement learning tailored to editing.The paper also explores task-specific system prompts distinguishing inpainting from subject-driven generation and prompt rewriting for instruction following.
5 Training Recipe and Evaluation
The paper emphasizes data quality as a decisive factor in model performance and identifies a need for specialized image-editing benchmarks. The supplied evaluation material also includes qualitative comparisons of editing with and without VAE latent conditioning.
- 5 Training Recipe and Evaluation: Data quality remains a decisive factor in determining overall model performance.
- 5 Training Recipe and Evaluation: Specialized image-editing benchmarks are needed to assess instruction following and consistency between generated images and reference inputs.
- 5 Training Recipe and Evaluation: Figure 6 qualitatively compares image-editing results from models with and without VAE latent conditioning.
6 Conclusion
BLIP3o-NEXT unifies text-to-image generation and image editing in a fully open-source architecture. Its findings emphasize scalable design, reinforcement learning, post-training, and high-quality data as drivers of progress toward controllable, high-quality native image generation.
- BLIP3o-NEXT achieves superior performance in both generation and editing tasks across diverse benchmarks.These results support the effectiveness of the Autoregressive + Diffusion paradigm.
- The model combines autoregressive instruction following and reasoning with diffusion-based fine-grained rendering to produce coherent, high-fidelity visual outputs.
- The study identifies scalable architectural simplicity, reinforcement learning, post-training, and high-quality large-scale data as central factors in advancing native image generation.
- The authors view unified architectures, reinforcement learning, and scalable post-training as promising directions for controllable and instruction-aligned image generation systems.