PaperlayerView exact revision on arXiv

Source-linked AI summary

Emerging Properties in Unified Multimodal Pretraining

Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, Haoqi Fan

arXiv:2505.14683v3cs.CV

TL;DR

Unified multimodal models need richer evidence than paired image-text data to support complex reasoning across images, text, and other modalities. BAGEL addresses this gap with an open-source decoder-only model trained on large-scale interleaved multimodal data and shared multimodal attention. It reports top-tier benchmark performance alongside emerging capabilities in complex visual reasoning and manipulation.

  • Problem

    Paired vision-text data falls short for complex in-context reasoning involving multiple images and intermediate text, while academic unified models lag proprietary systems.

  • Method

    BAGEL is a unified decoder-only multimodal model trained on large-scale interleaved data, using shared self-attention and diffusion forcing for interleaved multi-image generation.

  • Results

    BAGEL outperforms top-tier open-source VLMs on standard understanding benchmarks, delivers competitive text-to-image quality, and exhibits emerging complex multimodal reasoning abilities.

  • Takeaways & Limitations

    Scaling interleaved multimodal pretraining is associated with a progression from basic skills to complex editing, free-form manipulation, and long-context compositional reasoning.

  • Takeaways & Limitations

    Special-IP generation, complex text rendering, intricate human poses, multiple-instance generation, and large-scale simultaneous edits remain challenging; GPT-4o is more consistently successful in the reported examples.

Abstract

from arXiv · show

Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. In this work, we introduce BAGEL, an open-source foundational model that natively supports multimodal understanding and generation. BAGEL is a unified, decoder-only model pretrained on trillions of tokens curated from large-scale interleaved text, image, video, and web data. When scaled with such diverse multimodal interleaved data, BAGEL exhibits emerging capabilities in complex multimodal reasoning. As a result, it significantly outperforms open-source unified models in both multimodal generation and understanding across standard benchmarks, while exhibiting advanced multimodal reasoning abilities such as free-form image manipulation, future frame prediction, 3D manipulation, and world navigation. In the hope of facilitating further opportunities for multimodal research, we share the key findings, pretraining details, data creation protocal, and release our code and checkpoints to the community. The project page is at https://bagel-ai.org/

1 Introduction

BAGEL addresses the gap between academic and proprietary unified multimodal systems by scaling a unified model with carefully structured interleaved data. It combines shared multimodal context with modality-specific capacity to produce emerging reasoning and manipulation abilities.

  • Motivation: Academic unified models remain behind proprietary systems because many are trained predominantly on image-text pairs rather than richly structured interleaved data.The paper identifies scaling carefully structured multimodal interleaved data as the route it argues can help close this gap.
  • Architecture: The model uses a Mixture-of-Transformer-Experts architecture to provide modality-specific parameters while avoiding bottleneck connectors between understanding and generation.This design enables long-context interaction through shared multimodal processing.
  • Contribution: BAGEL is an open-source multimodal foundation model with 7B active parameters and 14B total parameters, trained on large-scale interleaved multimodal data.Its training combines multimodal data intended to support understanding and generation in one model.
  • Results: BAGEL outperforms top-tier open-source VLMs on standard understanding benchmarks and reaches competitive text-to-image quality, while showing stronger qualitative image editing and world-modeling capabilities.The reported extended capabilities include free-form visual manipulation, multiview synthesis, and world navigation.
  • Emerging properties: Scaling interleaved pretraining yields a progression from basic understanding and high-fidelity generation to complex editing, free-form manipulation, and long-context multimodal reasoning.The paper describes these abilities as emerging through synergy among previously independent atomic skills.

2 Model

BAGEL integrates understanding and generation in a decoder-only transformer while assigning separate experts and visual encoders to their distinct information needs. Its generalized causal attention supports interleaved multimodal generation, and controlled ablations favor the full Mixture-of-Transformers design.

  • Model design: BAGEL uses two transformer experts—one for multimodal understanding and one for generation—that operate on the same token sequence through shared self-attention.Text generation follows next-token prediction within this shared multimodal context.
  • Visual representation: The model uses separate visual encoders: a ViT for semantic image understanding and a VAE for image-generation latents.The ViT processes raw pixels into understanding tokens, while the VAE maps images between pixel and latent spaces.
  • Token conditioning: Interleaved generation lets later image or text tokens attend to clean VAE and ViT tokens from preceding images, excluding their noised VAE counterparts.This attention pattern distinguishes conditioning information from diffusion-training noise.
  • Generalized causal attention: Diffusion forcing assigns independent noise levels to different images and conditions each image on noisy representations of preceding images; consecutive images may also share full attention groups.The grouping strategy uses a common noise level within each group to enhance generation consistency.
  • Transformer design: In a controlled 1.5B experiment, the MoT variant consistently outperforms dense and MoE designs, converging fastest and reaching the lowest final generation MSE loss.Understanding CE loss fluctuates more, but MoT generally remains best; the experiment keeps hyperparameters and data configurations identical.

3 Data

BAGEL’s training corpus combines language, image, video, and web data, with new interleaved datasets designed to support sequential multimodal reasoning. Video supplies temporal dynamics, while web documents provide richly grounded multimodal structure through filtering, captioning, and rewriting.

  • Training corpus: The corpus spans language, image, video, and web data to support multimodal reasoning, in-context prediction, physical dynamics modeling, and future frame prediction.The paper adds new vision-text interleaved datasets from web and video sources for sequential multimodal reasoning.
  • Video data: Video data captures temporal and spatial dynamics, identity consistency, fine-grained visual detail, and complex motion useful for editing, navigation, and 3D manipulation.The video source is described as a natural simulator of real-world dynamics.
  • Web data: Web data includes illustrated encyclopedic articles, visual tutorials, and other interleaved documents that provide grounded supervision for multimodal reasoning.The web corpus builds on OmniCorpus and targets documents with strong text-image semantic alignment.
  • Data filtering: Video preprocessing uses temporal splitting, spatial cropping, quality filtering, and deduplication before constructing coherent training clips.Filters address borders, overlays, length, resolution, clarity, and motion stability.
  • Data construction: The web pipeline applies two-stage topic selection, quality filtering, per-image caption insertion, and summarization of long text segments to produce 20 million structured interleaved documents.Captions are inserted before images to provide localized conceptual scaffolds for image generation.
  • Data construction: The video pipeline generates inter-frame captions with a lightweight VLM distilled from high-quality examples, providing textual supervision for object motion and scene changes.These captions describe visual changes between consecutive frames while targeting scalable inference and reduced hallucination.

4 Training

BAGEL uses staged training over large-scale multimodal data, with interleaved data and objective balancing emphasized during continued training. Controlled studies guide the sampling ratio and learning-rate choices for jointly optimizing generation and understanding.

  • Training stages: BAGEL trains through Alignment, Pre-training, Continued Training, and Supervised Fine-tuning stages using dynamically mixed curated data.The recipe initializes the connector, performs large-scale pretraining, increases resolution and interleaved-data exposure, then applies high-quality supervised tuning.
  • Training stages: 2.5T tokens comprise the Pre-training corpus, spanning text, image–text pairs, multimodal conversations, web-interleaved data, and video-interleaved data.Native-resolution processing is used for multimodal understanding and generation, subject to image-size restrictions.
  • Training stages: Approximately 2.6T tokens are used in Continued Training, where higher visual resolution and a larger interleaved-data ratio emphasize cross-modal reasoning.The increased resolution is described as important for multimodal generation and understanding performance.
  • Hyper-parameter studies: Increasing generation-data sampling from 50% to 80% reduces MSE loss by 0.4% absolute, while CE loss shows no consistent sampling-ratio pattern.The study uses a 1.5B Qwen2.5 LLM and compares the 1g1u and 4g1u ratios.
  • Hyper-parameter studies: A larger learning rate accelerates MSE convergence, whereas a smaller learning rate benefits CE loss; separate objective weights reconcile this trade-off.The finding motivates the weighting factors listed in the training recipe.

5 Evaluation

BAGEL’s evaluation covers established multimodal understanding, text-to-image, and classical editing benchmarks, alongside IntelligentBench for free-form image manipulation requiring multimodal reasoning and task composition.

  • Multimodal understanding: Six understanding benchmarks span perception, cognition, and multimodal reasoning: MME, MMBench, MMVet, MMMU, MathVista, and MMVP.These benchmarks are selected as a compact testbed with discriminative power for ranking state-of-the-art models.
  • Text-to-image generation: Text-to-image evaluation uses GenEval and WISE, supplemented by qualitative comparisons with state-of-the-art models.WISE assesses complex semantic understanding and world-knowledge integration in text-to-image generation.
  • Image editing: GEdit-Bench evaluates practical image editing using authentic web-scraped user requests, automatic GPT-4.1 scoring, and qualitative examples.The benchmark contains diverse editing tasks intended to mirror real-world editing needs.
  • Intelligent image editing: IntelligentBench contains 350 examples pairing a question image and text with a reference answer image, and scores generated images on a normalized 0–100 scale.GPT-4o evaluates request fulfillment, visual consistency, and knowledge-grounded creativity from the complete quadruplet.

6 Emerging Properties

BAGEL’s scaling study defines emergence by later appearance of abilities and tracks historical checkpoints across understanding, generation, editing, and intelligent editing. Complex editing develops substantially later than basic capabilities and can show abrupt qualitative changes.

  • Definition and measurement: An ability is considered emerging when it is absent in earlier training stages but present in later pretraining.The study examines emergence within unified multimodal foundational models.
  • Definition and measurement: Historical-checkpoint evaluation uses average VLM performance, GenEval, GEdit, and IntelligentBench to track understanding, generation, naive editing, and complex reasoning.The authors note that phase-transition-like behavior cannot be predicted by extrapolating training loss curves.
  • Scaling behavior: Understanding and generation reach 85% of peak performance at approximately 0.18T and 0.68T tokens, while editing requires 2.64T tokens.Different tasks therefore exhibit distinct learning dynamics and saturation behavior.
  • Scaling behavior: Intelligent Editing reaches 85% only after 3.61T tokens and improves from 15 to 45 in later stages, unlike traditional editing’s limited response to resolution increases.Removing ViT tokens causes a 16% drop in Intelligent Edit but has minimal impact on GEdit-Bench.
  • Qualitative emergence: Before 3.5T tokens, Intelligent Editing often reproduces the input with minimal changes; after 3.5T tokens, qualitative behavior shifts toward the required new concept.Text rendering of “hello” and “BAGEL” emerges around 1.5T to 4.5T tokens.

7 Main Results

BAGEL is evaluated against open-source unified and specialized multimodal models across understanding, generation, and editing tasks. It reports strong benchmark performance, including higher understanding scores, competitive text-to-image quality, and support for Chinese or English prompts at arbitrary aspect ratios.

  • Evaluation overview: The evaluation spans established understanding, generation, and editing benchmarks plus IntelligentBench for complex multimodal reasoning.The experiments are organized around basic abilities, image editing, and reasoning-intensive generation and editing.
  • 7.1 Image Understanding: At 7B activated parameters, BAGEL improves over Janus-Pro by 14.3 points on MMMU and 17.1 points on MM-Vet.It also outperforms existing unified models and performs better on most benchmarks than specialized Qwen2.5-VL and InternVL2.5.
  • 7.2 Image Generation: 88% overall on GenEval places BAGEL above FLUX-1-dev at 82%, SD3-Medium at 74%, Janus-Pro at 80%, and MetaQuery-XL at 80%.Without an LLM rewriter, BAGEL reaches 82%, surpassing Janus-Pro-7B under the reported settings.
  • 7.2 Image Generation: On WISE, BAGEL exceeds all prior models except GPT-4o, indicating strong reasoning ability with world knowledge.WISE evaluates complex semantic understanding and world knowledge for text-to-image generation.
  • 7.2 Image Generation: Qualitatively, BAGEL generates higher-quality images than Janus-Pro 7B and SD3-medium while supporting Chinese and English prompts and arbitrary aspect ratios.The comparison includes BAGEL, Janus-Pro, SD3-medium, and GPT-4o.

7.3 Image Editing

BAGEL performs strongly on classical and reasoning-intensive image editing, with chain-of-thought substantially improving both generation and editing outcomes. Qualitative comparisons further show stronger instruction adherence than several competing models.

  • BAGEL achieves results competitive with Step1X-Edit on GEdit-Bench and outperforms Gemini 2.0.
  • 44.9 on IntelligentBench significantly surpasses Step1X-Edit by 30 points in complex image-editing reasoning.
  • BAGEL consistently outperforms Step1X-Edit and IC-Edit qualitatively, exceeds Gemini 2.0, and avoids GPT-4o’s unintended source-image modifications.
  • On WISE, chain-of-thought generation scores 0.70, exceeding non-CoT BAGEL by 0.18 and MetaQuery-XL by 0.15.
  • Adding thinking raises Intelligent Score from 44.9 to 55.3, with gains also reported on RISEBench and KRIS-Bench.

7.5 World Modeling

BAGEL is fine-tuned with more video and navigation data to improve long-sequence visual world modeling. It generates dynamic image sequences for navigation, rotation, and multi-frame tasks while generalizing beyond real-world street scenes.

  • Increasing video and navigation data improves BAGEL’s world modeling for long-sequence visual generation.
  • BAGEL generates dynamic numbers of images for navigation and rotation, and produces multiple images from prompts.
  • BAGEL generalizes world understanding from real-world street navigation to ink paintings, cartoons, and video games.

7.6 More Qualitative Results

Qualitative results indicate that BAGEL-1.5B can outperform larger comparison models, while scaling improves performance further. Persistent failures remain in complex generation, editing, and instruction-following scenarios.

  • BAGEL-1.5B surpasses larger JanusPro-7B and Step1X-Edit models qualitatively on text-to-image and image-editing tasks.
  • The qualitative gap between BAGEL-1.5B and BAGEL-7B indicates gains from model scaling.
  • Special-IP generation, complex text rendering, intricate human poses, and multiple-instance generation remain challenging for text-to-image systems.
  • Object-position swaps and simultaneous modification of many instances challenge most existing image-editing models.
  • In some complex scenarios, BAGEL and Gemini 2.0 similarly struggle with precise instruction adherence, whereas GPT-4o is most consistently successful.

8 Conclusion

The paper presents BAGEL as a unified multimodal model whose scaled pretraining yields strong benchmark performance and emerging reasoning and world-modeling capabilities. The authors release it openly to support further multimodal research.

  • BAGEL exhibits emerging capabilities as unified multimodal pretraining is scaled.
  • BAGEL achieves top-tier performance on standard multimodal understanding and generation benchmarks while showing world modeling and reasoning abilities.
  • The authors open-source BAGEL for the research community.

Appendix

The appendix documents BAGEL’s causal attention setup, scaling behavior, evaluation materials, and observed failure cases across multimodal tasks. It also includes benchmark comparisons and prompts for generating reasoning traces for image-editing tasks.

  • Causal masking: During interleaved image-text generation, each image attends exclusively to clean VAE and ViT tokens from preceding images, while attention is full within its own features.The causal-mask description defines VAE and ViT as feature types and identifies t=0 as the noise-free timestep.
  • Benchmark evaluations: The appendix reports comparisons on RISEBench and KRIS-Bench, with RISEBench results evaluated by GPT-4.1 and KRIS-Bench categories including perception, prediction, science, and reasoning.KRIS-Bench abbreviations include Attribute Perception, Spatial Perception, Temporal Prediction, Social Science, Natural Science, and Logical Reasoning.
  • Model scaling: Larger models demonstrate better prompt adherence and produce higher-quality images.
  • Failure cases: BAGEL and other models face challenges with certain IP, complicated text, counterfactual scenes, object swapping, and deblurring, whereas GPT-4o is more consistently successful.
  • Reasoning-trace data and evaluation: Tables provide prompts for generating reasoning traces from edit data, video-interleaved data, and conceptual-editing tasks, plus an IntelligentBench evaluation example clarifying that the human-answered image label refers to GPT-4o’s ground-truth reference.
Loading 2505.14683v3…