PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model

Zebin You, Xiaolu Zhang, Jun Zhou, Chongxuan Li, Ji-Rong Wen

arXiv:2603.01068v1cs.CVcs.LG

TL;DR

Multimodal diffusion must reconcile discrete text tokens with continuous visual latents, whose differing dynamics make naive joint training difficult. LLaDA-o uses Mixture of Diffusion with specialized experts and a shared attention backbone, plus adaptive-length training, and reports strong multimodal results including 87.04 on DPG-Bench.

  • Problem

    Text and images favor different diffusion dynamics, while naive joint training can create objective mismatch and gradient interference.

  • Method

    LLaDA-o decouples masked diffusion for text and visual encoder tokens from continuous diffusion for visual latent tokens, while sharing an attention backbone and using adaptive-length training.

  • Results

    LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and reaches 87.04 on DPG-Bench for text-to-image generation.

  • Takeaways & Limitations

    The results support LLaDA-o as a unified omni diffusion model for multimodal understanding and generation.

Abstract

from arXiv · show

We present \textbf{LLaDA-o}, an effective and length-adaptive omni diffusion model for multimodal understanding and generation. LLaDA-o is built on a Mixture of Diffusion (MoD) framework that decouples discrete masked diffusion for text understanding and continuous diffusion for visual generation, while coupling them through a shared, simple, and efficient attention backbone that reduces redundant computation for fixed conditions. Building on MoD, we further introduce a data-centric length adaptation strategy that enables flexible-length decoding in multimodal settings without architectural changes. Extensive experiments show that LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and generation benchmarks, and reaches 87.04 on DPG-Bench for text-to-image generation, supporting the effectiveness of unified omni diffusion modeling. Code is available at https://github.com/ML-GSAI/LLaDA-o.

1. Introduction

LLaDA-o addresses the difficulty of unifying multimodal understanding and generation by combining modality-specific diffusion processes within a shared framework. It reports strong understanding and generation performance, including 87.04 on DPG-Bench.

  • Diffusion language models use masked-token denoising to provide bidirectional context modeling and improved inference parallelism.
  • Text and images favor different diffusion dynamics, with masked diffusion suited to discrete language tokens and continuous diffusion standard for visual latent spaces.
  • Naively co-training both modalities can cause objective mismatch, gradient interference, unstable optimization, and suboptimal performance, while fixed-length understanding limits open-ended use.
  • LLaDA-o’s Mixture of Diffusion framework assigns masked diffusion to understanding and continuous diffusion to visual generation while sharing an attention backbone.
  • LLaDA-o is qualitatively described as achieving stronger multimodal understanding and producing images with richer fine-grained details than existing omni-diffusion models.
  • 87.04 on DPG-Bench is reported as state-of-the-art performance for text-to-image generation among the evaluated models.

2. Preliminaries

Diffusion models define forward corruption and learn reverse processes for generation. Continuous diffusion operates on visual data through learned flows, while discrete and masked diffusion progressively corrupt and reconstruct token sequences.

  • Continuous diffusion models corrupt data toward noise and learn reverse dynamics to generate samples from the noise distribution.
  • Rectified Flow connects noise and data distributions with a deterministic linear path and learns a velocity field matching its constant flow direction.
  • Sampling in Rectified Flow solves an ODE initialized from the noise distribution and numerically integrates the learned velocity field.
  • Discrete diffusion models progressively corrupt discrete states toward a simple prior and learn reverse dynamics to generate discrete data.
  • Masked diffusion independently leaves tokens unchanged or replaces them with a mask according to α_t, then iteratively reconstructs masked tokens during the reverse process.
  • Diffusion large language models demonstrate masked diffusion at scale while enabling parallel decoding and flexible generation control.

3. Method

LLaDA-o unifies multimodal understanding and generation by assigning modality-appropriate diffusion experts while sharing an attention backbone. It further improves inference efficiency and supports flexible-length decoding through attention reuse and adaptive length augmentation.

  • Mixture of Diffusion: Mixture of Diffusion uses masked diffusion for text and visual encoder tokens, while continuous diffusion handles visual latent tokens.The specialized experts address heterogeneous state spaces and corruption processes that can cause objective mismatch and gradient interference in dense co-training.
  • Mixture of Diffusion: The understanding expert combines a vision encoder, two-layer MLP, and diffusion language model to jointly process image and prompt tokens.The image is encoded into semantic visual tokens and projected into the language-token space before joint processing.
  • Mixture of Diffusion: The generation expert uses a VAE and diffusion Transformer, with the VAE mapping between images and visual latent tokens while remaining frozen during training.Input images and text in generation tasks are also processed by the understanding expert, whose parameters are jointly trained.
  • Intra-Modality Bidirectional Attention: Intra-modality bidirectional attention applies full attention within modality blocks and causal attention across blocks.Conditional blocks form a fixed prefix whose KV cache can be reused across denoising steps, avoiding redundant computation on long sequences.
  • Intra-Modality Bidirectional Attention: 5.9 times speedup is achieved compared with a representative baseline using globally bidirectional attention.The comparison is reported as a practical result in Fig. 4.
  • Adaptive Length: Adaptive length augmentation trains flexible decoding by appending EOS tokens or truncating responses, while block-wise inference extends generation when EOS is absent.The strategy preserves sample isolation, reuses the cached fixed prefix, and requires no architectural changes.

4. Experiments

LLaDA-o is evaluated across multimodal understanding, text-to-image generation, inference efficiency, variable-length decoding, and training-stage effects. It achieves strong benchmark performance, efficient MathVista inference, stable adaptive-length behavior, and improved generation across training stages.

  • Experimental Settings: LLaDA-o uses separate understanding and generation experts, initialized with LLaDA-8B-Instruct and a FLUX VAE-based visual encoder.The understanding expert uses SigLIP with a two-layer MLP projector; the generation expert uses a diffusion Transformer architecture initialized from LLaDA.
  • Experimental Settings: The three-stage training strategy scales data difficulty and image resolution, then activates adaptive-length augmentation with additional high-quality generation data.Generation resolution increases from up to 512 in Stage 1 to 1024 in Stage 2; Stage 3 jointly enables variable-length text decoding and stronger visual generation.
  • Benchmark Results: LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding benchmarks, with particular gains in mathematical reasoning and chart/document understanding.On MathVista, it scores 66.1 versus 59.7 for LLaDA-V and 73.1 for BAGEL.
  • Benchmark Results: LLaDA-o performs strongly on GenEval and achieves state-of-the-art DPG-Bench performance of 87.04 for text-to-image generation.It is particularly better on two-object and color-binding generation, while slightly trailing Lumina-DiMOO and Mogao overall on GenEval.
  • Further Analysis: A confidence threshold of 0.9 provides the optimal inference trade-off, while LLaDA-o achieves a 5.9× speedup over LLaDA-V with comparable MathVista performance.The threshold regulates the accuracy-throughput trade-off by selecting high-confidence tokens or accelerating generation.
  • Further Analysis: Adaptive length augmentation keeps output lengths relatively stable across block sizes while improving MathVista accuracy from 63.6% to 66.2%.Increasing block length from 32 to 96 reduces average generated tokens from 165 to 145, indicating that output length is mainly driven by input content.

5. Conclusion

LLaDA-o is a length-adaptive omni diffusion model for multimodal understanding and generation. Its framework combines a shared efficient attention backbone with adaptive-length training to support stable multimodal training and flexible-length generation.

  • LLaDA-o targets multimodal understanding and generation as a length-adaptive omni diffusion model.
  • Its Mixture of Diffusion framework uses a shared efficient attention backbone and a data-centric adaptive length training strategy.
  • LLaDA-o demonstrates strong performance on multimodal understanding and text-to-image generation tasks.
  • The paper positions LLaDA-o as a promising foundation for future omni diffusion approaches as masked diffusion language modeling advances.

B.1. Training Stages and Data Setup

LLaDA-o is trained through preliminary projector alignment followed by three progressive stages. The stages progressively establish baseline capabilities, add high-resolution reasoning, and activate variable-length generation with high-quality data.

  • Projector Alignment: Training begins with projector alignment, training an MLP projector while freezing the remaining components.
  • Stage 1: Foundation Setup: Stage 1 establishes baseline image understanding and generation capabilities using large-scale multimodal data, with generation restricted to 512 resolution.
  • Stage 2: High-Resolution and Reasoning: Stage 2 adds multimodal reasoning data and increases image-generation resolution to 1024 while keeping adaptive length augmentation disabled.
  • Stage 3: Variable-Length Refinement: Stage 3 activates variable-length generation through adaptive length augmentation and fine-tunes the model with high-quality understanding and generation data.

B.2. Multi-turn Dialogue Data and Interleaved Multimodal Data Setup

The data setup distinguishes dialogue-response supervision from interleaved multimodal generation supervision, while inference uses blockwise iterative denoising. Training resources are distributed across progressive stages, with the main cost concentrated in Stages 1 and 2.

  • Multi-turn Dialogue Data: LLaDA-o computes loss on every dialogue-turn response rather than randomly selecting one turn.
  • Interleaved Multimodal Data: For interleaved multimodal data, text serves only as an image-generation condition, so continuous diffusion loss is applied only to image tokens.
  • Qualitative Samples: Generated samples are organized in Fig. 1 by Image IDs, with IDs 1–5 in the top row and 6–10 in the bottom row.
  • Computational Resources: The primary training phases use 256 NVIDIA H800 GPUs, while final refinement uses 64 NVIDIA A100 GPUs.
  • Inference: Blockwise inference caches the fixed image-and-prompt prefix, appends masked blocks, and iteratively accepts tokens whose confidence exceeds threshold τ.

C. Additional Results

Additional qualitative examples show that LLaDA-o generates high-quality images that remain semantically aligned with user prompts while exhibiting fidelity and diversity.

  • Additional samples demonstrate high-quality image generation with semantic alignment, high fidelity, and diversity.

C.2. Qualitative Comparison with LLaDA-V under Mismatched Block Lengths

LLaDA-o adapts its response length to the user’s semantic request even when the predefined block length is mismatched. Compared with LLaDA-V, it avoids redundant output for simple extraction and overly brief output for detailed descriptions.

  • The qualitative comparison evaluates both models when semantic requirements conflict with predefined generation block lengths.
  • With L = 64 for simple text extraction, LLaDA-o accurately extracts the text and terminates correctly, unlike LLaDA-V’s redundant continuation.The comparison tests a long block length against a short semantic requirement.
  • With L = 16 for detailed image description, LLaDA-o dynamically adapts its output length, whereas LLaDA-V is constrained by the fixed window.

C.3. Effect of Block Length on Variable-Length Generation

LLaDA-o maintains largely stable output lengths across different predefined block lengths for the same image and prompt. These qualitative samples indicate that it dynamically adapts response length to the prompt and image rather than rigidly following the block length.

  • LLaDA-o dynamically adapts effective output length based on the user prompt and image content.
  • For a simple extraction request with L = 64, LLaDA-o produces the requested text instead of filling the window with redundant content.
  • For a detailed image-description request with L = 16, LLaDA-o adapts beyond the short block-length constraint rather than producing an overly brief response.
  • Across L ∈{16, 32, 64, 128}, LLaDA-o’s output length remains largely stable for the same image and prompt.The evaluation varies only the predefined generation block length while holding the image and prompt constant.
  • The qualitative examples include generated descriptions of mountain landscapes and other multimodal prompts under varied block-length settings.
Loading 2603.01068v1…