Source-linked AI summary
LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
Inclusion AI, Tiwei Bie, Haoxing Chen, Tieyuan Chen, Zhenglin Cheng, Long Cui, Kai Gan, Zhicheng Huang, Zhenzhong Lan, Haoquan Li, Jianguo Li, Tao Lin, Qi Qin, Hongjun Wang, Xiaomei Wang, Haoyuan Wu, Yi Xin, Junbo Zhao
TL;DR
LLaDA2.0-Uni addresses the challenge of combining multimodal understanding and generation without separate or mismatched visual modules. It unifies semantic visual tokens, an MoE diffusion language model, and a diffusion decoder, achieving strong benchmark performance while supporting interleaved generation and reasoning. The main scope boundaries are fine-grained visual detail preservation and the scaling needed for more complex interleaved capabilities.
Problem
Existing unified multimodal systems rely largely on autoregressive or decoupled designs, motivating a single diffusion framework for understanding and generation.
Method
The model combines SigLIP-VQ semantic tokens, a 16B MoE dLLM backbone with shared masked diffusion, and a diffusion decoder for image reconstruction.
Results
LLaDA2.0-Uni achieves strong performance across multimodal understanding, image generation, and editing benchmarks while naturally supporting interleaved generation and chain-of-thought reasoning.
Takeaways & Limitations
The unified discrete representation provides a flexible framework for jointly exploring multimodal understanding, generation, interleaving, and reasoning.
Takeaways & Limitations
SigLIP-VQ struggles to preserve fine-grained image details, and more training data and model capacity are needed for complex interleaved capabilities.
Abstract
from arXiv · showhide
We present LLaDA2.0-Uni, a unified discrete diffusion large language model (dLLM) that supports multimodal understanding and generation within a natively integrated framework. Its architecture combines a fully semantic discrete tokenizer, a MoE-based dLLM backbone, and a diffusion decoder. By discretizing continuous visual inputs via SigLIP-VQ, the model enables block-level masked diffusion for both text and vision inputs within the backbone, while the decoder reconstructs visual tokens into high-fidelity images. Inference efficiency is enhanced beyond parallel decoding through prefix-aware optimizations in the backbone and few-step distillation in the decoder. Supported by carefully curated large-scale data and a tailored multi-stage training pipeline, LLaDA2.0-Uni matches specialized VLMs in multimodal understanding while delivering strong performance in image generation and editing. Its native support for interleaved generation and reasoning establishes a promising and scalable paradigm for next-generation unified foundation models. Codes and models are available at https://github.com/inclusionAI/LLaDA2.0-Uni.
1 Introduction
LLaDA2.0-Uni unifies multimodal understanding and generation in a discrete diffusion framework, addressing limitations of autoregressive and decoupled designs. Its semantic tokenizer, MoE backbone, and diffusion decoder support strong multimodal performance, efficient inference, and interleaved generation and reasoning.
- Motivation: LLaDA2.0-Uni supports multimodal understanding and generation within one framework, unlike prior approaches that separate specialized models or combine incompatible paradigms.Masked diffusion offers parallel decoding and bidirectional context modeling, while a unified objective simplifies training.
- Unified Architecture: LLaDA2.0-Uni combines a semantic tokenizer, 16B MoE dLLM backbone, and diffusion decoder under a shared block-wise masked diffusion objective.The tokenizer discretizes visual inputs, while the decoder reconstructs generated visual tokens into images.
- Interleaved Capabilities: The unified discrete representation inherently supports interleaved generation and reasoning.This capability extends beyond separate understanding and generation tasks.
- Efficient Inference: The model further accelerates inference through dLLM decoding optimization and few-step diffusion-decoder distillation.The decoder synthesizes high-fidelity images in 8 inference steps.
- Results: LLaDA2.0-Uni achieves strong performance across multimodal understanding, image generation, and editing benchmarks, with performance on par with state-of-the-art unified models.It is competitive with specialized VLMs on visual question answering and document reasoning.
2 Model Design
The model uses semantic visual tokens, a modality-agnostic MoE diffusion language model, and a specialized diffusion decoder. Training-free SPRINT inference reduces denoising cost through modality-aware prefix retention and confidence-adaptive unmasking.
- Core Architecture: LLaDA2.0-Uni contains a SigLIP-VQ tokenizer, a 16B MoE diffusion language model, and a diffusion decoder for unified understanding and generation.The components support end-to-end processing of text and visual tokens within one framework.
- Tokenizer: SigLIP-VQ converts images into semantic discrete tokens and is trained on understanding tasks rather than pixel-level reconstruction.This representation is designed to preserve semantic information for multimodal understanding.
- Backbone: Block-wise attention balances quality and efficiency for diffusion-language-model training, while size tokens encode spatial information with 1D RoPE and support arbitrary resolutions.The design avoids unconstrained full attention and architectural changes for different image sizes.
- Diffusion Decoder: The diffusion decoder maps semantic tokens back to image space, using generated image tokens as conditioning instead of conventional text prompts.A specialized decoder is required because semantic VQ does not directly support pixel decoding.
- Inference Acceleration: SPRINT accelerates block-wise diffusion by pruning prefix KV caches and replacing fixed denoising schedules with confidence-adaptive unmasking.Prefix positions are scored using importance and confidence, with modality-specific keep ratios.
- Inference Acceleration: 1.6× speedup is achieved with negligible quality loss through sparse prefix retention and non-uniform token unmasking.The method reduces computation by retaining a shorter effective prefix and accepting sufficiently confident predictions earlier.
3 Data Preparation
LLaDA2.0-Uni is trained on large, filtered multimodal datasets spanning understanding, generation, editing, interleaved video-text data, and reasoning. Automated quality-control pipelines refine annotations, instructions, and visual samples.
- Pretraining Data: Pretraining uses extensive image-captioning data supplemented with OCR, grounding, and counting categories.OCR data combines PaddleOCR pseudo-labels with Qwen3-VL refinements for document understanding.
- Supervised Fine-Tuning Data: The SFT dataset contains approximately 60 million samples with a 1:5 ratio of text-only to multimodal data.It covers dialogues, multi-image scenarios, VQA, chart and table question answering, and mathematical reasoning.
- Image Generation Data: Generation training retains 140 million high-quality web images after metadata, aesthetics, and quality filtering.The source collection exceeds 200 million web images and emphasizes human-body and rendered-text content.
- Image Generation Data: Image captions incorporate informative original web text to produce richer and more accurate descriptions.Qwen3-VL evaluates whether source descriptions contain useful real-world information.
- Image Editing Data: Editing data combines open-source datasets with synthesized image-editing pairs, followed by filtering and instruction refinement.Quality control removes unchanged or artifact-prone edits and rewrites inaccurate instructions based on visual changes.
- Interleaved Data: Interleaved image-text data is built from filtered video clips and detailed frame-sequence captions with tailored user instructions.Filtering constrains duration, aesthetic and clarity quality, and motion.
- Reasoning Data: Approximately 8M reasoning samples support chain-of-thought image generation and multi-step reasoning across interleaved image-text sequences.The data comes from Flux-6M, Zebra-CoT, and Weave.
4 Model Training
LLaDA2.0-Uni uses a staged multimodal training pipeline combining block-level diffusion objectives, load balancing, complementary masking, decoder refinement, and efficiency-oriented data processing.
- Training Pipeline: The three-stage pipeline progresses from cross-modal alignment to multi-task pre-training and supervised fine-tuning.The stages progressively enhance model capabilities across alignment, multimodal training, and instruction following.
- Backbone Objective: Block-level masked diffusion enables parallel decoding while maintaining coherent context within each block.BDLM applies masking to blocks rather than individual tokens, supporting variable-length multimodal sequences.
- SFT Optimization: SFT reweighting uses the inverse square root of each sample’s masked-token count to balance gradients across response lengths.This addresses the contrasting effects of token-averaged and sample-level losses when sequence lengths vary by up to two orders of magnitude.
- SFT Optimization: Complementary masking constructs primary and inverse-mask instances so every token position appears uncorrupted once per pair.The strategy doubles effective information utilization and eliminates token-level sampling bias.
- Decoder Training: The diffusion decoder is trained through warm-up, multi-domain generalization, and high-fidelity refinement stages.The final stage focuses on aesthetic fidelity and fine-grained visual details.
- Decoder Efficiency: Consistency-based distillation enables 8-step CFG-free inference while maintaining high image quality.Only an auxiliary projection layer is added during distillation, and it is discarded at inference.
- Training Efficiency: Offline token pre-extraction and data packing reduce repeated encoder computation and padding waste during training.Packing concatenates shorter samples into fixed-length sequences, increasing effective token throughput.
5.1 Multimodal Understanding
LLaDA2.0-Uni is evaluated across 21 multimodal understanding benchmarks spanning general VQA, reasoning, OCR, document understanding, and other tasks. It outperforms diffusion-based unified baselines on representative categories and approaches specialized VLM performance.
- Evaluation Setup: The evaluation covers 21 multimodal understanding benchmarks across general VQA, reasoning, OCR/document understanding, and other multimodal tasks.The benchmark suite includes tasks such as MMStar, MMMU, ChartQA, DocVQA, CountBench, and VLRewardBench.
- Evaluation Setup: LLaDA2.0-Uni is compared with specialized VLMs and unified models using autoregressive and diffusion-based approaches.The baselines include Qwen2.5-VL-7B, LLaDA-V, BAGEL, and other unified systems.
- Results: MMStar reaches 64.1 versus 58.0 for Lumina-DiMOO, while MMMU reaches 50.1 versus 44.9, showing gains over diffusion-based unified models.The reported improvements span general VQA and complex reasoning categories.
- Results: On MMStar, LLaDA2.0-Uni scores 64.1 versus 63.9 for Qwen2.5-VL-7B, and on CountBench it scores 86.0 versus 84.9.The paper reports these as instances of slightly outperforming a specialized VLM on specific metrics.
- Results: LLaDA2.0-Uni maintains high performance on challenging OCR and document understanding scenarios where Lumina-DiMOO struggles.The paper describes this performance as part of its comprehensive multimodal understanding capability.
5.2 Text-to-Image Generation
LLaDA2.0-Uni is evaluated against specialized generation models and unified AR, discrete-diffusion, and hybrid baselines across text-to-image benchmarks. It achieves strong compositional, alignment, reasoning, layout, text-rendering, and reasoning-informed generation results, while dense text generation remains weaker.
- General Image Generation: 0.89 overall on GenEval, with the highest Position score of 0.90 across evaluated models.GenEval measures object-centric text-to-image generation with compositional prompts and diverse object attributes.
- General Image Generation: 87.76 overall on DPG-Bench, the state-of-the-art score among unified models, exceeding LLaDA-o at 87.04 and HunyuanImage-3.0 at 86.10.It also surpasses Z-Image-Turbo at 84.86 and leads the Entity and Other sub-metrics.
- General Image Generation: 0.505 overall on OneIG-EN, leading unified models in Alignment at 0.882 and Reasoning at 0.323 but trailing leaders on dense text generation.The result is comparable to top dedicated generation models such as Qwen-Image.
- General Image Generation: 79.63 overall on UniGenBench (EN), with advantages in Logic at 63.99 and Layout at 90.30.The model performs consistently across all ten dimensions and surpasses many specialized generation models on these dimensions.
- Text Rendering: 0.765 overall on CVTG-2K, leading unified models while showing slower performance decline as the number of text regions increases.CVTG-2K evaluates text rendering across multiple regions.
- Reasoning-Informed Generation: 0.68 overall on WISE-Bench, ranking first among unified models, with reasoning mode adding a 10% improvement.WISE-Bench evaluates semantic understanding and world knowledge in image generation.
5.3 Image Editing
LLaDA2.0-Uni is evaluated on instruction-based and multi-reference image editing against specialized and unified baselines. It leads unified models on instruction-following edits, preserves perceptual quality, and achieves the best multi-reference composition score.
- Evaluation Settings: Evaluation covers ImgEdit-Bench and GEdit-Bench quantitatively, with qualitative comparisons on MICo-Bench.Baselines include specialized editing models and unified models; multi-reference comparisons use models that natively support the task.
- General Image Editing: 3.92 overall on ImgEdit, ranking first among unified models and exceeding OmniGen2 at 3.44 and InternVL-U at 3.67.It also leads the unified category on Adjust and Hybrid tasks.
- General Image Editing: 6.61 on English and 6.66 on Chinese GEdit-Bench evaluations, with strong Perceptual Quality performance.The results indicate that edits can be executed without sacrificing the original image's visual quality.
- Multi-Reference Editing: 47.1 overall on MICo-Bench, exceeding OmniGen2 at 33.8, Qwen-Image at 35.9, and Lumina-DiMOO at 23.3.MICo-Bench evaluates multi-reference image composition.
5.4 Interleaved
LLaDA2.0-Uni is tested on interleaved generation and reasoning, enabled by a benchmark designed for practical interleaved applications. It generally outperforms Emu3.5 in generation and demonstrates preliminary reasoning over visual and textual sequences.
- Interleaved Generation: Interleaved generation outputs combine user prompts with generated text and images in a single sequence.The supplied qualitative example is presented as an interleaved-generation output.
- Interleaved Generation: InterGen comprises 150 samples across three main categories and uses Gemini-3 and Qwen3-VL judges to assess text coherence and text-image alignment.The benchmark addresses the absence of a standard evaluation for interleaved generation.
- Interleaved Generation: LLaDA2.0-Uni generally outperforms Emu3.5 on InterGen, leading in Story Telling and Time Series Forecasting while matching it on Explanation.Other interleaved-generation models such as NextFlow and Mogao are not open-source and are therefore not included in the primary comparison.
- Interleaved Reasoning: Interleaved reasoning enables logical strategy deduction in chess and step-by-step solutions to physics problems.The paper presents this as a preliminary exploration of reasoning-informed multimodal interaction.
5.5 Ablation Study
The ablation study examines inference acceleration through SPRINT and Diffusion Decoder Turbo. These methods substantially improve speed, with SPRINT trading a small average-score decrease for higher throughput and Turbo preserving competitive quality with fewer decoder steps.
- SPRINT: 39.8 TPS with SPRINT versus 24.3 TPS without it, a 1.6× acceleration alongside an average score change from 76.3 to 75.7 (−0.6).The largest reported benchmark decreases are OCRBench at −2.3 and DPG at −1.5.
- Ablation Setup: The ablation compares performance and throughput with and without SPRINT and compares the original diffusion decoder with its distilled Turbo version.The study also includes a visual comparison of the decoder variants.
- Diffusion Decoder Turbo: 11.4× speedup for Diffusion Decoder Turbo, reducing generation time from 32.95 s/img to 2.90 s/img.The comparison uses 50 decoder steps for the base and 8 steps for Turbo at 1024 × 1024 resolution, batch size 1, and BF16 precision.
- Diffusion Decoder Turbo: 0.87 GenEval and 87.24 DPG for Decoder Turbo, compared with 0.89 and 87.76 for the base decoder.Turbo remains on par with the original decoder on UniGenBench, OneIG-EN, and WISE benchmarks.
6 Conclusion and Future Directions
LLaDA2.0-Uni unifies multimodal understanding and generation within a single diffusion large language model, with strong benchmark performance and support for interleaved generation and reasoning. Future work targets visual detail, scaling, and reinforcement learning.
- LLaDA2.0-Uni enables multimodal understanding and generation within a single diffusion large language model.
- The unified architecture achieves strong performance across multimodal understanding, image generation, and editing benchmarks.
- The model naturally supports interleaved generation and chain-of-thought reasoning, demonstrating architectural flexibility and practicality.
- Future Directions: The SigLIP-VQ tokenizer struggles to preserve fine-grained image details, motivating better reconstruction techniques for detail-sensitive tasks such as image editing.
- Future Directions: Complex interleaved generation and reasoning require further scaling of training data and model capacity.
- Future Directions: Reinforcement-learning optimization for unified dLLMs remains challenging and requires further framework refinement.