Source-linked AI summary
Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei, Yuewen Cao, Keqi Wang, Yibin Wang, Jinbin Bai, Qian Yu, Dengyang Jiang, Yuandong Pu, Haoxing Chen, Le Zhuo, Junjun He, Gen Luo, Tianbin Li, Ming Hu, Jin Ye, Shenglong Ye, Bo Zhang, Chang Xu, Wenhai Wang, Hongsheng Li, Guangtao Zhai, Tianfan Xue, Bin Fu, Xiaohong Liu, Yu Qiao, Yihao Liu
TL;DR
Unified multi-modal models face slow generation, image-quality, and modality-unification challenges. Lumina-DiMOO uses a unified discrete diffusion framework for broad generation and understanding, reporting 32x faster text-to-image generation than Lumina-mGPT 2.0 and strong benchmark performance. Its open-source release and interactive retouching capability broaden its supported research and application scope.
Problem
Earlier unified models suffered slow autoregressive generation, suboptimal image quality, or reduced modality unification from added diffusion components.
Method
Lumina-DiMOO uses a unified discrete diffusion objective over mixed text-image token sequences, with masked training and iterative prediction, sampling, and remasking at inference.
Results
32x speed improvement in text-to-image generation over Lumina-mGPT 2.0 is reported, alongside additional 2x sampling acceleration from training-free ML-Cache and top performance across benchmarks.
Takeaways & Limitations
The open-source model supports broad multi-modal generation and understanding, including zero-shot inpainting and interactive retouching through precise regional annotations.
Takeaways & Limitations
Interactive Retouching lacks a guarantee that content outside user-specified annotations remains unchanged.
Abstract
from arXiv · showhide
We introduce Lumina-DiMOO, an open-source foundational model for seamless multi-modal generation and understanding. Lumina-DiMOO sets itself apart from prior unified models by utilizing a fully discrete diffusion modeling to handle inputs and outputs across various modalities. This innovative approach allows Lumina-DiMOO to achieve higher sampling efficiency compared to previous autoregressive (AR) or hybrid AR-Diffusion paradigms and adeptly support a broad spectrum of multi-modal tasks, including text-to-image generation, image-to-image generation (e.g., image editing, subject-driven generation, and image inpainting, etc.), as well as image understanding. Lumina-DiMOO achieves state-of-the-art performance on multiple benchmarks, surpassing existing open-source unified multi-modal models. To foster further advancements in multi-modal and discrete diffusion model research, we release our code and checkpoints to the community. Project Page: https://synbol.github.io/Lumina-DiMOO.
1. Introduction
Lumina-DiMOO addresses the speed and image-quality limitations of earlier unified models with an open-source discrete diffusion architecture for broad multi-modal generation and understanding. It reports faster inference, zero-shot inpainting with interactive retouching, and strong benchmark performance.
- Motivation: Earlier pure autoregressive unified models generated slowly and produced suboptimal image quality, while hybrid diffusion heads improved quality but weakened modality unification.The cited comparison describes a progression from pure AR to AR with a diffusion head and hybrid AR-discrete diffusion approaches.
- Contribution: Lumina-DiMOO is an open-source unified diffusion large language model supporting text-to-image, image-to-image, and image-understanding tasks.Supported image-to-image tasks include editing, style transfer, subject-driven, controllable, multi-view, and dense prediction generation.
- Efficiency: 32x speed improvement in text-to-image generation is achieved over Lumina-mGPT 2.0, with ML-Cache adding a further 2x sampling-speed boost.ML-Cache is training-free and reuses similar representations associated with tokens having high maximal logit values.
- Capabilities: Discrete diffusion enables zero-shot inpainting and Interactive Retouching, allowing users to refine specified regions through precise annotations.The paper presents this as a flexible application that is difficult for other methods to achieve.
- Evaluation: Lumina-DiMOO surpasses leading open-source unified models across multi-modal generation and understanding benchmarks and ranks first among open-source models on UniGenBench.The paper also reports extensive qualitative comparisons demonstrating superior performance.
2. Related Work
Prior work explores discrete diffusion and unified multi-modal generation and understanding, but existing systems often retain modality-specific designs or rely on external diffusion models. Masked diffusion provides a standard discrete-diffusion basis with iterative and parallel generation capabilities.
- Diffusion Large Language Models: Recent discrete diffusion language models commonly use masked diffusion, replacing selected data tokens with a special [mask] state and recovering them in reverse.The approach is described as simple and effective, with a reverse process analogous to BERT.
- Diffusion Large Language Models: Discrete diffusion language models offer bidirectional attention, iterative refinement, flexible generation order, parallel decoding, and infilling capabilities.These properties distinguish the paradigm from standard next-token generation.
- Unified Generation and Understanding: Unified multi-modal systems have used external continuous diffusion models after LLM image-feature regression, achieving visual generation while compromising true modality unification.The cited related-work passage identifies reliance on external models as the source of this compromise.
- Lumina-DiMOO: Lumina-DiMOO’s overview presents masked training on text and image tokens followed by progressive masked-token prediction during inference.The figure frames the model as a discrete diffusion system spanning both modalities.
- Unified Generation and Understanding: Other unified models share a transformer across modalities but still use modality-specific designs, complicating the model and reducing unity.The passage contrasts text autoregression with continuous or discrete diffusion for images before describing efforts to tokenize all modalities discretely.
3. Lumina-DiMOO
Lumina-DiMOO unifies text and image modeling through discrete diffusion, using masked-token prediction and iterative parallel refinement for generation and understanding. Its design combines multi-modal tokenization, resolution-preserving image representation, parallel sampling, and cache-based acceleration while balancing efficiency against per-step computational cost.
- Foundation Image Tokenizer: The model uses a 16×16-downsampling aMUSEd-VQ tokenizer to balance image reconstruction quality with high-resolution token-sequence efficiency.Chameleon-VQ produced slightly inferior reconstructions, while Open-MAGVIT2’s token format did not align with the modeling needs.
- Unified Discrete Diffusion Modeling: Lumina-DiMOO masks randomly selected tokens in mixed text-image sequences and predicts the original masked tokens from unmasked context and optional conditions.The same masked-diffusion formulation applies to textual and visual modalities, with masked cross-entropy training over sampled mask ratios.
- Sampling Strategies: At inference, generation starts fully masked and iteratively performs parallel prediction, sampling, confidence-based remasking, and refinement for T steps.Image sampling restricts predictions to the image vocabulary, uses highest-probability codebook entries, applies a cosine remasking schedule, and uses classifier-free guidance.
- Model Design: The model expands LLaDA’s vocabulary with 8,192 visual tokens and special image-boundary tokens, enabling text and image elements to share one token sequence.The combined vocabulary contains 126,345 text tokens, 8,192 visual tokens, and special tokens such as <IMAGE> and </IMAGE>.
- Arbitrary Resolution Image Representation: Row-delimiter <end-of-line> tokens preserve two-dimensional image structure after flattening, allowing different aspect ratios to be parsed and reconstructed without new positional embeddings.This addresses the problem that images such as 512×1024 and 1024×512 otherwise become equal-length one-dimensional sequences.
- Acceleration Sampling via Max Logit-based Cache: Because bidirectional attention makes each parallel-diffusion step costly, ML-Cache reuses representations and logits for tokens selected by maximal-logit confidence.Tokens with high maximal logits tend to have stable representations; the cache selects a configurable top cache_ratio fraction for reuse while recomputing the remainder.
4. Training Pipeline
Lumina-DiMOO is trained through four stages that progressively develop visual capability, multimodal alignment, image-to-image skills, instruction following, and unified generation-understanding reinforcement learning. Self-GRPO closes the loop between text-to-image generation and multimodal understanding while preserving diffusion sampling trajectories.
- Training Pipeline: The training pipeline comprises four stages, with Self-GRPO specifically designed for Lumina-DiMOO’s discrete diffusion and unified generation-understanding architecture.The four-stage pipeline is summarized in Table 2.
- Stage-I: Multi-Modal Pre-Training: Multi-modal pre-training concatenates text-image pairs into a single sequence and uses random masking to learn from unmasked text and image tokens.A progressive schedule addresses the complexity of long visual token sequences.
- Stage-II: Mid-Training: Mid-training integrates image-to-image tasks and specialized visual data, including editing, subject-driven generation, controllable generation, style transfer, and multi-view generation.Image-to-image tasks use 512 resolution for efficiency, while text-to-image tasks use 1024 resolution for finer details.
- Stage-III: Supervised Fine-Tuning: Supervised fine-tuning uses high-quality system-prompt, user-prompt, and answer triples to improve instruction alignment and multimodal generation and understanding quality.System and user prompts remain unchanged while answer tokens are masked for loss computation.
- Stage-IV: Self-Improving via GRPO: Self-GRPO jointly optimizes text-to-image generation and multimodal understanding using semantic rewards, selected trajectory steps, and reward-weighted objectives.Rewards count correct answers to questions conditioned on generated images, while selected timesteps reduce gradient computation and memory use.
- Stage-IV: Self-Improving via GRPO: Self-GRPO combines trajectory-consistent training, KL-regularized policy updates, memory-efficient training, and multimodal reward supervision to connect generation with understanding.The framework retains the complete sampling trajectory but computes gradients only at selected timesteps.
5. Data Construction
Lumina-DiMOO’s data construction spans large-scale text-image pre-training, specialized mid-training domains, supervised multimodal instruction data, and text-prompt data for Self-GRPO semantic alignment.
- Stage-I: Pre-Training Data: Approximately 80 million high-quality text-image pairs support pre-training, combining 30 million re-captioned public pairs with 50 million pairs from Lumina datasets.The data supports both image understanding and image generation pre-training.
- Stage-II: Mid-Training Data: Mid-training adds 3 million images covering table and chart comprehension, mathematics and geometry, user-interface parsing, and image-to-image tasks.These challenging-domain images are captioned using Qwen2.5-VL.
- Stage-III: Supervised Fine-Tuning Data: Supervised fine-tuning uses 15 million image-understanding samples, 15 million visual-generation samples, and additional datasets for image-to-image and low-level vision tasks.The image-to-image data includes subject-driven, controllable, dense-prediction, style-transfer, editing, and low-level vision examples.
- Stage-IV: Self-GRPO Data: Self-GRPO requires only text prompts, from which entity-relation-value triples are extracted and converted into single-choice questions for semantic alignment supervision.Distractor options are generated using global candidate pools.
6. Evaluation
Lumina-DiMOO performs strongly across text-to-image and image-to-image evaluations, leading several benchmarks and supporting diverse editing and generation tasks.
- Text-to-Image Generation: 88% overall on GenEval surpasses FLUX.1 [Dev] (82%), Lumina-mGPT 2.0 (69%), and unified models including GPT-4o (84%).The improvement over MMaDA is 25% overall, from 63% to 88%.
- Text-to-Image Generation: 86.04 overall on DPG surpasses all previous models, while MMaDA scores 69.97 under the same settings.DPG evaluates adherence to 1,065 dense prompts, including entities and relationships.
- Text-to-Image Generation: Lumina-DiMOO ranks first overall on UniGenBench, particularly excelling in Layout and Attribute across the benchmark’s evaluation dimensions.The leaderboard is evaluated and maintained by the Tencent Hunyuan team.
- Text-to-Image Generation: Lumina-DiMOO achieves the highest average score on OneIG-EN and ranks first in Alignment, Text, and Reasoning.On TIIF, it ranks second overall, surpassed only by FLUX.1 [dev].
- Image Generation: Qualitative comparisons show higher image quality than MMaDA and Janus-Pro 7B, with flexible resolution support beyond their fixed or restricted resolutions.The compared models include MMaDA at 512×512 and Janus-Pro at 768×768.
- Image-to-Image Generation: The model supports text-guided inpainting and extrapolation without fine-tuning, while controllable generation combines competitive controllability with superior visual quality and text consistency.It also performs well in object addition and replacement, background and style changes, and original-resolution preservation.
GPT-4o
The evaluation and visualizations compare Lumina-DiMOO with unified, specialized, and commercial multimodal systems across editing, understanding, sampling, and qualitative tasks.
- Image Editing: Lumina-DiMOO’s image editing comparison covers object addition, removal, replacement, background modification, and style modification against BAGEL and GPT-4o.The comparison emphasizes instruction adherence and resolution preservation.
- Style Transfer: Lumina-DiMOO exceeds OmniGen in style transfer, while qualitative results show OmniGen performing worse in most cases.The style-transfer comparison uses text alignment and style consistency.
- Multimodal Understanding: On multimodal understanding benchmarks, Lumina-DiMOO outperforms dedicated understanding models and achieves leading scores of 87.4 on POPE, 83.1 on SEED, and 58.6 on MMMU.The evaluated benchmarks cover perception, cognition, and multimodal reasoning.
- Multimodal Understanding: Visualizations cover OCR, captioning, mathematical geometry, and table understanding, highlighting text recognition, image description, geometry, and table analysis.Figure 11 presents these four understanding tasks.
- Sampling Efficiency: Sampling-time comparisons cover text-to-image generation and image understanding, using 768, 720, or 1024 image resolutions and 1024 tokens for understanding.The figure compares Lumina-mGPT 2.0, Emu3, and other models.
7. Ablation and Extension
Ablations show that Lumina-DiMOO benefits from diffusion sampling efficiency and ML-Cache, while an extension enables interactive localized retouching.
- Sampling Efficiency: Lumina-DiMOO samples text-to-image outputs several times faster than AR models and is roughly as fast as BAGEL under 64 sampling steps.Reducing the sampling steps further increases its speed advantage.
- ML-Cache: ML-Cache improves sampling efficiency by 2.05× for text-to-image generation and 1.87× for image understanding.For 1024×1024 image generation, GPU usage rises from 38.9 GB to 45.9 GB.
- Initialization Ablation: Training from scratch falls short for image generation and understanding, whereas LLaDA initialization supports both capabilities.The scratch-trained model often produces very large gradient norms during training.
- Interactive Retouching: Interactive Retouching masks user-annotated regions for regeneration while preserving information outside those annotations.Users can repeatedly modify selected areas through precise annotations.
- Interactive Retouching: The paper states that existing commercial editing models do not guarantee unchanged content outside user-specified annotations.This boundary is presented as a distinction of the interactive retouching capability.
8. Conclusion
Lumina-DiMOO is presented as a unified foundation model for multimodal understanding and generation, combining strong benchmark performance, fast sampling, and interactive retouching. The authors identify broader multimodal integration as future work, especially for video and audio.
- Lumina-DiMOO delivers top-tier performance on standard multimodal generation and understanding benchmarks.
- Its ultra-fast sampling speed and interactive retouching features distinguish the model within its supported capabilities.
- The model is open-sourced to advance research in multimodal and discrete diffusion modeling.
- Current capabilities focus on image generation and understanding, while future work aims to integrate video, audio, and additional modalities.The authors state that this expansion will require versatile tokenizers, temporal-information processing architectures, and advanced training techniques.