Source-linked AI summary
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework
Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Jiakang Yuan, Xinyue Li, Dongyang Liu, Xiangyang Zhu, Manyuan Zhang, Will Beddow, Erwann Millon, Victor Perez, Wenhai Wang, Conghui He, Bo Zhang, Xiaohong Liu, Hongsheng Li, Yu Qiao, Chang Xu, Peng Gao
TL;DR
Existing text-to-image models face limitations in multimodal fusion, task expansion, and high-quality captioning. Lumina-Image 2.0 addresses these issues with unified text-image processing, UniCap, and efficient training and inference strategies, achieving strong performance and promising results across public benchmarks and arenas.
Problem
Existing text-to-image models rely on cross-attention with fixed text features, require task-specific architectural changes, and lack dedicated captioning systems for accurate, sufficient descriptions.
Method
Lumina-Image 2.0 combines Unified Next-DiT joint attention over text and image tokens, UniCap for detailed image-text descriptions, and progressive training with efficient inference strategies.
Results
Lumina-Image 2.0 achieves significant improvements over Lumina-Next and promising performance on public benchmarks, with evaluations also reported through human-annotated online T2I arenas.
Takeaways & Limitations
The framework provides a unified, efficient, transparent, and reproducible approach for text-to-image generation and diverse visual generation tasks.
Takeaways & Limitations
The model struggles with complex structures and rare concepts, often produces disordered details in intricate textures, and current academic benchmarks may not reflect human-perceived image quality comprehensively.
Abstract
from arXiv · showhide
We introduce Lumina-Image 2.0, an advanced text-to-image generation framework that achieves significant progress compared to previous work, Lumina-Next. Lumina-Image 2.0 is built upon two key principles: (1) Unification - it adopts a unified architecture (Unified Next-DiT) that treats text and image tokens as a joint sequence, enabling natural cross-modal interactions and allowing seamless task expansion. Besides, since high-quality captioners can provide semantically well-aligned text-image training pairs, we introduce a unified captioning system, Unified Captioner (UniCap), specifically designed for T2I generation tasks. UniCap excels at generating comprehensive and accurate captions, accelerating convergence and enhancing prompt adherence. (2) Efficiency - to improve the efficiency of our proposed model, we develop multi-stage progressive training strategies and introduce inference acceleration techniques without compromising image quality. Extensive evaluations on academic benchmarks and public text-to-image arenas show that Lumina-Image 2.0 delivers strong performances even with only 2.6B parameters, highlighting its scalability and design efficiency. We have released our training details, code, and models at https://github.com/Alpha-VLLM/Lumina-Image-2.0.
1 Introduction
Lumina-Image 2.0 addresses limitations in multimodal fusion and T2I caption quality with a unified architecture, specialized captioner, and efficiency-oriented training and inference designs. It reports improvements over Lumina-Next on academic benchmarks and human-evaluated online arenas.
- Existing T2I models benefit from scalable Diffusion Transformers and large-scale, high-quality text-image datasets.
- Cross-attention-based models treat text embeddings as fixed external features, limiting multimodal fusion efficiency and potentially introducing uni-directional bias.
- Lumina-Image 2.0 combines Unified Next-DiT, UniCap, efficient training, and efficient inference into one T2I framework.
- Unified Next-DiT jointly processes textual and visual tokens through self-attention, supporting multimodal interaction and task expansion without core architecture changes.
- UniCap generates comprehensive multilingual descriptions at multiple granularities and dimensions to improve alignment between images and training text.
- Lumina-Image 2.0 uses progressive training, domain-specific prompts, auxiliary loss, and accelerated sampling techniques, and reports improvements over Lumina-Next across benchmarks and online arenas.
2 Related Work
Related work has shifted toward scalable Diffusion Transformers and unified multimodal representations, while vision-language models increasingly shape T2I systems. However, T2I-specific captioner research remains limited.
- Diffusion-based image generators have progressed from U-Net architectures to scalable Diffusion Transformers and unified multimodal representations.
- Vision-language model development has influenced T2I systems, with image-caption quality playing a critical role in model performance.
- Commonly used captioners include LLaVA, CogVLM, ShareGPT-4, and Qwen-VL, but they are general-purpose VLMs.
- Research developing captioners specifically tailored to text-to-image generation remains significantly underdeveloped.
3 Revisiting Lumina-Next
The paper revisits Lumina-Next’s architecture, data, and training strategy to identify limitations and scaling behavior. Lumina-Next uses Next-DiT with gated cross-attention, synthetic text-image data, progressive resolution training, and flow-model sampling schedules.
- Next-DiT is a scalable flow-based Diffusion Transformer using sandwich normalization, query-key normalization, and 2D Rotary Positional Encoding.
- Lumina-Next injects text embeddings through zero-initialized gated cross-attention for text-to-image generation.
- Lumina-Next trains on approximately 20M synthetic text-image pairs with captions generated from user prompts and vision-language models.
- Its progressive training proceeds through 256, 512, and 1024 resolutions to capture low- and high-frequency image information.
- During sampling, Lumina-Next uses flow-model time schedules to reduce ODE truncation errors and supports Euler and Midpoint solvers.
- Scaling the dataset from 20M to 200M samples while retaining Lumina-Next’s architecture and training strategy was used to examine whether insufficient data explained its performance gap.
4 Lumina-Image 2.0
Lumina-Image 2.0 unifies multimodal modeling and captioning through Unified Next-DiT and UniCap, while adding efficient training and inference strategies. The framework supports hierarchical text-image data, broader visual tasks, and faster sampling without visual degradation.
- Unified Next-DiT: Unified Next-DiT concatenates caption embeddings and noisy latents into one sequence for joint self-attention.This replaces zero-initialized gated cross-attention and enables unified text-image processing.
- Unified Captioner: UniCap re-captions diverse images with multi-granularity, multi-perspective, and multilingual descriptions to construct higher-quality training pairs.Its training data spans natural, web-crawled, synthetic, document, infographic, OCR-related, and multilingual content.
- Unified Captioner: Unified Next-DiT is designed to extend beyond text-to-image generation by incorporating visual-task annotations and prompt templates without modifying the core architecture.The training setup includes depth maps, pose maps, canny maps, and sketches arranged with paired images in composite grids.
- Unified Captioner: Increasing caption length acts as a controllable means of scaling model parameters, allowing capacity to be modulated during training and inference.The formulation represents text-to-image attention as an FFN whose dynamic hidden size is determined by caption length.
- Efficient Training: The efficient training framework combines progressive stages, hierarchical data, domain-specific system prompts, and auxiliary loss to improve refinement and convergence.Inference combines CFG-Renorm and CFG-Trunc, which address artifacts and redundant CFG calculations while extending the usable CFG-scale range.
- Efficient Inference: Over 20% sampling acceleration is achieved without visual degradation through the reported inference modification.The paper also reports complementary benefits when CFG-Renorm and CFG-Trunc are applied together.
5 Experiments
Lumina-Image 2.0 demonstrates strong benchmark, arena, multilingual, captioning, and image-generation performance, while progressive training improves metrics and CFG-Renorm with CFG-Trunc balances inference efficiency and quality.
- Quantitative Performance: Lumina-Image 2.0 achieves the second-best performance on several GenEval metrics and outperforms all compared models on DPG’s Entity, Relation, and Attribute sub-metrics.
- Quantitative Performance: Lumina-Image 2.0 ranks second only to FLUX Pro for prompt alignment and significantly outperforms Lumina-Next and other Chinese open-source models on AGI-Eval.
- Qualitative Performance: The model demonstrates zero-shot multilingual generation across Chinese, Japanese, English, Russian, and German, alongside robust text-image alignment for Chinese and English prompts.
- Qualitative Performance: UniCap generates detailed, accurate, multilingual captions across complex scenes, dense text, visual understanding, and spatial relationships, outperforming ShareGPT4V and Florence.
- Ablation Study: 86.6 on DPG and 0.71 on GenEval are reached within 1K high-quality tuning steps, up from 85.7 and 0.67 respectively.
- Ablation Study: The final solution adopts CFG-Renorm and CFG-Trunc to balance efficiency and quality in practical applications.
6 Limitation
Lumina-Image 2.0 still struggles with complex structures, rare concepts, intricate textures, and long complex text.
- The model struggles to consistently render correct complex structures and rare concepts, often produces disordered intricate details, and needs substantial improvement with long complex text.
7 Conclusion
Lumina-Image 2.0 is a unified and efficient text-to-image framework combining joint textual-visual processing, specialized captioning, and efficiency-oriented training and inference strategies.
- Unified Next-DiT integrates textual and visual information to generate high-quality images, while UniCap produces detailed and accurate descriptions for image-text training pairs.