Source-linked AI summary
ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement
Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, Hang Xu
TL;DR
Existing unified multimodal models struggle to combine understanding, generation, and editing while preserving both semantic alignment and image texture. ILLUME+ addresses this with DualViTok, unified coarse-to-fine representations, continuous visual inputs, diffusion decoding, and progressive training, achieving competitive cross-task performance with a 3B model. The paper positions this design as a scalable foundation for flexible-resolution multimodal applications.
Problem
Existing unified models have difficulty simultaneously supporting visual understanding, generation, and editing while preserving semantic alignment and image texture.
Method
ILLUME+ combines DualViTok semantic-and-texture tokenization, continuous visual inputs, unified coarse-to-fine representations, diffusion decoding, and progressive dynamic-resolution training.
Results
ILLUME+ with 3B parameters performs competitively across multimodal understanding, generation, and editing benchmarks while supporting flexible-resolution inputs and outputs.
Takeaways & Limitations
The unified design provides a flexible foundation for multimodal applications spanning understanding, generation, and editing.
Takeaways & Limitations
The authors identify scaling to larger models and developing more complex interleaved pretraining, datasets, and post-training strategies as future work.
Abstract
from arXiv · showhide
We present ILLUME+ that leverages dual visual tokenization and a diffusion decoder to improve both deep semantic understanding and high-fidelity image generation. Existing unified models have struggled to simultaneously handle the three fundamental capabilities in a unified model: understanding, generation, and editing. Models like Chameleon and EMU3 utilize VQGAN for image discretization, due to the lack of deep semantic interaction, they lag behind specialist models like LLaVA in visual understanding tasks. To mitigate this, LaViT and ILLUME employ semantic encoders for tokenization, but they struggle with image editing due to poor texture preservation. Meanwhile, Janus series decouples the input and output image representation, limiting their abilities to seamlessly handle interleaved image-text understanding and generation. In contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which preserves both fine-grained textures and text-aligned semantics while enabling a coarse-to-fine image representation strategy for multimodal understanding and generation. Additionally, we employ a diffusion model as the image detokenizer for enhanced generation quality and efficient super-resolution. ILLUME+ follows a continuous-input, discrete-output scheme within the unified MLLM and adopts a progressive training procedure that supports dynamic resolution across the vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible and efficient context-aware image editing and generation across diverse tasks. ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs and specialized models across multimodal understanding, generation, and editing benchmarks. With its strong performance, ILLUME+ provides a scalable and versatile foundation for future multimodal applications. Project Page: https://illume-unified-mllm.github.io/.
1 Introduction
ILLUME+ addresses the difficulty of unifying visual understanding, generation, and editing by combining semantic and texture-preserving tokenization with unified image representations and progressive training. Its 3B model supports flexible-resolution inputs and outputs while achieving competitive performance across these tasks.
- Limitations of Existing Models: VQGAN-based approaches share text and image vocabularies but lag in visual understanding, while semantic tokenizers improve alignment at the expense of texture preservation.Other designs also introduce separate heads or decouple understanding from generation, limiting modality switching and interleaved image-text tasks.
- Motivation: Unified models must combine visual understanding, high-quality generation, and instruction-following editing in one flexible foundation model.These capabilities are presented as core requirements for scaling across model capacity and task diversity.
- Key Designs: The model adopts continuous visual inputs before quantization and a unified coarse-to-fine representation that generates semantic tokens before pixel tokens.This design avoids quantization-induced input information loss, bridges text with visual textures, and supports a unified language-model head.
- Key Designs: ILLUME+ uses DualViTok to capture both text-aligned semantics and fine-grained textures for unified understanding, generation, and editing.Its semantic branch uses a pretrained text-aligned vision encoder, while the pixel branch combines semantic and pixel features and uses noise injection for robust decoding.
- Key Designs: A diffusion decoder improves generation fidelity and reduces artifacts while efficiently upscaling images during decoding.The decoder also mitigates token explosion for high-resolution autoregressive generation.
- Training: Progressive training increases resolution, task diversity, and task complexity across the vision tokenizer, MLLM, and diffusion decoder.This procedure supports flexible visual resolution while maintaining training stability and final performance.
- Results: ILLUME+ uses only a 3B LLM yet performs competitively against unified and specialized models across understanding, generation, and editing benchmarks.It supports high-resolution inputs and outputs, generation up to 1024×1024, stronger document-oriented performance, and improved texture preservation over ILLUME.
2 Related Work
Related work has pursued unified multimodal understanding and generation through autoregressive tokenization, unified architectures, and combinations of specialized components. These approaches improve capability but retain trade-offs in modality integration, efficiency, or task unification.
- MLLM for Image Understanding: MLLMs such as LLaVA, MiniGPT-4, QwenVL, and InternVL use aligned visual features and improved data or training strategies for image understanding.This line of work focuses primarily on visual perception rather than jointly unifying understanding, generation, and editing.
- Image Generation Models: Diffusion models generate images through iterative denoising, while autoregressive methods predict discrete image tokens produced by VQGAN-like tokenizers.ILLUME+ combines autoregressive multimodal processing with a diffusion decoder for improved reconstruction quality.
- Unified Multimodal Understanding and Generation: Unified multimodal models predict text and visual elements together, but non-unified modality optimization and additional diffusion components can limit feature integration or efficiency.VQ-tokenizer approaches unify vocabularies, yet the broader design space continues to balance integration, generation quality, and flexibility.
3 Method
ILLUME+ combines DualViTok, a unified MLLM, and a diffusion decoder to represent images with both semantic and texture information for understanding, generation, and editing. Its coarse-to-fine token sequence and progressive training support context-aware generation across flexible resolutions.
- Architecture: ILLUME+ combines a dual vision tokenizer, MLLM, and diffusion decoder to support visual understanding, generation, and editing.DualViTok captures deep semantic information and fine-grained texture details, while the diffusion decoder produces high-quality images.
- Dual Vision Tokenizer: DualViTok uses separate semantic and pixel branches, with 28× and 16× downsampling rates respectively, to preserve aligned semantics and fine-grained textures.The semantic branch uses a pretrained text-aligned QwenViT encoder, while the pixel branch follows a MoVQGAN-based architecture.
- Dual Vision Tokenizer: A 10% perturbation probability with 10% token replacement trains the tokenizer decoder to handle incorrect visual-token predictions more robustly.Noise injection is intended to reduce artifacts caused by erroneous tokens generated by the language model.
- Unified Multimodal Large Language Model: The unified MLLM uses continuous visual inputs and discrete visual outputs, aligning semantic and pixel features with the language-model input space through separate vision adaptors.Images are represented by discrete tokens for visual generation within a unified text-image framework.
- Unified Multimodal Large Language Model: Images are generated in a coarse-to-fine sequence, with semantic tokens preceding pixel tokens so content is determined before texture details are refined.Explicit image, semantic, and pixel boundary markers, row delimiters, and resolution indicators make the representation unambiguous.
- Training Procedure and Data Composition: Progressive training increases resolution, task complexity, and task diversity across the tokenizer, diffusion decoder, and MLLM.The pipeline pretrains dual-tokenizer reconstruction, fine-tunes diffusion decoding, and trains the MLLM through three stages with progressively broader capabilities.
4 Experiments
Experiments evaluate ILLUME+ across understanding, generation, editing, tokenizer reconstruction, and ablations. Results show competitive task performance, strong reconstruction and generation quality, and benefits from continuous input and dual-tokenizer design.
- Multimodal understanding: ILLUME+ achieves competitive performance against understanding-only and unified models on visual understanding benchmarks.
- Multimodal image generation: 6.00 FID on MJHQ-30K, with 0.72 accuracy in advanced GenAI-bench categories, demonstrates strong generation quality and text-image consistency.
- Multimodal image editing: ILLUME+ demonstrates strong image-editing performance, particularly on CLIP-T, while dual-codebook design improves consistency with source images.
- Image reconstruction: DualViTok achieves state-of-the-art reconstruction at 256 × 256 and improves rFID by 0.56 over VILA-U at 384x384.
- Ablation studies: Dual tokenization outperforms single tokenization for reconstruction, while coarse-to-fine codebook combination improves generation by 1.54 rFID.
- Ablation studies: Continuous visual input improves performance across all understanding benchmarks compared with discrete input.
- Ablation studies: DualViTok component changes reduce rFID from 1.83 to 1.33, with encoder-decoder scaling, DC blocks, larger codebooks, and random noise contributing improvements.
- Ablation studies: Random noise better reflects erroneous tokens predicted by LLMs, motivating its use despite both random and zero noise improving reconstruction over baseline.
5 Conclusion
ILLUME+ unifies visual understanding, generation, and editing through DualViTok, diffusion decoding, coarse-to-fine representation, and progressive dynamic-resolution training. Future work targets larger models, better interleaved pretraining, richer datasets, and post-training strategies.
- ILLUME+ integrates visual understanding, generation, and editing with DualViTok, diffusion decoding, and progressive training for dynamic resolution.
- Future work includes scaling beyond 3B parameters, improving image-text interleaved pretraining, constructing complex multimodal datasets, and exploring post-training strategies.