Source-linked AI summary
Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation
Chonghuinan Wang, Zhikai Chen, Chunwei Wang, Yecong Wan, Junwei Yang, Zhixin Wang, Wei Zhang, Jiaqi Xu, Renjing Pei, Xiaohe Wu, Fan Li, Wangmeng Zuo
TL;DR
Unified multimodal models need to generate free-form interleaved text-image sequences and be evaluated robustly despite variable output structures. ILLUME-X combines an interleaved data pipeline, progressive training, and ILScore, and outperforms previous unified models across multiple generation tasks.
Problem
Existing evaluation for interleaved text-image generation can fail when outputs deviate from predefined templates, limiting objective and robust measurement.
Method
ILLUME-X combines a 100K-sample multimodal data pipeline, shared-attention architecture, progressive interleaved training, and the ILScore evaluation protocol.
Results
ILLUME-X outperforms previous methods across multiple interleaved text-image generation tasks, including style transfer, image decomposition, and storytelling.
Takeaways & Limitations
The framework supports unified multimodal models that generate free-form interleaved text-image content and provides an evaluation protocol tailored to these sequences.
Takeaways & Limitations
ILLUME-X is primarily trained and evaluated at 512 resolution, while scaling interleaved generation to 1024 and above remains challenging.
Abstract
from arXiv · showhide
The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.
1 Introduction
ILLUME-X is a unified multimodal paradigm for high-quality, free-form interleaved text-image generation. It combines a 100K-sample data pipeline, progressive self-adaptive training, and ILScore evaluation, and outperforms previous methods across interleaved generation tasks.
- Unified multimodal models merge visual understanding and generation to model relationships between visual and textual information within one framework.
- ILLUME-X curates 100K high-fidelity interleaved training samples from authentic video sequences and synthetic multimodal-LLM data.The pipeline captures real-world temporal dynamics and diverse generative scenarios.
- ILLUME-X enables free-form N-to-M interleaved generation through progressive training with self-adaptive objectives for free-length multimodal token sequences.
- ILScore provides a multi-dimensional evaluation protocol specifically designed for complex text-image sequences across modalities.
- Extensive experiments show that ILLUME-X outperforms previous methods for unified generation of free-form interleaved content.The introduction highlights style transfer, image decomposition, and storytelling as interleaved text-image generation tasks.
2 Related Work
Related work organizes unified multimodal models into diffusion, autoregressive, and hybrid paradigms, but existing systems generally target one modality at a time. Although some studies explore interleaved image–text generation, dedicated validation of N-to-M scenarios remains limited.
- Unified multimodal models consolidate vision-language reasoning, text-to-image synthesis, and image editing within a single architectural framework.
- Existing unified models are broadly categorized as diffusion, autoregressive, or hybrid combinations of both paradigms.
- Hybrid models combine the compositional control of autoregressive decoding with the high-fidelity synthesis of diffusion.
- Despite generating both images and text, existing unified models typically require a single target modality specified in advance.
- Dedicated investigation and validation of N-to-M interleaved generation scenarios remain limited, despite studies demonstrating their feasibility.
3 Approach
ILLUME-X uses a unified decoder-only Transformer with modality-specific components to process free-form interleaved text-image sequences. Its training combines modality-specific objectives, guidance control, and complementary supervised data components.
- Unified Transformer: ILLUME-X uses separate ViT and VAE visual encoders with a decoder-only Transformer backbone for visual semantics, low-level details, and multimodal generation.The Transformer uses shared self-attention with modality-specific QKV projections and FFNs, alongside RMSNorm, QK-Norm, and SwiGLU.
- Interleaved Attention: Interleaved attention concatenates text, vision, and special tokens into one sequence, using BOI, EOI, and EOS to mark modality transitions.The framework explicitly organizes multimodal tokens with an attention mask for free-form interleaved contexts.
- Training Objective: Training jointly applies Cross-Entropy loss to text and control tokens and Rectified Flow-based MSE loss to vision tokens.This objective preserves language modeling and modality transitions while supporting high-fidelity image synthesis.
- Training Objective: Interleaved classifier-free guidance decomposes multimodal conditions by modality, using text-condition CFG as the anchor with separate γtxt and γimg coefficients.Training samples retain both conditions, drop only text, or drop both text and image conditions, while excluding losses for text tokens in the relevant guidance setup.
- Training Data: Supervised adaptation combines in-context image generation training with mixed interleaved text-image training across understanding, generation, and interleaved tasks.In-context generation improves sequential image generation, while heterogeneous joint training targets balance and cross-task generalization.
4 Interleaved Data Curation
ILLUME-X curates interleaved text-image data from existing resources and new pipelines designed to improve quality, semantic coverage, temporal representation, and narrative coherence. The curation includes video-derived sequences, collaboratively generated image sequences, and self-reflection examples.
- Dataset Expansion: Existing interleaved datasets from VINCIE, SEED-Story, and CoMM are supplemented because open-source resources have mediocre image quality, constrained instruction fidelity, and limited task diversity.The additional training data is curated to address these shortcomings and align with the research objectives.
- Video Data Extraction: The video pipeline extracts chronological representative frames and causally rich descriptions to capture major semantic transitions across actions, objects, and viewpoints.The resulting frames and descriptions train temporal evolution modeling for actions or events.
- Video Data Extraction: A multi-interval sliding-window sampler combines varying window sizes and temporal intervals to capture visual dynamics at multiple temporal scales.Examples include 2, 5, or 8-frame windows sampled every 1, 3, or 10 seconds, followed by refinement.
- Video Data Extraction: Frames undergo aesthetic and motion-based filtering, while Qwen-3-VL-32B produces global, per-frame, and inter-frame descriptions at multiple granularities.Filtering uses Laplacian variance and MANIQA scores to discard low-quality samples and reduce redundancy.
- Generated Image Sequences: A collaborative workflow uses textual planning, iterative image synthesis, and image-to-image feedback to generate associated multi-image sequences with coherent visual evolution.An external MLLM plans global and per-image semantics, while Qwen3-VL-32B analyzes differences that guide subsequent synthesis.
- Self-Reflection Data: The self-reflection dataset has a critic evaluate initial texts and images against user requirements, diagnose misalignment or aesthetic flaws, and formulate corrective guidance.The feedback is prepended to the original prompt to steer subsequent generation.
5 Experiments
Experiments evaluate ILLUME-X on interleaved text-image generation and standard text-to-image benchmarks. The model achieves leading interleaved-generation performance, strong ILScore results, and competitive text-to-image quality with lower deployment overhead.
- Interleaved Text-Image Generation: Qualitatively, ILLUME-X produces more visually aligned, narratively consistent outputs and follows complex instructions more effectively than Emu 3.5.Examples include cohesive ant-journey storytelling and smooth strawberry-to-juice transitions.
- Interleaved Text-Image Generation: ILLUME-X achieves an average ISG-Bench score of 6.26, outperforming Anole (2.81) and MiniGPT-5 (2.79) by over 120% in AVG score.It reaches state-of-the-art performance among unified models.
- Interleaved Text-Image Generation: ILLUME-X obtains the highest ILScore aggregate score of 5.34, narrowly surpassing Emu 3.5 (5.33) and exceeding Gemini 3 Pro (4.84).It ranks first in four of eight ILScore categories, including structurally complex tasks.
- Efficiency and Ablation: The model reduces training overhead and inference latency, supporting practical deployment, while its 7B+7B architecture handles dense prompts requiring fine-grained semantic alignment.Ablations focus on Visual Storytelling and Image-Text Complementation “howto” tasks because they rely on temporal logic and multimodal complementarity.
- Text-to-Image Generation: ILLUME-X scores 0.85 overall on GenEval, surpassing OmniGen2 (0.80) and BAGEL (0.82).It excels in Counting (0.81) and Colors (0.92).
- Text-to-Image Generation: ILLUME-X achieves the highest DPG-Bench Overall score of 86.38, surpassing BAGEL (85.07) and Janus-Pro (84.19).It leads Attribute (94.08) and Entity (91.80), with a +2.79 Attribute margin over BAGEL and a Global score of 90.28.
6 Limitations
ILLUME-X is mainly trained and evaluated at 512 resolution, while scaling interleaved generation to 1024 and above remains challenging because of architectural and context-length limitations.
- Resolution and scaling: ILLUME-X primarily operates at 512 resolution; scaling training and inference to 1024 and above remains difficult, leaving high-resolution quality needing improvement.The limitations arise from the underlying model architecture and finite context length.
7 Conclusion
ILLUME-X is a unified multimodal model for free-form interleaved text-image generation, supported by an efficient data pipeline and the ILScore evaluation metric. Extensive comparative experiments demonstrate the framework’s effectiveness.
- 7 Conclusion: ILLUME-X supports free-form interleaved text-image generation as a unified multimodal model.The framework is designed for generating interleaved sequences containing both text and images.
- 7 Conclusion: The framework includes an efficient multimodal interleaving data construction pipeline for producing training samples.
- 7 Conclusion: ILScore extends existing evaluation metrics to better reflect the requirements of interleaved text-image generation.
- 7 Conclusion: Extensive comparative experiments demonstrate the effectiveness of the proposed framework.
Supplementary Material
The supplementary material provides additional explanations and experimental results supporting the main paper, including setup details, evaluation prompts, visualizations, and custom interleaved dataset cases.
- Supplementary Material: The supplement covers detailed experimental setup, evaluation prompt templates, additional result visualizations, and custom interleaved datasets.These materials are organized into Sections A–D.
A Experiment Setting
The experiment setting is documented in Table S1, which presents ILLUME-X’s training recipe and detailed setup. The setup covers hyperparameters and training data.
- Experimental Setup: The detailed experimental setup includes hyperparameters.These implementation details are included in Table S1.
- Experimental Setup: The setup also specifies the training data used for ILLUME-X.Training data is identified as part of the experimental setup.
- Training Recipe: Table S1 presents the training recipe of ILLUME-X.It is the central reference for the experiment setting.
B Evaluation · C More Results
The evaluation introduces ILScore, a comprehensive metric for interleaved text-image sequences, and details its accuracy dimensions and scoring criteria. Additional results compare models across eight major multimodal tasks.
- B Evaluation: ILScore evaluates interleaved text-image generation across four dimensions: image-text, single image, image sequence, and text accuracy.The paper introduces ILScore in Sec. 5.1 and provides a detailed description of its dimensions.
- B Evaluation: Image-text accuracy measures coherence, content accuracy, relevance and responsiveness, logicality, and creativity and originality in multimodal generation.These metrics assess connections between modalities, factual correctness, responsiveness, reasoning, and originality.
- B Evaluation: Single-image accuracy evaluates conceptual consistency, relation matching, detail faithfulness, clarity and plausibility, and image aesthetics.These aspects assess whether described objects, relationships, details, and visual qualities are accurately represented.
- B Evaluation: Image-sequence accuracy considers cross-image content consistency and style consistency across images in the same sequence.The criteria assess stable object, character, and scene identity alongside consistent artistic style.
- B Evaluation: Text accuracy uses the same metric as image-text accuracy while focusing exclusively on the accuracy of the text itself.This dimension isolates textual correctness from multimodal generation quality.
- B Evaluation: Detailed scoring prompts are provided in Figures S1, S2, S3, S4, and S5.These supplementary figures document the prompts used for evaluation scoring.
- C More Results: The paper compares experimental results across Style transfer, Progressive, 3D Scene, Image Decomposition, Image-Text Complementation, Temporal Prediction, Visual Story Telling, and VQA.These comparisons are presented in Figures S6 to S13.
D Custom Dataset for Interleaved
The custom dataset spans diverse free-form interleaved text-image tasks, including ordered transformations, viewpoint changes, procedural instructions, storytelling, historical explanation, and visual question answering. Prompts consistently specify image counts, sequencing, and accompanying descriptions to structure multimodal responses.
- Ordered Transformations: Style-transfer prompts request three images applying rainy, night, and cloudy conditions in order, with a brief description after each image.The sequence begins from a photograph and requires descriptions after the generated images.
- Ordered Transformations: Progressive-morphing prompts require four PNG stages transitioning naturally from an initial image to a final desired state, with descriptions for each step.The prompt emphasizes natural, progressively illustrative transitions between states.
- Viewpoint and Procedures: Other prompts generate multiple object views from specified angles, decompose images, and produce four illustrated procedural steps with accompanying descriptions.The viewpoint task specifies 60 degrees left, 30 degrees left, 30 degrees right, and 60 degrees right; the procedural example concerns refrigerator dill pickles.
- Storytelling: Storytelling prompts continue an existing beach scene through four new images, requiring connecting phrases and descriptions for smooth transitions.The task begins with the text “Children are playing at the beach.” and asks the model to continue the story.
- Question Answering: Historical VQA prompts require a textual answer followed by three interleaved text-image sections, with each image matching preceding text and the final block being an image.The format explicitly separates every image with text content.