Source-linked AI summary
LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation
Zeyu Wang, Zilong Chen, Chenhui Gou, Feng Li, Chaorui Deng, Deyao Zhu, Kunchang Li, Weihao Yu, Haoqin Tu, Haoqi Fan, Cihang Xie
TL;DR
Unified multimodal models can be costly to train, and efficient connector-based approaches remain limited in performance and task generality. LightFusion fuses pretrained understanding and generation models with interleaved multimodal attention, reporting strong results across generation and editing benchmarks after training on ∼35B tokens.
Problem
Efficient unified multimodal approaches using lightweight VLM-to-DiT connectors remain limited in performance and task generality.
Source: §1 · p. 1Method
LightFusion interleaves zero-initialized multimodal self-attention across pretrained VLM and DiT blocks, retaining coordinated understanding and generation pathways.
Source: §1 · p. 2, §3.1 · p. 3Results
LightFusion reports strong results across text-to-image generation and image editing benchmarks after training on only ∼35B seen tokens.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that Double Fusion enables cross-modal feature interaction and integrates high-level semantics with low-level visual details while requiring fewer training tokens and compute than prior leading UMMs.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Unified multimodal models have recently shown remarkable gains in both capability and versatility, yet most leading systems are still trained from scratch and require substantial computational resources. In this paper, we show that competitive performance can be obtained far more efficiently by strategically fusing publicly available models specialized for either generation or understanding. Our key design is to retain the original blocks while additionally interleaving multimodal self-attention blocks throughout the networks. This double fusion mechanism (1) effectively enables rich multi-modal fusion while largely preserving the original strengths of the base models, and (2) catalyzes synergistic fusion of high-level semantic representations from the understanding encoder with low-level spatial signals from the generation encoder. By training with only ~ 35B tokens, this approach achieves strong results across multiple benchmarks: 0.91 on GenEval for compositional text-to-image generation, 82.16 on DPG-Bench for complex text-to-image generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By fully releasing the entire suite of code, model weights, and datasets, we hope to support future research on unified multimodal modeling.
1 INTRODUCTION
Unified multimodal models combine understanding and generation, but efficient open-source construction remains challenging: shallow connectors have limited performance and task generality, while leading approaches require substantial compute. LightFusion uses Double Fusion to connect specialized pathways and reports strong benchmark results with substantially fewer training tokens.
1 INTRODUCTION
- Shallow connectors from pretrained VLMs to diffusion transformers remain limited in performance and task generality.Earlier unified models also face optimization conflicts with single-stack training or substantial compute demands with separate pathways.Source: §1 · p. 1
- Double Fusion tightly couples VLM and DiT blocks through zero-initialized multimodal self-attention while retaining the base models’ autoregressive and diffusion capabilities.Coordinated pathways bridge understanding and generation as well as ViT- and VAE-based representations.Source: §1 · p. 2
- The paper reports that this design substantially outperforms shallow fusion and integrates high-level semantics with low-level visual signals for more accurate, consistent, and faithful image editing.Source: §1 · p. 2
- 0.91 on GenEval, 82.16 on DPG-Bench, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench are reported after training on only ∼35B seen tokens.The paper reports performance on par with or surpassing UniPiC and OmniGen2, which used orders of magnitude more tokens.Source: §1 · p. 2
2 RELATED WORKS
Prior work spans text-to-image generation, image editing, and unified multimodal models, with a recurring trade-off between cross-task capability and training demands. LightFusion builds on separate understanding and generation pathways while using deeper fusion and publicly available models and data.
Text-to-Image Generation
- Text-to-image systems moved from diffusion-based latent denoising to flow matching as an alternative for modeling vector fields between noise and data distributions.The passage cites Stable Diffusion and successors [Podell et al., 2023], DALL·E [Ramesh et al., 2022], and Imagen [Imagen 3 Team, 2024] as examples of large-scale pretraining and latent denoising.Source: §2 · pp. 2–3
Image Editing
- Image-editing research progressed from supervised instruction-source-target triplets in InstructPix2Pix to methods diversifying data or adding stronger control signals.OmniGen and UniReal process varied conditional inputs to unify common image-generation tasks.Source: §2 · p. 3
Unified Multimodal Models
- Early unified multimodal models used a single Transformer over interleaved image and text tokens, while later separate-branch designs eased optimization and improved performance but still relied on extensive pretraining.The paper notes that this scale constrains accessibility.Source: §2 · p. 3
- Efficient schemes coupling frozen multimodal LLMs to trainable diffusion decoders commonly use shallow or late-stage fusion, which the paper says restricts cross-modal alignment.LightFusion instead couples its VLM and DiT through double fusion.Source: §2 · p. 3
- LightFusion integrates high-level ViT features with low-level VAE latents in a dual-pathway design intended to produce more coherent, semantically grounded generation.Source: §2 · p. 3
- Unlike LMFusion and BAGEL, LightFusion uses publicly available fused base models and public data, with the paper describing its training regime as lightweight and accessible.The cited comparison says LMFusion and BAGEL require orders of magnitude more seen tokens and extensive private training data.Source: §2 · p. 3
3 METHODS
LightFusion interleaves multimodal self-attention across pretrained understanding and generation pathways, using public data and staged training. Its architecture combines text and ViT tokens with VAE latents while preserving the underlying model blocks.
3.1 MODEL ARCHITECTURE
- Multimodal self-attention blocks interleaved across VLM and DiT layers let text, ViT, and VAE tokens interact, while zero initialization preserves the base pathways’ starting feature distributions.The design uses QWen2.5-VL-7B and Wan2.2-TI2V-5B; the VLM’s final output is reused for the last two attention blocks because it has two fewer layers.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
- The architecture keeps pretrained VLM and DiT models intact, and combines high-level ViT semantics with low-level VAE signals for image editing.The authors also describe using hidden states across understanding and generation layers rather than compressing conditioning into a fixed-length representation.Source: §3.1 · p. 4
3.2 DATASET
- The training corpus comprises ∼45 million public and synthetic samples for text-to-image generation and image editing.It includes ∼4.5 million self-curated synthetic samples, and a VLM refines public editing instructions using source-target pairs.Source: §3.2 · p. 4
3.3 TRAINING
- Table 1 compares models across understanding, generation, editing, and in-context generation tasks.Source: Table 1 · p. 5
- Training freezes the understanding branch and uses three stages that progressively increase high-quality text-to-image and editing data.The passage says this staged setup benefits both text-to-image generation and image-editing performance; the branch is frozen to preserve QWen2.5-VL-7B’s understanding ability.Source: §3.3 · p. 5
4 EXPERIMENTS
LightFusion is evaluated on visual understanding, text-to-image generation, image editing, and design ablations. The experiments report competitive benchmark performance and compare fusion and training choices.
4.1 VISUAL UNDERSTANDING
- Freezing the understanding branch preserves QWen2.5-VL-7B reasoning, with scores of 83.5 on MMBench, 58.6 on MMMU, and 67.1 on MM-Vet.The paper also reports that LightFusion surpasses Ovis-U1, Show-o2, and Janus-Pro.Source: §4.1 · pp. 5–6
4.2 TEXT-TO-IMAGE GENERATION
- 0.91 on GenEval and 82.16 on DPG-Bench are reported for text-to-image generation.The GenEval result uses LLM-rewritten prompts; the DPG-Bench result is reported as exceeding BLIP3-o 8B (81.60) and UniWorld-V1 (81.38).Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
4.3 IMAGE EDITING
- 6.06 on GEdit-Bench-EN exceeds UniWorld-V1's 4.93, while 3.77 on ImgEdit-Bench exceeds UniWorld-V1's 3.26 and OmniGen2's 3.44.LightFusion's 6.34 semantic-consistency score is also reported on GEdit-Bench-EN.Source: §4.3 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 9
4.4 ABLATION STUDY
- 0%-depth fusion consistently outperforms shallow or early-layer fusion on text-to-image and image-editing tasks in a comparison with fixed parameter counts.At 0% depth, each DiT block is conditioned on the corresponding VLM block's output; Figure 5 presents the comparison.Source: Figure 5 · p. 9, §4.4 · p. 9
- Combining VAE low-level details with ViT high-level semantics yields more consistent and accurate image editing than either information source alone.The visual-tokenizer ablation reports that both sources are essential.Source: §4.4 · p. 9
- Shifts 2 and 4 outperform shift 1: DPG-Bench scores are 78.84 and 81.77 versus 76.67, and ImgEdit scores are 3.36 and 3.57 versus 3.07.The ablation reports better results for timestep shifts larger than 1.Source: Table 6 · p. 10, §4.4 · p. 10
5 CONCLUSION
LightFusion uses Double Fusion to support unified understanding and generation, reporting state-of-the-art performance across diverse tasks with substantially fewer training tokens and compute than prior leading UMMs.
5 CONCLUSION
- LightFusion reports state-of-the-art performance across diverse tasks while using substantially fewer training tokens and compute than prior leading UMMs.Source: §5 · p. 10
- Double Fusion enables effective cross-modal feature interactions and integrates high-level semantics with low-level visual details.Source: §5 · p. 10
- The authors release the code, models, and data to support reproducibility and accelerate progress in unified multimodal modeling.Source: §5 · p. 10