Source-linked AI summary
Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation
Shuhong Zheng, Aashish Kumar Misraa, Yu-Teng Li, Yu-Jhe Li, Igor Gilitschenski
TL;DR
Subject-driven generation must preserve subject identity while following text, but separate encoders limit multimodal reasoning and can produce copy-paste artifacts. The paper combines MLLM and VAE conditioning with DLA and multi-stage denoising, and reports harmonized multimodal understanding, identity preservation, and reduced copy-paste issues. Its evaluation scope remains limited for multi-subject scenarios and MLLM-DiT text-space alignment requires substantial resources and data for further improvement.
Problem
Separate text and image encoders limit multimodal reasoning, while MLLM-diffusion systems largely overlook identity preservation in subject-driven generation.
Method
The method jointly conditions diffusion on MLLM text-image features and VAE identity features, using modality-specific DLA aggregation and multi-stage denoising.
Results
The approach harmonizes multimodal understanding with identity preservation, mitigates copy-paste issues, and achieves superior human preference performance.
Takeaways & Limitations
Aggregating modality-specific information across all MLLM layers and combining it with VAE details supports flexible subject-driven generation.
Takeaways & Limitations
Multi-subject evaluation is limited by scarce high-quality multi-subject data, while stronger MLLM-DiT alignment would require substantial computation and text-to-image data.
Abstract
from arXiv · showhide
Subject-driven image generation aims to synthesize new images that preserve the identity of the given subject while following textual instructions. Existing approaches often encode text and reference images separately. This limits cross-modal reasoning abilities and causes copy-paste artifacts. Recent frameworks that connect multimodal models and diffusion models improve instruction following, but largely overlook identity preservation. To address these limitations, we condition diffusion models on Multimodal Large Language Models (MLLMs) that jointly encode text and reference images, and augment it with VAE-based identity conditioning. A novel Dual Layer Aggregation (DLA) module is designed to aggregate multi-level MLLM features for optimal conditioning, and a multi-stage denoising strategy is applied to progressively balance the semantic information from MLLM and fine-detail identity from VAE during inference. Extensive experiments demonstrate that our approach harmonizes multimodal understanding with identity preservation, mitigates copy-paste issues, and achieves superior performance regarding human preference on subject-driven image generation. Our project website is available at https://zsh2000.github.io/squeeze-mllm-subject-gen/.
1 Introduction
The paper unifies MLLM-based multimodal reasoning with VAE-based identity preservation for subject-driven generation. DLA aggregates multimodal features across MLLM layers, while staged denoising balances semantic conditioning and fine identity details.
- Existing diffusion methods often encode text and reference images separately, limiting multimodal reasoning and contributing to copy-paste artifacts.
- The proposed framework jointly encodes text and reference images in an MLLM space while adding VAE conditioning to preserve subject identity.
- DLA separately aggregates text and visual embeddings across all MLLM transformer layers instead of relying only on final-layer features.
- A multi-stage denoising strategy first uses MLLM guidance for global semantics, then combines MLLM and VAE signals, and finally emphasizes VAE-conditioned details.
2 Related Work
Subject-driven generation has progressed from per-subject fine-tuning to reference-image conditioning, but identity-focused pipelines remain weakly connected to multimodal reasoning. MLLM-integrated diffusion systems improve prompt interpretation while introducing a complementary route for joint text-image understanding.
- Figure 1 contrasts MLLMs’ joint text-image modeling with VAE-based methods’ separate encoding of input image and text.
- DreamBooth and Textual Inversion achieve strong identity fidelity through per-subject fine-tuning, but this limits scalability.
- Reference encoders and VAE-based token conditioning avoid retraining by extracting subject identity features from input images during inference.
- MLLM-integrated diffusion systems parse interleaved multimodal instructions by jointly reasoning over text and images.
3 Method
The method combines a DiT diffusion backbone with MLLM and VAE conditioning. LAP and DLA align multi-level modality-specific features, while timestep-aware denoising shifts from semantic guidance to identity refinement.
- 3 Method: The framework maps text and reference images through MLLM and VAE encoders before conditioning a DiT diffusion backbone.
- 3.2 Basic Module: Layerwise Attention Pooling: LAP summarizes MLLM feature maps across transformer layers through attention over the layer axis and adaptive layer weighting.
- 3.2 Basic Module: Layerwise Attention Pooling: Single LAP creates an identity-preservation and text-alignment tradeoff because text and image tokens exhibit different layerwise attention patterns.
- 3.3 Dual Layer Aggregator: DLA uses separate LAP modules for text and image tokens, emphasizing semantic fidelity for text and appearance and identity consistency for images.
- 3.3 Dual Layer Aggregator: DLA preserves cross-modal interaction because MLLM intermediate layers already transfer information between image and text tokens.
- 3.4 Multi-stage Timestep-aware Denoising: The VAE captures fine-grained identity details, whereas MLLMs provide stronger reasoning and layout understanding but weaker identity fidelity.
- 3.4 Multi-stage Timestep-aware Denoising: The denoising network uses timestep-dependent masks to activate MLLM conditioning early, both branches in the middle, and VAE conditioning late.
- 3.5 Two-stage Training Strategy: Figure 4 compares methods on subject identity, pose variation, and physical constraints, highlighting copy-paste failures in competing approaches.
4 Experiments
Experiments evaluate the method across standard benchmarks, copy-paste reduction, multimodal reasoning, human-aligned scores, user preference, and training or feature-connection ablations.
- Standard Benchmark Performance: The evaluation covers standard subject-driven generation benchmarks and comparisons with VAE-based, MLLM-based, and other recent methods.The benchmark comparison includes OminiControl, OmniGen2, UNO, XVerse, DreamO, USO, UMO, DreamEngine, Qwen-Image, and EasyRef, alongside a re-trained UNO baseline using UNO-1M.
- Copy-paste Issue Alleviation: The proposed evaluation measures copy-paste artifacts through azimuth and polar orientation differences between reference and generated subjects.It reports Recall@k° for k° ∈ {5°, 10°, 15°, 20°} and their average recall rate.
- Reasoning Capability: The method is evaluated on a 350-sample benchmark designed to measure multimodal reasoning in subject-driven generation.The reasoning evaluation targets prompts requiring joint interpretation of text and visual input, including concept binding between prompt terms and image regions.
- Human-aligned Evaluation and User Study: The experiments additionally assess human-aligned subject consistency on DreamBench++ and overall quality through a user study.The user study uses 30 volunteers, 1,500 votes, and samples from DreamBench and XVerseBench; its results report greater subjective preference for the proposed method.
- Ablation Study: Ablations examine MLLM-to-DiT feature-connection strategies and compare single-stage training with timestep-aware denoising.The feature studies include last-layer conditioning, feature blending, feature mixing, and single-LAP configurations; the training comparison highlights the two-stage strategy.
5 Conclusion
The conclusion identifies multi-level, modality-specific MLLM aggregation and coordinated MLLM–VAE denoising and training as central to balancing multimodal understanding with identity preservation.
- 5 Conclusion: Aggregating representations across all MLLM layers while aligning text and visual modalities separately is critical for multimodal understanding and identity preservation.The framework combines this aggregation with VAE-based fine-grained visual detail.
- 5 Conclusion: Multi-stage denoising and two-stage training harmonize MLLM conditioning with VAE identity details and provide users with more generation flexibility.The conclusion presents these mechanisms as complementary to the DLA-based conditioning design.
Technical Appendices and Supplementary Material
The supplementary material extends the paper with analyses of DLA layer contributions, layer-selection trade-offs, and additional related-work discussion.
- Technical Appendices and Supplementary Material: The appendix analyzes DLA layer contributions during inference and studies layer-selection strategies for training in terms of efficiency and performance trade-offs.It also discusses extended related work, including the development of text-to-
A Layer Analysis for DLA at Inference Time
Layer ablations show different modality sensitivities: early image layers support identity details, while later text layers are more important for prompt understanding.
- A Layer Analysis for DLA at Inference Time: Removing early MLLM image layers harms fine-grained visual detail and identity preservation, whereas using only early image layers can retain comparable identity consistency.The qualitative analysis specifically reports degradation when layers 0–19 are zeroed.
- A Layer Analysis for DLA at Inference Time: Text features are more robust to layer removal overall, but later text layers are critical for understanding and following prompts.Using only text layers 0–9 significantly weakens prompt comprehension and adherence.
- A Layer Analysis for DLA at Inference Time: Occasionally disabling one modality slightly improves performance when the other modality is partially removed, reinforcing the importance of balancing both modalities.This observation is reported from the layer-removal analysis rather than as a general training recommendation.
B Layer Selection for Training DLA
Training-time layer selection in DLA exposes a trade-off between identity preservation and text following. Restricting text layers is especially harmful, while some subset selections can improve identity metrics at the expense of semantic control.
- Pre-selecting early MLLM layers (0-9) improves identity metrics but reduces text-following performance.The analysis associates this pattern with increased reliance on identity images and copy-paste behavior.
- Almost all pre-selection strategies for the text modality degrade performance because textual information is distributed across all MLLM layers.
- Training with selected layer subsets can still produce comparable or improved performance for both text and image modalities.This suggests that some layers may be redundant while others are more informative depending on context.
- Figure D compares 16 separately retrained combinations of text and image layer ranges, including 0-9, 10-19, 20-28, and all layers (0-28).Rows represent text-modality settings and columns represent image-modality settings.
- Pre-selecting layers during training can increase reliance on identity images and worsen text following, especially when fewer than all 28 text layers are used.The comparison highlights a trade-off between efficiency and balanced identity preservation and multimodal understanding.
D Additional Sensitivity Analysis
The multi-stage denoising schedule uses MLLM features for early semantic reasoning and VAE features for later identity detail. Its thresholds expose a controllable trade-off between identity fidelity and pose diversity, while performance remains robust across settings.
- Early denoising steps use MLLM features for high-level reasoning, whereas later steps use VAE features for fine-grained identity detail.
- Higher denoising thresholds improve identity preservation but reduce pose diversity, while lower thresholds enable more creative poses with slightly weaker identity.
- Extreme CFG values can degrade image quality.
- The thresholds τ1 and τ2 and the CFG value control the trade-off between identity preservation and text following, with robust performance across parameter settings.
E Ablation on Different MLLM Selection
Across MLLM backbones, the framework maintains similar generation behavior and quality, with only minor quantitative variation. Larger models show slightly stronger grounding and semantic alignment, while the overall backbone choice has limited impact.
- InternVL3-8B, InternVL3-2B, Qwen2.5-VL-3B, and Qwen2.5-VL-7B exhibit similar performance trends across metrics with only minor variations.
- All four MLLM backbones produce comparable image fidelity, identity preservation, and text-following ability.
- Larger MLLM backbones show slightly stronger grounding and semantic alignment, but the overall performance trend remains consistent.
- The framework generalizes across different MLLM architectures, indicating that backbone choice has limited impact on final generation quality.
- The multi-subject extension achieves superior identity preservation and text compliance on multi-subject DreamBench samples compared with UNO, DreamO, and UMO.
- The constructed multimodal reasoning benchmark tests concept binding by asking models to locate a text-mentioned, non-salient object rather than assuming the most salient reference subject.
J Evaluation on More Benchmarks
Additional benchmarks and stress tests assess the method beyond the main evaluation. The reported results describe decent benchmark performance, competitive identity with varied poses, and tests involving multiple subject instances and long-context attribute binding.
- On XVerseBench and LAMICBench, the method is evaluated on single-subject and two-reference subsets using a model fine-tuned on two-subject data.
- The additional-benchmark results demonstrate decent performance on the subject categories evaluated.
- Stress tests consider prompts with multiple subject instances and attribute binding in long-context prompts.These settings require simultaneous identity preservation and satisfaction of multiple textual constraints.
- Qualitative comparisons report competitive subject identity with diverse pose variations while alleviating copy-paste issues associated with other VAE-based models.
M Discussions and Limitations
The framework’s main limitations concern text-space alignment and limited multi-subject evaluation. The paper also notes potential societal risks, while reporting promising qualitative performance and adaptation beyond the primary single-subject setting.
- Alignment limitation: Alignment between the MLLM text representation space and the DiT conditioning space remains a limitation of the current framework.The DiT conditioning space was originally designed for the T5 encoder, creating an alignment challenge for the MLLM-DiT system.
- Evaluation scope: Multi-subject evaluation is limited, partly because high-quality multi-subject data collections are scarce.The authors demonstrate adaptation to multi-subject scenarios but discuss them less extensively than single-subject generation.
- Societal impact: The authors identify digital forgery, copyright, and ethical concerns as potential negative societal impacts of the technique.These risks are presented alongside possible unintended or inappropriate use.
- Multi-subject adaptation: After lightweight fine-tuning on MUSAR-Gen, the model adapts to multi-subject scenarios with clearer identity separation, consistent posture, and more reliable concept binding.The comparison is reported against UNO, DreamO, and UMO.
- Qualitative results: Qualitative comparisons report better identity preservation and text alignment than several state-of-the-art subject-driven generation methods across diverse prompts.The comparisons include XVerse, EasyRef, DreamO, UMO, OminiControl, UNO, Qwen-Image, OmniGen2, and USO.