Source-linked AI summary
XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
Bowen Chen, Mengyi Zhao, Haomiao Sun, Li Chen, Xu Wang, Kang Du, Xinglong Wu
TL;DR
Multi-subject text-to-image personalization must preserve individual identities and semantic attributes without sacrificing DiT editability or coherence. XVerse addresses this by converting reference images into token-specific text-stream modulation offsets and using VAE features for detail enhancement, reporting strong multi-subject control, semantic generalization, and editability. The paper also identifies limited large-scale cross-image multi-subject data and unexplored image-modulation pathways as limitations.
Problem
Multi-subject generation requires preserving individual identity and semantic attributes while avoiding artifacts, distortions, attribute entanglement, and degraded structural coherence from direct image-feature injection.
Method
XVerse converts reference images into shared and per-block offsets for token-specific DiT text-stream modulation, with VAE-encoded features supporting fine-grained details.
Results
XVerse achieves outstanding multi-subject controlled generation performance, strong generalization for posture, lighting, and background, and high reference consistency while preserving editability.
Takeaways & Limitations
Modulation-offset conditioning enables per-subject control while maintaining image composition and reducing common artifacts and attribute entanglement.
Takeaways & Limitations
XVerse is limited by the lack of high-quality large-scale cross-image multi-subject datasets and by its primary focus on text-stream rather than image-modulation control.
Abstract
from arXiv · showhide
Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.
1 Introduction
XVerse addresses the challenges of preserving individual identities and reducing attribute entanglement in multi-subject generation by controlling DiT text-stream modulation. It combines reference-image modulation offsets with supporting VAE features, while reporting strong controllability, editability, semantic generalization, and benchmark coverage.
- Multi-subject personalization must preserve individual identity fidelity while alleviating attribute entanglement in coherent scenes.
- Direct reference-image injection into DiT attention can cause artifacts, distortions, attribute entanglement, and reduced structural coherence.
- XVerse transforms reference images into shared and per-block offsets for token-specific text-stream modulation, preserving the image’s underlying structural integrity.VAE-encoded features are used as supporting detail inputs rather than the main conditioning factor.
- XVerse is presented as a framework for fine-grained multi-subject controllable generation that maintains reference consistency and the original diffusion model’s editability.Its design maintains image composition while integrating reference images into modulation offsets.
- XVerse achieves outstanding multi-subject generation performance and generalizes to preserving semantic information such as posture, lighting, and background.
- XVerseBench evaluates single-subject and multi-subject controlled generation, including flexible editing, subject characteristics, and distinct identity preservation.
2 Related Work
Related work spans fine-tuning and tuning-free approaches for single-subject personalization, alongside modulation mechanisms that improve generative controllability. XVerse’s framework overview places reference-image processing within a modulation-based pipeline supplemented by VAE features.
- Fine-tuning methods such as Textual Inversion and DreamBooth adapt pretrained models to embed novel concepts from a few single-subject exemplar images.
- Tuning-free methods including IP-Adapter and Photoverse use vision encoders to inject subject identity directly.
- Modulation dynamically adjusts internal activations or parameters using conditioning information, with StyleGAN introducing AdaIN-based feature modulation through a style vector.
- Figure 2 depicts reference images processed by a T-Mod Resampler and injected into a per-token modulation adapter, while VAE-encoded features enter a single DiT block.
3 Method
XVerse uses text-stream modulation to integrate reference-image information while preserving the base model’s structure and editability. It supplements this control with constrained VAE features and regularization for detailed, disentangled multi-subject generation.
- Text-stream modulation: DiT modulation provides a separate pathway for integrating visual features into specific words without interfering with the denoising process.This separation supports fine-grained control over feature representations while preserving semantic directionality from text-based features.
- VAE feature integration: XVerse injects VAE-encoded image features only as an auxiliary detail-preservation module in selected FLUX blocks, limiting direct feature injection.The design preserves the global structure and key characteristics established by text-stream modulation while supplementing fine-grained features.
- Text-stream modulation: XVerse combines text and conditioning-image features through a T-Mod Adapter that produces token-specific offsets for refining the conditioning signal.The image is encoded with CLIP, resampled with text features, and added to the original modulation signal.
- Text-stream modulation: The image-derived offset is decomposed into a shared component and block-specific components, enabling adaptive control over text conditioning across DiT blocks.This structured decomposition gives different Transformer blocks customized conditioning rather than relying on one global offset.
- Regularization: Region preservation regularization trains the model to retain the text-to-image branch’s output in non-modulated regions while applying modulation to only one side of concatenated samples.This procedure regularizes subject-specific features and acts as multi-subject data augmentation for distinguishing and preserving subject characteristics.
4 Experiments
Experiments evaluate XVerse on XVerseBench across single- and multi-subject generation, using quantitative metrics, qualitative comparisons, and ablations. XVerse achieves the highest reported overall and average scores while preserving composition, subject consistency, editability, and semantic-attribute control.
- Experimental Setting: XVerseBench evaluates single- and multi-subject generation with editing, identity, object similarity, and aesthetic metrics.The benchmark includes 20 human identities, 74 objects, 45 animal species or individuals, and 300 prompts covering single-, dual-, and triple-subject settings.
- Quantitative Results: 73.40 Overall: XVerse outperforms all compared methods on the benchmark’s combined evaluation.The comparison includes MS-Diffusion, MIP-Adapter, OmniGen, UNO, OmniGen2, and DreamO.
- Quantitative Results: 76.72 AVG and 79.48 ID-Sim: XVerse leads single-subject generation and preserves condition-image identity strongly.DreamO achieves the best DPG score at 96.93, while XVerse remains competitive in similarity and aesthetic measures.
- Quantitative Results: 73.40 AVG: XVerse leads multi-subject generation, with text-stream modulation offsets supporting conditioning while preserving structural integrity.VAE-derived features are used for detail refinement rather than dominant conditioning, helping mitigate artifacts and distortions in multi-subject scenes.
- Qualitative Comparison: Qualitative comparisons show improved identity-object consistency, object-count accuracy, subject relationships, naturalness, and visual plausibility.The paper attributes these results to adjusting modulation offsets in the text-stream pathway.
- Ablation Studies: Ablations show that modulation offsets personalize reference subjects while preserving the original text-to-image composition, with VAE features adding details supportively.Together, the modules maintain subject consistency and editability in dual- and triple-subject generation.
- Semantic-Attribute Control: XVerse controls lighting, posture, clothing, and artistic style by injecting reference images toward targeted words.The paper reports this semantic-attribute control without extensive training data for specific attribute categories.
5 Conclusion
XVerse provides token-specific DiT modulation offsets for adaptive per-subject conditioning, addressing artifacts and attribute entanglement in controlled generation. The paper identifies limited cross-image multi-subject data and unexplored image-modulation pathways as remaining directions.
- Conclusion: XVerse injects reference-image features through modulation offsets to control token-specific representations within DiT blocks.This enables adaptive, per-subject conditioning during the diffusion process.
- Conclusion: The approach mitigates attribute entanglement and artifacts in both single-subject and multi-subject controlled generation.The conclusion reports particular excellence across both task types.
- Limitations: A major limitation is the lack of high-quality, large-scale cross-image multi-subject datasets for training and evaluation.The authors connect this gap to understanding and generating complex interactions between subjects.
- Limitations: The study focuses mainly on text-stream modulation, leaving image-modulation techniques for pixel-level or region-specific control unexplored.The authors identify this as an opportunity for future research.
A Samples for Training Dataset
The training data covers diverse multi-subject scenes and uses varied human reference-image selections. Randomly selecting facial or full-body references is intended to enhance generalization.
- Training Data Samples: Training examples span human-object interactions, human-animal compositions, and complex multi-person scenes.The dataset is designed around diverse scenarios for multi-subject controlled generation.
- Training Data Samples: Human-centric examples randomly select facial or full-body images as reference inputs.The paper states that this strategy can further enhance generalization performance.
B Impact of Prompt Variation on the Generated Image
Figure 9 shows how changing prompt wording affects subject-controlled generation while keeping the reference image fixed. More detailed or semantically aligned prompts preserve identity and attributes better, whereas major mismatches cause injection failure.
- The visualization compares the fixed reference and initial output with outputs produced under prompt variations highlighted in red.
- Detailed prompt descriptions improve preservation of the reference subject’s identity in the generated image.
- Prompts closely matching the reference can incorporate intricate details while retaining control over subject attributes, including gender changes.
- Large semantic mismatches, such as prompting for a dog or tree stump from a person reference, consistently cause the injection process to fail.
C Comparison of the CLIP-T and DPG scores
The paper contrasts CLIP-T’s broad semantic-alignment measurement with DPG’s focus on interpreting and executing detailed, complex prompts. It therefore presents DPG as more effective for intricate prompts.
- DPG is presented as more effective than CLIP-T for evaluating models on intricate prompts.
- CLIP-T assesses semantic consistency using CLIP image-text embeddings and provides a broad measure of semantic alignment.
- DPG is designed to evaluate a model’s ability to interpret and execute detailed and complex text prompts.
D Illustration of Region Preservation Loss
The region preservation loss regularizes unmodulated areas to match the text-to-image branch, helping reference-image injection remain localized. This supports precise control without altering irrelevant regions.
- Figure 10 illustrates the region preservation loss.
- Region preservation loss enforces L2 consistency between XVerse outputs and the text-to-image branch in unmodulated regions.
- Training samples concatenate two existing samples, merge their captions, and apply modulation randomly to only one side.
- The regularization helps inject reference images into specific areas without affecting irrelevant regions, enabling more precise generation control.
E Ablation Study for Text-Image Attention Loss
The ablation study examines text-image attention loss through generated outputs and attention maps. The qualitative results show that this regularization helps preserve the text-to-image branch’s structural and editable characteristics after modulation injection.
- Figure 11 presents qualitative comparisons for the words “woman,” “coffee cup,” and “pink suit” across methods.
- Removing text-image attention loss reveals its importance for preserving structural and editable characteristics after modulation injection.
- The ablation compares generated outputs and their corresponding attention maps with the regularization excluded.
F Broader Impacts
XVerse is presented as improving fidelity and editability for personalized, complex visual content, with potential applications in creative industries, education, training, and inclusive representation. The authors also acknowledge risks including misinformation, deepfakes, privacy concerns, and bias amplification.
- XVerse is described as enhancing fidelity and editability in multi-subject controllable text-to-image generation.
- The authors identify creative industries as a potential area for personalized and complex visual content creation.
- Potential negative impacts include misinformation, deepfakes, privacy concerns, and amplification of training-data biases.The authors state that future work will explore content detection and ethical guidelines as mitigation strategies.