Source-linked AI summary
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, Mike Zheng Shou
TL;DR
Multiple concept LoRAs are difficult to combine because decentralized customization can produce concept conflicts and identity loss. Mix-of-Show addresses this with ED-LoRA, gradient fusion, and regionally controllable sampling, and experiments show high-fidelity compositions of customized characters, objects, and scenes.
Problem
Jointly supporting multiple customized concepts with separately trained LoRAs remains challenging because concept conflicts and identity loss arise during tuning and fusion.
Method
Mix-of-Show uses ED-LoRA for single-client tuning, gradient fusion for center-node fusion, and regional-aware cross-attention for controllable multi-concept sampling.
Results
Experiments demonstrate high-fidelity composition of multiple customized characters, objects, and scenes, with identity loss decreasing from 0.094 to 0.025 under gradient fusion.
Takeaways & Limitations
Mix-of-Show supports theoretically limitless customized-concept fusion while addressing identity preservation and attribute binding within the reported setting.
Takeaways & Limitations
The decentralized setting focuses on merging single-concept models for scalability and reusability rather than co-training multiple concepts.
Abstract
from arXiv · showhide
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new concepts using low-rank adaptations (LoRAs). However, the utilization of multiple concept LoRAs to jointly support multiple customized concepts presents a challenge. We refer to this scenario as decentralized multi-concept customization, which involves single-client concept tuning and center-node concept fusion. In this paper, we propose a new framework called Mix-of-Show that addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.
1 Introduction
Mix-of-Show targets decentralized multi-concept customization, where individually tuned concept LoRAs must be fused and jointly sampled without concept conflicts or identity loss. It combines ED-LoRA, gradient fusion, and regionally controllable sampling to compose customized characters, objects, and scenes.
- 1 Introduction: Mix-of-Show supports complex compositions of multiple customized concepts, including characters, objects, and scenes.The framework is illustrated with compositions such as Harry Potter and Thanos in one image.
- 1 Introduction: Decentralized multi-concept customization combines private single-client concept tuning with center-node fusion of shared LoRA models.Clients retain private concept data, while the center node updates the pretrained model for joint sampling.
- 1 Introduction: Existing LoRA tuning and weight fusion face concept conflict and identity loss when multiple customized concepts are combined.Concept conflict reflects insufficient separation of embedding and LoRA-weight roles, while fusion can damage individual concept identity.
- 1 Introduction: Mix-of-Show uses embedding-decomposed LoRA for client tuning and gradient fusion for center-node concept fusion.ED-LoRA preserves more in-domain concept essence in expressive, layer-wise, multi-word embeddings; gradient fusion aligns individual concept inference behavior.
- 1 Introduction: Regionally controllable sampling injects region prompts through regional-aware cross-attention to address missing objects and attribute binding.It extends spatial control by using predefined spatial regions to guide concept-specific generation.
2 Related Work
Related work covers concept tuning, multi-concept customization, fast test-time customization, federated model fusion, and spatially controlled generation. Existing multi-concept approaches commonly require joint training or support only a small number of concepts.
- 2 Related Work: Concept customization extends pretrained diffusion models using few images through embedding tuning, joint embedding-weight tuning, or lightweight LoRA adaptation.LoRA can achieve comparable fidelity to full weight tuning while remaining lightweight.
- 2 Related Work: Existing multi-concept methods include co-training, constrained optimization, data augmentation, and concept-neuron addition.These approaches aim to prevent concept mixing or add multiple concepts to a pretrained model.
- 2 Related Work: Prior multi-concept methods are typically restricted to fusing 2-3 semantically distinct concepts, whereas Mix-of-Show targets theoretically limitless customized concepts.The claimed scope includes concepts within the same semantic category.
- 2 Related Work: Fast test-time customization methods pretrain category-specific encoders that extract features from a few representative concept images during inference.These features complement pretrained diffusion models for customized generation.
- 2 Related Work: Federated learning commonly uses FedAvg to average client model weights, but direct weight averaging is not ideal for fusing LoRAs of different concepts.This motivates alternatives that improve local training or fusion procedures.
- 2 Related Work: Text-only multi-concept generation suffers from missing objects and attribute binding, while spatial-control methods improve composition but do not fully solve concept binding.ControlNet and T2I-Adapter provide spatial hints such as keyposes and sketches.
3 Methods
Mix-of-Show addresses decentralized multi-concept customization with ED-LoRA for single-client tuning, gradient fusion for center-node fusion, and regionally controllable sampling for multi-concept generation.
- Task Formulation: Decentralized Multi-Concept Customization: Decentralized multi-concept customization combines private single-client concept tuning with center-node fusion of shared LoRAs.The center node updates the pretrained model using the gathered concept LoRAs.
- Preliminary: Embeddings capture in-domain concepts, whereas LoRA weights capture out-domain information such as styles and details absent from the pretrained model.Embedding tuning can encode in-domain identity, while LoRA weight shifts model out-domain information into the adapted model.
- Single-Client Concept Tuning: ED-LoRA: Existing LoRA tuning creates concept conflicts because LoRA weights encode most concept identity and map similar embeddings to visually distinct concepts.When multiple semantically similar concepts are fused, the model cannot reliably determine which concept to sample, causing individual identity loss.
- Single-Client Concept Tuning: ED-LoRA: ED-LoRA preserves more in-domain essence in embeddings through layer-wise and multi-word representations while using LoRA weights for remaining details.The method is designed to reduce concept conflict by decomposing the concept embedding and separating in-domain essence from other details.
- Center-Node Concept Fusion: Gradient Fusion: Weight fusion diminishes individual concept identity, whereas gradient fusion aligns single-concept inference behavior and improves identity preservation and consistent stylization.Gradient fusion decodes concepts with their respective LoRAs, extracts activations, and fuses LoRAs without accessing private concept data.
- Regionally Controllable Sampling: Regionally controllable sampling assigns subjects and attributes to specified regions, addressing missing objects and attribute binding in multi-concept generation.The method replaces global output features with regional features while maintaining a harmonious global context.
4 Experiments
Experiments evaluate Mix-of-Show for single-concept customization, multi-concept fusion, and regionally controllable sampling. Results indicate that ED-LoRA and gradient fusion preserve concept identity, while regional control improves attribute binding.
- Single-Concept Results: ED-LoRA achieves comparable object-customization performance to prior methods while maintaining better character identity.
- Multi-Concept Results: Mix-of-Show preserves individual concept identity after fusion more effectively than LoRA, which loses identity during weight fusion.
- Quantitative Comparison: Mix-of-Show and LoRA achieve superior image alignment with comparable text alignment in single-client tuned models.The authors attribute this to fine-tuning spatial-related layers in the U-Net, which helps capture complex concept identities.
- Embedding Expressiveness: Replacing LoRA embeddings with ED-LoRA reduces identity loss from 0.115 to 0.094 after weight fusion.Identity loss is measured by mean change of image alignment.
- Fusion Type: Gradient fusion reduces identity loss from 0.094 to 0.025 compared with weight fusion using the same ED-LoRAs.Human evaluation also reports a clear preference for gradient fusion.
- Regionally Controllable Sampling: Regionally controllable sampling overcomes attribute-binding problems that occur when directly sampling multiple concepts.The method produces correct attribute binding in multi-concept generation.
5 Conclusion
The conclusion frames decentralized multi-concept customization as limited by concept conflicts and identity loss, and presents Mix-of-Show as a framework addressing these challenges. Experiments show high-fidelity composition across characters, objects, and scenes.
- Mix-of-Show combines ED-LoRA for single-client tuning, gradient fusion for center-node fusion, and regional sampling for attribute binding.
- Experiments demonstrate successful high-fidelity generation of complex compositions involving customized characters, objects, and scenes.
6.1 Dataset and Implementation Details
The experiments use a dataset spanning real-world and anime characters, objects, and scenes, with specified pretrained models and optimization settings. Implementation includes ED-LoRA tuning, center-node fusion, and standardized sampling.
- Dataset: The curated dataset contains 19 concepts: 6 real-world characters, 5 anime characters, 6 real-world objects, and 2 real-world scenes.
- Pretrained Models: Chilloutmix2 is used for real-world concepts and Anything-v43 for anime concepts because of Stable-Diffusion v1-5 face-quality issues.All comparison methods use the same pretrained model for fair comparison.
- Single-Client Concept Tuning: ED-LoRA adds rank r = 4 LoRA layers to attention linear layers in the text encoder and U-Net.Adam learning rates are 1e-3 for embeddings, 1e-5 for the text encoder, and 1e-4 for the U-Net.
- Center-Node Concept Fusion: Center-node fusion optimizes LoRA-connected layers initialized from pretrained weights using LBFGS.Optimization uses 500 steps for the text encoder and 50 steps for the U-Net.
- Sampling: All experiments use DPM-Solver with 20 sampling steps and the same negative prompt for Mix-of-Show and comparison methods.
- Running Times: Single-client tuning takes approximately 10–20 minutes on two Nvidia-A100 GPUs, while fusing 14 concepts takes 30 minutes on one Nvidia-A100 GPU.
6.2 Quantitative and Qualitative Evaluation
Evaluation measures text and image alignment across single-concept and center-node fused models using varied prompts and qualitative comparisons. Mix-of-Show consistently preserves concept identity better than competing fusion approaches.
- Evaluation Metrics: Evaluation measures text alignment and image alignment for single-concept tuned and multi-concept fused models.Text alignment uses CLIP text-image similarity, while image alignment evaluates concept correspondence.
- Evaluation Prompts: Each concept is evaluated with 20 prompts covering recontextualization, restylization, interaction, and property modification.Each prompt generates 50 images, producing 1000 images per concept with fixed random seeds.
- Evaluation Results: Complete per-concept evaluation results are reported in Table 3, with category summaries in Table 1 of the main paper.
- Qualitative Comparison: Qualitative comparisons show Mix-of-Show best preserves concept identity after fusion, while P+ and Custom Diffusion sometimes produce oversaturation or semantic collapse.LoRA exhibits the most significant concept-identity loss after fusion.
6.3 Limitation and Future Work
Mix-of-Show has limitations in regional attribute isolation, center-node fusion time, and small-face detail generation. Future work considers broader composition scenarios, while the framework also raises misuse concerns.
- Limitations: Attributes from one region may influence another because some attributes are encoded in the global embedding.Region-specific negative prompts can partially alleviate this leakage.
- Limitations: Center-node concept fusion requires relatively lengthy optimization, primarily because Unet layers contain large spatial features.
- Limitations: Stable Diffusion may lose facial details in small face regions because of information loss in its VAE.Increasing the sample size is suggested as a potential mitigation.
- Future Work: Future work could apply Mix-of-Show to storybook generation involving character-object interactions across plots and to video or 3D concept customization.The latter directions are motivated by the framework's stable identity encoding.
- Potential Negative Society Impact: Malicious entities could exploit decentralized concept composition to create deceptive interactions with real-world figures and mislead the public.Suggested mitigations include anti-dreambooth-style perturbations and unseen watermarking.