Source-linked AI summary
Multi-Concept Customization of Text-to-Image Diffusion
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu
TL;DR
Users need to teach text-to-image models new personal concepts from only a few examples and compose those concepts together. Custom Diffusion tunes a small subset of cross-attention parameters with regularization and efficient multi-concept adaptation. It achieves strong single- and multi-concept generation while using 3% of model weights and taking 6 minutes on 2 A100 GPUs, but compositions involving three or more concepts remain challenging.
Problem
Text-to-image models do not reliably reproduce users’ personal concepts, which are absent from large-scale training data, from textual descriptions alone.
Method
Custom Diffusion fine-tunes key and value mappings in cross-attention layers, using regularization images and augmentation to adapt concepts efficiently and prevent forgetting.
Results
3% of model weights and 6 minutes on 2 A100 GPUs support efficient fine-tuning, while the method shows strong single-concept alignment and coherent composition of multiple concepts.
Takeaways & Limitations
The method generates novel variations of customized concepts in new contexts, preserves visual similarity, and coherently composes multiple new concepts.
Takeaways & Limitations
Compositions involving pet dogs and cats remain difficult, and composing three or more concepts together is challenging.
Abstract
from arXiv · showhide
While generative models produce high-quality images of concepts learned from a large-scale database, a user often wishes to synthesize instantiations of their own concepts (for example, their family, pets, or items). Can we teach a model to quickly acquire a new concept, given a few examples? Furthermore, can we compose multiple new concepts together? We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts while enabling fast tuning (~6 minutes). Additionally, we can jointly train for multiple concepts or combine multiple fine-tuned models into one via closed-form constrained optimization. Our fine-tuned model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings. Our method outperforms or performs on par with several baselines and concurrent works in both qualitative and quantitative evaluations while being memory and computationally efficient.
1. Introduction
Text-to-image models generate diverse, high-quality content, but they struggle to faithfully represent users’ personal concepts from text alone. Custom Diffusion addresses few-shot customization and multi-concept composition by tuning a small parameter subset efficiently.
- Motivation: Users’ personal concepts are absent from large-scale training data and difficult to reproduce faithfully through textual descriptions.Examples include family, pets, personal objects, and places.
- Challenges: Few-shot customization must preserve existing concept meanings while avoiding overfitting and reduced sampling variation.Adding “moongate,” for example, may alter the meaning of “moon.”
- Challenges: Composing multiple unseen concepts introduces additional difficulty because the concepts can become mixed.The paper considers compositions such as a pet dog in front of a moongate.
- Approach: Custom Diffusion updates key and value mappings in cross-attention layers and uses real images with similar captions plus augmentation during fine-tuning.The selected conditioning parameters are intended to update the new concept while the regularization images help prevent forgetting.
- Results: 6 minutes on 2 A100 GPUs and 3% of model weights are sufficient for fine-tuning, which is 2−4× faster than concurrent works.Experiments use as few as four training images; the method also composes multiple new concepts efficiently.
2. Related Work
Prior work spans text-conditioned generative modeling and direct editing of generative models. These lines of research motivate representing specific concepts inside pretrained systems while adapting them for user-directed generation and editing.
- Deep generative models: Earlier text-to-image systems were limited to a few classes, whereas recent models are trained at much larger scale.The paper situates Custom Diffusion within text-conditioned synthesis research.
- Image and model editing: Generative-model editing methods address specific images through per-image or per-edit optimization, including approaches that edit the generative model directly.The central challenge is representing the specific image in the pretrained model.
3. Method
Custom Diffusion personalizes a pretrained text-to-image diffusion model by updating a small subset of cross-attention parameters, using regularization to limit overfitting and language drift. It supports single- and multiple-concept customization through joint training or closed-form model merging.
- Single-Concept Fine-tuning: The method represents a personal instance with a learned modifier token placed before its general category name.For example, training can use captions such as “photo of a V∗dog.”
- Single-Concept Fine-tuning: Custom Diffusion updates only cross-attention key and value projection matrices, which are highly responsive during fine-tuning despite comprising 5% of model parameters.The method targets the mappings from text features to latent image features.
- Single-Concept Fine-tuning: Retrieved real images with similar captions form a regularization dataset that reduces overfitting and helps preserve existing concept meanings.Without regularization, fine-tuning can produce moongate-like images for the prompt “photo of a moon.”
- Multiple-Concept Compositional Fine-tuning: Multiple concepts can be trained jointly using distinct modifier tokens while optimizing the shared cross-attention key and value matrices.The combined training datasets contain examples for each individual concept.
- Multiple-Concept Compositional Fine-tuning: Separately fine-tuned concepts can be merged by solving a constrained least-squares objective over target and regularization text features.The optimization preserves mappings for target captions while incorporating values from the individually fine-tuned concept matrices.
- Training Details: The method uses data augmentation during fine-tuning and trains single concepts for 250 steps or two concepts jointly for 500 steps.Augmentation randomly resizes target images and appends prompts describing scale or framing.
4. Experiments
Experiments evaluate Custom Diffusion on single- and multi-concept customization using qualitative samples, alignment metrics, human preferences, and ablations. The method generally matches or exceeds baselines while using substantially less training time and storage.
- Evaluation: Experiments span ten target datasets, single- and two-concept customization, challenging prompts, and image- and text-alignment evaluation.The evaluation includes new scenes, art styles, object composition, and property changes.
- Single-concept results: Custom Diffusion captures target details while maintaining higher text-image alignment than Textual Inversion and performing on par with DreamBooth.It requires approximately 5× less training time and 75MB rather than 3GB of model storage.
- Multi-concept results: Custom Diffusion composes two new concepts coherently, generally outperforming DreamBooth and avoiding the omissions observed with Textual Inversion.Sequential training forgets the first concept, while fine-tuning only cross-attention parameters supports composition.
- Human preference and alignment: In alignment evaluations, Custom Diffusion is on par with or better than baselines despite a trade-off between target-image similarity and text alignment.Both joint training and optimization-based multi-concept methods outperform other baselines.
- Ablations: Removing augmentation lowers visual similarity, while omitting regularization or using generated regularization images produces worse KID and more forgetting.Real-image regularization helps preserve existing concepts and generated-image regularization shows signs of overfitting.
5. Discussion and Limitations
The method supports few-shot customization, novel variations, and coherent composition of multiple concepts. Difficult dog-and-cat compositions and compositions of three or more concepts remain challenging.
- Contributions: Custom Diffusion fine-tunes new concepts, categories, personal objects, or artistic styles from only a few image examples.It generates novel contexts while preserving visual similarity and stores only a small subset of model weights.
- Composition: The method can coherently compose multiple new concepts in the same scene.The conclusion identifies multi-concept composition as a supported capability.
- Limitations: Pet-dog and pet-cat compositions remain difficult, and combining three or more concepts is also challenging.The pretrained model faces similar difficulty for hard compositions, and Custom Diffusion inherits these limitations.
Appendix
The appendix extends the evaluation with a larger custom-concept dataset, derives optimization-based concept merging, and reports additional experiments and analyses.
- Appendix scope: Appendix A evaluates Custom Diffusion on CustomConcept101, a larger dataset of diverse custom concepts.The dataset and its concepts are described further in the appendix.
- Appendix scope: Appendix B derives the optimization-based method for merging multiple concepts into a single model.This addresses the multi-concept model-merging procedure.
- Appendix scope: Appendices C and D provide additional results, visualizations, evaluation details, and trends with training iterations.They also include implementation details.
A. CustomConcept101
CustomConcept101 broadens evaluation to 101 diverse concepts and 101 pairwise compositions. On this dataset, Custom Diffusion is competitive for single concepts and stronger than DreamBooth for multi-concept customization.
- Dataset: CustomConcept101 contains 101 concepts spanning toys, plushies, wearables, pets, scenes, and human faces.Images were collected from redistributable websites or captured by the authors.
- Evaluation protocol: The evaluation uses 20 prompts per single concept and 12 prompts per multi-concept composition, with prompts manually filtered and modified.The prompts test background changes, inserted objects, and stylistic variations.
- Results: On single-concept customization, Custom Diffusion performs on par with DreamBooth, with marginally lower image alignment and higher text alignment.Textual Inversion overfits target images and has low text alignment.
- Results: For multi-concept customization, both optimization-based Custom Diffusion and joint training outperform DreamBooth on average.Sample generations are provided in Figure 13.
B. Multi-Concept Optimization Based Method
Custom Diffusion combines multiple concepts by solving a constrained least-squares problem in closed form, preserving desired mappings while using text features from target and regularization captions.
- The method solves multi-concept composition through a constrained least-squares problem with a closed-form solution.The optimization enforces target mappings while remaining close to pretrained weights.
- C contains text features for target words across all concepts, while Creg contains features from approximately 1,000 randomly sampled regularization captions.
- The optimization assumes that Creg is non-degenerate when deriving the closed-form solution.
C. Experiments
Experiments evaluate Custom Diffusion on single- and multi-concept customization, compression, modifier-token choices, and difficult compositions. The method supports multi-concept synthesis but retains failure cases in challenging combinations.
- Datasets: CustomConcept101 includes 101 categories collected from Unsplash, redistribution-permitted websites, and authors’ own captures.
- Single- and multi-concept comparisons: Custom Diffusion achieves higher text alignment than DreamBooth and Textual Inversion, while DreamBooth retains more image alignment with target images in qualitative comparisons.
- Model compression: 5× model-storage compression preserves similar performance, but stronger compression decreases image alignment and moves the model toward pretrained behavior.
- Multi-concept fine-tuning: Custom Diffusion composes two new objects with a style or synthesizes three new objects in one scene.
- Limitations: Overlapping attention maps for “dog” and “cat” may contribute to worse compositions, including failures to place both personal concepts in the same scene.
- Regularization analysis: Generated images used for regularization can produce saturation artifacts for category-word prompts, potentially mitigated by a lower learning rate at the cost of longer training.
D. Evaluation
Evaluation shows a trade-off between text and image alignment during fine-tuning, while Custom Diffusion generally preserves performance on unrelated concepts and compares favorably with baselines.
- Alignment trends: Fine-tuning gradually decreases text alignment while increasing image alignment, exposing a trade-off between prompt fidelity and similarity to target images.
- Ablations: Random modifier-token initialization can shift original category words toward target images, while generated-image regularization worsens original-category samples.
- Baseline comparison: Custom Diffusion generally matches or exceeds baselines on averaged single-concept text- and image-alignment metrics.
- Unrelated-concept evaluation: Fine-tuned models retain similar MS-COCO FID to the pretrained model, indicating limited change to generated distributions for unrelated concepts.
E. Implementation and Experiment Details
The experiments use manually captured or downloaded datasets, short training schedules, augmentation, and CLIP-based alignment evaluation for single- and multi-concept customization.
- Datasets: Datasets are manually captured or downloaded from Unsplash, except for Moongate.
- Custom Diffusion: Custom Diffusion trains for 250 steps on single concepts and 500 steps on multi-concept tasks, using random resizing and size-related prompt augmentation.
- Evaluation: Single-concept evaluation uses 1K generated samples across 20 prompts per dataset for CLIP text- and image-alignment metrics.
- Evaluation: Multi-concept evaluation measures each composition pair on 400 generated images from 8 prompts using 200 DDPM steps and scale=6.
- Token selection: The modifier token is selected from rare alphabetic tokens occurring approximately 5–10 times in sampled LAION-400M captions and not embedded within another token.
- Societal impact: The method’s compute-efficient fine-tuning is presented as enabling customization by everyday users and easier sharing of fine-tuned concepts.
G. Change log
The change log records added dataset details, updated results, and new visual analyses covering compression, composition, prompt difficulty, and overfitting.
- The second version adds CustomConcept101 dataset details and results in Appendix A, Figure 12, Figure 13, and Table 4.
- The updated results include three-concept compositions shown in Figure 16.
- The compression analysis reports storage requirements from 75MB down to 0.08MB, with similar performance after 5× compression using the top 60% singular values.
- Textual Inversion struggles to compose two fine-tuned objects in the reported qualitative samples.
- Long text prompts do not enable the pretrained model to generate exact target images, motivating model fine-tuning.
- Regularization reduces prompt-template overfitting, producing less shift toward target images and greater diversity than the version without regularization.