Source-linked AI summary
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention
Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, Song Han
TL;DR
Existing subject-driven generation is costly to personalize and prone to blending identities across multiple subjects. FastComposer uses vision-derived subject embeddings, localized cross-attention supervision, and delayed conditioning without fine-tuning. It reports inference-only multi-subject generation with 300×-2500× speedup, memory savings, and zero extra storage for new subjects.
Problem
Subject-driven generation is limited by costly personalization and identity blending when multiple subjects are generated.
Method
FastComposer augments text conditioning with vision-encoder subject embeddings, supervises cross-attention with segmentation masks, and delays subject conditioning during denoising.
Results
300×-2500× speedup and 2.8×-6.7× memory saving are reported versus fine-tuning-based methods, with zero extra storage for new subjects.
Takeaways & Limitations
FastComposer provides inference-only personalized multi-subject generation while balancing identity preservation with textual editability.
Takeaways & Limitations
The FFHQ training set is small, human-face-centric, and long-tailed, limiting generation beyond three subjects and across other categories.
Abstract
from arXiv · showhide
Diffusion models excel at text-to-image generation, especially in subject-driven generation for personalized images. However, existing methods are inefficient due to the subject-specific fine-tuning, which is computationally intensive and hampers efficient deployment. Moreover, existing methods struggle with multi-subject generation as they often blend features among subjects. We present FastComposer which enables efficient, personalized, multi-subject text-to-image generation without fine-tuning. FastComposer uses subject embeddings extracted by an image encoder to augment the generic text conditioning in diffusion models, enabling personalized image generation based on subject images and textual instructions with only forward passes. To address the identity blending problem in the multi-subject generation, FastComposer proposes cross-attention localization supervision during training, enforcing the attention of reference subjects localized to the correct regions in the target images. Naively conditioning on subject embeddings results in subject overfitting. FastComposer proposes delayed subject conditioning in the denoising step to maintain both identity and editability in subject-driven image generation. FastComposer generates images of multiple unseen individuals with different styles, actions, and contexts. It achieves 300$\times$-2500$\times$ speedup compared to fine-tuning-based methods and requires zero extra storage for new subjects. FastComposer paves the way for efficient, personalized, and high-quality multi-subject image creation. Code, model, and dataset are available at https://github.com/mit-han-lab/fastcomposer.
1 Introduction
FastComposer addresses the cost of subject personalization and identity blending in multi-subject generation with a tuning-free, forward-pass approach. It combines subject-augmented text conditioning, cross-attention localization, and delayed conditioning to preserve identity and editability.
- Motivation: Existing subject-driven methods face costly personalization and identity blending when generating multiple subjects.Identity blending combines distinct characteristics, making one subject resemble another.
- Method: FastComposer replaces generic text tokens with identity embeddings extracted from reference images by a vision encoder.The augmented conditioning supports personalized generation from subject images and textual instructions using only forward passes.
- Method: Cross-attention localization supervises subject attention maps with segmentation masks to associate each subject with distinct image regions.This targets the unregulated attention that otherwise attends to multiple people and causes identity blending.
- Method: Delayed subject conditioning uses text-only conditioning early for layout, then subject-augmented conditioning to refine appearance.The approach preserves subject identity while following textual instructions, addressing subject overfitting.
- Results: 300×-2500× speedup and 2.8×-6.7× memory saving are reported versus fine-tuning-based methods, with zero extra storage for new subjects.FastComposer enables inference-only multi-subject generation across diverse scenarios.
2 Related Work
Related work includes optimization-based personalization, tuning-free subject conditioning, and multi-subject composition methods. These approaches remain constrained by fine-tuning costs, architecture limitations, or difficulties composing similar subjects.
- Subject-Driven Image Generation: DreamBooth, textual inversion, and Custom-Diffusion embed subjects through model fine-tuning or inversion into text tokens.These methods use optimization-based personalization of diffusion models.
- Tuning-Free Methods: Tuning-free methods project reference images into conditioning representations, but some architectures restrict applicability to multiple subjects.X&Fuse concatenates reference images with noisy latents, while ELITE and InstantBooth use mapping networks and reference features.
- Multi-Subject Image Generation: Custom-Diffusion supports multi-concept composition but struggles when composing subjects from similar categories.It may generate the same person twice when composing two different individuals.
3 Preliminaries
The preliminaries describe Stable Diffusion’s latent denoising pipeline and its text-conditioning mechanism. Cross-attention distributes text-token information across spatial latent locations, providing the basis for FastComposer’s conditioning design.
- Stable Diffusion: Stable Diffusion uses a VAE, U-Net, and text encoder to transform images into latents and predict denoising noise.At inference, random noise is iteratively denoised before the VAE decoder maps the final latent back to pixels.
- FastComposer Pipeline: The diffusion model is trained to generate multi-subject images from text and subject-augmented conditioning.Figure 3 summarizes the training and inference pipeline using an image encoder and augmented text tokens.
- Text Conditioning: Cross-attention projects spatial latent features and text embeddings into Query, Key, and Value matrices before computing attention outputs.The attention map represents information flow from each text token to spatial latent pixels.
4 FastComposer
FastComposer combines subject-augmented text conditioning with cross-attention localization and delayed subject conditioning for tuning-free personalized generation. These components target identity blending while balancing subject identity with adherence to textual instructions.
- Tuning-Free Subject-Driven Image Generation with an Image Encoder: A vision encoder extracts reference-image features to augment text conditioning, enabling subject-driven generation through forward passes.The approach trains the image encoder, MLP, and U-Net using subject-augmented image-text pairs.
- Localizing Cross-Attention Maps with Subject Segmentation Masks: Unregulated cross-attention can blend multiple subjects because each subject token attends to both people rather than a distinct image region.The paper identifies unrestricted cross-attention as the primary source of multi-subject identity blending.
- Delayed Subject Conditioning in Iterative Denoising: A 0.6–0.8 subject-conditioning ratio balances prompt consistency and identity preservation.Increasing the ratio improves identity preservation but reduces editability.
- Localizing Cross-Attention Maps with Subject Segmentation Masks: Cross-attention localization supervises subject-token maps to match segmentation masks, directing each subject’s features to separate regions.The localization loss is applied to downsampled cross-attention maps in the U-Net’s middle five blocks, with λ = 0.001.
- Delayed Subject Conditioning in Iterative Denoising: Delayed subject conditioning first uses text-only denoising for layout creation, then applies image-augmented conditioning to refine subject appearance.The time-dependent model switches from c to c′ after the layout-forming stage, with α controlling the subject-conditioning ratio.
- Evaluation: Figure 6 compares single-subject generation methods, including text-only StableDiffusion and Midjourney using scientists’ names as prompts.The evaluation also compares optimization-based approaches such as DreamBooth, Textual-Inversion, and Custom Diffusion.
5 Experiments
FastComposer is evaluated on single- and multi-subject generation, using FFHQ-derived benchmarks, standard baselines, quantitative metrics, and ablations of its core design choices.
- Evaluation Setup: The evaluation constructs a subject-augmented FFHQ-wild dataset using generated captions, panoptic masks, noun-phrase matching, and reserved validation and test images.Captioning uses BLIP-2, segmentation uses Mask2Former, and phrase-segment matching combines OpenCLIP and Sentence-Transformer similarity.
- Single-Subject Image Generation: Single-subject generation is compared with DreamBooth, Textual-Inversion, Custom Diffusion, and StableDiffusion as a text-only baseline.The evaluation uses subjects derived from Celeb-A and implementations from the diffusers library.
- Multi-Subject Image Generation: Multi-subject quality is assessed across 105 subject pairs and 21 prompts per pair using identity preservation and prompt consistency.Generated faces are matched to reference subjects, and identity preservation is calculated from FaceNet similarities after MTCNN detection.
- Multi-Subject Image Generation: FastComposer preserves distinct subject features better than optimization-based baselines while achieving prompt consistency on par with tuning-based approaches.The comparison reports frequent identity failure or blending for optimization-based methods and improved identity preservation for FastComposer.
- Ablation Study: A subject-conditioning ratio between 0.6 and 0.8 provides a favorable balance between identity preservation and editability.Increasing the ratio improves identity preservation but reduces editability.
- Ablation Study: Cross-attention localization supervision substantially improves identity preservation and helps prevent identity blending by focusing on particular reference subjects.The ablation compares against an otherwise similarly trained model without localization supervision.
6 Discussion and Conclusion
FastComposer combines tuning-free subject-driven generation with cross-attention localization and delayed subject conditioning, while its scope remains limited by training data and inherited biases.
- Discussion and Conclusion: FastComposer uses a pretrained vision encoder for tuning-free personalized multi-subject generation and requires only forward passes.The method is intended to make subject-driven generation efficient and accessible across platforms.
- Discussion and Conclusion: Cross-attention maps supervised with segmentation masks address identity blending, while delayed subject conditioning balances identity preservation with image-editing flexibility.These are the paper’s two principal mechanisms for multi-subject separation and editability.
- Limitations: Training on small, face-focused FFHQ data limits generation beyond three subjects and leaves the method primarily human-centric.The paper identifies more diverse multi-subject datasets as a route to broader actions, scenarios, and categories.
- Limitations: Because FastComposer is built on Stable Diffusion and FFHQ, it inherits their biases.This limitation is stated separately from the dataset’s subject and scale constraints.