Source-linked AI summary
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
Jing Shi, Wei Xiong, Zhe Lin, Hyun Joon Jung
TL;DR
Existing personalized image-generation methods typically require costly test-time finetuning for each concept, limiting efficiency and scalability. InstantBooth uses an image encoder for textual concept embeddings and adapter layers for rich identity features, trained only on text-image pairs. On unseen concepts, it produces language-aligned, identity-preserved images with a single forward pass and is reported as x100 faster than comparable test-time finetuning-based methods.
Problem
Personalized image generation requires learning concepts from image sets while existing approaches commonly depend on costly test-time finetuning for each concept.
Method
InstantBooth maps input images to a textual embedding with a learnable image encoder and adds adapter layers carrying rich visual features, trained using only text-image pairs.
Results
On unseen concepts, InstantBooth generates language-aligned, identity-preserved images with a single forward pass and is reported as x100 faster than DreamBooth-like test-time finetuning methods.
Takeaways & Limitations
Once trained, InstantBooth enables instant personalized generation without test-time finetuning for each new concept.
Abstract
from arXiv · showhide
Recent advances in personalized image generation allow a pre-trained text-to-image model to learn a new concept from a set of images. However, existing personalization approaches usually require heavy test-time finetuning for each concept, which is time-consuming and difficult to scale. We propose InstantBooth, a novel approach built upon pre-trained text-to-image models that enables instant text-guided image personalization without any test-time finetuning. We achieve this with several major components. First, we learn the general concept of the input images by converting them to a textual token with a learnable image encoder. Second, to keep the fine details of the identity, we learn rich visual feature representation by introducing a few adapter layers to the pre-trained model. We train our components only on text-image pairs without using paired images of the same concept. Compared to test-time finetuning-based methods like DreamBooth and Textual-Inversion, our model can generate competitive results on unseen concepts concerning language-image alignment, image fidelity, and identity preservation while being 100 times faster.
1. Introduction
Personalized text-to-image generation seeks to create new scenes or styles for an image-based concept while preserving identity and following language prompts. InstantBooth removes per-concept test-time finetuning by combining an image encoder that produces a textual embedding with adapter layers that retain fine identity details.
- Personalized generation learns a concept from images and produces new prompted scenes or styles while preserving the subject’s identity.
- Existing approaches commonly use pretrained text-to-image models and either invert images into textual representations or learn image-to-image mappings with text guidance.
- Test-time finetuning methods such as DreamBooth and Textual-Inversion adapt a model or textual embedding for each input concept before inference.
- InstantBooth learns an image encoder that maps input images to a compact textual embedding through offline training, supporting generalization to unseen concepts.
- Adapter layers inject rich visual features into the pretrained model to preserve fine-grained identity details without relying only on the compact embedding.
- Training uses only text-image pairs, yet the model generates unseen concepts with pose and location variation, identity preservation, and language-image alignment.
- Compared with methods such as DreamBooth, InstantBooth achieves comparable results while being x100 faster.
2. Related Work
Related work advances text-to-image synthesis with large pretrained generators and adds controllable visual conditions through new layers or mappings. InstantBooth differs from concurrent personalization methods by avoiding test-time finetuning for each new concept.
- Text-to-image synthesis progressed from GANs to transformer and diffusion models for generating complex scenes with improved fidelity and semantics.
- ControlNet and GLIGEN extend pretrained generators by adding trainable layers or injected conditions while retaining the original model capacity.
- Nevertheless, existing controllable personalization methods still rely on test-time finetuning for new concepts.
- InstantBooth generates identity-preserved results for new concepts without per-concept test-time finetuning, improving efficiency relative to those methods.
- Concurrent approaches differ in their training requirements: ELITE finetunes attention and U-Net parameters, UMM-Diffusion learns a visual mapping with a frozen generator, and SuTI uses massive paired data.
3. Method
InstantBooth personalizes a pre-trained text-to-image model without test-time finetuning by combining a compact concept embedding with rich patch features injected through trainable adapters. It uses masked, augmented inputs during training, supports arbitrary conditioning images at inference, and applies concept renormalization to preserve language control.
- Model architecture: InstantBooth maps input images to a compact textual concept embedding and injects rich patch features into U-Net adapters for identity preservation.The pre-trained backbone remains frozen while the image encoders and adapter layers provide concept and fine-grained visual conditioning.
- Data preprocessing and training: Training uses cropped, background-masked images with random augmentations, while the original uncropped images serve as ground truth.The model is trained with one image per concept because paired images of the same concept are unavailable.
- Prompt and concept embedding: The modified prompt inserts a unique identifier before the subject class noun, whose embedding is replaced by the averaged global image feature.The remaining prompt words are encoded by the frozen text encoder, producing the final textual condition for diffusion cross-attention.
- Rich representation learning: Each input image contributes 257 patch tokens, which are concatenated into a rich token sequence for the adapter.These patch features retain subject-related content details beyond the compact global concept embedding and therefore benefit identity preservation.
- Model inference: At inference, the adapter accepts an arbitrary number of conditioning images because concatenation and self-attention do not require a fixed token count.Inference masks backgrounds but omits the training-time augmentations.
- Concept renormalization: Concept renormalization rescales the concept token so it no longer dominates cross-attention, preserving style and background language control.Without renormalization, the model failed to generate the “witcher” style or “night” background; with it, noun attention became more balanced.
4. Experiment
Experiments evaluate InstantBooth on person and cat personalization using identity-preservation, reconstruction, and vision-language alignment measures, alongside qualitative comparisons and ablations. The method shows strong alignment and identity preservation, substantially faster testing, and trade-offs involving reconstruction, masking, adapter features, and input-image count.
- Dataset and metrics: Experiments cover person and cat categories using text-image pairs, segmentation masks, and prompts spanning category descriptions and varied visual conditions.The evaluation includes object categories, collected prompts, entity masks, and qualitative personalized generations.
- Dataset and metrics: Reconstruction measures input-output CLIP visual similarity, face distance targets person identity, and alignment measures CLIP similarity between prompts and generated images.Prompts test background changes, style changes, and composition.
- Comparison to SOTA Methods: InstantBooth achieves better vision-language alignment and face distance than comparison methods, while reconstruction is weaker because it suppresses background content.The method reconstructs the foreground portrait rather than replicating the full image background, lowering reconstruction without necessarily indicating weaker identity preservation.
- Comparison to SOTA Methods: With masked test inputs, both InstantBooth variants achieve better vision-language alignment and identity preservation.This evaluates the full setting in which background-masked images are provided during testing.
- Comparison to SOTA Methods: 100x faster testing is reported for InstantBooth because it does not require test-time finetuning.The comparison includes Textual Inversion and DreamBooth, both described as requiring heavy test-time finetuning.
- Ablation Study: Ablations show that removing the object mask retains more background, raising reconstruction but lowering alignment, while removing patch features heavily degrades reconstruction and identity.The patch feature is reported as crucial for foreground detail and identity preservation; all ablations perform worse visually than the full setting.
- Ablation Study: Removing the adapter branch or moving the concept token before CLIP worsens visual results, while tuning the CLIP visual encoder or U-Net worsens identity preservation.The authors attribute missing visual detail before CLIP to identity information being diffused by the frozen text encoder and conclude that pretrained weights should remain fixed.
- Ablation Study: Larger adapter weight β or concept-renormalization factor α improves identity preservation but weakens language comprehension, motivating β = 0.3 and α = 0.4 as a trade-off.The selected setting balances identity preservation and language comprehension.
5. Conclusion
InstantBooth personalizes images without test-time finetuning by combining textual concept tokens with adapter-based identity features. It generates language-aligned, identity-preserved images of unseen concepts in a single forward pass, while remaining limited to separate training per category and one concept at a time.
- InstantBooth converts input images into a textual token for general concept learning and uses adapter layers to retain fine-grained identity details.
- The model generates language-aligned, identity-preserved images of unseen concepts with only a single forward pass.
- Limitation and Future Work: The model must currently be trained separately for each category.
- Limitation and Future Work: The current adapter design accepts only a single concept for providing identity details.
A. Evaluation on Identity Preservation with Perceptual Face Distance
The evaluation notes that full-image reconstruction scores are inaccurate for identity preservation because shared background regions affect similarity. A face-specific embedding distance is therefore used for person images.
- Full-image reconstruction is inaccurate for identity preservation because shared background regions affect similarity between generated and input foreground objects.
- For person images, face detection and Inception-ResnetV1 embeddings are used to calculate perceptual face distance.The distance is the average L-2 norm between paired face embeddings.
B. Explanation on The Necessity of Using The ˆV Identifier
The ablation shows that the ˆV identifier is needed to align the input identity with the correct subject in the prompt and generated image. Without it, the model confuses the intended subject.
- Without training with the ˆV identifier, the model cannot correctly generate Joe Biden’s face and may confuse it with the input woman’s face.
- The ˆV identifier helps align input images with the correct subject specified in the prompt and generated images.
C. Details of Test Prompts
The person evaluation uses prompts that vary scenes, locations, artistic styles, and rendering formats while retaining the ˆV identifier and class noun.
- Test prompts place the ˆV subject in varied scenes and locations, including a swimming pool, New York, the moon, a kitchen, mountains, and a library.
- Other prompts evaluate artistic variation through Van Gogh-style painting, manga drawing, and colorful graffiti.