Source-linked AI summary
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, Kfir Aberman
TL;DR
DreamBooth personalization is effective but costly in time and storage. HyperDreamBooth uses a HyperNetwork to generate compact personalized weights from one face image, followed by fast fine-tuning; it reports faster, much smaller personalization while preserving subject details, styles, and semantic edits.
Problem
DreamBooth personalization requires substantial GPU time and storage, despite supporting high-fidelity subjects across diverse contexts and styles.
Method
HyperDreamBooth predicts Lightweight DreamBooth low-rank residuals from a single face image and applies rank-relaxed fine-tuning to refine them.
Results
HyperDreamBooth produces high-quality, diverse face images with different styles and semantic modifications while preserving subject details and model integrity.
Takeaways & Limitations
The method provides fast and lightweight subject personalization while retaining the diffusion model’s style prior, editability, and subject fidelity.
Abstract
from arXiv · showhide
Personalization has emerged as a prominent aspect within the field of generative AI, enabling the synthesis of individuals in diverse contexts and styles, while retaining high-fidelity to their identities. However, the process of personalization presents inherent challenges in terms of time and memory requirements. Fine-tuning each personalized model needs considerable GPU time investment, and storing a personalized model per subject can be demanding in terms of storage capacity. To overcome these challenges, we propose HyperDreamBooth - a hypernetwork capable of efficiently generating a small set of personalized weights from a single image of a person. By composing these weights into the diffusion model, coupled with fast finetuning, HyperDreamBooth can generate a person's face in various contexts and styles, with high subject details while also preserving the model's crucial knowledge of diverse styles and semantic modifications. Our method achieves personalization on faces in roughly 20 seconds, 25x faster than DreamBooth and 125x faster than Textual Inversion, using as few as one reference image, with the same quality and style diversity as DreamBooth. Also our method yields a model that is 10,000x smaller than a normal DreamBooth model. Project page: https://hyperdreambooth.github.io
1. Introduction
HyperDreamBooth addresses DreamBooth’s size and speed shortcomings while preserving model integrity, editability, and subject fidelity. It predicts personalized weights from a face image and then fine-tunes them for improved fidelity.
- Motivation: DreamBooth personalization preserves subject details and diverse styles, but its size and training speed limit broader use.Stable Diffusion DreamBooth fine-tunes more than 1GB of UNet and Text Encoder weights and takes about 5 minutes for 1,000 iterations.
- Contributions: HyperDreamBooth proposes Lightweight DreamBooth, a roughly 100KB personalized model based on a low-dimensional weight space inside LoRA weight space.The low-dimensional space is generated by a random orthogonal incomplete basis.
- Contributions: Rank-relaxed fine-tuning increases the LoRA rank during optimization to approximate high-level subject details and improve fidelity.The predicted weights are inserted into the model before higher-rank LoRA fine-tuning.
- Results: The method is reported as 25x faster than DreamBooth while achieving similar performance.The contribution passage attributes the speedup to HyperNetwork initialization followed by fast fine-tuning.
- Contributions: Its HyperNetwork predicts personalized weights from a single face image, providing an initialization for subsequent fast fine-tuning.Phase 1 uses pre-computed personalized weights and diffusion reconstruction losses for supervision; Phase 2 fine-tunes predicted weights with reconstruction loss.
2. Related Work
Related work spans text-to-image conditioning, generative-model personalization, HyperNetworks, and fine-tuning-based personalization. HyperDreamBooth differs by directly predicting low-rank network residuals for subject-specific adaptation.
- Text-to-Image Models: Text-to-image models generate images from text, while some also support image conditioning or additional inputs such as depth.The passage notes that current text- and image-based conditioning does not capture sufficient subject details.
- Personalization of Generative Models: Earlier GAN-based personalization methods manipulate subject images or fine-tune generative models, but often lack subject fidelity or diverse contexts.The cited approaches include pivotal tuning, StyleGAN personalization, and GAN conditioning with an input image.
- HyperNetworks: HyperNetworks predict neural-network weights and have been applied to image-generation tasks related to personalization, including StyleGAN inversion.They have also been used in language modeling.
- T2I Personalization via Finetuning: Fine-tuning methods optimize different personalization targets, ranging from text embeddings to full network weights and other parameterizations.Examples include Textual Inversion, DreamBooth, CustomDiffusion, SVDiff, LoRA, StyleDrop, and DreamArtist.
- Fast T2I Personalization: Fast personalization methods commonly combine encoder learning with complete network fine-tuning, whereas HyperDreamBooth directly predicts low-rank network residuals.The related-work passage identifies diversity, fidelity, and adherence to image distribution as continuing challenges.
3. Preliminaries
The paper builds on latent diffusion, DreamBooth, LoRA, and a lightweight DreamBooth variant to reduce the personalized parameter footprint. These preliminaries motivate predicting compact residual weights rather than fine-tuning full networks.
- Latent Diffusion Models: Latent diffusion models iteratively denoise a noise map into an image conditioned on a text embedding produced by a text encoder.Stable Diffusion is the specific latent-diffusion instantiation used in the work.
- DreamBooth: DreamBooth fine-tunes diffusion-network weights on subject images while using class-specific prior preservation to retain generalization.For Stable Diffusion, this involves fine-tuning the entire denoising UNet with over 1GB of parameters.
- Low Rank Adaptation: LoRA replaces full-weight updates with low-rank residuals, decomposing each residual as ΔW = AB with r much smaller than the layer dimensions.For Stable Diffusion 1.5, rank-1 LoRA-DreamBooth has approximately 386K parameters and is about 1.6MB.
- Lightweight DreamBooth: Lightweight DreamBooth further decomposes rank-1 LoRA residuals using a random orthogonal incomplete basis within LoRA weight space.The auxiliary layers are randomly initialized with row-wise orthogonal vectors and frozen.
4. Method
HyperDreamBooth combines an extremely compact personalized weight space with a HyperNetwork that predicts weights from one image, followed by fast finetuning to recover fine subject details. The design preserves subject fidelity, editability, style diversity, and prompt fidelity while reducing personalization storage and optimization demands.
- Lightweight DreamBooth: 10,000 times smaller than a DreamBooth model, LiDB uses only 30K variables and 120 KB of storage.The final weight space is also more than 10 times smaller than a LoRA DreamBooth model.
- Lightweight DreamBooth: LiDB further decomposes rank-1 LoRA residuals using frozen random orthogonal bases, leaving only Atrain and Btrain learnable.The residual is expressed as ΔW = A_aux A_train B_train B_aux, with the auxiliary matrices fixed.
- Lightweight DreamBooth: With a = 100 and b = 50, LiDB retains strong subject fidelity, editability, and style diversity using 30K trainable variables.These settings produce the 120 KB personalized model described above.
- HyperNetwork: The HyperNetwork takes a single image and predicts LiDB low-rank residuals for the diffusion model’s cross-attention and self-attention layers.Training combines diffusion denoising and weight-space losses, using domain-specific images.
- Supervisory Text Prompt: The method avoids learned token embeddings, using “a [V] face” for supervision while allowing semantic prompt modifications at inference.Variations such as “a [V] face in impressionist style” are used for recontextualization.
- HyperNetwork: A ViT image encoder and transformer decoder model dependencies among sequential diffusion-model layer weights during prediction.The image is encoded once for iterative prediction, which speeds training and inference without affecting result quality.
- Fast Finetuning: 40 iterations of fast finetuning are 25x faster than DreamBooth and LoRA DreamBooth after HyperNetwork initialization.The initial prediction captures broad semantic attributes, while finetuning recovers fine details; rank relaxation increases the capacity for high-frequency subject details.
5. Experiments
Experiments evaluate HyperDreamBooth’s personalization quality, editability, efficiency, and component contributions against established methods. Across qualitative comparisons, ablations, and user studies, it preserves identity and stylization while aggressive baseline training can degrade results.
- Subject Personalization Results: HyperDreamBooth generates diverse facial styles and semantic transformations while preserving subject characteristics and the diffusion model’s style prior.Examples include figurines and animated characters, with outputs retaining key facial characteristics.
- Comparisons: HyperDreamBooth generally outperforms Textual Inversion and DreamBooth and compares strongly with E4T in the single-reference regime.E4T can overfit to the reference pose and realistic appearance when outputs should be highly stylized.
- Quantitative Comparisons and Ablations: HyperDreamBooth outperforms Textual Inversion and DreamBooth across Face Rec., DINO, CLIP-I, and CLIP-T metrics on 30,000 samples.The evaluation uses 100 CelebAHQ identities and 30 prompts; face-recognition metrics are noted as relatively weak because stylized faces are out of distribution.
- Quantitative Comparisons and Ablations: Aggressive DreamBooth training without HyperNetwork initialization generally degrades results.The comparison uses 400-iteration and 40-iteration variants against 1200-iteration vanilla DreamBooth.
- Quantitative Comparisons and Ablations: The complete HyperDreamBooth setup achieves superior subject fidelity, while removing the HyperNetwork or finetuning changes the fidelity-prompt-following trade-off.The full setup performs best for fidelity metrics, whereas No Hyper achieves slightly better prompt following.
- User Study: Users generally prefer HyperDreamBooth for identity preservation and for combined stylization and identity compared with DreamBooth, Textual Inversion, and E4T.The identity study evaluates 1,000 sample pairs, while the E4T comparison evaluates 600 samples.
6. Conclusion
HyperDreamBooth is presented as a fast, lightweight personalization method that generates compact personalized parameters and refines them through fast finetuning. It produces diverse, high-quality facial images while preserving subject details and model integrity.
- 6. Conclusion: HyperDreamBooth uses a HyperNetwork to generate Lightweight DreamBooth parameters followed by rank-relaxed fast finetuning.This combination reduces personalization size and speed compared with DreamBooth and other optimization-based methods.
- 6. Conclusion: HyperDreamBooth produces high-quality, diverse facial images with different styles and semantic modifications while preserving subject details and model integrity.