PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman

arXiv:2208.12242v2cs.CVcs.GRcs.LG

TL;DR

Text-to-image models struggle to preserve a specific subject while placing it naturally in novel contexts. DreamBooth personalizes a pretrained diffusion model by binding a unique identifier to the subject from a few images, enabling photorealistic, prompt-guided renditions across scenes and applications while preserving key features.

  • Problem

    Text-to-image models cannot accurately reconstruct specific subjects from reference images while naturally synthesizing them in new contexts.

  • Method

    DreamBooth fine-tunes a pretrained diffusion model to bind a rare unique identifier and class descriptor to a subject, using prompts to generate personalized images.

  • Results

    The approach supports subject recontextualization, property modification, artistic rendering, and other text-based applications while preserving subject and prompt fidelity.

  • Takeaways & Limitations

    DreamBooth enables photorealistic subject-driven generation from only 3-5 images across diverse scenes, including animals and objects.

  • Takeaways & Limitations

    The method can suffer from context-appearance entanglement and overfitting to the original images when prompts resemble the training setting.

Abstract

from arXiv · show

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/

1 Google Research 2 Boston University

DreamBooth generates many images of a subject from only a few reference images and a text prompt. The generated images vary in context, articulation, and lighting while preserving the subject’s key visual features.

  • Subject-driven generation: DreamBooth typically uses just 3-5 images of a subject to generate many new images guided by a text prompt.The reference images are paired with textual guidance to produce varied subject renditions.
  • Subject recontextualization: The generated images place the subject in different contexts with natural interactions with the environment.The results show the subject integrated into novel surroundings rather than reproduced only in the reference settings.
  • Appearance preservation: The results introduce novel articulations and lighting variations while maintaining high fidelity to the subject’s key visual features.Variation occurs in both pose-like articulation and illumination without losing the defining appearance of the subject.

1. Introduction

The paper introduces subject-driven generation: personalizing text-to-image diffusion models to synthesize specific subjects in new contexts while preserving distinctive features. It proposes rare-token fine-tuning with class-specific prior preservation, and evaluates the resulting applications using a new dataset and protocol.

  • Motivation: Existing text-to-image models generate high-quality, diverse images from prompts but cannot accurately reconstruct a given subject’s appearance.Their semantic prior supports varied instances, poses, and contexts for subject classes, while shared language-vision embeddings still produce content variations rather than faithful reconstructions.
  • Approach: The proposed personalization approach expands the model’s language-vision dictionary by binding new words to user-specific subjects for photorealistic generation across scenes.The method aims to preserve the subject’s key identifying features while contextualizing it in novel scenes.
  • Approach: Given a few subject images, the method fine-tunes a pretrained diffusion model to associate a rare token identifier with the subject and its class.Prompts place the unique identifier before the class name, allowing class knowledge to support generation while separating the specific instance from the broader class.
  • Applications and evaluation: The approach enables subject recontextualization, property modification, original artistic renditions, and other text-based image-generation applications.The paper studies component contributions through ablations, compares alternative baselines and related work, and conducts a user study of subject and prompt fidelity.
  • Applications and evaluation: The paper introduces a dataset of subjects captured in different contexts and an evaluation protocol measuring subject fidelity and prompt fidelity.The dataset and protocol are publicly available through the project webpage.

2. Related work

Prior work spans image composition, text-guided editing and synthesis, controllable generative models, and personalized token embeddings. These approaches face limitations including rigid-object assumptions, restricted editing control, inconsistent subject identity, large data requirements, or dependence on frozen model expressiveness.

  • Image Composition: Image composition and 3D reconstruction place subjects into new scenes, but reconstruction usually targets rigid objects, requires many views, and struggles with lighting, shadows, contact, and novel scenes.The supplied passage contrasts these limitations with the proposed approach’s subject generation, though its continuation is truncated.
  • Text-to-Image Editing and Synthesis: GAN- and CLIP-based text-driven editing produces realistic manipulations but works best in structured scenarios and can struggle when subjects vary across diverse datasets.VQ-GAN-based training over more diverse data was proposed to alleviate this concern.
  • Text-to-Image Editing and Synthesis: Large text-to-image models achieve unprecedented semantic generation, yet provide limited fine-grained control and cannot consistently preserve a subject’s identity across synthesized images.The passage names Imagen, DALL-E2, Parti, CogView2, and Stable Diffusion as examples.
  • Controllable Generative Models: Controllable-generation methods use reference guidance, masks, inversion, or model fine-tuning, while personalized GAN approaches may require around 100 images, remain face-limited, or lose subject details.The cited approaches include diffusion-based image variations, mask-restricted editing, Pivotal Tuning, personalized face priors, and instance-conditioned GANs.
  • Controllable Generative Models: New-token methods personalize frozen text-to-image models with small embeddings, whereas fine-tuning embeds the subject within the model’s output domain but avoids the frozen model’s expressiveness limitation.The supplied passage states that this enables novel subject images preserving key features, but its final phrase is truncated.

3. Method

DreamBooth personalizes a pretrained text-to-image diffusion model from typically 3–5 casually captured subject images by binding a rare identifier to the subject and class descriptor. It fine-tunes the model with an autogenous class-specific prior-preservation loss to retain class knowledge, counter language drift, and preserve diverse novel outputs.

  • Task: The method targets high-detail subject fidelity from typically 3–5 casually captured images while enabling text-guided changes in location, properties, pose, and viewpoint.Input images require no restricted capture settings or textual descriptions.
  • Designing Prompts for Few-Shot Personalization: All subject images are labeled “a [identifier] [class noun]”, pairing a unique identifier with a coarse class descriptor such as cat, dog, watch, or similar.This prompt design implants the new identifier–subject pair without requiring detailed image descriptions.
  • Rare-token Identifiers: Rare-token identifiers are preferred because ordinary words carry existing meanings, while random character strings may be split by the tokenizer and inherit strong priors.The identifier should have weak priors in both the language and diffusion models.
  • Fine-tuning: Fine-tuning all model layers gives maximum subject fidelity but risks language drift and reduced diversity in generated poses and viewpoints.Small few-shot datasets can cause outputs to snap toward the limited training views.
  • Class-Specific Prior Preservation: The autogenous class-specific prior-preservation loss supervises fine-tuning with samples generated by the frozen pretrained model, encouraging diversity and countering language drift.The loss uses class prompts such as “a [class noun]”; about 1000 iterations with λ = 1 and learning rates 10^-5 for Imagen or 5 × 10^-6 for Stable Diffusion are reported as sufficient for good results.

4. Experiments

Experiments evaluate DreamBooth on a 30-subject dataset using subject- and prompt-fidelity metrics, and demonstrate recontextualization, artistic rendering, novel-view synthesis, and property modification. The method generally preserves subject identity across semantic changes, but can fail with difficult contexts, entanglement, overfitting, and rare subjects.

  • Applications: DreamBooth supports text-guided recontextualization, property and species modification, artistic rendering, and viewpoint changes while preserving distinctive subject features.The method generates new poses, scene structures, artistic variations, novel viewpoints, and cross-species renditions while retaining subject identity.
  • Dataset and evaluation: The evaluation dataset contains 30 subjects—21 objects and 9 live subjects or pets—and the suite generates four images per subject and prompt, totaling 3,000 images.The authors publicly release the dataset and evaluation protocol for future subject-driven-generation evaluation.
  • Dataset and evaluation: Subject fidelity is measured with CLIP-I and DINO, while prompt fidelity is measured with CLIP-T and CLIP-T-L; DINO is preferred because it better distinguishes subjects with similar descriptions.CLIP-I computes pairwise cosine similarity between generated and real CLIP embeddings, while DINO uses ViT-S/16 DINO embeddings.
  • Comparison: Compared with Textual Inversion, DreamBooth receives overwhelming user preference for both subject fidelity and prompt fidelity and performs better in qualitative comparisons.The reported preference aligns with Table 1, where DINO differences around 0.1 and CLIP-T differences of 0.05 are significant in user preference.
  • Limitations: Failures include inaccurate prompted contexts, context-appearance entanglement, overfitting to original settings, difficulty learning some subjects, limited variation for rare subjects, and hallucinated features.Incorrect or absent class nouns can also cause contention with, or failure to leverage, the class prior, harming convergence and subject fidelity.

5. Conclusions

The paper presents a subject-driven generation approach that binds a subject from a few images to a unique identifier in a text-to-image diffusion model, enabling novel text-guided renditions. The method works with only 3–5 subject images and supports applications involving animals and objects.

  • 5. Conclusions: The approach synthesizes novel subject renditions from a few subject images guided by a text prompt.It embeds the subject instance into the output domain of a text-to-image diffusion model.
  • 5. Conclusions: The method binds each subject instance to a unique identifier through fine-tuning.The identifier is used within the text-to-image diffusion model to represent the given subject.
  • 5. Conclusions: 3–5 subject images are sufficient for the fine-tuning process.This makes the technique particularly accessible for subject-driven generation.
  • 5. Conclusions: The paper demonstrates applications with animals and objects in generated photographs.These applications illustrate the approach's ability to produce varied renditions of subjects.

Supplementary Material · Background

Text-to-image diffusion models learn to generate images by reversing a noise-adding process, with conditioning vectors guiding denoising. Background approaches include cascaded or latent-space architectures and vocabulary encodings that influence visual quality and semantic fidelity.

  • Background: Diffusion models learn data distributions by gradually denoising Gaussian-sampled variables, corresponding to a learned reverse process of a fixed-length Markovian forward process.The conditional model is trained with a squared-error denoising loss.
  • Background: A conditional diffusion model denoises variably noised images z_t = α_t x + σ_t ϵ using a conditioning vector c and noise-schedule terms α_t, σ_t, and w_t.Here, x is the ground-truth image, c may come from a text prompt, and ϵ ∼ N(0, I).
  • Background: At inference, sampling begins from Gaussian noise and iteratively denoises intermediate states using either deterministic DDIM or stochastic ancestral sampling.The intermediate states are ordered by diffusion times from t_1 = 1 downward.
  • Background: Some state-of-the-art systems use cascaded diffusion models, combining a 64x64 base text-to-image model with super-resolution stages up to 1024 × 1024.The cited configuration uses stages of 64 × 64 → 256 × 256 and 256 × 256 → 1024 × 1024.
  • Background: Stable Diffusion instead performs forward and backward diffusion in a lower-dimensional latent space and decodes generated latent codes through a pretrained encoder-decoder architecture.The architecture is trained on a large image dataset before inference.
  • Background: Text conditioning is important for both visual quality and semantic fidelity in text-to-image diffusion models.Different systems use CLIP text embeddings with a learned prior or a pretrained T5-XXL language model.
  • Background: Vocabulary encoding transforms a tokenized text prompt into a conditioning embedding used by the diffusion model.The paper uses T5-XXL embeddings for this preprocessing step.

Dataset · Subject Fidelity Metrics

The evaluation suite covers 30 subjects and 25 prompts, producing 3,000 images for subject-driven generation. Subject fidelity is assessed with DINO and CLIP-I, with DINO argued to better track fine-grained human judgments.

  • Dataset: The dataset contains 30 subjects: 21 objects and 9 live subjects or pets.One sample image is provided for each subject; images were collected by the authors or sourced from Unsplash.
  • Dataset: The evaluation uses 25 prompts spanning recontextualization, accessorization, and property modification.Objects receive 20 recontextualization and 5 property modification prompts; live subjects/pets receive 10 recontextualization, 10 accessorization, and 5 property modification prompts.
  • Dataset: Four images are generated per subject and prompt, totaling 3,000 images for evaluating performance and generalization.The dataset and evaluation protocol are publicly available on the project webpage.
  • Subject Fidelity Metrics: DINO is proposed as superior to CLIP-I for measuring subject fidelity.The authors hypothesize that DINO’s self-supervised training distinguishes images modulo augmentations, whereas CLIP encodes descriptive text-image information that may omit fine details.
  • Subject Fidelity Metrics: The second real image achieves the highest CLIP-I and DINO scores among the illustrated examples.This example establishes the reference-image performance comparison used to examine metric behavior.
  • Subject Fidelity Metrics: DINO ranks the DreamBooth sample above the Textual Inversion sample, matching human judgments more closely than CLIP-I.Although the DreamBooth sample appears more similar to the reference, Textual Inversion receives a higher CLIP-I score.

User Study

The user study asked participants to compare generated items against reference images or text, evaluating subject identity and text-based descriptions. Across each study, 72 users answered 25 comparative questions, yielding 1,800 answers over 600 image pairs.

  • Subject fidelity: For subject fidelity, participants selected which generated item A or B reproduced the reference item’s identity and details.Accessories such as hats and outfits were instructed not to affect judgments.
  • Text-based evaluation: For text-based evaluation, participants selected which generated item A or B was best described by the reference text.Participants could choose “Cannot Determine / Both Equally” when unsure.
  • Study design: 72 users answered 25 comparative questions per study, totaling 1,800 answers across 600 evaluated image pairs.Each questionnaire was answered by 3 users.

Additional Applications and Examples

Additional examples demonstrate DreamBooth’s ability to preserve subject identity while changing expressions, accessories, properties, materials, viewpoints, and artistic or comic presentation. These capabilities extend personalized generation beyond photorealistic recontextualization, although some property modifications depend on the base model’s prior.

  • Additional Examples: Additional examples cover artistic renditions and novel viewpoints, with further recontextualization and comparisons against Imagen, Stable Diffusion, and Textual Inversion samples.These examples are presented in Figures 12, 13, and 15 and an annex HTML file.
  • Expression Manipulation: DreamBooth generates diverse subject expressions, spanning negative-to-positive emotions and arousal levels while preserving the dog’s asymmetric white facial streak.The examples include expressions absent from the original subject images.
  • Accessorization: DreamBooth accessorizes a Chow Chow with different accessories while preserving the dog’s identity and producing aesthetically pleasing results.The prompts follow the form “a [V] [class noun] wearing [accessory]”.
  • Property Modification: DreamBooth modifies subject properties such as color, generating novel subject instances while optionally changing the surrounding scene through descriptive prompts.Examples include color changes of a car, and other property modifications are possible.
  • Property Modification: Material modifications, such as rendering a dog as stone, are possible but vary in difficulty according to the prior of the base generation model.The passage explicitly notes that some property modifications are harder than others.
  • Comic Book Generation: DreamBooth supports non-photorealistic generation, including drawn media and a full comic with a persistent character across descriptively prompted frames.The comic examples include a cartoon character performing actions and speaking.

Additional Experiments

Additional experiments show that prior preservation loss maintains class variability, input-image needs depend on subject rarity, and fine-tuning super-resolution models with reduced noise augmentation improves subject-detail fidelity. Qualitative comparisons further evaluate the method against prior personalization and prompt-engineering approaches.

  • Prior preservation loss: Prior preservation loss preserves the variability of the class prior, avoiding language drift and generating new subject instances under generic class prompts.A naively fine-tuned model generates the subject dog for “a dog,” whereas the proposed loss preserves variability.
  • Number of input images: Common subjects such as the selected Corgi can sometimes be captured with only one or two input images, whereas rarer objects such as the backpack require more samples.The conclusions are supported by DINO subject-fidelity and CLIP-T prompt-fidelity results in Tables 5 and 6.
  • Super-resolution fine-tuning: Fine-tuning super-resolution models is necessary to avoid artifacts and hallucinated subject details that can hurt identity preservation.Without fine-tuning, SR models may be unfamiliar with instance-specific textures or may hallucinate incorrect or missing features.
  • Super-resolution fine-tuning: Reducing noise augmentation from 10^-3 to 10^-5 during 256 × 256 SR fine-tuning recovers fine-grained subject details and improves fidelity.The original noise level caused blurring of high-frequency patterns, while the lower level improved their reproduction.
  • Comparisons: Additional qualitative comparisons evaluate the method against Gal et al., vanilla Imagen with prompt engineering, and DALL-E 2 for subject recontextualization.The comparisons include a headless sculpture, cat toy, and a distinctive yellow alarm clock.

Societal Impact

The project aims to help users reconstruct personal subjects in varied contexts, addressing attribute biases in general text-to-image synthesis. It also raises concerns that malicious parties could use personalized images to mislead viewers, motivating continued investigation.

  • Benefits: The approach helps users synthesize personal animals and objects in different contexts and better reconstruct desirable subjects despite potential biases in general text-to-image models.The authors frame this as an effective tool for personalized subject synthesis.
  • Risks: Malicious parties might use personalized generated images to mislead viewers, paralleling concerns already associated with other generative models and content-manipulation techniques.The paper identifies misuse as a societal-impact risk rather than presenting it as unique to this method.
  • Future research: Future research in generative modeling, particularly personalized generative priors, should continue investigating and revalidating these concerns.This recommendation follows the paper’s discussion of potential misuse and broader content-manipulation risks.
Loading 2208.12242v2…