PaperlayerView exact revision on arXiv

Source-linked AI summary

BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

Dongxu Li, Junnan Li, Steven C. H. Hoi

arXiv:2305.14720v2cs.CVcs.AI

TL;DR

Subject-driven generation is limited by lengthy per-subject fine-tuning and difficulties preserving subject fidelity. BLIP-Diffusion pre-trains a multimodal encoder and diffusion model to combine subject images with text prompts, achieving zero-shot generation and up to 20x faster fine-tuning. It also supports extensions for structure-controlled generation and subject-driven editing, although common subject-driven and diffusion-model failures remain.

  • Problem

    Existing subject-driven models require lengthy fine-tuning and struggle to preserve subject fidelity when learning representations aligned with text.

  • Method

    BLIP-Diffusion uses a two-stage strategy that pre-trains BLIP-2 for text-aligned visual features and trains diffusion to generate subject renditions from those representations.

  • Results

    BLIP-Diffusion enables zero-shot generation and up to 20x faster fine-tuning than prior methods while maintaining comparable generation quality to DreamBooth.

  • Takeaways & Limitations

    The model combines subject and text control and can work with ControlNet and prompt-to-prompt for structure-controlled generation and image editing.

  • Takeaways & Limitations

    The model can produce incorrect context, overfit to training data, and inherit failures in text understanding and fine-grained composition from diffusion models.

Abstract

from arXiv · show

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/.

1 Introduction

BLIP-Diffusion addresses slow fine-tuning and subject-fidelity challenges by introducing pre-trained multimodal subject representations for zero-shot and few-step generation.

  • Existing subject-driven methods optimize placeholder embeddings through hundreds or thousands of fine-tuning steps for each new subject.
  • BLIP-Diffusion uses BLIP-2 to encode a subject image and category text into a text-aligned representation for latent diffusion.
  • Its two-stage pre-training first aligns visual features with text, then trains diffusion to generate novel subject renditions from those features.
  • 40-120 fine-tuning steps provide up to 20x speedup over DreamBooth, while the model also supports zero-shot generation and extensions with ControlNet and prompt-to-prompt.

2 Related Work

The section connects latent diffusion conditioning with subject-driven generation methods and introduces the paper’s two-stage pre-training context.

  • Latent diffusion trains a neural model to predict added noise from a noisy latent and text prompt, then decodes the denoised latent into an image.
  • BLIP-Diffusion additionally conditions latent diffusion on a subject representation, creating multimodal image-generation control.
  • The proposed pre-training illustration has a multimodal representation stage followed by subject representation learning using synthesized subject-background pairs.
  • Textual Inversion optimizes a placeholder embedding, while DreamBooth also fine-tunes the diffusion model to improve expressiveness and subject fidelity.

3 Method

BLIP-Diffusion learns text-aligned, subject-specific visual representations and injects them into diffusion prompts, while supporting structure-controlled generation and subject-driven editing.

  • BLIP-Diffusion adapts BLIP-2 to provide generic pre-trained subject representations for multimodal control.
  • Multimodal Representation Learning with BLIP-2: The first pre-training stage learns text-aligned generic image features so subject embeddings can cooperate with Stable Diffusion’s text embeddings.
  • Multimodal Representation Learning with BLIP-2: The multimodal encoder uses a frozen image encoder and Q-Former, reducing its output from 32 to 16 query features because 32 features can overpower CLIP text embeddings.
  • Subject Representation Learning with Stable Diffusion: The projected subject representation is appended to text-token embeddings as a soft visual subject prompt for diffusion generation.
  • Applications: ControlNet adds structure conditions without retraining its parameters, while attention control enables subject-driven editing through cross-attention manipulation.
  • Subject Representation Learning with Stable Diffusion: Prompted context generation composes subjects with random backgrounds to separate subject information from background context during pre-training.

4 Experiments

BLIP-Diffusion demonstrates zero-shot and efficient fine-tuned subject-driven generation, with strong subject fidelity, prompt relevance, and multimodal control. Experiments further show useful extensions to editing, style transfer, subject interpolation, and representation analysis.

  • Main Qualitative Results: 40-120 fine-tuning steps provide up to 20x greater efficiency than previous work while supporting high-fidelity subject re-contextualization and modification.The model also supports simultaneous structure and subject control when combined with ControlNet.
  • Comparisons on DreamBooth Dataset: BLIP-Diffusion achieves comparable or better generation quality than DreamBooth while requiring significantly fewer fine-tuning iterations on the DreamBooth dataset.It also achieves significantly better subject fidelity than Textual Inversion, Re-Imagen, and InstructPix2Pix.
  • Comparisons on DreamBooth Dataset: DINO, CLIP-I, and CLIP-T evaluations are consistent with the qualitative comparisons, showing superiority to Textual Inversion and Re-Imagen and comparable performance to DreamBooth.Fine-tuning improves subject alignment and average image-text alignment, although limited visual diversity can cause overfitting to target inputs.
  • Ablation Studies: Multimodal representation learning is critical because it bridges the representation gap between subject embeddings and text prompt embeddings.Freezing the diffusion text encoder worsens interaction with subject embeddings, while subject text and random embedding dropping improve results and generation ability.
  • Subject Representation Visualization: The learned subject embeddings capture both fine-grained local features and holistic visual aspects through different cross-attention patterns.Some embeddings focus on local features, while others encode holistic subject visuals.
  • Zero-shot Subject-driven Image Manipulation: The pre-trained subject representation enables zero-shot subject-driven manipulation, including style transfer, subject interpolation, and image editing.Interpolation blends subject appearances along embedding trajectories, while style transfer uses guiding subjects and edge-guided ControlNet.

5 Limitations and Failure Cases

BLIP-Diffusion shares common subject-driven generation failures and also inherits weaknesses from its underlying diffusion model, including prompt and composition errors.

  • The model can synthesize incorrect contexts and overfit to its training set, reflecting common subject-driven generation failures.
  • Because it inherits weaknesses of the underlying diffusion model, it may fail to understand text prompts and fine-grained composition relations.
  • The paper presents failure examples in Figure 10 and describes the technique as generic for future diffusion-model development despite these limitations.

6 Conclusion

BLIP-Diffusion combines pre-trained multimodal subject representation with diffusion generation to support zero-shot and efficient fine-tuned subject-driven synthesis. It also supports subject interpolation and integration with established generation and editing techniques.

  • BLIP-Diffusion uses built-in multimodal control powered by BLIP-2 and a two-stage strategy to learn progressively multimodal subject representations.
  • The model supports high-fidelity zero-shot generation and efficient fine-tuned subject-driven generation, with up to 20x fine-tuning speedup over prior methods at comparable quality.
  • Interpolating subject representations blends subject appearance while fitting the result coherently into a new context using the same denoising and decoder network.
  • Figure 10 presents example failure generations, with the subject images used for fine-tuning shown on the left.

A.1 Broader Impact

The paper discusses misuse and safety risks of image generation, including fake images and biased or inappropriate content. It describes data-removal and moderation measures intended to reduce these risks.

  • Person-related subjects were removed from training data, making the model weaker at generating fake images using people as subject controls.
  • Because Stable Diffusion was trained on web-scraped datasets, BLIP-Diffusion inherits risks of biased content, social stereotypes, and inappropriate NSFW outputs.
  • The paper recommends NSFW detectors, careful output inspection, content moderation, and regulation before user-facing deployment.

A.2 Competing Methods

The paper compares BLIP-Diffusion with fine-tuning and retrieval-augmented subject-driven methods on DreamBench, and qualitatively with an image-editing method. The competitors differ in tuning requirements and subject representation.

  • The evaluation compares fine-tuning-based and retrieval-augmented subject-driven models on DreamBench, plus qualitative comparisons with InstructPix2Pix.
  • 3,000 training steps are required by Textual Inversion to learn a new concept, taking around 30 minutes on an A100 GPU.
  • DreamBooth optimizes both a placeholder embedding and U-Net parameters, with a total budget of around 800 steps and roughly 6 minutes on an A100 GPU.
  • Re-Imagen requires no tuning but significantly underperforms other models and is not publicly available for qualitative comparison.
  • InstructPix2Pix edits a source image from an instruction and can support subject re-contextualization or property modification without explicitly representing subjects.

A.3 Evaluation Metrics

BLIP-Diffusion evaluates subject-driven generation with DINO, CLIP-I, and CLIP-T, which jointly measure subject fidelity and image-text alignment. Considering these metrics together avoids favoring copying or text alignment alone.

  • DINO and CLIP-I measure subject fidelity, while CLIP-T measures image-text alignment.DINO uses pairwise DINO embeddings, CLIP-I uses pairwise CLIP image embeddings, and CLIP-T compares prompt and image CLIP embeddings.
  • DINO is preferred for subject-fidelity evaluation because it is sensitive to differences between subjects of the same class.
  • Considering DINO, CLIP-I, and CLIP-T jointly helps avoid biased conclusions about subject-driven generation quality.Copying training images can score highly on DINO and CLIP-I but low on CLIP-T, whereas vanilla text-to-image generation can show the opposite pattern.

A.4 Pre-training Datasets

The appendix describes the datasets, optimization and inference settings, evaluation tables, qualitative checks, and stylization applications used to assess BLIP-Diffusion. Experiments cover zero-shot and fine-tuning setups on DreamBench, alongside controlled visual applications.

  • Pre-training Datasets: Multimodal representation learning uses 129M images from COCO, Visual Genome, CC3M, CC12M, SBU, and LAION400M, with synthetic CapFilt captions for web images.
  • Pre-training Datasets: Subject representation learning uses a filtered OpenImage-V6 subset that excludes unsuitable object configurations, viewpoints, aspect ratios, and object sizes.
  • Implementation Settings: Fine-tuning uses AdamW at learning rate 5e-6 on a single A100 GPU, while inference uses PNDM for 100 denoising steps with guidance scale 7.5.
  • Evaluation Results: Table 3 reports DreamBench fine-tuning steps, while Tables 4 and 5 report average metrics for zero-shot and fine-tuning setups.Metrics in Tables 4 and 5 average 10 experiment runs for each subject.
  • Qualitative Results: Figures 11–13 provide qualitative DreamBench results by mixing one genuine subject image among generated renditions to assess subject fidelity and photorealism.
  • Applications: Figures 14–15 demonstrate zero-shot subject-guided stylization by transferring a reference subject’s appearance style to other subjects with edge-driven ControlNet.
Loading 2305.14720v2…