PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamArtist++: Controllable One-Shot Text-to-Image Generation via Positive-Negative Adapter

Ziyi Dong, Pengxu Wei, Liang Lin

arXiv:2211.11337v4cs.CVcs.CLcs.MM

TL;DR

Personalizing text-to-image generation is difficult when new concepts are hard to describe and existing example-based methods struggle to preserve reference characteristics while generating diverse, high-quality images from one example. DreamArtist jointly learns positive and negative components from one reference using a pre-trained diffusion model. The paper reports substantial improvements over existing methods across content quality, style similarity, diversity, and detail quality.

  • Problem

    Existing one-shot personalization methods can overfit to a reference, limiting diversity, while prompt-tuning may fail to render details specified by prompt keywords.

    Source: §3.2 · p. 5

  • Method

    DreamArtist jointly learns positive and negative embeddings and adapters on a pre-trained diffusion model to capture reference characteristics and rectify positive components.

    Source: §1 · p. 2

  • Results

    DreamArtist substantially outperforms existing methods in content quality, style similarity, diversity, and detail quality.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DreamArtist generates high-quality, diverse images from one reference that can be controlled with additional text descriptions.

    Source: §7 · pp. 14–15

    Combining separately learned entity concepts can cause feature interference and make the concepts difficult to render as distinct entities.

    Source: §8 · p. 15

Abstract

from arXiv · show

State-of-the-arts text-to-image generation models such as Imagen and Stable Diffusion Model have succeed remarkable progresses in synthesizing high-quality, feature-rich images with high resolution guided by human text prompts. Since certain characteristics of image content \emph{e.g.}, very specific object entities or styles, are very hard to be accurately described by text, some example-based image generation approaches have been proposed, \emph{i.e.} generating new concepts based on absorbing the salient features of a few input references. Despite of acknowledged successes, these methods have struggled on accurately capturing the reference examples' characteristics while keeping diverse and high-quality image generation, particularly in the one-shot scenario (\emph{i.e.} given only one reference). To tackle this problem, we propose a simple yet effective framework, namely DreamArtist, which adopts a novel positive-negative prompt-tuning learning strategy on the pre-trained diffusion model, and it has shown to well handle the trade-off between the accurate controllability and fidelity of image generation with only one reference example. Specifically, our proposed framework incorporates both positive and negative embeddings or adapters and optimizes them in a joint manner. The positive part aggressively captures the salient characteristics of the reference image to drive diversified generation and the negative part rectifies inadequacies from the positive part. We have conducted extensive experiments and evaluated the proposed method from image similarity (fidelity) and diversity, generation controllability, and style cloning. And our DreamArtist has achieved a superior generation performance over existing methods. Besides, our additional evaluation on extended tasks, including concept compositions and prompt-guided image editing, demonstrates its effectiveness for more applications.

1 Introduction

Text-to-image models struggle with novel concepts and styles that are difficult to describe, while few-shot personalization methods can sacrifice diversity or prompt fidelity. DreamArtist learns from one reference using jointly optimized positive and negative components, and experiments report improvements over existing methods.

1 Introduction

  • DreamBooth’s fine-tuning with a few reference images often yields monotonous, low-diversity images, while TI often fails to faithfully render prompt-keyword details, even when references increase from one-shot to 3–5 instances.Multiple references may introduce unexpected ambiguities, and gathering object instances requires additional effort.Source: §1 · p. 2
  • DreamArtist learns a new concept from one reference image using a pre-trained diffusion model and positive-negative prompt tuning.Positive components capture reference characteristics, while negative components rectify limitations of the positive components.Source: §1 · p. 2
  • DreamArtist is presented as enabling creative image generation from a single reference image.Figure 2 compares DreamArtist with TI [27] on rendering visual content specified by text descriptions; Figure 3 contrasts one-reference DreamArtist generation with TI [27] and DreamBooth [28] using 3–5 references.Source: §1 · p. 2, Figure 2 · p. 3, Figure 3 · p. 3
  • DreamArtist’s positive-negative adapter combines positive guidance with complementary negative rectification for diverse, highly controllable generation.The positive part captures salient reference characteristics; the negative part helps rectify inadequacies in the positive components.Source: §1 · pp. 2–3
  • DreamArtist substantially outperforms existing methods in content quality, style similarity, diversity, and detail quality across natural and anime data.The experiments include qualitative and quantitative evaluations.Source: §1 · p. 3

2 Related Work

The related work describes diffusion models and guidance methods, then reviews text-to-image control and personalization approaches. It positions DreamArtist as a one-image alternative to methods that use 3–5 images and can overfit or offer poor controllability.

2.1 Diffusion-Based Generative Models

  • Diffusion models generate images through iterative denoising, but DDPM requires thousands of iterations; DDIM, K-Diffusion, and DPM solver were developed to accelerate sampling.Classifier guidance uses an auxiliary classifier, while classifier-free guidance learns conditional and unconditional generation.Source: §2.1 · p. 3

2.2 Text-to-Image Synthesis

  • Text-to-image diffusion models support high-quality generation, but prior spatial and image-text controls can struggle with user-given patterns and require complex descriptions.The section also describes Stable Diffusion as enabling high-resolution, high-quality text-to-image models.Source: §2.2 · p. 4

2.3 Few-Shot Text-to-Image Generation

  • Prior methods include TI prompt-tuning, DreamBooth fine-tuning, and an adapter fine-tuning a small parameter subset; they learn object features from few images but can overfit and offer poor controllability.Those methods require 3-5 images, while the authors’ method requires only 1 image to generate highly controllable personalized features usable with complex descriptions.Source: §2.3 · p. 4

3 Preliminary

Latent Diffusion Models perform denoising in encoded feature space. The section explains how existing few-shot methods overlook classifier-free guidance and the mismatch between reference and pretraining distributions, which can impair generation.

3.1 Latent Diffusion Model

  • LDM encodes images into feature space and uses a denoising U-Net to denoise the feature map, with text features conditioning generation through cross-attention.A pretrained AutoEncoder encodes and decodes images, and a pretrained text encoder produces the text features.Source: §3.1 · p. 4
  • The LDM training loss uses a fixed AutoEncoder while learning only the denoising network ϵθ.The passage identifies the loss as an ℓ2 loss.Source: §3.1 · p. 4

3.2 Deficiencies in Existing Methods

  • Prior few-shot generation methods overlooked classifier-free guidance during training, a gap linked to poor controllability or low-quality generated images.The section identifies CFG as a mechanism used by most text-to-image diffusion models, including LDM.Source: §3.2 · p. 5
  • TI fits pθ(w_t|S_p) while ignoring the difference between p(w) and p(z), which can produce incorrect guidance directions, unexpected features, and artifacts.The reference-image distribution p(w) differs from the pretraining distribution p(z).Source: §3.2 · p. 5
  • With a single reference point, maximizing pθ(w_t|S_p) can cause severe overfitting and limited diversity; fine-tuning and adapter-based methods likewise ignore the distribution distinction.The section names DreamBooth and LoRA as examples of those approaches.Source: §3.2 · p. 5

4 Methodology

DreamArtist jointly learns positive and negative prompts and adapters from one reference image to balance reference fidelity with text-guided controllability. It adds pixel-level reconstruction to improve detail and color learning.

4 Methodology

  • The method is designed to learn a new concept from a single reference image using a pretrained diffusion model.The paper frames this as balancing preservation of reference characteristics with text-guided controllability.Source: §4 · p. 5

4.1 Positive-Negative Prompt-Tuning

  • Positive-negative prompt tuning separates reference characteristics to capture from characteristics to exclude, learning both pseudo-words together.The positive branch captures desired reference traits; the negative branch supplies traits to exclude for rectification.Source: §4.1 · pp. 5–6

4.2 Reconstruction Constraint for Detail Enhancement.

  • A pixel-level reconstruction loss supplements feature-space constraints because prompt tuning alone can miss fine-grained image details.The loss compares the decoded approximate image with the reference using an ℓ1 loss.Source: §4.2 · p. 6

4.3 Positive-Negative Adapter

  • Jointly optimized positive and negative adapters improve fine-detail preservation while retaining diversity and textual controllability from one reference image.Generator adapters preserve fine details and learn features missing from pretraining; text-encoder adapters support precise control through descriptions.Source: §4 · p. 5, §4.3 · p. 7

5 Experiments

Experiments compare one-shot DreamArtist with TI, DreamBooth, and LoRA on natural and anime images, using similarity, diversity, detail, and controllability measures. The reported comparisons favor DreamArtist's balance of fidelity, diversity, and text control.

5.1 Experimental Settings

  • The evaluation uses image/style similarity, image diversity, detail, and controllability measures; similarity can trade off against diversity.The settings include LAION-2B for natural images and Danbooru for anime; the authors note that similarity to reference images can indicate overfitting bias.Source: §5.1 · pp. 7–8

5.2 One-Shot Text-guided Image Synthesis

  • DreamArtist produces realistic, detailed, and diverse outputs, while TI and DreamBooth show limited diversity and detail or overfitting to the reference.The comparison covers one-shot generation on natural and anime images.Source: §5.2 · p. 8
  • 0.89 CAS for DreamArtist versus 0.24 for DreamBooth and 0.37 for TI; DreamArtist also maintains similarity, diversity, and controllability.The authors report that DreamBooth and LoRA score better on LPIPS and style loss due to overfitting, while TI's higher CFV accompanies severe artifacts.Source: §5.2 · p. 8

6 Method Evaluation and Analysis

The analyses examine how γ, reconstruction, adapters, and embedding lengths affect DreamArtist, alongside controllability, human evaluation, and concept composition. They report trade-offs in fidelity, diversity, and control, with positive results for detail preservation and compositional generation.

6.1 Evaluation on γ

  • In Tab. 4, smaller γ yields better LPIPS and style loss, indicating greater fidelity to the input image; empirical results use γ = 3, while image-style learning uses γ = 5.Source: §6.1 · p. 9

6.2 Ablative Study

  • Without reconstruction loss, the model struggles with low-level color and detail; adding adapters improves fine-detail preservation without reported overfitting.The adapter comparison also reports improved image quality and feature controllability; Fig. 12 visualizes the comparison.Source: §6.2 · pp. 9–10, Figure 12 · p. 11, Figure 13 · p. 12

6.3 Analysis on Negative Branch

  • The negative branch rectifies deficiencies in positive-only generations, whose outputs can lack style and detail, and progressively improves image quality.Fig. 10 shows generations from positive prompts alone and with the negative branch.Source: §6.3 · p. 10, Figure 10 · p. 10

6.4 Analysis on Concept Decoupling

  • Learned pseudo-words and adapters can render reference characteristics such as light, material, and style in different text contexts.The authors describe this as implicit decoupling that supports controllability.Source: §6.4 · p. 11, Figure 17 · p. 13

6.5 Human Evaluation

  • 34.5% of participants mistook DreamArtist outputs for real images, exceeding the stated 30% Turing-test requirement; 52.31% and 83.13% preferred its natural and anime images.The Turing test involved 700 participants.Source: §6.5 · pp. 12–13

6.6 Analysis on Embedding Length

  • A positive embedding length of 3 balances learned visual features with text, while longer lengths can overfit; negative lengths of 3 or 6 are reported as optimal.Excessively long negative embeddings can degrade quality; length 3 is used for both embeddings for consistency.Source: §6.6 · p. 13

6.7 Extended Task: Concept Compositions

  • Combining learned pseudo-words can compose objects and styles in natural and anime images, with each component rendered in the output.Examples include a robot in an ancient-painting style and a dog with a robot body; the text contrasts this with TI's difficulty combining pseudo-words.Source: §6.7 · pp. 13–14

7 Conclusions

DreamArtist frames one-shot text-to-image generation around positive-negative prompt-tuning, with learned pseudo-words intended to support diverse, high-quality outputs controlled by added descriptions. The authors report substantial improvements over existing methods.

7 Conclusions

  • DreamArtist introduces one-shot text-to-image generation using positive-negative prompt-tuning to learn and rectify generation results from one reference image.The learned pseudo-words are reported to generate high-quality and diverse images.Source: §7 · p. 14
  • DreamArtist substantially outperforms existing methods in qualitative and quantitative analyses.The method is also described as highly controllable with complex descriptions.Source: §7 · p. 15
  • Learned pseudo-words can be controlled with additional text descriptions.The authors describe combining them with complex descriptions as a source of flexibility and potential for deployment with other models.Source: §7 · p. 15

8 Limitations and future work.

The authors identify cross-domain transfer and entity-concept composition as limitations. They propose continual learning for composition issues and identify domain shift as future work.

8 Limitations and future work.

  • Real-world-trained DreamArtist features transfer only partially to an anime base model, whose biases heavily influence generated outputs.The authors identify domain shift between reference images and pre-training images as an issue to address in future work.Source: §8 · p. 15
  • Composing separately learned entity concepts can cause interference, fusion of features, and failure to render distinct entities.The authors propose continual text-to-image generation with continual learning as a way to address this issue.Source: §8 · p. 15

Related papers

Loading 2211.11337v4…