PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models

Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, Jinwoo Shin

arXiv:2402.12004v2cs.CV

TL;DR

Low-shot personalization can impair the alignment and composition of text-to-image diffusion models with pretrained concepts. DCO limits fine-tuning deviation from the pretrained model, and the paper reports improved fidelity trade-offs and composition of customized subjects and styles. Its approach also has additional fine-tuning and sampling costs, and multi-subject composition is unstable for semantically similar subjects.

  • Problem

    Low-shot personalized T2I models can lose textual alignment and compositional capability, particularly when trained from only one or two reference images.

    Source: §1 · p. 1

  • Method

    DCO fine-tunes by controlling deviation from the pretrained model, with consistency guidance sampling to adjust the trade-off between reference-image consistency and textual alignment.

    Source: §1 · p. 2

  • Results

    DCO achieves a superior Pareto frontier in image-text similarity and image similarity, and its separately fine-tuned subject and style models can be merged more effectively than regular fine-tuned models post-optimized with methods such as ZipLoRA.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    DCO supports composing independently customized subject and style models, including generating a custom subject in a custom style.

    Source: §1 · p. 2, §6 · p. 10

    DCO multi-subject composition is unstable for semantically similar subjects, which can mix concepts and reduce subject consistency.

    Source: E Limitations · p. 22

Abstract

from arXiv · show

Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, can generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model or other fine-tuned models. To address this, we propose a novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation between fine-tuning and pretrained models to retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that our method positions itself on a superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom subjects in a custom style by composing separately customized subject and style models. Notably, we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.

1 Introduction

Low-shot personalization can create subject or style concepts, but fine-tuned models may lose textual alignment and compositional capability. The paper proposes DCO to limit deviation from the pretrained model and reports improved fidelity trade-offs and composition.

Motivation

  • Low-shot fine-tuning can reduce textual alignment and compositional generation capability, especially with only one or two reference images.Fine-tuning on a few images teaches new concepts but can impair composition with pretrained concepts.Source: §1 · p. 1
  • The authors hypothesize that knowledge forgetting causes low-shot fine-tuning to fail to recontextualize new concepts with pretrained concepts.Prior preservation can retain generalization but reduce subject fidelity, with a larger loss when fewer reference images are available.Source: §1 · p. 2

Contribution

  • DCO fine-tunes without additional data by controlling deviation from the pretrained model, and frames its objective as an implicit reward for consistency with reference images.The paper also proposes consistency guidance sampling to trade off subject consistency and textual alignment.Source: §1 · p. 2

Empirical results

  • DCO places the prompt-fidelity and subject-fidelity Pareto frontier above regular fine-tuning and prior-preservation baselines, and supports merging separately customized subject and style models.The merged models outperform post-optimized regular fine-tuned models in prompt and subject fidelity.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related work covers personalized text-to-image synthesis, methods for preserving pretrained knowledge, and reward-based fine-tuning. The paper distinguishes DCO as a data-free regularization approach and identifies consistency as a reward for personalization.

Personalized T2I synthesis

  • Personalized T2I methods adapt models from a few images, using approaches including soft prompts, LoRA, and adapter tuning.Prior preservation instead adds a class-specific dataset synthesized from the pretrained model; the paper describes its own alternative as regularizing with the pretrained model without such a dataset.Source: §2 · pp. 2–3

Fine-tuning T2I diffusion models with rewards

  • Reward-based T2I fine-tuning has used human preferences or aesthetic scores through reward weighting, reinforcement learning, or differentiation through reward models.Direct preference optimization has also been adapted to paired preference data without explicit reward models.Source: §2 · p. 3
  • The paper presents itself as the first study of consistency as a reward function for T2I personalization.This claim is made in the context of earlier reward-based fine-tuning approaches.Source: §2 · p. 3

3 Preliminaries

The preliminaries introduce diffusion and text-conditioned generation, then describe personalization and parameter-efficient adaptation. They also outline the prior-preservation approach that DCO is positioned against.

Diffusion models

  • A diffusion model adds noise to data over timesteps and generates by denoising random noise, with noise-prediction networks commonly used for training.The text describes the forward process and the corresponding score-network parameterization.Source: §3 · p. 3

Text-to-Image diffusion models

  • Text-to-image diffusion models condition image generation on text embeddings and commonly use classifier-free guidance to interpolate unconditional and conditional predictions.Higher guidance scale improves image-text alignment at the cost of image fidelity.Source: §3 · p. 3

Personalizing T2I models

  • DreamBooth personalizes a T2I model by fine-tuning on a few subject images, while prior preservation adds auxiliary images from the same class.The auxiliary images are used to retain pretrained prior knowledge.Source: §3 · p. 3

Parameter efficient fine-tuning

  • PEFT approaches include LoRA, which represents weight updates with a low-rank decomposition, and Textual Inversion, which learns a concept embedding.The passage also notes that PEFT methods are combined with DreamBooth for faster, more memory-efficient adaptation.Source: §3 · p. 3

4 Method

DCO regularizes low-shot personalization against the pretrained model, with an implicit consistency reward and an efficient noise-prediction implementation. The method also adds inference-time consistency guidance and recommends comprehensive reference captions to separate desired concepts from distractors.

4.1 Direct Consistency Optimization

  • DCO seeks personalized models that improve consistency with reference images while penalizing deviation from the pretrained model.The paper frames this objective as equivalent to optimizing an implicit consistency reward, avoiding explicit reward estimation from few samples and expensive RL fine-tuning.Source: §4.1 · pp. 4–5

4.2 Implementation and Analysis

  • The DCO implementation derives an efficient objective from the difference between fine-tuning and pretrained-model noise-prediction errors.Its gradient acts like diffusion-loss training scaled by an adaptive factor; when model deviation is large, the update is withheld. Unlike prior preservation, DCO regularizes on reference data and needs no auxiliary prior samples.Source: §4.2 · p. 5

4.3 Consistency Guidance Sampling

  • Consistency guidance adds a consistency term alongside text guidance, letting its weight trade off reference-image fidelity against prompt fidelity.Higher consistency guidance increases reference-image fidelity at the cost of prompt fidelity; lower weight can improve prompt fidelity.Source: §4.3 · pp. 5–6

4.4 Prompt Construction for Reference Images

  • Compact reference captions can cause the model to learn distractors such as background or style as part of the personalized concept.The figure contrasts a compact caption that produces a dog on a couch despite a lake prompt with a comprehensive caption that follows prompts more faithfully.Source: §4.4 · p. 6, Figure 2 · p. 6
  • Comprehensive captions describe the subject and other visual attributes, backgrounds, and styles to help associate attributes with text and separate them.For style customization, the paper recommends describing the subject so the model distinguishes style from subject.Source: §4.4 · p. 6

5 Experiments

Experiments evaluate DCO on subject and style personalization, their composition, and ablations using Stable Diffusion XL. Across these settings, the results compare fidelity and prompt alignment, including whether independently trained models can be merged.

5 Experiments

  • Experiments use Stable Diffusion XL and cover 30 DreamBooth subjects, StyleDrop style images, and combinations of separately personalized subject and style models.The subject dataset contains 4–6 images per subject.Source: §5 · p. 6, §5.1 · p. 6, §5.2 · p. 8

5.1 Subject Personalization

  • DCO generates custom subjects across visual attributes and styles; DreamBooth struggles with prompt following, while prior preservation improves alignment but often loses subject identity.DreamBooth’s overfitting to photographic style limits recontextualization into different outfits or styles.Source: Figure 3 · p. 7, §5.1 · p. 7
  • DCO lies on the upper-right frontier versus DreamBooth for subject customization and stylization, and improves image similarity over prior preservation while remaining comparable in image-text similarity.Figure 5a reports the averaged results; Figure 8 shows per-subject results.Source: Figure 5 · p. 8, §5.1 · p. 8

5.2 Style Personalization

  • DCO produces consistent styles without entangling them with reference-image content, while DreamBooth can import unwanted subject attributes into generated images.Style similarity is noisy because it can reflect the unexpected appearance of a subject in the style reference.Source: Figure 4 · p. 7, §5.2 · pp. 8–9

5.3 My Subject in My Style

  • DCO Merge produces high subject and style consistency with a simple arithmetic merge; DB Merge struggles with image quality, while DB ZipLoRA can lose subject or style consistency.The merge combines independently fine-tuned subject and style LoRAs.Source: Figure 6 · p. 9, §5.3 · p. 9
  • 0.462 vs. 0.386, 0.406 in subject similarity; 0.773 vs. 0.430, 0.729 in image-text similarity; and 0.651 vs. 0.672, 0.662 in style similarity for DCO Merge versus DB Merge and DB ZipLoRA.DCO Merge leads on subject and image-text similarity while retaining competitive style similarity.Source: §5.3 · p. 9, Table 1 · p. 10

5.4 Ablation Study

  • Across tested β values, DCO models form a better Pareto frontier than DreamBooth, while larger β tends to trade subject identity for image-text alignment.β=1000 works well overall, though the optimal value may vary by reference dataset.Source: Figure 7 · p. 10, §5.4 · p. 10
  • DCO reduces noise deviation from the pretrained model relative to DreamBooth, and increasing β reduces that deviation further.The analysis averages measurements over 100 random noises at each timestep.Source: Figure 7 · p. 10, §5.4 · p. 10

6 Conclusion

DCO controls how far low-shot fine-tuning deviates from the pretrained model. The conclusion reports stronger alignment and sample quality, improved consistency–similarity trade-offs, and easier composition of separately fine-tuned models.

6 Conclusion

  • DCO controls deviation from the pretrained model while learning new concepts, retaining pretrained capabilities during low-shot fine-tuning.Source: §6 · p. 10
  • DCO improves image-text alignment and sample quality over regular diffusion fine-tuning, and with consistency guidance achieves the strongest image-text and image-similarity Pareto frontier.Source: §6 · p. 10
  • DCO makes independently fine-tuned subject and style models easier to compose, and merged DCO models outperform post-processing methods such as ZipLoRA.Source: §6 · p. 10

A Derivation

The appendix derives Eq. (9) from a deviation definition and an epsilon-prediction expression for KL divergence, then discusses weighted deviation and sampling results.

A Derivation

  • The derivation defines deviation for a subseries z_t:1 and uses its relationship to KL divergence expressed with epsilon-prediction to establish Eq. (9).Source: A Derivation · p. 15
  • A weighted deviation is described as matching the general weighted epsilon-prediction loss through a monotonic weighting function.Source: A Derivation · p. 15
  • Figure 8 plots image similarity against image-text similarity for subject customization and stylization, distinguishing consistency-guidance sampling from conventional sampling.Source: Figure 8 · p. 16

B.1 Full comparison

The full comparison reports trade-offs across personalization baselines, guidance scales, and single-image examples. It also discusses merging personalized models and textual-embedding details.

B.1 Full comparison

  • DreamBooth with half the training steps improved image-text similarity but reduced image similarity, and its frontier did not improve over DreamBooth.The reported customization and stylization image-text similarities were 0.842 vs. 0.754 and 0.645 vs. 0.468; image similarities were 0.575 vs. 0.754 and 0.630 vs. 0.638.Source: §B.1 · p. 16
  • Reducing prior-preservation weight improved image similarity but lowered image-text similarity, without improving the consistency-guided frontier.For customization, image similarity changed from 0.532 to 0.547 while image-text similarity changed from 0.864 to 0.850; for stylization, the respective changes were 0.424 to 0.441 and 0.851 to 0.839.Source: §B.1 · p. 16
  • Users preferred DCO over DreamBooth for prompt fidelity while subject fidelity was maintained, and most favored DCO over prior-preservation DreamBooth for subject fidelity.The study collected 2,160 responses across comparisons; Table 2 reports the percentage of judgments favoring DCO.Source: §B.1 · p. 17, Table 2 · p. 17

B.2 Effect of Consistency Guidance Scale.

  • The optimal consistency-guidance scale varies by reference dataset and prompt, leaving the choice to user preference.The passage gives 5.0 and 2.0 as contrasting examples of suitable scales.Source: §B.2 · p. 18

B.3 1-shot personalization

  • On single synthetic reference images, DCO generated varied actions, backgrounds, and styles while preserving subject consistency, including conversions between photographic and animated styles.The experiments used images generated by SDXL and DALLE–3 and fine-tuned for 1,000 steps.Source: §B.3 · p. 18

C Extended Related Work

  • DCO models had cosine similarities comparable to DreamBooth models yet could be arithmetically merged without observed significant generation interference, questioning cosine similarity as a compatibility measure.The authors leave exploration of a different compatibility metric for future work.Source: Figure 10 · p. 18, C Extended Related Work · p. 19

D Experimental detail

  • New textual embeddings are initialized from original embeddings for inserted class tokens, then optimized with the diffusion models.Source: D Experimental detail · p. 19

D.1 Dataset

The evaluation uses DreamBooth subjects and StyleDrop images, with comprehensive captions and prompts testing subject customization and stylization.

D.1 Dataset

  • DreamBooth subject personalization covers 30 pets and unique objects, while style personalization uses 10 StyleDrop images.Source: §D.1 · p. 19
  • Comprehensive captions describe visual attributes, backgrounds, and styles alongside the subject, with LLaVA descriptions filtered and edited for non-visual details.Source: §D.1 · p. 19, Figure 11 · p. 20

D.2 Evaluation prompt

  • Evaluation prompts test subject attribute or background changes and subject stylization, with prompts tailored to specific object and subject categories.The examples include cube-shaped subjects, beach backgrounds, watercolor or origami styles, clocks, robots, cats, dogs, and wolf plushies.Source: §D.2 · p. 19, Figure 12 · p. 20

D.3 Additional implementation detail

The implementation uses LoRA fine-tuning with specified ranks and training and sampling settings, and the authors report robustness to training duration relative to DreamBooth.

D.3 Additional implementation detail

  • LoRA rank is 32 for subject personalization and 64 for style personalization.Training uses a batch size of 1 on a single A100 40GB GPU, for up to 2000 optimization steps.Source: §D.3 · p. 20
  • The authors report DCO is robust to training-step length, whereas DreamBooth often requires early stopping to prevent overfitting.The passage does not give a quantitative comparison of training duration.Source: §D.3 · p. 20
  • Sampling uses DDIM for 50 steps with a CFG guidance scale of 7.5 throughout the experiments.Source: §D.3 · p. 20

D.4 Evaluation metric

Evaluation covers subject customization and stylization using image similarity and image-text similarity metrics, with multiple sampling settings used to show trade-offs rather than relying on one score.

D.4 Evaluation metric

  • DINOv2 measures image similarity as mean cosine similarity between reference-image and synthesized-image embeddings, while SigLIP measures image-text similarity.For style personalization, image similarity is computed using the SigLIP image encoder; the supplied SigLIP formula is incomplete.Source: §D.4 · p. 20
  • The evaluation prompts cover subject customization by changing subject attributes or background and subject stylization by changing visual style.The listed prompts illustrate both categories, with the subject class token filling each {} placeholder.Source: Table 3 · p. 21, §D.4 · p. 21
  • Because image similarity can reach 1.0 through overfitting and image-text similarity can be high under underfitting, the authors report trends across sampling settings.They vary sampling parameters, such as consistency-guidance scale, and show Pareto frontiers rather than a single data point.Source: §D.4 · p. 21
  • In multi-subject composition, DCO outperforms DreamBooth and DreamBooth with prior preservation loss, but learned concepts can mix during generation.The reported mixing occurs more often for similar concepts: dog-dog is harder than dog-cat, which is harder than dog-toy.Source: Figure 13 · p. 21

E Limitations

DCO adds computational cost, and multi-subject composition remains unstable when the learned subjects are semantically similar.

E Limitations

  • DCO takes approximately 1.3 times longer per fine-tuning iteration than DreamBooth, while consistency-guidance sampling takes approximately 2 times longer than CFG sampling.The authors attribute the extra burden to pretrained-model inference during training and consistency-guidance inference during sampling.Source: E Limitations · p. 22
  • DCO can compose semantically distant subjects such as dog-car toy or dog-cat, but dog-dog compositions can mix concepts and produce subject inconsistency.The authors suggest limited disentanglement of semantically similar concepts as a partial explanation and say a better pretrained model could ameliorate the issue.Source: E Limitations · p. 22

F Broader Impact

The paper describes potential creative uses of T2I personalization while calling for responsible use to protect asset ownership and copyright. Its supplemental examples include one-shot personalization from synthetic images, and the checklist records decisions on reproducibility, statistical significance, and asset access.

F Broader Impact

  • Personalized T2I models could help creative directors generate visual assets from existing private visual assets, but responsible use is needed to protect ownership and copyright.Source: F Broader Impact · p. 22
  • DCO is demonstrated for one-shot subject personalization from synthetic SDXL and DALL·E-3 images, including recontextualization through prompts for accessories, backgrounds, or style.The SDXL examples show different actions and styles for a man and a pig; the DALL·E-3 example uses a baby otter image.Source: Figure 15 · p. 24, Figure 16 · p. 25

NeurIPS Paper Checklist

2. Limitations

  • The checklist reports that statistical significance information is not provided because experiments use the same commonly used random seed, while access to code and data is deferred until after acceptance.The checklist separately states that experimental details are specified and that a pseudocode implementation and Appendix D details are provided.Source: §2 · pp. 32–33

Related papers

Loading 2402.12004v2…