Source-linked AI summary
Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, Sungroh Yoon
TL;DR
Text-guided editing can miss the precise appearance of a user’s concept in reference images. Custom-Edit customizes a diffusion model with a few references before editing, and its results show improved source-reference trade-offs with language-relevant customization. The authors also identify settings where editing can fail, including undesired regions and complex backgrounds.
Problem
Text guidance can fail to capture the precise concept and fine-grained reference appearance users seek in image editing.
Source: §1 · p. 1Method
Custom-Edit customizes a diffusion model using a few reference images, then applies text-guided editing methods to source images.
Source: §1 · p. 1Results
Custom-Edit yields the best source-reference trade-off on P2P and SDEdit, while Textual Inversion has similar source similarity but lower reference similarity and Dreambooth has higher source but lower reference similarity.
Source: §4.2 · p. 4Takeaways & Limitations
The results support customizing language-relevant parameters for editing, and the paper offers design choices for customization and editing.
Source: §4.2 · p. 4, §5 · p. 4Custom-Edit sometimes edits undesired regions or fails on complex backgrounds, which the authors hypothesize relates to inaccurate Stable Diffusion attention maps and limited text controllability.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Text-to-image diffusion models can generate diverse, high-fidelity images based on user-provided text prompts. Recent research has extended these models to support text-guided image editing. While text guidance is an intuitive editing interface for users, it often fails to ensure the precise concept conveyed by users. To address this issue, we propose Custom-Edit, in which we (i) customize a diffusion model with a few reference images and then (ii) perform text-guided editing. Our key discovery is that customizing only language-relevant parameters with augmented prompts improves reference similarity significantly while maintaining source similarity. Moreover, we provide our recipe for each customization and editing process. We compare popular customization methods and validate our findings on two editing methods using various datasets.
1. Introduction
Text-guided editing can miss the fine-grained appearance of a user’s reference. Custom-Edit addresses this by customizing a diffusion model with a few references before text-guided editing.
1. Introduction
- Caption-based text guidance can fail to capture a reference’s fine-grained appearance, including concepts not encountered during large-scale training.The paper frames precise editing of a user’s unique concept as an open question.Source: §1 · p. 1, Figure 1 · p. 1
- Custom-Edit first customizes a model using a few reference images, then uses text-guided editing to transform images.The authors distinguish this image-to-image focus from prior customization work on random image generation.Source: §1 · p. 1
- Customizing language-relevant parameters with augmented prompts significantly enhances edited-image quality while addressing reference faithfulness and source-reference trade-offs.The paper presents design choices for customization and editing alongside its discussion of the trade-off.Source: §1 · p. 1
2. Diffusion Models
The paper describes a latent-space diffusion model conditioned on text embeddings. Its process combines noise prediction with a frozen text encoder and cross-attention conditioning.
2. Diffusion Models
- Stable Diffusion operates in a VAE latent space, reconstructing a clean latent representation from a perturbed one given text conditioning.The text condition is embedded with the CLIP text encoder.Source: §2 · p. 1
- The training objective minimizes squared error between added noise and the model’s predicted noise across time steps.The supplied equation expresses this noise-prediction objective.Source: §2 · p. 1
- A U-Net with attention blocks predicts noise, while text embeddings are projected to cross-attention keys and values and the text encoder remains frozen.The frozen encoder preserves its language-understanding capability.Source: §2 · pp. 1–2
- Custom-Edit’s overview separates reference-based model customization from source-image editing with the customized word.Customization optimizes language-relevant parameters and applies prior preservation; editing leverages P2P and Null-text inversion [7, 16].Source: Figure 2 · p. 2
3. Custom-Edit
Custom-Edit customizes language-relevant parameters using reference images, then edits source images with text-guided methods. Its editing recipe combines attention-based structure preservation with choices intended to manage source-reference similarity.
3. Custom-Edit
- Custom-Edit is a two-step process: customize the model on references, then edit images with textual prompts.The paper focuses on image-to-image editing with complex visual instructions.Source: §3 · p. 2
3.1. Customization
- Customization trains cross-attention keys and values plus a rare token, and augmented prompts use a modifier to focus learning on reference appearance.Prior preservation uses retrieved same-class images and captions to maintain language understanding during fine-tuning.Source: §3.1 · p. 2
3.2. Text-Guided Image Editing
- The editing process uses Prompt-to-Prompt attention injection to preserve source structure and supports local edits with automatically computed masks.Null-Text Inversion is applied before P2P to boost source preservation.Source: §3.2 · p. 2
- For P2P, the recipe favors prompt refinement over word swap and uses the source attention mask rather than the union mask.The authors say customized words may not favor swapped attention maps and may produce incorrect masks.Source: §3.2 · p. 2
- P2P strength controls a source-reference trade-off through self-attention injection; higher strength favors source similarity at the expense of reference similarity.For SDEdit, higher strength instead means higher reference similarity.Source: §3.2 · pp. 2, 4
- Qualitative examples show reference appearance transferred to source images while source structures are preserved.The examples use BLIP2 source prompts and, except for the pencil drawing, local P2P editing with automatically generated masks.Source: Figure 3 · p. 3
4. Experiment
The experiments evaluate customization methods with P2P and SDEdit, using qualitative examples and average source-reference similarity curves. Custom-Diffusion achieves the best trade-off, while the results also show how its edits transfer reference details and preserve source structure.
4. Experiment
- The evaluation uses eight references, averages five source images per reference, and measures source and reference similarity with CLIP ViT-B/32.It tests both P2P and SDEdit, with multiple strengths and samples per source image.Source: §4 · p. 4
4.1. Qualitative Results
- Custom-Edit transfers reference details while preserving source structure, including a wooden pot edited from a wine bottle and a cat in a pose absent from the references.Other examples include a tortoise plushy wearing a hat and a ceramic bird while preserving macarons.Source: §4.1 · p. 4
4.2. Quantitative Results
- Custom-Diffusion yields the best trade-off; Textual Inversion has similar source but lower reference similarity, while Dreambooth has higher source but lower reference similarity.SDEdit shows a similar customization tendency but lower source similarity than P2P.Source: §4.2 · p. 4, Figure 4 · p. 4
5. Discussion
The discussion presents Custom-Edit as a framework for fine-grained text-guided editing and identifies failures involving unwanted regions and complex backgrounds.
5. Discussion
- Custom-Edit supports fine-grained editing with textual prompts and presents design choices for customization and editing alongside the source-reference trade-off.The authors say these design choices can benefit future customization and editing work.Source: §5 · p. 4
- Custom-Edit sometimes edits undesired regions or fails on complex backgrounds; the authors hypothesize inaccurate attention maps and limited text controllability contribute to these failures.They suggest larger text encoders or extended controllability as potential solutions.Source: §5 · p. 4
A. Additional Results
The appendix provides additional Custom-Edit outputs and qualitative comparisons of customization methods on P2P and SDEdit.
A. Additional Results
- Additional Custom-Edit results are shown in Fig. A.Source: A. Additional Results · p. 6
A.1. Qualitative Comparisons
- On both P2P and SDEdit, Dreambooth fails to modify source images effectively, while Textual Inversion misses detailed reference appearance.Source: §A.1 · p. 6
A.2. Strength Control
The appendix reports how editing strength affects P2P and SDEdit results and gives ranges that produce good samples.
A.2. Strength Control
- P2P strengths of 0.4–0.6 and SDEdit strengths of 0.6–0.7 produce good samples, while users can adjust strength to match their preferences.Source: §A.2 · p. 6
A.3. Failure Cases
Custom-Diffusion improves text similarity, while the failure cases show limits in interpreting interactions and selecting targets in complex scenes.
A.3. Failure Cases
- In one failure, the cake becomes the V* cat instead of coffee; replacing “A cup” with “V* cat” resolves the issue.The authors speculate the cat token fails to attend to coffee, and note prior reports of less accurate Stable Diffusion attention maps than Imagen.Source: §A.3 · p. 6
- Custom-Edit handles the dolphin-to-tortoise edit but cannot select a rock in a complex scene without clarification about which rock to modify.Applying Custom-Edit to a grounding-input model such as GLIGEN [14] may solve this problem.Source: §A.3 · p. 6
A.4. Text Similarity
- Custom-Diffusion achieves significantly better text similarity than other methods, measured by CLIP ViT-B/32 against target text with V* omitted.The section frames this as a trade-off with source similarity.Source: §A.4 · p. 6
B. Implementation Details
The implementation details specify customization training settings and prompt augmentation for Dreambooth, Custom-Diffusion, and Textual Inversion.
B.1. Customization
- Dreambooth and Custom-Diffusion are trained for 500 optimization steps with a batch size of 2, and the same dataset is used for prior preservation loss; text inputs are augmented with templates.One template is “cropped in photo of a V* [modifier] [class]”.Source: §B.1 · p. 6
- For one out of three iterations, training randomly crops images and uses zoomed-in or close-up photo templates.The listed forms include “zoomed in photo of a V* [modifier] [class]” and “close up in photo of a V* [modifier] [class].”Source: §B.1 · p. 6
- Textual Inversion trains one token for 2,000 steps with batch size 4, using the text template from [6].For cat and dog categories, customizing without the modifier token already offered good results.Source: §B.1 · p. 6
B.2. Dataset
The appendix identifies the reference and source-image collections and provides qualitative comparisons for P2P and SDEdit, alongside a note on inversion.
B.2. Dataset
- Reference sets come from prior customization works, while source images come from Imagen, eDiffI [1], Imagic [10], Muse [4], Null-Text Inversion [16], and Text2Live [2].The reference concepts include wooden pot, tortoise plushy, cat, dog, ceramic bird, cat figurine, patterned teapot, and pencil drawing.Source: §B.2 · p. 6
C.1. Prompt-to-Prompt
- On P2P, Custom-Diffusion transfers the reference appearance to the source, whereas Dreambooth and Textual Inversion fail to capture it precisely.Each row uses the same strength; the listed strengths are [0.4, 0.8, 0.2, 0.4, 0.8] from first to last.Source: §C.1 · p. 8
- On SDEdit, Custom-Diffusion edits the source but changes some background because SDEdit diffuses the background as well.Dreambooth and Textual Inversion fail to capture the reference’s precise appearance; each row uses the same strength.Source: Figure C · p. 9
C.2. Null-Text Inversion
- Deterministic DDIM inversion [25] fails to reconstruct a real source image because classifier-free guidance [9] accumulates error.The section says editing real images requires deterministic mapping from image to noise.Source: §C.2 · p. 11