Source-linked AI summary
Paint by Example: Exemplar-based Image Editing with Diffusion Models
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, Fang Wen
TL;DR
The paper addresses the limited precision of text-guided editing by introducing exemplar-based editing with a self-supervised image-conditioned diffusion model. Information bottlenecks, strong augmentation, masking, and classifier-free guidance reduce trivial copying and improve controllability, while the method achieves high-fidelity in-the-wild editing in one forward pass. Its scope is limited for artificial images and rarer objects.
Problem
Text prompts can be ambiguous for fine-grained appearances, while self-supervised exemplar training can produce copy-and-paste artifacts and poor generalization to arbitrary references.
Method
The method uses self-supervised image-conditioned diffusion with compressed global exemplar conditioning, strong augmentation, arbitrary-shape masks, and classifier-free guidance.
Results
The method enables controllable, high-fidelity editing on in-the-wild images with favorable quantitative and subjective performance, using a single diffusion-model forward pass.
Takeaways & Limitations
Exemplar-based editing provides fine-grained semantic control by using images to specify desired customization more precisely than words.
Takeaways & Limitations
Because training data is mostly natural photos, the method performs poorly on some artificial images and struggles with rarer objects such as dinosaurs.
Abstract
from arXiv · showhide
Language-guided image editing has achieved great success recently. In this paper, for the first time, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. However, the naive approach will cause obvious fusing artifacts. We carefully analyze it and propose an information bottleneck and strong augmentations to avoid the trivial solution of directly copying and pasting the exemplar image. Meanwhile, to ensure the controllability of the editing process, we design an arbitrary shape mask for the exemplar image and leverage the classifier-free guidance to increase the similarity to the exemplar image. The whole framework involves a single forward of the diffusion model without any iterative optimization. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.
1. Introduction
The paper introduces exemplar-based image editing to provide finer semantic control than text prompts, using a self-supervised, image-conditioned diffusion model designed for realistic in-the-wild editing.
- Motivation: Text-guided editing remains ambiguous for fine-grained object appearances, motivating a more intuitive editing interface.The paper frames exemplar images as a way to convey desired customization more precisely than language.
- Contribution: Exemplar-based editing semantically transforms reference content before blending it into a source image, unlike harmonization methods that mainly adjust color and lighting.The transformation may involve changes in pose, deformation, or viewpoint while preserving contextual coherence.
- Method: The method trains a diffusion model conditioned on exemplar images using self-supervision because source–exemplar–edit triplets are difficult to collect.Naively cropping a source object as its own reference causes the model to learn copy-and-paste behavior that fails on real exemplars.
- Method: Global image conditioning, strong augmentation, irregular masks, and classifier-free guidance address trivial copying, train–test mismatch, practical masking, and reference similarity.The information bottleneck removes spatial tokens, while augmentation includes transformations such as flipping, rotation, blur, and elastic deformation.
- Results: The framework performs a single diffusion-model forward pass without image-specific optimization and reports favorable in-the-wild editing quality under quantitative and subjective evaluation.The authors describe a significant quality advantage over prior works in a similar setting.
2. Related Work
Prior image-composition methods mainly harmonize foreground and background appearance, while this paper targets semantic transformation of the composed content and demonstrates varied visual results.
- Image composition: Image harmonization methods typically adjust color or semantic features while assuming the foreground and background are already semantically compatible.The related work contrasts low-level color correction with the paper’s broader semantic-composition objective.
- Image composition: This paper addresses semantic image composition by transforming the reference content before blending it into another image.The target includes semantic inharmony rather than only appearance mismatch.
- Visual results: Figure 2 presents visual results across a wide variety of reference and source images.The caption identifies diversity of both image roles as the comparison to inspect.
3. Method
The method performs exemplar-based semantic editing by conditioning a diffusion model on a reference image and edit mask, using self-supervision to avoid copying artifacts. Information bottlenecks, strong augmentations, irregular masks, and classifier-free guidance improve generalization, mask control, and reference similarity.
- Problem: Exemplar-based editing synthesizes a source image region from a reference object while preserving the unmasked source and harmonizing the edited content.The task requires understanding reference shape and texture, transforming pose, size, or illumination, and producing a smooth merging boundary.
- Self-supervised training: Self-supervised training uses an object bounding box as the mask and the corresponding image patch as the reference because paired source-reference-output data is impractical to collect.The training data is simulated from images containing annotated object bounding boxes.
- Information bottleneck: The naive image-conditioned model learns to copy the reference patch, producing unnatural copy-and-paste artifacts and failing when test references are arbitrary rather than source-image patches.The failure occurs despite convergence on the training set, because the model does not learn reference-source relationships under the naive scheme.
- Information bottleneck: Using only the reference image’s CLIP class token, followed by learned feature decoding, compresses spatial information while retaining semantics and discouraging direct copying.The representation compresses a 224 × 224 × 3 image into a one-dimensional 1024-dimensional vector and is injected through cross-attention.
- Strong augmentation: Strong reference augmentations and irregular mask generation reduce the training-test gap and improve robustness to varied reference guidance and mask shapes.Reference augmentations include flipping, rotation, blur, and elastic transformation; masks are distorted from bounding boxes using sampled curve points and offsets.
- Control the similarity degree: Classifier-free guidance controls reference similarity: larger guidance scale s makes the generated result rely more on the conditional reference input.The framework uses a learnable unconditional vector during training and applies the modified prediction during inference denoising.
4. Experiments
Experiments show that the method produces high-quality, reference-consistent edits, while ablations identify image priors, augmentation, information bottlenecks, and classifier-free guidance as important components. Image conditioning preserves finer details than increasingly precise text, and stochasticity yields diverse in-the-wild results that retain reference identity.
- Comparisons: Qualitative comparisons show semantically consistent, high-perceptual-quality edits that transform exemplar appearance to fit the source image.Unlike direct harmonization, the method adapts exemplar shape, size, or pose rather than copying the exemplar unchanged.
- Comparisons: The method achieves the best performance across FID, QS, and CLIP score among the compared approaches.Image-based baselines preserve reference information but produce poor-quality results, whereas Stable Diffusion produces plausible images with weak conditional consistency.
- Ablation Study: Larger classifier-free guidance scale λ makes the generated region more similar to the reference, with λ = 5 used by default.The scale controls how strongly the fusion result relies on the conditional reference input.
- Ablation Study: The full method performs best when combining image prior, strong augmentation, information bottleneck, and classifier-free guidance.The baseline has boundary artifacts; the image prior improves FID, augmentation reduces copying, the bottleneck eliminates boundary artifacts, and guidance improves reference similarity and overall quality.
- From Language to Image Condition: Image conditioning maintains finer-grained details, including fur, expression, and collar features, beyond what progressively precise language descriptions preserve.The image-guided result remains more similar to the reference despite increasingly detailed textual guidance.
- In-the-wild Image Editing: Diffusion stochasticity enables multiple outputs from the same source and exemplar while preserving key reference identity traits.Examples retain shared attributes such as yellow fur, white chests, and drooping ears despite variation across outputs.
5. Conclusion
The paper introduces exemplar-based image editing for semantically altering image content and proposes self-supervised diffusion training to enable precise, high-quality editing.
- Exemplar-based image editing semantically alters image content according to an exemplar image.
- The method uses self-supervised training with a diffusion model to address the lack of paired training data.
- The approach addresses boundary artifacts and enables precise control with impressive performance on in-the-wild images.
Appendix A. Additional results
Additional results show that the method can edit arbitrary regions, harmonize generated objects with source images, and remain robust across different source images.
- The method edits different regions of the same source image and generates corresponding objects in the selected areas.
- Generated objects are highly in harmony with their source images.
- Results with the same object across different source images demonstrate robustness.
Appendix B. Implementation details
The implementation builds on Stable Diffusion and uses an inpainting modification with an image encoder, while the appendix also reports robustness across source images.
- Stable Diffusion v1-4 is used as the baseline and initialization for the text-driven image generation model.
- The baseline is modified into a text-driven inpainting model by adding five channels to the U-net’s first convolution layer.The added channels represent the encoded masked image and mask region, and the new weights are zero-initialized.
- The method demonstrates robustness for the same object across different source images.
Appendix C. Limitation
The method is limited by its natural-photo training distribution and struggles with artificial images and rarer objects.
- The method performs poorly on some artificial images, including oil paintings, because most training data consists of natural photos.
- The method can hardly understand rarer objects such as dinosaurs.
- Failure cases are presented in Figure 13.