PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Image-to-Image Translation

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, Jun-Yan Zhu

arXiv:2302.03027v1cs.CVcs.GRcs.LG

TL;DR

Editing real images with text-to-image diffusion models is difficult because complete prompts are cumbersome and prompt changes can disrupt unwanted content. Pix2pix-zero automatically discovers edit directions and applies cross-attention guidance through a pretrained diffusion model without per-edit training; experiments report stronger photorealism and content preservation than existing and concurrent works on real and synthetic images.

  • Problem

    Real-image editing is difficult because users must describe many visual details manually, while text-driven models can alter the input layout, pose, and unwanted content.

  • Method

    Pix2pix-zero combines automatically discovered source-to-target directions in text-embedding space with cross-attention guidance in a pretrained text-to-image diffusion model.

  • Results

    Pix2pix-zero outperforms existing and concurrent works in photorealism and content preservation across real and synthetic image-editing tasks.

  • Takeaways & Limitations

    The method enables structure-preserving edits without manual prompts, additional per-image training, or spatial masks.

  • Takeaways & Limitations

    Structure guidance is limited by cross-attention resolution and can fail for fine-grained details or atypical object poses.

Abstract

from arXiv · show

Large-scale text-to-image generative models have shown their remarkable ability to synthesize diverse and high-quality images. However, it is still challenging to directly apply these models for editing real images for two reasons. First, it is hard for users to come up with a perfect text prompt that accurately describes every visual detail in the input image. Second, while existing models can introduce desirable changes in certain regions, they often dramatically alter the input content and introduce unexpected changes in unwanted regions. In this work, we propose pix2pix-zero, an image-to-image translation method that can preserve the content of the original image without manual prompting. We first automatically discover editing directions that reflect desired edits in the text embedding space. To preserve the general content structure after editing, we further propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. In addition, our method does not need additional training for these edits and can directly use the existing pre-trained text-to-image diffusion model. We conduct extensive experiments and show that our method outperforms existing and concurrent works for both real and synthetic image editing.

1. Introduction

Pix2pix-zero addresses the difficulty of editing real images with text-to-image diffusion models by removing manual prompting and per-edit training while preserving input structure. It combines automatic edit-direction discovery with cross-attention guidance and supports diverse real and synthetic image translations.

  • Motivation: Text-to-image models generate diverse, realistic images but are difficult to repurpose for real-image editing because images lack complete text descriptions and edits can disrupt layout, shape, and pose.Users must otherwise describe numerous visual details, while changing prompts can synthesize new content rather than preserve the input.
  • Approach: Pix2pix-zero is a training-free, prompt-free diffusion-based image-to-image translation method that accepts only a source-to-target edit direction.It directly uses pretrained text-to-image diffusion models without additional training for each edit type or image.
  • Approach: The method automatically discovers robust edit directions by computing CLIP embedding differences between groups of sentences containing the source and target words.The direction is designed to generalize across input images and can be pre-computed in about 5 seconds.
  • Approach: Cross-attention guidance preserves input structure by encouraging the text-image cross-attention maps to remain consistent throughout diffusion.The method uses the observed relationship between cross-attention maps and generated-object structure to constrain translation.
  • Efficiency: Autocorrelation regularization improves inversion noise quality, while conditional GAN distillation targets faster interactive editing.The regularization keeps intermediate predicted noise closer to Gaussian during DDIM inversion; distillation addresses the cost of multi-step diffusion inference.
  • Results: Across object, attribute, and style translation tasks on real and synthetic images, pix2pix-zero outperforms existing and concurrent works in photorealism and content preservation.Examples include cat-to-dog translation, adding glasses, and sketch-to-oil-pastel style changes.

2. Related Work

Prior image-editing methods use task-specific GAN training, text-based control, masks, or diffusion modifications, but face limitations in generality, inversion, or structure preservation. Pix2pix-zero is positioned as a pretrained-diffusion approach that avoids per-image finetuning and spatial masks while retaining input structure.

  • Deep image editing with GANs: Conditional GAN editing methods often require task-specific training and struggle with high-quality inversion on complex images.Their success is described mainly for single-category curated datasets.
  • Text-to-Image models: Text-to-image models offer limited control outside text input, and changing words can cause unreliable edits that alter too much of the image.Some methods add spatial masks to constrain edits, whereas pix2pix-zero retains structure without one.
  • Image editing with diffusion models: Imagic requires finetuning the entire model per image, while prompt-to-prompt avoids finetuning but focuses mainly on synthetic editing and uses original cross-attention with edited-text values.Pix2pix-zero differs by automatically discovering and pre-computing edit directions in text-embedding space.

3. Method

pix2pix-zero edits images from a source-to-target domain direction by combining automatic text-embedding direction discovery with deterministic inversion and cross-attention guidance. The method uses reference attention maps to preserve input structure while applying the edit in latent diffusion space.

  • Editing pipeline: The pipeline applies the edit in latent space using Stable Diffusion, with noise-regularized inversion as input and an edited latent code as output.The algorithm takes xT, input text features c, an edit direction, and a cross-attention guidance weight, then returns the final edited latent code x0.
  • Inverting real images: Regularized DDIM inversion maps the input latent code to a noise map intended to reconstruct the original image during sampling.The inversion progressively noises latent code x0 until the final noised code xT becomes xinv, while autocorrelation regularization addresses non-Gaussian inverted noise.
  • Edit direction discovery: The method automatically computes an edit direction from source and target domains, allowing edits without manually writing an input-image prompt.It generates diverse sentences for both domains and takes the mean difference between their CLIP embeddings; the direction can be pre-computed on-the-fly.
  • Cross-attention guidance: During edited denoising, cross-attention guidance minimizes deviation between edited and reference maps, helping retain the input structure.The edited embedding is c + ∆cedit, and gradient updates move the latent toward matching the reference attention maps; unguided sampling can produce large structural deviations.
  • Cross-attention guidance: The method uses the original text embedding to obtain timestep-specific cross-attention maps that serve as references for the input image structure.Cross-attention maps relate text tokens to spatial locations and are observed to correspond closely to image structure.

4. Experiments

Experiments evaluate pix2pix-zero across diverse real and synthetic editing tasks, comparing structure and background preservation against diffusion baselines and testing its components and acceleration. The method applies edits while preserving input structure, with conditional GAN distillation providing a major inference-speedup path.

  • Evaluation: Evaluations cover object translation, object modification, and global style transfer on real and synthetic images.Tasks include cat→dog, horse→zebra, cat→cat with glasses, and sketch→oil pastel.
  • Qualitative Results: pix2pix-zero applies diverse edits while preserving the structure of real and synthetic input images.Examples include object-type changes, attribute modifications, and style transformations.
  • Ablation Study: Soft cross-attention guidance enables edits while preserving structure, unlike hard attention constraints that can prevent the desired edit.Without guidance, edited objects can appear in different poses and backgrounds.
  • Comparisons: pix2pix-zero combines high CLIP-Acc with low Structure Dist and BG LPIPS, indicating effective edits with retained structure and background details.BG LPIPS is relevant only to foreground-object editing tasks.
  • Ablation Study: Ablations attribute improved editing ability to regularized inversion and sentence directions, while cross-attention guidance improves structure and background preservation.Deterministic DDIM inversion also improves structure and background reconstruction over stochastic inversion.

5. Limitations and Discussion

The paper concludes that pix2pix-zero provides training-free, prompting-free, structure-preserving editing with pretrained text-to-image diffusion models, while acknowledging limits in fine-grained control and atypical poses.

  • Discussion: pix2pix-zero performs structure-preserving image editing with a pretrained text-to-image diffusion model without training or prompting.Its contributions include automatic edit-direction learning and cross-attention map guidance.
  • Limitations: The 64 × 64 Stable Diffusion cross-attention resolution may be insufficient for very fine-grained control of details such as legs and tails.Higher-resolution cross-attention maps could provide finer structure guidance.
  • Limitations: The method can fail when objects have atypical poses.The limitation is illustrated with a cat in an atypical pose.

Appendix

The appendix provides supplementary training details, baseline comparisons, regularization analyses, experiment details, societal-impact discussion, and additional qualitative results.

  • Appendix A: Appendix A details training of the fast distilled GAN model and provides additional results.
  • Appendices B–C: Appendices B and C provide further baseline comparisons and analyses of regularization effects.
  • Appendices D–E: Appendix D gives experiment details, while Appendix E discusses societal impacts and additional figures provide qualitative results.Additional qualitative results appear in Figures 13–16.

A. Fast Distillation

The method distills its diffusion editor into a faster feed-forward GAN using paired edited images, achieving comparable editing quality and structure preservation at much faster inference speed.

  • Fast Distillation: The distillation targets interactive editing by replacing costly multi-step diffusion inference with a fast feed-forward model.The main paper’s Section 4.5 motivates the distillation as an acceleration of the slow text-to-image diffusion model.
  • Fast Distillation: 15,000 pairs of input and edited images are collected and automatically filtered using segmentation overlap and CLIP-similarity thresholds.For cat→dog, the thresholds are 0.70 for segmentation overlap and 0.10 for CLIP increase.
  • Fast Distillation: A CoModGAN is trained on paired images with color, noise, and crop augmentation, using L1 and VGG-based LPIPS reconstruction losses.The training uses a learning rate of 0.001 and batch size 64.
  • Fast Distillation: The distilled GAN provides comparable edit quality and structure preservation to slow diffusion editing while substantially increasing inference speed.The comparison is reported for tree-to-winter-tree and tree-to-fall-tree tasks.

B. Comparisons to Baselines.

Comparisons on synthetic images show that pix2pix-zero preserves input structure while performing requested edits, whereas several baselines struggle with structure or edit fidelity; regularization reduces artifacts in smaller models.

  • Comparisons to Baselines: pix2pix-zero respects the synthetic input structure while performing the requested edit, consistent with its real-image results.The comparison is presented against SDEdit, DDIM with word swap, and prompt-to-prompt.
  • Comparisons to Baselines: Regularization is described as critical for reducing artifacts in edited results from smaller models.Figure 9 compares DiffusionCLIP editing with and without the regularization.
  • Comparisons to Baselines: SDEdit and DDIM with word swap struggle to preserve structure, while prompt-to-prompt performs better on synthetic images but sometimes misses desired edits.The cited example is zebra stripes that are not applied correctly.

C. Ablations

The ablation shows that autocorrelation regularization improves editing and reduces unwanted structure changes by making inverted noise more Gaussian, especially in smaller category-specific models.

  • Ablations: Regularization improves the CLIP Acc score, indicating a greater extent of editing during inversion and subsequent editing.The effect is reported in Table 2 of the main paper.
  • Ablations: The regularizer encourages inverted noise to be more Gaussian, making the edit direction more compatible and less prone to undesired structure changes.This links the inversion-noise property to the observed editing behavior within the method.
  • Ablations: Regularization effects are more pronounced for smaller diffusion models trained for specific categories, where removing it produces noticeable artifacts.Figure 9 illustrates the artifact increase without regularization in a smaller LSUN Bedrooms model.

D. Experiment Details

Experiments use filtered LAION 5B subsets and compare real and synthetic editing, including fast and slow variants, under standardized baseline and inference settings.

  • Experiment Details: Real-image experiments retrieve 250 LAION 5B images using source-text CLIP matching and an aesthetics filter of 9.For cat→dog, images are selected for high CLIP similarity with cat.
  • Experiment Details: The experiments include real image editing and synthetic image editing, with comparisons between input, fast GAN, and slow diffusion outputs.These categories and output variants are listed in the experiment materials.
  • Experiment Details: Baseline comparisons use official author code and recommended hyperparameters for Figure 5, Table 1, and Figure 12.This applies to the reported results for the compared methods.
  • Experiment Details: The method uses 100 DDIM inversion steps, 100 reconstruction steps, 100 editing steps, and five regularization iterations per timestep with λ = 20.Classifier-free guidance is also used for all editing results.

E. Societal Impact

pix2pix-zero is framed as a tool for democratizing controlled generative content creation, especially in creative applications. The authors also acknowledge potential misuse for fabricating malicious images and point to detectability research as a possible basis for downstream identification.

  • The method aims to democratize content creation by giving users precise control over desired edits.
  • Its primary intended use is in the creative industry, where users can create new content through controlled edits.
  • The authors acknowledge that the method could potentially be used to fabricate images for malicious practices.
  • Research on GANs and diffusion models suggests generated images can contain consistently detectable artifacts that enable downstream identification.
Loading 2302.03027v1…