PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructPix2Pix: Learning to Follow Image Editing Instructions

Tim Brooks, Aleksander Holynski, Alexei A. Efros

arXiv:2211.09800v2cs.CVcs.AIcs.CLcs.GRcs.LG

TL;DR

Instruction-based image editing requires training data for pairing human-written actions with before-and-after images, but such data is difficult to acquire at scale. The paper combines GPT-3 and Stable Diffusion to generate paired examples and trains InstructPix2Pix, which generalizes to real images and arbitrary human-written instructions across diverse edits.

  • Problem

    Instruction-based image editing lacks large-scale paired training data, while existing methods commonly require captions, descriptions, or example images rather than direct editing instructions.

  • Method

    The method combines GPT-3 and Stable Diffusion to generate paired editing data, then trains a conditional diffusion model to follow instructions given an input image.

  • Results

    The model achieves zero-shot generalization to real images and arbitrary human-written instructions, performing diverse edits across photographs and artwork.

  • Takeaways & Limitations

    Users can specify precise image changes in natural language without providing example images or descriptions of unchanged visual content.

  • Takeaways & Limitations

    Performance is limited by the generated dataset and underlying models, with particular difficulty in counting objects and spatial reasoning.

Abstract

from arXiv · show

We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.

1. Introduction

The paper presents a conditional diffusion model trained on synthetic image-editing pairs generated by GPT-3 and Stable Diffusion to follow human-written editing instructions. It edits images directly in one forward pass without per-example finetuning and generalizes zero-shot to real images, natural instructions, and diverse edits.

  • Motivation: The method combines GPT-3 and Stable Diffusion to generate paired training data for human-instructed image editing, addressing the difficulty of acquiring such data at scale.The models provide complementary knowledge about language and images.
  • Method: The conditional diffusion model takes an input image and text instruction, then generates the edited image directly in the forward pass without per-example finetuning.It also requires no additional example images or full input/output descriptions.
  • Results: Despite training entirely on synthetic instructions and imagery, the model generalizes zero-shot to arbitrary real images and natural human-written instructions.This supports inference beyond the generated training examples.
  • Results: The model follows instructions for diverse edits, including replacing objects and changing an image’s style, setting, or artistic medium.Selected examples are presented in Figure 1.

2. Prior work

Prior work combines large pretrained models for multimodal tasks, advances diffusion-based image generation, and develops text-guided image-editing methods. In contrast, this work focuses on editing from natural-language instructions and uses generative models to create training data.

  • Composing large pretrained models: Large pretrained models have been combined to solve multimodal tasks requiring both language-model and text-image-model knowledge.Approaches include joint finetuning, prompting-based communication, and composing probability distributions.
  • Diffusion-based generative models: Diffusion models have achieved state-of-the-art image synthesis and have been extended to video, audio, text, and network parameters.Text-to-image diffusion models generate realistic images from arbitrary text captions.
  • Generative models for image editing: Earlier image-editing models targeted single tasks, latent-space manipulation, or CLIP-guided editing with text.Recent methods also use pretrained text-to-image diffusion models, but targeted editing can be difficult because similar prompts may not produce similar images.
  • Generative models for image editing: Prompt-to-Prompt supports isolated edits to generated images, while SDEdit edits real images by noising and denoising them with a new target prompt.The paper uses Prompt-to-Prompt to generate training data and compares with SDEdit as a baseline.
  • Learning to follow instructions: Unlike prior text-based editing methods, this approach follows instructions specifying an action rather than labels, captions, or input/output image descriptions.Instruction-based editing lets users state what to do in natural written text without providing extra information such as example images.
  • Training data generation with generative models: Generative models are increasingly used as cheap and plentiful training-data sources when suitable supervised data, such as paired modalities, may not exist.This paper uses two different off-the-shelf generative models for training-data generation.

3. Method

The method generates paired image-editing data by combining GPT-3 with Stable Diffusion and Prompt-to-Prompt, then trains a conditional diffusion model on these examples. InstructPix2Pix thereby edits real images from arbitrary human-written instructions without per-example fine-tuning or inversion.

  • Method overview: The method first generates paired training examples containing an input image, an editing instruction, and the corresponding edited image, then trains a diffusion model on them.The two stages are dataset generation and diffusion-model training.
  • Dataset generation: Stable Diffusion and Prompt-to-Prompt transform caption pairs into corresponding before-and-after image pairs, with Prompt-to-Prompt encouraging image consistency across generations.This addresses the inconsistency of independently generated images under slightly different prompts.
  • Dataset generation: The procedure creates over 450,000 training examples and filters generated pairs with CLIP directional similarity to improve change consistency, diversity, quality, and robustness.The method generates 100 image pairs per caption pair with random p ∼U(0.1, 0.9) before filtering.
  • Method overview: Despite training on generated images and instructions, the model generalizes to real images and arbitrary human-written editing instructions.Inference performs edits in the forward pass without per-example fine-tuning or inversion.
  • Dataset generation: A finetuned GPT-3 converts real LAION image captions into editing instructions and modified output captions.GPT-3 is finetuned for one epoch on 700 manually labeled caption-instruction-output triplets sampled from LAION-Aesthetics V2 6.5+.
  • Diffusion model training: The model is initialized from Stable Diffusion, adds input-image latent channels, and uses the edit instruction as text conditioning for conditional image editing.New input-channel weights are initialized to zero, while available diffusion-model weights reuse pretrained checkpoints.

4. Results

InstructPix2Pix performs diverse instruction-based edits on real photographs and artwork, while comparisons show broader instruction-following and edit capabilities than prior methods. Quantitative analyses characterize trade-offs between input consistency and edit quality, and show how dataset design and guidance scales affect results.

  • Qualitative results: The model successfully performs diverse challenging edits, including object replacement, season and weather changes, background replacement, material modification, and artistic-medium conversion.Results cover real photographs and artwork with varied edit types and instruction wordings.
  • Comparisons: Compared with SDEdit and Text2Live, InstructPix2Pix follows editing instructions, whereas the baselines require an after-edit image description or have narrower edit formulations.SDEdit struggles with identity preservation and isolating individual objects, especially for larger changes; Text2Live is convincing for additive-layer edits but handles fewer edit categories.
  • Quantitative evaluation: The evaluation measures input-image consistency with CLIP image-embedding cosine similarity and edit quality with directional CLIP similarity, which are competing objectives.Higher values are better for both metrics, but stronger correspondence to the desired edit reduces similarity to the input image.
  • Ablations: Smaller training datasets reduce the ability to perform larger edits, while removing CLIP filtering produces a worse configuration than the proposed model.Reduced datasets instead favor subtle or stylistic adjustments with higher image similarity but lower directional scores.
  • Guidance-scale analysis: Increasing sT strengthens agreement with the instruction, increasing sI helps preserve input spatial structure, and the best typical ranges are sT 5−10 and sI 1−1.5.These are classifier-free guidance scales analyzed in the model.
  • Failure cases: The model has failure cases involving viewpoint changes, excessive image changes, object isolation, and reorganizing or swapping objects.These limitations are summarized in the reported failure-case figure.

5. Discussion

The method combines GPT-3 and Stable Diffusion to generate training data for instruction-based image editing and produces diverse, compelling edits, but remains constrained by dataset quality, generalization, and inherited biases. Future work includes spatial reasoning, multimodal conditioning, evaluation, and human feedback to improve alignment.

  • Contributions: The approach combines a large language model and a text-to-image model to generate data for training a diffusion model on written image-editing instructions.It produces diverse edits involving style, medium, and contextual changes.
  • Limitations: The model’s visual quality is limited by the generated dataset and the Stable Diffusion model used to create its imagery.Generalization also depends on the instructions used to fine-tune GPT-3 and GPT-3’s ability to generate instructions and modify captions.
  • Limitations: Edited images may inherit or introduce biases present in the underlying data and pretrained models, including correlations between profession and gender.These biases are illustrated in Figure 14.
  • Future Work: Future work includes following spatial-reasoning instructions, combining instructions with user interaction and other conditioning modalities, and evaluating instruction-based image editing.The paper also identifies human feedback and human-in-the-loop reinforcement learning as ways to improve alignment with people.

A. Implementation Details · A.1. Instruction and Caption Generation

The method fine-tunes GPT3 to generate edit instructions and edited captions, using structured prompts and controlled sampling during inference.

  • A.1. Instruction and Caption Generation: GPT3 is fine-tuned to generate edit instructions and edited captions.
  • A.1. Instruction and Caption Generation: During fine-tuning, the input prompt is the input caption followed by the separator token "\n##\n".
  • A.1. Instruction and Caption Generation: The completion concatenates the instruction and edited caption with "\n%%\n" between them.
  • A.1. Instruction and Caption Generation: The completion ends with "\nEND" as the stop token.
  • A.1. Instruction and Caption Generation: At inference, text completions are sampled given new input captions.
  • A.1. Instruction and Caption Generation: Inference sampling uses temperature=0.7 and frequency penalty=0.1.

A.2. Paired Image Generation

The method generates paired before/after training images from paired captions using Stable Diffusion and Prompt-to-Prompt, with shared latent noise to promote consistent edits. It filters generated pairs using image-image, image-caption, and directional CLIP thresholds.

  • Generation setup: Paired before/after training images are generated from paired captions with Stable Diffusion and Prompt-to-Prompt, using shared latent noise for each pair.Generation uses EMA weights from Stable Diffusion v1.5, improved ft-MSE autoencoder weights, 100 denoising steps, and an Euler ancestral sampler.
  • Attention replacement: Self-attention weights in the second image are replaced during the first p fraction of steps using one strategy across all edits.This differs from Prompt-to-Prompt’s edit-specific cross-attention replacement for word swaps, phrase additions, and word-weight changes.
  • Data filtering: 100 image pairs are generated for each caption pair and filtered with CLIP thresholds of 0.75 for image-image similarity, 0.2 for image-caption similarity, and 0.2 for directional similarity.The filters enforce visual consistency, caption correspondence, and agreement between caption changes and image changes.

A.3. Training InstructPix2Pix

InstructPix2Pix is trained for 10,000 steps at 256 × 256 resolution using large-scale A100 hardware and Stable Diffusion initialization. Despite training at 256 × 256, it generalizes to 512 × 512 inference and edits images in roughly 9 seconds on an A100 GPU.

  • Training setup: 10,000 training steps use 8× 40GB NVIDIA A100 GPUs over 25.5 hours, with 256 × 256 resolution and a total batch size of 1024.Training uses random horizontal flips, resize-and-crop augmentation, and a 10−4 learning rate without warmup.
  • Training setup: The model is initialized from EMA weights of the Stable Diffusion v1.5 checkpoint, with other settings adopted from the public Stable Diffusion code base.
  • Inference: 512 × 512 inference generalizes well from 256 × 256 training, using 100 denoising steps and an Euler ancestral sampler with Kerras et al.’s variance schedule.Editing an image takes roughly 9 seconds on an A100 GPU.

B. Classifier-free Guidance Details

The model applies classifier-free guidance separately to the input image and text instruction, using distinct scales to control their conditioning strengths. These scales shift probability mass toward samples favored by implicit classifiers learned from conditional and unconditional estimates.

  • Guidance formulation: Classifier-free guidance uses separate scales sI and sT to independently trade off the strengths of the input-image and text-instruction conditionings.The model is conditioned on both the input image cI and text instruction cT.
  • Guidance effects: The input-image guidance scale sI shifts probability mass toward data that an implicit classifier assigns high likelihood to image conditioning cI.This classifier is denoted pθ(cI|zt).
  • Guidance effects: The text guidance scale sT shifts probability mass toward data that an implicit classifier assigns high likelihood to text conditioning cT given cI.This classifier is denoted pθ(cT |cI, zt).
  • Implicit classifiers: The model learns these implicit classifiers from differences between estimates with and without the respective conditional input.The paper notes that alternative formulations are possible, including switching the positions of cT and cI.
Loading 2211.09800v2…