PaperlayerView exact revision on arXiv

Source-linked AI summary

Emu Edit: Precise Image Editing via Recognition and Generation Tasks

Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, Yaniv Taigman

arXiv:2311.10089v1cs.CVcs.AIcs.LG

TL;DR

Instruction-based image editing models often fail to follow natural-language instructions accurately and generalize poorly beyond their training tasks. Emu Edit addresses this with multi-task training across editing and recognition tasks plus learned task embeddings, achieving more precise editing and few-shot adaptation to new tasks. The paper also releases a seven-operation benchmark for evaluation.

  • Problem

    Existing instruction-based image editing models struggle with accurate instruction execution and generalization to tasks that differ from their training data.

  • Method

    Emu Edit trains a generative diffusion model across diverse editing and computer vision tasks and uses learned task embeddings to steer generation toward the correct edit type.

  • Results

    Emu Edit improves instruction compliance and preservation of original-image fidelity, while adapting to unseen tasks such as inpainting and super-resolution from few labeled examples.

  • Takeaways & Limitations

    Multi-task recognition, generation, and editing training together with task embeddings enables more precise and robust editing and efficient few-shot learning for new tasks.

  • Takeaways & Limitations

    Evaluation relies on Stable Diffusion-generated input images and GPT-3-generated instructions, so correspondence to real images and genuine user instructions remains unclear.

Abstract

from arXiv · show

Instruction-based image editing holds immense potential for a variety of applications, as it enables users to perform any editing operation using a natural language instruction. However, current models in this domain often struggle with accurately executing user instructions. We present Emu Edit, a multi-task image editing model which sets state-of-the-art results in instruction-based image editing. To develop Emu Edit we train it to multi-task across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks, all of which are formulated as generative tasks. Additionally, to enhance Emu Edit's multi-task learning abilities, we provide it with learned task embeddings which guide the generation process towards the correct edit type. Both these elements are essential for Emu Edit's outstanding performance. Furthermore, we show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples. This capability offers a significant advantage in scenarios where high-quality samples are scarce. Lastly, to facilitate a more rigorous and informed assessment of instructable image editing models, we release a new challenging and versatile benchmark that includes seven different image editing tasks.

1. Introduction

Emu Edit addresses instruction-following and generalization limitations in image editing through diverse multi-task training and learned task embeddings. It improves editing compliance and fidelity, adapts to unseen tasks with few examples, and introduces a seven-task benchmark.

  • Existing instruction-based image editing models often struggle to execute instructions accurately and generalize to tasks differing from their training data.
  • Emu Edit trains one model across sixteen image editing and computer vision tasks, including region-based, free-form, detection, and segmentation tasks formulated generatively.
  • Training a single model across all tasks outperforms independently trained expert models, while adding more tasks and computer vision training improves editing performance.
  • Learned task embeddings steer generation toward the appropriate edit type through cross-attention and timestep-embedding integration.
  • Task inversion adapts Emu Edit to unseen tasks by updating only a task embedding, with some few-example results nearly matching expert models trained on 100,000 examples.
  • The authors release a challenging benchmark containing seven image editing operations to support evaluation of instruction-based image editing.

3. Multi-Task Dataset for Image Editing

The dataset construction targets the scarcity and quality limitations of image-editing training data by organizing editing operations into region-based and free-form categories.

  • Manually collecting high-quality input, instruction, and edited-image examples is impractically time-consuming.
  • Region-based editing includes local, remove, add, texture, and background operations.
  • Free-form editing includes global changes, style transfer, and text editing operations.

3. Vision tasks

Emu Edit’s dataset combines vision tasks with image editing and uses task-specific instruction generation, image-pair construction, and filtering procedures to improve diversity and fidelity.

  • Vision task categories: Vision tasks include object detection, segmentation, color adjustment, and image-to-image translation across modalities such as sketches, depth maps, poses, and segmentation maps.
  • Dataset scale: The dataset contains sixteen distinct tasks and ten million examples, each pairing an input image, instruction, target image, and task index.
  • Task organization: Tasks are divided into region-based editing, free-form editing, and vision-task categories.
  • Instruction generation: Task-specific Llama 2 agents generate editing instructions, output captions, and object-update annotations using descriptions, exemplars, and image captions.
  • Image-pair construction: The pipeline develops task-specific image-generation techniques and applies filtering to preserve instruction adherence and image fidelity.

4. Method

Emu Edit is a diffusion model adapted for diverse generative editing and computer vision tasks, using task-specific conditioning to select the intended operation. It also supports few-shot adaptation to new tasks and thresholded multi-turn editing.

  • Emu Edit formulates region-based editing, free-form editing, detection, segmentation, and depth estimation as generative tasks within one diffusion model.
  • The model builds on Emu, whose architecture combines a large U-Net, text embeddings, and a high-resolution latent diffusion setup.The U-Net has 2.8 billion parameters, and Emu uses a 16-channel autoencoder.
  • Learned Task Embeddings: Each training task receives a learned embedding integrated into the U-Net through cross-attention and timestep embeddings.The embeddings are optimized jointly with the model weights, while a task predictor identifies the task during inference.
  • Learned Task Embeddings: Task conditioning addresses ambiguity by steering the model away from incorrect operations such as global editing instead of texture editing.Without task conditioning, complex or ambiguous instructions can trigger the wrong edit type.
  • Task Inversion: Task inversion adapts Emu Edit to a new task by learning a task embedding while freezing the model weights.The adapted embedding can support the new task while the original embeddings preserve the model’s existing tasks.
  • Sequential Edit Thresholding: Repeated editing can accumulate reconstruction and numerical errors, so per-pixel thresholding retains new pixels only when smoothed changes exceed α = 0.03.The change image is computed across RGB channels, and a low-pass filter smooths transitions between previous and current pixels.

5. Experiments

Emu Edit is evaluated for instruction following, visual fidelity, benchmark design, ablations, and few-shot generalization. The experiments show benefits from task conditioning, multi-task training, and adaptation to unseen tasks.

  • Evaluation: The evaluation measures instruction following and image preservation using automatic CLIP, L1, and DINO metrics alongside human ratings.Human raters separately assess image faithfulness and text alignment.
  • Baseline Comparisons: Emu Edit is compared with instruction-based and text-based editing baselines on the Emu Edit and MagicBrush test sets.Text-based methods receive input and output captions, which may give them an advantage because the automatic metrics use those captions.
  • Baseline Comparisons: Human raters consistently prefer Emu Edit over all baselines, and it outperforms existing baselines on automatic metrics except Null-Text Inversion.Null-Text Inversion uses ground-truth captions during inference.
  • Ablations: Detection and segmentation improve region-based editing, while image-to-image translation improves free-form editing.The authors hypothesize that recognition tasks improve recognition of the image content needed for editing.
  • Ablations: Conditioning on task type boosts performance, while the task predictor closes the gap with ground-truth task conditioning.Without task-type conditioning, the model may execute the wrong editing operation; changing the task embedding changes the executed task.
  • Ablations: Multi-task training outperforms task-specific expert models, and adding tasks improves performance even on tasks unrelated to those added.The Background task improves performance on Texture and Style tasks.
  • Few-Shot Learning of New Tasks: Fine-tuning with one example significantly improves unseen-task performance, while 100 examples nearly reach expert-level performance.Task inversion is comparable to full fine-tuning, whereas training from scratch overfits.

6. Conclusion

Emu Edit combines recognition and generation tasks to improve instructable image editing. It also generalizes to new tasks such as inpainting and super-resolution with minimal examples.

  • 6. Conclusion: Training on both recognition and generation tasks significantly enhances Emu Edit’s instructable image editing capabilities.The conclusion attributes the improvement primarily to this dual-focus training.
  • 6. Conclusion: Emu Edit generalizes to image inpainting and super-resolution with minimal examples, demonstrating versatility and advanced understanding.The paper notes potential future integration with a multimodal LLM for additional editing tasks.

7. Data

The dataset combines diverse editing and computer vision tasks, with task-specific generation pipelines designed to produce instruction-faithful image pairs. These pipelines use masks, attention control, inpainting, image transformations, and filtering tailored to each task.

  • Task coverage: The training dataset covers region-based, free-form, vision, and image-to-image translation tasks.The task distribution is shown in Figure 7, while the data-generation section describes task-specific procedures across these categories.
  • Region-based editing: Region-based tasks use detected object or location masks with mask-based attention control to localize edits.DINO extracts masks for relevant objects, while dilation, Gaussian blurring, and bounding-box masks address object replacement artifacts.
  • Free-form editing: Free-form tasks include local, texture, add, remove, background, global, style, and text editing procedures.These procedures use techniques including masked attention control, inpainting, Plug-and-Play, OCR, and blending.
  • Vision and translation tasks: Vision tasks formulate detection and segmentation as image generation, while color tasks apply filters such as brightness, contrast, saturation, hue, blur, and sharpening.Image-to-image translation procedures also generate depth maps, segmentation maps, poses, normal maps, and sketches.

8. Dataset Evaluation

The dataset-generation pipeline is evaluated against InstructPix2Pix using matched samples and automatic metrics. The section also reports task-count ablations and the benchmark’s per-task image allocation.

  • Pipeline comparison: The evaluation compares Emu Edit’s data-generation pipeline with InstructPix2Pix using 6,000 samples drawn from the same distribution.Each sample includes images, captions, an editing instruction, and edited-object annotations.
  • Dataset composition: The benchmark dataset records the number of images for each task and split.Table 6 summarizes the allocation across tasks and dataset partitions.
  • Task ablation: Figure 8 evaluates how performance changes as tasks outside Style and Texture are progressively excluded.The ablation uses CLIPdir as the performance metric.

9. Image Editing Benchmark

The Image Editing Benchmark is built from authentic images and human-written instructions, with multiple annotation stages to verify task relevance and support automatic evaluation.

  • Annotation process: MagicBrush images undergo instruction generation, instruction filtering, and caption annotation by crowd workers.Workers generate instructions, classify task type and relevance, and annotate captions for the input and desired edited images.
  • Evaluation support: The benchmark captions describe both the original and desired edited images, enabling automatic evaluation methods.Workers are instructed to make captions faithful to the image and aligned with the editing instruction.

10. Additional Results

Additional results examine vision-task performance, task-embedding control, few-shot adaptation, multi-task comparisons, and sequential-edit quality. The experiments show that task conditioning controls the executed operation and that thresholding balances image preservation with edit quality.

  • Vision tasks: Emu Edit is evaluated on detection, segmentation, and depth estimation using mAP@0.5, mIoU, and RMSE, respectively.Because the model was not trained on MS-COCO, ADE20K, or NYUv2, these are zero-shot results.
  • Task-embedding control: Changing the task embedding changes the task executed by the model while keeping the instruction and input image fixed.The model generates different outputs for the same input under different task embeddings.
  • Few-shot adaptation: Few-shot experiments evaluate super-resolution and contour detection alongside qualitative examples of other newly learned tasks.Figure 10 compares Emu finetuning, Emu Edit finetuning, task inversion, and an expert trained on 100k samples.
  • Sequential editing: Sequential-edit thresholding preserves output quality, but large threshold values can interfere with editing; the experiments use α = 0.03.Without thresholding, repeated edits produce general noise, whereas excessive thresholding harms edit quality.

11. Implementation Details

The implementation uses a diffusion-based model conditioned on text and task embeddings, with examples illustrating diverse editing prompts and qualitative comparisons. The materials also show in-context examples and evaluations across multiple editing scenarios.

  • Model and training: The model generates 512 × 512 images and conditions generation on text and task embeddings through cross-attention and timestep-embedding addition.Training uses Adam with batch size 512, a 2e-5 learning rate, cosine decay, and linear warmup.
  • Evaluation and generalization: The figures present qualitative comparisons with baselines and examples of unseen-task generation, sequential-edit thresholding, and the Emu Edit Test Set.Unseen-task examples include composed additions and detections, composed additions and styles, inpainting, contour detection, and super-resolution.
  • Prompting and in-context learning: The implementation materials include an in-context example for the Add task and prompt templates that vary verbs such as include, place, position, set, and add.The displayed examples pair input descriptions with structured edit, edited-object, and output fields.
Loading 2311.10089v1…