PaperlayerView exact revision on arXiv

Source-linked AI summary

MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, Yu Su

arXiv:2306.10012v3cs.CVcs.AIcs.CL

TL;DR

Existing editing methods depend on manually tuned zero-shot pipelines or noisy synthetic data, leaving a need for high-quality real-image supervision. The paper introduces MagicBrush, a large-scale manually annotated dataset spanning several editing settings, and fine-tunes InstructPix2Pix on it. The fine-tuned model outperforms other baselines in human preferences, although edited images remain notably inferior to ground truth.

  • Problem

    Zero-shot methods require extensive manual tuning, while synthetic datasets can contain annotation errors and fail to capture real-world editing diversity.

  • Method

    MagicBrush is a large-scale manually annotated dataset of real-image editing collected through qualified workers using interactive DALL-E 2 editing across diverse scenarios.

  • Results

    Fine-tuned InstructPix2Pix produces much better results and outperforms other baselines according to human preferences.

  • Takeaways & Limitations

    MagicBrush supports training and evaluation for real-world instruction-guided editing while exposing the challenge current methods face.

  • Takeaways & Limitations

    Although InstructPix2Pix fine-tuned on MagicBrush achieves the best results, its edited images remain notably inferior to ground truth images.

Abstract

from arXiv · show

Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop. However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise. Thus, they still require lots of manual tuning to produce desirable outcomes in practice. To address this issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing. MagicBrush comprises over 10K manually annotated triplets (source image, instruction, target image), which supports trainining large-scale text-guided image editing models. We fine-tune InstructPix2Pix on MagicBrush and show that the new model can produce much better images according to human evaluation. We further conduct extensive experiments to evaluate current image editing baselines from multiple dimensions including quantitative, qualitative, and human evaluations. The results reveal the challenging nature of our dataset and the gap between current baselines and real-world editing needs.

1 Introduction

Existing text-guided image editing methods rely either on manually tuned zero-shot pipelines or noisy synthetic data, motivating MagicBrush, a manually annotated real-image dataset with diverse editing scenarios. Fine-tuning InstructPix2Pix on MagicBrush improves human-preferred results, while evaluations expose a gap between current methods and real-world needs.

  • Zero-shot editing pipelines require substantial manual hyperparameter tuning to produce reasonable results.
  • Synthetic training data can contain annotation errors and miss real-world diversity, limiting editing and generalization abilities.
  • MagicBrush provides a large-scale, manually annotated dataset for instruction-guided real-image editing across single-turn, multi-turn, mask-provided, and mask-free scenarios.
  • The dataset construction workflow uses qualified crowd workers, interactive DALL-E 2 editing, continuous edit sessions, and manual quality checks, yielding 5,313 sessions and 10,388 turns.
  • Fine-tuned InstructPix2Pix produces much better images and outperforms other baselines according to human preferences, while evaluations reveal MagicBrush remains challenging.

2 Related Work

Prior image-editing datasets and methods provide specialized or synthetic supervision, but open-domain, high-quality data remain scarce. MagicBrush addresses this gap with a manually annotated dataset intended for real-world instruction-guided editing.

  • Text-guided Image Editing: Text-guided editing methods include global description-guided, local description-guided, and instruction-guided approaches.
  • Text-guided Image Editing: Local description-guided editing offers fine-grained regional control and preserves other image areas, but requires users to provide masks.
  • Image Editing Datasets: Prior datasets often repurpose closed-domain caption datasets, restricting image categories and limiting generalization for models trained on them.
  • Image Editing Datasets: Automatically synthesized silver data may have undesirable quality, whereas EditBench contains only 240 manually curated examples, insufficient for training and comprehensive evaluation.
  • Image Editing Datasets: The literature therefore identifies a need for a manually annotated, large-scale dataset covering open-domain editing.

3 MAGICBRUSH Dataset

MagicBrush defines instruction-guided editing across mask-free or mask-provided and single-turn or multi-turn settings, then constructs a diverse, manually annotated dataset through crowdsourcing. Its multi-turn setting is especially user-friendly but prone to accumulated errors, and sampled data receive strong consistency and quality scores.

  • Problem Definition: Mask-free editing uses a source image and textual instruction, while mask-provided editing additionally supplies a free-form mask restricting the editing region.
  • Problem Definition: Single-turn editing evaluates each generated image against its corresponding ground truth, whereas multi-turn editing evaluates the final image after sequential instructions.
  • Problem Definition: Mask-free multi-turn editing is the most user-friendly yet challenging setting because iterative edits can accumulate errors.
  • Dataset Construction: Source images are sampled from MS COCO across 80 balanced object classes, with person-included images comprising 34.0% of the final dataset.
  • Dataset Annotation Pipeline: Crowd workers use DALL-E 2 with prompts and masks to iteratively obtain desired targets, supported by tutorials, stringent selection, and manual annotation procedures.
  • Data Composition: MagicBrush contains over 5K edit sessions and more than 10K edit turns spanning object, action, color, text, pattern, and quantity changes.
  • Data Quality Evaluation: Quality review of 500 sampled turns reports average consistency and image-quality scores of 4.1 and 3.9, respectively, on 5-point scales.

4 Experiments

Experiments evaluate mask-free and mask-provided editing across quantitative, qualitative, and human assessments. Fine-tuning InstructPix2Pix on MAGICBRUSH improves performance, while multi-turn editing remains challenging because errors accumulate.

  • Experiment Setup: The evaluation covers mask-free and mask-provided baselines in single-turn and multi-turn settings, using shared test sessions and multiple image-editing metrics.Metrics include pixel differences, image similarity, and text-image alignment.
  • Quantitative Evaluation: Fine-tuned InstructPix2Pix achieves the best or second-best results under most mask-free metrics, whereas multi-turn editing degrades across methods.The authors attribute the multi-turn decline to error accumulation during iterative editing.
  • Quantitative Evaluation: Mask-provided methods score highly on CLIP-I and DINO but perform less satisfactorily on L1, L2, and CLIP-T, which evaluate local regions.Fine-tuned InstructPix2Pix surpasses Blended Diffusion on CLIP-I and DINO.
  • Qualitative Evaluation: Qualitative comparisons show that fine-tuning reduces excessive modification and improves photorealism, although outputs remain inferior to ground-truth images.Text2LIVE and GLIDE fail on the man’s face and clothes, while the original InstructPix2Pix changes images excessively.
  • Human Evaluation: Human evaluations consistently favor fine-tuned InstructPix2Pix over competing methods for instruction consistency and image quality.Multi-choice, one-on-one, and individual evaluations all report stronger results for the fine-tuned model.
  • Human Evaluation: The performance gap between fine-tuned InstructPix2Pix and ground truth generally widens as multi-turn editing continues.This pattern highlights the difficulty of iterative editing on MAGICBRUSH.

5 Conclusion and Future Work

The paper presents MAGICBRUSH as a manually annotated dataset for instruction-guided real image editing, and reports that fine-tuned InstructPix2Pix achieves the best results while remaining inferior to ground truth.

  • MAGICBRUSH is a large-scale, manually annotated dataset for instruction-guided real image editing.
  • Fine-tuned InstructPix2Pix achieves the best results in the reported experiments.
  • Its edited images remain notably inferior to the ground-truth images.

Appendices

The appendices provide supplementary discussions, implementation details, qualitative studies, annotation information, and shared-artifact documentation.

  • Section B: Discussions: The supplementary material includes discussions of limitations, model bias, social impacts, ethical considerations, and asset licensing.
  • Section C: Implementation Details: Implementation details cover baselines and fine-tuning InstructPix2Pix with MAGICBRUSH.
  • Section D: More Qualitative Study: Additional qualitative studies cover both single-turn and multi-turn editing scenarios.
  • Section E: Data Annotation: The data-annotation appendix describes dataset collection and image-quality evaluation.
  • The shared artifacts table notes that the test split is password-protected to prevent web crawling for model training.

B.1 Limitations

The limitations concern annotation artifacts, missing global-editing data, potential model bias, incomplete bias elimination, social risks, privacy safeguards, and worker-payment procedures.

  • Less than 5% of edits may contain unmentioned minor modifications or appear slightly unnatural.
  • MAGICBRUSH does not contain global-editing data such as style transfer because annotation relied on DALL-E 2.
  • Using one model for ground-truth generation may introduce bias inherent in that model.
  • The authors state that inherent biases are challenging to eliminate completely.
  • Advanced image-editing systems could be exploited to create realistic fake or harmful content, motivating safeguards and responsible-AI frameworks.
  • COCO sourcing, prohibitions on identifiable information, and DALL-E 2 content rules are described as minimizing privacy and harmful-content concerns.

C Implementation Details

The implementation appendix describes baseline methods, reproducible evaluation settings, MAGICBRUSH fine-tuning, and automatic conversion of instructions into formats required by other baselines.

  • C.3 Training Details: All baselines use official default hyperparameters, while MAGICBRUSH fine-tuning runs for 168 epochs on two 40GB NVIDIA A100 GPUs.
  • C.2 Baseline Details: The baseline suite includes Open-Edit, VQGAN-CLIP, SD-SDEdit, Text2LIVE, Null Text Inversion, InstructPix2Pix, and HIVE for mask-free editing.
  • C.2 Baseline Details: Open-Edit fine-tunes a GAN on each given image using reconstruction and consistency losses.
  • C.2 Baseline Details: VQGAN-CLIP optimizes VQGAN embeddings using CLIP similarity between generated images and target text.
  • C.2 Baseline Details: SD-SDEdit adds stochastic differential equation noise to the source image and denoises it toward the target description.
  • C.2 Baseline Details: Text2LIVE generates an edited object on an additional layer using Vision Transformer fine-tuning, data augmentation, and CLIP supervision.
  • C.2 Baseline Details: Null Text Inversion optimizes a DDIM trajectory before editing through text-image cross-attention control.
  • C.2 Baseline Details: InstructPix2Pix edits without test-time tuning by balancing instruction faithfulness and source-image similarity.

D More Qualitative Study

Qualitative evaluations show that current image-editing baselines struggle with precise edits in both multi-turn and single-turn settings. InstructPix2Pix follows instructions somewhat but often changes too much, sacrificing details or introducing incorrect modifications.

  • Top-performing baselines show consistent qualitative behavior in multi-turn editing and the comparison shown in Figure 5.
  • Most baselines fail to edit images precisely even in the relatively easier single-turn scenario.
  • InstructPix2Pix follows instructions to some extent but tends to over-edit images, losing important details or making incorrect changes.
  • Single-turn qualitative comparisons provide baselines with their requested input formats, including masks and local descriptions where applicable.
  • Multi-turn qualitative comparisons provide baselines with their requested input formats, including masks and local descriptions where applicable.

E Data Annotation

The annotation process uses a structured tutorial, staged workflow, DALL-E 2 synthesis, and iterative quality control. Workers prepare source images, perform initial and follow-up edits, and receive feedback throughout the process.

  • The annotation tutorial is a self-contained 5-page document supplemented by a video demonstration to guide workers through data collection.
  • The annotation workflow has three phases: Preparation, Initial Editing, and Follow-up Editing.
  • Preparation: During Preparation, workers access the source image, log in to DALL-E 2, and upload the image for editing.
  • Initial Editing: An Edit Instruction directs how workers want to alter the image, while a Global Description describes the image after editing and guides DALL-E 2 synthesis.
  • Follow-up Editing: Workers can perform follow-up edit turns on the image generated during the first turn using a similar process.
  • Quality control combines worker feedback, continuous data checks, and trial-period review before workers advance to later stages.

E.3 Human Evaluation

Human evaluation on AMT assesses dataset quality and generated images through individual ratings and comparative judgments. The evaluation interfaces measure consistency and image quality using progressively more comparative designs.

  • Human evaluation on AMT assesses both dataset quality and images generated by different models.
  • Individual evaluation: Individual evaluation uses a 5-point Likert scale to measure image quality.
  • Multi-choice comparison: Multi-choice comparison asks evaluators to compare four top-performing methods, including Text2LIVE, GLIDE, InstructPix2Pix, and MAGICBRUSH-fine-tuned InstructPix2Pix.
  • One-on-one comparison: One-on-one comparison evaluates fine-tuned InstructPix2Pix against other strong baselines and the ground truth.
  • Each human evaluation task assesses both consistency and image quality, presenting the original image and textual instruction first.
Loading 2306.10012v3…