PaperlayerView exact revision on arXiv

Source-linked AI summary

HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing

Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, Cihang Xie

arXiv:2404.09990v1cs.CVcs.AI

TL;DR

Instruction-based image editing lacks sufficiently large, diverse, and well-aligned high-quality image–instruction pairs. HQ-Edit addresses this gap with a scalable GPT-4V and DALL-E 3 pipeline that expands seed data, generates diptychs, and post-processes them, while introducing GPT-4V-based quality metrics. Models trained on HQ-Edit achieve state-of-the-art instruction-based editing performance, with HQ-Edit-finetuned InstructPix2Pix substantially outperforming its vanilla version.

  • Problem

    High-quality datasets pairing editing instructions with corresponding images remain limited for general instruction-based single-image editing.

  • Method

    HQ-Edit expands online seed triplets with GPT-4, generates diptych image pairs with DALL-E 3, refines alignment with GPT-4V, and evaluates pairs using Alignment and Coherence.

  • Results

    12.3 increase at Alignment and 5.64 enhancement at Coherence are achieved by HQ-Edit-finetuned InstructPix2Pix over its vanilla version.

  • Takeaways & Limitations

    Models trained on HQ-Edit set a new state-of-the-art performance in instruction-based image editing and can surpass models fine-tuned with human-annotated data.

Abstract

from arXiv · show

This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200,000 edits. Unlike prior approaches relying on attribute guidance or human feedback on building datasets, we devise a scalable data collection pipeline leveraging advanced foundation models, namely GPT-4V and DALL-E 3. To ensure its high quality, diverse examples are first collected online, expanded, and then used to create high-quality diptychs featuring input and output images with detailed text prompts, followed by precise alignment ensured through post-processing. In addition, we propose two evaluation metrics, Alignment and Coherence, to quantitatively assess the quality of image edit pairs using GPT-4V. HQ-Edits high-resolution images, rich in detail and accompanied by comprehensive editing prompts, substantially enhance the capabilities of existing image editing models. For example, an HQ-Edit finetuned InstructPix2Pix can attain state-of-the-art image editing performance, even surpassing those models fine-tuned with human-annotated data. The project page is https://thefllood.github.io/HQEdit_web.

1 Introduction

HQ-Edit addresses the limited availability of high-quality instruction–image pairs for general single-image editing by combining GPT-4V and DALL-E 3 in a scalable curation pipeline. The resulting dataset emphasizes high resolution, detailed instructions, and precise alignment, while its metrics and experiments demonstrate improved editing performance.

  • Motivation: HQ-Edit targets general instruction-based single-image editing without external attribute guidance, a challenge insufficiently addressed because high-quality paired datasets remain limited.The paper identifies dataset quality and availability as a major hurdle for training instruction-based editing models.
  • Approach: GPT-4V and DALL-E 3 are leveraged to build high-quality editing data with detailed instructions, rich visual content, and alignment between edits and image pairs.The intended data properties include preserving fidelity in regions not targeted for modification.
  • Dataset: HQ-Edit contains around 200,000 detailed edit instructions and images at approximately 900 × 900 pixels, nearly double the resolution of existing datasets.The dataset is synthetically generated through a scalable pipeline using GPT-4V and DALL-E 3.
  • Approach: The curation pipeline consists of Expansion, Generation, and Post-processing, progressing from diverse online seed triplets to refined diptych prompts and aligned image pairs.Post-processing includes decomposing diptychs, warping and filtering paired images, and refining instructions.
  • Evaluation: Alignment measures semantic consistency with edit prompts and preservation of image fidelity, while Coherence evaluates aesthetics, lighting, shadows, style, and edge smoothness.These metrics are introduced to quantitatively assess image edit-pair quality.

2 Related works

Instruction-based editing lets users modify images through descriptions of where, what, and how to edit, but existing datasets trade off quality, scale, or diversity. HQ-Edit instead uses foundation models to generate high-quality editing pairs at scale and rewrites instructions for closer image alignment.

  • Instruction-based Image Editing: Instruction-based image editing follows descriptions of where, what, and how an image should be changed without requiring elaborate descriptions or region masks.This makes image modification easier and more flexible for users.
  • Instruction-based Image Editing: InstructPix2Pix introduced instruction-based editing by fine-tuning Stable Diffusion on examples generated with GPT-3 and Prompt2Prompt.Subsequent work focused on improving the quality or quantity of the underlying datasets.
  • Instruction-based Image Editing Datasets: Manual labeling can preserve dataset quality but restricts scale and diversity; Magicbrush contains about only 10,000 edits and emphasizes object-level transformations over global changes.The cited examples include global edits such as style or weather changes.
  • Instruction-based Image Editing Datasets: HQ-Edit uses GPT-4 and DALL-E 3 to generate high-quality editing pairs at scale and GPT-4V to rewrite instructions for closer image alignment.This differs from approaches based on manual annotation or attribute guidance.

3 HQ-Edit Dataset

HQ-Edit builds a large instruction-based image-editing dataset through expansion, diptych generation, and post-processing. The resulting data broadens instruction diversity, improves image–instruction alignment, and scores higher on GPT-4V-based quality metrics than existing datasets.

  • 3.1 Expansion: The pipeline expands seed triplets, generates input–output diptychs, and post-processes images and instructions into aligned editing pairs.It uses GPT-4 for expansion, DALL-E 3 for diptych generation, and GPT-4V plus other methods for refinement.
  • 3.1 Expansion: 293 seed triplets are expanded with GPT-4 into around 100,000 diverse image-editing instances.The seeds include 203 online samples and 90 samples from the Emu Edit test set.
  • 3.2 Generation: 98,675 data samples comprise input–output text pairs, edit instructions, and diptych images after generation.DALL-E 3 receives refined diptych prompts designed to produce paired images side by side.
  • 3.3 Post-processing: Warping aligns decomposed diptych images, while filtering removes samples with excessive deformation and instruction refinement improves textual correspondence.The filtering rule excludes images with more than 50% deformation on any single dimension.
  • 3.3 Post-processing: Rewriting and inversion substantially increase instruction length and diversity, yielding broader coverage of global and local editing operations.HQ-Edit includes edits such as weather, background, and style changes alongside object-level transformations.
  • 3.4 Data Quality Assessment: HQ-Edit’s metrics correlate more strongly with human preferences than CLIP, and its sampled data score 92.80 Alignment and 91.87 Coherence.These scores exceed InstructPix2Pix, HIVE, and MagicBrush on 500 sampled data points.

4 Experiments

Experiments evaluate HQ-Edit through metric validation, model comparisons, qualitative examples, and post-processing ablations. Fine-tuning InstructPix2Pix with HQ-Edit improves quantitative and qualitative editing performance, while post-processing strengthens alignment and coherence.

  • Human Evaluation: Human evaluation assesses 1,651 DALL-E 3 image pairs using five levels describing how well image changes follow edit instructions.Alignment, Clip Directional Similarity, and Human Evaluation Score are normalized to a 0–100 scale before averaging and correlation analysis.
  • Human Evaluation: HQ-Edit’s Alignment metric significantly surpasses CLIP Directional Similarity for evaluating whether edits follow instructions and preserve image fidelity.The comparison highlights CLIP Directional Similarity’s difficulty capturing editing nuances and intricate image details.
  • Quantitative Evaluation: 12.30 Alignment points and 5.56 Coherence points separate HQ-Edit from vanilla InstructPix2Pix, increasing scores from 34.71 to 47.01 and 80.52 to 86.16.HQ-Edit also surpasses HIVE and MagicBrush, which are fine-tuned on InstructPix2Pix.
  • Qualitative Evaluation: HQ-Edit-trained models correctly apply spatially specified edits and support diverse operations, including adding flowers to the designated ground region.The qualitative examples indicate more precise manipulation of image content according to editing specifications.
  • Ablation Study: Filtering reduces Coherence by 1.4 points but remains superior to other baselines, while the ablation evaluates RAW, Rewrite, Inverse, Warp, and Filter strategies.Applying all four processing operations produces the HQ-Edit dataset.
  • Ablation Study: 11.79 Alignment points and 0.94 Coherence points are gained by the rewrite strategy over raw DALL-E 3 pairs, while inverse adds 5.2 Alignment and 0.94 Coherence points.The raw pairs improve Alignment but hurt Coherence; the post-processing techniques mitigate this issue.

5 Conclusion

The study presents an automatic approach for synthesizing a large-scale image editing dataset with GPT-4V and DALL-E 3. It also introduces GPT-4V-based evaluation metrics, and reports state-of-the-art instruction-based image editing performance for models trained on HQ-Edit.

  • HQ-Edit automatically generates, rewrites, and expands seed image-editing data using GPT-4V and DALL-E 3.
  • Two GPT-4V-based metrics evaluate edited-image alignment with instructions and coherence of image content.
  • Models trained on HQ-Edit achieve state-of-the-art performance on instruction-based image editing.

A Prompts

The appendix lists prompts used for HQ-Edit data expansion and evaluation, including an expansion prompt for generating diverse examples and dedicated prompts for generation and metrics.

  • A Prompts: The prompt inventory includes EXPAND PROMPT for Expansion, DIPTYCH PROMPT and REWRITE PROMPT for Generation, and ALIGNMENT PROMPT and COHERENCE PROMPT for evaluation.
  • EXPAND PROMPT (GPT-4): EXPAND PROMPT asks GPT-4 to generate a specified number of examples from given examples.
  • EXPAND PROMPT (GPT-4): The expansion prompt requires wide variation in human characteristics, animals, insects, furniture, tools, and other object types.
  • EXPAND PROMPT (GPT-4): It also requests diverse backgrounds, including countries, natural environments, landscapes, and skies.
  • EXPAND PROMPT (GPT-4): Editing attributes must be diverse, while examples should remain clear, concise, comprehensive, and easy for DALL-E 3 to generate.
  • EXPAND PROMPT (GPT-4): The prompt design combines diversity requirements with constraints on clarity and generation ease.

A.2 Step #2: Generation

The Generation step uses prompts that ask DALL-E 3 to create diptychs containing an original image and an edited counterpart. The revised prompt improves clarity and concision before generation.

  • The rewrite prompt asks for a clearer, more concise prompt that is easier for DALL-E 3 to use when generating a diptych.
  • The rewritten output must begin with the marker “REVISED”.
  • The diptych prompt places the original image on the left and an otherwise identical image on the right with the requested edit action.

A.3 Evaluation Metric

The Alignment metric rates whether edit text correctly and comprehensively describes the change between two images while preserving unrelated content.

  • Alignment scores the correctness and comprehensiveness of an edit description from 0 to 100.Correctness penalizes changes mentioned but not made; comprehensiveness penalizes changes made but omitted.
  • Stylization or lighting edits receive lower scores when content changes or image details are not preserved.
  • Local edits receive lower scores when irrelevant regions or the overall image style change.
  • Descriptions are penalized when the first image already contains the requested attribute or when the two images are unrelated.The prompt specifies scores below 80 for a pre-existing attribute and below 50 for unrelated images.

COHERENCE PROMPT (GPT-4V)

The Coherence prompt asks GPT-4V to score an edited image from 0 to 100 based on visual harmony, integration, and aesthetic consistency.

  • Coherence rates an image from 0 to 100, where 100 indicates perfect harmony among its components.The assessment considers whether elements blend naturally without conflicting or clashing.
  • The evaluation checks consistency in lighting and shadows, contextual fit of elements, edge smoothness, and integration with surroundings.
  • The prompt also evaluates aesthetic uniformity and visual flow, penalizing abrupt interruptions caused by disharmonious elements.
  • Scores of 90-100 are reserved for images that could pass as flawlessly captured scenes without discernible disharmony.
  • Moderate-to-high scores of 70-89 allow minor disharmony that does not significantly detract from overall harmony.

B.1 More visualization results

The visualizations compare datasets using edit instructions, Alignment, and Coherence, with HQ-Edit reported to have higher image quality and better image-text alignment.

  • HQ-Edit is visualized alongside InstructPix2Pix, MagicBrush, and HIVE using edit instructions, Alignment, and Coherence.
  • HQ-Edit is reported to possess higher image quality and better image-text alignment than the compared datasets.
  • Figures 11 through 14 show input images on the left and output images on the right for InstructPix2Pix, MagicBrush, HIVE, and HQ-Edit, respectively.
  • One HQ-Edit example changes a woman’s hair to blonde and adds waves, receiving Alignment 100 and Coherence 95.
  • Another HQ-Edit example replaces a heavy-duty power drill with a high-tech precision power tool, receiving Alignment 100 and Coherence 95.
Loading 2404.09990v1…