PaperlayerView exact revision on arXiv

Source-linked AI summary

ImgEdit: A Unified Image Editing Dataset and Benchmark

Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, Li Yuan

arXiv:2505.20275v1cs.CV

TL;DR

Open-source image-editing models face limited high-quality data and insufficient evaluation benchmarks. ImgEdit addresses these gaps with a curated dataset, quality-controlled construction pipeline, editing model, and multi-suite benchmark; ImgEdit-E1 outperforms existing open-source models on multiple tasks, while the benchmark analyzes factors associated with editing performance.

  • Problem

    Open-source image-editing models are constrained by limited high-quality datasets and insufficient benchmarks, including inadequate coverage of complex editing tasks.

  • Method

    ImgEdit combines an automated quality-control pipeline, a dataset with single-turn and multi-turn edits, ImgEdit-E1, and ImgEdit-Bench for evaluating editing performance.

  • Results

    ImgEdit-E1 substantially outperforms other open-source editing models in evaluated tasks, while ImgEdit-Bench examines instruction understanding, grounding, and editing ability.

  • Takeaways & Limitations

    ImgEdit provides high-quality editing data and comprehensive evaluation intended to help close the gap between open-source and closed-source image-editing approaches.

  • Takeaways & Limitations

    The paper does not provide detailed ablation studies of ImgEdit-E1’s architecture, training data, or training process, and its downstream text-generation capabilities are not yet optimal.

Abstract

from arXiv · show

Recent advancements in generative models have enabled high-fidelity text-to-image generation. However, open-source image-editing models still lag behind their proprietary counterparts, primarily due to limited high-quality data and insufficient benchmarks. To overcome these limitations, we introduce ImgEdit, a large-scale, high-quality image-editing dataset comprising 1.2 million carefully curated edit pairs, which contain both novel and complex single-turn edits, as well as challenging multi-turn tasks. To ensure the data quality, we employ a multi-stage pipeline that integrates a cutting-edge vision-language model, a detection model, a segmentation model, alongside task-specific in-painting procedures and strict post-processing. ImgEdit surpasses existing datasets in both task novelty and data quality. Using ImgEdit, we train ImgEdit-E1, an editing model using Vision Language Model to process the reference image and editing prompt, which outperforms existing open-source models on multiple tasks, highlighting the value of ImgEdit and model design. For comprehensive evaluation, we introduce ImgEdit-Bench, a benchmark designed to evaluate image editing performance in terms of instruction adherence, editing quality, and detail preservation. It includes a basic testsuite, a challenging single-turn suite, and a dedicated multi-turn suite. We evaluate both open-source and proprietary models, as well as ImgEdit-E1, providing deep analysis and actionable insights into the current behavior of image-editing models. The source data are publicly available on https://github.com/PKU-YuanGroup/ImgEdit.

1 Introduction

ImgEdit addresses shortcomings in open-source image-editing data and evaluation by combining a quality-controlled construction pipeline, a broad dataset, an editing model, and a benchmark. Its dataset covers diverse single-turn and multi-turn editing, while its released models and benchmark support analysis of editing performance.

  • Motivation: Open-source image editing is limited by low-quality datasets, weak prompt design, inadequate complex-task coverage, and inaccurate benchmarks.Existing data can contain poor resolution, simplistic prompts, negligible edit regions, inaccurate edits, concept imbalance, and imprecise filtering.
  • Approach: ImgEdit combines automated data construction, a large-scale dataset, an advanced editing model, and a comprehensive evaluation benchmark.The framework is presented as a unified response to the identified data and evaluation challenges.
  • Approach: The construction pipeline filters unsuitable images, adds object-level grounding, generates diverse prompts, creates edited pairs with task-specific workflows, and evaluates prompt adherence and visual fidelity.Filtering considers aesthetics, resolution, and editable-region size; grounding uses detection and segmentation models.
  • Dataset: ImgEdit contains 1.1 million single-turn samples spanning ten edit tasks and 110,000 multi-turn samples covering three novel interaction types.The dataset is designed to include both representative single-turn operations and challenging multi-turn interactions.
  • Models and evaluation: ImgEdit-Bench evaluates editing models through basic, challenging, and multi-turn suites, while ImgEdit-E1 and ImgEdit-Judge provide an editing model and human-preference-aligned evaluator.The benchmark and models extend the framework beyond dataset construction.

2 Related Work

Prior image-editing datasets vary in automation and human quality control, but remain constrained in scale, domain coverage, or data quality. Existing benchmarks often rely on generic similarity metrics that correlate poorly with human judgments and may omit common or complex settings.

  • Datasets: Existing datasets range from fully synthetic or automated pipelines to datasets with varying degrees of human quality control.MagicBrush adds human annotations but contains only 10,000 pairs, while InstructPix2Pix is confined to a synthetic P2P domain.
  • Benchmarks: Current image-editing benchmarks remain rudimentary because generic metrics such as CLIP score, PSNR, and SSIM correlate poorly with human judgments.Task-specific benchmarks broaden coverage, but some still measure performance primarily through similarity.
  • Benchmarks: Existing benchmarks also have scope gaps, including neglect of common settings or insufficient coverage of highly complex scenes.The cited comparison contrasts SmartEdit’s complex-scene focus with its omission of most common settings.

3 ImgEdit: A High Quality Dataset

ImgEdit defines a broad set of single- and multi-turn editing tasks and builds its dataset through automated filtering, grounding, prompt generation, tailored in-painting, and post-processing. The resulting dataset contains 1.2 million high-quality pairs across 13 editing categories, including multi-turn examples and subject-consistent extraction and visual-edit subsets.

  • 3.1 Edit Type Definition: ImgEdit divides editing into single-turn tasks covering practical edits and multi-turn tasks covering continuous interactions.Single-turn editing is organized into local, global, visual, and hybrid categories; multi-turn editing addresses content memory, content understanding, and version backtracking.
  • 3.1 Edit Type Definition: Single-turn editing includes local operations, global transformations, reference-image visual edits, and hybrid multi-object instructions.Local operations include add, remove, replace, alter, motion change, and object extraction; global edits include background replacement and style or tone transfer.
  • 3.1 Edit Type Definition: Multi-turn editing tests content memory, pronoun-dependent content understanding, and version backtracking across editing results.These challenges cover persistent constraints, omitted subjects in later instructions, and edits based on earlier versions.
  • 3.2 Automatic Dataset Pipline: The data pipeline filters low-quality images, creates object-level grounding and segmentation information, generates localized instructions, performs scenario-specific in-painting, and applies precise post-processing.The pipeline uses aesthetic, resolution, editable-region, object-area, CLIP, and prompt-guided quality criteria across preparation and post-processing.
  • 3.3 ImgEdit-E1: ImgEdit-E1 combines a vision-language model, vision encoder, and Diffusion-in-Transformer backbone to process editing instructions and original images.Qwen2.5-VL-7B, SigLIP, and FLUX are identified as the model components in the architecture.
  • 3.4 Dataset Statistics: 1.2 million high-quality image-editing pairs span 13 editing categories, including 110k multi-turn examples.The object extraction and visual edit subsets are described as the first editing tasks with high subject consistency.

4 ImgEdit-Bench: A Comprehensive Benchmark

ImgEdit-Bench evaluates image-editing systems across basic, challenging single-turn, and multi-turn scenarios. Its scoring framework measures instruction adherence, editing quality, and detail preservation, using rubric-based VLM ratings and a learned evaluation model.

  • 4 ImgEdit-Bench: A Comprehensive Benchmark: ImgEdit-Bench evaluates basic editing, complex understanding-grounding-editing, and multi-turn capabilities.The multi-turn suite covers content understanding, content memory, and version backtracking.
  • Basic-Edit Suite: The basic suite covers nine common editing tasks across six semantic super-categories and manually collected Internet images.Tasks include add, remove, alter, replace, style transfer, background change, motion change, hybrid edit, and cut-out.
  • Understanding-Grounding-Editing Suite: The challenging single-turn suite uses 47 complex images with occlusion, repeated instances, camouflage, uncommon subjects, and demanding spatial or multi-object instructions.Prompts require spatial reasoning, multi-object coordination, compound or fine-grained operations, and large-scale modifications.
  • Multi-Turn Suite: The multi-turn suite evaluates content memory, content understanding, and version backtracking using 10 images per task and three interaction turns per case.The suite targets real-world continuous editing use cases with manually designed prompts.
  • 4.2 Evaluation Metrics: Scores cover instruction adherence, image-editing quality, and detail preservation, with the latter two capped by instruction-adherence scores.Instruction adherence measures prompt comprehension and conceptual understanding; quality measures target manipulation precision, while detail preservation measures unchanged-region fidelity.
  • 4.3 ImgEdit-Judge: ImgEdit-Judge is trained on 200k post-processed rating records and compared with human annotators, Qwen2.5VL-7B, and GPT-4o-mini in a human study.The evaluator is intended to provide an open-source VLM-based alternative for image-editing assessment.

5 Experiments

The experiments compare open- and closed-source image-editing models across single-turn and multi-turn settings, then analyze qualitative behavior and performance factors. ImgEdit-E1 performs strongly on instruction adherence, preservation, localization, and extraction, while current multi-turn support remains inadequate.

  • Multi-turn Evaluation: GPT-4o-Image and Gemini-2.0-Flash support version backtracking within two turns but provide inadequate overall support for multi-turn edits.The models have minimal content memory and may misunderstand references or fail to retain premises.
  • Qualitative Evaluation: ImgEdit-E1 and GPT-4o-Image preserve intricate details while accurately following instructions across representative editing tasks.They successfully preserve snow during color changes and perform accurate removal and color-modification edits.
  • Discussion: ImgEdit-E1 and Step1X-Edit substantially outperform other open-source models, underscoring the importance of stronger text encoders.Conventional T5- or CLIP-based encoders handle simple tasks but perform poorly on complex, region-specific edits.
  • Discussion: The benchmark identifies instruction understanding, grounding, and editing as three key factors influencing image-editing performance.Understanding depends largely on the text encoder, grounding on localization and visual perception, and editing on training-data quality, size, and diversity.
  • Discussion: ImgEdit-E1 exhibits superior performance on precise-localization tasks such as Attribute Alteration and Object Extraction.The discussion links these tasks to instruction understanding, grounding, and spatial information in prompts.

6 Conclusion

The conclusion presents ImgEdit as a unified dataset, pipeline, model, and benchmark intended to address data-quality and evaluation limitations in image editing. It reports strong ImgEdit-E1 performance and emphasizes the framework’s broader value for model analysis and development.

  • Conclusion: ImgEdit addresses existing data-quality limitations, introduces practical editing categories, and provides a robust pipeline for future dataset construction.The paper positions the framework as a contribution spanning dataset quality and editing-task coverage.
  • Conclusion: ImgEdit-Bench evaluates image-editing models across novel dimensions, providing insights into data selection and architectural design.The conclusion presents the benchmark as part of the paper’s broader contribution to model development.
  • Model Design: ImgEdit-E1 uses Qwen2.5-VL and Siglip vision features as inputs to Flux’s text branch.Its vision features are concatenated with Vision-Language Model features before being supplied to Flux.
  • Model Design: Flux serves as the generative foundation because its dual-stream architecture integrates semantics and images effectively.Its pretrained weights also reduce training costs.

A.4 Details of Training Strategy

ImgEdit-E1 is trained in two stages that progressively connect Qwen2.5-VL and Siglip to Flux through trainable MLP components.

  • A.4 Details of Training Strategy: Stage one freezes Qwen2.5-VL and Flux while training only the connecting MLP.Training uses global batch size 128, Prodigy, and learning rate 1.0.

A.5 Limitations

The paper limits its analysis of ImgEdit-E1 because the model is not the core contribution, leaving detailed studies of its structure, data, and training for future work.

  • A.5 Limitations: The authors did not perform detailed ablations of ImgEdit-E1’s model structure, training data, or training process.They identify these aspects as areas for deeper future investigation.
  • A.5 Limitations: ImgEdit-E1’s editing capabilities are not yet optimal for downstream scenarios such as text generation.The paper states that this limitation is shared by current open-source models.

B.1 Additional Details of Dataset Statistic

ImgEdit provides task-level dataset statistics and word-cloud summaries, while retaining post-processing results for further filtering. These materials characterize the collected editing tasks and support inspection of dataset composition.

  • ImgEdit presents word clouds for different editing tasks.
  • Table 3 reports detailed statistics for all ImgEdit editing types.
  • Post-processing results are included so users can apply additional filtering.

B.2 Samples of Collected Data

The collected data cover diverse single-turn and multi-turn edits, generated through metadata-aware prompting and task-specific inpainting workflows. The pipeline uses specialized controls and processing choices for replacement, object editing, style transfer, and iterative edits.

  • Samples of Collected Data: ImgEdit includes samples from all edit tasks in both single-turn and multi-turn data.
  • Samples of Collected Data: GPT-generated instructions combine captions, objects, image resolution, bounding-box coordinates, and task-specific prompts.
  • Samples of Collected Data: The dataset pipeline adopts task-tailored ComfyUI inpainting workflows for a lightweight, user-friendly implementation.
  • Samples of Collected Data: Replacement and background-change edits soften mask edges to reduce abrupt transitions and produce seamless results.
  • Samples of Collected Data: Add and Remove use inverse mask-inpainting processes with prompts containing terms such as "empty" and "blank".
  • Samples of Collected Data: Attribute alteration uses Canny edges, Canny LoRA, and ControlNet, followed by edge softening to improve visual quality.
  • Samples of Collected Data: Object extraction and reference replacement invert their roles: generated clean-background objects replace real-image objects through Flux-Redux in-context processing.
  • Samples of Collected Data: Style transfer uses SDXL for fidelity and Canny edge detection to preserve finer image details.

D.2 Multi-Turn Qualitative Analysis

The qualitative analysis includes multi-turn test cases for GPT-4o-Image and Gemini-2.5-flash. These examples are presented as figures for examining their multi-turn editing behavior.

  • The analysis lists GPT-4o-Image multi-turn test cases in Figure 11.
  • The analysis lists Gemini-2.5-flash multi-turn test cases in Figure 12.
Loading 2505.20275v1…