Source-linked AI summary
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, Baobao Chang
TL;DR
Existing instruction-based image editing datasets are limited by instruction diversity, biases from synthetic images, and insufficient region-based data. ULTRAEDIT addresses these gaps with an automated pipeline combining human examples, LLMs, real-image anchors, and automatic region annotations. Models trained on ULTRAEDIT achieve strong results on MagicBrush and Emu Edit, while MagicBrush evaluation can favor models trained on its own data.
Problem
Existing image editing datasets have limited instruction diversity, implicit biases from text-to-image-generated images, and missing region-based editing data.
Method
ULTRAEDIT automatically curates editing data using LLM-expanded human examples, real images as anchors, prompt-to-prompt diffusion, and automatically generated region annotations.
Results
Models trained on ULTRAEDIT outperform canonical baselines on MagicBrush and improve with dataset scale on Emu Edit, setting a new record against a proprietary 10M-data baseline.
Takeaways & Limitations
Real-image anchors and region-based editing data are key design principles for building large-scale instruction-based image editing datasets.
Takeaways & Limitations
MagicBrush is biased toward its training set, so models trained on it can overfit the benchmark and lose general editing ability on other datasets.
Abstract
from arXiv · showhide
This paper presents UltraEdit, a large-scale (approximately 4 million editing samples), automatically generated dataset for instruction-based image editing. Our key idea is to address the drawbacks in existing image editing datasets like InstructPix2Pix and MagicBrush, and provide a systematic approach to producing massive and high-quality image editing samples. UltraEdit offers several distinct advantages: 1) It features a broader range of editing instructions by leveraging the creativity of large language models (LLMs) alongside in-context editing examples from human raters; 2) Its data sources are based on real images, including photographs and artworks, which provide greater diversity and reduced bias compared to datasets solely generated by text-to-image models; 3) It also supports region-based editing, enhanced by high-quality, automatically produced region annotations. Our experiments show that canonical diffusion-based editing baselines trained on UltraEdit set new records on MagicBrush and Emu-Edit benchmarks. Our analysis further confirms the crucial role of real image anchors and region-based editing data. The dataset, code, and models can be found in https://ultra-editing.github.io.
1 Introduction
ULTRAEDIT addresses limited instruction diversity, image biases, and missing region-based editing data with an automated pipeline combining human examples, LLMs, real-image anchors, and region annotations.
- Motivation: Prior datasets trade off human-aligned instruction diversity against scalability, limiting either dataset size or the variety of novel instructions.Human raters provide diverse instructions but are difficult to scale, while LLM-assisted generation is scalable but may have limited variety.
- Motivation: Text-to-image-generated source and target images can introduce implicit biases into instruction-based editing datasets.These biases motivate using real images as anchors rather than synthesizing every image with text-to-image models.
- Approach: ULTRAEDIT combines LLMs with in-context human-written examples to generate diverse editing instructions and target captions.The pipeline uses human-created instructions as context before expanding them with LLMs.
- Approach: The pipeline uses diverse real image-caption pairs as anchors and automatically generates editing regions for region-based samples.Prompt-to-prompt control produces free-form pairs, while modified inpainting uses generated region annotations.
- Dataset: ULTRAEDIT contains approximately 4 million editing samples spanning more than 9 editing types and supports both free-form and region-based editing.The dataset is presented as a large-scale, automatically curated resource intended to address shortcomings in existing datasets.
2 The ULTRAEDIT Dataset
ULTRAEDIT is constructed as a large-scale dataset with free-form and region-based editing samples, combining LLM-generated instructions, real-image anchors, diffusion editing, automatic region extraction, and quality filtering.
- Dataset formulation: ULTRAEDIT represents each sample with source and target images, an editing instruction, optional region mask, and source and target captions.The model maps the source image, instruction, and mask to the edited target image.
- Instruction generation: The pipeline combines human-written examples and LLM expansion to generate editing instructions, then uses captions and instructions to construct editing samples.This stage targets broader instruction variety while retaining human-provided context.
- Free-form editing: Approximately 1.6 million real image-caption pairs from datasets such as COCO and NoCaps serve as anchors for free-form image generation.Real anchors are used to mitigate biases associated with synthesizing all images with text-to-image models.
- Region-based editing: Region-based samples identify editable objects with recognition and language models, derive boxes and masks with GroundingDINO and SAM, and use modified inpainting.The resulting soft mask helps smooth transitions between the inpainting area and the rest of the image.
- Region-based editing: The modified pipeline alternates regular diffusion with masked inpainting to guide edits within the region while avoiding edge artifacts.It is compatible with prompt-to-prompt control and SDXL-Turbo and uses 3-7 diffusion steps for a target image.
- Dataset statistics: 4,108,262 editing samples comprise 4,000,083 free-form instances and 108,179 region-based instances, with 757,879 unique edits across over 9 instruction types.ULTRAEDIT is described as the largest publicly released instruction-based image editing dataset.
- Quality assessment: Quality filtering evaluates source-target similarity, caption-image alignment, and directional consistency between image changes and caption changes.The metrics include DINOv2 similarity, CLIP image similarity, SSIM, CLIP similarity, and CLIP Directional Similarity.
3 Experiments
The experiments evaluate ULTRAEDIT-trained diffusion editors on MagicBrush and Emu Edit Test, using benchmark comparisons, qualitative examples, and ablations of real-image anchors and region-based data. Results show strong general and region-based editing performance, with dataset scale, real-image anchors, and region annotations contributing to outcomes.
- Experimental Setup: ULTRAEDIT-trained diffusion models are evaluated against instruction-guided and description-guided baselines on MagicBrush and Emu Edit Test.The setup uses the InstructPix2Pix editing diffusion model with Stable Diffusion v1.5 and evaluates image fidelity, instruction alignment, and similarity metrics.
- MagicBrush Results: Training on approximately 450K free-form ULTRAEDIT samples significantly improves MagicBrush performance over canonical editing baselines.Adding approximately 100K region-based and 350K free-form samples further boosts general editing, while region input produces the best multi-turn results.
- Emu Edit Test: ULTRAEDIT models outperform InstructPix2Pix on Emu Edit Test, with advantages expanding as the training dataset scales.The strongest gains occur on CLIPdir and CLIPout, and the model sets a new record relative to the baseline trained on a proprietary 10M dataset.
- Qualitative Evaluation: Qualitatively, ULTRAEDIT better follows novel and multi-turn instructions while preserving source context than several baselines.Baselines often overedit or fail on edits such as blurring and special styles; region information gives ULTRAEDIT the best multi-turn performance.
- Insights and Analysis: Real-image anchors generally produce better models across 450K, 1M, and 1.5M scales, and scaling effects appear only when anchors are used.The authors hypothesize that removing real anchors increases image biases and hinders further scaling.
- Insights and Analysis: Region-based data improves free-form editing despite its smaller scale, while substantial region-data volume is needed for peak region-editing performance.The ablation varies free-form data from 200K to 400K and region-based data from 30K to 90K; qualitative results show more precise localized edits.
4 Related Work
Prior image-editing work has advanced text-guided editing, but available instruction-based datasets remain limited in scale and quality. Existing datasets rely on manual annotation or automatic generation with trade-offs in scalability, diversity, and bias.
- Image Editing via Generation: Text-guided editing methods use diffusion-based transformations, attention control, inversion, and related techniques to edit images according to descriptions.Representative methods include SDEdit and Prompt-to-Prompt, alongside later image-editing approaches.
- Image Editing Dataset: Existing image-editing datasets are scarce, with the largest cited dataset containing only around 300,000 samples.EditBench has 240 manually curated examples, while MagicBrush is manually annotated and limited in size.
5 Conclusion
ULTRAEDIT is presented as a large-scale, real-image-based dataset designed to address limited instruction diversity, image bias, and missing region-based editing data. Its construction combines human and LLM instruction generation with real-image anchors and automatically extracted editing regions.
- Conclusion: ULTRAEDIT combines LLM creativity with in-context human-rater examples to generate diverse editing instructions.Human raters first create editing examples, which language models expand into instructions and corresponding captions.
- Conclusion: The dataset uses real images and captions as anchors to reduce generation-model biases and improve reliability for real-world image editing.The source collection contains approximately 1.6 million high-quality image-caption pairs from diverse public datasets.
- Conclusion: ULTRAEDIT supports region-based editing through automatically produced region annotations in addition to free-form editing.This design addresses the lack of region-based editing data in prior datasets.
- Conclusion: The resulting pipeline generates 4.16 million text-only instruction-caption examples before image-pair construction.The language model is queried with sampled editing instructions and in-context edit examples.
A.3 Region-based Data Generation
Region-based ULTRAEDIT data is generated by identifying editable objects, localizing them with detection and segmentation models, and applying fused soft masks during image generation. The dataset records both free-form and region-based editing examples with diverse instruction types.
- Region Target Identification: The pipeline uses object recognition and LLM queries to determine which source-image objects an instruction targets.Object-oriented instructions receive localized targets; otherwise, the entire image is treated as the editing area.
- Region Mask Construction: Grounding DINO produces coarse bounding boxes, while SAM converts them into fine-grained object masks for editing regions.The fine-grained masks are expanded to create contour masks around the selected objects.
- Region Mask Construction: A soft mask fuses fine-grained and bounding-box masks to smooth color transitions between edited and unedited areas.The mask is used to guide image generation and reduce irregular boundaries.
- Masked Generation: During generation, the mask alternates between preserving the noisy latent outside the region and applying the denoising model within it.The inpainting-rate hyperparameter s is set between 0.2 and 0.8.
- Dataset Statistics: The dataset contains over 4,000,000 free-form instances and 100,000 region-based instances, making it the largest open-source image-editing dataset described here.It is also presented as the first large-scale dataset focused on region-based image editing.
B.1 Comparison with other dataset
ULTRAEDIT is compared with InstructPix2Pix as a higher-quality dataset for free-form image editing, emphasizing performance across task types and broader real-world editing coverage.
- ULTRAEDIT outperforms the InstructPix2Pix dataset across all evaluated task types on automatic data-quality metrics.The comparison is reported in Table 12.
- InstructPix2Pix may achieve higher CLIPimg scores on some tasks despite weaker visual similarity, revealing a limitation of semantic similarity as a data-quality signal for editing.
- Free-form and region-based examples demonstrate ULTRAEDIT’s editing capabilities across varied real-world scenarios.The dataset uses real images as anchors, while region-based examples emphasize preservation of image elements.
- ULTRAEDIT examples cover text modification, abstract object editing, multi-step instructions, and fine-grained editing.
C.2 Details on Benchmarks and Metrics
The benchmarks evaluate editing quality using complementary datasets and metrics, while MagicBrush carries a documented bias toward DALLE-2-generated edits and instructions.
- MagicBrush evaluates single-turn and multi-turn editing with annotator-defined instructions, editing masks, and DALLE-2-generated ground-truth images.
- MagicBrush metrics include L1, L2, CLIP similarity, DINO similarity, and CLIP-T for pixel differences, overall similarity, and local text-image alignment.
- MagicBrush is biased toward generated images and instructions that the DALLE-2 editor can successfully follow, potentially limiting diversity and complexity.
- Emu Edit Test uses source-to-edited-image L1 distance, CLIP image similarity, and DINO similarity without ground-truth images.
D.1 Human Evaluation
Human evaluation compares editing consistency, instruction alignment, and image quality across models on MagicBrush and Emu Test benchmarks, with ULTRAEDIT-trained models preferred over baselines.
- Evaluators compare edited images for consistency, instruction alignment, and image quality using pairwise choices of first image, second image, or tie.
- TrueSkill ratings are used to aggregate human-evaluation results on the MagicBrush and Emu Test benchmarks.
- The ULTRAEDIT-trained model produces more preferable editing results than the baselines, including the MagicBrush baseline.The paper notes that MagicBrush is reported to overfit on its test set.
- Qualitative evaluations include single-turn and multi-turn editing tasks on MagicBrush and varied editing tasks on Emu Edit Test.
D.3 Qualitative Evaluation on Real Image Anchors
Qualitative analyses show that real image anchors improve diversity, stability, visual consistency, and control, while region-based training and mask processing improve localized editing precision and boundary quality.
- Real image anchors substantially increase generated-image diversity and make results more stable and aligned with editing instructions.The anchors provide information beyond image captions alone.
- Real image anchors preserve visual consistency between source and target images and support more fine-grained edits.
- Training with region-based editing enables more precise background and localized edits even when no region input is provided at evaluation.
- The region-based pipeline filters oversized, undersized, and fragmented masks and adjusts fine-grained masks with soft masks.
- The proposed mask processing reduces artifacts and abrupt boundaries between edited regions and the rest of the image.
E.1 Limitations
ULTRAEDIT’s automatically generated data may contain biases, instruction errors, and occasional region-annotation inaccuracies. Its benchmark gains are reported as benchmark-specific, and ethical risks remain around fairness, misuse, and privacy.
- Automatically generated instructions and data may introduce biases and errors, while human raters and LLMs may miss nuances of creative editing.The authors also report occasional inaccuracies or inconsistencies in automatically produced region-based editing data.
- ULTRAEDIT’s diffusion-baseline improvements are benchmark-specific and may not generalize across all editing scenarios.The authors call for validation across a broader range of editing tasks.
- Real-image anchors and automatic generation may preserve biases that perpetuate stereotypes or unfair representations.The concern includes biases from the generation process, LLMs, human raters, and source images.
- The editing capability could be misused to create misleading or deceptive content and contribute to misinformation.The authors recommend safeguards and responsible use to mitigate this risk.
- Real image anchors may contain identifiable information, leaving a residual risk of unintentional privacy breaches despite anonymization efforts.
- The authors encourage bias checks, transparency, accountability, and ongoing community dialogue when developing or applying image-editing technologies.
F Datasheet for ULTRAEDIT
The datasheet describes ULTRAEDIT’s purpose, composition, distribution, maintenance, and intended uses, alongside documentation of noise, detoxification, and responsible-use constraints. The dataset is publicly distributed and intended primarily for training image-editing models.
- ULTRAEDIT was created to support image editing from natural-language instructions and optional region masks.
- The dataset was created by researchers from Peking University, BIGAI, Tsinghua University, UCLA, and UIUC.
- F.2 Distribution: The dataset is publicly available online, with downloads provided through Hugging Face.
- F.4 Composition: Instances include source and edited images, captions, instructions, edited objects, evaluation metrics, and optional editing regions in Apache Parquet columns.
- F.4 Composition: ULTRAEDIT contains approximately 4 million samples, including approximately 100,000 region-based editing instances and the remainder as free-form edits.
- The entire database is intended for training rather than predefined training, validation, and test splits.
- The authors identify possible noise and redundancies, while stating that offensive content was detoxified through an automated procedure.
- ULTRAEDIT is primarily intended for training capable instruction-following image-editing models and may also support benchmarking.