Source-linked AI summary
SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing
Yuying Ge, Sijie Zhao, Chen Li, Yixiao Ge, Ying Shan
TL;DR
Instruction-guided image editing needs models to understand language and images, follow iterative requests, and preserve realism, but high-quality large-scale training data are scarce. SEED-Data-Edit combines automated, real-world, and expert-annotated multi-turn examples, and the authors fine-tune SEED-X with the dataset. They report that SEED-X-Edit follows editing instructions more accurately than models trained with existing datasets.
Problem
Training instruction-guided editing models is constrained by scarce high-quality, large-scale datasets for learning to interpret and execute edits while preserving realism and semantic consistency.
Source: §1 · p. 2Method
Results
SEED-X-Edit adheres to editing instructions more accurately than models trained with existing image-editing datasets.
Source: §4 · p. 7Takeaways & Limitations
The authors present SEED-Data-Edit and SEED-X-Edit as released resources for language-guided image editing.
Source: §5 · p. 7Multi-turn editing sequences were not used during fine-tuning; the multi-turn data were treated as single-turn examples.
Source: §4 · p. 7
Abstract
from arXiv · showhide
In this technical report, we introduce SEED-Data-Edit: a unique hybrid dataset for instruction-guided image editing, which aims to facilitate image manipulation using open-form language. SEED-Data-Edit is composed of three distinct types of data: (1) High-quality editing data produced by an automated pipeline, ensuring a substantial volume of diverse image editing pairs. (2) Real-world scenario data collected from the internet, which captures the intricacies of user intentions for promoting the practical application of image editing in the real world. (3) High-precision multi-turn editing data annotated by humans, which involves multiple rounds of edits for simulating iterative editing processes. The combination of these diverse data sources makes SEED-Data-Edit a comprehensive and versatile dataset for training language-guided image editing model. We fine-tune a pretrained Multimodal Large Language Model (MLLM) that unifies comprehension and generation with SEED-Data-Edit. The instruction tuned model demonstrates promising results, indicating the potential and effectiveness of SEED-Data-Edit in advancing the field of instructional image editing. The datasets are released in https://huggingface.co/datasets/AILab-CVC/SEED-Data-Edit.
1 Introduction
Instruction-guided image editing lets people manipulate images with natural-language instructions, but training is constrained by a lack of high-quality, large-scale datasets. SEED-Data-Edit combines automated, real-world, and human-annotated multi-turn data, and the authors fine-tune SEED-X to demonstrate the dataset's potential.
1 Introduction
- A shortage of high-quality, large-scale datasets is a major hurdle for training models to interpret and execute image-editing instructions.The task also requires understanding language and visual content, handling iterative instructions, and preserving realism and semantic consistency.Source: §1 · p. 2
- SEED-Data-Edit combines diverse automated editing pairs, real-world user requests, and human-annotated multi-turn edits to train language-guided editing models.The three sources respectively provide volume and diversity, real-world user intentions, and examples of iterative editing.Source: §1 · p. 2
- The automated pipelines produce millions of editing pairs, while the real-world and multi-turn components add user requests and iterative edits.The real-world collection has 52K pairs; the human-annotated multi-turn collection has 95K pairs, with up to five rounds per sequence.Source: §1 · p. 2
- The authors fine-tune the pretrained MLLM SEED-X with SEED-Data-Edit and report promising language-guided editing results.SEED-X unifies multimodal comprehension and generation; its visual de-tokenizer uses the source image alongside high-level image features to recover fine-grained details.Source: §1 · p. 2
2 Related Work
Existing editing datasets differ in their image sources, real-world coverage, and multi-turn annotations. SEED-Data-Edit combines automated, real-world, and expert-annotated data at larger scale and with longer editing sequences than the compared datasets.
2 Related Work
- SEED-Data-Edit contains 3.7M image-editing pairs and 21K multi-turn sequences of up to five rounds, combining automated, real-world, and expert-annotated data.InstructPix2Pix has only single-turn model-generated pairs; MagicBrush has 10K pairs and up to three rounds; HQ-Edit uses generated diptychs that may not preserve fine details or realism.Source: §2 · p. 3
- InstructPix2Pix uses Prompt-to-Prompt to generate paired images from LAION-Aesthetics captions, but its data are model-generated and single-turn.It does not include real images.Source: §2 · p. 3
- MagicBrush uses crowd workers to annotate MS COCO images with DALL-E 2, but contains 10K pairs and sequences of at most three rounds.Source: §2 · p. 3
- HQ-Edit generates diptychs with GPT-4V and DALL-E 3, but diptych generation does not guarantee fine-detail preservation and the generated images lack realism.Source: §2 · p. 3
3 Dataset
SEED-Data-Edit brings together automated edits, real-world user requests, and expert-performed multi-turn edits. Its automated pipelines provide diverse pairs, while the other sources reflect real intentions and iterative editing.
3 Dataset
- The dataset combines automated examples, real-world editing requests fulfilled by Photoshop experts, and multi-turn edits by experts on real images.Source: §3 · p. 3
3.1 Automated Pipeline-generated Data
- Two automated pipelines generate edits: one removes objects and reverses the operation to create additions; the other produces changes in style, object, color, material, or expression.Pipeline (a) uses segmentation and inpainting, while pipeline (b) uses image-guided text-to-image generation; filtering yields 1.5M and 2.0M pairs, respectively.Source: §3.1 · pp. 3–4
3.2 Real-world Scenario Data
- The real-world collection contains 52K pairs from four websites, with human annotators rewriting instructions to match expert-edited images.The requests are described as more complex and imaginative, reflecting real-world user intentions.Source: §3.2 · p. 4
3.3 Multi-turn Editing Data
- Photoshop experts created 21K multi-turn rounds, yielding 95K editing pairs across sequences of up to five rounds.Edits include object replacement, addition and removal, action changes, text or pattern changes, and count changes.Source: §3.3 · p. 4
4 SEED-X-Edit
SEED-X-Edit is a fine-tuned multimodal model for language-guided image editing. The report says it follows instructions more accurately than models trained with existing editing datasets, while noting that multi-turn sequences were used as single-turn examples during training.
4 SEED-X-Edit
- The authors fine-tune pretrained SEED-X with SEED-Data-Edit, InstructPix2Pix, and MagicBrush using LoRA.SEED-X decodes predicted ViT features with a pretrained visual de-tokenizer to understand multimodal inputs and generate an edited target image.Source: §4 · p. 7
- Multi-turn editing sequences are not used as sequences in fine-tuning; their data are instead treated as single-turn examples.The authors state that multi-turn image editing will be explored in future work.Source: §4 · p. 7
- SEED-X-Edit adheres to editing instructions more accurately than the compared models trained with existing editing datasets.Examples include adding sunglasses to one animal rather than both and removing a bottle when baseline models fail to do so.Source: §4 · p. 7
5 Conclusion
SEED-Data-Edit combines three types of image-editing data for language-guided editing and is used to train SEED-X-Edit, which achieves promising results. The dataset and model are released.
- SEED-Data-Edit combines large-scale automated edits, real-world examples, and human-annotated multi-turn edits.Source: §5 · p. 7
- Fine-tuning pretrained SEED-X with the dataset produces SEED-X-Edit, which achieves promising results.Source: §5 · p. 7
- The dataset and instruction-tuned SEED-X-Edit model are released.Source: §5 · p. 7