Source-linked AI summary
OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision
Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, Wenhu Chen
TL;DR
Existing instruction-guided image editing methods are limited by biased task coverage, poor data filtering, and fixed square-image training. OMNI-EDIT combines specialist supervision, multimodal-model importance scoring, EditNet, and varied aspect-ratio data, and evaluations report significant improvement over existing models. Its performance remains constrained by the underlying SD3 base model.
Problem
Existing image editing methods have limited skills, noisy training data from weak filtering, and poor support for non-square images.
Method
OMNI-EDIT trains one generalist using seven specialist models, distilled multimodal-model scoring, EditNet, and images with varied aspect ratios.
Results
OMNI-EDIT significantly outperforms existing models in automatic and human evaluations across diverse editing tasks, resolutions, and aspect ratios.
Takeaways & Limitations
Specialist-to-generalist supervision provides a framework for building image editors with broader editing skills and improved instruction adherence.
Takeaways & Limitations
The approach remains constrained by the weakness of the SD3 base model and may achieve greater potential with stronger base models such as Flux.
Abstract
from arXiv · showhide
Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases. In this paper, we present \omniedit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) \omniedit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that \omniedit can significantly outperform all the existing models. Our code, dataset and model will be available at https://tiger-ai-lab.github.io/OmniEdit/
1 INTRODUCTION
The paper identifies biased task coverage, poor synthetic-data filtering, and fixed square resolutions as barriers to practical image editing, then introduces OMNI-EDIT to address them. OMNI-EDIT combines seven specialist models, multimodal-model importance scoring, EditNet, and varied aspect-ratio training, with evaluations showing improved performance over existing models.
- 1 INTRODUCTION: Existing methods inherit limited editing capabilities because their synthetic data are biased toward the strengths and weaknesses of underlying generation models.Prompt2Prompt struggles with localized edits, whereas SD-Inpaint and DALLE-2 are ineffective at global edits.
- 1 INTRODUCTION: CLIP-score and DINO-score filtering correlate poorly with actual data quality, allowing noisy or artifact-prone samples into training datasets.The paper links this weak filtering to suboptimal training data and degraded model quality.
- 1 INTRODUCTION: Current models are trained on square image pairs, so their generalization to non-square images is poor.OMNI-EDIT instead incorporates images with varying aspect ratios and high resolution during training.
- 1 INTRODUCTION: OMNI-EDIT combines seven specialist models, multimodal-model data scoring, EditNet, and varied aspect-ratio training to address limited skills, noisy data, and square-image constraints.The proposed system targets task coverage, training-data quality, editing architecture, and image-shape versatility.
- 1 INTRODUCTION: OMNI-EDIT-BENCH contains diverse resolutions and prompts spanning the listed editing skills, and automatic plus human evaluations show significant improvement over existing baselines.The reported baselines include CosXL-Edit and UltraEdit.
2 PRELIMINARIES
The preliminaries describe diffusion models as latent-variable models that progressively add noise and learn to reverse that process. They then formulate instruction-based image editing as supervised learning over source images, editing instructions, and target images.
- 2.1 TEXT-TO-IMAGE DIFFUSION MODELS: Diffusion models progressively add Gaussian noise to image latents through a predefined forward Markov chain and learn a reverse denoising process.The latent dimensionality remains consistent across the original and noisy representations.
- 2.1 TEXT-TO-IMAGE DIFFUSION MODELS: The diffusion training objective minimizes a weighted mean squared error between the model prediction and the original data, conditioned on a text prompt.The timestep weighting factor is commonly set to 1 in prior work.
- 2.2 INSTRUCTION-BASED IMAGE EDITING IN SUPERVISED LEARNING: Instruction-based image editing is formulated as supervised learning using paired editing instructions and images before and after the edit.The source image, editing instruction, and target image form the supervised training example.
- 2.2 INSTRUCTION-BASED IMAGE EDITING IN SUPERVISED LEARNING: The OMNI-EDIT pipeline trains task-specific specialists, uses them to generate high-resolution multi-aspect-ratio data, scores pairs with a distilled multimodal model, and trains a weighted generalist.The pipeline is organized into four stages from specialist training through final generalist training.
3 LEARNING WITH SPECIALIST SUPERVISION
OMNI-EDIT uses specialist models for seven editing categories and importance-weighted supervision to train a generalist editor. Its weighting function favors high-quality edits, using distilled multimodal scoring instead of CLIP-based filtering.
- 3.1 LEARNING OBJECTIVE: The learning framework samples an editing pair, selects the specialist associated with its instruction, and weights the resulting demonstration for generalist training.The specialist distribution supplies edited examples while the importance function controls their contribution to training.
- 3.2 CONSTRUCTING SPECIALIST MODELS: OMNI-EDIT trains or builds a specialist model for each of seven image-editing categories to generate task-specific examples.The task groups and example instructions are summarized in Table 2.
- 3.3 IMPORTANCE WEIGHTING: The importance function assigns higher weights to samples likely to match the ground-truth editing distribution and lower weights to unlikely samples.This weighting is used as a quality measure for up-weighting high-quality training examples.
- 3.3 IMPORTANCE WEIGHTING: Unlike CLIP-score filtering, the proposed scoring approach addresses metrics reported to correlate poorly with actual training-data quality and human judgments.The paper motivates replacing simplified automatic filtering because it can produce noisy or suboptimal training data.
- 3.3 IMPORTANCE WEIGHTING: GPT-4o scores are distilled into InternVL2, which filters millions of synthesized examples by retaining samples scored at least 9, producing a 1.2M-example training dataset.The fine-tuned InternVL2 is reported to detect unusual distortions and edits that fail to follow instructions while being more efficient for large-scale scoring.
- 3.4 EDITNET: EditNet is compared with ControlNet and InstructPix2Pix as an architecture that lets control and original branches interact through intermediate representations.The comparison also states that EditNet updates text representations to improve task understanding.
4 EDITNET
Direct channel-wise image concatenation can compromise a pretrained diffusion model’s original representations. EditNet addresses this by adding a replicated control branch that interacts with the original diffusion-transformer blocks.
- 4 EDITNET: Directly fine-tuning a pretrained diffusion model such as SD3 with channel-wise image concatenation compromises its original representational capabilities.The paper refers to Figure 10 and Section 5.2 for the detailed comparison.
- 4 EDITNET: EditNet transforms diffusion-transformer models into editing models by replicating each original DIT block as a control branch.The control branch interacts with original DIT tokens, conditional image tokens, and editing prompts.
5 EXPERIMENTS
OMNI-EDIT is evaluated on diverse resolutions, aspect ratios, and editing tasks using automatic and human assessments, with ablations of importance sampling and EditNet. It outperforms baselines in instruction adherence, perceptual quality, and robustness across image formats.
- 5.1 MAIN RESULTS: OMNI-EDIT preserves base-model generation capabilities while producing sharper, cleaner edits, including a distinct NASA logo added to a T-shirt.Qualitative comparisons cover multiple editing tasks and image formats.
- 5.1 MAIN RESULTS: OMNI-EDIT achieves the highest Perceptual Quality score among the evaluated models.Table 3 reports the comparison on Omni-Edit-Bench.
- 5.1 MAIN RESULTS: OMNI-EDIT significantly outperforms the best baseline in following editing instructions accurately and minimizing over-editing.Both VIEScore and human evaluations support this comparison.
- 5.1 MAIN RESULTS: OMNI-EDIT maintains robust editing capabilities across diverse aspect ratios, while baselines struggle with blurry outputs and non-square images.The comparison includes 3:4 images and models trained at 512x512 or 256x256 resolutions.
- 5.2 ABLATION STUDY: Removing importance scoring and filtering causes a significant decrease in VIEScores for both perceptual quality and semantic consistency.The ablation compares OMNI-EDIT with a same-architecture baseline sampled directly from the pre-filtering dataset.
- 5.2 ABLATION STUDY: EditNet improves task-intent capture because intermediate representation interaction complements text-representation updates in the control branch.ControlNet variants still struggle with content removal despite updating text representations.
6 RELATED WORK
Prior image-editing datasets rely heavily on synthetic pairs or costly annotation, but their curation methods leave gaps in task coverage, alignment, accessibility, or global editing ability.
- Image Editing Datasets: Existing image-editing datasets commonly use synthetic data because expert-annotated editing pairs are difficult to collect.Several datasets build pairs with prompt-to-prompt, DALLE-2, DALLE-3, or other generative methods.
- Image Editing Datasets: Prior curation approaches face limitations including weak global editing, missing pixel alignment, and inaccessible proprietary models.The cited examples include MagicBrush, HQ-Edit, Emu Edit, and UltraEdit.
7 DISCUSSION
The discussion concludes that specialist-supervised generalist editing improves broad skill coverage, while the current base model limits the approach’s attainable quality.
- 7 DISCUSSION: OMNI-EDIT combines multiple specialist approaches and weighted loss to supervise a single generalist model across broad editing skills.The authors report a significant quality boost across the evaluated skills.
- 7 DISCUSSION: The SD3 base model limits OMNI-EDIT’s current potential, motivating future evaluation with Flux or other more capable base models.The authors state that output quality is highly influenced by the underlying base model.
A APPENDIX
Table 4 summarizes the OMNI-EDIT training dataset before and after importance scoring and filtering.
- A APPENDIX: Table 4 reports training-dataset sample counts before and after importance scoring and filtering with o-score ≥9.The table describes the dataset statistics used for training.
A.1.1 OBJECT REPLACEMENT
The object-replacement specialist masks an existing object and uses an inpainting model to fill the region with a specified target object. GPT-4o proposes replacement scenarios, which are converted into instruction-based training examples.
- Object Replacement: The replacement expert is trained to reconstruct the original source image from its masked version while receiving the mask and target caption.This trains the inpainting model to fill the masked region with the requested object.
- Data Generation: GPT-4o proposes five source-object and target-object replacement scenarios for each sampled image.The process samples 200K images and uses GPT-4o to identify source objects and suggest corresponding targets.
- Data Generation: GroundingDINO and SAM generate the source-object mask, which is then dilated before editing.Mask dilation expands the mask boundaries before the expert model performs replacement.
- Object Replacement: The specialist replaces a masked source object with a target object specified by caption Ctrg obj.The edited image is generated from the masked source image, the object mask, and the target-object caption.
- Data Generation: Each generated edit becomes an instruction-based pair containing the source image, edited image, and replacement instruction.The initial instruction is later expanded into more detailed natural-language instructions by large vision-language models.
A.1.2 OBJECT REMOVAL
The object-removal and related specialist pipelines create edits by masking regions and using inpainting or masked Prompt-to-Prompt generation. The broader pipeline applies importance sampling and evaluates OMNI-EDIT against specialists and baseline models.
- Object Removal: Object removal masks a selected object and inpaints the region with predicted background content Ctrg background.The removal expert receives the masked image, object mask, and predicted background content.
- Object Addition: Object addition is constructed as the inverse of removal by swapping the source and edited images and changing the instruction to add the object.This reuses removal examples to create addition examples.
- Attribute Modification: Attribute modification uses masked Prompt-to-Prompt generation to edit only the selected object while preserving the background.At each diffusion step, the mask combines edited-object latents with source-background latents.
- Environment Modification: Environment modification masks the foreground to preserve its details while changing the background during the early diffusion steps.The mask operation is applied for the first τenv = 0.7 of the timesteps.
- Data Selection: Importance sampling uses an overall score from large multimodal-model evaluation to construct the training dataset.The scoring prompts are illustrated in Figures 6 and 7, and the resulting dataset is described in Table 4.
- Evaluation: OMNI-EDIT performs comparably to specialist models on their specialized tasks and consistently outperforms other baseline models.These comparisons are reported in Table 5 and Figure 8.