Source-linked AI summary
AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea
Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, Yueting Zhuang
TL;DR
Instruction-based image editing is limited by scarce, low-quality datasets with narrow editing types and weak support for complex multimodal and reference-based instructions. AnyEdit addresses this with a large, diverse dataset and adaptive collection pipeline, then trains AnySD with task-aware routing; experiments show superior semantic and visual similarity on standard benchmarks.
Problem
Existing instruction-editing datasets are scarce, low-quality, and limited in editing types, constraining support for complex multimodal and reference-based instructions.
Method
AnyEdit combines a multi-type, multi-scene dataset with adaptive data pipelines, automated result selection, and AnySD’s task-aware routing for unified image editing.
Results
0.872 CLIPim on EMU-Edit Test and 0.881 DINO on MagicBrush Test show AnySD surpasses SOTA methods in semantic and visual similarity.
Takeaways & Limitations
AnyEdit and AnySD support high-quality instruction following across diverse editing tasks while preserving unchanged image elements.
Takeaways & Limitations
AnyEdit-Test is designed to expose weaknesses on complex and multi-scene editing tasks, where existing models can fail or show notable performance drops.
Abstract
from arXiv · showhide
Instruction-based image editing aims to modify specific image elements with natural language instructions. However, current models in this domain often struggle to accurately execute complex user instructions, as they are trained on low-quality data with limited editing types. We present AnyEdit, a comprehensive multi-modal instruction editing dataset, comprising 2.5 million high-quality editing pairs spanning over 20 editing types and five domains. We ensure the diversity and quality of the AnyEdit collection through three aspects: initial data diversity, adaptive editing process, and automated selection of editing results. Using the dataset, we further train a novel AnyEdit Stable Diffusion with task-aware routing and learnable task embedding for unified image editing. Comprehensive experiments on three benchmark datasets show that AnyEdit consistently boosts the performance of diffusion-based editing models. This presents prospects for developing instruction-driven image editing models that support human creativity.
1. Introduction
Instruction-based image editing seeks natural-language control over selected image elements, but existing datasets and models are limited by scarce high-quality data, narrow editing types, and weak support for complex multimodal instructions.
- High-quality instruction-editing data is scarce, hindering development of powerful instruction-guided image editing models.
- Prior datasets have limited editing types and low data quality, with weak support for complex multimodal and reference-based instructions.
- AnyEdit categorizes editing into local, global, camera-move, implicit, and visual groups to cover diverse editing capabilities.
- AnySD uses task-aware routing and learnable task embeddings to support diverse editing types within a unified instruction-based editing model.
- AnyEdit and AnySD report improved instruction adherence and image fidelity across diverse editing types and scenarios.
2. Related Work
Existing instruction-based editing datasets differ in collection strategy but generally provide narrower coverage, while AnyEdit combines multiple data types to address complex and multimodal editing needs.
- Existing datasets use either automatic collection or human curation, with MagicBrush containing only 10K manually annotated editing pairs because of labor costs.
- Prior datasets commonly neglect complex customized editing, unified implicit editing, viewpoint changes, or multimodal inputs.
- AnyEdit combines five data groups covering 25 editing types and introduces counterfactual synthetic scenes to address data bias.
3. AnyEdit
AnyEdit constructs a broad editing dataset through diverse source data, instruction generation, task-specific editing pipelines, and automated quality filtering.
- The construction pipeline prepares real-world and synthetic data, generates instructions with an LLM, validates instructions, edits adaptively, and assesses image quality.
- The dataset begins with approximately 680K real-world image-caption pairs enriched by multimodal language models.
- Adaptive pipelines customize editing methods for specific edit types and input formats instead of relying on a shared fixed pipeline.
- Quality enhancement combines instruction-validation pre-filtering with image-quality post-filtering to remove unsuitable editing pairs.
- AnyEdit contains 2.5 million high-quality editing pairs across five domains and 25 distinct editing types.
- +25.2% in DINOv2 and +16.0% in CLIPin versus UltraEdit indicate improved pixel-level consistency and instruction reflection.
4. Method
AnySD is designed to unify instruction-based editing across diverse tasks and conditioning types. It combines task-aware routing and learnable task embeddings with AnyEdit data to control task-specific editing behavior.
- AnySD Architecture: AnySD addresses training challenges involving multimodal semantic alignment, edit granularity, and coordination across diverse editing tasks.Its architecture includes a visual prompt projector, task-aware routing, and learnable task embedding.
- AnySD Architecture: AnySD supports original images, editing instructions, and visual prompts for various editing tasks.
- Task-aware Routing: Task-aware routing uses Mixture-of-Experts blocks to adapt visual attention to task-specific editing requirements and edit granularity.Experts share text attention while the router distributes task-dependent visual attention weights.
- Learnable Task Embeddings: Learnable task embeddings guide task-specific granularity by conditioning the visual input to MoE blocks and routing weights across experts.They are inserted before the MoE block to reduce confusion between instructions and editing types.
- Training: AnySD extends classifier-free guidance to three conditioning signals and trains in stages for general instruction understanding followed by task-specific tuning.The second stage fine-tunes MoE visual weights, routing, visual projection, and task embeddings.
5. Experiments
Experiments evaluate AnyEdit and AnySD on standard benchmarks and the more challenging AnyEdit-Test. The results report stronger semantic and visual editing performance, including on complex and visual-conditioned tasks.
- Evaluation Setup: Experiments cover standard benchmarks, AnyEdit-Test, qualitative evaluation, and architecture ablations to assess quality, generalization, and scalability.Standard evaluation uses EMU-Edit Test and MagicBrush with semantic and visual similarity metrics.
- Standard Benchmarks: 0.872 CLIPim and 0.285 CLIPout on EMU-Edit Test show that SD-1.5 trained only on AnyEdit outperforms SOTA methods in semantic performance.This comparison uses no additional mask supervision.
- Standard Benchmarks: 0.881 DINO on MagicBrush Test accompanies AnySD’s stronger semantic and visual similarity than SOTA methods.AnySD reports 0.872 CLIPim on EMU-Edit Test in the same comparison.
- AnyEdit-Test: AnyEdit outperforms SOTA methods across AnyEdit-Test editing categories, including complex tasks and visual editing conditions.Compared with Uni-ControlNet, it consistently performs better on visual editing tasks.
- Qualitative Evaluation: Qualitative results show more accurate target-region editing and better consistency in irrelevant areas without mask guidance across complex and visual instructions.The method also distinguishes foreground and background when modifying backgrounds.
- Ablation Study: Removing text-aware routing reduces CLIPim from 0.872 to 0.838 and increases L1 from 0.070 to 0.154 on EMU-Edit Test.
6. Conclusion
The paper introduces AnyEdit, a large multi-type, multi-scene editing dataset and benchmark, and AnySD, a unified model built to use it. Experiments support accurate complex editing while preserving unmodified image content.
- Contributions: AnyEdit contains 2.5M editing samples across 25 distinct types and includes the AnyEdit-Test benchmark.
- Contributions: The dataset is constructed with adaptive pipelines to produce high-quality data for diverse editing tasks in a low-resource manner.
- Conclusions: AnySD and AnyEdit perform strongly across standard and challenging benchmarks, executing complex instructions while preserving consistency in unmodified elements.
Supplementary Material
The supplementary material provides additional dataset, benchmark, implementation, experimental, and qualitative details.
- Supplementary Contents: The supplement details AnyEdit collection, statistics, examples, AnyEdit-Test, per-type results, implementation, and additional qualitative and human evaluations.
B. Detailed Dataset Collection Process
AnyEdit’s collection process combines task-specific instruction generation with adaptive, editing-type-specific pipelines to produce diverse editing pairs. The dataset spans 25 editing types and uses varied synthesis, transformation, and reference-based procedures.
- Task Organization: AnyEdit organizes its collection around five primary categories containing 25 distinct editing types.
- Instruction Generation: Task-specific prompts, handcrafted examples, and iterative example-pool expansion guide the LLM toward diverse, consistent editing instructions.Five in-context examples are randomly selected during subsequent generation rounds.
- Adaptive Editing Pipeline: Adaptive pipelines select procedures suited to each editing type rather than applying one shared process to all instructions.Figure 7 illustrates the specific pipelines used for the different task types.
- Object-Level Editing: Object removal uses GroundingDINO and Segment Anything masks, followed by SD-Inpaint with mask dilation and Gaussian filtering.
- Object-Level Editing: Attribute edits use attention differences to identify masks, while action changes use mutual self-attention and masked cross-attention for fine-grained modifications.
- Global and Reference Editing: Global and specialized edits include caption-controlled text changes, tone and style transfer, foreground-preserving background changes, movement and resizing, implicit instruction conversion, and image references.Style transfer retains intuitive style instructions, while image-reference pairs reuse edited objects and corresponding visual conditions.
E. AnyEdit-Test Benchmark
AnyEdit-Test is a manually curated benchmark designed to evaluate editing models across diverse, representative, and complex task scenarios. It selects 50 example pairs for each supported editing type.
- Benchmark Construction: AnyEdit-Test selects 50 example pairs from each editing type supported by AnyEdit.
- Benchmark Construction: The resulting benchmark covers diverse editing challenges and task complexities for a more rigorous assessment of model performance.Its design broadens evaluation beyond narrower existing task coverage.
F. Detailed Experiments of AnyEdit-Test
AnyEdit-Test evaluates whether editing models accurately perform diverse instructions while preserving image content. AnySD addresses varied task demands with task-aware visual conditioning and separate guidance over image, text, and visual prompts.
- Evaluation: Evaluations of 25 editing types focus on editing accuracy and content consistency.
- Evaluation: Existing models show reduced accuracy on complex tasks such as action change, rotation, and outpainting, while fine-grained edits can degrade content preservation.The reported examples include reduced CLIPim and nearly doubled L1 degradation for action change and textual change.
- AnySD Architecture: AnySD uses a Mixture of Experts architecture so different editing tasks can selectively focus on visual elements such as local objects or global style.A router distributes expert weights according to task embeddings.
- AnySD Architecture: Decoupled cross-attention injects visual features through an additional attention layer while preserving the text-editing condition.The visual branch uses separate key and value projections derived from visual conditioning.
- Diffusion Formulation: AnySD conditions latent diffusion on the input image, text instruction, and visual prompt with task embedding.The model predicts noise for a noisy latent produced from the encoded image.
- Classifier-Free Guidance: Separate classifier-free guidance scales control the strengths of image, text, and visual conditioning during inference.The scales are denoted s_I, s_T, and s_V.
- Task Embeddings: Each supported task uses a distinct learned task embedding whose size matches the dimensionality of CLIP.
G.5. Training Details
The training and evaluation setup combines staged AnySD training with established instruction-based, specific-editing, and visual-condition baselines. Performance is assessed using pixel, image-similarity, text-alignment, and instruction-following metrics.
- Training: Stage I trains AnySD for 110,000 steps at 256 × 256 resolution using four 48GB NVIDIA A6000 GPUs for 280 hours.The training uses batch size 1024 and learning rate 10^-4.
- Training: Stage II trains on the entire AnyEdit dataset at 512 × 512 resolution to adapt task-specific editing granularity.Task embeddings and expert components are used in this stage.
- Baselines: Baselines include instruction-based methods such as InstructPix2Pix, HIVE, UltraEdit, EMU-Edit, and MagicBrush, alongside specific and visual-condition methods.Baseline configurations follow official repository or Hugging Face defaults.
- Metrics: L1 measures pixel-level differences, while CLIP, DINO, CLIP-T, CLIPdir, and related similarities assess image fidelity, text alignment, and instruction following.
H.1. Human Evaluation
Human evaluations consistently favor AnySD over competing methods on instruction consistency and image quality. The results also reveal that automatic metrics may not fully reflect human preferences.
- Multi-Choice Comparison: Human evaluators selected AnySD as the best method for consistency and image quality in multi-choice comparisons.The evaluation sampled 100 AnyEdit-Test images and compared four methods.
- Evaluation Implications: Automatic metrics can diverge from human judgments because they emphasize image quality and may miss instruction consistency and human preferences.
- One-on-One Comparison: AnySD consistently outperformed alternatives in one-on-one comparisons, with most evaluators preferring its results.
- Individual Evaluation: AnySD achieved higher average scores than the baselines in individual 5-point Likert evaluations of consistency and overall quality.
H.2. Qualitative Evaluation on Different Benchmarks
Qualitative evaluations show that AnySD performs fine-grained edits across diverse benchmarks and editing scenarios. It can preserve unrelated content, edit specific regions without masks, and handle varied instructions and image inputs.
- EMU-Edit Test: AnySD separates foreground and background to modify backgrounds while preserving foreground content in EMU-Edit examples.
- MagicBrush Benchmark: On MagicBrush, AnySD performs specific local edits without mask inputs, unlike DALLE-2’s mask-supervised editing process.
- AnyEdit-Test: AnySD produces high-quality, fine-grained edits across varied images and instructions, including reflection and appearance modifications involving world knowledge.
- Editing Diversity: The qualitative examples span local, global, camera movement, visual, and implicit editing categories.
- Additional Scenarios: Additional examples cover multi-turn editing, real-image inputs, and comparisons with other instruction-based image-editing methods.