Source-linked AI summary
Diffusion Model-Based Image Editing: A Survey
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, Shifeng Chen
TL;DR
Diffusion-model image editing lacks a focused survey that systematically covers its methods, conditions, and tasks. This paper reviews and categorizes over 100 methods, introduces EditEval with LMM Score, and identifies performance variability across seven editing tasks while discussing current limitations and future directions.
Problem
Existing diffusion-model surveys focus mainly on generation, restoration, enhancement, or video, while image-editing coverage is often cursory.
Method
The survey categorizes over 100 methods by learning strategy, input conditions, and editing-task type, and develops the EditEval benchmark with LMM Score.
Results
Across seven editing tasks, no single method outperforms all others, and most methods show substantial score variability.
Takeaways & Limitations
The survey provides a structured resource for comparing diffusion-based image-editing methods and identifying future research directions.
Takeaways & Limitations
Most diffusion-based models require many inference steps, making them time-consuming and computationally costly for deployment and user experience.
Abstract
from arXiv · showhide
Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provide an exhaustive overview of existing methods using diffusion models for image editing, covering both theoretical and practical aspects in the field. We delve into a thorough analysis and categorization of these works from multiple perspectives, including learning strategies, user-input conditions, and the array of specific editing tasks that can be accomplished. In addition, we pay special attention to image inpainting and outpainting, and explore both earlier traditional context-driven and current multimodal conditional methods, offering a comprehensive analysis of their methodologies. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval, featuring an innovative metric, LMM Score. Finally, we address current limitations and envision some potential directions for future research. The accompanying repository is released at https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods.
1 INTRODUCTION
Diffusion model-based image editing has become an increasingly important research area, but existing surveys do not provide a focused, detailed synthesis. This survey addresses that gap by systematically organizing methods, input conditions, and editing tasks.
- Diffusion models generate desired data by learning to reverse gradual noise addition, motivating their growing use in image editing.The reverse process starts from random noise and iteratively produces data matching the source distribution.
- Existing diffusion-model surveys mainly cover image generation, restoration, enhancement, or video generation rather than image editing.Surveys mentioning image editing often provide only cursory overviews.
- The survey reviews over 100 papers across training-based, testing-time finetuning, and training-and-finetuning-free approaches.These classes are organized according to whether methods require training, inference-time finetuning, or neither.
- It identifies 10 input conditions, including text, masks, reference images, layouts, poses, sketches, segmentation maps, audio, and dragging points.The conditions demonstrate the range of user inputs used for image editing.
- The survey groups editing tasks into semantic, stylistic, and structural categories covering 12 specific types.Figure 1 summarizes publication distributions across learning strategies, input conditions, and editing tasks.
- The paper aims to categorize and critically assess the field while synthesizing findings and guiding future research.It also discusses current challenges and potential future trends.
2 BACKGROUND
Diffusion models transform data through progressive noise addition and learn an iterative reverse process for generation and editing. Stable Diffusion further uses text-image cross-attention and image self-attention to control prompts while preserving spatial structure.
- Diffusion Models: Diffusion models gradually add noise to data and iteratively reverse that process to generate samples matching the original distribution.The reverse process starts from Gaussian noise and proceeds through time steps until reaching the generated data sample.
- Diffusion Models: The forward process introduces noise with stepwise variances β1:T, while z_t can also be sampled directly without intermediate steps.The maximal time step is T, and the initial data distribution is represented by z_0.
- Diffusion Models: Reverse diffusion learns pθ(z_t−1 | z_t) with a UNet, and conditional formulations can use the source image as an external variable.Training minimizes a KL divergence between the forward and reverse joint distributions, with DDPMs using a reweighted denoising loss in practice.
- DDIM Sampling and Inversion: DDIM inversion approximately maps a real image z_0 to a noisy latent z_T before editing and sampling produce an edited image.The approximation replaces the unknown z_t on the inversion equation’s right-hand side with z_t−1 under the assumption z_t−1 ≈ z_t.
- Text Condition and Classifier-Free Guidance: Text-conditional diffusion uses prompt embeddings during iterative denoising, while classifier-free guidance combines conditional and unconditional noise predictions.The guidance scale w controls textual influence, with higher values producing stronger text-driven control.
- Attention in Stable Diffusion: Stable Diffusion cross-attention aligns text tokens with image regions, whereas self-attention relates spatial features and can preserve original spatial structure during editing.Cross-attention derives queries from noisy-image features and keys and values from text embeddings; self-attention uses image-derived queries and keys.
- Related Tasks: Conditional image generation, restoration, and composition are related tasks but differ from directly modifying existing image content and are outside the survey’s primary focus.The survey distinguishes these objectives from image editing rather than treating them as the same task.
3 SCOPE AND CATEGORIZATION
The survey defines its scope around diffusion-based modification of existing images and organizes editing methods by task type. Its categorization distinguishes semantic, stylistic, and structural changes, while Table 1 provides a multi-perspective overview.
- Scope: Included works must modify the appearance, structure, or content of existing images without becoming significant new-image generation, and must depend on diffusion models.These are the survey’s two primary inclusion criteria.
- Semantic Editing: Semantic editing changes an image’s content or narrative through tasks such as object addition, removal, replacement, background change, and emotional-expression modification.This category concerns the depicted scene’s story, context, or thematic elements.
- Stylistic Editing: Stylistic editing changes visual style or aesthetics without altering narrative content, including color, texture, and overall style changes.The category covers both artistic and realistic styles.
- Structural Editing: Structural editing changes spatial arrangement, viewpoint, or object characteristics through movement, size or shape, action or pose, and perspective changes.This category emphasizes the organization and presentation of objects within a scene.
- Overview: Table 1 summarizes the surveyed papers’ categorization across multiple perspectives to provide a quick search resource.
4 TRAINING-BASED APPROACHES
Training-based approaches require substantial large-dataset training and are organized by application scope, training conditions, supervision, and editing methodology. The survey covers domain-specific editing, reference and attribute control, and instruction-based multimodal editing.
- Training-based methods require substantial training on large datasets before deployment for specific editing tasks.
- Domain-Specific Editing: Because large-scale training is computationally expensive, earlier methods often used specialized datasets for faces, animals, objects, and artistic styles.
- Self-Supervised Editing: Self-supervised methods extract image attributes or information as training conditions for reference-based composition and attribute-controlled editing.
- Instructional Editing: Instruction-based methods use humanized editing commands rather than descriptions of the desired final image, with InstructPix2Pix establishing a foundational framework.
- Instructional Editing: Multimodal instruction methods extend editing conditions beyond text by integrating audio, images, and text through multimodal encoders or large language models.
5 TESTING-TIME FINETUNING APPROACHES
Testing-time finetuning methods optimize models or conditioning components for each input image rather than relying on a traditional pre-training phase. The survey covers model, embedding, latent, and hybrid finetuning strategies.
- Testing-time finetuning adapts the model during inference for each image, enabling more precise and personalized edits.
- Denoising Model Finetuning: Methods may finetune the entire denoising model or selected layers to learn image-specific features and improve editing accuracy.
- Embedding Finetuning: Null-text embedding finetuning addresses DDIM inversion reconstruction failures while preserving the user’s target prompt set.
- Latent Finetuning: Latent optimization directly adjusts noisy latents, while human-guided variants let users move specified image points during editing.
- Hybrid Finetuning: Hybrid methods combine multiple finetuning components sequentially or simultaneously for targeted editing, including joint text-embedding and model optimization.
6 TRAINING AND FINETUNING FREE APPROACHES
Training- and finetuning-free methods aim to edit images quickly and cheaply by modifying inputs, inversion or sampling procedures, attention, masks, or noise. Their techniques exploit diffusion-model mechanisms without dataset training or source-image finetuning.
- Training- and finetuning-free methods require neither dataset training nor source-image finetuning during editing, prioritizing speed and low cost.
- Input Text Refinement: Input-text refinement uses captioning and prompt injection to expand minimal user inputs into detailed prompts for more accurate contextual editing.
- Inversion/Sampling Modification: Inversion and sampling modifications improve reconstruction consistency by preserving inversion information or using null-text guidance during sampling.
- Attention Modification: Attention modification edits spatially relevant regions by manipulating cross-attention or self-attention information while maintaining consistency with the source image.
- Mask and Noise Guidance: Masks steer sampling toward localized regions, while multi-noise redirection combines multiple editing directions into a single noise prediction.
- Object-Aware Editing: OIR-Diffusion uses object-aware inversion and reassembly to edit specific objects while preserving nonedited regions, especially in intricate multi-object scenes.
7 INPAINTING AND OUTPAINTING
Image inpainting and outpainting are distinct image-editing subtasks with different objectives and challenges. The survey divides them into two major types for separate discussion.
- The survey treats inpainting and outpainting as two major types of image editing with distinct objectives and challenges.
7.1 Traditional Context-Driven Inpainting
Traditional context-driven inpainting methods use diffusion processes to reconstruct masked content from uncorrupted image information, while posterior-guidance methods enforce consistency with measurements or observations.
- Palette trains a conditional diffusion framework for multiple image-to-image translation tasks, including inpainting and outpainting.Its conditioning concatenates the low-quality reference with the denoised result from the previous step for noise prediction.
- Context Prior Integration: Context-prior methods exploit unmasked structures and textures to maintain global consistency during masked-region generation.Repaint samples masked regions from a pretrained unconditional diffusion model and restores unmasked regions from the input at each reverse step.
- Posterior-guided methods estimate measurement-conditioned diffusion distributions and can add projection steps to enforce data consistency.DPS approximates the posterior using the predicted clean image, while MCG projects onto the measurement subspace.
7.2 Multimodal Conditional Inpainting
Multimodal conditional inpainting adds text, segmentation, or reference-image conditions to improve control beyond surrounding visual context. Methods differ in how they provide masks, personalize content, blend inputs, and handle high resolution.
- Text-to-image diffusion models adapt inpainting by injecting multimodal conditions such as text descriptions, segmentation maps, and reference images.Stable Diffusion and Imagen replace the noisy background with a noisy version of the original image during reverse diffusion.
- SmartBrush improves mask control by letting users choose coarse masks containing an object or detailed masks outlining its shape.The method addresses random masking that may cover regions unrelated to the text prompt, especially when masked areas are small.
- Reference-based methods support personalized inpainting by extracting subject information or jointly using text and reference images.PbE uses CLIP for subject extraction, while PhD integrates text and reference images in training.
- Training-free or pretrained-model methods combine diffusion sampling with blending, segmentation, inversion, attention guidance, or classifier optimization.Inpaint Anything supports removal, replacement, and filling through SAM and inpainting models, while MagicRemover targets precision and stability with fewer steps.
- HD-Painter targets high-resolution inpainting through image completion followed by inpainting-specialized super-resolution.Its completion stage modifies attention in Stable Diffusion’s latent space to improve prompt alignment.
7.3 Outpainting
Outpainting generates new pixels beyond an image’s boundaries and is often treated as a specialized form of inpainting. Existing text-to-image diffusion models can be adapted because they were trained on varied image sizes and shapes.
- Outpainting extends image boundaries with newly generated pixels that are intended to blend seamlessly with the original image.Its objective differs from inpainting, which fills missing regions within an existing image.
- Stable Diffusion and DALL-E can be generalized to outpainting, while Palette trains diffusion models by combining cropped images with generated content.
8 BENCHMARK AND EVALUATION
The survey introduces EditEval to address limited coverage in existing image-editing benchmarks and evaluates methods across seven tasks using LMM Score and user judgments. Results show substantial task and sample variability, while LMM Score most consistently aligns with user evaluations.
- 8.1 Benchmark Construction: EditEval evaluates general diffusion-based image editing on 150 curated images and 7 common editing tasks.The dataset contains diverse professional photographs with source and target prompts and corresponding editing instructions.
- 8.1.3 Metric Design and Selection: LMM Score uses large multimodal models to assess editing performance across different tasks alongside a user study for subjective evaluation.The framework evaluates editing accuracy, contextual preservation, visual quality, and logical realism.
- 8.2.2 Comparative Analysis: Most methods handle semantic and stylistic editing better than structural editing, consistent with difficulties in accurate spatial awareness.
- 8.1.3 Metric Design and Selection: LMM Score combines four sub-scores, each ranging from 1 to 10, using a weighted average to represent overall editing quality.The weighting formula is described as balancing the relative importance of the evaluation factors.
- 8.2.2 Comparative Analysis: No method dominates every task, and broad score ranges and large standard deviations indicate sample-dependent performance.Some methods remain stable and impressive on particular tasks.
- 8.2.1 Method Selection: The benchmark selects 4 to 8 open-source, text-conditioned methods per task and excludes domain-specific methods to support broader comparison.
- 8.2.2 Comparative Analysis: LMM Score consistently achieves the highest correlation with user evaluations across tasks, whereas CLIPScore and Directional CLIP Similarity show limited correlation.TIFA Score performs better for semantic editing but struggles with stylistic editing.
9 CHALLENGES AND FUTURE DIRECTIONS
Future work must address efficiency, faithful complex-structure and lighting edits, and high-resolution editing precision. These challenges reflect computational costs, artifacts, and limited concrete evidence for high-resolution performance.
- Fewer-Step Model Inference: Diffusion-based editing often requires many inference steps, increasing runtime and computational cost for deployment and user experience.Few-step and one-step models, including distillation and consistency-model approaches, are proposed to improve efficiency.
- Efficient Models: Training realistic diffusion models is computationally intensive and requires large amounts of high-quality data.Efficient architectures, partial parameter training, and added layers over pretrained models are proposed to reduce training costs.
- Complex Object Structure Editing: Complex structures such as fingers, logos, and scene text remain prone to noticeable artifacts during editing.Negative prompting is not robust enough, while layouts, edges, and dense labels provide structural guidance.
- Complex Lighting and Shadow Editing: Realistic and consistent lighting or illumination editing remains challenging.The survey identifies complex lighting and shadow editing as an unresolved task.
- High-Resolution Image Generation and Editing: Specialized high-resolution image-editing methods are scarce, and few claimed capabilities have demonstrated concrete results.At resolutions of 1024x1024 pixels or higher, precise pixel-level modifications, edge refinement, and fine-detail preservation become increasingly demanding.
10 CONCLUSION
The survey organizes diffusion-based image-editing research by learning strategy and editing task, then evaluates seven tasks through EditEval and LMM Score. It concludes by identifying broad potential and future research directions.
- Learning Strategies: The survey categorizes over 100 methods into training-based, test-time fine-tuning, and training- and fine-tuning-free approaches.The classification is based on whether methods require training, inference-time fine-tuning, or neither.
- Editing Tasks: Image-editing tasks are grouped into semantic, stylistic, and structural categories covering 12 specific types.These categories organize the range of editing tasks examined in the survey.
- Benchmark: EditEval evaluates seven tasks alongside recent state-of-the-art methods and introduces LMM Score for comparative analysis.The benchmark focuses on evaluating diffusion-based image-editing methods.
- Future Research: The survey highlights broad potential in image editing and suggests directions for future research.Its conclusion follows the review and benchmark analysis of diffusion model-based editing methods.