PaperlayerView exact revision on arXiv

Source-linked AI summary

SR-Edit: Region-Aware Image Editing via Self-Refinement

Andong Wang, Zehua Chen, Yuxuan Jiang, Jun Zhu

arXiv:2609.02504v1cs.CV

TL;DR

Faithful image editing remains challenging because automatically inferred regions can be inaccurate or unstable and heuristic preservation corrections can introduce artifacts. SR-Edit alternates self-consistent pixel-space region estimation with sampling-aligned Doob’s h-transform corrections, achieving better preservation and visual quality than representative baselines. Its scope is localized editing, where preservation can still interfere with intended edit semantics.

  • Problem

    Faithful image editing remains challenging because automatically inferred regions can be inaccurate or unstable and heuristic preservation corrections can introduce artifacts.

  • Method

    SR-Edit alternates self-consistent pixel-space region estimation with Doob’s h-transform masked residual corrections aligned with diffusion and flow-matching sampling.

  • Results

    SR-Edit achieves better non-edit preservation and visual quality than representative baselines across multiple editors and metrics while maintaining instruction consistency.

  • Takeaways & Limitations

    The framework provides a training-free, mask-free approach for iterative balancing of preservation and edit expressiveness.

  • Takeaways & Limitations

    SR-Edit focuses on localized editing, and overestimated non-edit regions or soft preservation can interfere with intended edit semantics.

Abstract

from arXiv · show

With the recent rapid progress in generative models, image editing has made remarkable advances, yet achieving faithful edits that precisely modify only the target regions while strictly preserving all other regions remains challenging. Since externally provided region annotations are often difficult to obtain in practice, a growing body of work seeks to improve preservation by automatically inferring edit and non-edit regions, and then enforcing consistency on the latter. However, these approaches still suffer from inaccurate region estimation and heuristic correction strategies that distort the native inference process, making methods designed for fidelity themselves a new source of artifacts. We propose SR-Edit, an image editing framework that overcomes these issues via iterative self-refinement. Specifically, at each iteration, SR-Edit first (i) extracts progressively precise and self-consistent region separation from the model's own predictions by lightweight post-processing, and then (ii) enforces preservation in non-edit areas through correction updates that remain aligned with the original sampling dynamics. Extensive experiments demonstrate that SR-Edit achieves superior preservation and overall image quality compared to existing editing techniques.

1 Introduction

SR-Edit addresses faithful image editing by automatically separating edit and non-edit regions and preserving the latter without disrupting native sampling dynamics. Its iterative self-refinement improves region localization and preservation, yielding better visual quality than prior editing methods.

  • Motivation: Faithful editing remains difficult because global sampling dynamics can alter backgrounds, textures, identity cues, and geometric details outside the intended edit.These unintended changes arise even when the semantic edit succeeds.
  • Motivation: External masks can provide spatial control, but manual annotation is costly and often unavailable, motivating automatic mask-free region inference.Prior approaches use attention signals or source-versus-target generation differences to estimate editable regions.
  • Limitations of prior methods: Automatic region estimates may be imprecise or unstable, while heuristic feature interventions can distort native sampling dynamics and introduce artifacts.These limitations can prevent preservation methods from reliably improving fidelity.
  • SR-Edit: SR-Edit uses the model’s own predictions and lightweight pixel-space post-processing to construct sharper, more stable edit and non-edit separation.Pixel-space differences are used to refine self-consistent difference maps rather than relying on latent or attention discrimination.
  • SR-Edit: SR-Edit enforces non-edit preservation with a Doob’s h-transform-based masked residual correction that remains aligned with diffusion and flow-matching sampling dynamics.The correction is derived from a distribution transform rather than heuristic state intervention.
  • Results: Across iterative self-correction, region separation and preservation guidance are repeatedly updated, producing progressively refined results with superior preservation and overall visual quality.The paper reports improvements over prior region-aware and mask-free editing techniques.

2 Related Work

Prior image-editing systems use diverse generative backbones and preservation mechanisms, but preserving irrelevant content and automatically identifying edit regions remain challenging. SR-Edit builds on inference behavior and pixel-space differences to obtain more localized, stable region estimates.

  • Image editing with generative models: Generative image editors include forward-noising and conditional-denoising methods, training-based instruction editors, and inversion-based trajectory-modification methods.These paradigms differ in how they incorporate the input image and enforce edit intent.
  • Source preservation: Noisy initialization can induce global drift, so prior work improves preservation through inversion, reconstruction, masks, feature manipulation, or structural constraints.These mechanisms aim to reduce deviation or spatially align edited outputs with source images.
  • External masks: User-provided masks offer direct spatial control and strong results when accurate, but manual annotation limits real-world applicability.Mask quality also constrains the resulting separation between edit and non-edit regions.
  • Region identification: SR-Edit’s region pipeline constructs a difference map, applies Otsu thresholding, performs morphological refinement, and filters connected components.The incremental contribution of these post-processing components is visualized in Fig. 4.
  • Text-driven localization: Text-driven methods infer regions from source-target inference differences or cross-attention maps, reducing reliance on external masks.Their localization signals are derived from semantic or text-conditioned representations.
  • Inference behavior and region identification: Inference-behavior methods exploit trajectory information for region control, but existing approaches can rely on single-step reconstruction and intermediate-feature interpolation.SR-Edit instead uses output-driven pixel-space cues and post-processing to refine region estimates.

3 SR-Edit

SR-Edit iteratively infers edit and non-edit regions from model predictions, refines them in pixel space, and applies preservation guidance aligned with the sampling dynamics. Its pipeline combines lightweight mask processing with Doob’s h-transform corrections to improve region fidelity.

  • Iterative self-refinement: SR-Edit repeatedly probes provisional edits, infers regions, applies preservation guidance, and refines the result across stacked blocks.Each block uses few-step inference followed by region estimation and guided refinement.
  • Precise region identification: The method uses decoded pixel space because latent features are patch-level and their nonlinear decoding makes latent variation an unstable proxy for pixel changes.Pixel-space processing also enables mature image post-processing operations.
  • Precise region identification: Pixel-space region identification computes per-pixel input–reference differences, thresholds them with Otsu’s method, and produces a binary change mask.The mask marks changed pixels as edited and unchanged pixels as non-edited.
  • Precise region identification: Morphological opening, closing, and connected-component filtering remove speckles, fill holes, connect gaps, and suppress small residual components.The final filtering step removes components smaller than Amin.
  • Region preservation: Doob’s h-transform models preservation as a vanishing-noise observation on non-edit pixels and adds a guidance correction to the sampling dynamics.Under a plug-in approximation, the correction becomes a masked residual on the model’s clean-image prediction.
  • Region preservation: The resulting additive guidance can augment reverse-time diffusion scores and is implemented as a time-dependent masked-residual correction.The same preservation idea is applied to diffusion and flow-based samplers.

4 Experiments

SR-Edit is evaluated on localized editing benchmarks using multiple diffusion- and flow-based backbones, with metrics covering semantic alignment, source preservation, and structural fidelity. Results report consistent source-preservation gains while largely maintaining semantic alignment, alongside stronger qualitative fidelity and realism.

  • Datasets: ImgEdit-Bench evaluation covers 497 localized single-turn editing cases spanning replace, adjust, background, remove, add, and action tasks.The benchmark is derived from ImgEdit and focuses on edits confined to specific image areas.
  • Datasets: PIE-Bench evaluation uses 578 cases with human-annotated edit masks and captions for separate regional preservation and semantic-consistency assessment.Style edits and viewpoint transformations are excluded because they cannot be meaningfully localized.
  • Experimental setup: The experiments span four open-source backbones from diffusion- and flow-based editor families, including InstructPix2Pix, AnyEdit, Qwen-Image-Edit 2511, and Step1X-Edit v1p2.The study compares SR-Edit with Follow-Your-Shape and SpotEdit as representative training-free region-aware methods.
  • Metrics: Evaluation measures instruction alignment with CLIP and ImgEdit scores, while PSNR, SSIM, DISTS, LPIPS, and PIE-Bench metrics assess preservation and structural fidelity.PIE-Bench additionally reports region-specific metrics over human-annotated non-edit areas.
  • Inference configuration: With two iterations and stride-3 few-step inference, SR-Edit uses specified morphology, observation-noise, guidance-schedule, and self-refinement settings.Self-refinement begins at 30% of the model’s inference steps.

5 Results

SR-Edit improves source and non-edit-region preservation across backbones and benchmarks while maintaining semantic alignment and edit quality. Quantitative and qualitative results support higher fidelity and more realistic edits than competing methods.

  • SR-Edit consistently improves source preservation across backbone models while largely maintaining semantic alignment.
  • SSIM increases from 0.70 to 0.85 on AnyEdit and from 0.68 to 0.77 on Step1X-Edit v1p2.LPIPS decreases from 0.41 to 0.35 and from 0.39 to 0.27, respectively.
  • On Qwen-Image-Edit 2511, SR-Edit achieves the best PSNR, DISTS, and LPIPS while improving CLIP from 26.16 to 26.80.The reported results indicate stronger preservation does not generally weaken the intended edit.
  • Non-edit SSIM improves from 0.76 to 0.86 on InstructPix2Pix and from 0.87 to 0.94 on Qwen-Image-Edit.PIE-Bench also reports reductions in LPIPS, MSE, and structural distance, while semantic scores are improved or largely preserved.
  • SR-Edit achieves higher input fidelity and more realistic, higher-quality edits than competing methods.

6 Ablation Study

Ablations show that few-step inference, iterative refinement, complementary post-processing, and increasing guidance schedules each support region identification or preservation. SR-Edit can still produce blending artifacts when preservation overestimates the non-edit region.

  • Region estimation: Few-step inference consistently outperforms reconstruction in region-identification IoU and F1, with smoother curves across sampling steps.Among few-step settings, stride-2 yields the best performance.
  • Iterative refinement: Two SR-Edit blocks improve region alignment because masks and preservation guidance are re-estimated from progressively refined predictions.A single fixed early mask is typically over-inclusive and inaccurate, weakening preservation and potentially instruction adherence.
  • Region post-processing: The four region post-processing components are complementary: removing Otsu, opening, closing, or connected-component filtering harms mask quality in different ways.The reported effects include lower precision, reduced overlap, weaker spatial coherence, and noisy fragments.
  • Failure cases: SR-Edit sometimes introduces unintended blending between preserved content and the target edit.Examples include a dark upper-right artifact and a hybrid cabin–tent structure caused by incomplete replacement.
  • Guidance schedule: A linearly increasing guidance schedule consistently improves preservation over Constant and Linear Down with minor semantic and overall-quality variation.The schedule matches the reported decrease in uncertainty during sampling and strengthens correction later in inference.

7 Computational Efficiency Analysis

SR-Edit adds negligible memory overhead and a moderate inference-time increase. The authors characterize this cost as generally acceptable given the improvement in source fidelity and note a downstream distillation use.

  • SR-Edit can construct large-scale, high-fidelity editing pairs for distilling source preservation into the base model.

8 Scope and Limitations

SR-Edit currently targets localized editing, and its main observed failures occur when preservation interferes with intended edit semantics. Overestimated non-edit regions and soft correction can produce blending or residual semantic failures.

  • Scope: SR-Edit currently focuses on localized editing as its primary target setting.
  • Failure modes: Overestimating the non-edit region can cause abnormal mixing between preservation guidance and edit modification.This failure is associated with unintended blending in the edited result.
  • Failure modes: Soft preservation correction may not fully reset drifted non-edit latents, allowing residual interference to propagate into the edited region.The authors state this can produce semantic failures and that preservation may remain weaker than hard replacement.

9 Conclusion

SR-Edit combines pixel-space region identification with iterative self-refinement to support mask-free, faithful image editing. Its pipeline uses lightweight post-processing and inference procedures summarized for diffusion and flow backbones.

  • Conclusion: SR-Edit infers and sharpens edit regions from pixel-space predictions without external masks.The framework is described as region-aware and mask-free, with iterative self-refinement for region estimation.
  • Conclusion: The region-identification pipeline constructs a difference map, thresholds it, applies morphological refinement, and filters connected components.Algorithm 1 includes difference-map construction, thresholding, opening and closing operations, and minimum-area filtering.
  • Conclusion: The unified inference procedure covers both diffusion and flow-based backbones.Algorithm 2 is presented as the inference component paired with the pixel-space region-identification pipeline.

2 Additional Details

The additional-details section introduces the unified SR-Edit inference algorithm and its required inputs and initialization steps. It also describes the evaluation subset used for quantitative analysis.

  • Additional Details: The quantitative analysis uses 198 manually selected MagicBrush test samples whose masks accurately match the edited region.Cases without a realizable target for mask-free editing, such as task add, are excluded.
  • Additional Details: Algorithm 2 defines unified SR-Edit inference for diffusion and flow backbones.The algorithm requires an input image, editing condition, terminal time, self-correction start time, correction duration, refinement-block count, and probe stride.
  • Additional Details: Inference encodes the input image into a source latent, initializes the terminal latent, and runs stabilization inference before refinement.These operations precede the iterative refinement blocks.

I. Few-step probe for region estimation

SR-Edit probes intermediate predictions to estimate edit regions, then applies masked residual correction during iterative inference. The analysis reports stable mask behavior, resolution robustness, and a preservation–instruction-adherence trade-off.

  • I. Few-step probe for region estimation: Each refinement block decodes a few-step prediction, identifies a pixel-space region, and downsamples that mask to latent space.The probe uses stride K and produces a provisional reference image for region identification.
  • I. Few-step probe for region estimation: Self-correction defines the latent non-edit mask as 1 − Mlatent and uses it to guide subsequent sampling updates.The residual guidance is incorporated into score estimation or the vector field over the correction interval.
  • I. Few-step probe for region estimation: The quantitative analysis reports 198 samples after excluding cases whose masks do not define realizable targets for mask-free editing.The selected masks accurately match the edited region.
  • I. Few-step probe for region estimation: Across backbones and metrics, source preservation and instruction adherence show a clear negative relation, while nearby mask configurations vary smoothly.The passage characterizes this smooth variation as indicating strong SR-Edit stability.
  • I. Few-step probe for region estimation: SR-Edit robustly improves fidelity across resolutions, and fixed mask parameters remain effective in the reported robustness analysis.The analysis compares scaled and unscaled mask parameters and several plug-in estimates on Qwen-Image-Edit 2511.
Loading 2609.02504v1…