PaperlayerView exact revision on arXiv

Source-linked AI summary

CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing

Dinh-Khoi Vo, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

arXiv:2506.18438v3cs.CV

TL;DR

Real-image editing with diffusion models remains difficult when edits must target specific objects while preserving background details. CPAM combines self- and cross-attention manipulation with mask guidance for zero-shot editing. The paper reports top user ratings for object retention, background retention, and realism, while noting that results remain bounded by the pretrained model’s capacity.

  • Problem

    Existing methods can change the overall image rather than edit individual objects, limiting precise real-image manipulation.

    Source: §I · p. 2

  • Method

    CPAM uses self-attention preservation adaptation, cross-attention localized extraction, and mask-guidance strategies for zero-shot editing.

    Source: §I · p. 2, D. Mask-Guidance Strategy · p. 6

  • Results

    CPAM received the highest user ratings for object retention, background retention, and realism, and significantly outperformed other methods across most metrics.

    Source: 6) Quantitative Results · p. 11

  • Takeaways & Limitations

    CPAM consistently outperforms leading editing techniques on complicated, non-rigid real-image edits, according to the reported empirical results.

    Source: §VI · p. 12

    Because CPAM manipulates attention in a pretrained model without additional training, its performance is bounded by that model’s capacity, and large pose or viewpoint changes may not align with the prompt or desired structure.

    Source: H. Limitations and Discussion · p. 11

Abstract

from arXiv · show

Editing natural images using textual descriptions in text-to-image diffusion models remains a significant challenge, particularly in achieving consistent generation and handling complex, non-rigid objects. Existing methods often struggle to preserve textures and identity, require extensive fine-tuning, and exhibit limitations in editing specific spatial regions or objects while retaining background details. This paper proposes Context-Preserving Adaptive Manipulation (CPAM), a novel zero-shot framework for complicated, non-rigid real image editing. Specifically, we propose a preservation adaptation module that adjusts self-attention mechanisms to preserve and independently control the object and background effectively. This ensures that the objects' shapes, textures, and identities are maintained while keeping the background undistorted during the editing process using the mask guidance technique. Additionally, we develop a localized extraction module to mitigate the interference with the non-desired modified regions during conditioning in cross-attention mechanisms. We also introduce various mask-guidance strategies to facilitate diverse image manipulation tasks in a simple manner. CPAM can be seamlessly integrated with multiple diffusion backbones, including SD1.5, SD2.1, and SDXL, demonstrating strong generalization across different model architectures. Extensive experiments on our newly constructed Image Manipulation BenchmArk (IMBA), a robust benchmark dataset specifically designed for real image editing, demonstrate that our proposed method is the preferred choice among human raters, outperforming existing state-of-the-art editing techniques. The source code and data will be publicly released at the project page: https://vdkhoi20.github.io/CPAM

I. INTRODUCTION

Real-image editing must preserve object identity and background while enabling specific, complex edits, but existing approaches can alter whole images or struggle with non-rigid changes. CPAM uses attention manipulation and mask guidance for tuning-free editing, and the paper reports that it outperforms leading methods in experiments and user studies.

I. INTRODUCTION

  • Existing methods can modify the overall image rather than a specific object, while local latent blending may produce rigid edits.Real-image editing also risks losing original textures and identity during inversion and editing.Source: §I · pp. 1–2
  • On IMBA, experiments and a user study report that CPAM significantly outperforms state-of-the-art editing methods.The paper introduces IMBA with enriched annotations for real-image editing evaluation.Source: §I · pp. 2–3
  • CPAM is a tuning-free method for zero-shot real-image editing that manipulates both self-attention and cross-attention.It is demonstrated across SD1.5, SD2.1, and SDXL without architectural modifications.Source: §I · p. 2
  • Preservation adaptation adjusts self-attention to retain object attributes and background while independently controlling them.The preserved attributes include pose, view, texture, identity, structure, and color.Source: §I · pp. 2–3
  • Localized extraction limits target-prompt influence on non-desired regions in cross-attention, enabling edits to specific objects.This addresses the tendency of cross-attention to condition all image pixels on the prompt.Source: §I · pp. 2–3
  • Mask-guidance strategies support flexible manipulation and object-shape tracking through mask adjustments.The source and target masks can be computed differently for different editing needs.Source: §I · pp. 2–3

II. RELATED WORK

Prior image-editing methods include optimization- and training-based approaches as well as zero-shot methods that edit during diffusion. The paper frames attention manipulation as a route to control content and context, while noting that existing approaches leave specific editing needs unresolved.

A. Image Manipulation Methods

  • Image-manipulation methods include per-image embedding or model optimization, fine-tuning, and training on source-reference pairs, unlike CPAM’s tuning-free approach.These approaches can require time-consuming optimization or extensive training.Source: A. Image Manipulation Methods · p. 3

B. Zero-Shot Methods

  • Zero-shot methods edit during denoising, but latent blending can yield rigid edits and existing methods may fail at controllable object removal.CPAM instead separates object and background through mask-guided attention adaptation for removal without architectural modifications.Source: B. Zero-Shot Methods · p. 3

III. PRELIMINARY ANALYSIS OF ATTENTION MECHANISM IN STABLE DIFFUSION

  • Stable Diffusion’s denoising U-Net uses self-attention and cross-attention, with queries derived from spatial features and keys and values derived from spatial or textual features.The source distinguishes self-attention’s spatial projections from cross-attention’s textual projections.Source: §III · p. 3

A. Insights from cross-attention

Cross-attention links image pixels with prompt tokens, while the paper’s observations motivate assigning prompts to selected regions and using null text for background pixels. CPAM combines this localized conditioning with self-attention control in a mask-guided editing pipeline.

A. Insights from cross-attention

  • Cross-attention connects pixels to prompt tokens, and CPAM directs object pixels to the target prompt while background pixels attend to null text.The paper notes that latent regions attending to null text can be perfectly reconstructed.Source: A. Insights from cross-attention · p. 3

B. Insights from self-attention

A. Overview

  • CPAM takes a source mask and target prompt, then uses preservation adaptation and localized extraction to retain content outside the edit while restricting prompt effects.In the Messi example, localized extraction preserves the background while changing the soccer ball into a rugby ball.Source: A. Overview · pp. 4–5, Figure 4 · p. 5

B. Preservation Adaptation

Preservation adaptation uses self-attention to derive background and foreground semantics from intermediate latent noise, then controls their adaptation separately. Background content is retained, while object content can adapt to a new shape, pose, or view.

B. Preservation Adaptation

  • The section frames self-attention adaptation as independently adapting semantic content from intermediate latent noise to the edited image while preserving the original appearance.Source: B. Preservation Adaptation · p. 5

1) Background preservation adaptation:

  • Background preservation retains query features and uses key and value features from intermediate latent noise to extract background semantics under the inverse source mask.The attention mechanism is guided by 1 − Ms.Source: 1) Background preservation adaptation · p. 5

2) Object preservation adaptation:

  • Object preservation adapts the original object's semantic features to new shapes, poses, or views after step T = 3 and layer L = 8.Otherwise, attention uses the current noise's key and value features.Source: 2) Object preservation adaptation · p. 5

3) Location adaptation:

Location adaptation combines separately derived foreground and background semantics under the target mask, while occasional normal self-attention helps maintain coherence. Localized extraction confines prompt conditioning to the selected object, and mask settings support several editing tasks.

3) Location adaptation:

  • The target mask combines foreground and background semantic content for region-specific edits, leaving other areas intact.Foreground and background semantics are derived separately; normal self-attention is randomly applied in 10% of layers to support coherence and more natural synthesis.Source: 3) Location adaptation · p. 5

C. Localized Extraction

  • Localized extraction directs selected object pixels to the target prompt and the remaining pixels to the null text prompt.It addresses distortions in non-desired regions caused when all image pixels attend to the target prompt.Source: C. Localized Extraction · p. 6

D. Mask-Guidance Strategy

  • For object replacement or pose and view changes, the target mask comes from aggregated cross-attention maps or an expanded source-mask hull.Source: D. Mask-Guidance Strategy · p. 6
  • Background alteration reuses the source mask, object removal sets the target mask to zero, and spatial-region edits use or slightly expand the source mask.For object manipulation, the target mask can be refined during denoising using aggregated cross-attention maps; the source mask is used for the first Tm steps before transitioning to the target mask.Source: D. Mask-Guidance Strategy · p. 6

E. Discussion on Differences with Prior Methods

CPAM separates object and background representations and limits prompt influence to target regions, contrasting with prior attention-control methods that propagate image information jointly. The implementation uses SD-1.5, SD-2.1, and SDXL without architectural modification.

E. Discussion on Differences with Prior Methods

  • Unlike MasaCtrl, which propagates object and background information together, CPAM separates their queries for independent control of edited and preserved regions.PnP also propagates object and background features jointly, limiting object-level control; DiffEdit can produce rigid edits for non-rigid pose or view changes.Source: E. Discussion on Differences with… · p. 6
  • CPAM combines mask-guided attention adaptation with localized extraction to disentangle object and background representations and restrict prompt influence to target regions.Together, these components support flexible object-level editing while preserving background consistency.Source: E. Discussion on Differences with… · p. 6

A. Implementation Details

  • CPAM was tested with publicly available SD-1.5, SD-2.1, and SDXL models using DDIM inversion, guidance scale 7.5, and 50 inference steps.The reported experiments used a single A100 GPU.Source: A. Implementation Details · p. 6

B. Image Manipulation BenchmArk (IMBA) Dataset

IMBA extends TEdBench with annotations for controllable object-level editing, including masks, prompts, and editing preferences. The paper evaluates text-image alignment and background preservation, and compares CPAM with established editing approaches using tuned settings.

B. Image Manipulation BenchmArk (IMBA) Dataset

  • IMBA extends TEdBench with alteration masks, object prompts, editing-preference labels, and localized, multi-object editing cases.Its 104 samples include 43 object-retention cases, 97 object-modification cases, and 7 background-alteration cases.Source: B. Image Manipulation BenchmArk (IMBA)… · p. 7

C. Evaluation Metrics

  • Evaluation uses CLIPScore for text-image alignment and LPIPS, DreamSim, and RMSE on the ground-truth background mask for background preservation.The paper reports LPIPS and DreamSim alongside RMSE; it describes DreamSim as better correlated with human perceptual judgments.Source: C. Evaluation Metrics · p. 7

D. Comparison with State-of-the-Arts

  • The comparison covers prompt-based editing, attention manipulation, feature injection, and fine-tuning methods, with publicly available official code used except for Imagic.The authors grid-searched each method's specified hyperparameter ranges while keeping other settings at defaults.Source: 1) Compared Methods · p. 7, D. Comparison with State-of-the-Arts · p. 7

1) Compared Methods:

  • SDEdit uses 25 steps at strength 0.5 (50 total) or 40 at strength 0.8 (default 50); MasaCtrl tests three mask configurations, while DiffEdit uses explicit ground-truth masks.Imagic uses 500 text-embedding optimization steps and 1000 fine-tuning steps, with α from 0.1 to 2.0. PnP uses 0.5–0.8 self-attention replacement and feature-injection ratios with 50 inference steps; FPE uses a 0.5–0.8 self-attention replacement ratio.Source: 1) Compared Methods · p. 7

2) Qualitatitve Evaluation:

CPAM is reported as the only compared method supporting all listed functional capabilities, and qualitative comparisons show coherent edits that preserve background structure across multiple diffusion backbones. The examples span varied real-image editing tasks.

2) Qualitatitve Evaluation:

  • CPAM is the only compared method reported to support localized editing, object removal, and editing without the original image caption.Table I summarizes functional capabilities, including mask control and compatibility with high classifier-free guidance scales.Source: 2) Qualitatitve Evaluation · p. 7, Table I · p. 8
  • Across SD1.5, SD2.1, and SDXL, CPAM produces more coherent edits while preserving background structure and scene layout.The qualitative examples include changes to pose, viewpoint, background context, and object appearance.Source: Figure 7 · p. 8, 2) Qualitatitve Evaluation · p. 8
  • CPAM is applied across different diffusion architectures without architectural modification, requiring adaptation to their attention-layer configurations and feature representations.This compatibility is reported for SD1.5, SD2.1, and SDXL.Source: 2) Qualitatitve Evaluation · p. 8

E. Quantitative Evaluation

CPAM shows strong background preservation across diffusion backbones, while ablations indicate that both preservation adaptation and localized extraction contribute to localized editing and scene consistency. The paper also describes selective high guidance and intermediate step-layer scheduling as ways to balance edit flexibility with preservation.

E. Quantitative Evaluation

  • CPAM-SDXL achieves LPIPS 0.118 and RMSE 18.90 for background preservation, with CLIPScore 29.77; CPAM-SD2.1 has the best DreamSim score and second-best RMSE.The authors report strong performance across backbones and improved performance with stronger models such as SD2.1 and SDXL.Source: E. Quantitative Evaluation · pp. 8–9

1) Effectiveness of Proposed Components:

  • Removing LE raises LPIPS from 0.118 to 0.204 and RMSE from 18.90 to 33.26; removing PA raises RMSE to 38.33 and DreamSim to 0.181.The ablation reports that either removal degrades background preservation, with PA described as particularly important for scene consistency.Source: 1) Effectiveness of Proposed Components · p. 9, Table III · p. 9

2) Effectiveness of High Classifier-Free Guidance Scale:

  • CPAM applies high classifier-free guidance only to regions requiring modification, reducing distortion artifacts associated with globally over-editing the image.The paper reports faithful edits from simple prompts without detailed descriptions.Source: 2) Effectiveness of High… · p. 9

3) Step and Layer Scheduling:

  • An intermediate schedule of T = 3 and L = 7–9 balances semantic consistency with flexible transformations, and the strategy remains stable across backbones.For SDXL, the authors select approximately layers 30–35 to retain a proportional middle-layer position.Source: 3) Step and Layer Scheduling · pp. 9–10, Figure 10 · p. 10

G. User Study

The user study compares prompt-based editing methods using four ratings that capture object and background preservation, realism, and prompt alignment. Twenty participants evaluated the methods, and the reported table states that CPAM significantly outperforms existing methods.

1) Metrics:

  • The user study evaluates object retention, background retention, realism, and satisfaction with the text prompt.Object and background retention assess preservation of the main object and its surroundings; realism focuses on non-rigid edits, while satisfaction measures alignment with the prompt.Source: 1) Metrics · p. 10

2) Participants:

  • Twenty participants, ages 16–22, rated the methods; they were new to AI generative tasks, though some had joined prior AI user studies.The group included 17 males and three females from the research community, including students with AI knowledge and people from outside the industry.Source: 2) Participants · p. 10, Figure 11 · p. 11

G. User Study

  • Table IV reports participants’ opinion scores and states that CPAM significantly outperforms existing methods.The table uses a 1-to-6 rating scale, with bold and underlining marking the best and second-best scores.Source: Table IV · p. 11

3) Setup:

Participants rated editing outputs in a blinded, randomized study, where CPAM received the highest ratings for object retention, background retention, and realism. The discussion also identifies boundaries tied to pretrained-model capacity, mask estimation and accuracy, and editing small objects.

3) Setup:

  • Participants evaluated four methods per sample, selected from a pool of 104, with method identities blinded and presentation order shuffled.Twenty batches each drew 50 samples; participants rated methods from 1 to 6 and left object-retention ratings blank when preservation was not required.Source: 3) Setup · p. 11, 4) Tasks · p. 11

6) Quantitative Results:

  • CPAM received the highest ratings for object retention, background retention, and realism, while Imagic excelled in satisfaction but had background-retention and realism challenges.The ratings are reported as Mean Opinion Scores in Table IV, with detailed statistical ratings shown in Fig. 12.Source: 6) Quantitative Results · p. 11, Figure 12 · p. 11

H. Limitations and Discussion

  • Because CPAM is bounded by its pretrained model’s capacity, significant pose or viewpoint changes may not align with the prompt or achieve the desired structural transformation.The figure comparison states that tuning-free methods cannot generate novel object views and poses aligned with the prompt, unlike fine-tuning method Imagic.Source: H. Limitations and Discussion · p. 11, Figure 13 · p. 12
  • Target-mask tracking can become unreliable with major structural changes, inaccurate masks, or attention drift, and may localize unintended objects in multi-object scenes.Major transformations require cross-attention-map aggregation to estimate the target mask; relative attention-based estimates may not capture changed object structure precisely.Source: H. Limitations and Discussion · pp. 11–12
  • Inaccurate source masks can omit object parts or include multiple salient objects, leading to lost regions or unintended edits.The examples include a dog mask that misses its legs and masks that capture salient objects beyond the intended target.Source: H. Limitations and Discussion · p. 12, Figure 14 · p. 12
  • The method remains robust to moderate mask inaccuracies: a 10% mask expansion does not significantly degrade editing quality, while more substantial mask deviations can introduce artifacts or distort object appearance.Source: H. Limitations and Discussion · p. 12
  • Editing small or less prominent objects can be unstable, though increasing guidance scale can strengthen the text prompt’s influence in those regions.The paper links this challenge to diffusion-model training data that emphasizes highly salient objects over minor spatial details.Source: H. Limitations and Discussion · p. 12

VI. CONCLUSION

CPAM combines self- and cross-attention to support zero-shot real-image editing, retaining object attributes and background context while limiting prompt effects in undesired regions. The conclusion reports strong results on complex, non-rigid edits, and the accompanying mask illustration shows sensitivity to mask precision.

VI. CONCLUSION

  • CPAM combines self- and cross-attention mechanisms, object-preservation control, background preservation, and localized prompt conditioning for zero-shot editing.The paper also introduces IMBA for comprehensive image-manipulation evaluation and reports that CPAM consistently outperforms leading editing techniques on complex, non-rigid edits.Source: §VI · p. 12
  • Mask precision affects outputs: 10% expansion leaves the result unchanged, while 20% expansion distorts the cat’s left eye and a coarse mask changes a sweater to red.The illustration compares different mask-generation methods and their impact on editing performance.Source: Figure 15 · p. 13

Related papers

Loading 2506.18438v3…