PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Object Removal via Attention Masking, Latent Anchoring, and Refinement

Arman Taghizadeh, Ulf Krumnack, Kai-Uwe Kühnberger

arXiv:2609.28342v1cs.CV

TL;DR

Removing an object requires suppressing residual features while preserving the surrounding scene and generating context-consistent replacement content. This paper uses a zero-shot constrained-inpainting pipeline with frozen Stable Diffusion; results show effective removal, with masked NTI helping complex scenes and refinement reducing residual traces and boundary artifacts.

  • Problem

    Object removal must suppress residual object features, preserve the unedited scene, infer occluded background, and maintain consistency across the mask boundary.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The framework combines mask construction, caption conditioning, DDIM inversion, background-weighted masked NTI, attention masking, outside-mask latent anchoring, and localized refinement with frozen Stable Diffusion.

    Source: §1 · p. 3

  • Results

    The pipeline removes selected objects while preserving most surrounding content; masked NTI is particularly useful near structural boundaries, and refinement reduces residual traces and improves seam quality.

    Source: §5.3 · pp. 7–8, §7 · p. 8

  • Takeaways & Limitations

    Background-weighted masked NTI is especially beneficial when replacement content must agree with nearby structure, texture, or scene geometry.

    Source: §5.3 · p. 8

    The method depends strongly on mask quality, and shadows or reflections outside the mask remain unchanged unless included in the editable region.

    Source: §6 · p. 8

Abstract

from arXiv · show

Removing an object from a real image requires more than synthesizing plausible content within a mask: the method must suppress residual object features, preserve the unedited scene, and generate replacement content that is consistent with the surrounding background. This paper approaches object removal from a stage-based perspective and proposes a zero-shot framework for constrained latent inpainting with a frozen pretrained Stable Diffusion model, requiring no task-specific training or model fine-tuning. The method integrates SAM-based mask construction, BLIP image-caption conditioning, DDIM inversion, background-weighted masked null-text optimization, decoder self-attention masking, hard outside-mask latent anchoring, and localized renoise--denoise refinement into a unified pipeline. The method is evaluated through qualitative examples, quantitative local-consistency metrics, and ablation studies. The results demonstrate effective object removal and context-consistent replacement content. The ablations indicate that background-weighted masked NTI is particularly beneficial for structurally complex backgrounds, whereas the no-NTI variant is sufficient in other evaluated examples. Repeated refinement further reduces object remnants and boundary artifacts remaining after the primary editing pass.

1. Introduction

Prior zero-shot editing methods address different subsets of preservation, harmonization, suppression, and background generation; the paper frames object removal as requiring these stages jointly. It proposes a unified, zero-shot constrained-inpainting pipeline using frozen Stable Diffusion.

1. Introduction

  • Sampling-based inpainting methods use known-region replacement, optimization, or guidance; RePaint, Blended Latent Diffusion, DiffEdit, GradPaint, CoPaint, and PILOT illustrate these strategies.Their mechanisms include repeated resampling, latent blending, mask guidance, gradient losses, posterior constraints, and latent-space optimization.Source: §1 · p. 1
  • LanPaint treats inpainting as partial conditional sampling, using Bidirectional Guided Score for mutual adaptation and Fast Langevin Dynamics for faster convergence.The method is described as training-free and asymptotically exact for ODE-based diffusion samplers and rectified-flow models.Source: §1 · p. 2
  • Textual and embedding-based approaches manipulate prompts or attention for suppression and localized editing, while Uni-paint combines multiple guidance types but uses masked fine-tuning (Yang, Chen, et al. 2023).The reviewed methods also include MagicRemover, negative-prompt analysis, SuppressEOT, HD-Painter, and ZONE.Source: §1 · p. 2
  • Attention-guided methods use self-attention perturbations or redirection to suppress object information, while AdaEraser notes that indiscriminate suppression can also disrupt background generation (Liu 2026).SAG and PAG establish training-free sampling guidance from attention-informed perturbations.Source: §1 · pp. 2–3
  • Object removal jointly requires localization, real-image reconstruction, background preservation, object-feature suppression, replacement generation, and boundary refinement.Prior approaches address different subsets, with trade-offs between preservation, harmonization, spatial control, and plausible background generation.Source: §1 · p. 3
  • The proposed framework combines mask construction, image-caption conditioning, inversion, masked null-text optimization, attention masking, latent anchoring, and localized refinement without task-specific training or fine-tuning.Its components localize synthesis, establish a background anchor, suppress target-feature leakage, preserve unedited regions, and refine replacement content.Source: §1 · p. 3

2. Preliminaries

The preliminaries explain how latent trajectories support real-image reconstruction, how null-text optimization addresses guidance mismatch, and why attention and denoising time matter for localized editing.

2.1. Latent Diffusion and DDIM Inversion

  • Latent diffusion operates in a pretrained VAE’s latent space, and DDIM inversion reverses its deterministic timestep mapping to provide source-image references at corresponding noise levels.DDIM inversion maps the source latent to a noisy latent along a time-aligned trajectory.Source: §2.1 · pp. 3–4

2.2. Classifier-Free Guidance and Null-Text Inversion

  • Classifier-free guidance combines conditional and unconditional predictions, while Null-Text Inversion optimizes timestep-specific unconditional embeddings to improve compatibility with the image’s inversion trajectory.The conditional prompt and diffusion model remain fixed during this optimization.Source: §2.2 · p. 4

2.3. Self-Attention and Denoising-Time Dynamics

  • Self-attention transfers information among spatial tokens, and its control must account for denoising stage: early steps shape layout, while later steps and higher-resolution features shape local detail.These dynamics motivate timestep-dependent decoder attention masking and intermediate-timestep refinement.Source: §2.3 · p. 4

3. Problem Definition and Challenges

The task is to remove a selected object while generating plausible, context-consistent content inside its mask and leaving the outside region unchanged. The formulation must balance this goal against reconstruction, suppression, mask, and boundary challenges.

3. Problem Definition and Challenges

  • Given a real image and binary object mask, the goal is to remove the selected object while synthesizing plausible content inside the mask and preserving the outside region.The mask convention assigns 1 to object pixels and 0 otherwise.Source: §3 · p. 4
  • Latent features and attention can retain object information, global denoising can alter the background, and inversion that improves reconstruction can encourage the masked object to reappear.The edit must preserve background while leaving enough freedom to resynthesize the masked region.Source: §3 · p. 4
  • Mask quality and boundary consistency constrain the edit: incomplete masks may leave fragments, oversized masks require more regeneration, and local mismatches can create seams or halos.The overall formulation balances object suppression, outside-mask preservation, context-consistent synthesis, and boundary refinement.Source: §3 · p. 4

4. Methodology

The pipeline performs zero-shot constrained latent inpainting with a frozen Stable Diffusion model, coordinating mask localization, background preservation, object-feature suppression, and context-aware refinement.

4.1. Pipeline Overview

  • The pipeline uses a time-aligned inversion trajectory and distinct controls for localization, object-feature suppression, outside-mask preservation, and local consistency.The mask defines the editable region, while inversion and hard anchoring preserve the source scene as synthesis proceeds.Source: §4.1 · p. 4

4.2. Mask Construction and Text Conditioning

  • SAM-based mask construction defines the editable region, while a BLIP image caption conditions inversion, optimization, editing, and refinement without a target-specific prompt.The mask is downsampled to latent resolution, and the caption describes the input image while inference-time constraints control removal.Source: §4.2 · pp. 4–5

4.3. Background-Weighted Masked NTI

  • Masked NTI prioritizes reconstruction outside the object mask, while weak in-mask weighting avoids forcing accurate reconstruction of the target object.Timestep-specific unconditional embeddings align guided denoising with the preserved source trajectory.Source: §4.3 · p. 5

4.4. Decoder Self-Attention Masking

  • Decoder self-attention masking blocks object-region keys as context sources while allowing masked-region queries to use background keys, shifting from soft early masking to hard later masking.The restriction strengthens later in denoising, when object remnants and local textures are more likely to appear.Source: §4.4 · pp. 5–6

4.5. Primary Editing with Hard Background Anchoring

  • At every editing step, hard projection restores the outside-mask latent from the source inversion trajectory while allowing synthesis inside the mask.This constrains drift in the preserved region without preventing generation in the editable region.Source: §4.5 · p. 6

4.6. Localized Renoise–Denoise Refinement

  • Repeated localized renoise–denoise rounds revise masked content while keeping the outside region anchored, addressing remnants, texture inconsistencies, and boundary artifacts.A smaller renoise level permits more revision but costs additional denoising and stability; a larger level mainly supports minor corrections.Source: §4.6 · pp. 6–7

5. Experiments

Experiments assess qualitative removal and local consistency, finding that anchoring preserves much of the surrounding scene, refinement reduces residual artifacts, and masked NTI helps most in structurally challenging examples.

5.1. Implementation Details

  • The implementation uses Stable Diffusion v1.4, DDIM inversion, SAM-assisted masks, BLIP captions, and 50 denoising steps, with a distinct no-NTI variant using a fixed empty-prompt embedding.Refinement uses 10 rounds from the middle of the denoising trajectory, with hard outside-mask anchoring.Source: §5.1 · p. 7

5.2. Evaluation Protocol

  • The evaluation combines qualitative inspection with local boundary and context-consistency metrics because object removal lacks a unique ground-truth completion.The metrics target local failure modes rather than complete removal success.Source: §5.2 · p. 7

5.3. Qualitative Results and Ablation Analysis

  • Hard outside-mask anchoring keeps surrounding structures largely aligned, while refinement reduces residual object traces and improves local texture and seam quality.Some artifacts remain in scenes with dense structures, neighboring objects, shadows, or fine background details.Source: §5.3 · pp. 7–8
  • Without masked NTI, inversion, self-attention masking, hard anchoring, and refinement are often sufficient for homogeneous backgrounds or regular, repetitive textures.Figure 4 shows the no-NTI pipeline after refinement.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Figure 5 reports that masked NTI removes a visible artifact and is more useful when the fill must match nearby structural boundaries, textures, or scene geometry.Its outside-dominant objective improves compatibility between guided denoising and the inversion trajectory while leaving the masked region relatively unconstrained.Source: Figure 5 · p. 8, §5.3 · p. 8
  • Negative prompting produced no consistent improvement under the stronger spatial controls and was omitted; more aggressive optimization or refinement increases runtime and may reduce stability.More aggressive settings include additional masked-NTI iterations, more refinement rounds, or refinement from an earlier timestep.Source: §5.3 · p. 8

6. Limitations

The method has practical limits tied to mask quality, fixed outside-mask content, optimization cost, and the pretrained model's capacity.

6. Limitations

  • Incomplete masks may leave object fragments, while oversized masks require more content to be regenerated.Shadows and reflections outside the mask remain unchanged, so they must be included in the editable region when removal is intended.Source: §6 · p. 8
  • Masked NTI requires per-image, per-timestep optimization; the faster no-NTI variant may be less reliable in structurally complex scenes..Source: §6 · p. 8
  • Replacement quality remains limited by the pretrained diffusion model, and the local-consistency metrics do not fully assess whether the target object remains visible.Complex geometry, fine structures, or weakly represented content may produce artifacts, semantic errors, or inconsistent textures.Source: §6 · p. 8

7. Conclusion

The paper concludes that coordinated inference-time control of a frozen Stable Diffusion model can support zero-shot object removal. Masked NTI helps particularly in structurally complex scenes, while no-NTI works in many evaluated examples.

7. Conclusion

  • The framework performs constrained latent inpainting with a frozen pretrained Stable Diffusion model, without task-specific training or fine-tuning.Its pipeline combines SAM masks, BLIP captions, DDIM inversion, masked NTI, decoder self-attention masking, hard outside-mask anchoring, and localized refinement.Source: §7 · p. 8
  • The coordinated components suppress target-object information, preserve the surrounding scene, and improve consistency between synthesized content and background..Source: §7 · p. 8
  • Background-weighted masked NTI is particularly beneficial in structurally complex scenes, while the no-NTI variant is sufficient in many evaluated examples..Source: §7 · p. 8

Zero-Shot Object Removal via Attention-Guided Diffusion Inpainting

This section evaluates decoder self-attention masking, refinement, and masked NTI across examples using visual results and local-consistency metrics, with outcomes varying by case and incomplete mask coverage sometimes leaving object parts.

  • Masked NTI produces cleaner refined removals in some examples, but other cases show little or no qualitative advantage over no-NTI.The balloon and reflection and the woman’s shoe region show clear gains; the cigarette, baby, porch, and wall-and-door examples show no clear gain or persistent structural errors.Source: Zero-Shot Object Removal via… · pp. 12–17
  • For the balloon example, masked NTI removes the object and its reflection more effectively, and refinement yields a clean result where no-NTI leaves a minor artifact.Metrics favor masked NTI on boundary-gradient mismatch, local feature distance, SSIM, and LPIPS, while no-NTI is better on color jump, outside-ring consistency, and CLIP alignment.Source: Zero-Shot Object Removal via… · p. 12
  • Refinement makes no-NTI sufficient in several examples, removing remaining artifacts in the cigarette, baby, porch, bench, and hockey-stick removals.Across these cases, quantitative comparisons are mostly close or mixed, with only small differences and no substantial or clear NTI advantage.Source: Zero-Shot Object Removal via… · pp. 13, 14, 16, 19, 20
  • For the girl-removal example, masked NTI gives a cleaner intermediate and final result, while metrics are mixed and do not uniformly favor either variant.Perceptual and feature-based measures capture some improvement, but local seam metrics do not fully reflect the qualitative reduction of artifacts inside the removed region.Source: Zero-Shot Object Removal via… · p. 15
  • The man-removal example is a failure case: both variants leave wall and door-frame inconsistencies, and local metrics do not fully capture these visible structural errors.The passage attributes the difficulty to the structural complexity of the background and emphasizes interpreting metrics together with visual inspection.Source: Zero-Shot Object Removal via… · p. 17
  • In the bathtub example, refined masked NTI is visually cleaner, while no-NTI refinement introduces an extra wine glass despite mixed metrics that favor no-NTI on most local measures.This illustrates that local metrics can miss semantic inconsistency; stronger refinement or more masked-NTI inner iterations may further improve the result.Source: Zero-Shot Object Removal via… · p. 18
  • In another example, masked NTI improves the edited region’s appearance, but quantitative metrics are mixed and the benefit is concentrated in a specific problematic region.Source: Zero-Shot Object Removal via… · p. 21
  • For the small table object, decoder self-attention masking alone produces a clean removal, with no visible improvement from refinement or masked NTI.The variants’ quantitative metrics are nearly identical, supporting the qualitative finding that masked NTI adds no meaningful advantage in this simple case.Source: Zero-Shot Object Removal via… · p. 22
  • For the skiing example, refinement removes artifacts left by no-NTI, but an under-masked shadow is restored by hard background locking, and masked NTI offers no substantial improvement.This illustrates that hard background locking preserves unmasked content even when it belongs to the object being removed.Source: Zero-Shot Object Removal via… · p. 23
  • For the beach example, masked NTI removes artifacts more effectively than no-NTI, and refinement further improves the result, though minor artifacts remain.Quantitative results partly support the visual advantage: masked NTI improves boundary-gradient consistency, while other metrics favor no-NTI.Source: Zero-Shot Object Removal via… · p. 24
Loading 2609.28342v1…