AdaEraser: Training-Free Object Removal via Adaptive Attention Suppression

1. Introduction

• Object removal requires eliminating specified objects while plausibly reconstructing background content in the vacated regions.The two goals can conflict when self-attention in object regions is suppressed indiscriminately.Source: §1 · p. 1
• Token-wise self-attention-map similarity between the original image and denoising process correlates with object presence and serves as a heuristic for adaptive suppression.The maps are Softmax-normalized, making them comparable across the original image and denoising process.Source: §1 · p. 2
• AdaEraser is a training-free object-removal framework that monitors target presence, computes token-wise presence scores, and uses them to modulate self-attention suppression.The suppression coefficients are based on the presence scores.Source: §1 · p. 2
• Across varying benchmarks, AdaEraser consistently outperforms prior inpainting and object-removal methods in removal and visual-quality preservation, including training-based approaches.The experiments cover varying benchmarks.Source: §1 · p. 2
• The introduction identifies adaptive regulation based on residual object presence as central to high-quality removal and presents AdaEraser as a new perspective on the task.The paper describes token-wise self-attention similarity as an effective proxy for that presence.Source: §1 · p. 2

2.1. Object Removal

• Training-based removal methods use adapters, finetuning, or dedicated datasets, but acquiring high-quality data is described as prohibitively costly.Some methods also use learnable embeddings or textual prompts as additional guidance.Source: §2.1 · p. 3
• Training-free methods exploit pretrained diffusion priors; MagicRemover reduces target-token cross-attention scores (Yang et al., 2023), while AttentiveEraser blocks image-to-object self-attention (Sun et al., 2025).The cited discussion says AttentiveEraser's background restoration suffers from low visual quality because attention within the region is lacking.Source: §2.1 · p. 3

2.2. Attention Mechanism within Diffusion Models

• Self-attention maps tend to reflect spatial layout during generation, and query-key pairs within the same object tend to have larger scores.Prior work is cited for self-attention's role in controlling spatial arrangement (Tewel et al., 2023; Tumanyan et al., 2023; Nam et al., 2024).Source: §2.2 · p. 3
• The paper analyzes changes in token-wise self-attention maps for removal regions to guide a balance between object removal and background reconstruction.Source: §2.2 · p. 3

3. Method

• AdaEraser adapts self-attention suppression token by token to remove the masked object while enabling background restoration.The task also aims to keep content outside the mask unchanged.Source: §3 · p. 3

3.1. Self-Attention Maps in Object Removal

• Tokens for different parts of the same object can focus on distinct regions, supporting token-wise rather than object-wide attention operations.Cow head, abdomen, and tail tokens highlight different regions.Source: §3.1 · p. 4, Figure 2 · p. 4
• As denoising progresses, attention-map similarity between a target-region token before and during removal decreases as different content occupies that region.Early maps reflect coarse contours and remain similar; later maps increasingly capture the semantic concept at the location.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Motivation Clarification

• Full suppression can remove the object but over-rely on local background cues, producing an incorrect reconstruction such as grass instead of a dog’s leg.The design goal is to retain effective removal while recovering global context.Source: §3.2 · pp. 4–5
• AdaEraser estimates object presence relatively by comparing attention maps during denoising with an unsuppressed reference, rather than directly measuring presence in latent space.The paper notes that direct latent-space measurement is challenging; the reference map still reflects target-object features.Source: §3.2 · p. 5

3.3. AdaEraser

• For reference maps, the method diffuses the source image to each timestep and denoises once, avoiding a full inversion search for optimal latent noise.Gaussian noise is introduced at multiple scales, one for each forward-diffusion timestep, to preserve source-image feature information under noisy conditions and avoid cumulative errors.Source: §3.3 · p. 5
• The token-wise presence score is a heuristic relative signal from comparing source and target attention maps, not a calibrated measure of true object existence.The method flattens maps and uses cosine similarity in its experiments.Source: §3.3 · p. 6
• AdaEraser uses the presence score to adaptively modulate self-attention, then iteratively combines the target foreground with the source background before decoding the result.The denoising loop proceeds from t = T to 1; the final latent is passed to the VAE-Decoder.Source: §3.3 · p. 6

4.2. Comparisons with State-of-the-art Methods

• On Mulan and OABench, AdaEraser outperforms training-based and training-free competitors across all reported metrics.The comparison evaluates object-removal quality and background coherence.Source: Table 1 · p. 7, §4.2 · p. 8
• Baseline methods show unintended objects, incomplete removal, blur, detail loss, or context-inconsistent fills, whereas AdaEraser removes targets while preserving scene context.Figure 6 compares these qualitative outcomes.Source: Figure 6 · p. 7, §4.2 · p. 8

4.1. Datasets and Evaluation Protocols

• Mulan provides source images, object masks, and object-removed ground truths, while OABench contains real-world removal examples; evaluation includes image fidelity and region-coherence metrics.The study also uses user rankings and reports Average Human Ranking.Source: §4.1 · pp. 7–8

4.3. Analysis

• Compared with AttentiveEraser, AdaEraser adds parallel presence-score estimation before suppression, with computational and memory overhead within 15%.Source and target latent denoising runs in parallel, with suppression applied only to the target.Source: Table 2 · p. 8, §4.3 · p. 8
• Presence scores decline progressively during removal with layer- and token-specific patterns; token-wise suppression outperforms timestep- and region-based alternatives, and the target-timestep reference performs best.The reported reference result attributes its performance to precise noise-level alignment.Source: Figure 7 · p. 8, §4.3 · pp. 8–9, Table 3 · p. 8

4.4. Discussion

• AdaEraser is robust to slightly loose masks, but incomplete masks can leave uncovered objects or their shadows, reflections, and context changes visible.Mask quality and coverage affect the editing result.Source: §4.4 · p. 9
• Similar textures, repetitive patterns, and overlapping same-class objects can make presence scores less discriminative, although AdaEraser performs well in many such cases.The paper attributes the challenge to shared local appearance between targets and surrounding content.Source: §4.4 · p. 9
• When removed content is structurally complex or semantically ambiguous, reliance on the generative model's prior can yield semantic inaccuracies, structural distortions, or degraded visual quality.The paper suggests stronger structural guidance or semantic priors as a future direction.Source: §4.4 · p. 9, Figure 8 · p. 9

5. Conclusion

• AdaEraser is a training-free framework that adaptively balances object elimination and background restoration.It adjusts suppression strength according to estimated object presence to avoid drawbacks of indiscriminate suppression.Source: §5 · p. 9
• The paper investigates token-wise self-attention dynamics in pretrained diffusion models as a basis for adaptive suppression.Source: §5 · p. 9
• Comprehensive experiments and user studies validate AdaEraser's effectiveness on challenging object-removal benchmarks.Source: §5 · p. 9

Overview

• The appendix covers theoretical justification, ablations, mask quality, accelerated models, architecture generalization, user studies, and additional qualitative results.Source: Overview · p. 13

A. Theoretical Justification of AdaEraser

• The theoretical discussion is explanatory rather than a formal guarantee of convergence or optimality.It aims to explain the presence score and adaptive suppression under realistic diffusion priors without claiming exact latent-semantic recovery.Source: A. Theoretical Justification of… · p. 13

A.1. Presence Score as a Monotonic Semantic Indicator

• The presence score is an order-preserving proxy for relative semantic persistence, not an unbiased estimate of true semantic density.The analysis states that its monotonic mapping preserves semantic ordering.Source: §A.1 · pp. 13–14

A.2. Robustness under Diffusion Noise

• Using the same Gaussian noise in both branches induces positively correlated perturbations and stabilizes token-wise presence-score rankings.The analysis considers the full range ρ ∈[0, 1].Source: §A.2 · p. 14
• The high-dimensional approximation holds with high probability as D →∞, while the tD terms damp noise and can preserve presence-score ordering even when ρ = 0.Ordering remains stable when the underlying semantic signal has a finite margin.Source: §A.2 · pp. 14–15

A.3. Variational Interpretation of Adaptive Attention Suppression

• Adaptive suppression is interpreted as a KL-regularized optimization, not as a claim of global optimality for full diffusion dynamics.The parameter-free gate η(j) = 1 − p(j) is presented as a monotone approximation of exp(−βp(j)).Source: §A.3 · p. 15

A.4. Bounded Residual Behavior under Iterative Suppression

• The UUB interpretation allows semantic residue to approach a bounded neighborhood rather than vanish, consistent with a small late-step presence-score floor.The proposed interpretation motivates relaxing suppression to avoid over-suppression artifacts while preventing re-emergence.Source: §A.4 · p. 15

B.2. Reference Selection

• A reference with the same noise level as the target yields the best removal results, while fixed-noise references can cause over-deletion, incomplete removal, or insufficient detail restoration.The passage attributes the advantage to more accurate presence-score computation when noise levels match.Source: §B.2 · p. 16

B.1. Self-Attention Suppression Strategy

• Token-level adaptive suppression yields higher-quality removal than timestep-based suppression, which can leave objects incompletely removed or cause structural abnormalities, and region-based suppression, which lacks fine-grained token perception.Figure 9 presents the visual comparison.Source: §B.1 · p. 15, Figure 9 · p. 16

C. Effect of Mask Quality on AdaEraser

• Incomplete masks may leave object-induced shadows, reflections, or context changes visible, so AdaEraser is more sensitive to under-masking.The reported editing results are better when masks cover both the object and its associated side effects.Source: C. Effect of Mask Quality on AdaEraser · p. 16
• Slightly loose masks can retain stable removal quality because full-image attention lets background tokens inside the mask use context from outside it.The method maintains a global receptive field rather than hard-isolating the masked region.Source: C. Effect of Mask Quality on AdaEraser · p. 16

D. Compatibility with Accelerated Distilled Models

• With SDXL-Lightning at 4 denoising steps, removal and reconstruction quality degrade compared with standard SDXL, with more artifacts in edited regions.The setting significantly reduces inference time.Source: D. Compatibility with Accelerated… · p. 16
• The authors believe compressed denoising weakens adaptive erasing because AdaEraser relies on sufficiently rich multi-step attention evolution.They identify compatibility with highly accelerated distilled models as future work.Source: D. Compatibility with Accelerated… · p. 17

E. Generalizability to Other Architectures

• SDXL is the main experimental model because the authors consider its generation-quality and efficiency balance favorable.Source: E. Generalizability to Other… · p. 17
• AdaEraser achieves stable erasure across SD1.5, SD2.1, SDXL, and FLUX, with cross-backbone results reported on Mulan.The authors describe the strategy as not tied to a specific backbone.Source: Table 5 · p. 17, E. Generalizability to Other… · p. 17

F. Details about User Study

• For each user-study evaluation instance, the eight candidate results were randomly shuffled to avoid ordering bias.Figure 14 shows a questionnaire example.Source: F. Details about User Study · p. 17

• Across varied scenarios, AdaEraser is described as stable, though adaptive suppression might inadvertently suppress background cues when object and background textures are highly similar.Source: G. More Results · p. 17, Figure 15 · p. 20
• Overlapping same-class objects are especially challenging because the target and remaining content resemble each other in appearance and local texture, yet AdaEraser still performs well.Source: G. More Results · p. 17
• Figures 11–12 cover different and loose masks; when the source includes the target and its side effects, removal quality depends heavily on the provided mask.Source: Figure 11 · p. 18, Figure 12 · p. 18