PaperlayerView exact revision on arXiv

Source-linked AI summary

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

Sanghyun Jo, Donghwan Lee, Eunji Jung, Seong Je Oh, Kyungsu Kim

arXiv:2512.21545v2cs.CV

TL;DR

Object removal must restore occluded background structure while preventing the target from reappearing, but dataset-free methods can confuse non-target foregrounds with background and blur distinct subtypes. EraseLoRA uses MLLM-guided foreground separation and test-time adaptation to integrate background cues; it nearly halves unwanted foreground re-generation and retains gains across alternative backbones and MLLMs.

  • Problem

    Dataset-free attention-manipulation methods can misinterpret non-target foregrounds as background and blur or misalign distinct background subtypes.

    Source: §2.2 · p. 5

  • Method

    EraseLoRA uses an MLLM to separate target foreground, non-target foregrounds, and clean background, then applies LoRA-based test-time optimization to integrate background cues without explicit attention blocking.

    Source: §1 · p. 3, §3.1 · p. 7

  • Results

    EraseLoRA nearly halves unwanted foreground re-generation, with gains retained when diffusion backbones and MLLMs are replaced by alternatives.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The framework provides a dataset-free, plug-in approach for coherent background reconstruction across diffusion backbones.

    Source: §1 · p. 3

    The full fixed-500-iteration optimization takes approximately three minutes per test image on SD3.5-M, adding inference latency and memory use.

    Source: §4.3 · pp. 14–15

Abstract

from arXiv · show

Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches manipulate the diffusion model's internal self-attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a multimodal large-language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives without explicit attention intervention. As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.

1 Introduction

Object removal must recover the occluded background faithfully, yet existing methods can hallucinate content or mishandle non-target foregrounds and diverse background subtypes. EraseLoRA combines MLLM-guided foreground exclusion with test-time adaptation to reconstruct background cues without explicit attention blocking.

1 Introduction

  • Existing dataset-free methods can mistake non-target foregrounds for background and apply uniform attention constraints that blur textures and misalign distinct subtypes.More broadly, inpainting methods may generate plausible textures or new objects instead of restoring the underlying background structure.Source: §1 · p. 2
  • EraseLoRA uses an MLLM to separate target foreground, non-target foregrounds, and background, then adapts with LoRA to integrate background subtypes without attention blocking.It is designed as a plug-in for diverse pretrained diffusion backbones and requires no additional data or retraining.Source: §1 · p. 3
  • The paper introduces three-label ground-truth annotations and two evaluation metrics for unpaired object-removal settings.The labels distinguish target foreground, non-target foreground, and background.Source: §1 · p. 4
  • At least 23% higher background fidelity than previous dataset-free methods, with unwanted foreground re-generation nearly halved.The gains remain when diffusion backbones and MLLMs are replaced by alternatives.Source: §1 · p. 4

2 Related Work

Generative inpainting can produce plausible completions without restoring original scene content, while object-removal methods must also preserve target-unrelated regions. EraseLoRA uses MLLMs to infer occluded background cues and aggregates subtypes through test-time adaptation.

2.1 Image Inpainting with Generative Models

  • GAN- and diffusion-based inpainting can create realistic textures but may hallucinate new objects rather than restore the underlying background structure.Diffusion inpainting methods fine-tuned on paired datasets still tend to favor context-consistent completion.Source: §2.1 · p. 4

2.2 Diffusion Models for Object Removal

  • Object removal requires erasing the masked target while restoring occluded background structure and preserving target-unrelated regions.Dataset-driven methods rely on additional, often paired data, whereas dataset-free methods operate directly on pretrained diffusion models.Source: §2.2 · p. 4
  • Dataset-free attention-manipulation methods can regenerate non-target foregrounds and disrupt fine textures or integration of multiple background cues.They treat the masked area as foreground while leaving background subtypes undifferentiated.Source: §2.2 · p. 5
  • EraseLoRA uses external visual reasoning to exclude non-target foregrounds and produce clean background cues, then aggregates inferred subtypes through test-time adaptation without explicit attention blocking.The framework is plug-and-play across diffusion backbones.Source: §2.2 · p. 5

2.3 MLLMs for Visual Reasoning in Image Editing

  • Unlike MLLM editing and inpainting approaches that reason over visible context or generate new content, EraseLoRA infers the background behind the target and identifies non-target foregrounds.This produces background cues not directly visible in the input image.Source: §2.3 · p. 6

3 Method

EraseLoRA combines MLLM-guided foreground/background separation with test-time adaptation to reconstruct the masked region from clean background cues. BFE excludes non-target foregrounds, while BRSA aggregates background subtypes using complementary reconstruction and puzzle losses.

3.1 Background-aware Foreground Exclusion

  • BFE uses MLLM reasoning and Tag2Mask localization to separate target foreground, non-target foregrounds, and clean background from an image-mask pair.It treats occluded objects or scene components behind the target as background subtypes and uses the remaining pixels as clean background.Source: §3.1 · p. 7

3.2 Background-aware Reconstruction with Subtype Aggregation

  • BRSA adapts LoRA at test time to aggregate multiple background subtypes and reconstruct the masked region with structural and contextual consistency.Its total objective combines background reconstruction and background puzzle losses, softly regulating attention without hard attention blocking.Source: §3.2 · pp. 7–8
  • The background reconstruction loss anchors reconstructed latents to the original image only at clean-background locations, promoting fidelity and globally coherent reconstruction.The loss is applied over the clean background mask MB.Source: §3.2 · p. 8
  • The background puzzle loss treats each subtype as a puzzle piece and concentrates its attention on target or clean-background regions to improve spatial consistency.The strongest attention response across inferred background subtypes is selected at each location.Source: §3.2 · pp. 8–9

4 Experiments

Across OpenImages V7 and RORD, EraseLoRA improves background alignment and suppresses foreground regeneration relative to prior methods, while preserving unmasked regions. The experiments also examine component effects, model flexibility, occlusion cases, and inference limitations.

4.1 Experimental Setup

  • Evaluation uses 200 OpenImages V7 samples and 343 RORD frames, refined three-label annotations, and DINO-based FG Sim. and BG Sim. plus unmasked-region SSIM.The implementation freezes backbone parameters and optimizes inserted LoRA adapters for 500 iterations.Source: §4.1 · pp. 9–10

4.2 Comparison with State-of-the-art Approaches

  • BG Sim. rises from 0.605 to 0.743 on OpenImages V7 and from 0.582 to 0.779 on RORD; FG Sim. falls from 0.286 to 0.151 and 0.319 to 0.138, respectively.EraseLoRA also has the highest BG Sim. and lowest FG Sim. among dataset-driven methods on both benchmarks, with background preservation around 0.90, about 0.18 higher than all five methods.Source: Table 2 · p. 10, §4.2 · p. 10

4.3 Discussion

  • Adding BFE to prior dataset-free methods improves background similarity by up to 6.6% and reduces foreground similarity by 8.6%.The module excludes non-target foregrounds from reference regions without modifying those methods’ inference pipelines.Source: §4.3 · p. 11, Table 3 · p. 12
  • Lrecon improves BG Sim. from 0.605 to 0.736 (+21.7%) but can leave faint foreground traces; Lpuzzle alone suppresses artifacts but may miss fine structure and global patterns.Together, the losses produce coherent, detail-preserving restoration.Source: §4.3 · p. 12, Table 3 · p. 12, Figure 10 · p. 13
  • EraseLoRA gains on both similarity metrics across diffusion backbones; even 7B MLLMs yield gains up to 20.3% in BG Sim. and 42.7% in FG Sim.The 78B MLLM is the default because it provides the strongest background-aware reasoning and best overall performance.Source: §4.3 · pp. 12–13, Table 4 · p. 13
  • Across Tag2Mask variants, BG Sim. gains are at least 10.0% and FG Sim. reductions at least 28.3% over SD3.5-M; Grounded SAM2 reaches +22.8% and −47.2%.Source: Table 4 · p. 13, §4.3 · p. 14
  • EraseLoRA reconstructs occluded content such as paintings and stairs, while human-guided masks and background tags can correct challenging failure cases and bypass MLLM overhead.The interactive workflow accepts points or bounding boxes to refine non-target foreground masks.Source: Figure 9 · p. 12, §4.3 · p. 14, Figure 11 · p. 14
  • Inference takes approximately three minutes per SD3.5-M test image; early stopping averages about 141 iterations, giving over 3.5× speedup while retaining over 96.5% of BG Sim.Complex scenes with subtle semantic boundaries or heavy occlusion can yield incomplete or inaccurate removal.Source: §4.3 · pp. 14–15

5 Conclusion

EraseLoRA is presented as a dataset-free, MLLM-guided framework for faithful background restoration. The preliminaries describe diffusion denoising and how self- and cross-attention relate image tokens to one another and to conditioning tokens.

5 Conclusion

  • EraseLoRA uses MLLM-guided background-aware reasoning and test-time adaptation to restore backgrounds without paired data, and is designed to work across diffusion backbones.The conclusion identifies video object removal, where shared background context across frames could amortize per-image optimization, as future work.Source: §5 · p. 15

A Preliminaries

  • Diffusion models iteratively denoise a latent representation and decode the resulting latent into an image.The input image is first encoded into a latent; denoising may be conditioned on text or other external signals.Source: A Preliminaries · p. 20
  • Self-attention models dependencies among latent tokens, while cross-attention links latent tokens to external condition tokens such as text.Modern backbones may compute attention over larger patch tokens rather than individual latent tokens.Source: A Preliminaries · p. 20

B.1 Implementation Details

The implementation adapts only LoRA attention adapters at test time, with configurations spanning diffusion backbones, MLLMs, and tag-to-mask models. Evaluation uses similarity and preservation measures for unpaired removal.

B.1 Implementation Details

  • Only LoRA adapters inserted into attention blocks are updated during test-time optimization; backbone parameters remain frozen.Baselines use official implementations when available, or are re-implemented from their paper descriptions.Source: §B.1 · p. 20
  • The reported cross-attention subtype maps average responses across attention layers, heads, and the text tokens for each subtype, then normalize across subtype tags.The dominant aggregation map takes the strongest normalized subtype response at each spatial location; matching it to target and clean-background regions encourages aggregation there and suppresses non-target foreground responses.Source: §B.1 · p. 21

Configurations.

  • The default configuration uses SD3.5-M, InternVL3-78B, and Grounding DINO with SAM2; LoRA rank 32 gives the best results across experiments, while alternatives are tested for each component family.Source: Configurations · pp. 21, 23

Metric Details.

  • For unpaired evaluation, BG Sim. measures similarity between the reconstructed masked region and the input background, while FG Sim. measures unintended foreground reference.FG Sim. is weighted by (1 − BG Sim.); BG Pres. evaluates preservation of the unmasked region using SSIM.Source: Metric Details · pp. 23–24

Limitations of Attention Surgery.

Attention surgery can damage local detail and fails to distinguish different background subtypes. BRSA instead injects image-specific background cues through test-time LoRA adaptation of a frozen backbone.

Limitations of Attention Surgery.

  • Blocking masked-region self-attention disrupts short-range activations needed for fine detail, producing blur and, in patch-level models, checkerboard artifacts.Uniform constraints also treat unmasked context as one background, which can misapply cues across subtypes and cause textural blurring, structural misalignment, and unnatural boundaries.Source: Limitations of Attention Surgery · pp. 24–25

Effect of Test-Time Adaptation.

  • BRSA adapts a frozen backbone with LoRA to inject BFE-inferred background cues, rather than explicitly blocking attention or relying on paired training data.Its complementary objectives aggregate image-specific cues with textual and structural consistency, reducing blur, checkerboard artifacts, and subtype-misaligned textures across diffusion backbones.Source: Effect of Test-Time Adaptation · p. 25

C.1 Experimental Setup

The evaluation setup includes additional baselines and paired benchmarks, but excludes OmniPaint-Bench because its available samples do not provide reliable ground-truth backgrounds. Metrics assess fidelity, though they do not establish semantic removal success on their own.

C.1 Experimental Setup

  • OmniPaint-Bench is excluded because 1,000 of its 1,300 samples overlap with RORD and the remaining 300 contain input-to-ground-truth inconsistencies, including color mismatches in unmasked regions.The evaluation also includes RemovalBench, which provides 68 paired samples with ground-truth images after removal.Source: §C.1 · p. 25, Figure 14 · p. 26
  • The extended paired evaluation reports PSNR, SSIM, and LPIPS on masked regions to measure pixel accuracy, structural consistency, and perceptual similarity.FLUX.1-Fill is added as a competitive baseline beyond the main comparison.Source: §C.1 · pp. 25–26
  • Image-fidelity metrics cannot determine whether the target truly disappears without traces or whether the reconstructed background is contextually plausible.GPT-Metric evaluates these semantic aspects with a removal success rate and a semantic perceptual score on a 0–100 scale.Source: §C.1 · p. 26

C.2 Quantitative Analysis

Extended evaluations show EraseLoRA improves background fidelity and reduces unwanted foreground regeneration across multiple datasets, while remaining competitive with dataset-driven methods. Its inference cost is comparable to or lower than some alternatives, with additional BRSA computation incurred during adaptation.

C.2 Quantitative Analysis

  • EraseLoRA consistently surpasses dataset-free approaches across OpenImages V7, RORD, and RemovalBench while remaining competitive with dataset-driven methods.The extended results report improved background fidelity and less unwanted foreground regeneration without noise while preserving the background.Source: §C.2 · p. 26, Table 5 · p. 27, Table 6 · p. 27
  • The extended quantitative comparisons cover OpenImages V7 and RORD in Table 5, and paired RemovalBench results plus inference-time cost in Table 6.Source: Table 5 · p. 27, Table 6 · p. 27
  • At inference, merged LoRA weights add no extra cost; EraseLoRA has comparable or lower overhead than some methods despite extra BRSA computation during adaptation.The authors characterize the quality gains as making the trade-off advantageous, while noting dataset-driven models may have lower inference cost.Source: §C.2 · p. 27, Table 6 · p. 27

D.1 Flexibility

EraseLoRA is evaluated across diffusion backbones, MLLMs, and Tag2Mask models, with consistent improvements across these choices. The section also examines adaptation capacity and discusses multiple-object removal, effect handling, and dependence on accurate background cues.

D.1 Flexibility

  • Across SD1.5, SDXL, SD3.5-M, and FLUX.1, EraseLoRA raises BG similarity by at least 17.8% and decreases FG similarity by 28.8%.Qualitative results also show foreground-free restoration across SD1.5, FLUX.1, and SD3.5-M.Source: §D.1 · p. 28, Table 7 · p. 28, Figure 15 · p. 28
  • Across MLLMs, background-aware reasoning and accurate subtype cues matter more for removal quality than the tested hallucination-mitigation model alone.Qwen2.5-VL-72B and InternVL3-78B provide precise subtype cues, whereas MARINE is reported to fail because of inaccurate cues.Source: §D.1 · p. 29, Figure 16 · p. 29
  • Across Tag2Mask variants, EraseLoRA gains at least 10.0% in BG Sim. and reduces FG Sim. by 28.3% over the SD3.5-M baseline.Grounded SAM2 performs best, improving BG Sim. by up to 22.8% and reducing FG Sim. by up to 47.2%.Source: §D.1 · p. 30, Figure 17 · p. 30

D.2 Adaptation Capacity

  • Rank 32 gives the best trade-off for foreground suppression and consistent reconstruction; larger ranks offer no meaningful gains and cost more to optimize.The tested ranks were 16, 32, 64, and 128.Source: §D.2 · p. 30, Figure 18 · p. 31
  • The method adopts 500 adaptation iterations as a practical balance because longer optimization yields diminishing gains relative to added time.Source: §D.2 · p. 31, Figure 18 · p. 31

D.3 Discussion

  • Multiple masked objects can be removed jointly because adaptation operates per background rather than per instance, without framework modification.Source: §D.3 · p. 31
  • EraseLoRA edits only the masked area to preserve unrelated regions; object-induced effects outside it may require mask expansion or interactive control.The method does not explicitly handle effects such as shadows or reflections outside the mask, and inaccurate subtype classifications can also leave residual object traces.Source: §D.3 · pp. 31–32, Figure 20 · p. 32

E.1 Limitations

The discussion identifies mask coverage and computational cost as practical boundaries, and explores reusing background-specific adaptation across video frames. A single-frame adaptation produces outputs comparable to per-frame optimization when frames share similar background context.

E.1 Limitations

  • Physical effects outside the mask, including shadows, lighting distortion, and reflections, may leave subtle traces unless the mask includes them.An interactive mask encompassing both the object and its effects is presented as a practical solution.Source: §E.1 · p. 32, Figure 21 · p. 33
  • MLLM reasoning and LoRA test-time optimization add computation, with optimization cost scaling by the number of distinct background contexts.Using LLaVA-7B instead of InternVL3-78B reduces parameter scale by approximately 11× while maintaining over 98.0% of BG Sim. and 91.4% of FG Sim.; early stopping reduces average optimization from 500 to approximately 140 iterations while preserving more than 96.5% of final BG Sim.Source: §E.1 · pp. 32–34

E.2 Future Works

  • For video frames sharing similar backgrounds, adapting one representative frame and reusing the model yields outputs comparable to adapting every frame.This reuse reduces adaptation cost by the number of frames, as described for the video extension.Source: Figure 21 · p. 33, §E.2 · p. 34

F Additional qualitative results

Additional qualitative comparisons report artifact-free background textures and removal without semantic traces for EraseLoRA, contrasting with foreground-like patterns or overly smooth textures from prior dataset-free methods. The figures extend comparisons across OpenImages V7, RORD, and RemovalBench.

F Additional qualitative results

  • EraseLoRA removes targets without semantic traces and reconstructs artifact-free textures, while prior dataset-free methods tend to produce foreground-like patterns or overly smooth textures.Source: F Additional qualitative results · p. 34
  • Figure 22 presents additional qualitative comparisons on OpenImages V7.The comparison includes dataset-driven and dataset-free methods.Source: Figure 22 · p. 35
  • Figure 23 presents additional qualitative comparisons on RORD and RemovalBench.The comparison includes dataset-driven and dataset-free methods.Source: Figure 23 · p. 36

Related papers

Loading 2512.21545v2…