PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data

Runpu Wei, Zijin Yin, Shuo Zhang, Lanxiang Zhou, Xueyi Wang, Chao Ban, Tianwei Cao, Hao Sun, Zhongjiang He, Kongming Liang, Zhanyu Ma

arXiv:2501.07397v3cs.CV

TL;DR

Object removal still faces missing realistic supervision for shadows and reflections and can produce unintended objects or artifacts. OmniEraser combines video-frame data with separate object and background guidance, and the authors report that it significantly outperforms previous methods.

  • Problem

    Existing object-removal data often lacks realistic shadows and reflections, and background-only guidance can produce unintended objects or artifacts.

    Source: §1 · p. 2

  • Method

    OmniEraser trains on video-frame-derived data and uses foreground-object and background images as independent diffusion guidance signals.

    Source: §3.1 · p. 3, §5 · p. 9

  • Results

    OmniEraser significantly outperforms previous methods in experiments and a user study.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that OmniEraser effectively removes objects and their visual effects using object masks.

    Source: §5 · p. 9

    OmniEraser may alter unrelated elements, such as other objects’ shadows or background color, because it omits latent blending to encourage removal beyond the mask.

    Source: §A.2 · p. 12

Abstract

from arXiv · show

Inpainting algorithms have achieved remarkable progress in removing objects from images, yet still face two challenges: 1) struggle to handle the object's visual effects such as shadow and reflection; 2) easily generate shape-like artifacts and unintended content. In this paper, we propose Video4Removal, a large-scale dataset comprising over 100,000 high-quality samples with realistic object shadows and reflections. By constructing object-background pairs from video frames with off-the-shelf vision models, the labor costs of data acquisition can be significantly reduced. To avoid generating shape-like artifacts and unintended content, we propose Object-Background Guidance, an elaborated paradigm that takes both the foreground object and background images. It can guide the diffusion process to harness richer contextual information. Based on the above two designs, we present OmniEraser, a novel method that seamlessly removes objects and their visual effects using only object masks as input. Extensive experiments show that OmniEraser significantly outperforms previous methods, particularly in complex in-the-wild scenes. And it also exhibits a strong generalization ability in anime-style images. Datasets, models, and codes will be published.

1. Introduction

Object removal lacks realistic training pairs that capture shadows and reflections, while existing methods can also generate unintended objects and artifacts. OmniEraser combines video-derived data with Object-Background Guidance and is reported to outperform previous methods.

1. Introduction

  • Synthetic training data often lacks physical effects such as shadows and reflections, while realistic alternatives can be laborious to annotate or limited in scale and availability.RORD includes shadow regions in its masks, and ObjectDrop is limited in scale and is not publicly available.Source: §1 · p. 2
  • Methods guided only by background context can generate new objects or artifacts because they lack explicit awareness of the target region.The paper gives adding a new bowl as an example.Source: §1 · p. 2
  • Video4Removal contains 134,281 high-quality realistic images created as triplet data from video frames.The dataset is presented as a large-scale source of realistic training data.Source: §1 · p. 2
  • Object-Background Guidance uses foreground-object and background images as separate diffusion guidance signals to improve context and prevent unintended objects.Video4Removal masks exclude effect regions, encouraging models to learn object-effect associations end to end.Source: Figure 2 · p. 2, §1 · p. 2
  • OmniEraser removes objects and their visual effects using object masks, and experiments report performance superiority over previous methods.The paper also introduces RemovalBench for evaluation.Source: §1 · pp. 2–3

2. Related work

Prior work spans general inpainting, object removal, and scene-layer decomposition. The paper contrasts its realistic, scalable paired data and direct background generation with limitations in existing removal datasets and layer-based approaches.

Image Inpainting

  • GAN-based inpainting methods are often trained for specific domains, while diffusion inpainting methods can generate uncertain content and blurred textures.Blended Latent Diffusion, SmartBrush, and SD-Inpaint concatenate masks and source images; BrushNet adds a trainable branch and blending.Source: §2 · p. 3

Object Removal

  • Object-removal methods commonly fine-tune diffusion models on triplet data, but many datasets lack realistic shadows and reflections.RORD masks include object effects, while ObjectDrop is limited in size, image quality, and accessibility; the paper's paired data is described as large-scale and scalable.Source: §2 · p. 3

Layer Decomposition

  • Layer-decomposition work models backgrounds, objects, and effects in separate layers, whereas this paper trains directly to generate a background image.The cited approaches include Omnimatte [27], MULAN [46], Alfie [32], LayerDiffusion [55], and LayerDecomp [50].Source: §2 · p. 3

3. Method

The method combines Video4Removal, a video-derived triplet dataset, with OmniEraser’s Object-Background Guidance to remove objects and their visual effects. The section describes the dataset construction and model design, including mask augmentation for robustness.

3.1. Dataset Construction

  • The dataset pipeline separates moving-object foreground frames from background frames, pairs each foreground with a corresponding background, and generates masks for the foreground objects.The construction figure describes pairing with the temporally closest background frame; the method uses MOG for frame separation and filters instances with motion-blurred objects.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • Video4Removal contains 134,281 photorealistic triplets at 1920×1080 resolution, constructed from fixed-viewpoint real-world video frames to reduce collection labor.Foreground frames are paired with corresponding background frames, and object masks complete each triplet.Source: §3.1 · pp. 3–4

3.2. Model Design

  • Object-Background Guidance uses foreground-object and background latents as joint denoising conditions to provide target-region and background context.The latents, noise, and mask are concatenated and passed through a trainable linear layer.Source: Figure 4 · p. 4, §3.2 · pp. 4–5
  • OmniEraser achieves the best performance across all metrics in the comparison, with SDXL and SD3 variants also delivering competitive results.The table caption does not provide metric values or identify individual comparisons.Source: Table 1 · p. 5
  • Training randomly applies dilation, erosion, box, or no change to input masks to improve robustness to irregular user-provided masks.Source: §3.2 · p. 5

4. Experiments

OmniEraser is evaluated on RemovalBench and RORD-Val, with qualitative tests on in-the-wild and anime-style images. The experiments compare its results with existing methods and examine dataset, guidance, and mask-augmentation choices.

4.1. Setting

  • The experiments use RemovalBench and RORD-Val as primary benchmarks, comparing methods with official weights and default recipes.RemovalBench contains 70 samples; RORD-Val uses 344 selected images from the RORD validation set.Source: §4.1 · pp. 5–6, Figure 5 · p. 5

4.2. Comparison with Existing Methods.

  • On RemovalBench, OmniEraser improves FID from 55.49 to 39.52, LPIPS from 0.146 to 0.133, and CMMD from 0.232 to 0.208; on RORD-Val it achieves 43.71 FID, 0.153 CMMD, 22.13 PSNR, and 4.99 AS.The reported comparisons cover image quality and masked-region preservation across both benchmarks.Source: §4.2 · p. 6
  • OmniEraser is reported to perform consistently better across two mask settings: masks containing the object and its effects, and masks containing only the object.The qualitative comparison is shown in Figure 6.Source: §4.2 · pp. 6–7, Figure 6 · p. 6
  • In collected in-the-wild images with prominent shadows and reflections, OmniEraser removes objects and their effects; it also generalizes to anime-style images despite training solely on real-world image data.The authors report that earlier methods may generate unintended content around shadows or fail to erase object shadows.Source: Figure 7 · p. 6, §4.2 · p. 7, Figure 8 · p. 7
  • In a study with 50 participants judging 55 examples, OmniEraser was preferred by more users for overall image quality and accuracy of object-and-effect removal.Participants chose among results from five models.Source: §4.2 · p. 7, Figure 10 · p. 8

4.3. Ablation Study and Analysis

  • The dataset comparison reports that Video4Removal achieves the best performance on RemovalBench, while the caption identifies Table 2 as comparing training datasets.The authors state that this supports the importance of training on high-quality, real-world data.Source: §4.3 · p. 8, Table 2 · p. 8
  • Object-Background Guidance significantly outperforms background-only and full-image guidance in visual quality and removal accuracy, and the qualitative comparison shows it removing both objects and their shadows.Background-only guidance tends to produce shape-like artifacts, while full-image guidance removes the object but not its shadow.Source: §4.3 · p. 8, Table 3 · p. 8, Figure 9 · p. 8
  • Applying all mask-augmentation strategies yields the best results when tested with incomplete or excessively covered masks.Table 4 is identified as the mask-augmentation ablation.Source: §4.3 · p. 8, Table 4 · p. 8

5. Conclusion

OmniEraser removes objects and their visual effects using object masks, supported by the Video4Removal dataset and Object-Background Guidance. Experiments and a user study report that it significantly outperforms previous methods.

5. Conclusion

  • OmniEraser is presented as removing objects and their visual effects, with experiments and a user study reporting significant gains over previous methods.The paper also introduces the RemovalBench benchmark for evaluation.Source: §5 · p. 9
  • Video4Removal contains over 100,000 high-quality image pairs extracted from video frames.The dataset is presented as supporting object-removal training involving objects and their visual effects.Source: §5 · p. 9
  • Object-Background Guidance conditions the diffusion model on object and background images independently to prevent unintended new objects and artifacts.OmniEraser uses object masks as its input for removal.Source: §5 · p. 9

Supplementary Material

Figure 1 illustrates the mask-augmentation training strategy and reports robustness to irregular masks.

Supplementary Material

  • Figure 1 presents qualitative results for mask-augmentation training and indicates robustness to irregular masks.The supplied supplementary passage also includes the paper's arXiv version and date.Source: Figure 1 · p. 12

A.1. Robustness to Irregular Mask

The mask-augmentation strategy is qualitatively tested with under-covered and over-covered masks. Without augmentation the model can erase irrelevant elements or fail to remove the target, while the proposed method handles these irregular masks.

A.1. Robustness to Irregular Mask

  • Without mask augmentation, the model tends to erase irrelevant elements and fail to remove the target under irregular masks.The tests use under-covered and over-covered masks; all models perform well with normal masks.Source: §A.1 · p. 12
  • With mask augmentation, OmniEraser consistently removes the target even when masks are under-covered or over-covered.The comparison is qualitative and uses a baseline without augmentation.Source: §A.1 · p. 12
  • The authors characterize this irregular-mask robustness as making OmniEraser more user-friendly and adaptable.Source: §A.1 · p. 12

A.2. Limitation of OmniEraser

OmniEraser may alter unrelated elements, including other objects’ shadows or background color, because latent blending is omitted to allow removal of effects outside the mask.

A.2. Limitation of OmniEraser

  • OmniEraser may alter unrelated elements, such as other objects’ shadows or background color, because latent blending is omitted to encourage removal of effects outside the mask.Latent blending would fuse the output latent’s foreground with the original latent’s background.Source: §A.2 · p. 12

A.3. Comparison with Instruction-based Methods.

On Emu-Edit test samples, OmniEraser effectively removes target objects while preserving background integrity; the compared instruction-based approaches struggle with both.

A.3. Comparison with Instruction-based Methods.

  • Instruction-based image inpainting methods can erase objects according to input text.The supplied passage introduces these methods as having this capability.Source: §A.3 · p. 12
  • On Emu-Edit test samples, OmniEraser effectively removes target objects and preserves background integrity, unlike the compared instruction-based approaches.The compared methods are Emu-Edit, Inst-Inpaint, MGIE, and OmniGen.Source: §A.3 · p. 12
  • The comparison presents qualitative results using samples from the Emu-Edit test dataset.The section identifies Emu-Edit, Inst-Inpaint, MGIE, and OmniGen as the instruction-based methods compared.Source: §A.3 · p. 12

B. More Visualization Results.

Additional visual examples span Video4Removal, RemovalBench, challenging in-the-wild scenes, and anime-style images; the authors report object removal, background preservation, robustness, and generalization.

B. More Visualization Results.

  • Additional examples cover Video4Removal, RemovalBench, challenging in-the-wild cases, and diverse anime-style samples, collectively demonstrating object removal, background preservation, robustness, and generalization.The examples appear in Figures 4–11.Source: B. More Visualization Results · p. 13

Related papers

Loading 2501.07397v3…