Source-linked AI summary
EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing
Yang Fu, Yike Zheng, Ziyun Dai, Henghui Ding
TL;DR
Video object removal must erase target objects and their visual effects, but existing methods and datasets inadequately capture these effects across dynamic scenes. The paper introduces VOR and EffectErase, which jointly learns removal and inverse insertion with region guidance and consistency objectives. EffectErase achieves state-of-the-art performance across datasets and metrics, producing high-quality removal of objects and effects in complex scenes.
Problem
Existing video object removal methods struggle with complex visual effects, while available datasets lack large-scale paired videos covering such effects across varied scenes.
Method
VOR provides hybrid captured and synthesized paired videos, and EffectErase jointly learns removal and inverse insertion using Task-Aware Region Guidance and Effect Consistency loss.
Results
EffectErase achieves state-of-the-art performance across datasets and evaluation metrics, including best FVD, QScore, and user feedback ratings.
Takeaways & Limitations
VOR and EffectErase support effect-aware video object removal across diverse and complex scenes.
Takeaways & Limitations
Existing methods still lack explicit spatiotemporal modeling of object–effect correlations, limiting robustness and stable localization in complex real-world scenes.
Abstract
from arXiv · showhide
Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds. Recent diffusion-based video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds. Beyond method limitations, progress is further hampered by the lack of a comprehensive dataset that systematically captures common object effects across varied environments for training and evaluation. To address this, we introduce VOR (Video Object Removal), a large-scale dataset that provides diverse paired videos, each consisting of one video where the target object is present with its effects and a counterpart where the object and effects are absent, with corresponding object masks. VOR contains 60K high-quality video pairs from captured and synthetic sources, covers five effects types, and spans a wide range of object categories as well as complex, dynamic multi-object scenes. Building on VOR, we propose EffectErase, an effect-aware video object removal method that treats video object insertion as the inverse auxiliary task within a reciprocal learning scheme. The model includes task-aware region guidance that focuses learning on affected areas and enables flexible task switching. Then, an insertion-removal consistency objective that encourages complementary behaviors and shared localization of effect regions and structural cues. Trained on VOR, EffectErase achieves superior performance in extensive experiments, delivering high-quality video object effect erasing across diverse scenarios.
1. Introduction
Video object removal methods often miss object-induced effects, while progress is also constrained by limited video datasets. The paper introduces VOR and EffectErase, combining a diverse benchmark with joint removal–insertion learning for effect-aware erasure.
- Existing methods can remove masked object bodies but often overlook side effects such as reflections.
- Reliance on input masks overlooks object-induced effects and limits precise localization of affected regions in complex scenes.
- Image-level datasets cannot teach temporal consistency, while prior video datasets omit effects or simplify object motion.
- VOR combines camera-captured and synthesized videos with diverse objects, scenes, effects, and dynamic multi-object settings.The dataset includes 293 captured scenes and over 150 synthesized 3D scenes.
- EffectErase jointly learns removal and insertion using task-aware region guidance and consistency between shared effect regions and structural features.The framework uses cross-attention to model object–effect spatiotemporal correlations and a task token for switching tasks.
- The work establishes VOR-Eval and VOR-Wild and reports state-of-the-art quantitative and visual performance for EffectErase.
2. Related Work
Related work progresses from image and video inpainting toward object removal with explicit visual-effect modeling. Existing approaches still face limitations in realistic side effects, dataset scale, and temporal consistency.
- Video inpainting methods use convolutional spatiotemporal modeling, optical flow, and text guidance to reconstruct masked regions.ProPainter uses recurrent flow completion to improve controllability and temporal consistency.
- Object removal requires modeling object-induced effects, but image datasets are often limited in scale or fail to reproduce realistic shadows and reflections.
- Video object removal additionally requires temporal consistency, while Minmax-Remover implicitly models effects and lacks a large, high-quality dataset.
3. Methodology
VOR combines captured and synthesized videos to represent diverse object-induced effects, while EffectErase jointly learns removal and insertion with region-aware guidance and consistency supervision.
- VOR Dataset: VOR combines camera-captured videos for physical realism with 3D-synthesized videos modeling dynamic cameras and multi-object interactions.The dataset includes diverse scenes, object categories, motions, and weather conditions, with synthesized data adding controlled dynamic interactions.
- VOR Dataset: VOR characterizes five representative object-induced effects, including occlusion, shadow, lighting, and reflection.Occlusion is subdivided into opaque, semi-transparent, and transparent cases.
- VOR Dataset: Validated object masks are propagated across sequences with SAM2 and manually refined before forming triplet training data.The pipeline uses point prompts on key frames, segmentation verification, sequence propagation, and manual cleaning.
- EffectErase: EffectErase jointly trains removal and insertion through a shared denoising backbone, using complementary supervision for affected regions and structural cues.Removal uses the background latent, while insertion uses the object-present latent within the shared diffusion-training framework.
- EffectErase: Task-Aware Region Guidance injects task and foreground tokens through cross-attention to localize effect-related regions and switch between removal and insertion.A lightweight adaptor first fuses noisy latents with task conditions, and projected foreground embeddings replace the object placeholder token.
- EffectErase: Effect Consistency aligns removal and insertion attention maps because the inverse tasks share affected regions.The loss uses cross-attention maps from both branches to encourage consistent localization of object and induced environmental effects.
4. Experiments
Experiments compare EffectErase with image inpainting, video inpainting, and video object removal methods across multiple datasets and metrics. Results show stronger temporal and visual quality, effective effect removal, and complementary benefits from consistency loss, region guidance, synthesized data, and insertion adaptation.
- Comparison with State-of-the-Art Methods: EffectErase is evaluated against image inpainting, video inpainting, and video object removal methods on ROSE-Benchmark, VOR-Eval, and VOR-Wild.Ground-truth datasets use PSNR, SSIM, LPIPS, and FVD; VOR-Wild uses user ratings and QScore.
- Comparison with State-of-the-Art Methods: EffectErase achieves state-of-the-art performance across all datasets and evaluation metrics, including the best FVD, QScore, and user feedback ratings.The reported FVD result indicates superior temporal smoothness and consistency, while QScore and ratings assess visually convincing removal.
- Comparison with State-of-the-Art Methods: EffectErase removes target objects and associated effects more cleanly than competing methods in occlusion, shadow, lighting, reflection, and deformation scenarios.Qualitative comparisons report artifacts or incomplete side-effect removal for competing inpainting and object-removal approaches.
- Ablation Studies: Adding EC loss improves all metrics, with FVD decreasing from 368.664 to 354.545.The consistency loss encourages removal and insertion to focus on the same side-effect regions.
- Ablation Studies: Adding TARG improves SSIM from 0.737 to 0.780 by capturing spatiotemporal correlations and localizing affected regions.The ablation validates the module’s contribution to higher-quality erasure results.
- Ablation Studies: Training with real and synthesized data reduces LPIPS from 0.193 to 0.170 and improves VOR-Eval generalization.The mixed setup exposes the model to broader appearance variations and motion patterns, producing cleaner backgrounds and more stable temporal restoration.
- Object Insertion: EffectErase adapts to object insertion by changing the task prompt without additional training, generating realistic shadows and light reflections.Examples include inserted leaves and traffic cones with shadows, plus reflections on glossy ceramic tiles.
5. Conclusion
The conclusion presents VOR and EffectErase as a framework for effect-aware video object removal and reports strong performance in complex scenes. It also identifies mask dependence as a limitation and points to text- or speech-based interaction as future work.
- VOR combines camera-captured and synthesized videos covering object-induced effects, with VOR-Eval and VOR-Wild as evaluation benchmarks.
- EffectErase jointly learns video object removal and insertion using Task-Aware Region Guidance and Effect Consistency loss.
- Extensive experiments and ablations validate the components, while EffectErase achieves state-of-the-art high-quality removal in complex scenes and naturally extends to object insertion.
- EffectErase requires an input mask to specify the removal region, motivating future support for text and speech interactions.
Supplementary Material for EffectErase
The supplement describes the captured portion of VOR, emphasizing paired recordings, broad scene and object coverage, and controlled camera-motion augmentation. These design choices support training data with consistent targets and varied visual conditions.
- The supplement provides additional dataset details, method descriptions, and qualitative results for EffectErase.
- Captured Data: Each VOR pair contains a video with the target object and effects and a counterpart without them, recorded with locked exposure, focus, duration, triggering, and camera position.
- Captured Data: Captured data spans 293 scenes and more than 45 scene categories, with objects ranging from sports balls and tools to children, teenagers, and vehicles.
- Captured Data: The extended Ken Burns augmentation provides fourteen camera-motion patterns, including zoom, pan, tilt, combined motion, walk-bob, and random combinations.
A.2. Synthesized Data
The synthesized-data section constructs varied 3D environments, animated objects, and camera trajectories for video-pair generation. It also describes capture controls and mask propagation used to maintain consistent training and evaluation data.
- Synthesized Environments: The synthesized environments include indoor and outdoor settings such as streets, farms, coastal areas, mountains, parking lots, classrooms, and forests.
- Data Capture: The capture app locks exposure and focus, matches file names and durations, enables Bluetooth triggering, and uses a tripod to remove camera shake.
- Synthesized Environments: Animated content includes dancing humans, walking bears, moving boats, and flying balloons across realistic, anime, and game-style domains.
- Camera Trajectories: Manually designed camera paths and natural zoom and pan behaviors aim to mimic human-operated filming practices.
- Mask Processing: Masks are initialized with a verified first-frame point prompt, propagated with SAM2, and reviewed to remove sequences lacking stable, complete coverage.
A.4. Dataset Statics
The dataset comparison emphasizes VOR’s broader scene and object coverage, longer videos, and substantially more paired sequences than prior datasets. The section also describes standardized scene categorization and implementation details for evaluation and model adaptation.
- VOR offers richer scene diversity, broader object coverage, longer video durations, and more paired sequences than existing removal datasets.
- The scene taxonomy organizes indoor and outdoor environments into 67 categories for standardized dataset comparison.
- The model uses a 3D convolutional adaptor with 1×2×2 kernels and strides, followed by a projector built from two sequential MLP blocks.
- A mapper aggregates cross-attention maps across DiT layers and predicts an effect-area distribution map for each frame.
- During inference, the model supports both removal and insertion, using video and masks for removal or background and object videos for insertion.
B.4. Metric details
Removal quality is evaluated with both an automated QScore and a user study. QScore combines removal completeness and visual artifacts, while human ratings average judgments from 20 volunteers.
- QScore averages Qwen-VL assessments of removal completeness and visual artifacts for each removed video.
- 20 volunteers each score 195 generated videos from VOR-Wild, with the final user-study score averaging ratings across participants.
C.1. Effect-region Erasing Evaluation
The evaluation focuses on whether EffectErase removes visual effects outside the object mask, while additional results cover insertion, in-the-wild removal, and qualitative comparisons with state-of-the-art methods.
- EffectErase evaluates effect-region erasing outside the object mask, computing metrics only over the corresponding effect regions.
- Additional figures report EffectErase results for the insertion task and in-the-wild data.
- Further qualitative comparisons assess EffectErase against state-of-the-art methods.
C.5. Failure Cases and Analysis
The reported failures occur when visual effects or accessories cannot be confidently assigned to the target object. Examples involve ambiguous lighting, entangled shadows, and unclear leash ownership.
- Failure cases mainly arise when effects or accessories are ambiguously associated with the target object.
- Residual lighting may originate from other light sources, although it can remain visually natural after removal.
- Dog and person shadows can become heavily entangled, making the dog’s shadow difficult to separate reliably.
- A leash may be impossible to assign clearly to either the dog or the person.