PaperlayerView exact revision on arXiv

Source-linked AI summary

EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal

Feier Wu, Wanke Xia, Xu He, Zilang Zhou, Si Chen, Dongxia Liu, Liyang Chen, Qimeng Wu, Zhengbo Zhang, Wenming Yang, Zhiyong Wu

arXiv:2608.05565v1cs.CV

TL;DR

Video object removal must remove induced effects while preserving visual fidelity and temporal coherence, but existing methods have limited coverage of complex real-world interactions. EffectLearner combines semantic object-effect reasoning, motion-aware restoration, and the EffectWorld curriculum, achieving strong reconstruction quality, perceptual fidelity, temporal coherence, and open-world generalization.

  • Problem

    Existing video object removal methods provide limited coverage of complex object-induced effects, including compositional, detached, long-tail, and dynamically evolving interactions.

  • Method

    EffectLearner uses a VLM-based Object-Effect Reasoner with target-highlighted videos to guide a DiT-based Video Eraser, supported by motion-aware stabilization and EffectWorld curriculum training.

  • Results

    EffectLearner demonstrates strong reconstruction quality, perceptual fidelity, temporal coherence, and open-world generalization across ROSE-Bench, EffectWorld-Eval, and EffectWorld-Wild.

  • Takeaways & Limitations

    Semantic reasoning and motion-aware stabilization support reliable removal of objects and their induced effects in complex videos.

Abstract

from arXiv · show

Video object removal must eliminate not only the target object but also its induced effects while maintaining high-fidelity and spatiotemporally coherent restoration. Existing methods mainly learn object-effect correspondences implicitly from predefined effect categories and fixed data distributions, limiting their generalization to complex real-world scenes involving compositional effects, spatially detached or weakly correlated effects, long-tail physical phenomena, and dynamically evolving interactions. We propose EffectLearner, a semantic-reasoning-enhanced framework that combines a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser. Guided by a structured effect-analysis prompt, the Reasoner performs cross-modal reasoning over a target-highlighted video and extracts compact effect-aware context, which guides the Video Eraser toward comprehensive object-effect removal. Motion-aware mask guidance and motion-consistency supervision further improve removal coverage and spatiotemporal stability under object motion and evolving scene dynamics. To fully exploit the framework in challenging real-world scenarios, we further construct EffectWorld, a paired video dataset specifically designed for complex object-induced effects, and introduce a progressive training curriculum that combines common supervision with complex-effect data. On the standard ROSE-Bench, EffectLearner outperforms existing baselines on most metrics and achieves clear advantages on both EffectWorld-Eval and the challenging EffectWorld-Wild, demonstrating its ability to deliver high-quality video object removal in complex real-world scenes.

1 Introduction

The introduction frames video object removal as requiring elimination of object-induced effects alongside the target, with complete and spatiotemporally consistent restoration in dynamic real-world scenes. It presents EffectLearner, EffectWorld, motion-aware mechanisms, and progressive training as solutions for semantic effect reasoning and complex-scene generalization.

  • Motivation: High-quality video object removal must eliminate target objects and their induced effects while preserving visual fidelity and spatiotemporal coherence across evolving scenes.Object-induced effects may extend beyond the target region and change with object motion and scene dynamics.
  • EffectLearner: EffectLearner integrates a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser to guide high-fidelity, generalizable removal in complex real-world videos.A structured effect-analysis prompt guides the Reasoner in extracting effect-aware context for semantic video restoration.
  • EffectLearner: Motion-aware mask guidance and motion-consistency loss improve removal coverage and temporal stability under object motion and scene dynamics.The mechanisms target missed removals, residual traces, and temporal flickering by preserving mask coverage and encouraging consistent neighboring-frame restoration.
  • EffectWorld: EffectWorld is a physics-based object-effect video dataset covering compositional, dynamic, spatially detached, and long-tail physical effects.The dataset is constructed from carefully designed 3D scenes rendered in Unreal Engine to address limited coverage in existing datasets.
  • Training and Evaluation: A progressive training curriculum combines conventional data with EffectWorld data to support transition from basic removal learning to complex-scene generalization.Experiments on standard benchmarks and challenging complex-effect settings demonstrate state-of-the-art removal quality, robustness, and spatiotemporal stability.

2 Related Work

Prior video object removal methods primarily use temporal video inpainting to reconstruct explicitly masked regions, often overlooking additional effects. Recent VLM-guided video editing methods improve controllability through multimodal reasoning but mainly address explicit instructions or object-level manipulation.

  • Video Object Removal: Video object removal methods aim to eliminate undesired objects while preserving visual fidelity and temporal consistency, with early propagation-based approaches exploiting temporal correspondence for reconstruction.These approaches include methods by Zhou et al. (2023), Zeng, Fu, and Chao (2020), Liu et al. (2021), and Li et al. (2022).
  • VLM Guidance in Video Editing: Recent VLM-based video editing methods use multimodal reasoning for controllable generation, covering instruction-based editing, reference-guided manipulation, object interaction deletion, and reward-based alignment.RACCOON, Kiwi-Edit, Void, and Viva represent these respective directions.

3 Method

EffectLearner combines a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser to remove target objects and their induced effects while reconstructing affected regions faithfully. It uses target-highlighted inputs, structured effect-analysis prompts, effect-aware conditioning, region-weighted training, and motion-aware designs for dynamic scenes.

  • Dynamic-Scene Consistency: EffectLearner adds motion-aware spatiotemporal consistency mechanisms to address moving targets, evolving effects, temporal mask compression, incomplete removal, residual traces, and flicker.The framework is designed to promote stable restoration in dynamic scenes where target locations and induced effects vary across neighboring frames.
  • Framework Overview: EffectLearner couples a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser for joint object-effect removal and faithful affected-region reconstruction.The Reasoner analyzes object-scene interactions and provides effect-aware semantic context to guide the Eraser.
  • Object-Effect Reasoner: The Reasoner uses a target-highlighted video and structured prompts to analyze target appearance, motion, scene interactions, induced effects, temporal evolution, and preserved content.Alpha blending makes the target location explicit while retaining visual evidence, and learnable effect queries compress VLM hidden states into compact conditions.
  • Video Eraser: The Video Eraser combines source latent and downsampled object-mask conditions with effect-aware context tokens that guide DiT cross-attention.The source latent preserves appearance, texture, structure, and temporal information, while the semantic context specifies effects, the post-removal state, and content to preserve.
  • Effect-Aware Supervision: A region-weighted flow-matching objective emphasizes both the explicitly specified object region and broader regions affected by induced effects.The joint object-effect mask is derived from absolute differences between paired source and target videos, with separate weights for object and effect regions.

4 Experiments

Experiments evaluate EffectLearner on standard and complex-effect benchmarks, showing stronger video-level coherence and more complete open-world removal than competing methods. Ablations further show that semantic reasoning, region-weighted flow matching, and motion-consistent loss address complementary failure modes.

  • Experimental Setup: Experiments use ROSE-Bench, EffectWorld-Eval, and EffectWorld-Wild to assess fidelity, temporal coherence, and open-world video removal.Training uses 16,663 ROSE triplets and 11,048 EffectWorld triplets; the benchmarks contain 60, 33, and 70 videos or cases, respectively.
  • Quantitative Results: EffectLearner achieves competitive reconstruction quality, the best LPIPS, and substantially lower FVD on ROSE-Bench, improving video-level coherence.ROSE performs strongly on pixel-level metrics but has high FVD, whereas category-specific supervision limits its coherence and generalization.
  • Qualitative Results: Traditional inpainting leaves spatially extended or detached effects and visible artifacts, while EffectLearner better preserves surrounding content and natural scene dynamics.EffectLearner’s restored regions show fewer artifacts and better integration with adjacent content in challenging EffectWorld-Wild videos.
  • Ablation Study: Removing VLM guidance degrades reconstruction and video-level metrics because object masks cannot identify weakly correlated or spatially detached effects.Effect-aware semantic context helps the DiT distinguish target-induced changes to remove from content to preserve.
  • Ablation Study: The largest deterioration occurs without motion-consistent loss, particularly in LPIPS and FVD, showing that temporal residual matching suppresses flicker and preserves cross-frame consistency.Without region-weighted flow matching, background regions dominate optimization; VAE compression can also cause incomplete removal or residual traces for fast-moving objects.

5 Conclusion

EffectLearner combines semantic reasoning and motion-aware stabilization to remove target objects and their induced effects from complex videos. Experiments on ROSE-Bench, EffectWorld-Eval, and EffectWorld-Wild support its reconstruction quality, perceptual fidelity, temporal coherence, and open-world generalization.

  • 5 Conclusion: EffectLearner uses a VLM-based Object-Effect Reasoner and DiT-based Video Eraser for semantic-reasoning-enhanced object-and-effect removal.The Reasoner converts target-highlighted visual evidence and a structured effect-analysis prompt into compact effect-aware context that guides the Video Eraser.
  • 5 Conclusion: Experiments on ROSE-Bench, EffectWorld-Eval, and EffectWorld-Wild demonstrate reconstruction quality, perceptual fidelity, temporal coherence, and open-world generalization.The conclusion identifies semantic reasoning and motion-aware stabilization as complementary components for reliable video object-and-effect removal.

Technical Supplement of EffectLearner: World-Aware Object-Effect … A.3 EffectWorld-Wild

The supplement documents EffectLearner’s benchmark suite and EffectWorld construction, including paired and open-world evaluation settings for complex object-induced effects. It also details Wild-video sourcing, diagnostic annotations, category labels, and semi-automatic mask verification.

  • Removal: The supplement covers benchmark details, training data, EffectWorld construction, implementation, Wild evaluation, failure analysis, and ethics, with visual demonstrations provided in an attached webpage archive.The archive is accessed through index.html after unzipping the Media Supplement.
  • A.1 ROSE-Bench: ROSE-Bench contains 60 paired source-mask-target video triplets evaluated over 49 frames and spanning shadow, light source, reflection, mirror, and translucent effects.It evaluates object-removal accuracy, effect handling, and generalization to real-world scenes.
  • A.2 EffectWorld-Eval: EffectWorld-Eval contains 33 paired complex-effect videos, including 29 compositional-effect cases, eight spatially detached or weakly correlated relations, and motion in all samples.The benchmark retains ROSE’s five non-exclusive effect labels for category-level analysis.
  • A.3 EffectWorld-Wild: EffectWorld-Wild contains 70 open-world source-mask pairs from 59 curated real-world videos and 11 DAVIS videos, without object-removed targets for no-reference and subjective evaluation.Every sample includes a manually verified target-object mask.
  • A.3 EffectWorld-Wild: The 59 non-DAVIS Wild videos were collected from Pexels and screened for identifiable targets, sufficient temporal visibility, and observable target-surroundings interactions.Diagnostic annotations include 12 physical-trace cases and six fastmotion cases.
  • A.3 EffectWorld-Wild: Wild-video category analysis retains ROSE’s five non-exclusive labels, yielding 41 shadow, 11 mirror, eight reflection, seven light, and five translucent cases.These counts describe the retained effect-label distribution in EffectWorld-Wild.
  • A.3 EffectWorld-Wild: Target masks are obtained by zero-shot category-prompt detection, SAM2 propagation, manual overlay and contact-sheet inspection, and corrective reruns using manually specified box prompts.Corrections address tracking errors, missing object parts, and background leakage.

B Training Data Details … E.2 LLM-as-a-Judge Evaluation

EffectLearner is trained with paired EffectWorld and ROSE data, uses structured effect reasoning and motion-aware video-erasing mechanisms, and is evaluated with fine-grained human/LLM criteria. On GPT-5.4 judging, it ranks first overall and leads target removal, effect removal, and temporal rendering.

  • B Training Data Details: 11,048 EffectWorld triplets and 16,663 ROSE triplets form a 27,711-sample training manifest for complex-effect supervision.The complete EffectWorld UE collection contains 11,092 valid paired triplets, but the formal manifest uses 11,048.
  • C EffectWorld Construction Details: The construction pipeline identifies rose-tagged targets, creates 20 randomized sequences per target, and renders aligned source, target, and mask outputs.Sequences use target and camera transform tracks in a shared LevelSequence rendered at 15 fps.
  • C EffectWorld Construction Details: EffectWorld generates paired source, object-removed target, and mask sequences, covering compositional effects, long-tail phenomena, and weak object–effect correlations.The standardized dataset contains 11,092 valid triplets, including 3,428 90-frame sequences and 7,664 150-frame sequences.
  • D Framework Implementation Details: The framework supplies a structured effect-analysis prompt to its VLM-based reasoner for analyzing target-highlighted videos and producing effect-aware guidance.The prompt includes analysis-role, general effect-analysis, and sample-level removal instructions.
  • D.2 Mask Construction and Objectives: Frame masks are resized and merged across latent-time windows into the four-channel Munion condition, while dilation produces Mloss for loss weighting.The effect region combines thresholded latent differences with dilated supervision masks.
  • D.2 Mask Construction and Objectives: Motion-consistency supervision compares motion-compensated temporal residuals within dilated effect and adjacent object regions, with weight 0.20.Centroid displacement between adjacent masks provides the translational pseudo-flow.
  • D.3 Target-Highlighted Video Input: Target-highlighted inputs blend masked pixels with red and mark mask boundaries yellow, while temporal mask union aligns frame-level masks to latent positions.The highlight blending coefficient is α = 0.15, and nearest-neighbor resizing is used for latent mask alignment.
  • E.1 Fine-grained Object Removal Evaluation Protocol: The evaluation protocol scores Target Removal, Effect Removal, Background Preservation, and temporal rendering across complete anonymized candidate sequences.Effect Removal includes shadows, reflections, illumination changes, ripples, smoke, and motion traces beyond the object mask.

E.3 Human Evaluation

Human volunteers evaluated EffectWorld-Wild videos using averaged 1–4 ratings across each evaluation dimension. EffectLearner achieved the highest overall human-evaluation score, leading in effect removal and background preservation while tying for target removal and temporal rendering.

  • Evaluation protocol: 20 volunteers rated each method’s EffectWorld-Wild videos on every evaluation dimension using integer scores from 1 to 4.Ratings were averaged to produce each method’s human-evaluation score.
  • Human-evaluation results: EffectLearner achieved the highest overall human-evaluation score on EffectWorld-Wild.The evaluation used averaged volunteer ratings across the evaluation dimensions.
  • Human-evaluation results: EffectLearner ranked first in effect removal and background preservation, while tying for best performance in target removal and temporal rendering.The comparison was conducted against various methods on EffectWorld-Wild.

F Failure Cases and Analysis · G Ethical Statement

EffectLearner exhibits failures under large foreground occlusion, severe motion blur, and repeated occlusion, while the paper states data-provenance and licensing safeguards alongside acknowledging potential misuse. The section also documents structured prompting and fine-grained evaluation materials.

  • F Failure Cases and Analysis: Large foreground occlusion over textured backgrounds can produce over-smoothed textures and local distortions near unremoved objects.The cat example combines target removal with occlusion of a textured sofa and part of another cat.
  • F Failure Cases and Analysis: Severe motion blur and repeated occlusion can cause incorrect removal behavior in the dog example.The reported dog failure involves repeated occlusion and severe motion blur; the supplied passage ends before specifying the exact incorrect behavior.
  • G Ethical Statement: EffectWorld is rendered in Unreal Engine without captured footage or biometric information from real individuals.Public ROSE data, DAVIS, and free-to-use Pexels videos are used under their respective terms or licenses for evaluation.
  • G Ethical Statement: Video object removal supports benign applications such as editing and post-production, but the paper acknowledges that it may be misused.The supplied ethical statement ends after this acknowledgment and does not specify misuse scenarios.
  • G Ethical Statement: The structured analysis prompt asks the assistant to identify the target’s appearance, position, motion, visible interval, and causally produced environmental effects.Its stated role is fine-grained analysis of video objects and their effects on surrounding environments.
  • G Ethical Statement: The evaluation protocol uses a semi-transparent reference mask and anonymous candidate videos A-F from different methods.The protocol evaluates mask-guided video object removal results.
  • G Ethical Statement: Figure VI provides the complete prompt for an LLM judge in the fine-grained object-removal evaluation protocol.The supplied passage identifies the figure’s purpose but does not describe the judge’s criteria.
  • F Failure Cases and Analysis: Representative failures arise from large foreground occlusion, fast motion, and repeated occlusion by thin structures, producing local structural artifacts.Figure VII compares original inputs with EffectLearner removal results.
Loading 2608.05565v1…