Source-linked AI summary
VOID: Video Object and Interaction Deletion
Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool, Zhuoning Yuan, Ta-Ying Cheng
TL;DR
Existing video object removal methods struggle when deleting an object should alter downstream physical interactions, rather than only reconstructing appearance. VOID addresses this gap with counterfactual Kubric and HUMOTO data and VLM-guided video diffusion, and evaluations on synthetic and real-world data report better modeling of complex dynamics and broad generalization. Standard perceptual metrics may overlook physically implausible artifacts in dynamic editing settings.
Problem
Existing video object removal methods handle hidden content and photometric effects but struggle to correct downstream physical interactions, producing implausible scene dynamics.
Method
VOID trains on paired counterfactual removals from Kubric and HUMOTO and uses VLM-generated quadmasks to guide video diffusion toward physics-informed outcomes.
Results
VOID outperforms prior inpainting and text-guided video models on synthetic and real-world evaluations and generalizes to scenarios far from its training data.
Takeaways & Limitations
VOID is a starting point for transferring world-modeling capabilities into video editing through high-level causal reasoning.
Takeaways & Limitations
Standard perceptual similarity metrics can miss physically implausible artifacts and incorrect scene dynamics in dynamic video inpainting.
Abstract
from arXiv · showhide
Existing video object removal methods excel at inpainting content "behind" the object and correcting appearance-level artifacts such as shadows and reflections. However, when the removed object has more significant interactions, such as collisions with other objects, current models fail to correct them and produce implausible results. We present VOID, a video object removal framework designed to perform physically-plausible inpainting in these complex scenarios. To train the model, we generate a new paired dataset of counterfactual object removals using Kubric and HUMOTO, where removing an object requires altering downstream physical interactions. During inference, a vision-language model identifies regions of the scene affected by the removed object. These regions are then used to guide a video diffusion model that generates physically consistent counterfactual outcomes. Experiments on both synthetic and real data show that our approach better preserves consistent scene dynamics after object removal compared to prior video object removal methods. We hope this framework sheds light on how to make video editing models better simulators of the world through high-level causal reasoning.
1 Introduction
Video object removal must model how deleting an object changes other objects and scene dynamics, not merely reconstruct hidden pixels or remove photometric effects. VOID addresses this challenge with counterfactual data, VLM-guided guidance, and video diffusion, and reports stronger results across synthetic and real-world evaluations.
- Challenge: Removing an object can require predicting downstream physical changes, such as later dominoes remaining standing after middle tiles disappear.The task therefore demands high-level causal reasoning rather than reliance on low-level visual features.
- VOID: VOID extends video object removal to dynamic scenarios by modeling how removing a specified object affects other objects in the scene.The framework is designed to elicit high-level causal reasoning from a video diffusion model.
- VOID: VOID combines Kubric and HUMOTO counterfactual data, quadmask conditioning, a second-pass appearance refiner, and VLM-generated pixel-space guidance.The quadmask identifies regions that may change after removal, while the refiner addresses artifacts such as unwanted object morphing.
- Evaluation: Evaluations on synthetic and real-world videos use perceptual metrics, user studies, and VLM-based judging to compare VOID with inpainting and general video-editing models.The benchmark includes videos with diverse and complex interactions.
- Results: VOID reportedly generalizes to unseen effects, including balloon motion and blender contents remaining still when the interacting person is removed.The authors attribute these extrapolations to high-level reasoning and world knowledge from the VLM and video diffusion model.
- Limitations of prior work: Existing decomposition and inpainting methods handle shadows, reflections, and other photometric effects but cannot model complex physical interactions between objects.These methods primarily recover or synthesize appearance-level content and fail when an object breaks or moves another object.
3 Approach
VOID reframes video object removal as counterfactual scene rewriting: the model removes the target, regenerates affected regions, and preserves unaffected content. It combines physics-based paired data, interaction-aware masks, VLM guidance, and a two-pass diffusion process to synthesize plausible downstream motion while reducing deformation.
- 3 Approach: VOID models object removal as generating a counterfactual video in which the target and all induced interactions are removed.The framework must eliminate the target, regenerate affected regions, and preserve unaffected regions rather than merely fill a spatial hole.
- 3 Approach: Kubric and HUMOTO provide paired videos with and without the target object, supervising rigid-body and articulated interaction changes.Kubric re-simulates collisions, falling, and structural dependencies; HUMOTO captures dynamic human-object manipulations.
- 3 Approach: Quadmask conditioning identifies the object, removal-affected regions, and regions that should remain unchanged, resolving ambiguities in prior trimask guidance.VOID focuses the affected region on likely effects and handles overlap between the removed object and newly changing areas.
- 3 Approach: The first pass typically captures broad motion hypotheses such as free-fall and continued motion, but newly synthesized motion can cause bending, stretching, or structural drift.Flow-warped noise derived from the first-pass optical flow is used to encourage temporally consistent denoising along predicted trajectories.
- 3 Approach: A VLM expands the user’s object mask to affected regions, after which VOID predicts a counterfactual trajectory and optionally applies a deformation-stabilizing second pass.The second pass uses motion-aligned warped noise; examples show improved rigidity for newly moving guitars and balls.
4 Results
VOID is evaluated on real-world and synthetic videos involving object manipulation, support removal, collisions, articulated interactions, and appearance effects. Across human, VLM-judge, qualitative, and synthetic evaluations, it generally produces more physically plausible counterfactual videos than baselines, while detailed masking and mixed training data improve performance.
- Experimental setup: The evaluation covers 75 real-world videos and a synthetic set combining 30 Kubric/HUMOTO videos with existing object-removal datasets.The real-world videos include manipulation, support removal, collisions, articulated interactions, and shadow/reflection removal.
- Human preference study: VOID was selected in 64.8% of human comparisons, substantially outperforming all baselines, including Runway with additional counterfactual guidance.The study used 25 participants, each evaluating 5 scenarios, for 125 total comparisons.
- VLM-as-a-judge evaluation: Across three VLM judges, VOID achieved the highest total score, with the strongest and most consistent gains in Interaction & Physics.Its visual quality was at least on par with the Runway general video editing model.
- Real-world comparisons: On qualitative real-world comparisons, VOID maintained object structure, plausible motion, accurate removal, and artifact-free physical evolution where baselines showed deformation, incomplete removal, or implausible outputs.Examples include preventing implausible car collisions and avoiding paint appearing after roller removal.
- Generalization: VOID generalized to unseen interactions, including released objects falling, prevented collisions, balloons floating upward, and blenders remaining off after the operator was removed.These effects were not represented in the training data described for the corresponding examples.
- Synthetic evaluation and ablations: VOID achieved the strongest synthetic performance on all metrics except LPIPS, while mixed Kubric-HUMOTO training and detailed quadmasks improved performance across evaluation categories.LPIPS can penalize counterfactual effects generated in slightly incorrect regions; the trimask ablation degraded performance across all categories.
5 Conclusion
VOID generates counterfactual videos for object removal and models complex dynamics that follow from removing an object. Its remaining limitations include domain gaps, short video lengths, and limited resolution.
- VOID generates counterfactual videos corresponding to object removal using paired datasets derived from Kubric and HUMOTO.
- A VLM-guided quadmask pipeline directs VOID toward physics-informed counterfactual video generation.
- Extensive synthetic and real-world evaluations show that VOID models complex dynamics following object removal and generalizes beyond its training scenarios.
- VOID is presented as a starting point for transferring world-modeling capabilities to video editing.
- VOID remains constrained by unusual or close camera viewpoints, few-second video lengths, and resolutions that could be improved.
VOID: Supplementary Material
The supplementary material documents additional analyses, evaluation procedures, interfaces, and prompts supporting the main paper.
- The supplement analyzes VLM-based mask generation, second-pass refinement, user interfaces, human evaluation, video similarity metrics, and VLM evaluation prompts.
(i) Mask Generation with Different VLM Reasoners
The mask-generation analysis compares three VLM reasoners using identical user clicks and evaluates resulting videos across six quality dimensions. Gemini 3-Pro produces the most reliable masks.
- Qwen3-32B, GPT 5.2, and Gemini 3-Pro receive identical user clicks to generate masks guiding the inpainting process.
- The evaluation scores interaction physics, object removal, background artifacts, temporal consistency, scene preservation, and sharpness.
- Gemini 3-Pro consistently produces the most reliable masks, particularly improving interaction physics and background reconstruction.
(ii) Second-Pass Refinement Analysis
VOID uses an optional second refinement pass for cases requiring substantial scene reconfiguration. On flagged real-world samples, refinement improves interaction reasoning, object removal, and overall scores.
- Among 75 real-world test videos, the VLM flagged 10 cases as requiring the optional refinement pass.
- The refinement step improves interaction reasoning and object removal quality, producing higher overall scores.
(iii) User Interface for Mask Generation
VOID’s mask-generation interface lets users select sparse points on the target object, after which a vision-language model generates an interaction-aware removal mask using scene context.
- Users select a small number of points on the object targeted for removal.
- The VLM uses the scene context and sparse user inputs to generate an interaction-aware removal mask.
(iv) User Study Interface
The human evaluation interface presents participants with example cases, sampled scenarios, highlighted removal targets, and outputs from seven models for comparative selection.
- Participants first read instructions and view three example object-interaction removal cases from the training dataset.
- Each participant evaluates five randomly sampled scenarios showing the original video and highlighted objects to remove.
- Table 5 evaluates Gemini 3 as a judge on 75 real-world videos across different mask-generation VLMs.
- Table 6 reports per-category average scores out of 5 for 10 samples selected by the VLM for pass 2 refinement.
- Participants compare outputs from seven models and select the result with the best inpainting outcome.
(v) Limitations of Standard Video Metrics
Standard perceptual similarity metrics can miss task-specific artifacts in interaction-aware video inpainting, sometimes favoring visually implausible results over more realistic outcomes.
- LPIPS, DreamSim, and DINOv2 measure visual fidelity and perceptual similarity but may miss dynamic interaction and causal artifacts.
- DiffuEraser and ProPainter receive better similarity scores despite leaving a clear shadow artifact in the example shown in Figure 9.
- Similarity metrics primarily assess appearance-level correspondence rather than physical plausibility or correct scene dynamics.
(vi) VLM Judge Prompt
The VLM judge first structures scene understanding around the target, interactions, consequences, background, and visual effects, then scores inpainting outputs across six dimensions.
- Stage 1 Prompt: Stage 1 asks the VLM to identify the removal target, its physical interactions, affected objects, background, and effects to remove.
- Stage 1 Prompt: The prompt emphasizes causal physics, such as a mug falling when its holder is removed or supported objects remaining stable when a ball is removed.
- Stage 1 Prompt: The structured context records the target object, interactions, physical consequences, expected background, removable effects, unchanged content, and interaction difficulty.
- Stage 2 Prompt: Stage 2 provides the result video and structured context to the VLM, which returns scores from 0–5 across six dimensions, totaling at most 30.
- Scoring Dimensions: The judge is instructed to inspect object movements and trajectories frame by frame and report only clearly observed motion.
- Scoring Dimensions: Interaction Physics is the primary category and assesses trajectories, gravity, momentum, and shadow or reflection removal.
- Scoring Dimensions: The remaining categories assess object removal, background artifacts, temporal consistency, scene preservation, and sharpness or blur.