PaperlayerView exact revision on arXiv

Source-linked AI summary

InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion

Hoiyeong Jin, Hyojin Jang, Junha Hyung, Jeongho Kim, Kinam Kim, Dongjin Kim, Huijin Choi, Hyeonji Kim, Jaegul Choo

arXiv:2512.17504v2cs.CVcs.AI

TL;DR

Video object insertion still lacks both 4D-aware placement and realistic optical interactions such as shadows and reflections. InsertAnywhere propagates a user-anchored 3D placement through video and uses optics-aware synthesis and ROSE++ training data; it reports realistic insertions that significantly outperform research and commercial tools.

  • Problem

    Video object insertion lacks 4D-aware placement and optics-aware synthesis, including realistic shadows and reflections.

    Source: §1 · p. 2

  • Method

    InsertAnywhere propagates a single-frame 3D placement into video masks and uses Optics-Aware Representation Alignment and ROSE++ to synthesize optical effects beyond the object boundary.

    Source: §1 · pp. 2–3

  • Results

    InsertAnywhere produces geometrically plausible, photometrically realistic insertions and significantly outperforms existing research and commercial generative tools.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework is presented for practical applications including virtual product placement and visual effects.

    Source: §5 · p. 15

    Complex physics-aware interactions, such as dropping a heavy object onto a soft sofa, remain outside the framework's scope.

    Source: §5 · p. 15

Abstract

from arXiv · show

Recent advances in diffusion models have enabled impressive video editing capabilities, yet production-grade Video Object Insertion (VOI) remains challenging due to inadequate 4D scene understanding and a lack of proper optical interactions, such as shadows and reflections. To address these limitations, we present InsertAnywhere, a comprehensive VOI framework that achieves geometrically grounded object placement and optics-aware video synthesis. Our approach first leverages a 4D-aware mask generation module that allows users to anchor an object's 3D pose in a single frame. The framework automatically propagates this placement across the video, accurately handling local scene dynamics and occlusions. To synthesize realistic physical lighting interactions, we introduce Optics-Aware Representation Alignment, a novel strategy that utilizes an extended mask to guide feature extraction, enabling optical effects to seamlessly extend beyond the inserted object's boundary. Finally, to overcome the lack of training data for such phenomena, we construct and open-source ROSE++, a specialized quadruplet dataset tailored for the supervised learning of optical effects. Extensive experiments demonstrate that InsertAnywhere produces geometrically plausible and photometrically realistic insertions in complex real-world scenarios, significantly outperforming existing research and commercial generative tools.

1 Introduction

Video object insertion must combine controllable 4D placement with realistic optical effects, yet existing approaches lack one or both capabilities. InsertAnywhere propagates a single 3D placement through a video and uses optics-aware synthesis; the paper reports realistic insertions that outperform research and commercial tools.

1 Introduction

  • Existing VOI methods fall short in 4D-aware placement and optics-aware synthesis, while accurate insertion also requires handling moving support surfaces and foreground occlusions.Mask-free models may hallucinate optical effects but can damage unedited background regions; mask-conditioned models constrain edits to the mask boundary.Source: §1 · p. 2
  • The 4D-aware mask module lets users anchor an object's 3D pose and scale in one frame, then propagates placement through scene dynamics and occlusions to form a temporally coherent mask video.The 3D object is reprojected into 2D, and the resulting mask conditions video generation.Source: §1 · pp. 2–3
  • Optics-Aware Representation Alignment uses an extended mask to guide feature alignment, allowing mask-conditioned diffusion to synthesize shadows and reflections beyond the object's boundary.The generation pipeline also uses first-frame anchoring.Source: §1 · p. 3
  • ROSE++ is an open-source dataset of source videos, target videos, mask sequences, and reference images tailored to supervised training of optical effects.The paper reports that models fine-tuned on ROSE++ generalize effectively to real-world VOI tasks.Source: §1 · p. 3
  • InsertAnywhere produces geometrically plausible, photometrically realistic insertions and significantly outperforms existing research and commercial generative tools.The reported scope includes diverse real-world scenarios.Source: §1 · p. 3

2 Related Work

Prior VOI approaches trade off precise spatial control and background preservation against the ability to synthesize optical effects and handle scene dynamics. InsertAnywhere combines 4D-aware mask propagation with optics-aware synthesis.

2 Related Work

  • Mask-free approaches can hallucinate optical interactions but lack precise control over object location, scale, and boundaries, and can degrade backgrounds or fail under dynamic occlusions.They also struggle to maintain temporal consistency without explicit geometric guidance.Source: §2 · p. 3
  • Mask-conditioned approaches preserve the background through localized inpainting, but rigid mask boundaries constrain shadows and reflections, while missing 4D understanding can make dynamic placements brittle.The cited settings include moving support surfaces and occlusion.Source: §2 · p. 4
  • InsertAnywhere combines an automatically generated 4D-aware mask sequence for occlusion-robust placement with optics-aware synthesis that extends lighting interactions beyond the object boundary.Its architecture propagates a single 3D placement into a mask sequence, which conditions the diffusion model.Source: §2 · p. 4, Figure 2 · p. 4

3 Method

InsertAnywhere pairs 4D-aware object placement and mask propagation with optics-aware video synthesis. Its method uses scene geometry to form coherent masks and trains generation to extend optical effects beyond the object mask.

3.1 Overview

  • InsertAnywhere couples 4D-grounded mask propagation with diffusion synthesis, targeting controllable placement and optical effects beyond the primary mask.The two stages produce a temporally consistent spatial condition and a photorealistic video with expanded shadows and reflections.Source: §3.1 · p. 5

3.2 Geometrically Grounded Mask Propagation

  • 4D reconstruction separates static and dynamic scene points; users place a 3D object, and KNN scene flow follows nearby moving surfaces before reprojection creates coherent masks.The propagation accounts for scene dynamics, camera motion, and occlusions.Source: §3.2 · pp. 5–6

3.3 Optics-Aware Video Synthesis

  • ROSE++ repurposes object-removed and object-present ROSE videos as source and target, and filters generated reference-image candidates by DINO similarity.The generated references omit scene-dependent lighting cues so the model must infer illumination from the target scene rather than copy it from the reference.Source: §3.3 · pp. 7–8
  • Optics-Aware Representation Alignment derives an extended mask from source-target differences and aligns fine-mask features with extended-mask features covering shadows and reflections.The extended-mask branch acts as a stop-gradient teacher during training, enabling optical variations at inference when only the fine mask is provided.Source: §3.3 · p. 8
  • The video diffusion model is LoRA-fine-tuned on ROSE++; optional first-frame anchoring conditions generation on an edited image to propagate appearance and illumination cues temporally.The anchoring technique uses an image-insertion model to synthesize the first frame.Source: §3.3 · p. 8

4 Experiments

Experiments test InsertAnywhere across diverse VOIBench conditions and show strong insertion quality, while ablations attribute optical-effect improvements to Optics-Aware Representation Alignment and ROSE++ design.

4.1 Experimental Setup

  • VOIBench contains 200 clips spanning scene, occlusion, and lighting axes; baselines include commercial, open-source, and mask-conditioned VACE methods, evaluated on identity, video quality, and multi-view consistency.The benchmark includes indoor, outdoor, and natural scenes; none, dynamic, and initially occluded cases; and daylight, indoor, and low-light conditions.Source: §4.1 · pp. 9–10

4.2 Qualitative Results

  • With dynamic occlusion, VACE and InsertAnywhere preserve the moving tube and background, but VACE’s inserted boat has implausible reflections and weak cast shadows.Kling can misinterpret the moving tube as the inserted paper boat, while Señorita’s insertion degrades over time under occlusion.Source: §4.2 · p. 10

4.3 Quantitative Results

  • InsertAnywhere leads CLIP-I, DINO-I, overall VBench, and five of six metrics; both ROSE++-tuned VACE variants underperform it, while Motion Smoothness closely matches Kling (0.9925 vs. 0.9940).The Motion Smoothness gap is described as within VBench’s per-clip noise.Source: §4.3 · p. 11, Table 2 · p. 11

4.4 Ablation Study

  • Progressive ablations show 4D-aware masks resolve dynamic occlusions, first-frame anchoring improves identity, and ROSE++ fine-tuning stabilizes appearance before alignment adds external lighting effects.The full combination yields geometrically accurate, photometrically realistic insertions with high object fidelity across the sequence.Source: §4.4 · p. 11
  • With the same ROSE++ training, the aligned model’s fine-mask attention activates beyond the object boundary, closely matching an extended-mask baseline and supporting synthesis of surrounding effects.The alignment strategy is reported to enable surrounding optical effects such as cast shadows without an explicitly expanded inference mask.Source: Figure 7 · p. 13, §4.4 · p. 13
  • In the optics region, alignment improves PSNR and SSIM and lowers LPIPS; optics-region PSNR rises by +7.3 dB while unedited-region PSNR remains nearly identical.Object-region appearance fidelity is largely insensitive to the alignment loss.Source: §4.4 · p. 13, Table 4 · p. 14
  • ROSE++ reference images achieve 0.5865 Multi-View Consistency versus 0.5295 for random-frame references, supporting the dataset’s goal of discouraging copy-and-paste artifacts.The paper attributes this advantage to training that requires inference of 4D geometry and lighting rather than direct image copying.Source: Table 4 · p. 14, §4.4 · p. 14

5 Conclusion

InsertAnywhere combines geometrically grounded mask propagation with optics-aware generation to improve geometric consistency and photorealistic rendering, while leaving complex physics-aware interactions outside its current scope.

5 Conclusion

  • InsertAnywhere combines geometrically grounded mask propagation and optics-aware diffusion to produce precise spatial control and photorealistic rendering, including physical lighting effects beyond the mask boundary.Evaluations report improved geometric consistency and photometric realism over existing baselines.Source: §5 · pp. 14–15
  • From one 3D anchor, the method handles camera motion and dynamic occlusions; scene-flow propagation maintains temporally consistent alignment with moving objects.The conclusion describes automatic handling of complex camera motions and dynamic occlusions; the propagation figure contrasts this with a static mask.Source: §5 · p. 14, Figure 9 · p. 15
  • Interactions involving complex physics, such as a heavy object dropping onto a soft sofa, remain outside the current scope.Source: §5 · p. 15

Supplementary Material

The supplementary material covers implementation details, a user study, retrieval prompts, and additional qualitative comparisons.

Supplementary Material

A Implementation Detail.

The implementation fine-tunes Wan2.1-VACE-14B with LoRA and reports training and inference costs, alongside steps used to reduce noise in extended masks.

A Implementation Detail.

  • Wan2.1-VACE-14B is fine-tuned with LoRA for 6,500 iterations at learning rate 1 × 10−4 and rank 128, using one NVIDIA H200 GPU for about 60 hours.Training and generation use 832 × 480 resolution and 81 frames per clip.Source: A Implementation Detail · p. 18
  • On one NVIDIA B200 GPU, inference takes about 4 minutes for 4D masks and 4.5 minutes for video synthesis at 832 × 480 × 81, with 48 GB peak memory.The reported pipeline time is comparable to VACE-14B’s 4.5 minutes and orders of magnitude faster than manual graphics pipelines.Source: A Implementation Detail · p. 18
  • Extended-mask noise is reduced by frame-differencing edits, retaining nearby connected components using EDT proximity, and applying morphological closing.The passage notes that simulator-rendered ROSE masks already have small extended-mask noise.Source: A Implementation Detail · p. 18

B User Study

The user study compared InsertAnywhere with three baselines across six perceptual criteria, with a second study focused on clips rich in shadows and reflections.

  • Twenty participants evaluated randomly ordered results from four models on videos sampled from 10% of the test set.The candidates were InsertAnywhere, Kling, Señorita, and VACEour mask.Source: B User Study · p. 18
  • The six questions assessed object realism, lighting consistency, occlusion integrity, object–video consistency, background preservation, and overall naturalness.Source: B User Study · pp. 18–19
  • The first study also asked participants to judge artifact absence and preference for real-world deployment.Source: B User Study · p. 19
  • In the optics-focused study, participants preferred InsertAnywhere across all six criteria on 50 shadow- and reflection-rich clips.The study involved 30 participants.Source: B User Study · p. 19

C ROSE++ Prompt Details.

The ROSE++ prompt details describe generating multiple visually consistent object candidates for each video and choosing one as the reference image.

D Additional Qualitative Comparisons.

The qualitative comparisons examine insertion behavior across real-world scenes and report that InsertAnywhere preserves scene content while synthesizing optical effects.

Loading 2512.17504v2…