PaperlayerView exact revision on arXiv

Source-linked AI summary

PISCO: Precise Video Instance Insertion with Sparse Control

Xiangbo Gao, Renjie Li, Xinghao Chen, Yuheng Wu, Suofei Feng, Qing Yin, Zhengzhong Tu

arXiv:2602.08277v2cs.CVcs.AI

TL;DR

Precise video instance insertion requires exact spatial-temporal control, realistic scene interaction, preserved background dynamics, and minimal annotation effort. PISCO introduces a sparse-keyframe video diffusion framework with conditioning and stabilization mechanisms for this task. It consistently outperforms strong baselines and improves monotonically as additional control frames are provided.

  • Problem

    Precise video instance insertion remains difficult because existing methods do not reliably provide exact placement, physical scene adaptation, preserved background dynamics, and low-effort control.

  • Method

    PISCO propagates appearance, motion, and interaction from sparse keyframes using Variable-Information Guidance, Distribution-Preserving Temporal Masking, and geometry-aware conditioning.

  • Results

    PISCO consistently outperforms strong inpainting, video editing, and agentic baselines on PISCO-Bench, with monotonic gains from additional sparse control frames.

  • Takeaways & Limitations

    PISCO provides a practical and scalable approach to professional-grade video editing under sparse user control.

  • Takeaways & Limitations

    Pure I2V approaches can hallucinate backgrounds, while geometry-heavy alternatives face scarce 4D-labeled data and difficulty with dynamic object insertion.

Abstract

from arXiv · show

The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.

1 Introduction

Precise video instance insertion requires exact spatial-temporal placement, realistic scene interaction, preserved background dynamics, and low-effort control. PISCO addresses these requirements with sparse keyframe conditioning and mechanisms for stable, scene-consistent propagation.

  • Precise instance insertion requires specifying when and where an object appears while preserving the original footage’s identity and dynamics.
  • The task also requires coherent object motion, physically consistent effects such as shadows and reflections, and unchanged background dynamics.
  • Existing inpainting methods require dense per-frame masks and have limited ability to adjust surrounding scenes physically consistently.
  • PISCO enables single, endpoint, or arbitrary-timestamp keyframe control while propagating appearance, motion, and interaction in a scene-consistent manner.
  • Its multi-channel adapter uses RGB, masks, depth, and availability signals, while VIG, DPTM, and geometry-aware conditioning address sparse-conditioning instability and scene consistency.
  • PISCO-Bench evaluations show consistent superiority over strong baselines and monotonic gains as additional sparse control frames are provided.

2 Related Works

Prior approaches address parts of video instance insertion but remain limited in temporal control, background preservation, or flexible dynamic-object handling. PISCO targets these gaps using sparse-controlled video diffusion with original-video context.

  • 2.1 Video Inpainting for Video Instance Insertion: Video inpainting reconstructs masked regions using spatio-temporal models, optical flow, attention, and diffusion-based generation.
  • 2.2 Reference-Guided Video-to-Video Editing: Reference-guided editing uses instance images or text prompts, but general video-to-video methods struggle with exact object position, posture, and start time.
  • 2.3 Image-to-Video (I2V) Generation and Propagation: Generate-then-animate workflows edit one keyframe before using I2V models to predict future frames.
  • 2.3 Image-to-Video (I2V) Generation and Propagation: Pure I2V insertion can hallucinate the background, causing temporal drift and inconsistency between edited foregrounds and original background pixels.
  • 2.4 4D Reconstruction and Manipulation: Geometry-heavy methods use depth, flow, or 4D representations for physical placement and occlusion handling.
  • 2.4 4D Reconstruction and Manipulation: These approaches face scarce 4D-labeled data and difficulty inserting dynamic objects, whereas PISCO supports dynamic insertion without heavy reconstruction.

3 Methodology

PISCO performs precise video instance insertion from sparse keyframe conditions by propagating appearance, motion, and interaction while using geometry-aware signals and temporal stabilization. Its training mechanisms address variable user guidance and distribution shifts caused by sparse conditioning.

  • PISCO synthesizes an inserted instance in a clean background video with spatially accurate placement, temporally coherent motion, and geometrically plausible occlusion.
  • The model accepts RGB instance clips, masks, background depth, and availability signals to condition insertion across sparse timestamps.Background depth remains unmasked, reflecting fully observable scene geometry alongside sparse instance-level guidance.
  • Variable-Information Guidance: Variable-Information Guidance samples single-frame, sparse, dense, and full-frame supervision so the model propagates motion under sparse control while preserving appearance under dense control.The hybrid sampling strategy creates a trade-off between temporal propagation and appearance fidelity.
  • Distribution-Preserving Temporal Masking: Distribution-Preserving Temporal Masking interpolates missing frames in pixel space, then masks originally unobserved tokens to preserve temporal-VAE input statistics while retaining observability information.An availability channel is aligned to compressed token resolution and concatenated with other conditions.
  • Geometry- and Appearance-Robust Training: Depth-aware conditioning supplies background and instance depth cues to improve relative depth ordering and occlusion handling during insertion.The method combines depth-aware conditioning with occlusion-aware completion and relighting augmentation for geometric and appearance robustness.

4 Implementation Details

PISCO training uses curated real-world videos, paired target/background constructions, depth estimates, amodal completion data, and staged adaptation of pretrained video diffusion backbones.

  • The training tuples contain target videos, paired background videos, segmented instance clips, masks, and background and instance depth maps.The target video contains the instance, while the paired background video represents the same scene without it.
  • The training corpus contains 16,642 filtered real-world video clips with foreground masks and at least 49 frames each.Videos come from ROSE, VPData, MOSE, and DAVIS, with ambiguous or non-detachable masks discarded.
  • Paired background videos are generated by removing target instances with a side-effect-aware video removal model, preserving shared scene dynamics for insertion training.This construction produces target/background pairs that differ in the presence of the instance.
  • Depth maps are estimated with Depth Anything V3 separately for background videos and cropped instance regions.
  • Amodal completion data pairs fully visible instances with synthetically occluded modal views to train reconstruction of complete instance appearances.A pretrained image-editing framework is used as the base generator for the completion model.
  • PISCO-1.3B and PISCO-14B build on Wan2.1-VACE-1.3B and Wan2.2-VACE-Fun-A14B, using staged training to stabilize multi-condition adaptation while preserving pretrained priors.Early stages warm up and then finetune the VACE context adapter before broader adaptation.

5 Experiments

PISCO-Bench evaluates sparse-control instance insertion across curated real-world videos using reference-based and reference-free metrics. PISCO consistently outperforms baselines, with additional sparse frames further improving quality and trajectory alignment.

  • Evaluation Data: PISCO-Bench contains 100 manually inspected BURST videos with corrected instance masks and clean backgrounds produced by side-effect-aware instance removal.The benchmark spans diverse scenarios and has no overlap with training data.
  • Experimental Setup: PISCO-1.3B and PISCO-14B are evaluated with first-frame or first-and-last-frame controls, while several baselines require dense per-frame masks.All experiments use 50 diffusion denoising steps.
  • Reference-based Evaluation: PISCO-14B with first-and-last control reduces whole-video FVD from 371 to 204 and LPIPS from 0.103 to 0.097 versus VACE.It also achieves foreground FVD 138 and LPIPS 0.022, demonstrating strong visual quality and temporal alignment.
  • Reference-based Evaluation: PISCO-14B with five sparse frames improves whole-video FVD to 136 and foreground LPIPS to 0.015.The setting uses the first, last, and three random intermediate frames.
  • Reference-free Evaluation: PISCO-14B achieves Subject Consistency 91.57, exceeding VACE’s 90.29 and inpainting models’ approximately 89.PISCO remains competitive in Background Consistency at 94.20 while offering stronger motion and instance fidelity.
  • Reference-free Evaluation: Adding three intermediate frames raises PISCO-14B to Subject Consistency 91.98 and Aesthetic Quality 51.45.The improvement appears across both reference-based and reference-free evaluations.
  • Qualitative Evaluation: Qualitative comparisons show agentic pipelines degrade background consistency, inpainting methods hallucinate objects, and generic V2V models lose spatial control or the instance.PISCO’s first-and-last control resolves trajectory ambiguity that can remain under first-frame-only control.

6 Beyond Instance Insertion: Broader Applications of PISCO

PISCO’s instance-level conditioning and temporal propagation extend beyond insertion to controllable video editing and simulation. The paper illustrates applications that preserve or modify instance motion, placement, scale, and scene interactions.

  • Broader Applications: PISCO supports background change, repositioning, rescaling, speed change, and dynamics simulation through instance-level conditions.These applications preserve instance identity and motion or create counterfactual trajectories, depending on the task.
  • Broader Applications: Repositioning and rescaling adjust instance location or size while maintaining scene-consistent interactions such as occlusions and shadows.Speed changes are induced by temporally subsampling instance-related conditions.

7 Conclusion

The paper presents PISCO as a sparse-control video diffusion model for precise instance insertion. Its benchmark results show stronger performance than several baselines and monotonic gains from additional sparse control frames.

  • Conclusion: PISCO combines Variable-Information Guidance, Distribution-Preserving Temporal Masking, and geometry-aware conditioning to address sparse-conditioning distribution shift and temporal instability.These mechanisms target the challenges of applying sparse conditions to pretrained video diffusion models.
  • Conclusion: Experiments on PISCO-Bench show consistent outperformance over inpainting, video editing, and agentic baselines, with monotonic gains as sparse control frames increase.The conclusion positions PISCO as a practical and scalable solution for professional-grade video editing.
Loading 2602.08277v2…