PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Models

Jinshu Chen, Xinghui Li, Xu Bai, Tianxiang Ma, Pengze Zhang, Zhuowei Chen, Gen Li, Lijie Liu, Songtao Zhao, Bingchuan Li, Qian He

arXiv:2509.17627v1cs.CV

TL;DR

Mask-free video insertion faces data scarcity, subject-scene balancing, and harmonization challenges. OmniInsert combines diverse paired-data construction with a unified insertion framework and harmonization techniques; the authors report that it outperforms closed-source commercial solutions on InsertBench.

  • Problem

    Mask-free video insertion lacks paired training examples and existing approaches can require complex controls while struggling with subject consistency, scene preservation, and natural interactions.

    Source: §1 · p. 4

  • Method

    InsertPipe constructs diverse cross-pair data, while OmniInsert combines distinct source and subject conditioning, progressive training, subject-focused supervision, preference optimization, and context-aware prompt rephrasing.

    Source: §1 · p. 4

  • Results

    OmniInsert is reported to outperform closed-source commercial solutions on InsertBench.

    Source: §1 · p. 4, §6 · p. 13

  • Takeaways & Limitations

    The paper presents a data pipeline, insertion framework, and benchmark as a comprehensive study intended to support future MVI research and development.

    Source: §6 · p. 13

    Generated results may retain slight color discrepancies and minor physical implausibilities, which IPO reduces but does not fully resolve.

    Source: D Limitation and Future Work · p. 20

Abstract

from arXiv · show

Recent advances in video insertion based on diffusion models are impressive. However, existing methods rely on complex control signals but struggle with subject consistency, limiting their practical applicability. In this paper, we focus on the task of Mask-free Video Insertion and aim to resolve three key challenges: data scarcity, subject-scene equilibrium, and insertion harmonization. To address the data scarcity, we propose a new data pipeline InsertPipe, constructing diverse cross-pair data automatically. Building upon our data pipeline, we develop OmniInsert, a novel unified framework for mask-free video insertion from both single and multiple subject references. Specifically, to maintain subject-scene equilibrium, we introduce a simple yet effective Condition-Specific Feature Injection mechanism to distinctly inject multi-source conditions and propose a novel Progressive Training strategy that enables the model to balance feature injection from subjects and source video. Meanwhile, we design the Subject-Focused Loss to improve the detailed appearance of the subjects. To further enhance insertion harmonization, we propose an Insertive Preference Optimization methodology to optimize the model by simulating human preferences, and incorporate a Context-Aware Rephraser module during reference to seamlessly integrate the subject into the original scenes. To address the lack of a benchmark for the field, we introduce InsertBench, a comprehensive benchmark comprising diverse scenes with meticulously selected subjects. Evaluation on InsertBench indicates OmniInsert outperforms state-of-the-art closed-source commercial solutions. The code will be released.

1 Introduction

Mask-free video insertion remains challenged by data scarcity, balancing subject consistency with preservation of unedited scenes, and natural interactions. OmniInsert addresses these challenges with a data pipeline, a unified insertion framework, and harmonization methods, and is reported to outperform commercial solutions on InsertBench.

1 Introduction

  • Mask-free video insertion lacks paired training data and must balance inserted-subject consistency with unchanged scenes and plausible interactions.Earlier approaches either add an inversion stage or rely on complex controls, and reported methods still struggle with consistency or naturalness [23] [54] [45] [55] [28] [53].Source: §1 · p. 4
  • InsertPipe automatically constructs diverse cross-pair training data through real-video, generated-data, and rendering-based pipelines.Its pipelines target scene diversity and complex interactions; cross-video subject pairs are used to prevent copy-paste issues associated with key-frame references [3] [16].Source: §1 · p. 4
  • OmniInsert supports mask-free insertion from single or multiple subject references, using distinct condition injection, progressive training, and subject-focused supervision.The injection mechanism separates source-video and subject conditions; progressive training balances their injection, while Subject-Focused Loss targets subject detail.Source: §1 · p. 4
  • Insertive Preference Optimization and the Context-Aware Rephraser are designed to improve insertion harmonization in complex scenes.IPO uses paired videos reflecting human preferences, while CAR enriches inference-time prompts with scene details such as layout and interaction cues.Source: §1 · p. 4
  • On InsertBench, OmniInsert achieves clear quantitative and qualitative advantages over state-of-the-art commercial solutions.InsertBench comprises 120 videos paired with selected subjects and prompts.Source: §1 · p. 4
  • The paper introduces InsertBench, a benchmark of 120 videos paired with carefully selected subjects and corresponding prompts.The authors report using it for extensive quantitative and qualitative evaluation.Source: §1 · p. 4

2 Related Work

Related work describes the shift from image-centric video diffusion models toward DiT-based methods, alongside video-insertion approaches that use inversion or control signals. Mask-free editing works and commercial systems motivate closing the gap in insertion quality and applicability.

2 Related Work

  • DiT-based video-generation methods process spatiotemporal patches with Transformers and are described as improving video quality and consistency.Earlier approaches extend image models with temporal modules, while VDM extends 2D U-Net to 3D and AnimateDiff adds temporal attention [9].Source: §2 · p. 5
  • Prior insertion methods use DDIM inversion or mask and point controls, while newer mask-free editing works still face subject infidelity and incoherent interactions [23] [54] [45] [55] [28] [53].The inversion approaches add computational overhead; VideoAnyDoor and GetInVideo require control signals.Source: §2 · p. 5
  • The authors identify commercial software as a current state-of-the-art reference point and frame bridging the academic-commercial gap as important.The passage names commercial software.Source: §2 · pp. 5–6

3 Preliminary

The preliminary section describes a Diffusion Transformer denoiser trained with Flow Matching, which linearly interpolates between noise and clean video. Its stacked blocks combine spatial and spatiotemporal attention.

  • The model uses a Diffusion Transformer (DiT) as its denoising network to refine diffusion latents.It inherits video diffusion transformers trained using Flow Matching.Source: §3 · p. 6
  • Flow Matching linearly interpolates between noise and clean video, and trains the model to regress the target velocity.At time t, z_t = (1 − t)z_0 + tϵ, where z_0 is clean video and ϵ is Gaussian noise.Source: §3 · p. 6
  • Stacked DiT blocks use 2D self-attention for spatial modeling and 3D self-attention for spatiotemporal fusion.The diffusion-model output is vθ, and y denotes the condition.Source: §3 · p. 6

4 Methodology

OmniInsert pairs an automated cross-pair data pipeline with a mask-free insertion framework for source videos and one or more subject references. It combines condition-specific feature injection, staged training, subject-focused supervision, preference optimization, and inference-time prompt rephrasing.

4.1 InsertPipe

  • InsertPipe creates paired prompt, reference-image, source-video, and target-video samples through real-capture, generated, and rendered-interaction pipelines.RealCapture uses detection, tracking, and erasing; SynthGen produces subject-scene pairs and videos; SimInteract renders complex interactions using asset and motion libraries.Source: §4.1 · pp. 6–7

4.2 OmniInsert Framework

  • Condition-Specific Feature Injection handles source-video and subject references with distinct temporal alignment strategies, combining them in one framework for mask-free insertion.Source-video latents are concatenated with noisy target latents along the channel dimension; subject latents are concatenated along the temporal dimension.Source: §4.2 · pp. 7–8

4.3 OmniInsert Training Pipeline

  • Progressive Training moves from subject modeling to full insertion, identity refinement, and preference optimization to balance subject and source-video features.The authors describe four stages: subject-and-prompt training, full-task pretraining, portrait and rendering refinement, then preference-based fine-tuning.Source: §4.3 · p. 8
  • Subject-Focused Loss directs flow-matching attention toward tracked subject regions, while phase-four IPO uses human-selected preferences to improve visual stability and physical plausibility.The paper states that IPO is applied in phase four and reports that 500 preference pairs suffice for substantial performance gains at this stage.Source: §4.3 · pp. 8–9

4.4 OmniInsert Inference Pipeline

  • At inference, Context-Aware Rephraser uses VLM-generated scene and subject descriptions plus creative constraints to produce prompts for more coherent insertion.The constraints include preserving the source setting and maintaining spatial consistency in subject size and position.Source: §4.4 · pp. 9–10

5 Experiments

OmniInsert is evaluated against Pika-Pro and Kling using visual, metric-based, and user-study comparisons, with ablations assessing its proposed modules. The reported results favor OmniInsert overall, while ablations associate each module with a distinct benefit.

5.1 Setup

  • The evaluation compares OmniInsert with Pika-Pro and Kling across subject consistency, text-video alignment, and video quality.InsertBench contains 120 videos across varied environments; each is about 5 seconds and 121 frames. Experiments use 480p resolution.Source: §5.1 · p. 10

5.2 Qualitative Analysis

  • Visual comparisons report stronger subject/background consistency and natural integration for OmniInsert; Pika struggles in wearable and human-centric scenes, while Kling shows portrait copy-paste issues and scene inconsistency.The compared methods do not support multiple references; additional multi-subject results are provided in supplementary materials.Source: §5.2 · p. 10

5.3 Quantitative Comparisons

  • OmniInsert leads on CLIP-I*, DINO-I*, and ViCLIP-T, shows superior video quality, and has significant user-study advantages, though Kling scores higher on face similarity in portrait scenarios.The user study used 40 test samples and four criteria: subject consistency, text alignment, insertion rationality, and comprehensive evaluation.Source: §5.3 · p. 10

5.4 Ablation Studies

  • Ablations associate PT with greater insertion stability, SL with subject preservation, CAR with more natural compositions, and IPO with insertion rationality and fewer visual artifacts.The reported ablation evidence is in Fig. 7 and Table 3.Source: §5.4 · p. 13, Figure 7 · p. 13, Table 3 · p. 13

6 Conclusion

The paper presents OmniInsert for mask-free video insertion, addressing data scarcity, subject-scene equilibrium, and insertion harmonization, and introduces InsertBench. Experiments report that OmniInsert outperforms closed-source commercial solutions.

6 Conclusion

  • OmniInsert combines InsertPipe, CFI, progressive training with Subject-Focused Loss, IPO, and CAR, and experiments on InsertBench show it outperforms closed-source commercial solutions.InsertBench is introduced to address the lack of a benchmark for the task.Source: §6 · p. 13

A Implementation Details

The appendix specifies the IPO and Subject-Focused Loss hyperparameters used in training.

A.1 Detailed Parameters

  • The appendix introduces the hyperparameter listing as the settings used in the experiments.Source: §A.1 · p. 18
  • IPO loss hyperparameters γ, λ, and β are set to 10, 1, and 1, respectively, for phase 4 optimization.Source: §A.1 · p. 18
  • Subject-Focused Loss hyperparameters λ1 and λ2 are both set to 1.Source: §A.1 · p. 18

A.2 Training and Inference Details

The appendix reports training durations, data composition, model configuration, inference time, and the user-study procedure. Training uses four phases and preference data; user-study responses were randomized and filtered for validity.

A.2 Training and Inference Details

  • Training runs for 70k, 30k, 10k, and 8k iterations across four phases, using about 1M samples for phases 1–2, 50k for phase 3, and 0.5k preference pairs for phase 4.Phase GPU-hour estimates are about 2700, 1500, 500, and 2300, respectively.Source: §A.2 · p. 18
  • LoRA rank 256 in DiT blocks yields 600M trainable parameters, and generating one 5-second 480P video takes about 90 seconds on 8 A100 GPUs.Source: §A.2 · p. 18

A.3 User Study

  • The user study presented 40 randomly ordered result sets per participant and retained 30 valid surveys after excluding specified invalid responses.Participants judged subject consistency, prompt match, insertion rationality, and overall preference.Source: §A.3 · p. 18

B InsertBench Details

InsertBench pairs videos with selected subjects and prompts, spanning diverse scene categories. The section also shows benchmark examples and describes the videos’ frame rate and duration.

B InsertBench Details

C More Visual Results

Supplemental results show OmniInsert’s insertions and comparisons with baselines. On InsertBench at 480P, the authors report robust insertions across varied formats and multi-subject scenarios, while Pika-Pro and Kling show failures and unnatural artifacts.

C More Visual Results

  • Figures 10 and 11 show OmniInsert inference results, while Figures 12 and 13 compare it with baselines.The corresponding videos are available in supplementary attachments.Source: C More Visual Results · p. 20
  • Pika-Pro and Kling show insertion failures and unnatural artifacts, while OmniInsert demonstrates superior robustness on InsertBench at 480P.The authors also report that other baselines lack or perform poorly on multi-subject scenarios, which OmniInsert handles.Source: C More Visual Results · p. 20
  • Supplemental results report stable, semantically coherent insertions with strong prompt fidelity across diverse aspect ratios and multi-subject scenarios.The authors attribute multi-subject support to concatenating multiple subjects’ IP information along the temporal dimension for unified training.Source: C More Visual Results · p. 20

D Limitation and Future Work

The authors report occasional color discrepancies and physically implausible results that remain unresolved, alongside an inference-time comparison with closed-source baselines. They identify preference optimization and general video-diffusion acceleration as directions for improvement.

D Limitation and Future Work

  • OmniInsert can still produce slight color discrepancies and minor physically implausible phenomena, which IPO reduces but does not completely resolve.The authors say IPO depends on high-quality model-generated samples, limiting selectable examples, especially in diversity, when artifacts are prevalent.Source: D Limitation and Future Work · p. 20, Figure 14 · p. 24
  • The authors identify more advanced preference-optimization techniques as a direction for addressing artifact-related limitations.Source: D Limitation and Future Work · p. 20
  • A 480P, 121-frame video takes approximately 90 seconds to infer, versus over 180 seconds for the compared baselines.Because the baselines are closed-source, the authors could not conduct a fair comparison by running them offline; they suggest general video-diffusion acceleration techniques for future improvement.Source: D Limitation and Future Work · p. 20
Loading 2509.17627v1…