PaperlayerView exact revision on arXiv

Source-linked AI summary

InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction

Yuhui Wu, Liyi Chen, Ruibin Li, Shihao Wang, Chenxi Xie, Lei Zhang

arXiv:2503.20287v2cs.CV

TL;DR

Instruction-based video editing needs larger and higher-quality training triplets than existing datasets provide. The paper constructs InsViE-1M with automated two-stage filtering and trains InsViE using multi-stage learning; experiments report advantages over state-of-the-art methods.

  • Problem

    Existing instruction-based video-editing datasets are limited in quality, quantity, resolution, duration, or source-video realism.

  • Method

    The paper constructs InsViE-1M using curated videos and images, two-stage GPT-4o and optical-flow filtering, and multi-stage training of a video-generation-based InsViE model.

  • Results

    InsViE achieves the best results across all eight evaluated metrics, including a CLIP temporal-consistency score of 0.956 and the lowest optical-flow EPE.

  • Takeaways & Limitations

    InsViE-1M supports instruction-based video editing with high-quality triplets and a model designed to preserve temporal coherence and motion dynamics.

Abstract

from arXiv · show

Instruction-based video editing allows effective and interactive editing of videos using only instructions without extra inputs such as masks or attributes. However, collecting high-quality training triplets (source video, edited video, instruction) is a challenging task. Existing datasets mostly consist of low-resolution, short duration, and limited amount of source videos with unsatisfactory editing quality, limiting the performance of trained editing models. In this work, we present a high-quality Instruction-based Video Editing dataset with 1M triplets, namely InsViE-1M. We first curate high-resolution and high-quality source videos and images, then design an effective editing-filtering pipeline to construct high-quality editing triplets for model training. For a source video, we generate multiple edited samples of its first frame with different intensities of classifier-free guidance, which are automatically filtered by GPT-4o with carefully crafted guidelines. The edited first frame is propagated to subsequent frames to produce the edited video, followed by another round of filtering for frame quality and motion evaluation. We also generate and filter a variety of video editing triplets from high-quality images. With the InsViE-1M dataset, we propose a multi-stage learning strategy to train our InsViE model, progressively enhancing its instruction following and editing ability. Extensive experiments demonstrate the advantages of our InsViE-1M dataset and the trained model over state-of-the-art works. Codes are available at \href{https://github.com/langmanbusi/InsViE}{InsViE}.

1. Introduction

Instruction-based video editing is user-friendly but limited by scarce, low-quality training data. The paper introduces InsViE-1M and a two-stage filtering pipeline, alongside a video-generation-based InsViE model.

  • Instruction-based editing requires only textual instructions, unlike methods requiring masks or edited first frames.
  • Existing datasets are limited by low resolution, short videos, small scale, synthetic sources, and unsatisfactory editing quality.
  • InsViE-1M contains 1M high-quality training triplets comprising source videos, edited videos, and instructions.
  • The dataset uses two-stage filtering: GPT-4o selects edited first frames, then GPT-4o and optical flow assess propagated videos.
  • The InsViE model uses video generation models and multi-stage learning to improve editing ability and motion consistency.

2. Related Work

Prior video-editing methods include training-free, one-shot, and training-based approaches, each using different mechanisms to balance generalization, efficiency, and consistency. The paper’s pipeline builds on training-based instruction editing and automated filtering.

  • Training-free video editing: Training-free methods rely on pretrained image or video models but are limited in generalization, runtime, or input requirements.
  • One-shot video editing: One-shot methods optimize each individual video to improve visual effects, increasing per-video adaptation requirements.
  • Training-based video editing: The proposed related pipeline edits a video’s first frame, screens candidates, then filters propagated videos with GPT-4o and optical flow.
  • Training-based video editing: Training-based methods construct datasets using masks, edited first frames, or textual instructions.

3. InsViE-1M Dataset Consruction

InsViE-1M combines high-resolution real-world videos, image-editing pairs, and static videos through a two-stage generation-and-screening pipeline. The process uses first-frame editing, video propagation, GPT-4o evaluation, and optical-flow filtering.

  • Dataset overview: InsViE-1M contains 1M high-quality triplets generated from real-world videos, image-editing pairs, and static videos derived from images.
  • Triplet generation from real-world videos: The first stage edits each video’s first frame with varied CFG values and screens candidates before video propagation.
  • Triplet generation from real-world videos: Real-world videos are recaptioned with language models to produce detailed captions and varied editing instructions.
  • Triplet generation from real-world videos: The second stage propagates the selected frame, scores sampled frames with GPT-4o, and evaluates whole-video motion using optical flow.
  • Triplet generation from image editing pairs: Image-editing pairs from InstructPix2Pix and MagicBrush are converted into source and edited videos before filtering.
  • Generate static video triplets from images: Static videos created from real-world images use manual camera operations and produce 450k training triplets.

4. InsViE Model Training

InsViE is trained by finetuning CogVideoX-2B with source-video and instruction conditioning in a latent diffusion framework. The training design combines a denoising objective, video-generation architecture, multi-stage learning, and dual-condition classifier-free guidance.

  • Training setup: InsViE training finetunes CogVideoX-2B on InsViE-1M using source videos, edited videos, and instructions as training inputs.Source and edited videos are encoded into latents, while instructions are converted into text conditions.
  • Training objective: The latent diffusion objective trains a denoising network to predict noise from the noisy latent, source-video latent, and text condition.The denoising network is denoted by ϵθ.
  • Model architecture: Unlike image-model finetuning, InsViE finetunes a video-generation model to improve motion consistency during editing.The architecture encodes the source video and instruction, processes them through an embedding layer and DiT, then decodes the denoised latent into the output.
  • Multi-stage training: Multi-stage training progressively enhances editing capability, beginning with general instruction-based editing on the filtered Set-S1 dataset.The first stage applies L2 loss to the model output.
  • Classifier-free guidance: Dual-condition classifier-free guidance uses separate scales for source-video and instruction conditioning during training and sampling.The scales sV and sT correspond to input-video and instruction conditioning, and both conditions are randomly dropped in 5% of training examples.

5. Experiment

Experiments evaluate InsViE against representative video-editing methods using temporal consistency, textual alignment, and video-quality metrics. InsViE achieves strong quantitative and visual results, while ablations support the filtering and multi-stage training design.

  • 5.1. Experimental Settings: The experimental setup fine-tunes CogVideoX-2B with LoRA on InsViE-1M using three training sets and evaluates against training-free and training-based baselines.Training uses 64 NVIDIA A100 GPUs for 40k iterations at 720 × 480 resolution; evaluation considers temporal consistency, textual alignment, and video quality.
  • 5.2. Quantitative Results: InsViE achieves the best results across all eight evaluation metrics, including a CLIP temporal-consistency score of 0.956 and the lowest optical-flow EPE.The evaluation covers temporal consistency, textual alignment, and video quality; the authors note that optical flow is needed because GPT-4o does not fully capture motion dynamics.
  • 5.3. Visual Comparisons: InsViE shows stronger visual editing than RAVE and InsV2V in swapping, local color, global, and mixed editing tasks.It completely swaps the target parrot, edits the rug locally without unrelated artifacts, balances editing strength with content preservation, and follows mixed-editing keywords.
  • 5.4. Ablation Study: Removing either the GPT-4o or optical-flow filter reduces performance across metrics, with GPT-4o most affecting textual alignment and optical flow most affecting temporal consistency.The ablation therefore supports using both filters for content coherence and motion quality.
  • 5.4. Ablation Study: Multi-stage training progressively improves optical-flow EPE, textual alignment, video quality, and editing quality compared with earlier training stages.The full three-stage setting increases DOVER by 0.052 and Pick by 0.26 over Stage 1&2, while the 5:1 static-to-real video ratio provides the best reported alignment and visual quality.
  • 5.4. Ablation Study: Retraining InsV2V on InsViE-1M improves its results, demonstrating that the dataset contributes to model performance.The comparison is reported on the TGVE and TGVE+ benchmarks, where the retrained model is denoted InsV2V∗.

6. Conclusion

The paper presents InsViE-1M, a one-million-triplet dataset built from curated real-world videos and images, together with a filtered construction pipeline and multi-stage InsViE training strategy. Experiments demonstrate superior video-editing performance, while the approach remains constrained by GPT-4o’s visual understanding capability.

  • 6. Conclusion: InsViE-1M contains 1 million training triplets sourced from high-quality real-world videos, images, and image editing pairs.The triplets consist of source video, edited video, and instruction.
  • 6. Conclusion: The construction pipeline separates first-frame editing from video propagation and uses GPT-4o and optical flow to filter edited samples and videos.GPT-4o screens candidate first-frame edits, while GPT-4o and optical flow evaluate propagated videos.
  • 6. Conclusion: A multi-stage training strategy progressively trains an InsViE video-editing model whose superior performance is demonstrated through extensive experiments.
  • Limitations: The filtering process is constrained by GPT-4o’s visual understanding capability.

Supplementary Material

The supplementary material provides additional dataset-construction, training, testing, visualization, and ablation details, along with a demo video.

  • Supplementary Material: The supplementary file adds construction-pipeline details, model training and testing settings, visual comparisons, ablation studies, and a demo video.The demo video contains additional visual comparisons and requires software that can open MOV files.

7. Details of InsViE-1M Dataset

The supplementary dataset section details how captions, instructions, filtered edits, and triplets are generated from videos, image pairs, and real-world images. It also documents screening prompts and classifier-free guidance selection.

  • 7.1. Recaptioning and Instruction Generation: A large vision-language model refines video captions and generates editing instructions from three extracted key frames and initial captions.The process targets concise descriptions of subjects, actions, colors, materials, weather, and style, producing instructions for objects, styles, colors, and weather.
  • 7.1. Recaptioning and Instruction Generation: The supplementary material lists refined caption–instruction examples organized by editing type.
  • 7.2. Prompts for Screening and Filtering: The screening and filtering prompts ask GPT-4o to assess instruction adherence, edit integration, naturalness, artifacts, subject consistency, and frame consistency.The screening prompt selects the best edited image, while the filtering prompt evaluates video-edit quality and consistency.
  • 7.3. Examples of Triplets Construction Process: Triplets from real-world videos, image editing pairs, and real-world images are illustrated through simplified construction examples.For real-world images, camera-motion generation replaces video editing and filtering’s video-generation step.
  • 7.3. Examples of Triplets Construction Process: Camera motion is synthesized by gradually changing crops, using a minimum crop size of 90% for zoom and directional movements.
  • 7.4. CFG Case Study: A CFG sweep on 10K images found most best edited samples between 3.0 and 8.0, leading to six generated samples with CFG in [8].

8. Training and Testing

The paper details the training setup and evaluates edited videos with GPT-4o using distinct testing prompts and frame-selection procedures. Testing scores assess temporal consistency and textual alignment.

  • Training and Testing: Training uses 8 nodes with 8 Nvidia A100 GPUs, batch size 128, and 40k optimization steps.The implementation follows CogVideoX defaults and uses Adam with EMA.
  • Training and Testing: Testing supplies all video frames to GPT-4o instead of three sampled key frames used during data filtering.The authors state that the larger test-set resource cost is acceptable.
  • Training and Testing: GPT-4o testing introduces scores across three evaluation aspects as a new metric.The revised testing prompts differ from those used in screening and filtering.
  • Training and Testing: Temporal consistency evaluation checks composition coherence and content consistency across original and edited frames.The prompt requests an integer score from 1 to 5, where 5 denotes the highest temporal consistency.
  • Training and Testing: Textual alignment evaluation assesses strict adherence to the instruction and integration of the requested edits.The supplied passage introduces these criteria but does not include the complete scoring specification.

9. More Visual Results

Additional qualitative results show diverse InsViE-1M triplets and compare InsViE with prior methods. The comparisons report stronger editing performance and more visually pleasing videos across varied instructions.

  • More Visual Results: InsViE-1M samples cover removal, substitution, addition, and stylization editing instructions.The examples are presented as source video, edited video, and instruction triplets.
  • More Visual Results: InsViE achieves better editing performance than previous methods across various editing instructions in visual comparisons.The passage reports that the resulting videos are more visually pleasing.

10. More Ablation Studies

The additional ablations examine LPIPS loss and the static-to-real video ratio in the final training stage, showing their reported effects on quality and alignment metrics.

  • More Ablation Studies: LPIPS loss is added in Stage 3 to enhance detail generation and contributes more to video quality metrics.The first two training stages use L2 loss.
  • More Ablation Studies: A static-to-real ratio of 0:1 produces results similar to Stage 1&2, indicating limitations of using real videos alone.The comparison concerns the final training stage.
  • More Ablation Studies: Increasing the static-to-real ratio to 0.5:1 improves all reported metrics over Stage 1&2.The passage reports a broad improvement but does not provide the metric values.
  • More Ablation Studies: At a 1:1 static-to-real ratio, performance stabilizes with better DOVER and GPT quality scores.The authors characterize static videos as beneficial for visual quality.
  • More Ablation Studies: The largest reported gain occurs at a 5:1 static-to-real ratio, especially for textual alignment and video quality.This ratio is evaluated in the final training stage.
Loading 2503.20287v2…