Source-linked AI summary
Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists
Bojia Zi, Penghui Ruan, Marco Chen, Xianbiao Qi, Shaozhe Hao, Shihao Zhao, Youze Huang, Bin Liang, Rong Xiao, Kam-Fai Wong
TL;DR
Existing video editing methods face slow or inconsistent inversion-based editing and limited high-quality pairs for end-to-end training. The paper introduces Señorita-2M, a roughly two-million-pair dataset built with specialized experts, instruction rewriting, and filtering, and reports high-quality editing models with improved ablation metrics. Its impact statement notes that trained models can edit videos, with deepfake detection identified as a risk-reduction measure.
Problem
Existing video editing methods are limited by slow, inconsistent inversion-based editing and insufficient high-quality training pairs for end-to-end methods.
Method
The paper builds Señorita-2M using four specialized video editing experts, LLM-generated instructions, and a filtering pipeline for approximately two million editing pairs.
Results
Señorita-2M trains high-quality video editing models with strong visual quality, frame consistency, and text alignment; an ablation improves CLIPScore from 0.2366 to 0.2596 and Temporal Consistency from 0.9712 to 0.9783 over InsV2V.
Takeaways & Limitations
The dataset provides large-scale instruction-based training data for general video editing across 18 tasks.
Takeaways & Limitations
Models trained on the dataset are capable of editing videos, creating a misuse risk that the authors suggest reducing with deepfake detection methods.
Abstract
from arXiv · showhide
Recent advancements in video generation have spurred the development of video editing techniques, which can be divided into inversion-based and end-to-end methods. However, current video editing methods still suffer from several challenges. Inversion-based methods, though training-free and flexible, are time-consuming during inference, struggle with fine-grained editing instructions, and produce artifacts and jitter. On the other hand, end-to-end methods, which rely on edited video pairs for training, offer faster inference speeds but often produce poor editing results due to a lack of high-quality training video pairs. In this paper, to close the gap in end-to-end methods, we introduce Señorita-2M, a high-quality video editing dataset. Señorita-2M consists of approximately 2 millions of video editing pairs. It is built by crafting four high-quality, specialized video editing models, each crafted and trained by our team to achieve state-of-the-art editing results. We also propose a filtering pipeline to eliminate poorly edited video pairs. Furthermore, we explore common video editing architectures to identify the most effective structure based on current pre-trained generative model. Extensive experiments show that our dataset can help to yield remarkably high-quality video editing results. More details are available at https://senorita-2m-dataset.github.io.
1. Introduction
Video editing methods face a trade-off between flexible but slow inversion-based editing and faster end-to-end editing limited by training-pair quality. Señorita-2M addresses this gap with specialized expert models, filtering, and approximately two million video editing pairs.
- Motivation: Video editing is dominated by inversion-based methods that can require long editing durations and produce frame inconsistencies.End-to-end methods are attracting attention as an alternative.
- Dataset construction: Señorita-2M contains approximately 2 million high-quality video editing pairs constructed with expert models and a filtering pipeline.The dataset uses videos sourced from Pexels and filters failed samples.
- Contributions: The contribution is positioned as a large-scale instruction-based video editing dataset, contrasting with prior datasets focused on local edits or synthetic generation.The original videos are sourced from the Internet.
- Dataset construction: The dataset uses four specialized experts for global stylization, local stylization, object removal, and object swapping.Each expert is reported to achieve state-of-the-art performance in its task.
- Results: Experiments show that models trained on Señorita-2M achieve high visual quality, strong frame consistency, and text alignment.The authors also evaluate multiple video editing architectures using the dataset.
2. Related Works
Prior video editing methods emphasize frame consistency or prompt-guided generation, while video editing datasets remain scarce and often rely on limited local annotations or synthetic pairs. Señorita-2M is introduced against this data-quality gap.
- Video editing methods: Video editing methods include fine-tuning, attention-based consistency, first-frame feature injection, and noise-based style transfer approaches.Examples include Tune-A-Video, Pix2Video, TokenFlow, AnyV2V, Gen3, and SORA.
- Editing datasets: Image editing datasets commonly use synthetic data, human annotations, or large-scale generated pairs with LLM-generated instructions.Examples include InstructPix2Pix, MagicBrush, HQ-Edit, EmuEdit, and UltraEdit.
- Editing datasets: Only a few video editing datasets exist, including RACCooN, VIVID-10M, and InsV2V.RACCooN and VIVID-10M use inpainting models, whereas InsV2V uses generated original and target videos.
- Editing datasets: InsV2V’s generated video pairs are described as having insufficient data quality for strong performance.This motivates improving the quality of training pairs for video editing.
3. Methodology
Señorita-2M combines specialized video editing experts, sourced and processed video data, instruction rewriting, and cascaded filtering. Its tasks span local and global editing, with filtering designed to remove failed, misaligned, or imperceptible edits.
- Expert models: The dataset construction uses four video experts: global stylizer, local stylizer, text-guided inpainter, and object remover.The broader pipeline includes data collection, inference for local and global pairs, and filtering.
- Global edit: The global stylizer edits an initial frame with image ControlNet, then uses Canny, HED, and depth controls to generate remaining frames.An LLM converts style prompts into actionable instructions.
- Local edit: The local stylizer combines inpainting and ControlNet, feeding control conditions to one branch and mask conditions to the main branch.It uses CogVideoX-2B as the pretrained model.
- Local edit: Local editing covers object swap, local style transfer, object addition, object removal, inpainting, and outpainting.Object swap uses FLUX-Fill and an inpainter, while LLMs rewrite task instructions.
- Global edit: Global editing includes style transfer, object grounding, and conditional generation across ten video-to-video translation tasks.These tasks include deblurring, edge and depth control, detection, upscaling, scribble conversion, and colorization.
- Filtering: A filtering pipeline removes failed edits, poor text alignment, and videos with subtle or absent changes.It combines quality classifiers with CLIP-based prompt and change comparisons.
4. Experiments
The experiments construct and evaluate Señorita-2M through expert-generated pairs, filtering, model training, quantitative comparisons, ablations, and architecture studies. Results report stronger editing quality, consistency, alignment, and user preference, while first-frame guidance and dataset enhancement improve architecture performance.
- 4.1. Construction of Dataset: The dataset construction pipeline collects and annotates videos, applies editing experts, generates instructions with LLMs, and filters failure cases.The experiments also describe implementation settings for expert inference and training.
- 4.3.1. Quantitative Comparison: The trained editor achieves Ewarp 9.42, CLIPScore 0.2895, and Temporal Consistency 0.9775, outperforming previous methods across these metrics.Evaluation uses Ewarp, Temporal Consistency, and CLIPScore on DAVIS with randomly generated editing prompts.
- 4.3.1. Quantitative Comparison: Against InsV2V, the method improves Ewarp from 16.50 to 9.42, CLIPScore from 0.1675 to 0.2895, and Temporal Consistency from 0.9727 to 0.9775.Both methods require video-pair training, whereas the other compared methods are zero-shot.
- 4.3.2. Qualitative Results: 53.17% user preference is reported for the method, compared with 19.40% for the next-best approach.The questionnaire sequence was randomly shuffled to ensure fairness.
- 4.3.3. Ablation Study: Using Señorita-2M instead of InsV2V raises CLIPScore from 0.2366 to 0.2596 and Temporal Consistency from 0.9712 to 0.9783 under the same training duration.Increasing Señorita-2M samples from 60K to 120K further raises CLIPScore to 0.2641 and lowers Ewarp from 8.44 to 7.95.
- 4.3.4. Different Editing Architectures: FF-Control-Edit reaches 37.12% user preference, while Control-Edit with Omni-Edit reaches Temporal Consistency 0.9802 and FF-Ins-Edit reaches the lowest Ewarp at 8.44.The architecture study compares InstructPix2Pix and ControlNet variants with and without first-frame guidance and dataset enhancement.
5. Conclusion
The paper presents Señorita-2M as a large instruction-based video editing dataset built from specialized experts, computer-vision controls, filtering, and LLM-generated instructions. Experiments report improvements in visual quality, frame consistency, and text alignment.
- 5. Conclusion: Señorita-2M contains 18 video editing tasks and approximately 2 million video pairs created with advanced editing models and computer-vision experts.A cascade of filtering algorithms and LLM prompt transformation are used to support dataset quality and instruction precision.
- 5. Conclusion: Experiments show that the dataset can produce high-quality video editing models with improved visual quality, frame consistency, and text alignment.The paper validates the dataset by training four editing models using four widely adopted editing architectures.
Impact Statement
The dataset uses legally sourced real videos and is described as posing no harm to the community. Models trained on it can edit videos, creating a risk that the paper says could be reduced by deepfake detection methods.
- Impact Statement: The real videos are legally sourced from Pexels.com through authenticated APIs.The impact statement describes the dataset itself as posing no harm to the community.
- Impact Statement: Models trained on the dataset are capable of editing videos, and the authors identify deepfake detection methods as a way to reduce the associated risk.This is presented as a risk of the trained models rather than of the dataset itself.
B.1. The Construction of Expert Training Dataset
The expert-training dataset is built from WebVid-10M with object recognition, segmentation masks, control conditions, and detailed captions. These annotations support training specialized inpainter, remover, and stylizer models.
- B.1. The Construction of Expert Training Dataset: The dataset is based on WebVid-10M, with CogVLM2 recognizing objects and Grounded-SAM2 generating phrase names and object-mask sequences.The detected object names are separated by commas and used as Grounded-SAM2 input prompts.
- B.1. The Construction of Expert Training Dataset: Object-mask annotations are used to train the inpainter, remover, and local stylizer models.Canny, HED, and depth detectors provide control conditions for global and local stylizer training.
- B.1. The Construction of Expert Training Dataset: CogVLM2 generates detailed video captions for inpainter training, while object-recognition prompts request brief comma-separated object names.The caption prompt requests descriptions longer than 20 words and shorter than 60 words.
B.2. The Construction of Global Stylizer
The global stylizer combines image-based first-frame stylization with video ControlNet propagation to improve text-guided style transfer while preserving video structure. Its expert model outperforms all baselines across warping, text alignment, and temporal consistency.
- Architecture: The global stylizer addresses weak style understanding in video generation models by first editing the initial frame with image ControlNet.The edited first frame then guides video ControlNet for completing subsequent frames.
- Architecture: The model integrates DiT ControlNet with CogVideoX-5B-I2V to propagate first-frame style features through subsequent frames.Control features are injected into the main branch through zero convolution across DiT blocks.
- Training: The training pipeline initializes the main and control branches from CogVideoX-5B-I2V, using six control DiT blocks and partially frozen layers.Training uses one epoch, batch size 8, learning rate 1e-5, and weight decay 1e-4.
- Inference: The inference setup generates 33-frame videos at 336×592 resolution within 2 minutes on an Nvidia RTX 4090.The method uses classifier-free guidance of 4 and appends the required style prompt to the video description.
- Results: The expert model achieves the lowest Ewarp (9.02), highest CLIPScore (0.3145), and best Temporal Consistency (0.9781).These results indicate strong visual quality, text alignment, and temporal smoothness relative to the baselines.
B.3. The Construction of Local Stylizer
The local stylizer combines inpainting and ControlNet to modify appearance, style, or texture in selected video regions while preserving the original background. It achieves strong stylization, temporal consistency, and background fidelity against the compared methods.
- Architecture: The local stylizer combines inpainting and ControlNet for regional appearance modification, stylization, and texture manipulation while keeping the background unchanged.Its design is inspired by SparseControl, CoCoCo, and AVID.
- Architecture: The local stylizer uses the same ControlNet architecture as the global stylizer but changes the base model and input condition.It uses CogVideoX-2B and accepts original-video, masked-video, and mask latents as input.
- Training: Training uses masks and phrases from the training data, with randomly composed pronoun-based sentences for one epoch.The setup uses batch size 32, learning rate 1e-5, weight decay 1e-4, and 33-frame videos at 336×592 resolution.
- Inference: The model edits 33-frame videos at 336×592 resolution within 1 minute on an Nvidia RTX 4090.Inference uses classifier-free guidance scale 6 and prompts such as “It’s a yellow house.”
- Results: The expert model achieves the lowest Ewarp (6.50), highest CLIPScore (0.2944), and best Temporal Consistency (0.9828).It also obtains the highest PSNR (28.29) and SSIM (0.9843), with LPIPS (0.0346) and MSE (108.25) supporting background preservation.
B.4. The Construction of Text-Guided Video Inpainter
The text-guided video inpainter is developed for object addition, modification, and removal using diffusion-based video completion and varied mask construction. Its object-swap expert leads the comparison in text alignment and temporal consistency.
- Motivation: Existing video inpainting methods often rely on outdated foundation models and can produce artifacts and inconsistencies, motivating a new CogVideoX-based inpainter.The paper contrasts its approach with VIVID, whose inpainter was not open-sourced.
- Training: The inpainter trains with random mask positions and shapes, including rectangles with varying aspect ratios whose locations shift across subsequent frames.This design addresses overfitting to specific mask shapes.
- Training: The first frame of each mask sequence is set to zeros so the model can use guidance from the edited image.The inpainter is initialized from CogVideoX-5B-I2V and does not require a control branch.
- Inference: During inference, Flux-Fill edits the first frame, while the inpainter generates the remaining frames from prompts, dilated masks, and the input video.LLM-generated object names provide the replacement target.
- Results: The expert model achieves the highest CLIPScore (0.3186) and Temporal Consistency (0.9782) for object swap.The comparison notes that InsV2V has lower Ewarp (8.80) but often fails to follow instructions and swap the object.
B.5. The Construction of Remover
The remover is trained to break the correlation between generated content and mask shape by using masks from unrelated videos and mixed removal or generation instructions. It balances removal effectiveness, temporal stability, and background preservation better than the compared methods.
- Motivation: Prior inpainting methods can reproduce the masked shape during object removal, so the remover is designed to break the correlation between generated content and mask shape.This limitation was observed for CoCoCo during object-removal experiments.
- Training: The remover assumes input objects are unrelated to the sampled mask and phrase, training the model to remove the assumed object while generating the input video’s object.The training mixture uses 90% unrelated masks with “Remove” instructions and 10% input-video masks with “Generate” instructions.
- Training: The remover is built on CogVideoX-2B and trained for one epoch with 33-frame, 336×592 videos using AdamW.Training uses batch size 32, learning rate 1e-5, weight decay 1e-4, and a 90% task-irrelevant versus 10% task-relevant mask split.
- Inference: The removal process completes within 1 minute on an Nvidia RTX 4090 for 33-frame videos at 336×592 resolution.Inference uses positive “Remove {object name}” and negative “Generate {object name}” prompts with guidance scale 2.
- Results: The expert model achieves the lowest Ewarp (4.21) and Relevance (0.1554), plus the highest Temporal Consistency (0.9864).Propainter has stronger pixel-level metrics but poor removal because removed regions become blurry; the expert model balances removal and background preservation.
C. Construction of Señorita-2M Dataset
Señorita-2M is constructed from diverse source videos and specialized editing models to create supervised video-editing pairs. The construction covers stylization, grounding, translation, object manipulation, and inpainting or outpainting, with LLM-generated instructions.
- Source videos: The source collection contains 388,909 Pexels videos, primarily at 720p–4K resolution, with most exceeding 500 frames.
- Specialized editing tasks: Global stylization combines 290 style prompts with masks and control conditions such as HED, depth, and Canny.
- Instruction construction: LLMs convert generated prompts into concise editing instructions, producing source–target–instruction triplets from original and edited videos.
- Specialized editing tasks: The dataset includes object-grounding pairs that highlight instruction-relevant instances while masking unrelated regions.
- Specialized editing tasks: Ten video-to-video translation tasks include deblurring, edge, depth, HED, scribble, upscaling, detection, and colorization variants.
- Specialized editing tasks: Object stylization, swapping, removal, addition, and inpainting or outpainting provide additional edited-video pair types.
D. Data Selection and Cleaning
The data-selection pipeline removes failed, poorly aligned, and minimally changed edits. It combines learned quality classification with text–video similarity and frame-level original–edited comparisons.
- Filtering pipeline: A quality-classifier stage removes corrupted and failed generated samples before subsequent filtering.
- Quality classification: The classifier is trained from 5,000 edited videos, including approximately 1,000 manually annotated failures, with a validation set of 120 successful and 60 failed samples.
- Quality classification: Two CLIP-based feature classifiers are ensembled from frame-wise features extracted across 17 frames per video.
- Quality classification: Samples are retained when classifier confidence exceeds 0.6, with a lower threshold used for object addition.
- Text alignment: Task-specific CLIP text–video similarity thresholds remove edits whose content is unrelated to their prompts, accounting for masked-region effects.
- Edit strength: Pairs with original–edited similarity above 0.95 are removed to exclude unchanged or only subtly modified videos.