PaperlayerView exact revision on arXiv

Source-linked AI summary

RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing

Bojia Zi, Xiaoyan Yang, Yu Zhou, Ruijie Sun, Lihan Zhang, Bin Liang, Kam-Fai Wong, Haibin Huang, Chi Zhang, Xuelong Li

arXiv:2608.26101v1cs.CV

TL;DR

Existing video-editing datasets often use model-generated targets and lack visual references, motivating more reliable and controllable supervision. RefVideo-6M addresses this gap with expert-constructed video and image pairs, diverse references, and RefMoT; experiments report state-of-the-art performance and improved reference-based editing results.

  • Problem

    Existing datasets may contain artifacts from model-generated targets and typically lack visual guidance alongside textual instructions.

  • Method

    RefVideo-6M constructs 5 million video pairs and 1 million image pairs using expert editing pipelines, reversed video roles, filtering, and 6 million references across 10 types.

  • Results

    Models trained on RefVideo-6M achieve state-of-the-art performance, while RefMoT improves reference-based editing and reduces training computation by 50%.

  • Takeaways & Limitations

    RefVideo-6M supports reference-based video editing with supervision designed for reliability, visual controllability, and reference consistency.

Abstract

from arXiv · show

Recent advances in video editing have been largely driven by large-scale instruction-based datasets. However, existing datasets still suffer from two critical limitations. First, target videos are commonly produced by automatic editing models, which may introduce visible artifacts and unreliable supervision signals. Second, most public datasets rely primarily on textual instructions, while lacking visual references that are crucial for precise, identity-preserving, and controllable editing. To address these limitations, we introduce RefVideo-6M, a large-scale reference-guided editing dataset containing 5 million video editing samples and 1 million image editing samples. To ensure reliable supervision, our dataset uses a construction pipeline that treats artifact-free real videos as editing targets and generates quality-filtered input conditions with multiple editing experts. In addition, it provides approximately 6 million visual references, covering diverse reference types and editing scenarios, thereby enabling models to learn fine-grained visual correspondence beyond text-only instructions. Based on RefVideo-6M, we further train a reference-guided video editing model, Ref-MoT, to evaluate the effectiveness and scalability of the proposed dataset. Extensive experiments demonstrate that RefVideo-6M provides substantially more reliable supervision than existing datasets and enables the training of powerful editing models with improved visual quality, controllability, and reference consistency. The open-source dataset is available at https://huggingface.co/datasets/RefVideo6M/RefVideo6M.

1 Introduction

RefVideo-6M addresses unreliable supervision and limited visual guidance in existing video-editing datasets with a large, reference-based dataset and the RefMoT model. It combines expert-generated, quality-filtered editing pairs with diverse references for controllable editing.

  • Existing video-editing datasets commonly use model-generated targets and primarily textual instructions, limiting supervision reliability and visual guidance.
  • RefVideo-6M contains 5 million 720p video pairs with 81 to 129 frames and 1 million image pairs.
  • The dataset reverses source and edited videos for most tasks, using edited videos as sources and original videos as targets.This construction is intended to reduce artifacts from editing pipelines.
  • RefMoT adapts an instruction-based editing model for reference-based editing, reducing training computation by 50% while maintaining comparable inference cost.
  • Training on RefVideo-6M enables state-of-the-art performance among the reported baselines and variants.
  • RefVideo-6M provides 6 million references across 10 types, covering both editing location and editing appearance.Reference types include region guidance such as bounding boxes and circles, plus style, texture, background, object, lighting, outpainting, and clothing references.

2 Related Works

Prior video-editing datasets vary in scale, task coverage, and construction strategy, but many rely on existing editing models to generate targets and mainly support textual guidance. Video-editing methods increasingly use instruction-based and unified multimodal frameworks, while performance remains closely tied to training-data quality.

  • 2.1 Video Editing Datasets: Existing datasets span approximately 300K to 2M video pairs and cover different numbers of editing tasks and categories.Senorita-2M covers 18 tasks, InsViE contains 1M pairs, Ditto contains 1M pairs, ReCo contains 500K pairs, and FFP contains 300K pairs.
  • 2.1 Video Editing Datasets: Many datasets use existing editing models to generate edited videos as ground-truth targets, which may introduce annotation artifacts.
  • 2.1 Video Editing Datasets: These datasets typically accept textual instructions rather than both textual and visual guidance.
  • 2.2 Video Editing Methods: Instruction-based editing methods require training but offer faster inference, simpler prompts, and improved visual quality compared with early inversion-based approaches.
  • 2.2 Video Editing Methods: Unified models increasingly combine video generation, editing, and understanding within a single framework using multiple modalities.
  • 2.2 Video Editing Methods: For a given model architecture, higher-quality editing datasets generally lead to stronger editing performance.

3 Methodology

RefVideo-6M combines expert-produced, quality-filtered editing data with diverse visual references, while RefMoT adapts instruction-based editing models for efficient reference-guided control.

  • Dataset Construction: RefVideo-6M contains 5 million video samples across 26 tasks and 1 million image samples across 6 tasks.The video samples use 720p resolution and 81 to 129 frames.
  • Reference Data: The dataset provides six million references spanning location, direction, and appearance controls for precise editing of object, human, background, and other content.References include bounding boxes, circles, masks, direction images, and appearance references.
  • Dataset Construction: Twelve editing experts produce the video subset, while LLM filtering removes visible artifacts and checks consistency among source, target, references, and instructions.The filtering process is iteratively refined with three human annotators per task.
  • RefMoT Architecture: RefMoT routes reference tokens through trainable Mixture-of-Tokens layers while freezing the main branch and sharing AdaLN layers with vision tokens.This preserves Stage-1 instruction understanding and edit localization while encoding references as visual guidance.

4 Experiments

Experiments show that RefVideo-6M improves instruction-based editing and that RefMoT achieves strong reference-based editing performance, while ablations support both the data and architecture designs.

  • Instruction-based Editing: The model achieves the best overall performance under both GPT-5.5 and Gemini-3-Pro evaluations for instruction-based editing.Under Gemini-3-Pro, its Overall Score is 4.56 versus 4.28 for the second-best method.
  • Reference-based Editing: The reference-based model improves Overall Score from 3.73 to 4.03 against UniVideo under GPT-5.5 and from 3.98 to 4.16 under Gemini-3-Pro.It leads on most reported metrics, although UniVideo slightly outperforms it in Attribute Alignment under Gemini-3-Pro.
  • Ablation Study: On ablations using up to 200K videos per dataset, RefVideo-6M raises Overall Score from 3.32 to 3.88 under GPT-5.5 and from 3.21 to 3.90 under Gemini-3-Pro.Overall instruction alignment also improves from 2.42 to 3.68 and from 2.16 to 3.48, respectively.
  • Ablation Study: The full RefMoT model further raises Overall Score from 3.97 to 4.22 under GPT-5.5 and from 4.08 to 4.56 under Gemini-3-Pro.The ablation compares the full model with the strongest architectural variant.

5 Conclusion

RefVideo-6M is a reliable, reference-based editing dataset with broad task and reference coverage, and experiments report strong performance from models trained on it and from RefMoT.

  • Dataset: RefVideo-6M contains 5M video pairs and 1M image pairs covering 26 video tasks and 6 image tasks.The video subset is produced by 12 editing experts at 720p with 81 to 129 frames.
  • Dataset: Using edited videos as sources and original high-quality videos as targets for most tasks reduces artifacts from editing pipelines.The dataset supports 10 reference types that specify editing location and appearance.
  • Results: Models trained on RefVideo-6M achieve state-of-the-art performance, while RefMoT further improves results for reference-guided editing.

A Training Details of RefMoT

The video editor is trained in two stages: first for general source-to-target instruction editing, then for reference-conditioned adaptation using auxiliary visual streams.

  • Stage 1: Stage 1 learns source-to-target editing from paired videos and natural-language instructions.Source and target videos are encoded into the latent space of a pretrained video VAE.
  • Stage 2: Stage 2 adds reference, bounding-box, circle, and relighting images as visual conditions for reference-guided editing.Condition images are repeated temporally, encoded by the same VAE, and injected as auxiliary transformer streams.
  • Training Objective: Both stages use flow matching: noise is added to target video latents and the transformer predicts the corresponding velocity target.The source video is the main visual condition and the editing instruction is encoded by a frozen multimodal text encoder.
  • Training Strategy: The two-stage design separates general editing alignment from reference-conditioned adaptation, supporting both text-guided and reference-guided video editing.The separation is intended to make training more stable and modular.
  • Implementation: Training uses AdamW, mixed precision, warm-up, and periodic validation sampling with different classifier-free guidance scales.

B.1 Global Stylizer

The global stylizer propagates edits made to a video's first frame across the full sequence, while addressing motion preservation, structural control, and training–test misalignment.

  • Global Stylizer: First-frame edits are propagated across the entire video, but extracted propagation conditions can discard motion cues and fine-grained background details.This can produce results with degraded motion and background fidelity.
  • Architecture: The architecture adapts Wan2.1-1.3B by placing edited-first-frame reference latents in the first temporal slot and concatenating noisy and source latents.A vision embedder extracts reference tokens for conditioning.
  • Training Data: Training on Ditto videos can produce overly static edits because extracted conditions lose fine-grained source details and motion dynamics.The filtering strategy uses frame-feature similarity as an indicator of motion variation.
  • Dataset Construction: Editing experts generate source videos for local stylization, leaving original videos as artifact-free targets for training pairs.This reverses the usual source–target arrangement to avoid editing-induced flaws in targets.
  • Architecture: A ControlNet-style inpainting design combines structural conditions with video inpainting to modify color and texture while preserving edited-region structure.The control and inpainting branches exchange hidden states throughout the network.
  • Limitations: At test time, spatial shifts or local distortions in edited first frames create a train–test discrepancy that can impair reliable edit propagation.Pixel-aligned training makes the model sensitive to misalignment.

B.4 Human Swap

The human swap model uses pose-conditioned control and mixed mask geometry to replace subjects while preserving pose and reducing sensitivity to mask shape.

  • Motivation: Existing object swap editors often fail to preserve pose consistency in human swaps, motivating a dedicated human swap model.The limitation is specific to transferring general object-swap methods to human subjects.
  • Architecture: The model uses a control branch conditioned on pose and noisy latents, while the main branch receives only noisy latents.Pose guidance is injected through the control branch.
  • Data Preparation: Training videos are collected from Pexels, filtered for valid human subjects, and processed with OpenPose and foreground masks.OpenPose extracts a pose map for each person.
  • Mask Design: Inpainting results correlate with input-mask shape, which can entangle mask geometry with the source subject's appearance attributes.A woman-to-man swap may incorrectly retain long hair when using the woman's original mask.
  • Mask Design: Using rectangular masks for 50% of samples and original masks for 50% makes the model less sensitive to detailed mask geometry.Rectangles provide less shape-specific constraints while original masks preserve accurate foreground regions.
  • Training: The model trains with simple target-category prompts, relying on pose conditioning to preserve human structure and motion.Examples include prompts such as “a man” or “a woman”.

C Video Dataset Construction Details

The video construction pipeline samples source videos, generates varied edits and visual references, propagates first-frame changes when applicable, and filters outputs for quality and consistency.

  • Editing Process: First-frame edits are generated with Flux2-Klein-9B or Nano Banana Pro and propagated to full videos using the distilled global stylizer.Several pipelines use 720P videos with 81 frames and 10 sampling steps.
  • Quality Filtering: GPT-5.2 filtering removes visible artifacts and motion inconsistencies after human annotators refine the filtering instructions.The process is applied after edited videos are generated.
  • Instruction Generation: The pipeline samples videos from a pool and uses language models to generate editing styles, factors, trajectories, and instructions.Prompts cover styles, weather, season, time of day, relighting, and camera trajectories.
  • Reference Construction: Reference images are generated or rendered for the editing task and verified for correspondence with the intended condition.Style references are checked by GPT-5.2, while relighting references encode direction explicitly.
  • Reference Construction: Relighting references are constructed as rendered arrows that specify the desired lighting direction on a blank canvas.Qwen3-VL-8B parses the instruction before OpenCV renders the reference.
  • Camera Trajectory: Camera-trajectory data uses one randomly selected trajectory to synthesize a source video while retaining the original video as the target.GPT-5.2 estimates the original trajectory for target annotations.

C.1.5 VFX Editing

The construction details cover VFX, object removal and addition, recolor, retexture, and object-swap data, using edited sources, original targets, references, and task-specific filtering.

  • VFX Editing: 654 successful VFX examples were generated across 7 instruction categories using Runway Gen-4 Aleph.Each final video has 121 frames at 720P.
  • Object Removal: Object-removal data uses 800K video–mask–object triplets, with object-removed videos as sources and original videos as targets.GPT-5.2 and human feedback filter the resulting pairs.
  • Reference Construction: Reference conditions include bounding boxes, circles, and object images, with GPT-5.2 used to verify generated references or filter failed generations.These references support object addition and related localized editing tasks.
  • Object Addition: Object-addition data generates a compatible object in the first frame and propagates it through the video, while original videos serve as targets for removal pairs.The first-frame-guided pipeline improves controllability of object insertion.
  • Local Editing: Object recolor and retexture use 800K triplets each, with local stylization producing altered source videos and original videos retained as targets.Recolor and texture references include geometric references and generated object or texture images.
  • Object Swap: Object-swap data selects 1.8M triplets and uses VACE to generate replacement videos, followed by filtering for artifacts, motion, blur, and similarity issues.The relatively low editing success rate requires a large candidate set.

D Image Dataset Construction Details

The image dataset covers six reference-based editing tasks, using reference-conditioned target generation, source derivation, instruction generation, and quality filtering. Different subsets encode object identity, texture, human identity, or style through source–reference–target relationships.

  • Dataset structure: Six image-editing subsets cover style transfer, object addition, object retexture, object swap, human swap, and virtual try-on.Each example is organized as a source image, reference image, target image, and editing instruction.
  • Construction pipeline: The general construction pipeline generates a reference-conditioned target, derives the source image, and automatically generates instructions with visual quality checks.
  • Style transfer: Style-transfer samples derive realistic sources by reversing stylization, suppressing residual color and texture, and recoloring the grayscale result.This procedure is designed to preserve the underlying image content while removing the target stylization.
  • Style transfer: Style-transfer references encode representative texture and color attributes extracted from the stylized target and combined with style and content descriptions.The final triplet pairs a realistic source with a style reference and stylized target.
  • Object-centered edits: Object-addition samples use clean reference images on white backgrounds, then derive source images by removing the referenced object from the generated target.
  • Object-centered edits: Retexture samples preserve object identity, layout, and background while changing the object's texture between source and target; filtering checks reference–target texture consistency and source–target difference.The reference image specifies texture rather than object identity.
  • Identity edits: Object swap and human swap preserve scene context while replacing the source object or person with the reference-specified entity in the target.Human swap extends object replacement to identity-specific categories and preserves pose context and unrelated regions.

D.6 Virtual Try-On

The virtual try-on subset adapts the human-swap protocol to clothing references, generating target outfits and source images with alternative garments while preserving the person and scene.

  • Virtual try-on: Virtual try-on uses clothing references rather than identity references, covering garments and accessories such as skirts, hats, and coats.
  • Virtual try-on: The target image depicts a person wearing the reference clothing, while the source uses an alternative garment and preserves the person, pose, background, and global layout.Each example therefore represents a clothing replacement task guided by a visual reference.
  • Evaluation configuration: Matched relative timestamps are used for source and output frames in the automatic evaluation configuration.

E Alignment Between MLLM-based Video Evaluation and Human Preference

The study evaluates whether MLLM judges align with human preferences for instruction- and reference-guided video editing, using shared outputs, strict criteria, and complementary agreement metrics. GPT-5.5 and Gemini-3-Pro achieve high agreement with human labels, while the user study separately compares systems across editing criteria.

  • Evaluation design: The alignment study uses 30 shared editing inputs and outputs from three systems, producing 90 videos evaluated by humans and two MLLM judges.Reference-guided cases include an injected reference image when enabled by the metadata.
  • Human preference: Human annotations include binary labels, rationales, and confidence levels; all 90 samples were completed, with 35 positives, 55 negatives, and high confidence on 80 samples.
  • Evaluation criteria: Only perfect or nearly perfect outputs receive positive labels; evaluation jointly considers task completion, reference use, preservation, temporal consistency, artifacts, and visual quality.
  • MLLM alignment: 94.44% GPT-5.5 agreement and 92.22% Gemini agreement with human binary preferences support MLLM-based evaluation for large-scale binary screening within this study.
  • MLLM alignment: Gemini achieves higher positive recall at 97.14%, whereas GPT-5.5 achieves higher negative specificity at 96.36% and performs better on most aggregate classification measures.
  • User study: The user study recruits 33 participants for two 12-case parts, comparing 15 methods without references and three methods with reference images.Part 2 adds reference identity as a criterion alongside text alignment, visual quality, motion consistency, and background preservation.
  • User study: The user-study interfaces separately support instruction-based editing without visual references and reference-based editing with visual references.

G Details of the MLLM-Based Benchmark Evaluation

The benchmark uses two independent MLLM judges to evaluate instruction- and reference-based video editing under a shared protocol and diverse visual guidance conditions.

  • Benchmark configuration: The benchmark contains 100 instruction-based and 100 reference-based editing cases, with reference guidance including objects, textures, bounding boxes, and circle-marked regions.GPT-5.5 and Gemini-3-Pro receive the same input construction, question set, and scoring procedure.

G.1 Evaluation Inputs

The evaluation presents sampled edited-video frames, editing instructions, and optional visual references to an MLLM judge. Task-specific and task-agnostic questions are scored on a five-point scale across instruction adherence, attribute matching, preservation, visual quality, and undesired-object avoidance.

  • Evaluation Inputs: Each edited video is represented by three temporally ordered frames, resized while preserving aspect ratio, alongside its editing instruction and optional visual reference.References may depict an object, texture, bounding box, or circle-marked image.
  • Evaluation Questions: Task-specific questions assess whether the requested edit and attributes are satisfied while unrelated regions remain preserved.Examples cover texture re-editing and replacing a chick with a ginger kitten.
  • Evaluation Questions: Two task-agnostic questions assess visible artifacts and unintended objects in every evaluation case.These questions are appended after the task-specific questions.
  • Evaluation Inputs: The prompt template omits the reference-image block for instruction-based cases that do not require visual guidance.Image placeholders are replaced by the corresponding inputs in the multimodal request.
  • Evaluation Dimensions: Scores use five dimensions: Instruction Alignment, Attribute Alignment, Background Preservation, Visual Quality, and No Undesired Object.Each question receives an integer score from 1 to 5, with 1 indicating worst fulfillment and 5 indicating best fulfillment.
  • Aggregation and Reporting: Valid question scores are averaged within each dimension and reported separately for GPT-5.5 and Gemini-3-Pro.Per-video predictions, question-level scores, and textual justifications are retained for reproducibility and qualitative error analysis.
Loading 2608.26101v1…