EasyV2V: A High-quality Instruction-based Video Editing Framework

• EasyV2V unifies data processing, architecture, and control for instruction-based video editing.Source: Figure 1 · p. 1
• Figure 1 illustrates flexible inputs, including text, masks, and edit timing, across diverse editing tasks.Source: Figure 1 · p. 1

• Instruction-based video editors still lag image-based counterparts in visual fidelity and control, despite broader edit coverage than task-specific systems.Training-free editors are described as fragile and slow, while training-based systems often target narrower tasks.Source: §1 · p. 1

Data

• The data strategy composes existing experts with fast inverses, lifts image edits into pseudo-video pairs using shared smooth camera transforms, and mines dense-captioned clips for action and transition supervision.Image-derived pairs increase scale and instruction variety but provide weak motion supervision; shared camera trajectories add temporal structure without changing semantics.Source: §1 · p. 2

Architecture

• Sequence concatenation preserves separate source and target streams and improves edit quality, while LoRA fine-tuning adapts the pretrained backbone with less overfitting than full training.The design also uses patch-embedding routes for source video and masks, a frozen VAE, and optional reference images.Source: §1 · p. 2

Flexible control

• A single editable mask video unifies spatial and temporal control: pixels mark where to edit, while frames specify when edits begin and how they evolve.The mask sequence is lightweight but required for this control mode.Source: §1 · p. 2

• EasyV2V consistently improves edit quality, motion, and instruction following over recently published methods while supporting flexible input combinations.Source: §1 · p. 2

Instruction-based visual editing datasets

• Video-edit datasets remain harder to create than image-edit datasets because they must maintain temporal coherence while faithfully applying edits; prior synthetic efforts still lag in quality and diversity.Source: §2 · p. 3

Instruction-based visual editing

• Training-free diffusion editing often has low output quality, slow inference, or low success rates, while paired-data training can outperform it.Source: §2 · p. 3
• Prior video editors span training-free pipelines, image-model adaptations, propagated first-frame edits, and task-specific conditioning; EasyV2V systematically studies training data and adds reference-image and spatiotemporal control.Source: §2 · p. 3

Input video conditioning

• Source-video tokens are concatenated with noisy-target tokens along the sequence dimension, a strategy found more robust and effective than channel-wise concatenation.The source tokens precede target tokens to support in-context editing behavior resembling video continuation.Source: §3 · pp. 3–4

Architecture

• Separate patch embeddings encode source video and mask signals; the frozen backbone is adapted by optimizing the new embeddings and LoRA weights rather than all model weights.Figure 3 shows mask tokens added to input-video tokens and the DiT trained with LoRA.Source: §3 · p. 3, Figure 3 · p. 4

Mask conditioning

• A binary mask marks where and when to edit, with encoded mask latents added to source-video tokens; without a supplied mask, inference defaults to a blank mask.Source: §3 · p. 4

Training data

• The training-data summary reports approximately 4.3M collected open-source and licensed examples plus 3.4M curated examples, totaling approximately 8M.Source: Table 1 · p. 4

Reference image conditioning

• Optional reference images can come from the target video, an external image editor, or the user; random reference dropping during training improves robustness to absent or noisy references.Reference tokens are placed near target tokens for stronger guidance while preserving a fixed source-to-target token distance.Source: §3 · p. 4

4.1. V2V data

• Expert pipelines generate paired videos for human animation, object removal or insertion, actor transmutation, stylization, and controllable generation.Human animation uses pose, facial-region, and reference-image conditioning; object edits use detection, segmentation, and inpainting; other pipelines expand actor categories and styles.Source: §4.1 · pp. 4–5
• Transition data varies edit onset and applies transition operators, such as linear blending, to supervise how edits unfold over time.Frame-wise masks activate edits after the chosen onset.Source: §4.1 · p. 6

4.2. I2I data

• Image-edit pairs become pseudo-video pairs by applying the same smooth affine camera trajectory to source and target images, preserving the edit while adding temporal structure.Trajectories combine small rotations, zooms, and translations, with adjustments to avoid out-of-frame regions.Source: §4.2 · p. 6

4.3. Dense-captioned text-to-video data

• Dense-captioned clips become source-video, target-video, and imperative-instruction triples by pairing frames before a caption’s time window with frames containing its described action.Windows that are too long, too short, or contain scene cuts are discarded; curated action edits support accurate and complex human-action edits.Source: §4.3 · p. 6

5. Implementation details

• Wan-2.2-TI2V-5B is the base model, paired with Wan-2.2-VAE, and reported results use 81×832×480 resolution.The supplement also provides results fine-tuned at 81×1280×704.Source: §5 · p. 6
• LoRA uses rank 256, a constant 1e−4 learning rate, and AdamW, with training run on 32 NVIDIA H100s.Source: §5 · p. 6
• All newly introduced parameters are zero-initialized, and reference-image dropout and video-transition augmentation each occur with 50% probability.Source: §5 · p. 6

6.1. Benchmarks and metrics

• Evaluation covers 160 videos across 16 supported EditVerseBench edit types, excluding unsupported tasks such as camera-pose changes, and uses GPT-4o VLM scores as the primary metric.The benchmark has 20 edit types; the study also reports frame/video text alignment and PickScore.Source: §6.1 · p. 6

6.2. Baseline comparison

• 7.73/9 VLM score without guidance: EasyV2V outperforms the previously best published method, concurrent work, and a commercial solution.The benchmark comparison is reported in Table 2.Source: §6.2 · p. 6, Table 2 · p. 7
• Qualitative comparisons report higher-quality outputs and better instruction following than baselines, including examples of missed edits and removed background trees.The comparison includes EditVerse, InsViE-1M, Señorita-2M, and Lucy Edit Dev; the passage also describes their specific limitations.Source: §6.2 · p. 7, Figure 5 · p. 8

6.3. Ablation study

• Sequence-wise concatenation of control latents outperforms patch-embedding addition, while full-model training tends to overfit and LoRA quickly adapts a text-to-video model for video editing.Architecture protocols are evaluated by VLM score at 20K and 40K training steps.Source: §6.3 · p. 7, Table 3 · p. 7
• Affine transformations improve downstream video-editing performance, and joint training on image-editing data outperforms training solely on video-editing data.The ablation compares models trained with equal numbers of image-to-image and video-to-video pairs.Source: §6.3 · p. 7, Table 4 · p. 7
• Performance improves with training-data size for both seen and unseen edits, while 10K examples already yield fair performance.The study tests 10K, 100K, and 1M samples.Source: §6.3 · p. 8, Figure 6 · p. 8

7. Conclusion

• The framework combines diverse data strategies with a parameter-efficient architecture and unified mask-based spatiotemporal control, with optional reference images.The conclusion describes this combination as a recipe for high-quality editing with extensive controllability.Source: §7 · p. 8
• Inference takes around one minute, precluding EasyV2V's use in real-time applications.Source: §7 · p. 8
• Geometric cinematographic camera-pose control is identified as a potential extension to broaden creative applications.Source: §7 · p. 9

A. Performance on image editing

• The evaluation treated each image as a single-frame video at 1 × 832 × 480 resolution, although the model was not designed specifically for image editing.Source: A. Performance on image editing · p. 10
• 0.54: EasyV2V outperformed EditVerse, surpassing all baselines on nearly all ImgEdit Bench subtasks and matching leading closed-source and commercial systems.Source: A. Performance on image editing · p. 10
• EasyV2V also showed strong action and hybrid editing results, alongside a training pipeline combining image-editing and human-action video-caption data.Source: A. Performance on image editing · p. 10

B.1. Mask conditioning

• Tested mask strategies include adding encoded masks to source or target latents, downsampling masks, and concatenating source, target, mask, and reference latents.Source: §B.1 · p. 10
• EasyV2V's mask-token addition to the source latent performs best; both it and sequence concatenation outperform WanVACE on mask-conditioned editing.Source: §B.1 · p. 10
• Figure II compares mask-guided edits on EditVerseBench samples; WanVACE shows weaker prompt generalization and spatial and temporal mask adherence than EasyV2V.Source: §B.1 · p. 10

B.2. LoRA Rank

• LoRA performance improves with rank, but the gap between ranks 128 and 256 is marginal; rank 256 is adopted as performance begins to saturate.Source: §B.2 · p. 10

B.3. Time and memory profiling

• Training and inference costs were profiled on one NVIDIA H100 across four full-model or LoRA tuning and sequence-concatenation or token-addition strategies.FlashAttention was enabled for both training and inference; the provided passages do not report the cost comparisons themselves.Source: §B.3 · pp. 10–11

C. Additional details on data pipelines

• Prompt assets cover human animation and object categories, scenes, and actions; video stylization uses 350 generated prompts.Source: C. Additional details on data pipelines · pp. 11, 13
• A manually filtered 15K-sample in-the-wild video collection is paired through model-free and model-based transformations, including pose, edges, depth, color changes, blur, and masks.Source: C. Additional details on data pipelines · p. 13
• Dense-captioned clips are filtered for length, action duration, preceding frames, and scene cuts; Qwen-3-4B converts suitable action descriptions into imperative edit instructions.Source: C. Additional details on data pipelines · p. 13
• For image-edit pairs, Qwen3-4B-Instruct generates up to five diverse edit instructions per caption, image editors execute them, and Gemma-3-27B filters candidate pairs.Source: C. Additional details on data pipelines · p. 13
• Image-edit pairs become pseudo-videos by applying the same interpolated affine camera trajectory to source and target, with bounds ensuring transformed content stays in frame.Trajectories vary rotation, zoom, and translation; half are reversed to balance zoom directions.Source: C. Additional details on data pipelines · p. 14

D. Impact of classifier free guidance

• Higher prompt-only CFG generally improves text alignment but can reduce temporal consistency and video quality; dual-condition CFG strengthens appearance and semantic control, with best scales between 3.0 and 5.0.The authors use s = 3.0 by default for prompt-only guidance and note that dual-condition guidance requires careful scale tuning.Source: D. Impact of classifier free guidance · p. 14, Table V · p. 14, Table VI · p. 15

E. User study

• EasyV2V outperforms other methods across instruction alignment, preservation of unedited regions, and video quality in the user study.The benchmark contains 160 horizontal and vertical videos spanning 18 edit types.Source: E. User study · pp. 14–15, Figure V · p. 16

F.1. Robustness to Reference Image

• EasyV2V is reported robust to the choice of reference image, can rectify zoom and pose inconsistencies, and preserves the source background better without an external reference.The tested references are derived from the source video's first, middle, or last frame.Source: §F.1 · p. 15, Figure VI · p. 17

F.2. Comparison on Human Animate and Flow Edit Datasets

• Training on Human Animate often yields better visual detail, pose preservation, unseen human pose-to-video generalization, and facial-expression preservation than training on Flow Edit.Human Animate contains human-centric edits; Flow Edit focuses on actor transmutation edits.Source: §F.2 · p. 15, Figure VII · p. 18

F.3. Exhibition Gallery and Visualization

• Dense-captioned video data supports human-action edits, while frame-wise temporal masks elicit smooth, realistic transitions despite simple blending-transition training.The transition evaluation restricts edits to the second half of the video.Source: §F.3 · p. 15
• EasyV2V edits approximately 720P videos at 81 frames; high-resolution training uses about 6 million samples and converges rapidly.The reported resolution is 81 × 1280 × 704, the maximum supported by Wan-2.2-TI2V-5B.Source: §F.3 · p. 15