Source-linked AI summary
Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
Qingyan Bai, Qiuyu Wang, Hao Ouyang, Yue Yu, Hanlin Wang, Wen Wang, Ka Leong Cheng, Shuailei Ma, Yanhong Zeng, Zichen Liu, Yinghao Xu, Yujun Shen, Qifeng Chen
TL;DR
Instruction-based video editing is limited by scarce large-scale, high-quality paired data and difficult trade-offs among diversity, fidelity, coherence, and cost. Ditto combines image-guided in-context video synthesis, efficient distilled generation, temporal enhancement, and agentic curation to build Ditto-1M, then trains Editto with curriculum learning. The resulting model demonstrates superior instruction following and state-of-the-art instruction-based video editing performance.
Problem
Large-scale, high-quality, and diverse paired data for end-to-end instruction-based video editing remains scarce, while existing synthesis methods trade off scalability against diversity, temporal consistency, and visual quality.
Method
Ditto generates edited reference frames with an instruction-based image editor, uses them to guide an in-context video generator, reduces cost with distillation and a temporal enhancer, and applies VLM-based curation.
Results
Editto trained on Ditto-1M demonstrates superior instruction-following ability and establishes a new state-of-the-art in instruction-based video editing.
Takeaways & Limitations
Ditto-1M provides a million-scale collection of high-fidelity instruction-video examples, while modality curriculum learning enables Editto to perform edits from text instructions alone.
Abstract
from arXiv · showhide
Instruction-based video editing promises to democratize content creation, yet its progress is severely hampered by the scarcity of large-scale, high-quality training data. We introduce Ditto, a holistic framework designed to tackle this fundamental challenge. At its heart, Ditto features a novel data generation pipeline that fuses the creative diversity of a leading image editor with an in-context video generator, overcoming the limited scope of existing models. To make this process viable, our framework resolves the prohibitive cost-quality trade-off by employing an efficient, distilled model architecture augmented by a temporal enhancer, which simultaneously reduces computational overhead and improves temporal coherence. Finally, to achieve full scalability, this entire pipeline is driven by an intelligent agent that crafts diverse instructions and rigorously filters the output, ensuring quality control at scale. Using this framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of one million high-fidelity video editing examples. We trained our model, Editto, on Ditto-1M with a curriculum learning strategy. The results demonstrate superior instruction-following ability and establish a new state-of-the-art in instruction-based video editing.
1. Introduction
Instruction-based video editing trails image editing because temporally coherent, high-quality paired training data remains scarce. Ditto addresses this through scalable synthesis, efficient generation, agentic curation, and the Ditto-1M dataset.
- Motivation: Video editing must modify content while propagating changes coherently across frames, making paired training data difficult to obtain.The field’s video capabilities lag behind image editing, where instruction-based methods have advanced substantially.
- Motivation: Existing synthetic-data pipelines trade off editing diversity, temporal consistency, visual quality, and scalability.Prior approaches include computationally expensive per-video optimization and training-free image-to-video propagation.
- Ditto framework: Ditto uses an edited reference frame as a visual prior for an in-context video generator, producing edits that match the instruction with temporal coherence.The reference frame comes from advanced instruction-based image editors and guides video synthesis.
- Ditto framework: 20% of the original computational cost is achieved by combining a distilled video model with a temporal enhancer while preserving temporal stability and avoiding visual artifacts.The architecture targets the efficiency–coherence trade-off in high-fidelity generation.
- Dataset: Over 12,000 GPU-days produced Ditto-1M, a dataset of over one million source-instruction-edited video triplets curated for broad task coverage and quality.A VLM agent supports instruction consistency and high aesthetic quality during curation.
- Model training: Editto is trained on Ditto-1M with modality curriculum learning that gradually removes edited-reference-image guidance to enable instruction-only inference.Training begins with text plus an edited reference image and anneals the visual scaffold over time.
2. Related Work
Prior work advanced instruction-based image editing and video editing through increasingly capable generative models, but video editing still faces temporal-consistency and scalability challenges. Ditto builds on image-editor progress while using in-context generation primarily for scalable, high-quality data synthesis.
- Instruction-based image editing: InstructPix2Pix showed that generated image triplets and language-model instructions can support instruction-based image editing.This work helped establish text-guided, user-directed image modification as a practical paradigm.
- Instruction-based image editing: Recent image-editing systems support targeted local edits, character consistency across multiple turns, and complex scene transformations without requiring fine-tuning.Ditto uses a state-of-the-art instruction-based image editor to manipulate keyframes for subsequent video editing.
- Instruction-based video editing: Instruction-based video editing must preserve temporal consistency and background content, motivating inversion-based and zero-shot approaches.Inversion-based methods can be computationally intensive, while Tune-A-Video lacks scalability despite enabling personalized edits.
- Ditto’s position: Ditto differs from concurrent EditVerse by using in-context generation primarily for high-quality data synthesis rather than unifying editing tasks.Its Modality Curriculum Learning strategy bridges multimodal synthesis and text-only inference for Editto.
3. Ditto-1M
Ditto-1M is constructed through a scalable pipeline that generates instruction-video triplets using language, edited-frame, and depth guidance, then filters and enhances them. The resulting dataset contains approximately one million edited videos with broad task coverage and high-resolution outputs.
- Pipeline design: Ditto addresses limited editing diversity and fidelity by combining image-editor priors with depth-guided in-context video generation.Edited key-frames guide appearance, while depth video provides spatiotemporal structure.
- Instruction generation: The pipeline generates instructions in two VLM stages: dense video captioning followed by creative, context-grounded edit commands.This produces global style edits and localized object modifications.
- In-context video generation: The in-context generator synthesizes edited videos from the instruction, edited key-frame, and depth video without costly per-video optimization.The edited frame supplies appearance guidance, while depth constrains motion and structure.
- Curation and enhancement: The pipeline applies VLM filtering and denoising enhancement to improve instruction fidelity, source-video preservation, and visual quality.These operations form the final curation and refinement stage.
- Training pipeline: Curriculum learning trains context blocks by gradually annealing and eventually dropping the reference frame.This training design transitions from visual guidance toward instruction-driven editing.
- Ditto-1M dataset: Over 12,000 GPU-days produced approximately 1M edited videos, including about 700k global-editing and 300k local-editing triplets.Final samples are 1280x720, 101 frames, and 20 FPS.
4. Modality Curriculum Model Learning
Editto adapts an in-context video generator to combine visual context with textual instructions, then uses modality curriculum learning to enable text-only video editing.
- Architecture: The adapted architecture uses a Context Branch for source-video and reference-frame features and a DiT-based Main Branch for synthesizing edited videos.The Main Branch receives joint visual-context and textual guidance.
- Modality Curriculum Learning: Modality curriculum learning initially supplies the edited reference frame as a visual scaffold alongside the instruction, then gradually removes it during training.This annealing shifts the model toward learning the mapping from text instructions alone.
- Training Objective: The model is trained with a flow-matching objective that predicts a vector field from a noised latent toward the clean edited-video latent.Here, z0 is the clean target latent, zt is its noised version, and c contains text and visual conditioning.
5. Experiments
Experiments evaluate Editto through automatic metrics, human judgments, qualitative comparisons, additional transfer results, and ablations of data scale and curriculum learning.
- Quantitative Comparison: The out-of-distribution test set contains 50 online videos, each paired with 5 distinct editing instructions, and is evaluated with automatic metrics and a user study.The user study includes 1,000 votes from postgraduates and researchers.
- Quantitative Comparison: Editto significantly outperforms all baselines across automatic metrics and receives strong human preference for instruction adherence, temporal smoothness, and visual quality.The automatic evaluation uses CLIP-T, CLIP-F, and a VLM score; human evaluation measures Edit-Acc, Temp-Con, and Overall.
- Qualitative Comparison: Qualitatively, Editto better handles complex stylizations and local attribute edits while preserving temporal coherence, identity, and background details.For the black-suit example, Gen4-Aleph slightly changes identity and other methods largely fail.
- Additional Results: The learned model supports synthetic-to-real translation and outperforms the raw data generator on newly emerged content beyond key frames.The authors attribute this superiority to scaled training, curriculum learning, and filtered high-quality data.
- Ablation Studies: Increasing training-data scale improves stylistic-edit quality and content-and-motion fidelity, while removing MCL often harms interpretation of the instruction’s full semantic intent.These ablations support the value of both Ditto-1M scale and modality curriculum learning.
6. Conclusion
Ditto addresses video-editing data scarcity through scalable synthetic data generation, enabling Ditto-1M and state-of-the-art Editto performance via modality curriculum learning.
- Ditto combines strong image-editing priors, a distilled in-context video generator with a temporal enhancer, and autonomous VLM-based quality control.
- The pipeline resolves prior fidelity-diversity and efficiency-coherence trade-offs in synthetic video-editing data generation.
- Ditto enables construction of the large-scale, high-quality Ditto-1M dataset.
- Modality curriculum learning transitions Editto from visual-textual conditioning to purely instruction-driven inference.
- Editto achieves state-of-the-art performance using Ditto-1M and the proposed modality curriculum learning strategy.