Source-linked AI summary
Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing
Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, Zhibo Chen
TL;DR
Existing instruction-based video editing datasets largely omit structural and simultaneous multi-task edits. Goku introduces a 2-million-pair dataset, Goku-Edit, and a human-verified benchmark, with Goku-Edit achieving up to an 8% improvement over existing state-of-the-art models in instruction following.
Problem
Existing video editing datasets largely remain limited to single-task, appearance-level modifications, omitting complex structural transformations and simultaneous multi-task editing.
Method
Goku combines a 2-million-pair dataset, a decomposed and progressively filtered synthesis pipeline, Goku-Edit, and the human-verified Goku-Bench.
Results
Up to 8% improvement over existing state-of-the-art models is achieved in instruction following on Goku-Bench.
Takeaways & Limitations
Goku extends instruction-based video editing evaluation to structural, multi-task, reference-guided, semantic, spatial, and temporal capabilities.
Abstract
from arXiv · showhide
Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.
1 Introduction
Goku addresses the narrow, appearance-focused scope of existing instruction-based video editing datasets with a 2-million-pair dataset spanning multi-task, structural, and reference-guided edits. It also introduces a scalable synthesis pipeline, Goku-Edit, and the human-verified Goku-Bench for complex video editing.
- Motivation: Existing datasets largely limit instruction-based video editing to single-task, appearance-level modifications, neglecting structural transformations and simultaneous multi-task editing.Examples include object removal and single-attribute alteration, while narrow problem definitions restrict task diversity.
- Goku Dataset: Goku contains 2 million high-quality, instruction-aligned video editing pairs spanning basic, multi-task, structural, and reference-guided editing.Structural categories include camera movement and subject movement, incorporated alongside fundamental appearance-level edits.
- Data Pipeline: Goku’s scalable, automated pipeline decomposes complex structural and multi-task instructions into independently controllable sub-problems handled by task-specific expert models.The pipeline is designed to ensure semantic precision and temporal coherence and uses VACE for basic appearance-level synthesis.
- Goku-Edit: Goku-Edit uses an MLLM text encoder and a dedicated mask prediction branch to connect high-level semantic reasoning with precise spatial manipulation.The mask branch enhances spatial grounding while allowing the main branch to focus on fine-grained rendering.
- Goku-Bench: 1,000 human-verified test cases and 7 specialized metrics define Goku-Bench, where Goku-Edit achieves up to an 8% improvement over existing state-of-the-art models in instruction following.The benchmark is designed for rigorous evaluation of complex video editing tasks and emphasizes reliability and fairness.
2 Related Work
Related work on instruction-based video editing addresses limited paired data through synthetic training paradigms and model designs, while existing datasets still face inconsistent filtering and uneven quality. Prior methods span triplet synthesis, training-free propagation, consistency modeling, and MLLM–video diffusion integration.
- Instruction-Based Video Editing: Instruction-based video editing has attracted attention for modifying videos with natural-language commands, but paired training data remains scarce.InsV2V synthesizes video triplets, and InsViE scales this paradigm.
- Instruction-Based Video Editing: Prior methods explore triplet synthesis, training-free first-frame editing with image-to-video propagation, and shape or temporal consistency.These approaches include InsV2V, InsViE, AnyV2V, and StableV2V.
- Video Editing Datasets: Existing video editing datasets often depend on editing models for generation but lack unified or sufficiently rigorous filtering criteria.InsViE and Señorita-2M can contain residual static videos and failed editing samples, producing uneven data quality.
3 The Goku Dataset
Goku is a large-scale, multifaceted instruction-based video editing dataset that extends beyond single-task appearance edits to structural, multi-task, and reference-guided challenges. Its automated collection and progressive filtering pipeline target semantic fidelity, temporal consistency, and instruction-following precision.
- Goku covers complex structural deformations, simultaneous multi-task interactions, and reference-guided editing beyond conventional appearance-level modifications.
- Data collection: 1 million high-quality video clips are curated from Koala-36M using shot transition detection, aesthetic scoring, motion analysis, OCR watermark removal, and content-richness screening.
- Data collection: Gemini2.5-Pro generates diverse structured instructions by identifying operable objects across over 200 categories for Add, Remove, Swap, and Subject Movement tasks.
- Data generation: The synthesis pipeline decomposes difficult edits into tractable sub-problems, including image-level editing followed by conditional video generation for object relocation.
- Progressive filtering: A three-tier progressive filtering system applies quality gates throughout preprocessing and synthesis, while human evaluation uses 100 samples per task and 3 annotators.
4 Goku-Edit Model
Goku-Edit uses a dual-branch architecture that combines video generation with auxiliary mask prediction for controllable structural editing. RoPE-aligned cross-attention, bidirectional branch interaction, and SpatialCFG address resolution misalignment, evolving structural guidance, and editing spillover.
- RoPE-Aligned Spatial Cross-Attention: RoPE-aligned spatial cross-attention resolves coordinate-grid misalignment when the mask branch operates at 1/n spatial resolution.Direct cross-resolution attention conflicts with RoPE’s dependence on relative token offsets.
- Dual-Branch Model Architecture: The model adapts pre-trained Wan2.2-5B in a primary video-editing branch and an auxiliary mask-prediction branch that provides spatial guidance.The video branch synthesizes the edited video, while the mask branch predicts the targeted editing region.
- Dual-Branch Model Architecture: A frozen Qwen3VL-8B processes text prompts for both branches to improve comprehension of complex editing instructions.A pre-trained VAE encodes the source, edited video, downsampled source, reference inputs, and mask into branch-specific composite latents.
- Bidirectional Cross-Attention: Video-to-mask attention closes the feedback loop by letting mask predictions refine from the main branch’s evolving state during denoising, reducing flickering and boundary incoherence.Without V2M, the mask branch relies only on the downsampled source video and can diverge from edited content.
- SpatialCFG: SpatialCFG is a training-free inference strategy that amplifies cross-branch spatial constraints by contrasting coupled and decoupled predictions.Its guidance suppresses unintended modifications outside the editing region and refines mask boundary consistency, while composing with standard text CFG.
5 Goku-Bench
Goku-Bench is a comprehensive benchmark designed to evaluate complex video editing capabilities beyond single-task appearance editing. It contains 1,000 diverse test cases and 7 editing-specific metrics, including measures for camera motion and style transfer.
- Benchmark scope: Goku-Bench addresses the limitations of appearance-focused benchmarks by evaluating multi-task and structural editing capabilities.It is designed to provide fine-grained evaluation for complex editing tasks.
- Benchmark scope: 1,000 diverse test cases and 7 editing-specific metrics comprise the Goku-Bench evaluation suite.The benchmark is intended to assess complex editing capabilities comprehensively.
- Test-set diversity: The final test set covers multi-person scenes, full- and half-body humans, animals, common objects, and natural scenes.Examples include dogs, cats, sharks, birds, clothing, vehicles, and buildings.
- Editing-specific metrics: For camera motion editing, optical-flow analysis identifies and evaluates motion types across video frames.This metric assesses camera-motion editing by analyzing motion across frames.
- Editing-specific metrics: For style transfer, DINO feature similarity between the reference style image and generated video frames quantifies style-transfer accuracy.The reference style image corresponds to the style text used for the editing task.
6 Experiment
Experiments on Goku-Bench show that Goku-Edit outperforms prior video editing methods across most metrics and task-related evaluations. Qualitative and ablation studies further demonstrate its advantages in structural, complex, and multi-task editing, as well as the importance of spatial alignment and SpatialCFG.
- Quantitative comparison: Goku-Edit achieves the best performance on most Goku-Bench evaluation metrics and significantly higher editing fidelity than other methods.The comparison includes representative open-source methods and closed-source commercial models.
- Qualitative comparison: Goku-Edit correctly pans the camera rightward and plausibly inpaints newly revealed regions, whereas several baselines produce no effective camera motion.The baselines are InsV2V, InsViE, Omni-Video, and LucyEdit.
- Qualitative comparison: Goku-Edit precisely removes a red hat while preserving the surrounding background, while baseline methods mislocalize the target and damage nearby content.This result comes from a complex-scene structural editing example.
- Data ablation: Models trained on existing datasets show limited multi-task editing performance because those datasets primarily focus on single-task appearance editing.The comparison controls network architecture, training scale at 50k samples and 6k steps, and evaluation on Goku-Bench.
- Component ablation: 0.819 SR is achieved with RoPE-aligned spatial cross-attention versus 0.789 without alignment, while SpatialCFG provides gains of +0.020 PR and +0.019 IF.RoPE alignment improves cross-branch positional correspondence and boundary precision; SpatialCFG suppresses editing spillover and enhances instruction following.
7 Conclusion
The work introduces Goku, a large-scale, high-fidelity dataset designed to address key challenges in instruction-based video editing through broad editing-pattern coverage and rigorous data creation.
- Dataset: Goku is a large-scale, high-fidelity dataset for instruction-based video editing.It is introduced to address key challenges hindering progress in the field.
- Dataset: The dataset comprehensively covers camera movement, subject movement, reference edit, and multi-task patterns.These patterns expand coverage across diverse editing scenarios.
- Data creation: A rigorous data creation pipeline is intended to ensure exceptional quality and diversity.The pipeline addresses static content, semantic mismatches, and poor visual quality found in existing datasets.