Source-linked AI summary
Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance
Yiqi Lin, Guoqiang Liang, Ziyun Zeng, Zechen Bai, Yanzhe Chen, Mike Zheng Shou
TL;DR
Text-only video editing lacks precise visual control, while reference-guided editing is constrained by scarce high-quality training quadruplets. The paper addresses both problems with an automated RefVIE data pipeline, RefVIE-Bench, and the Kiwi-Edit multimodal architecture. The resulting system achieves state-of-the-art controllable video editing performance.
Problem
Natural language cannot reliably describe precise visual details, and existing resources lack scalable high-quality source-video, instruction, reference-image, and target-video quadruplets.
Method
The paper synthesizes reference images from existing video editing pairs, constructs RefVIE and RefVIE-Bench, and uses a unified MLLM-DiT architecture with learnable queries and latent visual features.
Results
Kiwi-Edit achieves state-of-the-art performance on instruction-only and reference-guided video editing tasks.
Takeaways & Limitations
The data-centric pipeline and unified model provide an open foundation for more controllable and accessible video content creation.
Abstract
from arXiv · showhide
Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.
1. Introduction
Instruction-based video editing has advanced, but text-only control cannot reliably express precise visual details and reference-guided editing lacks scalable paired data. The paper addresses these gaps with RefVIE, RefVIE-Bench, and Kiwi-Edit.
- Text-only instructions struggle to specify precise textures, object identities, and stylistic characteristics, motivating visual references.
- Existing datasets provide instruction-based video pairs but generally lack reference images, while accessible reference-guided resources remain scarce.
- RefVIE synthesizes missing reference images from existing source-target video pairs using vision-language models, image editors, filtering, and de-duplication.
- Kiwi-Edit integrates a frozen MLLM with a DiT through query and latent connectors to guide video generation with multimodal conditions.
- The paper also introduces RefVIE-Bench and reports state-of-the-art performance across instruction-only and reference-guided tasks.
2. Related Work
Prior video editing methods increasingly add reference images to textual instructions because natural language alone cannot capture detailed visual intent. This direction supports more precise editing but requires models that can process multimodal inputs effectively.
- Pretrained text-to-image-based video editing methods commonly use fine-tuning or inversion, but often have limited temporal consistency.
- Reference-guided methods combine images with text to represent precise spatial relationships, visual references, and temporal dynamics.
- This multimodal direction aims to overcome the limitations of text-only control in video editing.
3. RefVIE Dataset and Benchmark
RefVIE is built by converting large-scale video editing pairs into reference-guided quadruplets through automated localization, synthesis, filtering, and quality control. Its statistics, benchmark design, and evaluation framework target diverse editing tasks and reference adherence.
- Scalable Data Generation Pipeline: Reference-guided training requires quadruplets comprising source videos, instructions, reference images, and target videos, which are expensive to curate manually.
- Scalable Data Generation Pipeline: The automated pipeline augments existing video editing triplets with synthesized references and processes 3.7M raw samples through four stages.
- Scalable Data Generation Pipeline: Grounding and segmentation localize the editing region in the target frame before image editing models synthesize task-specific reference images.
- Scalable Data Generation Pipeline: Quality control verifies semantic alignment and applies CLIP-based global de-duplication, yielding 477K high-quality instruction-reference-video quadruplets.
- Dataset Statistics: RefVIE covers balanced local addition, replacement, and background-change tasks, with most clips containing 80 to 110 frames.
- Benchmark and Evaluation: RefVIE-Bench uses manually verified samples to assess reference adherence, instruction compliance, and temporal consistency, while Gemini3 scores outputs across three dimensions.
- Benchmark and Evaluation: Kiwi-Edit combines an MLLM and DiT, injecting source-video features element-wise and reference-image features by sequence concatenation.
4. Methodology
Kiwi-Edit combines MLLM semantic conditioning with DiT video generation, using separate query and latent pathways for instruction intent, reference detail, and source structure. Progressive training aligns these components before instruction tuning and reference-guided fine-tuning.
- Architecture: The framework uses an MLLM to encode source video, instructions, and optional references into conditioning signals that guide a DiT to generate edited videos.The MLLM provides semantic understanding while the DiT performs video generation.
- Architecture: Learnable query tokens distill editing intent, while a Latent Connector extracts visual features from reference images for complementary conditioning.The query pathway targets semantic intent, whereas latent features provide dense visual information.
- Structural conditioning: Source-video latents are added element-wise to the noisy latent and modulated by a time-dependent factor to preserve spatial-temporal structure.The method avoids feature concatenation, which the authors report causes training instability.
- Reference conditioning: Reference-image patches are concatenated to the DiT input sequence, extending the attention context to support direct copying of reference texture details.This pathway is distinct from source-video control, which uses element-wise latent injection.
- Training objective: Flow Matching trains the model by minimizing mean squared error between predicted velocity and the ground-truth drift, conditioned on multimodal inputs.The target video latent, Gaussian noise, and multimodal conditioning define the training variables.
- Training curriculum: The progressive curriculum first aligns MLLM and DiT bridge components, then jointly tunes the DiT, and finally fine-tunes on RefVIE quadruplets for reference-guided editing.Instruction tuning uses a resolution curriculum from 480p to 720p, while all stages sample at most 81 video frames.
5. Experiments
Experiments evaluate Kiwi-Edit on instruction editing, reference-guided editing, and architectural and training ablations. The results report strong open-source instruction-editing performance, competitive reference fidelity, and benefits from the proposed conditioning and curriculum choices.
- Instruction Editing: Kiwi-Edit achieves an Overall score of 3.11 on OpenVE-Benchmark, surpassing the previous best OpenVE-Edit score of 2.50.Its Background Change score is 2.67, exceeding Runway Aleph’s 2.62.
- Qualitative results: Qualitative comparisons show accurate localization of edited objects and subject consistency during substantial background style changes.The reported examples include hat and table localization and reference preservation under background changes.
- Condition Design: The Add w/ time-dependent factor configuration outperforms alternatives, while sharing patch embeddings reduces results to 1.01.The strongest configuration also reports remove 2.63 and add 2.01 instruction-task scores.
- Training Curriculum: Skipping alignment causes a catastrophic performance drop, while excluding image co-training reduces Removal score from 2.84 to 2.58.The ablation attributes the latter difference to missing fine-grained spatial supervision for local manipulations.
- Reference Condition Design: Adding Reference Latent features improves the baseline score from 3.20 to 3.30, supporting joint use of sparse queries and dense visual latents.Queries capture high-level editing intent, while latent features inject detailed reference information.
6. Conclusion
The work addresses scarce reference-guided editing data with a scalable synthesis pipeline, RefVIE-Bench, and Kiwi-Edit. Together, these contributions achieve state-of-the-art controllable video editing and support more accessible video creation.
- The pipeline transforms existing video pairs into instruction-reference quadruplets, addressing the scarcity of high-quality reference-guided editing data.
- RefVIE-Bench standardizes evaluation for reference-guided video editing.
- Kiwi-Edit achieves state-of-the-art performance in controllable video editing.
- The released data-centric resources aim to support more controllable and accessible video content creation.
A. Outlines
The supplementary material expands the paper with workflows, benchmark results, reference-dataset visualizations, and further comparisons with state-of-the-art methods.
- Section B: Section B details the data construction process with workflows.
- Section C: Section C details the benchmark results.
- Section D: Section D provides additional visualizations of the reference dataset.
- Section E: Section E presents further quantitative and qualitative comparisons with state-of-the-art methods.
B. Dataset Details
The dataset-construction details describe automated grounding and reference-image quality filtering for creating and validating reference-guided editing data.
- The supplementary material uses MLLM grounding and reference-image score filtering.
- Grounding: Editing-area grounding generates bounding boxes for regions edited between source and edited images.
- Grounding: Foreground grounding identifies corresponding foreground objects or people across the first and second images.
- Reference filtering: The reference-image evaluator checks whether the reference matches the instructed edit and edited result.
- Reference filtering: Reference-image quality filtering also assesses clarity, realism, artifacts, lighting, texture, and subject focus.
- Reference filtering: The scoring procedure outputs one overall integer score from 1 to 10.
C. Benchmark Details
The benchmark details define reference-guided subject and background editing evaluations using human-style 5-point ratings across fidelity, preservation, temporal stability, and visual harmony.
- RefVIE-Bench evaluation uses Gemini for reference-guided editing assessment.
- Subject reference editing: Subject-reference editing is rated on a 5-point scale for identity consistency and compliance.
- Background reference editing: Background-reference editing evaluates background replacement while checking foreground preservation and reference fidelity.
- Background reference editing: Background-editing ratings include reference fidelity and preservation, matting quality and temporal stability, and visual harmony and perspective.
- Background reference editing: The highest background-editing criterion requires an exact semantic and stylistic reference match with pixel-perfect foreground preservation throughout the video.
D. Sample Visualization
Figures 8–10 provide additional visual samples of the high-quality RefVIE dataset.
- Figures 8–10 show additional RefVIE samples.The passage identifies these figures as visual examples of the dataset.
- The samples are presented as examples of RefVIE's high quality.
- The visualization section focuses on qualitative dataset examples rather than quantitative evaluation.
E. Qualitative Comparison
The qualitative comparison evaluates instruction-only editing across SoTA methods, while the examples illustrate reference-guided local and background edits. In the reported comparison, Kiwi-Edit precisely follows a compound instruction involving both object replacement and background modification.
- Qualitative Comparison: The comparison uses instruction-only inputs against SoTA methods in Figures 11–12.Instruction-reference-guided demo videos are provided separately in the supplementary material.
- Reference-guided Editing: Reference-guided examples cover local object changes, local additions, and background changes.The examples include replacing clothing or objects, adding tracked elements, and changing video backgrounds.
- Qualitative Comparison: The qualitative section includes further RefVIE examples alongside the comparison figures.
- Reference-guided Editing: Additional examples include replacing a chef coat with a blue shirt and changing the background to a forest classroom.
- Qualitative Comparison: Only Kiwi-Edit precisely replaces the man with a robot and changes the background tree to a red maple tree.The figure caption highlights this compound instruction-following result in a red box.