PaperlayerView exact revision on arXiv

Source-linked AI summary

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

Yiqi Lin, Guoqiang Liang, Ziyun Zeng, Zechen Bai, Yanzhe Chen, Mike Zheng Shou

arXiv:2603.02175v4cs.CVcs.AI

TL;DR

Text-only video editing lacks precise visual control, while reference-guided editing is constrained by scarce high-quality training quadruplets. The paper addresses both problems with an automated RefVIE data pipeline, RefVIE-Bench, and the Kiwi-Edit multimodal architecture. The resulting system achieves state-of-the-art controllable video editing performance.

  • Problem

    Natural language cannot reliably describe precise visual details, and existing resources lack scalable high-quality source-video, instruction, reference-image, and target-video quadruplets.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    The paper synthesizes reference images from existing video editing pairs, constructs RefVIE and RefVIE-Bench, and uses a unified MLLM-DiT architecture with learnable queries and latent visual features.

    Source: §1 · p. 2

  • Results

    Kiwi-Edit achieves state-of-the-art performance on instruction-only and reference-guided video editing tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The data-centric pipeline and unified model provide an open foundation for more controllable and accessible video content creation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.

1. Introduction

Instruction-based video editing has advanced, but text-only control cannot reliably express precise visual details and reference-guided editing lacks scalable paired data. The paper addresses these gaps with RefVIE, RefVIE-Bench, and Kiwi-Edit.

  • Text-only instructions struggle to specify precise textures, object identities, and stylistic characteristics, motivating visual references.Source: §1 · p. 2
  • Existing datasets provide instruction-based video pairs but generally lack reference images, while accessible reference-guided resources remain scarce.Source: §1 · p. 2
  • RefVIE synthesizes missing reference images from existing source-target video pairs using vision-language models, image editors, filtering, and de-duplication.Source: §1 · p. 2
  • Kiwi-Edit integrates a frozen MLLM with a DiT through query and latent connectors to guide video generation with multimodal conditions.Source: §1 · p. 2
  • The paper also introduces RefVIE-Bench and reports state-of-the-art performance across instruction-only and reference-guided tasks.Source: §1 · p. 2

2. Related Work

Prior video editing methods increasingly add reference images to textual instructions because natural language alone cannot capture detailed visual intent. This direction supports more precise editing but requires models that can process multimodal inputs effectively.

  • Pretrained text-to-image-based video editing methods commonly use fine-tuning or inversion, but often have limited temporal consistency.Source: §2.1 · p. 3
  • Reference-guided methods combine images with text to represent precise spatial relationships, visual references, and temporal dynamics.Source: §2.2 · p. 3
  • This multimodal direction aims to overcome the limitations of text-only control in video editing.Source: §2.2 · p. 3

3. RefVIE Dataset and Benchmark

RefVIE is built by converting large-scale video editing pairs into reference-guided quadruplets through automated localization, synthesis, filtering, and quality control. Its statistics, benchmark design, and evaluation framework target diverse editing tasks and reference adherence.

Scalable Data Generation Pipeline

  • Reference-guided training requires quadruplets comprising source videos, instructions, reference images, and target videos, which are expensive to curate manually.Source: §3.1 · p. 3
  • The automated pipeline augments existing video editing triplets with synthesized references and processes 3.7M raw samples through four stages.Source: §3.1 · p. 3, Figure 3 · p. 4
  • Grounding and segmentation localize the editing region in the target frame before image editing models synthesize task-specific reference images.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Quality control verifies semantic alignment and applies CLIP-based global de-duplication, yielding 477K high-quality instruction-reference-video quadruplets.Source: §3.1 · p. 4, Figure 3 · p. 4

Dataset Statistics

  • RefVIE covers balanced local addition, replacement, and background-change tasks, with most clips containing 80 to 110 frames.Source: §3.2 · p. 4, Figure 4 · p. 4

Benchmark and Evaluation

  • RefVIE-Bench uses manually verified samples to assess reference adherence, instruction compliance, and temporal consistency, while Gemini3 scores outputs across three dimensions.Source: §3.3 · pp. 4–5
  • Kiwi-Edit combines an MLLM and DiT, injecting source-video features element-wise and reference-image features by sequence concatenation.Source: Figure 5 · p. 5

4. Methodology

Kiwi-Edit combines MLLM semantic conditioning with DiT video generation, using separate query and latent pathways for instruction intent, reference detail, and source structure. Progressive training aligns these components before instruction tuning and reference-guided fine-tuning.

Architecture

  • The framework uses an MLLM to encode source video, instructions, and optional references into conditioning signals that guide a DiT to generate edited videos.The MLLM provides semantic understanding while the DiT performs video generation.Source: §4.1 · p. 5
  • Learnable query tokens distill editing intent, while a Latent Connector extracts visual features from reference images for complementary conditioning.The query pathway targets semantic intent, whereas latent features provide dense visual information.Source: §4.1 · p. 5

Structural conditioning

  • Source-video latents are added element-wise to the noisy latent and modulated by a time-dependent factor to preserve spatial-temporal structure.The method avoids feature concatenation, which the authors report causes training instability.Source: §4.1 · pp. 5–6

Reference conditioning

  • Reference-image patches are concatenated to the DiT input sequence, extending the attention context to support direct copying of reference texture details.This pathway is distinct from source-video control, which uses element-wise latent injection.Source: §4.1 · p. 6

Training objective

  • Flow Matching trains the model by minimizing mean squared error between predicted velocity and the ground-truth drift, conditioned on multimodal inputs.The target video latent, Gaussian noise, and multimodal conditioning define the training variables.Source: §4.1 · p. 6

Training curriculum

  • The progressive curriculum first aligns MLLM and DiT bridge components, then jointly tunes the DiT, and finally fine-tunes on RefVIE quadruplets for reference-guided editing.Instruction tuning uses a resolution curriculum from 480p to 720p, while all stages sample at most 81 video frames.Source: §4.2 · p. 6

5. Experiments

Experiments evaluate Kiwi-Edit on instruction editing, reference-guided editing, and architectural and training ablations. The results report strong open-source instruction-editing performance, competitive reference fidelity, and benefits from the proposed conditioning and curriculum choices.

Instruction Editing

  • Kiwi-Edit achieves an Overall score of 3.11 on OpenVE-Benchmark, surpassing the previous best OpenVE-Edit score of 2.50.Its Background Change score is 2.67, exceeding Runway Aleph’s 2.62.Source: §5.2 · p. 7

Qualitative results

  • Qualitative comparisons show accurate localization of edited objects and subject consistency during substantial background style changes.The reported examples include hat and table localization and reference preservation under background changes.Source: §5.3 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

Condition Design

  • The Add w/ time-dependent factor configuration outperforms alternatives, while sharing patch embeddings reduces results to 1.01.The strongest configuration also reports remove 2.63 and add 2.01 instruction-task scores.Source: §5.4 · pp. 7–8

Training Curriculum

  • Skipping alignment causes a catastrophic performance drop, while excluding image co-training reduces Removal score from 2.84 to 2.58.The ablation attributes the latter difference to missing fine-grained spatial supervision for local manipulations.Source: §5.4 · p. 8

Reference Condition Design

  • Adding Reference Latent features improves the baseline score from 3.20 to 3.30, supporting joint use of sparse queries and dense visual latents.Queries capture high-level editing intent, while latent features inject detailed reference information.Source: §5.4 · pp. 8–9

6. Conclusion

The work addresses scarce reference-guided editing data with a scalable synthesis pipeline, RefVIE-Bench, and Kiwi-Edit. Together, these contributions achieve state-of-the-art controllable video editing and support more accessible video creation.

  • The pipeline transforms existing video pairs into instruction-reference quadruplets, addressing the scarcity of high-quality reference-guided editing data.Source: §6 · p. 9
  • RefVIE-Bench standardizes evaluation for reference-guided video editing.Source: §6 · p. 9
  • Kiwi-Edit achieves state-of-the-art performance in controllable video editing.Source: §6 · p. 9
  • The released data-centric resources aim to support more controllable and accessible video content creation.Source: §6 · p. 9

A. Outlines

The supplementary material expands the paper with workflows, benchmark results, reference-dataset visualizations, and further comparisons with state-of-the-art methods.

Section B

Section C

Section D

Section E

  • Section E presents further quantitative and qualitative comparisons with state-of-the-art methods.Source: A. Outlines · p. 11

B. Dataset Details

The dataset-construction details describe automated grounding and reference-image quality filtering for creating and validating reference-guided editing data.

Grounding

  • Editing-area grounding generates bounding boxes for regions edited between source and edited images.Source: B. Dataset Details · p. 11
  • Foreground grounding identifies corresponding foreground objects or people across the first and second images.Source: B. Dataset Details · p. 11

Reference filtering

C. Benchmark Details

The benchmark details define reference-guided subject and background editing evaluations using human-style 5-point ratings across fidelity, preservation, temporal stability, and visual harmony.

Subject reference editing

Background reference editing

  • Background-reference editing evaluates background replacement while checking foreground preservation and reference fidelity.Source: C. Benchmark Details · p. 12
  • Background-editing ratings include reference fidelity and preservation, matting quality and temporal stability, and visual harmony and perspective.Source: C. Benchmark Details · pp. 12–13
  • The highest background-editing criterion requires an exact semantic and stylistic reference match with pixel-perfect foreground preservation throughout the video.Source: C. Benchmark Details · p. 13

D. Sample Visualization

Figures 8–10 provide additional visual samples of the high-quality RefVIE dataset.

E. Qualitative Comparison

The qualitative comparison evaluates instruction-only editing across SoTA methods, while the examples illustrate reference-guided local and background edits. In the reported comparison, Kiwi-Edit precisely follows a compound instruction involving both object replacement and background modification.

Qualitative Comparison

  • The comparison uses instruction-only inputs against SoTA methods in Figures 11–12.Instruction-reference-guided demo videos are provided separately in the supplementary material.Source: E. Qualitative Comparison · p. 13

Reference-guided Editing

  • Reference-guided examples cover local object changes, local additions, and background changes.The examples include replacing clothing or objects, adding tracked elements, and changing video backgrounds.Source: E. Qualitative Comparison · pp. 14–16

Qualitative Comparison

Reference-guided Editing

Qualitative Comparison

  • Only Kiwi-Edit precisely replaces the man with a robot and changes the background tree to a red maple tree.The figure caption highlights this compound instruction-following result in a red box.Source: Figure 12 · p. 18

Related papers

Loading 2603.02175v4…