PaperlayerView exact revision on arXiv

Source-linked AI summary

VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization

Xiaoyan Cong, Haotian Yang, Angtian Wang, Yizhi Wang, Yiding Yang, Canyu Zhang, Chongyang Ma

arXiv:2512.16906v1cs.CV

TL;DR

Existing video-editing systems rely on simplistic paired edits, limiting generalization to diverse and complex instructions. VIVA combines a VLM instructor, Edit-GRPO reward optimization, and synthetic paired-data construction, and reports superior instruction following, generalization, and editing quality over state-of-the-art methods.

  • Problem

    Paired training data for video editing is often limited to simple operations, restricting generalization to complex real-world instructions.

  • Method

    VIVA uses a VLM instructor for grounded multimodal conditioning, Edit-GRPO for relative reward optimization, and a synthetic pipeline for diverse paired video-instruction data.

  • Results

    VIVA achieves superior instruction following, generalization, and editing quality over state-of-the-art methods.

  • Takeaways & Limitations

    Grounded multimodal representations and edit-specific reward optimization provide VIVA’s framework for instruction-faithful, content-preserving video edits.

  • Takeaways & Limitations

    Future work must expand supported edit types, interactive editing, arbitrary resolution and duration, and more general video-editing rewards.

Abstract

from arXiv · show

Instruction-based video editing aims to modify an input video according to a natural-language instruction while preserving content fidelity and temporal coherence. However, existing diffusion-based approaches are often trained on paired data of simple editing operations, which fundamentally limits their ability to generalize to diverse and complex, real-world instructions. To address this generalization gap, we propose VIVA, a scalable framework for instruction-based video editing that leverages VLM-guided encoding and reward optimization. First, we introduce a VLM-based instructor that encodes the textual instruction, the first frame of the source video, and an optional reference image into visually-grounded instruction representations, providing fine-grained spatial and semantic context for the diffusion transformer backbone. Second, we propose a post-training stage, Edit-GRPO, which adapts Group Relative Policy Optimization to the domain of video editing, directly optimizing the model for instruction-faithful, content-preserving, and aesthetically pleasing edits using relative rewards. Furthermore, we propose a data construction pipeline designed to synthetically generate diverse, high-fidelity paired video-instruction data of basic editing operations. Extensive experiments show that VIVA achieves superior instruction following, generalization, and editing quality over state-of-the-art methods. Website: https://viva-paper.github.io

1 Brown University 2 Intelligent Creation, ByteDance

The paper includes a comparison figure and notes Xiaoyan Cong’s internship affiliation and project-lead role.

  • Figure 1 compares VIVA with Runway Gen-4 Aleph [53].
  • VIVA accepts an optional reference image, illustrated by an inset teddy bear.
  • The work notes that Xiaoyan Cong was an intern at ByteDance and that one contributor was the project lead.

1. Introduction

VIVA addresses the limited generalization of video editors trained on simplistic paired edits through grounded multimodal instruction encoding, Edit-GRPO optimization, and synthetic paired data.

  • Motivation: Existing paired-data pipelines focus on simplified edits, limiting generalization to complex real-world instructions.
  • VLM Instructor: The VLM instructor jointly encodes text and visual inputs into grounded tokens for the diffusion transformer.It represents target entities, spatial regions, and desired attribute changes while combining visual-linguistic understanding with video generation.
  • Edit-GRPO: Edit-GRPO generates multiple candidate videos and updates the model using relative rewards for instruction fidelity, identity preservation, and structural consistency.The post-training stage emphasizes semantic correctness rather than pixel-level reconstruction alone.
  • Data Construction: The data pipeline creates high-quality paired source and edited videos with accurate instructions across diverse basic editing operations.
  • Evaluation: VIVA reports superior instruction following without degrading video quality compared with strong baselines.

2. Related Work

Related work spans instruction-based video editing, vision-language models for visual grounding, and reinforcement learning for generative alignment.

  • Instruction-based video editing: Earlier video-editing methods used inversion-based approaches, while later systems targeted additions, inpainting, try-on, and character animation [11] [16] [19] [26] [30] [33] [42] [48] [63] [68].
  • VLMs for visual understanding and generation: VLMs learn joint visual-text representations, and instruction-tuned variants improve visual grounding and context-aware comprehension [49] [32] [1] [37] [45] [4].
  • VLMs for visual understanding and generation: VIVA uses a VLM-based instructor to encode textual instructions and visual context for improved controllability and instruction fidelity.
  • Reinforcement learning for generative models: Reinforcement-learning methods including PPO, RPO, and GRPO have been used for alignment in language and visual generation, while image-editing work applies reward-guided fine-tuning [54] [51] [55] [39] [40] [65] [69] [44] [67].

3. Method

VIVA combines a context-aware VLM instructor with a DiT editing backbone, supervised training on synthetic pairs, and Edit-GRPO reward optimization. The framework uses multimodal conditioning, source-video context, and rewards for instruction following, preservation, and human preference alignment.

  • 3. Method: VIVA uses a VLM instructor and DiT generation branch to interpret multimodal editing conditions and produce edited videos.The VLM processes instructions, source-video frames, and optional reference images, while the DiT performs generation.
  • 3.1. Context-Aware VLM Instructor: The VLM combines text with the source video's first frame and optional reference image to provide grounded, fine-grained conditioning for editing.This replaces language-only conditioning with multimodal representations that capture target entities, spatial regions, and attribute changes.
  • 3.2. Instruction-based Supervised Fine-Tuning: Source-video VAE latents are channel-concatenated with noise tokens, giving the denoiser spatially and temporally aligned guidance for preserving structure and motion.A trainable projector aligns the resulting contextual tokens with the DiT feature dimension.
  • 3.2. Instruction-based Supervised Fine-Tuning: Supervised fine-tuning trains the DiT, patchify modules, projector, and token refiner on synthetic paired editing data, supplemented with one-frame image-editing examples.The synthetic dataset addresses scarce paired video-editing data and limited editing diversity.
  • 3.3. Edit-GRPO: Edit-GRPO samples multiple edited videos, scores instruction following, source preservation, and human preference alignment, then updates a DiT LoRA using relative advantages.The reward system combines these objectives with balancing weights, while LoRA avoids full backbone fine-tuning for efficiency.
  • 3.4. Data Preparation: The data pipeline constructs 1.5 million paired videos covering diverse local editing types to support scalable training.The dataset targets the field’s lack of large-scale, high-quality paired video-editing examples and uses a dedicated video-input branch for synthesis.

4. Experiments

VIVA is evaluated on diverse instruction-based video edits using VIE-Bench, VLM-based metrics, qualitative comparisons, user preferences, and ablations. It outperforms open-source baselines, matches the commercial Runway model quantitatively, and benefits from its VLM instructor, masked loss, image-data mixing, and Edit-GRPO.

  • Implementation Details: Experiments fine-tune pretrained HunyuanVideo-T2V-13B for 12,000 steps on paired data, mixing video and image samples with probabilities of 0.4 and 0.6.Training uses a learning rate of 2 × 10^-5 and global batch size 128.
  • Qualitative Comparisons: VIVA handles instruction-only and reference-based edits more accurately than baselines, including complex subject addition, local material changes, subject removal, and reference-image alignment.The VLM instructor provides visual context for interpreting complex semantics and aligning reference images with source videos.
  • Quantitative Comparisons: VIVA outperforms all open-source baselines across VLM-based evaluations and performs on par with Runway Gen-4 Aleph.Table 1 evaluates VIE-Bench, while reference-based editing is compared only with Runway because open-source baselines lack reference-image control.
  • User Study: The user study compares VIVA and baselines through paired expert judgments of instruction following, source preservation, and editing quality.Fourteen domain experts evaluate sampled instruction-only and reference-image-controlled edits; outcomes are reported as wins, losses, or ties in percentages.
  • Ablation Study: The VLM instructor significantly improves all evaluation metrics by supplying visually grounded semantic context that better guides the DiT.The ablation study also evaluates masked loss, image-data mixing, and Edit-GRPO, with the final row representing the full method.
  • Ablation Study: Masked loss improves spatial accuracy and convergence, image-data mixing broadens edit coverage and visual quality, and Edit-GRPO improves instruction following, source consistency, and preference alignment.The full model combines these components and achieves comprehensive gains in overall editing performance.

5. Conclusion

VIVA combines a visually grounded VLM instructor with Edit-GRPO to improve instruction-based video editing. The framework achieves strong instruction following, edit quality, and generalization, while future work targets richer edits, interactive use, arbitrary resolutions and durations, and unified rewards.

  • Conclusion: VIVA achieves state-of-the-art instruction following, edit quality, and generalization while outperforming existing open-source baselines.Its VLM instructor encodes textual and visual inputs into grounded multimodal tokens, and Edit-GRPO provides consistent performance gains.
  • Future Work: Future work will expand supported edit types, interactive editing, arbitrary resolution and duration, and unified reward modeling for video editing.The paper identifies these directions alongside supplementary-material discussions of limitations and failure cases.

Supplementary Material

The supplementary material describes Flow Matching and GRPO foundations used for video generation and reward-based post-training. It explains how continuous flows transport noisy latents toward data and how grouped relative rewards stabilize optimization.

  • DiT-Based Video Generation and Editing: Flow Matching trains a velocity field that transports a simple prior distribution toward the target video-data distribution through a probability-flow ODE.DiTs use this process as a multimodal-conditioned backbone for video generation and editing.
  • DiT-Based Video Generation and Editing: Rectified Flow defines the trajectory as a linear interpolation between noise and data samples, with the model regressing the corresponding target velocity.This continuous flow provides smoother temporal dynamics and faster convergence than conventional noise-prediction objectives.
  • Group Relative Policy Optimization: GRPO samples groups of trajectories, evaluates them with task-specific rewards, and converts those rewards into relative advantages for optimization.The relative formulation reduces variance and stabilizes alignment without requiring a separate critic network.
  • Group Relative Policy Optimization: Flow-GRPO introduces stochasticity by converting deterministic Flow-ODE sampling into an equivalent Flow-SDE, while coefficient-preserving sampling modifies noise injection to reduce artifacts.These changes support the diverse trajectory sampling required by GRPO-based post-training.

B. Implementation Details

Edit-GRPO updates only lightweight LoRA adapters while keeping the supervised DiT frozen, and uses fixed inference settings balancing quality and speed.

  • Training: Edit-GRPO inserts LoRA modules into DiT self-attention and cross-attention layers, updating only those adapters while freezing the supervised model parameters.The configuration uses rank r = 64 and scaling factor α = 128.
  • Inference: Inference uses classifier-free guidance scale 2.0 only for instruction conditions and 50 timesteps to balance performance with inference speed.

C.1. Detailed Architecture for Data Preparation

The data-preparation architecture adds a lightweight control branch to a pretrained MMDiT, enabling mutual attention between conditioning signals and noisy video latents. Supplementary results show robustness across guidance scales and improvements from Edit-GRPO.

  • Detailed Architecture for Data Preparation: The paired-data synthesis network inserts an additional control branch into the pretrained MMDiT every four blocks, combining condition tokens with noisy latents for full mutual attention.RoPE alignment between condition and latent tokens supports spatial and temporal correspondence.
  • Detailed Architecture for Data Preparation: Only the added control branch is trained while pretrained DiT weights remain frozen, preserving the foundation model’s generative capabilities.
  • Guidance Robustness: The model remains free from saturation artifacts across a wide CFG range, unlike standard ControlNet, while retaining instruction following at settings where ControlNet is sensitive.The authors attribute this robustness to bidirectional attention between conditional inputs and noisy latents.
  • Edit-GRPO Data: The pipeline uses source videos and Gemini 2.5 Pro to generate complex editing instructions and source and target captions for Edit-GRPO training.This produces the triplet (Vsrc, tsrc, tedit) used in the post-training stage.
  • Edit-GRPO Ablation: Edit-GRPO improves structural integrity, realism, source consistency, instruction adherence, aesthetic fidelity, and human-preference alignment in challenging edits.Qualitative before-and-after comparisons show fewer distortions, collapsing artifacts, and unnatural edited regions after post-training.

D.3. Complex Tasks

VIVA handles complex hybrid edits, non-rigid effects, and reference-conditioned transformations while maintaining foreground–background coherence. The supplementary material also documents evaluation dimensions, qualitative comparisons, ablations, and failure cases.

  • Complex Tasks: VIVA performs complex hybrid instructions and non-rigid effects including flames, smoke, fireworks, stylization, and watermark removal.These tasks are presented as challenging tests of generalization beyond the synthetic data-construction pipeline.
  • Complex Tasks: For background replacement, VIVA preserves coherence between the foreground subject and new environment by synthesizing realistic shadows on the changed terrain.
  • Complex Tasks: The method’s complex-editing performance is attributed to VLM interpretation of text–visual context and transfer of image-editing generalization through mixed training data.
  • Failure Cases: Failure cases include over-editing under global transformations, blur during rapid motion, and residual artifacts from under-editing removal tasks.
  • Evaluation: The VLM evaluator scores instruction following, source-video preservation, editing quality, and subject similarity on a 0–10 scale.Subject similarity applies to reference-based edits.

E. Limitations

VIVA’s failures cluster around rapid motion and difficulty calibrating edit intensity. Fast motion can produce blur, while global transformations may be over-edited and removal tasks may retain artifacts.

  • Rapid motion can produce blurry outputs, including blurred hands.
  • Global transformations such as weather or style changes may be over-edited.
  • Removal tasks may be under-edited, leaving residual artifacts such as cast shadows.
Loading 2512.16906v1…