PaperlayerView exact revision on arXiv

Source-linked AI summary

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer

Ruonan Yu, Zhenxiong Tan, Zigeng Chen, Songhua Liu, Xinchao Wang

arXiv:2603.15478v1cs.CV

TL;DR

Video control and editing are limited by scarce paired video data and high training costs. ViFeEdit adapts video diffusion transformers using only 2D images through spatial-temporal decoupling and dual-path conditioning, achieving promising performance across diverse editing tasks with limited image data and minimal cost.

  • Problem

    Video control and editing require spatially coherent changes and temporally consistent propagation, while paired video data is scarce and expensive to construct.

  • Method

    ViFeEdit reparameterizes video DiTs to decouple spatial adaptation from frozen 3D temporal modeling and uses a dual-path pipeline with separate timestep embeddings.

  • Results

    ViFeEdit achieves promising performance across a wide range of video editing tasks using limited paired image data and minimal computational cost.

  • Takeaways & Limitations

    Video diffusion transformers can be adapted for diverse video editing tasks without any video training data, using solely 2D images.

Abstract

from arXiv · show

Diffusion Transformers (DiTs) have demonstrated remarkable scalability and quality in image and video generation, prompting growing interest in extending them to controllable generation and editing tasks. However, compared to the image counterparts, progress in video control and editing remains limited, mainly due to the scarcity of paired video data and the high computational cost of training video diffusion models. To address this issue, in this paper, we propose a video-free tuning framework termed ViFeEdit for video diffusion transformers. Without requiring any forms of video training data, ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images. At the core of our approach is an architectural reparameterization that decouples spatial independence from the full 3D attention in modern video diffusion transformers, which enables visually faithful editing while maintaining temporal consistency with only minimal additional parameters. Moreover, this design operates in a dual-path pipeline with separate timestep embeddings for noise scheduling, exhibiting strong adaptability to diverse conditioning signals. Extensive experiments demonstrate that our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data. Codes are available https://github.com/Lexie-YU/ViFeEdit.

1 Introduction

Video control and editing require both spatially coherent changes and temporally consistent propagation, but paired video data is costly and scarce. ViFeEdit addresses this gap by tuning video DiTs with only 2D images through spatial-temporal decoupling and a dual-path design.

  • Video editing is harder than image editing because modifications must remain coherent across both space and time.Paired video datasets also require expensive frame-level annotation for temporal alignment.
  • Recent paired video-data efforts can consume over 10,000 GPU days, motivating video-free adaptation.
  • ViFeEdit tunes video diffusion transformers for diverse editing and control tasks using only 2D images and minimal training cost.
  • The method decouples spatial token modeling from temporal modeling, learning spatial editing behaviors while retaining pretrained temporal modules.
  • Complementary spatial attention blocks and separate timestep embeddings support stable adaptation without disrupting the original 3D attention behavior.The spatial blocks reuse pretrained spatial priors, while the dual-path pipeline separates latent and conditional processing.
  • Extensive experiments across six fine-grained video editing tasks show promising performance with limited image data and minimal computational cost.

2 Related Works

Video editing methods trade off temporal modeling, efficiency, editing capacity, and supervision requirements. Existing approaches range from temporal adaptation and inference-time modulation to fully supervised end-to-end video training.

  • The literature therefore includes temporal adaptation, training-free attention or latent modulation, and supervised end-to-end video editing paradigms.
  • Temporal-Adaptation Methods: Temporal-adaptation methods add temporal modules or recurrent connections to image diffusion models to improve cross-frame consistency.They typically require additional training or per-video fine-tuning and are computationally expensive.
  • Attention- and Latent-Modulation Methods: Attention- and latent-modulation methods reuse frozen image backbones for greater efficiency and lower memory cost.Their editing capacity is largely limited to appearance-level refinements rather than structural transformations.
  • End-to-End Methods: End-to-end methods train video-conditioned generators on large paired or synthetic video datasets for strong editing and generalization.Their performance requires substantial data and computational resources.

3 Method

ViFeEdit adapts text-to-video DiTs for editing without video training data by separating spatial adaptation from pretrained temporal modeling. Its dual-path pipeline injects clean source-video guidance while preserving distinct noise states.

  • 3.1 Preliminary: Video DiTs process noisy video latents and text tokens, with full 3D attention jointly modeling spatial and temporal dependencies.The video latent token count is N = f × h × w.
  • 3.1 Preliminary: ViFeEdit is a video-free tuner that adapts text-to-video DiTs to video editing and control tasks using minimal additional parameters.
  • 3.2 Spatio-Temporal Decoupling: Directly fine-tuning full 3D attention with 2D images can disrupt temporal dynamics, while attention-head roles vary across prompts and denoising steps.
  • 3.3 Dual-Path Pipeline: A dual-path pipeline separately processes noisy latents and source-video conditions, allowing the source video to provide explicit spatial reference guidance.The two streams use separate processing arrangements rather than being jointly concatenated through all attention layers.
  • 3.2 Spatio-Temporal Decoupling: The method adds complementary positive and negative 2D spatial attention blocks while freezing original 3D attention to preserve pretrained temporal capability.Their residual outputs cancel at initialization, retaining the original model behavior as adaptation begins.
  • 3.3 Dual-Path Pipeline: During inference, the source video can optionally initialize the noisy latent as a noise prior controlled by α, with flow matching starting from t = α.
  • 3.3 Dual-Path Pipeline: Separate timestep embeddings assign t to noisy latents and 0 to the clean source video, preventing conditional guidance from being blurred by mismatched noise levels.

4 Experiments

ViFeEdit evaluates video editing and control using only paired 2D image data, across six editing tasks and multiple experimental settings. Results indicate high-fidelity, temporally consistent edits, unified multi-task support, and benefits from its architectural components.

  • Experimental Setup: ViFeEdit uses the open-source Wan2.1-T2V-1.3B as its base model and evaluates six video editing tasks with paired image data.The tasks include consistent style transfer, rigid and non-rigid object replacement, color alteration, object addition, and object removal.
  • Experimental Setup: Style-transfer experiments use five generated videos per prompt across 3D Chibi, Ghibli Studio, and American Cartoon styles, evaluated with VBench and VLM-based consistency measures.The evaluation covers subject, background, temporal flickering, motion smoothness, color, structural consistency, and motion consistency.
  • Consistent Style Transfer: ViFeEdit produces high-quality style transfer with temporal and spatial consistency, stable color behavior, and faithful target-style capture across evaluated settings.The method preserves spatial structure and temporal coherence while avoiding the abrupt changes and color drift observed in competing approaches.
  • Video Editing Results: Across rigid and non-rigid replacement, color alteration, addition, and removal, ViFeEdit delivers detailed edits, localized color changes, preserved quality, and plausible background completion.The method preserves motion and background integration for replacement, avoids disturbing unrelated regions during color changes, and completes object removal plausibly.
  • Video Editing Results: Multi-task LoRA supports color alteration, object addition, and object removal within one unified framework instead of requiring separate LoRAs or models.The multi-task setting is evaluated on FiVE-Bench alongside single-task LoRA variants.
  • Ablation Studies: Ablations attribute improvements to separate timestep embeddings, spatio-temporal decoupling, and the dual-path pipeline, which together yield consistent motion, background preservation, and high-quality style transfer.Separate timestep embeddings accelerate learning, while decoupling preserves temporal consistency and the dual path further improves spatial consistency.

5 Conclusion

ViFeEdit adapts video diffusion transformers to diverse video editing tasks using only 2D images and minimal training cost. Its architectural reparameterization and dual-path pipeline preserve visual control, temporal consistency, and stable optimization.

  • ViFeEdit adapts text-to-video DiTs to diverse video editing tasks using solely 2D paired images.The framework is designed for video-free tuning with minimal training costs.
  • Architectural reparameterization decouples spatial interactions from full 3D attention without compromising temporal consistency.
  • A dual-path pipeline with separate timestep embeddings enhances background and motion consistency.
  • Limited paired image data yields promising performance across a wide range of video editing tasks.The training process also benefits from more stable optimization and faster convergence.

A More Experimental Details

The experiments use paired image data for training and multiple video-generation setups for evaluation. The method is compared with established controllable and editing baselines across several tasks.

  • Consistent style transfer uses OmniConsistency subsets containing 140, 100, and 124 image pairs for 3D Chibi, Ghibli, and American Cartoon styles.
  • Other editing tasks use 250 GPT-5-generated prompts per task, with FLUX.1-dev producing source images and Qwen-Image-Edit-2509 producing targets.
  • Evaluation videos are generated with Wan2.1-T2V models using VBench and FiVE Bench prompts, with task-specific sample counts.
  • ViFeEdit inserts complementary 2D spatial attention modules into each 3D full-attention block, initialized from the corresponding 3D module and combined by subtraction.
  • Baselines include VACE with style-specific LoRA tuning, OmniConsistency, SDEdit, VidToMe, Pyramid-Edit, and Wan-Edit.The baselines span end-to-end video editing and alternative image- or video-based editing approaches.

A.3 More Training Details

Training fine-tunes selected modules with LoRA while keeping most of the network frozen. Style-transfer experiments also compare the proposed method with a high-capacity baseline using extensive video supervision.

  • LoRA fine-tuning updates only the 2D spatial attention and FFN modules, while the remaining network stays frozen.The LoRA rank is 32, training uses three NVIDIA RTX 6000 Ada GPUs, and peak memory is about 18 GiB.
  • Training hyperparameters are kept consistent across tasks except for the number of epochs.
  • Visual comparisons report high consistency and style fidelity for the proposed method using only paired image data.

A.4 More Evaluation Details

Evaluation combines VBench-based style-transfer assessment with FiVE-Bench editing metrics across full videos. Comparisons indicate that ViFeEdit maintains strong style-transfer consistency and fidelity with far less training data than Ditto.

  • VBench evaluation scores structural consistency, motion consistency, and style fidelity using prompts that request values from 0 to 100.
  • FiVE-Acc quantifies rigid and non-rigid object replacement, color modification, object addition, and object removal over entire videos.The benchmark question set was reviewed and refined to improve reliability.
  • Ditto uses Wan2.1-VACE-14B and 1M high-quality video pairs, while ViFeEdit uses the officially released style-transfer LoRA for comparison.
  • Using only 100–200 image pairs, ViFeEdit achieves high consistency and style fidelity across different styles.Ditto preserves motion coherence and temporal stability but often misses accurate style, color correspondence, and spatial alignment.

B.2 Results on Arbitrary Style Transfer

ViFeEdit handles multiple style transfers jointly using paired 2D image data, preserving spatial and temporal consistency while distinguishing closely related styles.

  • B.2 Results on Arbitrary Style Transfer: 317 paired images spanning Pixel Art, Oil Painting, and Van Gogh support mixed-style fine-tuning.The Wan2.1-T2V-1.3B model is fine-tuned for 20 epochs.
  • B.2 Results on Arbitrary Style Transfer: ViFeEdit jointly transfers multiple styles while maintaining high spatial and temporal consistency.The method handles several styles in one fine-tuning procedure.
  • B.2 Results on Arbitrary Style Transfer: The method distinguishes closely related styles, such as Van Gogh and classical oil painting, without cross-style interference.It also produces high-quality style transformations.

B.3 Results on Depth-to-Video Task

ViFeEdit extends to depth-conditioned video generation without video training data, closely following depth structure while achieving visual fidelity comparable to a video-trained baseline.

  • B.3 Results on Depth-to-Video Task: The depth-to-video evaluation uses 250 generated prompts paired with depth maps as supervision signals.Fine-tuning lasts 20 epochs under the settings in Table 4.
  • B.3 Results on Depth-to-Video Task: ViFeEdit closely follows input depth conditions and captures fine-grained structural cues with high precision.The experiment evaluates depth-to-video generation using generated source videos and depth maps.
  • B.3 Results on Depth-to-Video Task: Generated-video aesthetic quality and visual fidelity are comparable to Wan2.1-VACE-1.3B, despite that baseline being trained on large-scale video datasets.Wan2.1-VACE-1.3B serves as the comparison baseline.

B.4 Results on Challenging Cases

ViFeEdit remains temporally coherent and structurally stable in challenging editing cases, supporting high-quality video editing with limited paired image data rather than large-scale paired video training.

  • B.4 Results on Challenging Cases: ViFeEdit preserves coherent temporal consistency and stable structure under motion blur and significant occlusion.These challenging cases are evaluated through qualitative edited results.
  • B.4 Results on Challenging Cases: Across video editing and control tasks, the method achieves high-quality, consistent outputs using only a small amount of paired image data.The approach does not require large-scale paired video data or extensive training.
  • B.4 Results on Challenging Cases: The method supports global and local editing as well as controllable generation.The conclusion reports this versatility alongside consistently high-quality outputs.
Loading 2603.15478v1…