Source-linked AI summary
Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, Chunhua Shen
TL;DR
Video editing lacks an accessible way to extend image diffusion success while preserving prompt alignment, frame fidelity, and temporal consistency without costly video training. Vid2vid-zero addresses this gap with off-the-shelf image diffusion models, null-text inversion, cross-frame modeling, and spatial regularization, producing promising text-aligned and faithful edits in real-world videos. Its main limitation is inherited from image models that lack temporal and motion priors, preventing effective direct editing of video actions.
Problem
Video editing remains difficult to extend from image diffusion because existing approaches can require substantial text-video data and computation, while edits must align with prompts, preserve original content, and remain temporally consistent.
Method
Vid2vid-zero uses off-the-shelf image diffusion models without video training, combining null-text inversion, cross-frame temporal modeling, and spatial regularization.
Results
Experiments show promising, high-quality, text-aligned, and faithful editing of attributes, subjects, and places in real-world videos.
Takeaways & Limitations
The zero-shot design preserves image diffusion creativity while enabling test-time bi-directional temporal modeling for video editing.
Takeaways & Limitations
Because the method inherits image diffusion models without temporal and motion priors, it cannot directly edit video actions effectively.
Abstract
from arXiv · showhide
Large-scale text-to-image diffusion models achieve unprecedented success in image generation and editing. However, how to extend such success to video editing is unclear. Recent initial attempts at video editing require significant text-to-video data and computation resources for training, which is often not accessible. In this work, we propose vid2vid-zero, a simple yet effective method for zero-shot video editing. Our vid2vid-zero leverages off-the-shelf image diffusion models, and doesn't require training on any video. At the core of our method is a null-text inversion module for text-to-video alignment, a cross-frame modeling module for temporal consistency, and a spatial regularization module for fidelity to the original video. Without any training, we leverage the dynamic nature of the attention mechanism to enable bi-directional temporal modeling at test time. Experiments and analyses show promising results in editing attributes, subjects, places, etc., in real-world videos. Code is made available at \url{https://github.com/baaivision/vid2vid-zero}.
I. INTRODUCTION
Video editing must align with text prompts, preserve the original video's frame content, and maintain temporal consistency and quality. Existing approaches can require substantial video data or per-video fine-tuning, motivating the training-free vid2vid-zero pipeline.
- Text-driven video editing must satisfy text-to-video alignment, fidelity to corresponding original frames, and temporal consistency with high quality.
- Video diffusion approaches require significant paired text-video data and computational resources, limiting accessibility.
- Tune-A-Video reduces training demands but still fine-tunes more than 70M parameters per video and may degrade generation quality.
- Frame-wise image editing causes severe flickering because it lacks temporal modeling, even with content-preserving inversion and cross-attention guidance.
- Vid2vid-zero directly uses pre-trained image diffusion models without video training, combining null-text inversion, spatial regularization, and cross-frame modeling.
- The method uses dense spatial-temporal attention for bi-directional temporal modeling and shows promising results across real-world video edits.
II. RELATED WORK
Diffusion models have advanced image and video generation, but video methods face higher complexity, limited datasets, and expensive training. Vid2vid-zero differs from prior editing methods by bypassing video diffusion training and using off-the-shelf image diffusion models for zero-shot editing.
- Video generation is harder than image generation because of higher-dimensional complexity and limited high-quality datasets.
- Video diffusion systems such as Video Diffusion Models and Imagen Video improve temporal coherence and photorealism but require costly data and computation.
- Image diffusion models have achieved strong text-driven image editing by leveraging pre-trained models and spatial information in cross-attention maps.
- Prior video editing methods fine-tune video diffusion models or require training conditioned on structure and prompts to maintain consistency.
- The framework replaces U-Net self-attention with shared-weight cross-frame attention to support temporal modeling.
- Vid2vid-zero bypasses expensive video training resources and large text-video datasets by using off-the-shelf image diffusion models for zero-shot editing.
III. METHOD
Vid2vid-zero receives an input video with source and target prompts, inverts the video, and then samples an edited result using the inversion outputs. Its method is organized around video inversion, spatial regularization, and cross-frame modeling.
- The method takes an input video, source prompt, and target prompt, then produces an edited video in a zero-shot setting.
- Vid2vid-zero contains video inversion for text-to-video alignment, spatial regularization for video-to-video fidelity, and cross-frame modeling for temporal consistency.
- The implementation accepts the input video, source and target prompts, and null-text and attention-injection thresholds.
- The algorithm uses DDIM inversion and null-text optimization before sampling the edited video from the inverted latent representation.
A. Real Video Inversion
The method first inverts each video frame into noise space, then optimizes null-text embeddings to align reconstruction with the video content and source prompt.
- Real Video Inversion: DDIM inversion maps each input video frame to noise space across T timesteps for subsequent editing.The inversion trajectory represents the latent variables used for reconstruction and editing.
- Real Video Inversion: Null-text optimization addresses the mismatch between inverted latent noise and the user-provided source prompt.Without alignment, sampling from the inverted latent and source prompt may reconstruct a video substantially different from the original.
- Real Video Inversion: The optimized null-text embedding trajectory is updated during inversion to improve text-to-video alignment.The method uses the DDIM inversion trajectory and source prompt as inputs to optimize the embedding.
- Real Video Inversion: A shared null-text embedding across video frames preserves consistent information during reconstruction.This sharing strategy is applied across different frames rather than optimizing wholly independent embeddings.
B. Temporal Modeling
The temporal modeling design must provide bidirectional temporal context without departing substantially from the spatial behavior of pretrained image-diffusion self-attention. ST-Attn addresses this trade-off by combining spatial querying with attention to all video frames.
- Temporal Modeling: Temporal modeling is essential for producing temporally coherent video editing results.The section contrasts causal and temporal-only alternatives because video editing requires bidirectional temporal modeling while retaining spatial modeling.
- Temporal Modeling: ST-Attn lets each frame attend to all frames, using spatial features from the query frame and features across the video for keys and values.This provides access to both previous and future frames while preserving spatial querying.
- Temporal Modeling: The proposed attention uses pretrained projection weights for queries, keys, and values shared across spatial and temporal locations.The formulation is Q = W_Qx_i, K = W_Kx_1:T, and V = W_Vx_1:T.
- Temporal Modeling: ST-Attn provides bidirectional temporal modeling while spatial querying reduces discrepancy from pretrained image-diffusion sampling.The design balances temporal context with compatibility with the original self-attention process.
- Temporal Modeling: The implementation uses ST-Attn in selected U-Net blocks and SC-Attn in other self-attention layers.The 2D residual block is inflated to 3D by copying its weights across frames for video inference.
C. Spatial Regularization
Spatial regularization preserves fidelity to the input video by injecting reconstruction-time cross-attention maps into the editing process. The resulting guidance is temporal-aware, limiting abrupt attention changes across frames.
- Spatial Regularization: Spatial regularization uses cross-attention maps from input-video reconstruction to focus editing on prompt-related spatial regions.The method treats these maps as spatial guidance and injects them during denoising.
- Spatial Regularization: The denoising process injects the reconstruction attention mask M_t together with the optimized null-text embedding.The mask guides noise prediction during editing.
- Spatial Regularization: Temporal-aware cross-attention masks prevent sudden changes in guidance across different video frames.This follows from the method’s zero-shot temporal modeling.
IV. EXPERIMENTS
The experiments demonstrate zero-shot editing with Stable Diffusion-based implementations on short, fixed-resolution videos and show examples spanning style, subject, and scene changes. The displayed results are intended to illustrate prompt alignment and temporal consistency while preserving source-video content.
- Implementation Details: The implementation uses latent diffusion models with publicly available Stable Diffusion weights.The default setup uses 8 frames at 512×512 resolution, DDIM sampling for 50 steps, and classifier-free guidance at scale 7.5.
- Experiments: Figure 4 presents edited videos intended to be temporally consistent, prompt-aligned, and faithful to the original videos.The figure includes results from various input videos and prompts.
B. Main Results
vid2vid-zero edits video style, attributes, backgrounds, and subjects while preserving the original video’s semantics, content, and other attributes. It also supports multiple edits simultaneously while maintaining alignment with the target text.
- Editing Style: vid2vid-zero transforms all video frames to a target style without altering the original video’s semantics.
- Editing Attributes: The method edits attributes such as a person’s age and a car’s model while preserving other properties, including color and pose.
- Editing Background: vid2vid-zero changes backgrounds to beaches, urban streets, and deserts while retaining the generative power of the pretrained Stable-Diffusion model.
- Replacing Subjects: The method replaces subjects, including a man with Stephen Curry and a horse with a dog, while maintaining text alignment and fidelity to the original videos.
- Replacing Subjects: vid2vid-zero can replace subjects and edit backgrounds simultaneously, such as changing a man to Curry and the background to Times Square.
C. Analysis
Ablations show that temporal modeling, spatial attention guidance, and null-text inversion each address different video-editing failures. Dense spatial-temporal attention captures information across frames and locations, supporting bidirectional temporal modeling while avoiding artifacts and blur.
- Ablations on Each Module: Removing temporal modeling makes both foreground cars and background mountains inconsistent over time.
- Ablations on Each Module: Removing spatial attention guidance reduces fidelity, changing car colors and omitting background trees.
- Ablations on Each Module: Removing null-text inversion makes background mountains and trees blurry because optimized null-text embeddings preserve fine-grained details.
- Ablations on Temporal Attention Modeling: Replacing dense spatial-temporal attention with Sparse-Causal Attention deforms early frames because errors propagate from previous frames.
- Ablations on Temporal Attention Modeling: Temporal-only Attention reduces error propagation but creates blurry frames because it diverges from the pretrained model’s spatial self-attention distribution.
- Spatial-Temporal Attention Visualization: The proposed attention attends to car-light regions in both previous and future frames, demonstrating bidirectional spatial-temporal information capture.
D. Comparison
Compared with frame-wise PnP editing and fine-tuned Tune-A-Video, vid2vid-zero balances temporal consistency with fidelity to the input video. User studies report higher scores than both baselines across quality, text alignment, and fidelity, while its CLIP score is comparable to Tune-A-Video.
- Qualitative Comparison: PnP preserves similar poses but suffers flickering, deformation, and repeated tails across edited frames.
- Qualitative Comparison: Tune-A-Video achieves better temporal consistency but does not faithfully preserve the input video’s pose and motion.
- Qualitative Comparison: vid2vid-zero replaces the horse with a dog while maintaining both temporal consistency and faithfulness to the input video.
- Quantitative Comparison: In user preference studies on 32 videos, vid2vid-zero scores higher than both PnP and Tune-A-Video on quality, text-to-video alignment, and fidelity.
- Quantitative Comparison: vid2vid-zero obtains a CLIP score on par with Tune-A-Video fine-tuned on the input video.
E. Applications on Customized Video Editing
vid2vid-zero extends customized image generation models to customized video editing without video training. It replaces foreground objects while preserving customized identity more effectively than TAV in the reported examples.
- Customized video editing replaces video objects with objects of a specific identity of interest to the user.
- The method uses DreamBooth to customize image generation models, then applies the resulting model for video customization editing.
- vid2vid-zero replaces a cow with an orange cat and a rabbit with a dog in the customized editing examples.
- Compared with TAV, vid2vid-zero better preserves customized-object appearance without the reported repeating-tail and color-changing artifacts.TAV preserves customized identity to some extent but exhibits a repeating cat tail and a dog changing from yellow to black.
- The method may inherit image-model limitations and cannot directly edit video actions because it lacks temporal and motion priors.