Source-linked AI summary
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy
TL;DR
Applying image diffusion models to video leaves temporal consistency across frames as a central challenge. This paper introduces a zero-shot framework that combines hierarchical cross-frame constraints for key-frame translation with temporal-aware patch propagation, achieving global and local temporal consistency without training.
Problem
Applying image diffusion models to video makes temporal consistency across frames a central challenge.
Method
A zero-shot framework adapts pre-trained image diffusion models with hierarchical cross-frame constraints for key-frame translation, then propagates key frames using temporal-aware patch matching and frame blending.
Results
The framework achieves both global and local temporal consistency without training and balances video quality with efficiency.
Takeaways & Limitations
The framework adapts image diffusion models to videos and is compatible with existing image diffusion techniques.
Takeaways & Limitations
The method relies on optical flow, so inaccurate or changed flow can cause artifacts or incorrect movement; small details and subtle motions may also be poorly preserved.
Abstract
from arXiv · showhide
Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.
1. Introduction
Applying image diffusion models to video requires zero-shot operation, compatibility with existing image tools, and temporal consistency beyond global style. The proposed framework combines hierarchical cross-frame constraints with key-frame translation and propagation to preserve global and local coherence.
- Existing zero-shot video methods reduce training costs and retain image-model compatibility, but often preserve global style without low-level structures and textures.
- The target application requires temporal consistency, no training or fine-tuning, and compatibility with off-the-shelf customized image models.
- Hierarchical cross-frame constraints use optical flow, previous frames, and an initial anchor to regulate shapes, textures, colors, and global style across diffusion stages.
- The framework translates key frames with adapted diffusion models, then propagates them using temporal-aware patch matching and frame blending.
- The resulting zero-shot framework targets global and local temporal consistency without training and remains compatible with pre-trained image diffusion models.
2. Related Work
Related work spans text-to-image diffusion, customized image models, video diffusion, zero-shot editing, and patch-based propagation. The paper combines coherent diffusion-generated exemplars with propagation to address the limitations of these approaches.
- 2.1. Text Driven Image Generation: Diffusion models generate images through gradual denoising, with latent diffusion improving efficiency by operating in a low-resolution latent space.
- 2.1. Text Driven Image Generation: Textual Inversion, DreamBooth, and LoRA customize image models, while ControlNet adds edge, depth, or pose conditions for fine-grained generation.
- 2.2. Video Editing with Diffusion Models: Video diffusion methods extend image models with temporal modeling but generally require large-scale video data, substantial computation, or model redesign.
- 2.2. Video Editing with Diffusion Models: Single-video fine-tuning and optimization-based editing methods are less efficient, whereas zero-shot methods avoid training and support flexible conditions.
- 2.2. Video Editing with Diffusion Models: Patch-based video translation preserves fine details, but temporal consistency depends on coherence among the exemplar frames.
3. Preliminary: Diffusion Models
Stable Diffusion performs text-conditioned generation in an autoencoder’s latent space through iterative diffusion and denoising. ControlNet augments its noise prediction with spatial conditions such as edges, depth, and pose.
- Stable Diffusion: Stable Diffusion encodes an image into a latent feature x0, applies diffusion in the autoencoder latent space, and decodes the recovered latent into an image.
- Stable Diffusion: The forward process progressively adds noise over time steps, eventually making the latent approximate a standard Gaussian distribution.
- Stable Diffusion: A trained U-Net predicts noise conditioned on the latent, time step, and text prompt, enabling iterative recovery with deterministic DDIM sampling.
- Stable Diffusion: During inference, DDIM starts from Gaussian noise and decodes the recovered latent x0 into the generated image I′.
- ControlNet: ControlNet adds an extra condition cf to the U-Net noise prediction, providing spatial guidance alongside the text prompt.
4. Zero-Shot Text-Guided Video Translation
The framework translates selected video key frames with adapted image diffusion models and hierarchical cross-frame constraints, then interpolates non-key frames from neighboring key frames. This design uses frame relationships to improve temporal consistency.
- Framework Overview: Given input frames {Ii}N_i=0, the goal is to render a new video in an artistic expression specified by text prompts or customized Stable Diffusion models.
- Key Frame Translation: Key Frame Translation adapts pre-trained image diffusion models with four hierarchical cross-frame constraints to render coherent key frames.
- Full Video Translation: Full Video Translation interpolates non-key frames from their neighboring two key frames, exploiting relationships across frames for temporal consistency.
4.1. Key Frame Translation
Key frame translation adapts image diffusion to video by injecting anchor- and previous-frame guidance at different sampling stages, then preserving pixel-level fidelity during latent fusion. The design combines global style, rough shape, fine texture, and color consistency mechanisms.
- Sampling pipeline: The pipeline initializes each key frame from a noisy version of its input latent, with T controlling how much input detail is preserved.Smaller T retains more detail; the first frame serves as an anchor while the previous frame provides temporal guidance.
- Global style consistency: Cross-frame attention uses the first and previous frames as key-value references to regularize global style across generated key frames.Unlike self-attention within one frame, it matches and fuses corresponding patches from other frames.
- Shape-aware cross-frame latent fusion: Optical-flow warping and latent fusion add local shape guidance, using the anchor frame because it provides better guidance than the previous frame.Fusion is restricted to early sampling steps because late-step interpolation can blur and distort shapes.
- Pixel-aware cross-frame latent fusion: Mid-step pixel-aware fusion encodes warped anchor and previous outputs for fine texture alignment, but lossy autoencoding can accumulate distortion and color bias.The method addresses this with fidelity-oriented encoding that predicts iterative information loss for compensation and masks potential artifacts.
- Pixel-aware cross-frame latent fusion: Fidelity-oriented encoding preserves image information after repeated encoding and decoding, while late-step AdaIN further maintains coherent color style across key frames.The reported preservation test repeats encoding and decoding 10 times.
- Pixel-aware cross-frame latent fusion: Structure-guided inpainting combines warped frame references with ControlNet structure guidance inside the masked region to produce pixel-coherent key frames.The resulting fused frame provides a pixel reference for subsequent sampling.
4.2. Full Video Translation
Full video translation renders uniformly sampled key frames with the adapted diffusion model, then propagates them to neighboring frames through Ebsynth-based correspondence and blending. This hybrid design balances diffusion quality with efficient propagation while omitting a blending step that can introduce artifacts.
- Key frame and non-key frame translation: The framework renders uniformly sampled key frames with the adapted diffusion model and interpolates the remaining frames using neighboring stylized key frames.Key frames are sampled every K frames; each non-key frame is interpolated from its two neighboring stylized key frames.
- Frame propagation: Frame propagation warps each stylized key frame to neighboring frames using guided path matching with color, positional, edge, and temporal guidance.Each key frame is propagated to its preceding K −1 and succeeding K −1 frames, producing two results for an intermediate frame.
- Frame blending: Frame blending combines propagated colors and gradients by selecting lower-error patches, then applies contrast-preserving blending to produce the initial blended image.The framework uses the first two Ebsynth blending steps for each non-key frame.
- Frame blending: The framework omits Poisson blending because it can cause artifacts in non-flat regions and is relatively time-consuming.The initial blended image is therefore used as the final result.
5. Experimental Results
Experiments evaluate runtime, comparisons, ablations, encoding, key-frame frequency, control, applications, and failure cases. Results support improved temporal consistency and flexible rendering, while exposing dependencies on optical flow, key-frame coverage, and sampling choices.
- Runtime: 14.23s and 1.49s per frame are required for key-frame and non-key-frame translations, respectively, on 512×512 videos.Overall runtime is 1.49 + 12.74/K seconds per frame.
- Quantitative comparison: The method achieves the best temporal consistency and second-best frame editing accuracy among the evaluated methods.The evaluation reports Fram-Acc, Tmp-Con, and Pixel-MSE, alongside a 30-participant user study.
- Hierarchical cross-frame constraints: Pixel-aware latent fusion coherently renders local details such as hairstyles and acne, whereas global constraints and shape-aware fusion address less complete forms of consistency.Cross-frame attention maintains global style, adaptive latent adjustment can preserve hair color, and shape-aware fusion handles local movement but not pixel-level consistency for complex motion.
- Hierarchical cross-frame constraints: Pixel-aware latent fusion restores local details disrupted by diffusion noise and denoising, reducing ghosting artifacts in interpolated non-key frames.Consistent key frames provide the pixel information used during propagation and blending.
- Fidelity-oriented image encoding: The fidelity-oriented image encoding reduces error accumulation, detail loss, and color bias across repeated encoding and decoding.It alleviates artifacts observed with the tested Stable Diffusion VAEs and improves reconstruction error on MS-COCO images.
- Frequency of key frames K: K ∈[5, 20] is recommended because larger K improves pixel-level temporal consistency through interpolation but harms frame editing quality.The trade-off is reflected by improved temporal consistency and lower Fram-Acc at larger key-frame intervals.
- Flexible structure and color control: The pipeline supports content and color control by changing the noisy initialization, adjusting T, and applying color correction.Using a noisy latent version of the input preserves more details than Gaussian-noise initialization.
- Applications: Applications include text-guided pet editing, photorealistic and non-photorealistic rendering, and dynamic virtual characters.The examples use customized models and real human video inputs for some generated characters.
6. Conclusion
The framework adapts image diffusion models for video translation while enforcing temporal consistency, and supports stylization through existing customized image models. Its limitations include optical-flow failures, content–prompt trade-offs, and failed propagation without good key frames.
- The framework adapts image diffusion models to produce high-quality, temporally coherent videos.It uses hierarchical cross-frame constraints to enforce consistency in global style and low-level textures.
- Off-the-shelf image models customized for different styles can be used to accurately stylize videos.
- Optical-flow estimation limits the method on videos with large motions.
- The method faces a trade-off between preserving content and following the textual prompt.
- Propagation can fail when suitable key frames are unavailable.