PaperlayerView exact revision on arXiv

Source-linked AI summary

FateZero: Fusing Attentions for Zero-shot Text-based Video Editing

Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, Qifeng Chen

arXiv:2303.09535v3cs.CV

TL;DR

Real-world video editing with diffusion models is challenging because generation is random and image models do not inherently preserve temporal consistency. FateZero performs zero-shot text-driven editing by fusing attention maps captured during inversion, blending source cross-attention, and using spatial-temporal attention. The paper reports temporally consistent style, attribute, and shape editing without per-prompt training or user-specific masks, while very different shape compositions remain unsatisfactory.

  • Problem

    Real-world video editing is difficult because diffusion generation is highly random and pretrained image models lack temporal-aware information such as motion and 3D shape.

  • Method

    FateZero captures and fuses inversion self- and cross-attention maps, uses source cross-attention for adaptive blending, and introduces spatial-temporal attention.

  • Results

    FateZero enables zero-shot temporal-consistent video style and local attribute editing with a pretrained text-to-image model, plus shape-aware editing with a video diffusion model.

  • Takeaways & Limitations

    The framework provides a general zero-shot editing approach applicable to widely existing text-to-image and text-to-video models.

  • Takeaways & Limitations

    Zero-shot editing performs poorly for new concept compositions or very different shapes, such as changing a black swan into a yellow pterosaur.

Abstract

from arXiv · show

The diffusion-based generative models have achieved remarkable success in text-based image generation. However, since it contains enormous randomness in generation progress, it is still challenging to apply such models for real-world visual content editing, especially in videos. In this paper, we propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. To edit videos consistently, we propose several techniques based on the pre-trained models. Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information. These maps are directly fused in the editing process rather than generated during denoising. To further minimize semantic leakage of the source video, we then fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt. Furthermore, we have implemented a reform of the self-attention mechanism in denoising UNet by introducing spatial-temporal attention to ensure frame consistency. Yet succinct, our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model. We also have a better zero-shot shape-aware editing ability based on the text-to-video model. Extensive experiments demonstrate our superior temporal consistency and editing capability than previous works.

1. Introduction

FateZero addresses the challenge of temporally consistent, zero-shot text-driven video editing without per-prompt training or user-specific masks. It fuses inversion attention and spatial-temporal attention to preserve motion and structure while enabling style, attribute, and shape editing.

  • Challenges: Video editing with image diffusion models is difficult because image models lack temporal awareness, causing obvious flickering when applied frame by frame.Existing video methods may additionally require atlas learning, keyframe selection, or per-prompt tuning.
  • FateZero: FateZero performs zero-shot video editing without training separately for each target prompt or using a user-specific mask.The framework targets style, attribute, and shape editing in real-world videos.
  • FateZero: The method stores and fuses self- and cross-attention maps throughout DDIM inversion and denoising to preserve motion and structural consistency.Self-attention provides motion information, while cross-attention supports spatial blending and structure preservation.
  • FateZero: An Attention Blending Block uses source-prompt cross-attention to prevent semantic leakage and improve shape-editing capability.This design supports more localized and shape-aware edits.
  • Applications: The framework supports video style editing, local editing, and object replacement, using pretrained text-to-image and video diffusion models.The paper reports applications with both image-based and video-based generative models.

2. Related Work

Related work spans diffusion-based image and video generation, image editing, and video editing. These approaches provide useful generative priors but often lack temporal consistency or require examples, optimization, learned atlases, keyframes, or stronger video models.

  • Video Editing: Video stylization methods may rely on example styles or frame-wise image transfer, while post-processing is used to reduce temporal inconsistency.Example-based methods may fail when tracking is lost, and perceptual-distance style transfer can remain imperfect.
  • FateZero Overview: FateZero’s overview takes source and target prompts plus video latents, performs DDIM inversion, and stores attention maps for later editing.The editing stage fuses source and target attention information during denoising, including adaptive spatial blending from source cross-attention.
  • Image and Video Generation Models: Video generation is more difficult than image generation and commonly relies on larger cascaded models and datasets.Some methods initialize video models from text-to-image models, while others use stronger video-generation architectures.
  • Image Editing in Diffusion Model: Diffusion image-editing methods use inversion, noise blending, feature reconstruction, cross-attention guidance, prompt reweighting, or optimization.These methods generally edit images by modifying the denoising process under a target prompt.
  • Image Editing in Diffusion Model: Naively applying image-editing methods independently to video frames produces flickering.This limitation motivates methods that explicitly preserve temporal information during video editing.

3. Methods

FateZero adapts pretrained diffusion models for zero-shot video editing by preserving inversion-time structure and motion, blending attention to limit semantic leakage, and introducing spatial-temporal self-attention for frame consistency.

  • 3. Methods: FateZero targets zero-shot text-driven video editing without per-prompt optimization or user-provided masks, using pretrained text-to-image and video diffusion models.The framework supports style, attribute, and shape editing across Stable Diffusion and Tune-A-Video.
  • 3.3. Shape-Aware Video Editing: The method applies Stable Diffusion for video appearance editing and a pretrained video diffusion model for more challenging object shape editing.Reported examples include local attributes, global styles, and zero-shot object-shape changes in videos.
  • 3.1. Preliminary: Latent Diffusion and Inversion: DDIM inversion maps clean latents to noise, which can then be denoised under an edited prompt to reconstruct the modified content.The standard editing pipeline uses deterministic inversion and target-prompt denoising, with attention constraints from source-prompt reconstruction.
  • 3.2. FateZero Video Editing: FateZero stores self- and cross-attention maps at every inversion step and replaces corresponding denoising maps to preserve source motion and structure.Inversion attention provides more accurate structure guidance than fusion only during reconstruction.
  • 3.2. FateZero Video Editing: Cross-attention from the source prompt is thresholded into a binary mask that blends source and edited self-attention, limiting semantic leakage while retaining unchanged structure.The mask supports local attribute and shape editing by combining target content with source pose, background, and structure.
  • 3.2. FateZero Video Editing: Spatial-temporal self-attention reshapes pretrained self-attention to capture both within-frame structure and temporal correspondence without changing pretrained weights.The mechanism uses warped middle-frame features as key and value for attribute and style editing, with f = 2 frames used as key and value.

4. Experiments

Experiments show that FateZero supports style, attribute, and shape editing while preserving temporal consistency and achieving strong comparative performance.

  • Applications: The framework performs zero-shot style and local attribute editing with a pretrained text-to-image model, preserving image structure and temporal motion.Feather texture and color changes remain consistent across frames, while Ukiyo-e and Makoto Shinkai styles preserve structure and motion.
  • Quantitative Evaluation: The method achieves the best temporal consistency and comparable framewise editing accuracy against an optimization-based image editing baseline in CLIP-based evaluation.The evaluation uses Tem-Con for consecutive-frame similarity and Frame-Acc for framewise editing accuracy.
  • Applications: FateZero also supports difficult shape-aware editing with a pretrained video model, while avoiding inaccurate shapes and wrong motion caused by full-resolution attention fusion.The proposed Attention Blending Block addresses the failures observed in the ablation study.
  • Ablation Studies: Using reconstruction attention instead of inversion attention fails to detect the boat position and provide suitable motion guidance for zero-shot video editing.The comparison uses the prompt transition ‘deserted shore’ to ‘glacier shore’.
  • Quantitative Evaluation: User studies with 20 subjects show that FateZero receives the best average ranking for editing quality, image fidelity, and temporal consistency.Each study included nine comparison sets.
  • Baseline Comparisons: Compared with baselines, framewise SDEdit lacks localization and consistency, Null-text optimization is costly and inconsistent, and NLA struggles with new shapes or 3D structures.Tune-A-Video ranks second in editing quality and image fidelity but has difficulty reproducing exact motion and spatial position.
  • Ablation Studies: Without self-attention fusion, input-video details disappear, while unmasked full self-attention lets the original jeep structure mislead Porsche generation.The ablation highlights the role of spatially masked attention blending in preserving unedited details and enabling shape changes.

5. Conclusion

FateZero is a temporal-consistent, zero-shot framework for editing video attributes, styles, and shapes. It combines attention guidance during DDIM inversion and editing with spatial-temporal self-attention, but cannot easily generate radically new motions or shapes.

  • Conclusion: FateZero performs temporally consistent zero-shot editing of video attributes, styles, and shapes.The framework uses pretrained diffusion models rather than per-prompt training.
  • Conclusion: The framework fuses cross-attention and spatial-temporal self-attention during DDIM inversion to provide fine-grained motion and structure guidance at denoising steps.An Attention Blending Block further improves shape-editing performance.
  • Limitation & Future Work: During shape editing, the method struggles to generate totally new motion or very different shapes because motion comes from a one-shot video diffusion model.Examples include ‘swim’ to ‘fly’ and ‘swan’ to ‘pterosaur’.

A. Implementation Details

Implementation details define attention-fusion hyperparameters and use different settings for style or attribute editing versus shape editing.

  • Hyperparameters: The implementation uses ts, tc, and τ to control self-attention blending, cross-attention fusion, and the blending-mask threshold.The algorithm uses these hyperparameters during attention fusion.
  • Hyperparameters: Smaller ts fuses more inversion self-attention to preserve source structure and motion.ts denotes the last timestep of self-attention blending.
  • Hyperparameters: Smaller tc fuses more inversion cross-attention to preserve the spatial semantic layout.tc denotes the last timestep of cross-attention fusion.
  • Hyperparameters: Smaller τ uses more editing self-attention to improve shape-editing results.τ is the threshold for the blending mask.
  • Task-Specific Settings: For style and attribute editing, the settings are ts = 0.2T, tc = 0.3T, and τ = 1.0; for shape editing, they are ts = 0.5T, tc = 0.5T, and τ = 0.3.The first setting preserves more source structure and motion, while the second allows more freedom for new motion and 3D shape generation.

B. Demo Video

The demo video presents FateZero’s editing results, animation, baseline comparisons, and additional applications, alongside the full inversion and denoising algorithm.

  • Demo Contents: The demo showcases style, local attribute, and shape editing results to validate the proposed method.It also includes animation explaining the method and baseline comparisons in video.
  • Algorithm: Algorithm 1 takes source latent and prompts, performs DDIM inversion while recording attention maps, then denoises with attention fusion to produce the edited latent.The output is the final edited latent code.
  • Attention Fusion: Attention fusion uses an edited-word cross-attention mask and derives the self-attention blending mask by thresholding source cross-attention.The fusion is applied through ATT-FUSION during denoising.

C. Limitation and Future Work

FateZero struggles with new concept composition and large shape changes, motivating stronger video diffusion models as future work.

  • Limitation: FateZero performs poorly on new concept composition and generation of very different shapes.Editing “black swan” to “yellow pterosaur” is reported as unsatisfactory.
  • Limitation and mechanism: The reported limitation concerns zero-shot editing rather than the attention-fusion procedure itself.The cited algorithm uses cross-attention fusion and self-attention blending with threshold-based timing conditions.
  • Future work: A stronger video diffusion model is proposed as a possible way to alleviate the limitation and is left for future work.
Loading 2303.09535v3…