Source-linked AI summary
FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, Sen He
TL;DR
Text-to-video editing must alter a source video while keeping its frames visually consistent, yet spatio-temporal attention can aggregate irrelevant patches. FLATTEN introduces optical-flow-guided attention that links patches along shared trajectories without training, and experiments report state-of-the-art benchmark performance, especially for visual consistency.
Problem
Text-to-video editing requires consistent edited frames and preserved source motion, while spatio-temporal attention can introduce irrelevant patch information.
Method
FLATTEN uses optical flow from the source video to make patches on the same cross-frame trajectory attend to each other during diffusion.
Results
Experiments report new state-of-the-art performance on existing text-to-video editing benchmarks, especially in maintaining visual consistency.
Takeaways & Limitations
FLATTEN is training-free and can be integrated into existing diffusion-based text-to-video editing methods to improve visual consistency.
Takeaways & Limitations
The approach excels at style transfer, coloring, and texture editing but is relatively limited in dramatic structure editing.
Abstract
from arXiv · showhide
Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's U-Net to address the inconsistency issue for text-to-video editing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual consistency in the edited videos. Additionally, our method is training-free and can be seamlessly integrated into any diffusion-based text-to-video editing methods and improve their visual consistency. Experiment results on existing text-to-video editing benchmarks show that our proposed method achieves the new state-of-the-art performance. In particular, our method excels in maintaining the visual consistency in the edited videos.
1 INTRODUCTION
Text-to-video editing must maintain visual consistency and source motion, but inflated spatio-temporal attention can aggregate irrelevant patches. FLATTEN uses optical-flow-guided attention to connect patches along shared trajectories without training, improving consistency.
- Challenge: Text-to-video editing changes a source video's visual appearance from a textual prompt while requiring consistent content and preserved source motion.The edited video should remain visually smooth and structurally close to the source.
- Challenge: Inflating spatial self-attention into spatio-temporal attention combines patches across frames, but each patch may attend to irrelevant video patches.This approach enables temporal context with few or no fine-tuning steps, yet remains insufficient for visual consistency.
- Method: FLATTEN guides attention by enforcing interactions among patches on the same optical-flow path across different frames.It estimates source-video optical flow, computes patch trajectories, and uses them to guide attention.
- Method: FLATTEN integrates into existing diffusion-based text-to-video editing approaches without training or fine-tuning.The framework is designed to preserve per-frame feature distributions during the diffusion process.
- Results: Experiments report new state-of-the-art performance on existing text-to-video editing benchmarks, especially for visual consistency.The contribution passages characterize the method as a framework for high-quality and highly consistent editing.
2 RELATED WORK
Prior work extends image-generation and image-editing methods into video, but training costs, complex motion, and inconsistent outputs remain concerns. FLATTEN instead uses optical flow to guide attention and integrates with existing video-editing frameworks without training or fine-tuning.
- Image and Video Generation: Diffusion models have become prominent for image generation, with methods operating in pixel space or autoencoder latent space.The related work situates later video methods as extensions of image-generation models.
- Image and Video Generation: Video generation extends image generation into an additional dimension, with recent models adapting text-to-image approaches to spatio-temporal domains.Examples include spatio-temporal factorized U-Nets and latent-space video diffusion models.
- Text-to-Image Editing: Text-to-image editing methods manipulate images using noise-denoising, cross-attention guidance, or feature injection, but exclusive reliance on them can produce inconsistent video outputs.The passage lists SDEdit, Pix2pix-Zero, Prompt-to-Prompt, and PNP-Diffusion as representative approaches.
- Text-to-Video Editing: Text-to-video editing includes structure-driven, layered, fine-tuned, and training-free approaches, but existing methods face training costs or difficulties modeling complex motion.Recent methods extend pretrained image diffusion models into text-to-video editing.
- Proposed Approach: FLATTEN differs by using optical flow to guide attention during diffusion and integrating into existing video-editing frameworks without training or fine-tuning.The framework is presented as improving overall visual consistency for text-to-video editing.
3 METHODOLOGY
The framework inflates a 2D diffusion U-Net temporally and combines dense spatio-temporal attention with optical-flow-guided FLATTEN without adding parameters. FLATTEN samples latent patch trajectories from optical flow, restricts attention to patches sharing a trajectory, and supports training-free, consistent video editing.
- Preliminaries: DDIM inversion converts the input latent video into z_T for reconstruction and subsequent editing.
- Framework: The framework expands a text-to-image diffusion U-Net along the temporal axis and incorporates FLATTEN without introducing new parameters.DDIM inversion is used in the latent space to retain generated-video fidelity.
- FLATTEN Integration: Dense spatio-temporal attention first produces H, which supplies FLATTEN's keys and values while the U-Net weights remain frozen.
- Optical Flow Estimation: RAFT estimates optical flow between consecutive RGB frames, whose displacement fields are downsampled to latent-space resolution.
- Patch Trajectory Sampling: Latent patch trajectories are sampled across frames from the downsampled displacement fields, with new trajectories created for appearing patches.During occlusions, one trajectory is randomly continued and conflicting trajectories are stopped, ensuring unique patch assignment.
- Attention Process: For each query, FLATTEN gathers embeddings from other patches on the same trajectory and applies multi-head attention without additional positional encoding.
- Attention Process: Flow-guided attention connects patches across frames, eliminates irrelevant-patch aggregation effects, and can be integrated into diffusion-based editing without additional training.
4 EXPERIMENTS
Experiments evaluate FLATTEN on TGVE-D and TGVE-V using automatic metrics, qualitative comparisons, compatibility tests, ablations, and a user study. The results indicate stronger editing performance and visual consistency, including when FLATTEN is integrated into another method.
- Experimental Settings: The evaluation uses 53 videos from LOVEU-TGVE, with 16 DAVIS videos in TGVE-D and 37 Videvo videos in TGVE-V.Videos are resized to 512 × 512, contain 32 frames, and include a ground-truth caption plus four creative editing prompts.
- Experimental Settings: The study measures textual alignment with CLIP-T, visual consistency with flow warping error Ewarp, and combined editing performance with Sedit = CLIP-T/Ewarp.The authors note that Ewarp alone can favor unchanged source videos, motivating the combined score.
- Quantitative Comparison: FLATTEN outperforms the compared methods on CLIP-T, PickScore, and Sedit across both TGVE-D and TGVE-V, while its Ewarp is 0.1 × 10^-3 lower than TokenFlow.The authors also report superior overall performance across the evaluation metrics.
- Qualitative Results: Qualitatively, FLATTEN produces prompt-consistent videos while avoiding structural errors and intermittent visual artifacts observed in competing methods.Examples include incorrect truck counts, failure to edit, and a white sun appearing and disappearing across frames.
- Plug-and-Play FLATTEN: Integrating FLATTEN into ControlVideo reduces Ewarp from 6.81 to 4.78 and increases Sedit from 40.70 to 56.42, while CLIP-T changes from 27.72 to 26.97.The accompanying example shows a fish disappearing without FLATTEN but remaining visually consistent with it.
- Ablation Study: The ablation compares DSTA, FLATTEN, and two combinations, finding that the second combination performs better and is adopted as the final solution.The first combination sometimes causes blurring, whereas the second directly uses DSTA outputs as FLATTEN queries, keys, and values.
5 CONCLUSION
The paper concludes that FLATTEN provides a training-free framework for highly consistent text-to-video editing and achieves state-of-the-art benchmark performance. It can also be integrated into other diffusion-based editing methods to improve their visual consistency.
- Conclusion: FLATTEN is a training-free flow-guided attention framework that achieves state-of-the-art performance on existing text-to-video editing benchmarks.The reported strength is especially visual consistency in edited videos.
- Conclusion: FLATTEN can be seamlessly integrated into other diffusion-based text-to-video editing methods to improve their visual consistency.
A DDIM INVERSION WITH FLATTEN
FLATTEN is also applied during DDIM inversion to improve latent-noise estimation and source-video reconstruction. Reconstructions with FLATTEN preserve more visual details and achieve higher PSNR and SSIM.
- DDIM Inversion: Using FLATTEN during DDIM inversion improves the estimated latent noise and restores more source-video details during reconstruction.The fish eyes and other details are reconstructed with FLATTEN but are missing without it.
- DDIM Inversion: 33.89dB PSNR and 0.9159 SSIM are achieved with FLATTEN, compared with 32.74dB and 0.8974 without it.
B ADDITIONAL QUALITATIVE RESULTS
Additional qualitative comparisons report that the training-free FLATTEN framework produces high-quality, highly consistent text-to-video edits. The videos also exhibit strong semantic alignment.
- Additional Qualitative Results: Additional qualitative results show that flow-guided attention supports high-quality and highly consistent text-to-video editing without training.
- Additional Qualitative Results: Videos produced by FLATTEN exhibit strong visual consistency and semantic alignment alongside high visual quality.
C USER STUDY DETAILS
The user study compares text-to-video editing methods from semantic alignment, visual consistency, and motion and structure preservation perspectives. Qualitative figures provide additional comparisons and examples of consistent outputs.
- User study setup: Thirty source videos from TGVE-D and TGVE-V were edited using six text-to-video approaches and FLATTEN.The study involved 16 participants per group.
- Evaluation criteria: Participants evaluated edited videos for semantic alignment, visual consistency, and motion and structure preservation.These criteria respectively assess prompt matching, smooth adjacent frames, and alignment with the source video's motion or structure.
- Qualitative comparisons: Figure 10 presents a qualitative comparison between advanced text-to-video editing approaches and FLATTEN.
- Qualitative comparisons: Figure 11 shows that the approach can output highly consistent videos under different textual prompts.
D LIMITATIONS
FLATTEN is designed around optical flow from the source video, which supports style, coloring, and texture edits but limits dramatic structural transformations. The paper illustrates this boundary with shark-to-drone editing, where the result remains a mechanical shark.
- Scope boundary: FLATTEN excels at style transfer, coloring, and texture editing because it uses optical flow from the source video.
- Scope boundary: The method is relatively limited in dramatic structure editing when the target shape differs substantially from the source.In the reported failure case, sharks become mechanical sharks rather than quadrotor drones.
- Flow estimation: RAFT provides robust trajectory estimates for videos with large and abrupt motions, while more precise flow estimation could improve trajectory prediction.
F ROBUSTNESS TO FLOWS
FLATTEN remains effective when the optical flow guidance is corrupted with noise or replaced by flow from another estimator. These tests indicate robustness to moderate differences in patch trajectories, although noise introduces some artifacts.
- Noisy flow: Adding random Gaussian noise to the pre-computed flow introduces a few artifacts, but FLATTEN still outperforms the baseline without optical-flow guidance.
- Robustness conclusion: The robustness results support adaptability to small differences in patch trajectories and corrupted flow fields.
- Alternative flow estimators: Replacing RAFT flow with GMA flow produces no obvious difference in the output videos.
G RUNTIME EVALUATION
The runtime evaluation measures the time required to edit a 32-frame video across different text-to-video editing models on a single A100 GPU. FLATTEN has a relatively short sampling-stage runtime, though further improvement remains possible.
- Evaluation setup: Runtime was measured for editing a single 32-frame video with different text-to-video editing models on one A100 GPU.
- Runtime results: FLATTEN has a relatively short runtime during the sampling stage.
- Runtime results: The paper identifies scope for further improvement in runtime.