Source-linked AI summary
TokenFlow: Consistent Diffusion Features for Consistent Video Editing
Michal Geyer, Omer Bar-Tal, Shai Bagon, Tali Dekel
TL;DR
The paper tackles limited visual quality and control in video editing, especially the difficulty of preserving motion and layout while changing content. It introduces TokenFlow, which propagates edited diffusion features using inter-frame correspondences, and reports temporally consistent, state-of-the-art edits on diverse videos. The method is designed for motion-preserving edits and does not handle edits requiring structural changes.
Problem
Video models lag behind image models in visual quality and user control, while existing image-diffusion video editing approaches inadequately preserve temporal consistency.
Method
TokenFlow uses a fixed pretrained text-to-image diffusion model, jointly edits sampled keyframes, and propagates their edited tokens according to original inter-frame feature correspondences.
Results
The method achieves state-of-the-art editing results on diverse videos with complex motions and significantly improves temporal consistency over existing baselines.
Takeaways & Limitations
Consistent diffusion feature representations provide a practical route to coherent text-driven video editing with a text-to-image model.
Takeaways & Limitations
TokenFlow preserves the original motion and therefore cannot handle edits requiring structural changes; it also depends on the underlying image-editing method preserving frame structure.
Abstract
from arXiv · showhide
The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos. Webpage: https://diffusion-tokenflow.github.io/
1 INTRODUCTION
The paper addresses the gap between controllable image editing and temporally consistent video editing by using diffusion features to preserve inter-frame structure. TokenFlow enforces feature correspondences across frames and reports state-of-the-art results on diverse videos with complex motion.
- Existing video models remain limited in resolution, length, or the complexity of dynamics they can represent.
- TokenFlow explicitly enforces the original inter-frame correspondences on edited diffusion features to improve temporal consistency.The approach relies on a correlation between temporal redundancy in RGB videos and diffusion feature representations.
- TokenFlow increases temporal consistency in videos generated by a text-to-image diffusion model by enforcing semantic correspondences across frames.
- The paper presents an empirical analysis of diffusion features across videos.
- The method achieves state-of-the-art editing results on diverse videos depicting complex motions.
2 RELATED WORK
Prior approaches use frame-wise editing, temporal post-processing, feature manipulation, or attention-based video generation, but face limitations with varying content and complex dynamics. TokenFlow builds on temporal redundancy in natural videos and shows that analogous redundancy exists in diffusion features.
- Consistent video stylization: Frame-wise stylization followed by post-processing reduces high-frequency flicker but struggles when frames vary substantially in content.
- Consistent video stylization: Patch-based propagation methods struggle with illumination changes and complex dynamics, while requiring manually consistent keyframe edits.
- Consistent video stylization: TokenFlow leverages temporal redundancies shared by natural videos and text-to-image diffusion feature spaces to achieve consistency.
- Controlled generation via diffusion features manipulation: Diffusion feature manipulation has been adapted to image editing and generation tasks, including semantic appearance swapping through feature correspondences.
- Feature correspondences: Source-frame diffusion features can reconstruct nearby target frames through nearest-feature swapping or RGB warping based on feature-derived correspondences.
- TokenFlow: TokenFlow jointly edits sampled keyframes and propagates their edited tokens across the video using precomputed inter-frame correspondences.
- Controlled generation via diffusion features manipulation: Extending self-attention across frames can produce globally coherent video edits, but this does not by itself ensure the desired temporal consistency.
3 PRELIMINARIES
The paper introduces diffusion models as progressive denoising systems and describes Stable Diffusion as a latent-space text-to-image model with attention-based feature processing. DDIM inversion provides intermediate noisy representations used by the framework.
- Diffusion Models: Diffusion models approximate a data distribution through progressive denoising from Gaussian noise.
- Diffusion Models: DDIM inversion starts from a clean image and obtains intermediate noisy images across diffusion steps.
- Stable Diffusion: Stable Diffusion operates in a latent image space using a pretrained encoder and decoder around a U-Net architecture.
- Stable Diffusion: Stable Diffusion’s U-Net includes residual, self-attention, and cross-attention blocks, with cross-attention conditioning features on the text prompt.
- Stable Diffusion: Self-attention projects features into queries, keys, and values before computing attention-weighted layer outputs.
4 METHOD
TokenFlow edits keyframes with an image-editing method, then propagates their diffusion features across the video using original-video token correspondences. Randomized keyframe selection and distance-weighted propagation support consistent feature representations and smooth transitions.
- The framework applies a fixed text-to-image diffusion model to video frames while preserving the input video's motion and semantic layout.
- 4.1 KEYFRAME SAMPLING AND JOINT EDITING: TokenFlow builds a base set of edited keyframe tokens, with each keyframe querying the others and aggregating their information for a unified appearance.The base tokens are defined for each layer of the diffusion network.
- 4.2 EDIT PROPAGATION VIA TOKENFLOW: For each frame, TokenFlow finds nearest-neighbor token correspondences in the original video and uses adjacent future and past keyframes for propagation.Correspondences are defined over spatial token locations using cosine distance.
- 4.2 EDIT PROPAGATION VIA TOKENFLOW: Propagation linearly combines tokens from adjacent keyframes using a distance-proportional weight, including for sampled keyframes, to ensure smooth transitions.The same token-modification rule is applied across frames and self-attention layers.
- At each denoising step, it randomly samples keyframes, edits them with an image-editing technique and extended attention, then propagates the edit across the entire video with TokenFlow.The procedure alternates keyframe editing with full-video propagation.
5 RESULTS
TokenFlow produces temporally consistent video edits that adhere to target prompts while preserving scene motion and layout, outperforming prominent baselines qualitatively and according to temporal-consistency evaluations.
- 5 RESULTS: TokenFlow edits diverse DAVIS and Internet videos while maintaining temporal consistency and adherence to the target text prompt.Evaluations cover animals, food, humans, and objects in motion at 384×672 or 512×512 resolution over 40–200 frames.
- 5.1 QUALITATIVE EVALUATION: Compared with prominent baselines, TokenFlow better balances prompt adherence and temporal consistency, whereas competing methods exhibit jitter, inconsistent content, or reduced frame quality.The comparison includes Tune-A-Video, per-frame PnP editing, Gen-1, Text2Video-Zero, and Fate-Zero.
- 5.2 QUANTITATIVE EVALUATION: User studies and quantitative metrics evaluate temporal consistency through warp-error and human judgments, while CLIP similarity measures fidelity to the target prompt.The reported evaluation also notes that optical-flow-based warp-error can miss artifacts and long-range inconsistencies.
- 5 RESULTS: The method cannot handle edits requiring structural deviations because it follows feature correspondences from the original video.This limitation arises from preserving the original motion and structure.
- 5.2 QUANTITATIVE EVALUATION: TokenFlow’s ablations replace feature propagation with extended attention or fixed keyframes to assess their roles in enforcing temporal consistency.The experiments compare TokenFlow against extended attention and remove random keyframe selection.
6 DISCUSSION
TokenFlow represents videos in diffusion feature space and uses that representation to improve reconstruction and temporal consistency, while remaining constrained by structure-preserving edits and decoder artifacts.
- 6 DISCUSSION: TokenFlow reconstructs videos more effectively than vanilla DDIM inversion according to PSNR and LPIPS evaluations.The reconstruction excludes keyframe editing and evaluates the TokenFlow representation directly.
- 6 DISCUSSION: The framework preserves original motion but cannot support edits requiring structural changes.It also depends on an image-editing method that preserves frame structure; otherwise propagated correspondences can create visual artifacts.
- 6 DISCUSSION: The LDM decoder introduces some high-frequency flickering, although the authors identify improved decoders as a possible remedy.The paper characterizes this flickering as minor and states it can be eliminated with existing techniques.
A IMPLEMENTATION DETAILS
The implementation uses Stable Diffusion with DDIM inversion and keyframe-based sampling, while reducing per-frame editing time through sparse attention computation.
- StableDiffusion: Stable Diffusion v2.1 serves as the pretrained text-to-image model.The checkpoint is provided through the official HuggingFace webpage.
- DDIM inversion: DDIM inversion uses deterministic 50-step sampling, classifier-free guidance scale 1, and 1000 forward steps for video inversion.Self-attention input tokens are extracted during inversion.
- Runtime: Computing self-attention outputs only on keyframes reduces per-frame editing time by 20%.The 1000-step inversion is the main runtime bottleneck, while substantially fewer steps can often suffice.
- Hyper-parameters: The sampling configuration uses classifier-free guidance scale 7.5 and randomly selected keyframes within frame intervals of 8.The interpolation weights in equation 5 use neighboring future and past keyframes through a sigmoid function.
- Baselines: Baseline comparisons use official implementations or platforms for Tune-a-Video, Gen-1, Text-to-Video-Zero, and Fate-Zero.Gen-1 outputs videos with different lengths and frame rates, preventing warp-error computation on its results.