Source-linked AI summary
Pix2Video: Video Editing using Image Diffusion
Duygu Ceylan, Chun-Hao Paul Huang, Niloy J. Mitra
TL;DR
Text-guided video editing with image diffusion must preserve source content and temporal coherence despite image models being trained on individual frames. Pix2Video uses depth-conditioned editing, self-attention feature injection, and guided latent updates to propagate anchor-frame edits. It achieves results on par with or superior to baselines without additional preprocessing or finetuning, while longer videos can degrade as anchor distance increases.
Problem
Image-based editing workflows produce inconsistent video frames, while temporal propagation must preserve appearance and motion coherence.
Method
Pix2Video edits an anchor frame with a pretrained structure-guided image diffusion model, then propagates changes using self-attention feature injection and guided latent updates.
Results
Pix2Video is on par with or superior to baselines across various inputs and editing tasks without additional preprocessing or finetuning.
Takeaways & Limitations
Training-free reuse of pretrained image generation models supports realistic local and global text-guided video edits without video-specific finetuning.
Takeaways & Limitations
Temporal coherence can degrade in longer videos as the distance from the anchor frame increases.
Abstract
from arXiv · showhide
Image diffusion models, trained on massive image collections, have emerged as the most versatile image generator model in terms of quality and diversity. They support inverting real images and conditional (e.g., text) generation, making them attractive for high-quality image editing applications. We investigate how to use such pre-trained image models for text-guided video editing. The critical challenge is to achieve the target edits while still preserving the content of the source video. Our method works in two simple steps: first, we use a pre-trained structure-guided (e.g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model. We then consolidate the changes by adjusting the latent code for the frame before continuing the process. Our approach is training-free and generalizes to a wide range of edits. We demonstrate the effectiveness of the approach by extensive experimentation and compare it against four different prior and parallel efforts (on ArXiv). We demonstrate that realistic text-guided video edits are possible, without any compute-intensive preprocessing or video-specific finetuning.
1. Introduction
Pix2Video adapts pretrained image diffusion models for text-guided video editing without additional training. It propagates an edit from an anchor frame while balancing prompt faithfulness, temporal coherence, and source-video plausibility.
- Image diffusion models support high-quality conditional generation and inversion-based editing of real images.
- Applying image-editing workflows independently to video frames produces inconsistent results across time.
- Pix2Video edits one frame with textual guidance, then propagates the change across the remaining frames.
- Self-attention feature injection and guided latent updates preserve appearance while respecting motion dynamics.
- Across local and global edits, Pix2Video is on par with or better than several baselines without compute-intensive preprocessing or video-specific finetuning.
- The training-free method uses pretrained image generation models for video editing without preprocessing or additional inference overhead.
2. Related Work
Related work spans image diffusion control, video generation, layered representations, and patch-based stylization. Existing video approaches remain constrained by immature video models, expensive per-video training, domain assumptions, or unseen content.
- Diffusion models have become prominent for high-quality image generation and support conditioning through signals such as sketches, segmentation, lines, and depth.
- Video-generation methods often target unconditional generation or use guidance channels while being trained for specific domains.
- Video generation models based on transformers or diffusion remain in their infancy because temporal generation is complex and large annotated video datasets are limited.
- Layered neural representations enable text-guided video editing but may require extensive per-video training lasting 7–10 hours.
- Patch-based video stylization can achieve high quality but is limited when regions are absent from provided style keyframes.
3. Our Method
Pix2Video edits videos with a fixed image diffusion model by combining structure conditioning, cross-frame self-attention feature injection, and guided latent updates. DDIM inversion initializes frames, while temporal guidance and early-step updates promote coherent propagation.
- The method targets coherently edited frame sequences using a pretrained, fixed image diffusion model without example-specific finetuning.
- Depth predictions provide a structure channel that helps the image model capture motion dynamics and geometric changes.
- Self-attention feature injection lets the current frame attend to earlier-frame features and preserve appearance changes across the sequence.
- The method concatenates anchor and previous-frame features for key and value computation, using the first frame as the anchor.
- Guided latent updates compare predicted clean images between consecutive frames with an L2 loss before continuing diffusion.
- Latent updates are applied during the first 25 of 50 diffusion steps because later updates can reduce image quality.
- DDIM inversion supplies the initial noise for each frame, with a caption for the first frame used as the source inversion prompt.
4. Evaluation
Pix2Video is evaluated on DAVIS videos against image- and video-editing baselines using faithfulness and temporal-coherency metrics, qualitative examples, user preferences, and ablations. It supports diverse edits while avoiding heavy preprocessing and finetuning, and its design choices improve temporal consistency and structural preservation.
- Dataset: Pix2Video is evaluated on DAVIS videos ranging from 50 to 82 frames, using prompts from prior work or user consultation.
- Baselines: The evaluation compares Pix2Video with optical-flow propagation, neural-atlas editing, per-frame diffusion editing, and video-specific finetuning baselines.The baselines include Jamriska et al., Text2Live, SDEdit, and Tune-a-Video.
- Efficiency: Pix2Video requires neither heavy preprocessing nor per-video or per-edit finetuning.The method uses an existing depth-conditioned image diffusion model and performs video inversion once per video.
- Metrics: The evaluation measures prompt faithfulness with CLIP-Text and temporal coherency with CLIP-Image, optical flow, and Pixel-MSE.CLIP-Text compares prompt and frame embeddings, while CLIP-Image compares embeddings of consecutive frames.
- Qualitative results: Pix2Video handles localized and global edits, multiple foreground objects, and reflections without explicit masks.Examples include changing object attributes and overall video style.
- Quantitative results: Pix2Video attains the highest CLIP-Text score and fairly good temporal-coherency scores, while per-frame editing has the lowest CLIP-Image and highest Pixel-MSE.Baseline limitations include failures under unreliable flow, multiple foreground objects, or absent structure cues.
- User study and ablations: A user study found Pix2Video more faithful and generally preferred, while ablations favored combining a fixed anchor with the previous frame and guided latent updates.Removing latent updates worsened CLIP-Image and Pixel-MSE, and weaker injection choices produced temporal or structural artifacts.
5. Conclusion and Future Work
Pix2Video edits video clips with a pretrained, fixed text-to-image model and performs comparably to or better than baselines without preprocessing or finetuning. The authors identify temporal coherency and longer-video quality degradation as areas for future improvement.
- Pix2Video uses a pretrained and fixed text-to-image generation model for editing video clips.
- The method is on par with or superior to baselines without additional preprocessing or finetuning.
- Temporal coherency remains an area for improvement, with patch-based and CLIP similarity proposed as future latent-update guidance.
- Quality can degrade in longer videos as the distance from the anchor frame increases.
Appendices
The appendices provide details on the perceptual study, comparison setup, and additional ablations, while video results are available on the accompanying webpage.
- The document details the perceptual study, comparison setup, and additional ablations.
- Video-based results are available on the accompanying webpage.
A. Comparison Details.
The comparison details describe how neural atlases and foreground masks are prepared for Text2Live comparisons and identify the implementation used for Tune-a-Video comparisons.
- Text2Live comparisons use author-provided neural atlases for the swan, dog, and car examples.
- For other Text2Live examples, masks are computed with MaskRCNN for detected categories or per-frame foreground object selection.
- Tune-a-Video comparisons use the implementation provided by its authors.
B. Perceptual Study
The perceptual study compares Pix2Video with four baselines using randomized result placement and asks viewers which edit better represents the prompt.
- Each survey shows the input video alongside two edited results, one from Pix2Video and one randomly selected from four baselines.
- The two edited results are randomly placed in the middle or right columns to avoid bias.
- One survey question asks which edited result better represents the prompt.
C.1. Effect of layer choice in feature injection.
Feature injection is most effective in decoder self-attention layers, while extending it to the bottleneck or all UNet layers offers limited benefit and can reduce sharpness.
- Layer selection: Injecting features into deeper decoder layers, especially layers 13-16, produces significant structural and appearance consistency.The UNet contains 16 layers, including six encoder blocks, nine decoder blocks, and one bottleneck.
- Layer selection: The ablation compares feature injection across all UNet layers against injection restricted to the decoder.Table 3 reports the quantitative comparison between these two layer choices.
- Layer selection: Decoder-only injection balances CLIP-Image, Pixel-MSE, and CLIP-Text performance better than injecting features at all layers.All-layer injection yields on-par CLIP-Image and slightly better Pixel-MSE, but produces blurrier outputs and slightly worse CLIP-Text errors.
D. More Results and Discussion
Pix2Video improves temporal consistency over naive per-frame image editing and can be combined with keyframe-based style propagation for more stable results.
- Additional comparisons: Per-frame Null-inversion and Prompt-to-Prompt editing produce inconsistent results across video frames.Prompt2Prompt can localize edits but cannot guarantee consistency across images.
- Style propagation: Using every third Pix2Video frame as a keyframe for style propagation improves temporal stability without visibility-change artifacts from a single keyframe.The method's depth cue preserves input structure, supporting this post-processing combination.
- Feature injection: Feature injection into decoder layers 8-16 improves consistency, while adding the bottleneck layer does not yield significant improvement.Figure 9 compares decoder-only injection with injection that also includes bottleneck layer 7.