Source-linked AI summary
Dreamix: Video Diffusion Models are General Video Editors
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, Yedid Hoshen
TL;DR
Video editing methods based on diffusion had been far less developed than image editing, especially for changing motion while preserving the original video. Dreamix adapts a text-conditioned video diffusion model using degraded-video conditioning and mixed finetuning, and the authors report broad editing, image-animation, and subject-driven generation capabilities, with computational cost and prompt-dependent success as limitations.
Problem
Few diffusion-based methods perform text-based editing of general videos, where alignment, fidelity, quality, motion synthesis, and temporal consistency are all required.
Method
Dreamix combines degraded input-video conditioning with finetuning on the original video and unordered frames using masked temporal attention, then applies the editor to image animation and subject-driven generation.
Results
The authors report remarkable video-editing and image-animation abilities, including general real-world image-driven editing and subject-driven video generation, with superior results against state-of-the-art baselines.
Takeaways & Limitations
Dreamix extends text-conditioned diffusion editing beyond appearance to general video motion editing and supports image animation and subject-driven video generation.
Takeaways & Limitations
Editing success varies with object and action frequency in the training dataset, and finetuning billions of parameters takes around 30 minutes per video on large hardware accelerators.
Abstract
from arXiv · showhide
Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtaining high-fidelity to the original video requires retaining some of its high-resolution information, we add a preliminary stage of finetuning the model on the original video, significantly boosting fidelity. We propose to improve motion editability by a new, mixed objective that jointly finetunes with full temporal attention and with temporal attention masking. We further introduce a new framework for image animation. We first transform the image into a coarse video by simple image processing operations such as replication and perspective geometric projections, and then use our general video editor to animate it. As a further application, we can use our method for subject-driven video generation. Extensive qualitative and numerical experiments showcase the remarkable editing ability of our method and establish its superior performance compared to baseline methods.
1. Introduction
Dreamix addresses the limited availability of video-editing methods by adapting text-conditioned video diffusion models for general appearance and motion editing. It also introduces mixed finetuning, image animation, and subject-driven video generation.
- Very few existing methods use text-driven diffusion for video editing, despite established text-based image-editing approaches.
- The method preserves fidelity by initializing from degraded input video information and finetuning on the original video.
- Mixed finetuning combines full temporal training with unordered-frame training using masked temporal attention, improving motion-edit quality.
- Dreamix extends video editing to text-guided image animation and subject-driven video generation from image collections.
- Dreamix is presented as the first method for general text-based appearance and motion editing of real-world videos.
2. Related Work
Prior work established diffusion-based image editing and video-generation foundations, but video editing remained limited. Dreamix uses a video backbone to edit both appearance and motion while targeting temporal smoothness.
- Diffusion models became a powerful paradigm for image generation and were extended toward text-to-video generation and image reconstruction.
- Diffusion-based image-editing methods modify inputs through targeted corruption or related mechanisms, but their video applications remain sparse.
- Text-to-image backbones can edit video appearance but generally cannot edit motion and may lack full temporal consistency.
- Dreamix instead uses a text-to-video backbone to enable motion editing while maintaining video smoothness.
3. Background: Video Diffusion Models
Video diffusion models learn to denoise noisy videos conditioned on text and, when applicable, a low-resolution video. Cascaded architectures and preprocessing support high-resolution, multi-application generation and editing.
- The denoising model recovers a ground-truth video from noisy input, a time or noise level, a text prompt, and an optional conditioning video.
- Video diffusion models are trained on large corpora of video–text pairs using a denoising objective.
- Sampling uses stochastic DDIM steps that estimate noise, remove part of it, and add random Gaussian noise.
- High-resolution text-to-video generation is handled with a cascade comprising base generation followed by spatial and temporal super-resolution models.
- Dreamix preprocessing converts inputs into coarse videos, which are corrupted and then upscaled by a finetuned text-guided video diffusion model.
4. General Editing by Video Diffusion Models
Dreamix edits general videos by degrading the input into low spatio-temporal information, guiding diffusion with text, and finetuning on the original video to improve fidelity. Mixed finetuning combines video reconstruction with frame-wise reconstruction to balance motion and appearance.
- 4.1. Text-Guided Video Editing by Inverting Corruptions: Dreamix corrupts an input video by downsampling and adding noise, then uses text-guided cascaded diffusion to synthesize high-resolution edited output.The degraded video supplies low-resolution details, while diffusion generates new high spatio-temporal information aligned with the prompt.
- 4.1. Text-Guided Video Editing by Inverting Corruptions: Dreamix can significantly change subject actions and motions while maintaining temporal consistency and preserving unedited details.Figure 5 illustrates a puppy edited to leap.
- 4.2. Mixed Video-Image Finetuning: The degraded input alone lacks sufficient high-resolution information for preserving fine textures and object identity, motivating preliminary finetuning on the input video.Finetuning is performed once per video, after which multiple prompts can be used without further finetuning.
- 4.2. Mixed Video-Image Finetuning: Full temporal-attention finetuning reconstructs the input video and updates the model prior for both motion and appearance.The first objective requires exact reconstruction of the input video from noisy versions.
- 4.2. Mixed Video-Image Finetuning: Masked-attention finetuning reconstructs frames individually while restricting attention within each frame, enhancing the appearance prior separately from motion.Temporal attention layers use fixed masks, and residual temporal convolution blocks are masked.
- 4.2. Mixed Video-Image Finetuning: The two finetuning objectives are trained jointly with a mixing weight α, while few iterations and a low learning rate reduce overfitting.The method exposes α alongside finetuning steps and learning rate as finetuning hyperparameters.
5. Applications of Dreamix
Dreamix extends general video editing to image animation and subject-driven video generation. It creates coarse videos from images or image collections, then applies text-conditioned video editing to add motion, camera effects, or subject actions.
- Dreamix for Single Images: Simple image-processing operations such as frame replication and geometric transformations provide the coarse video input for general text-conditioned image-to-video editing.This framework supports editing objects and backgrounds and creating dynamic camera motion.
- Dreamix for Single Images: The image-animation framework transforms an image or image set into a coarse video and edits it with Dreamix under a text prompt.For a single image, frame replication creates a static video that can then receive appearance and motion edits.
- Dreamix for Subject-Driven Video Generation: Dreamix uses image collections containing the same subject for text-conditioned video generation.The method applies its finetuning approach to the collection and then generates from text without a conditioning video.
- Dreamix for Subject-Driven Video Generation: Greater diversity of viewing angles and backgrounds in the image collection is beneficial for subject-driven video generation.The collection may also contain different frames from the same video when they show the same subject.
6. Experiments
Experiments show that Dreamix supports text-guided appearance and motion editing, image animation, and subject-driven video generation. Qualitative and user-rated comparisons indicate strong alignment, fidelity, quality, and temporal consistency relative to baselines.
- Video Editing: Dreamix edits motion, appearance, objects, effects, backgrounds, and camera movements while retaining coarse video attributes.Examples include changing a monkey to a dancing bear, adding objects, replacing objects, changing backgrounds, and zooming out.
- Ablation Study: Mixed finetuning is important for motion editing and background, color, or texture changes.The ablation compares no finetuning, video-only finetuning, and the proposed mixed objective.
- Hyperparameter Tradeoffs: Higher noise enables larger edits but reduces fidelity, while more finetuning iterations improve fidelity at higher noise levels.The best qualitative results use high noise with many finetuning iterations.
- Image-driven Videos: Dreamix animates general real-world images by adding moving objects, injecting motion, and creating camera movements.Examples include camel motion, turtle and coffee animation, and buffalo camera movement.
- Subject-driven Video Generation: Dreamix generates videos featuring subjects from image collections, including new actions and surroundings.Demonstrations include a toy fireman weight-lifting, bears walking and drinking, and caterpillars placed in new environments.
- Qualitative Results: Dreamix produces text-aligned video edits while preserving original details and maintaining high visual quality.In qualitative comparisons, Imagen-Video loses resemblance to the input, while PnP has low temporal consistency.
7. Discussion
Dreamix’s limitations concern usability, evaluation, editability, and computational cost, while several future applications remain promising. The authors identify automation and faster inference as key directions.
- Limitations: Optimal hyperparameters can vary between prompts, so automatic selection from input-video and prompt features remains future work.The authors suggest learning a regressor and creating a user-judged training set of optimal per-edit hyperparameters.
- Limitations: Automatic metrics such as CLIP Score for alignment are imperfectly correlated with human preference.The authors call for improved video text-editing metrics, which could also support hyperparameter-selection datasets.
- Limitations: Not all prompt-video pairs produce successful edits, with more frequent training-set objects and actions yielding better preliminary results.The authors suggest predicting successful pairs and using automatic prompt engineering.
- Limitations: Finetuning the billions-parameter VDM requires large hardware accelerators for around 30 minutes per video.Reducing runtime and computational cost would broaden the method’s applications.
- Future Applications: Potential future applications include motion interpolation between image pairs, text-guided inpainting, and outpainting.
8. Conclusion
The paper presents a general approach for text-conditioned editing with video diffusion models, extends it to image animation, and applies it to subject-driven video generation. Extensive experiments report unprecedented results.
- Conclusion: Dreamix provides a general approach for text-conditioned editing using video diffusion models.
- Conclusion: The framework also supports image animation and subject-driven video generation.
- Conclusion: Extensive experiments demonstrated the unprecedented results of the method.
9. Social Impact
Dreamix aims to help users animate personal content, while its personalized approach may align content with user intent despite biases in general VDMs. The method also carries risks associated with capable media generation.
- Opportunities: The work aims to advance tools that enable users to animate their personal content.
- Opportunities: Personalization may better align generated content with user intent despite potential biases in general VDMs.
- Risks: Edited videos could be misused to mislead viewers or engage in targeted harassment.The authors state that future research should continue investigating these concerns.
A.1. Architecture
The architecture uses a cascaded Imagen-Video system with text conditioning, a base video diffusion model, and six spatial or temporal super-resolution models. Implementation choices include selective finetuning, mixed objectives, distillation, DDIM sampling, chunked processing, and tuned noise strengths.
- Architecture: All experiments use Imagen-Video, a pretrained cascaded video diffusion model.
- Architecture: A T5-XXL text encoder computes prompt embeddings used as conditioning by the other models.
- Architecture: The base model generates text-conditioned videos at 16 × 24 × 40 × 3 resolution and 3 fps.
- Finetuning: The models are pretrained on videos and images, and mixed finetuning treats video frames as images.
- Finetuning: Only the base model and first two SSR models are finetuned, using non-distilled versions for those SSR models.
- Finetuning: The mixed video/video-frame finetuning configuration uses α = 0.35 with FTsteps ∈[200, 300].
- Sampling: All experiments use learning rate 6 · 10−6, DDIM sampling with stochastic noise correction, and 32-frame sub-chunks for the final highest-resolution SSR.The sub-chunk outputs are concatenated into 128-frame videos.
B. Human evaluations details
Human evaluations compared edited videos across visual quality, text alignment, and preservation of unedited details, using 10 raters and 29 videos with 127 edit prompts.
- 10 human raters evaluated 29 videos containing 127 edit prompts across animals, people, vehicles, and other objects.The videos came from YouTube-8M and were shown at 350 × 200 resolution.
- The ablation analysis asked raters to select the best edited video among 12 hyperparameter combinations, using α = 0.35, FTsteps = 300, and s = 1 for baseline evaluation.
- The baseline comparison presented the original video alongside an edited video for evaluation.
- Raters scored the edited video's overall visual quality and smoothness.
- Raters judged how well each edited video matched its textual edit description.
- Raters assessed how well edited videos preserved unedited details from the original videos.