Source-linked AI summary
Make Pixels Dance: High-Dynamic Video Generation
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, Hang Li
TL;DR
Existing video-generation methods often produce high-fidelity clips with limited motion, motivating alternatives to text-only conditioning. PixelDance combines text with first- and last-frame image instructions in a diffusion model and reports strong synthesis of complex scenes and intricate motions.
Problem
Existing text- and image-conditioned video generators often produce limited motions, while detailed textual annotations are costly to collect and difficult for models to learn.
Method
PixelDance is a diffusion-based video-generation approach conditioned on text plus first- and optionally last-frame image instructions.
Results
PixelDance trained mainly on public WebVid-10M data achieves FVD scores of 381 on MSR-VTT and 242.8 on UCF-101 with text-derived first-frame instructions.
Takeaways & Limitations
PixelDance demonstrates strong synthesis of complex scenes, actions, and motion-rich videos, with last-frame instructions especially important for intricate out-of-domain generation.
Takeaways & Limitations
The authors identify open-domain data quality, domain adaptation, annotated motion text, and model scaling as opportunities for further improvement.
Abstract
from arXiv · showhide
Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.
1. Introduction
PixelDance addresses limited motion in text- and image-conditioned video generation by combining text with first- and optionally last-frame image instructions. The approach uses tailored training and inference techniques and reports strong performance on complex, motion-rich videos.
- Existing video-generation methods often produce simple scenes with limited motion despite improved visual fidelity from image inputs.
- Image instructions are easier to obtain than highly descriptive motion annotations, allowing training to focus on video dynamics and improving generalization to out-of-domain instructions.
- PixelDance conditions latent diffusion on text, first-frame, and optionally last-frame instructions to guide video content and ending status.The first frame enforces adherence to the starting scene, while the last frame provides additional control.
- PixelDance uses noisy, randomly selected last-frame instructions and sometimes drops them during training; inference applies them only during the first τ denoising steps.This design supports coarse user guidance while promoting temporal coherence.
- With 1.5B parameters and mainly WebVid-10M training, PixelDance reports FVD scores of 381 on MSR-VTT and 242.8 on UCF-101 using text-derived first-frame instructions.
- The model generates motion-rich videos, continuous clips, and complex out-of-domain scenes, with last-frame instructions identified as critical for intricate actions and transitions.
2. Related Work
Related work spans generative video models, video editing, and long-video generation. PixelDance is positioned as addressing limited motion and reference-video constraints while supporting continuous, consistent clips.
- Prior video-generation research uses GANs, VQ-VAE-based transformers, and diffusion models, including latent diffusion to reduce computational burden.
- Video Generation: Video editing modifies content while preserving structure and motion, but finding a suitable reference video is time-consuming and restricts entirely novel synthesis.
- Long Video Generation: Long-video generation requires seamless transitions and long-term consistency, while autoregressive methods can accumulate errors over time and hierarchical methods require difficult-to-obtain long training videos.
- Long Video Generation: PixelDance generates continuous clips autoregressively and reports superior long-term frame consistency compared with existing models.
3. Method
PixelDance injects first- and last-frame image conditions into a latent diffusion model alongside text. Its training and inference procedures deliberately balance endpoint control with temporal coherence.
- PixelDance integrates image instructions for both endpoint frames with text instruction to generate videos with richer dynamics.
- A latent diffusion model denoises perturbed video representations, using a 2D UNet architecture with spatial downsampling, upsampling, and skip connections.
- Image Instruction Injection: VAE-encoded first- and last-frame instructions are arranged with padding and concatenated with noised latent z along the channel dimension.
- Training uses the ground-truth first frame for strict adherence but avoids exact replication of the last frame so inference can accommodate coarse user drafts.
- The last-frame instruction is randomly dropped during training and applied only during the first τ of T denoising steps at inference.
- τ controls dependence on the last-frame instruction, including the option to generate high-dynamic videos without it when τ = 0.
4.1. Implementation Details
PixelDance is trained on video-text and image-text data, including WebVid-10M and LAION-400M, and evaluated through zero-shot comparisons on MSR-VTT and UCF-101. The supplied implementation passages emphasize dataset construction and evaluation setup rather than full numerical tables.
- Data: WebVid-10M provides about 10M short clips averaging 18 seconds, but its coarse captions are weakly correlated with video content and its videos contain watermarks.
- Data: PixelDance supplements video-text training with 500K self-collected watermark-free clips and jointly trains on LAION-400M image-text data.
- Training: Training samples 16 consecutive video frames at 4 fps, freezes the text encoder and VAE, and uses DDPM with T = 1000 training steps.
- Evaluation: Table 1 compares zero-shot text-to-video performance on MSR-VTT at 256×256 spatial resolution, with the best results bolded.
- Evaluation: Table 2 compares zero-shot text-to-video performance on UCF-101 at 256×256 spatial resolution, with the best results bolded.
4.2. Video Generation
PixelDance is evaluated for zero-shot video generation using text and image instructions, with qualitative analyses examining how each instruction contributes to motion, consistency, and control.
- Quantitative Evaluation: PixelDance achieves state-of-the-art MSR-VTT results in FVD and CLIPSIM, including an FVD of 381 versus ModelScope’s 550.On UCF-101, it also outperforms other models across IS, FID, and FVD.
- Qualitative Analysis: Text instructions specify body, facial, object, camera, and visual-effect motions while helping preserve cross-frame consistency of described elements.The polar bear’s black hat and red bow tie are cited as examples of consistent key elements.
- Qualitative Analysis: First-frame instructions provide finer visual details, support consecutive clips, and enable more motion-rich videos when combined with text.The first frame supplies scene content that is difficult to describe succinctly and precisely in text alone.
- Qualitative Analysis: Last-frame instructions add control over a clip’s concluding state, supporting intricate motions, out-of-domain generation, and natural shot transitions.They are particularly important for complex scenes and actions outside the training distribution.
- Last Frame Guidance: Training and inference techniques prevent videos from ending by exactly replicating an imprecise last-frame draft, producing more fluent and temporally consistent results.Without the techniques, the generated video abruptly ends at the supplied last frame.
- Qualitative Analysis: PixelDance generates high-quality videos in non-realistic styles despite limited training videos in categories such as science fiction, comics, and cartoons.The paper attributes this generalization to learning motion dynamics and temporal consistency from image instructions.
4.3. Ablation Study
The ablation study examines PixelDance’s instruction components and reports that omitting either text or last-frame guidance substantially degrades video quality.
- Experimental Setup: The study compares a same-data text-to-video baseline with variants that analyze the text and last-frame instructions.The first-frame instruction is treated as indispensable for generating continuous video clips.
- Instruction Ablation: Omitting either the text or last-frame instruction causes significant deterioration in video quality.The ablation is conducted on UCF-101 under the zero-shot evaluation setting.
- Instruction Ablation: A model trained with last-frame instruction outperforms one trained without it even when the evaluation omits that instruction.This result indicates that last-frame training contributes to modeling motion dynamics and temporal consistency beyond direct inference conditioning.
4.4. Long Video Generation
PixelDance extends generation across successive clips by conditioning each new clip on the preceding clip’s last frame, targeting temporal consistency in long and composite videos.
- Quantitative Evaluation: PixelDance generates 1024-frame UCF-101 videos with lower FVD scores and smoother temporal variations than autoregressive and hierarchical baselines.The comparison includes TATS-AR, LVDM-AR, and LVDM-Hi.
- Quantitative Evaluation: The long-video construction uses the last frame of a preceding clip as the first-frame instruction for the subsequent clip.This conditioning enforces adherence to the connecting frame between clips.
- Qualitative Analysis: Composite shots string together multiple continuous clips to represent realistic long videos composed of numerous shots.The qualitative examples include complex camera movements, animation effects, and controlled rocket trajectories.
- Qualitative Analysis: Figure 9 illustrates continuous-clip compositions involving Arctic camera movements, a polar bear in a hot-air balloon, and a rocket trajectory.These examples emphasize user-directed construction of desired video sequences.
- Qualitative Analysis: PixelDance synthesizes a three-minute video with a coherent story and consistent portrayal of its main character.The result is presented as an example of composing longer, temporally consistent sequences.
4.5. More Applications
PixelDance’s image-instruction framework extends beyond ordinary frames to sketches and enables zero-shot video editing through edits to a video’s boundary frames.
- Sketch Instruction: Fine-tuning PixelDance with an image sketch as the last-frame instruction allows a simple sketch to guide video generation.The sketch is demonstrated as an alternative image-instruction type.
- Zero-shot Video Editing: PixelDance performs zero-shot video editing by transforming the task into image editing and modifying the provided video’s first and last frames.The generated video is reported to remain temporally consistent and aligned with user expectations.
5. Conclusion
PixelDance is a diffusion-based video generation approach that uses first- and last-frame image instructions with text to synthesize complex scenes and actions. The authors identify open-domain data, domain adaptation, annotated motion descriptions, and larger models as directions for further advancement.
- PixelDance combines first- and last-frame image instructions with text instructions in a diffusion-based video generation approach.
- The approach demonstrates proficiency in synthesizing videos with complex scenes and actions, setting a new standard in video generation.
- Further progress may come from higher-quality open-domain data, domain-specific fine-tuning, annotated motion descriptions, and scaling beyond 1.5B parameters.