Source-linked AI summary
Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, Jun Zhu
TL;DR
Text-to-video models had been limited to short durations, motivating Vidu’s use of a U-ViT-based diffusion architecture for scalable long-sequence modeling. Vidu generates 1080p videos up to 16 seconds with strong coherence and dynamism, reports performance comparable to Sora to some extent, and shows promising controllable-generation results.
Problem
Earlier video-generation models primarily focused on a single limited duration, such as 4 seconds.
Method
Vidu uses a video autoencoder and U-ViT noise-prediction network, with transformer processing for variable-length video sequences and captioning support for training and inference.
Results
Vidu generates 1080p videos of all lengths up to 16 seconds and exhibits strong coherence, dynamism, realistic and imaginative generation, and promising controllable-generation results.
Takeaways & Limitations
Vidu demonstrates a high-definition text-to-video system spanning variable durations, coherent dynamic content, photography techniques, and several controllable-generation settings.
Takeaways & Limitations
Vidu still has occasional detail flaws and sometimes produces interactions between subjects that deviate from physical laws.
Abstract
from arXiv · showhide
We introduce Vidu, a high-performance text-to-video generator that is capable of producing 1080p videos up to 16 seconds in a single generation. Vidu is a diffusion model with U-ViT as its backbone, which unlocks the scalability and the capability for handling long videos. Vidu exhibits strong coherence and dynamism, and is capable of generating both realistic and imaginative videos, as well as understanding some professional photography techniques, on par with Sora -- the most powerful reported text-to-video generator. Finally, we perform initial experiments on other controllable video generation, including canny-to-video generation, video prediction and subject-driven generation, which demonstrate promising results.
1 Introduction
Vidu is a text-to-video diffusion model using U-ViT to overcome limited video durations and generate 1080p videos up to 16 seconds. It shows strong coherence, dynamism, realism, imaginative generation, photography techniques, and promising controllable-generation results, with performance reported as comparable to Sora to some extent.
- 1080p videos up to 16 seconds can be generated in a single generation, including single-frame images as videos.
- Vidu exhibits strong coherence and dynamism while generating both realistic and imaginative videos.
- Vidu shows preliminary understanding of transitions, camera movements, lighting effects, and emotional portrayal.
- Initial canny-to-video, video prediction, and subject-driven-generation experiments demonstrate promising results.
- Vidu’s generation performance is reported as comparable to Sora to some extent.
2 Text-to-Video Generation
Vidu uses a video autoencoder and U-ViT noise-prediction network to model compressed videos as token sequences. Transformer sequence processing supports videos with variable durations, while captioning and re-captioning improve text-video supervision and inference inputs.
- Vidu reduces video spatial and temporal dimensions with an autoencoder before applying U-ViT to predict noise in compressed representations.
- U-ViT splits compressed videos into 3D patches, treats time, text, and noisy patches as tokens, and uses long skip connections.
- Transformer processing of variable-length sequences enables Vidu to handle videos with variable durations.
- A dynamic-information-focused video captioner automatically annotates training videos, while re-captioning reformulates user inputs for inference.
2.1 Generating Videos of Different Lengths
Vidu generates 1080p videos at all lengths up to 16 seconds, including single-frame images, and demonstrates strong 3D consistency during camera rotation.
- 1080p videos of all lengths up to 16 seconds can be generated, including images represented as single-frame videos.
- During camera rotation, generated objects maintain 3D consistency through different-angle projections and natural occlusion.
2.3 Generating Cuts
Vidu can generate videos with cuts that switch camera angles and present different perspectives of the same scene while maintaining subject consistency.
- Vidu generates cuts that switch camera angles to show different perspectives of the same scene while maintaining subject consistency.
2.4 Generating Transitions
Vidu can generate transitions within a single video, connecting two different scenes in an engaging manner. It also produces camera movements such as zoom, pan, and dolly.
- Vidu generates transitions within a single video generation.These transitions can connect two different scenes in an engaging manner.
- Vidu generates camera movements including zoom, pan, and dolly.
2.6 Lighting Effects
Vidu generates lighting effects that enhance atmosphere and can evoke mystery and tranquility. It also depicts emotions including happiness, loneliness, embarrassment, and joy.
- Vidu generates lighting effects that enhance the overall atmosphere.The generated videos can evoke atmospheres of mystery and tranquility.
- Vidu can express emotions such as happiness, loneliness, embarrassment, and joy.
2.8 Imaginative Ability
Vidu can generate imaginative scenes that do not exist in the real world, in addition to depicting real-world scenes.
- Vidu generates scenes that do not exist in the real world.This demonstrates imaginative ability alongside its generation of real-world scenes.
2.9 Comparison with Sora
Because Sora is not publicly accessible, the authors compare Vidu with it using Sora-released example prompts. The resulting figures indicate that Vidu's generation performance is comparable to Sora to some extent.
- The comparison uses Sora's released example prompts because Sora is not publicly accessible.
- Vidu's generation performance is comparable to Sora to some extent.Figures 10 and 11 illustrate the comparison using vintage televisions and a vintage SUV scene.
3 Other Controllable Video Generation
Vidu supports several controllable video-generation settings beyond text-to-video, including control-guided generation, video prediction, and subject-driven generation. Initial experiments show promising results, including faithful subject recreation from image-only finetuning.
- Initial experiments at 512 resolution on canny-to-video generation, video prediction, and subject-driven generation demonstrate promising results.
- 3.1 Canny-to-Video Generation: Vidu adds control using techniques similar to ControlNet for canny-to-video generation.
- 3.2 Video Prediction: Given an input image or several input frames, Vidu generates subsequent video frames for video prediction.
- 3.3 Subject-Driven Generation: Vidu performs subject-driven video generation by finetuning solely on images, without requiring videos.
- 3.3 Subject-Driven Generation: Using DreamBooth, Vidu designates a learned subject as <V> and faithfully recreates it in generated videos.
4 Conclusion
Vidu is a high-definition text-to-video generator with strong duration, coherence, and dynamism capabilities, while occasional detail flaws and physically implausible subject interactions remain. The authors suggest further scaling as a way to address these issues.
- Vidu demonstrates strong abilities in video duration, coherence, and dynamism, on par with Sora.
- Occasional detail flaws and interactions between subjects that deviate from physical laws remain limitations.
- The authors believe further scaling can effectively address these issues.