Source-linked AI summary
Phenaki: Variable Length Video Generation From Open Domain Textual Description
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, Dumitru Erhan
TL;DR
Text-to-video generation is constrained by the computational cost of video and the limited quantity of high-quality text-video data, while longer videos require prompts that can vary over time. Phenaki addresses these challenges with a causal variable-length video tokenizer, a bidirectional masked transformer, and joint image-video training. It generates open-domain, temporally coherent videos, including long videos and videos conditioned on sequences of prompts.
Problem
High-quality text-to-video generation is challenging because video requires severe computation and substantially less high-quality text-video data, while longer videos need prompts that vary over time.
Method
Phenaki combines C-ViViT, a causal temporal-spatial video tokenizer, with a bidirectional masked transformer and joint training on text-image and text-video data.
Results
Phenaki generates temporally coherent and diverse open-domain videos, including videos lasting minutes and videos conditioned on a story or sequence of prompts.
Takeaways & Limitations
Phenaki demonstrates open-domain video generation with arbitrary-length outputs and time-variable text conditioning.
Takeaways & Limitations
Phenaki may enable maliciously fake content and non-consensual videos, and its training data includes undesirable biases from LAION-400M.
Abstract
from arXiv · showhide
We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.
1 INTRODUCTION
Phenaki addresses the difficulty of open-domain text-to-video generation by combining variable-length video modeling with sequence-of-prompt conditioning. It introduces C-ViViT and demonstrates temporally coherent, diverse videos, including long videos and story-conditioned generation.
- Motivation: High-quality text-to-video generation remains challenging because video requires more computation and substantially less high-quality text-video data than image generation.The paper contrasts billions of image-text pairs with substantially smaller text-video datasets such as WebVid’s approximately 10M videos.
- Motivation: A single short prompt cannot fully describe a longer video, motivating arbitrary-length generation conditioned on prompts that vary over time.The paper frames a sequence of prompts as a story narrating what happens across scenes.
- Capabilities: Phenaki is trained on both text-video and text-image data to generate temporally coherent, diverse videos from open-domain prompts, including new compositions of concepts.The model can generate videos lasting minutes despite training on 1.4-second videos at 8 fps.
- Capabilities: Phenaki generates videos conditioned on a story, meaning a sequence of prompts that changes over time.The paper identifies this as story-based conditional video generation and illustrates it in Figures 1 and 5.
- Architecture: C-ViViT provides temporal-spatial compression while remaining autoregressive in time, enabling variable-length video encoding and decoding.Its causal structure supports the variable-length capability required by Phenaki.
- Architecture: C-ViViT exploits temporal redundancy to improve reconstruction quality over a per-frame model while compressing the number of video tokens by 40% or more.The contribution combines improved reconstruction with lower token counts than per-frame encoding.
2 THE PHENAKI MODEL
Phenaki combines a variable-length video tokenizer with masked bidirectional token generation. Its C-ViViT encoder-decoder compresses videos into discrete tokens, while text-conditioned generation and autoregressive extension support long, story-based videos.
- Model overview: Phenaki uses an encoder-decoder to compress videos into discrete tokens and a transformer to translate text embeddings into video tokens.Text embeddings are produced with the pre-trained T5X language model, and generated tokens are decoded into video.
- Video tokenizer: The encoder separates the first frame from later spatio-temporal tokens, which depend autoregressively on previous frames.Input videos are compressed from tx + 1 frames into a representation of (tz + 1) × wz × hz tokens.
- Video tokenizer: The decoder reverses the temporal and spatial transformations, then linearly projects token representations back into pixel space.Tokens are first transformed into embeddings, followed by temporal and spatial transformers.
- Video tokenizer: Video tokens are quantized through a learned codebook, and training combines vector-quantization, adversarial, image-perceptual, video-perceptual, and L2 losses.The stated objective is L = LVQ + 0.1 × LAdv + 0.1 × LIP + 1.0 × LVP + 1.0 × L2.
- Video tokenizer: C-ViViT uses causal temporal attention, allowing variable numbers of input frames and autoregressive extrapolation into the future.This replaces ViViT's all-to-all temporal attention, which requires fixed-length video inputs.
- Text-to-video generation: A masked bidirectional transformer predicts masked video tokens in parallel, reducing generation to typically 12–48 sampling steps instead of sequential autoregressive sampling.During inference, predicted tokens are retained while others are re-masked and predicted again; additional frames are generated autoregressively from the last K frames.
3 EXPERIMENTS
Phenaki is evaluated across text-conditioned generation, joint image-text training, video reconstruction, image-conditioned prediction, and image animation. The experiments show qualitative control and generalization, competitive prediction performance, improved spatio-temporal reconstruction with fewer tokens, and trade-offs from combining image and video data.
- Text conditional video generation: Phenaki generates videos with control over actors, backgrounds, appearance, and style, including unseen compositions and stylized outputs.The model generalizes from still-image data to video despite the video dataset lacking stylized examples.
- Text conditional video generation: Phenaki achieves comparable generation quality to prior text-to-video methods on Kinetics-400 in a zero-shot setting.The comparison uses a same-size 0.9B-parameter model, while competing models were trained or fine-tuned on Kinetics-400.
- Joint training: Joint image-text and video-text training trades video dynamics against alignment and image quality as the image-data ratio increases.Video-only training yields significantly better FVD, while more image data improves text-video and text-image alignment and significantly improves image FID.
- Image and story-conditioned generation: Phenaki generates coherent videos from unseen images while following text prompts and can extend videos using dynamically changing prompts.The model is evaluated by animating phone-captured images and by generating long videos in story mode from sequences of prompts.
- Video encoding: C-ViViT achieves better spatio-temporal reconstruction dynamics than per-frame image methods while compressing videos into fewer tokens.Per-frame methods slightly outperform C-ViViT on FID, but have significantly lower FVD; fewer tokens reduce downstream transformer computation.
- Image conditional video generation: Phenaki is competitive with state-of-the-art video prediction methods on Kinetics-600 despite not being specifically designed for video prediction.The evaluation tests prediction of 11 frames from 5 frames on Kinetics-600 and 15 frames from one frame on BAIR Robot Pushing.
4 RELATED WORKS
Phenaki builds on auto-regressive and diffusion approaches for text-conditioned image and video generation, while addressing temporal dynamics and variable video length through a temporal video representation. Prior work includes fixed-size, frame-independent, and pixel-space video generation methods.
- Auto-regressive methods: Auto-regressive methods map text tokens to discrete image or video tokens, but sequential sampling becomes impractical for long video sequences.GODIVA uses local attention for many frame tokens; NUWA generates fixed-size outputs, while NUWA-Infinity adds autoregressive computation for variable sizes.
- Temporal modeling: NUWA, NUWA-Infinity, and CogVideo treat videos as sequences of independent images, which can produce poor dynamics modeling and motion artifacts.NUWA-Infinity uses the previous frame during decoding, whereas Phenaki treats videos as temporal image sequences.
- Diffusion models: Diffusion video models apply a 3D space-time diffusion process directly to pixels, but conventional formulations are limited to fixed-size videos.VDM introduces an autoregressive extension to address this issue.
- Video-generation approaches: Video-generation research also includes deterministic recurrent-convolutional models, stochastic variational methods, discrete representations, and autoregressive models.These approaches are discussed in the broader context of image-conditioned and unconditional video generation.
- Evaluation settings: Earlier methods commonly focus on robotic-video prediction or short fixed-size clips, motivating comparisons with models designed for broader video generation.The paper positions its evaluation against these established settings.
5 CONCLUSION
Phenaki generates variable-length videos from sequences of open-domain text prompts using the temporally autoregressive C-ViViT encoder. Joint image-video training supports generation quality and diversity, while the model also supports starting-frame conditioning and coherent story-like video generation.
- Core capabilities: Phenaki generates variable-length videos conditioned on sequences of open-domain text prompts.Its C-ViViT video encoder provides temporal-spatial compression while remaining autoregressive in time.
- Architecture: C-ViViT enables variable-length generation by combining temporal-spatial compression with autoregressive processing in time.The conclusion identifies C-ViViT as crucial to Phenaki’s variable-length capability.
- Training: Joint training on images and videos improves generation quality and diversity by leveraging the much larger image-text dataset.The conclusion motivates combining modalities because image-text data contains orders of magnitude more samples.
- Applications: Phenaki supports video prediction, long videos from text, conditioning on both text and a starting frame, and coherent video stories from prompt sequences.The model is presented as a potential creative tool for visualizing more complex narratives.
ETHICS STATEMENT
The ethics statement discusses potential misuse and dataset-related harms associated with Phenaki. These concerns motivate withholding the models, code, data, and demo until filtering, bias measurement, and mitigation improve.
- Creative use: Phenaki may support creative work, but downstream applications were not explored and generated samples may not perfectly match captions or user intent.The authors nevertheless anticipate time savings when at least one generated sample aligns with the user’s intent.
- Potential misuse: Phenaki’s ease of use could facilitate maliciously fake videos, including videos generated of people without their consent or knowledge.The authors note that generated videos are not yet indistinguishable from real videos, but that threshold may be possible for some samples.
- Dataset risks: Phenaki incorporates LAION-400M despite known issues involving violence, pornography, and gore, although the authors observed better results with its inclusion.A current training version uses datasets intended to minimize these problems.
- Release decision: The authors withheld the underlying models, code, data, and interactive demo while prioritizing data, prompt, and output filtering and explicit bias measurement.They plan to mitigate identified issues through data, model, and pre/post-processing changes.
B.1.1 NETWORK ARCHITECTURE
The C-ViViT network uses spatio-temporal patch embeddings and separate spatial and temporal transformer processing. Its approximately 50M-parameter encoder-decoder uses a shared reverse architecture for decoding latent embeddings back to image space.
- Baselines: All encoder-decoder baselines use approximately 50M parameters, enabling architectural comparisons at similar model scale.The convolutional and ViT baseline configurations are described alongside C-ViViT.
- Transformer encoder: The C-ViViT encoder contains four spatial and four temporal transformer layers with 512 hidden units, eight attention heads, and 2048 MLP hidden units.Its embedding dimension is 32.
- Decoder: The decoder reverses the encoder architecture to map latent embeddings back into image space.The same decoder design is used for all compared models.
B.1.2 TRAINING
The training setup uses Adam with scheduled learning-rate warmup and decay, while a patch-based tokenization scheme captures spatial and temporal structure.
- Adam training uses β1 = 0.9, β2 = 0.99, a 1×10−4 peak learning rate, and decoupled weight decay of 1 × 10−4.The learning rate warms up over 100,000 steps and then follows cosine decay for the remaining 900,000 steps.
- Video tokenization applies 4 × 4 spatial patches to the first image and 2 × 4 × 4 spatio-temporal patches thereafter.The spatial encoder has 8 layers, while the temporal encoder has 6 layers.
B.2.2 KINETICS-600 C-VIVIT ARCHITECTURE
The Kinetics-600 C-ViViT setup retains the surrounding training configuration while using 8-layer spatial and temporal encoders.
- Both the spatial encoder and temporal encoder consist of 8 layers in the Kinetics-600 C-ViViT architecture.
B.2.3 MASKGIT ARCHITECTURE
The MaskGIT configurations vary by experiment, using unconditional latent-space prediction for video baselines and text conditioning through cross attention for text-to-video generation.
- The latent-space video-prediction transformer has 24 layers, 768 hidden units, 16 attention heads, and 3072 MLP hidden units.Dropout and attention dropout rates are both 0.1.
- C-ViViT is trained without FPS downsampling, and MaskGIT models video tokens unconditionally without frame or text inputs.
- Text-conditional MaskGIT receives T5X embeddings through cross attention with video tokens.The base model is reduced for quantitative comparison against NUWA, while the main experiments use a larger architecture.
- The main MaskGIT architecture uses 24 transformer layers, 2048 hidden units, 32 attention heads, and 8192 MLP hidden units, totaling 1.8B parameters.
- All text-conditional video-generation experiments use the training parameters specified in Table 7.