Source-linked AI summary
Video Generation with Predictive Latents
Yian Zhao, Feng Wang, Qiushan Guo, Chang Liu, Xiangyang Ji, Jian Zhang, Jie Chen
TL;DR
Existing video VAEs reconstruct effectively, but reconstruction optimization does not necessarily produce more diffusible generative latents. PV-VAE jointly trains reconstruction and prediction by withholding future frames, achieving faster convergence and improved generation while strengthening temporal and motion representations.
Problem
Existing video VAEs achieve strong reconstruction quality, but improving reconstruction does not necessarily improve generation, leaving video-latent diffusability unresolved.
Method
PV-VAE randomly discards future frames, encodes partial past observations, and trains the decoder to reconstruct the complete video through predictive reconstruction.
Results
52% faster convergence and 34.42 FVD improvement over Wan2.2 VAE were reported on UCF101, alongside consistent gains in downstream video understanding.
Takeaways & Limitations
PV-VAE produces a more temporally structured, motion-aware, and generation-ready video latent space.
Takeaways & Limitations
PV-VAE has a reconstruction gap on dense text, likely because the current data distribution contains few text-heavy samples.
Abstract
from arXiv · showhide
Video Variational Autoencoder (VAE) enables latent video generative modeling by mapping the visual world into compact spatiotemporal latent spaces, improving training efficiency and stability. While existing video VAEs achieve commendable reconstruction quality, continued optimization of reconstruction does not necessarily translate into improved generative performance. How to enhance the diffusability of video latents remains a critical and unresolved challenge. In this work, inspired by principles of predictive world modeling, we investigate the potential of predictive learning to improve the video generative modeling. To this end, we introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction. Specifically, we randomly discard future frames and encode only partial past observations, while training the decoder to reconstruct the observed frames and predict future ones simultaneously. This design encourages the latent space to encode temporally predictive structures and build a more coherent understanding of video dynamics, thereby improving generation quality. Our model, termed Predictive Video VAE (PV-VAE), achieves superior performance on video generation, with 52% faster convergence and a 34.42 FVD improvement over the Wan2.2 VAE on UCF101. Furthermore, comprehensive analyses demonstrate that PV-VAE not only exhibits favorable scalability, with generative performance improving alongside VAE training, but also yields consistent gains in downstream video understanding, underscoring a latent space that effectively captures temporal coherence and motion priors.
1 Introduction
Existing video VAEs reconstruct well, but reconstruction optimization does not necessarily improve generation, leaving latent diffusability unresolved. PV-VAE addresses this gap by combining predictive learning with reconstruction, producing stronger generation and temporally structured latents.
- Video VAEs provide compact spatiotemporal latents that improve generative-model training efficiency and stability.
- Reconstruction quality does not necessarily translate into better generation, making video-latent diffusability an unresolved challenge.
- PV-VAE randomly discards future frames, encodes partial observations, and reconstructs the complete sequence to combine visual reconstruction with temporal prediction.
- 52% faster convergence and 34.42 FVD improvement over Wan2.2 VAE were achieved on UCF101.
- PV-VAE improves downstream optical flow, next-frame prediction, and point tracking, while its latent visualizations reveal motion-aware structures aligned with video dynamics.
- The contributions include a predictive reconstruction objective, improved class-conditional and unconditional generation, and diagnostics linking predictive accuracy with generative quality.
2 Related Work
Prior work studies video VAEs, latent diffusability, and predictive learning separately. PV-VAE positions predictive reconstruction as a way to make video latents autonomously capture structured temporal dynamics.
- Video VAEs map high-dimensional videos into compact latent spaces used by modern latent video generative pipelines.
- Latent diffusability concerns whether a latent space is suitable for diffusion, often improved through structured constraints or semantic priors.
- Video-level latent structuring remains constrained by architectural and computational bottlenecks, with prior approaches using heuristic constraints.
- PV-VAE differs by using predictive reconstruction to encourage autonomous capture of structured temporal dynamics in video latents.
- Predictive learning models future states from existing information and has been applied to sequence, action, trajectory, language, and visual modeling.
3 Approach
PV-VAE reformulates VAE training as partial-to-complete reconstruction: future groups are withheld from the encoder, padded latents are decoded, and the full video is reconstructed. The implementation combines causal spatiotemporal compression, staged training, and multiple reconstruction losses.
- PV-VAE jointly learns predictive and reconstruction objectives to enhance the diffusability of video latent spaces.
- The model represents video pixels and sampled latents with spatial and temporal compression ratios and a latent channel dimension.
- Integrating predictive learning into reconstruction: Training partitions clips into temporal groups, randomly drops future groups, encodes retained observations, and pads the latent sequence before decoding the full video.
- Integrating predictive learning into reconstruction: With dropped frames withheld from the encoder, the model must infer subsequent video evolution from past observations and encode predictive information in the latents.
- Model design: PV-VAE uses 3D causal convolutions with 16× spatial downsampling, 4× temporal downsampling, and 64 latent channels.
- Training and loss functions: Training includes image pretraining, predictive video training, decoder fine-tuning, and MSE, LPIPS, GAN, and KL losses.
4 Experiments
PV-VAE is evaluated across generation, reconstruction, efficiency, latent structure, temporal dynamics, prediction, and ablation settings. The experiments show stronger generation and motion-aware temporal representations while retaining competitive reconstruction quality.
- Generation comparison: PV-VAE achieves the best overall generation performance on UCF101 and RealEstate10K, with improved speed and memory efficiency over comparable models.On UCF101, it improves FVD over Wan2.2 VAE and SSVAE while using a higher latent-channel dimension.
- Reconstruction comparison: PV-VAE delivers comparable reconstruction performance to existing 4×16×16 video VAEs, slightly underperforming Wan2.2 VAE while outperforming SSVAE.It also achieves faster inference and lower memory consumption than Hunyuan-VAE and Wan2.2 VAE at 256 × 256 resolution.
- Qualitative comparison: PV-VAE produces superior visual fidelity, fewer motion artifacts, and enhanced temporal coherence over Wan2.2 VAE under identical generative training settings.Its reconstruction remains competitive, with dense-text reconstruction identified as a remaining limitation associated with scarce text-heavy samples.
- Latent-space analysis: PCA visualizations show PV-VAE latent activations aligned with optical flow, allocating more representational bandwidth to dynamic foregrounds and smoother representations to static areas.The analysis compares principal components of the latent channels with optical flow across non-adjacent video frames.
- Temporal analysis: Predictive accuracy correlates with generative performance, and PV-VAE scales consistently with more training data unlike the pure reconstruction objective.Its latents also exhibit smoother short-term transitions and monotonic long-term trajectories across expanding frame intervals.
- Frame prediction: PV-VAE predicts plausible future frames that align with underlying video dynamics, including relative spatial shifts between subjects and backgrounds.The model reconstructs observed frames while generating frames from the unobserved latter half of each sequence.
- Ablation study: Generative performance consistently improves as the maximum dropping ratio increases, leading to a training setting with r = 100%.The ablation interprets stronger perturbation as stronger predictive regularization that encourages more robust representations.
5 Discussion and Conclusion
The discussion frames video VAE design as a balance between reconstruction fidelity, latent diffusability, and computational efficiency. PV-VAE provides a temporally structured latent space, while Transformer-based variants offer faster inference but retain a generative-performance gap.
- High-dimensional latents can facilitate generation when they are appropriately structured, challenging the traditional reconstruction–generation trade-off.Rich latent information improves reconstruction but can complicate generative modeling; highly compressed latents facilitate generation while sacrificing detail.
- Decoder fine-tuning consistently improves reconstruction while preserving latent diffusability when the encoder remains frozen.The frozen encoder also allows decoder fine-tuning to proceed in parallel with diffusion-backbone training.
- Transformer-based video VAEs address the computational overhead and limited global modeling of 3D causal convolutions, despite a current generative gap.The architecture is explored as a minimalist alternative to predominantly convolutional video VAEs.
- 87% faster inference makes the Transformer variant promising for long video sequences, while its representational flexibility supports future self-supervised objectives.The authors identify optimized architectures and training recipes as future work for unlocking this potential.
- PV-VAE jointly optimizes visual fidelity and temporal dynamics to produce more temporally structured, generation-ready latents and motion-aware representations.The conclusion connects these representations with substantial gains in video generation performance.