Source-linked AI summary
Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, Luc Van Gool
TL;DR
High-resolution video generation is difficult because temporal dynamics, memory demands, and unstable training compound the challenges of image generation. The paper applies progressive growing with spatiotemporal convolutions and a Sliced Wasserstein GAN loss, reaching 256x256x32 videos and a reported UCF-101 inception score of 14.56.
Problem
Unsupervised video GANs were limited to tiny 64 × 64 clips because video generation faces computational, memory, and network-instability challenges.
Method
The paper progressively grows video GANs from low-resolution, short clips using 3-D convolutions and applies a Sliced Wasserstein GAN loss.
Results
The model generates videos up to 256x256 resolution and 32 frames and reports a state-of-the-art UCF101 inception score of 14.56.
Takeaways & Limitations
Progressively increasing video-generation complexity can be effective for unsupervised video generative models.
Takeaways & Limitations
Evaluation relies on a C3D model with approximately 75% clip-level accuracy, while using only the central 32 UCF101 frames for training.
Abstract
from arXiv · showhide
The extension of image generation to video generation turns out to be a very difficult task, since the temporal dimension of videos introduces an extra challenge during the generation process. Besides, due to the limitation of memory and training stability, the generation becomes increasingly challenging with the increase of the resolution/duration of videos. In this work, we exploit the idea of progressive growing of Generative Adversarial Networks (GANs) for higher resolution video generation. In particular, we begin to produce video samples of low-resolution and short-duration, and then progressively increase both resolution and duration alone (or jointly) by adding new spatiotemporal convolutional layers to the current networks. Starting from the learning on a very raw-level spatial appearance and temporal movement of the video distribution, the proposed progressive method learns spatiotemporal information incrementally to generate higher resolution videos. Furthermore, we introduce a sliced version of Wasserstein GAN (SWGAN) loss to improve the distribution learning on the video data of high-dimension and mixed-spatiotemporal distribution. SWGAN loss replaces the distance between joint distributions by that of one-dimensional marginal distributions, making the loss easier to compute. We evaluate the proposed model on our collected face video dataset of 10,900 videos to generate photorealistic face videos of 256x256x32 resolution. In addition, our model also reaches a record inception score of 14.57 in unsupervised action recognition dataset UCF-101.
Introduction
Video generation is harder than image generation because temporal dynamics add modeling complexity, while high resolution, memory demands, and unstable training limit existing unsupervised methods. This work applies progressive growing to video GANs, adds a stability-oriented loss, and introduces a facial-dynamics dataset.
- Video contains temporal information unavailable in images, but its computational and modeling complexity has kept computer-vision research focused largely on static images.
- Existing unsupervised video GANs generally generate only 64 × 64 clips because of computational limitations and network instability.
- The work generalizes Progressive Growing of GANs to video generation using simple 3-D convolutions for spatiotemporal modeling.The authors examine whether robust image-GAN training measures generalize to video frameworks.
- Higher-resolution video generation is constrained by insufficient datasets, severe memory and computation demands, and unstable training or convergence.
- 1.1 Focus of this Work: The stated contributions are progressive video GAN training, an improved loss function for stability, and a 300 × 300 facial-dynamics dataset containing 10,910 clips.
2.1 GAN
GANs learn a data distribution through an adversarial generator–discriminator game, but conventional objectives can produce instability, mode collapse, and convergence problems. Wasserstein-based objectives and gradient penalties are presented as approaches for smoother optimization and Lipschitz enforcement.
- GAN training uses a generator to produce samples and a discriminator to distinguish generated from real data in a non-cooperative game.The generator maps latent noise to samples, while the discriminator acts as a classifier or critic.
- GANs can suffer from vanishing gradients, instability, mode collapse, and lack of convergence despite improvements such as least-squares loss, feature matching, and minibatch discrimination.
- 2.1 GAN: Wasserstein GAN replaces the JS-divergence objective with Wasserstein distance, which behaves smoothly when real and generated distributions have non-overlapping support.
- 2.1 GAN: The Wasserstein formulation uses a dual optimization problem because directly exploring all joint couplings is infeasible, with a K-Lipschitz constraint on the critic function.
- 2.1 GAN: Gradient penalties enforce the Lipschitz constraint by penalizing gradient norms, using samples drawn uniformly along lines between real and generated samples.
2.3 Conditional GANs
The supplied passages position conditional GANs and related video-generation methods as background for the paper’s focus on progressive growing and Sliced Wasserstein GANs. They also motivate these directions through the difficulty of stable, high-dimensional video modeling.
- 2.3 Conditional GANs: Conditional GANs impose conditions such as class labels or parts of original data so generators produce samples with desired properties.
- 2.3 Conditional GANs: Applications include generating specified MNIST digits, faces with particular attributes, and future video frames.
- The paper focuses on higher-resolution video generation with progressive growing of Sliced Wasserstein GANs.
- Prior work explored curriculum learning, multiple generators or discriminators, and multi-stage GANs to manage complexity, mode collapse, stability, or sequential refinement.
- Sliced Wasserstein GANs use Sliced Wasserstein Distance, motivated by Wasserstein distance’s suitability for non-overlapping supports and the difficulty of enforcing Lipschitz constraints.
3.1 Progressive Growing of GAN
Progressive growing trains GANs from lower-resolution samples and incrementally adds layers to learn higher-resolution video generation. The approach extends progressive growing from images to videos while contrasting prior architectures that separate temporal, spatial, foreground, or background components.
- Progressive Growing of GAN: Progressive growing first trains generators and discriminators on lower-resolution samples, then introduces layers to increase resolution and complexity.The transition is smoothed by combining higher-resolution and downscaled inputs, with the higher-resolution contribution increasing from 0 to 1.
- Progressive Growing of GAN: The progressive-growing scheme had not previously been applied to video generation, motivating its exploration for progressively growing video generative networks.
- Video GAN: Video GAN uses parallel foreground and background streams, but assumes a stable background across video frames.
- Temporal GAN: Temporal GAN uses a cascade in which a 1-D temporal generator creates frame-specific latent codes for a 2-D image generator, without requiring background stabilization.
3.4 MoCoGAN
MoCoGAN and related video-generation methods separate temporal and spatial or motion and content representations, while the proposed progressive approach addresses instability and complexity in higher-resolution video generation.
- MoCoGAN: MoCoGAN uses a cascade of recurrent temporal and image generators, concatenating motion codes with newly sampled content codes and adding an image discriminator.
- Other Related Works: Video Pixel Networks model raw-pixel distributions for future prediction, whereas several other works address future prediction rather than unsupervised video generation.
- Motivation: Existing video GANs face greater instability at higher resolutions because video generators and discriminators contain many parameters, making improved losses alone potentially insufficient.
- Progressive Video Generation: Progressive video generation begins with low-resolution, short clips and gradually adds layers to learn finer spatial and temporal details.The implementation downsamples 32 × 256 × 256 videos to 4 × 4 × 4 and initially generates at that scale.
- Transition Phase: During resolution transitions, upscaled low-resolution outputs are linearly blended with higher-resolution outputs while their weights change smoothly from 1 to 0 and 0 to 1.
4.2 Minibatch Standard Deviation
The method uses minibatch feature statistics to discourage mode collapse and pixel-vector normalization to control feature magnitudes, alongside Sliced Wasserstein objectives that reduce distribution-distance estimation to one-dimensional projections.
- Minibatch Standard Deviation: Minibatch standard deviation computes feature variability at each spatio-temporal location across samples, averages it, and concatenates the resulting statistic to all locations and features.Because it has no additional learnable parameters, the approach is computationally cheaper while remaining efficient as argued in [19].
- Feature Normalization: Pixel feature-vector normalization extends local response normalization to spatiotemporal features and avoids parameter explosion in the generator and discriminator.The normalization is not guaranteed to improve performance, but it prevents feature magnitudes from causing parameter-related instability.
- Sliced Wasserstein GAN Loss: Sliced Wasserstein Distance replaces a high-dimensional distribution comparison with Wasserstein distances between one-dimensional projected marginals, which are easier to approximate.The one-dimensional Wasserstein distance has a closed-form solution, and SWD was proven equivalent to WD in.
- Sliced Wasserstein GAN Loss: The SWGAN discriminator encodes samples into a latent space, projects the encodings with orthogonal transform matrices into one-dimensional marginals, and approximates the resulting integral with a mapping function.The discriminator is defined as D = f ◦ E, with an additional gradient regularizer to avoid gradient explosion and vanishing for E.
Evaluation Metrices
The section reviews Inception Score as a quantitative GAN metric based on class-prediction distributions, emphasizing visual quality, class diversity, and its video-generation extension.
- Evaluation Metrices: GAN evaluation is non-trivial, motivating quantitative metrics such as Inception Score and FID alongside earlier human-survey approaches.
- Inception Score: Inception Score is motivated as a metric that correlates with the visual quality of generated samples.
- Inception Score: Inception Score uses the KL divergence between p(y|x) and the marginal class distribution p(y) to evaluate generated samples.The marginal distribution is approximated from N generated samples.
- Inception Score: The metric favors samples that are confidently classifiable while maintaining a uniform distribution across classes, encouraging realism and discouraging mode collapse.
- Video Evaluation: Video Inception Score generalizes the metric using a C3D model trained on Sports-1M and fine-tuned on UCF101.Its computation requires a model trained for the relevant class labels.
6.2 Fr´echet Inception Distance
The section presents Fréchet Inception Distance as a feature-statistics comparison between real and generated samples and describes its video adaptation using C3D features.
- Fréchet Inception Distance: FID compares the means and covariances of CNN features extracted from real and generated samples using Fréchet distance.
- Fréchet Inception Distance: FID is reported to correlate with visual quality and can evaluate generative models across datasets because it uses lower-level features.
- Video FID: For video evaluation, C3D features from an action-recognition model can be used, with the first fully connected layer preferred over the very high-dimensional final pooling output.
- Evaluation Setup: The experiments evaluate TrailerFaces qualitatively and use FID for the Golf and Aeroplane datasets.
7.1 Trailer Face Dataset
The paper introduces TrailerFaces to address the scarcity of large, high-resolution facial-dynamics video datasets, alongside existing action and object-video benchmarks.
- Dataset Examples: Figures show example frames from TrailerFaces, UCF101, Golf, and Aeroplane datasets, while Figure 7.1 depicts the dataset-construction pipeline.
- Motivation: TrailerFaces addresses the lack of publicly available high-resolution datasets containing facial dynamics.Existing facial datasets and common video datasets provide limited resolution or coverage for this setting.
- Dataset Description: The dataset is compared with existing facial-dynamics datasets in Table 7.1 and organized by clip frame counts in Table 7.2.
- Collection: Approximately 6000 Hollywood movie trailers were downloaded to capture emotionally charged and demographically diverse facial clips.
- Collection: Consecutive frames with successful face detection were retained, and SIFT matching removed clips containing different identities across frames.
- Comparison Datasets: Evaluation also uses UCF101, Golf, and Aeroplane datasets, with UCF101 providing labeled action videos and Golf and Aeroplane providing 128x128 clips.
Experiments and Results
Progressive training improves generated-video detail and avoids traditional instability issues, while SWGAN achieves the strongest reported UCF101 Inception Score; object appearance remains a major limitation.
- 8.1 Qualitative Results: Progressive training produces increasingly detailed structures over time on TrailerFaces, as illustrated by Figure 8.1.
- 8.1 Qualitative Results: The generated TrailerFaces videos look reasonable, but their quality remains below image-generation quality.
- 8.1 Qualitative Results: Moving aeroplanes, humans, and animals appear as indistinct blobs, although their temporal dynamics appear more reasonable in qualitative comparisons.
- 8.1 Qualitative Results: Latent-space interpolation across datasets indicates good generalization and no observed overfitting.
- 8.1 Qualitative Results: Progressive growing avoids mode collapse and instability during higher-resolution video generation, but does not fully solve blob-like moving objects.
- 8.2 Inception score on UCF101: Inception Score decreased with further training, possibly because of smaller minibatches at higher resolution, but the cause remained inconclusive.
8.3 FID
The Progressive Video GAN outperforms TGAN and VideoGAN on FID comparisons across Golf and Aeroplane datasets, while achieving strong UCF101 results despite a comparatively weaker discriminator.
- FID comparisons used 10,000 generated samples per model and C3D fc-6 features; 64x64 outputs from TGAN and VideoGAN were upscaled to 128 × 128.
- Progressive Video GAN performs significantly better than TGAN and VideoGAN on both Golf and Aeroplane datasets by FID score.The advantage is especially prominent on Aeroplane, where TGAN and Progressive Video GAN outperform VideoGAN.
- VideoGAN’s weaker Aeroplane performance is consistent with its static-background assumption, unlike TGAN and Progressive Video GAN.
- 14.56 Inception Score on UCF101 is reported as state of the art, alongside better FID and Inception Score performance than existing approaches.The authors describe this as an improvement over the state-of-the-art score reported in [40].
Appendix A
Appendix A documents the generator and discriminator architectures for 256x256x32 video generation, while supplementary figures visualize latent-space interpolation across three datasets.
- Appendix A: Table A.1 specifies the generator architecture used to produce 256x256x32 videos.
- Appendix A: Table A.2 specifies the corresponding discriminator architecture for 256x256x32 video generation.
- Appendix A: Latent-space interpolation samples are shown for Golf, Aeroplane, and TrailerFaces datasets across Figures B.1–B.6.