PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Video Generation on Complex Datasets

Aidan Clark, Jeff Donahue, Karen Simonyan

arXiv:1907.06571v2cs.CVcs.LGstat.ML

TL;DR

Video generation has lagged behind image generation because complex natural videos impose greater data and computational demands. DVD-GAN addresses this with a scalable dual-discriminator architecture and evaluation on synthesis and prediction tasks. It achieves state-of-the-art results on UCF-101 synthesis and Kinetics-600 prediction while establishing a strong Kinetics-600 synthesis baseline.

  • Problem

    Natural video generation is constrained by increased data complexity and computational requirements, leaving prior work concentrated on simpler datasets or strongly conditioned tasks.

  • Method

    DVD-GAN uses an efficient spatio-temporal decomposition of its discriminator to scale GAN training to complex, high-resolution video.

  • Results

    DVD-GAN achieves state of the art for video synthesis on UCF-101 and prediction on Kinetics-600, while establishing a strong baseline for synthesis on Kinetics-600.

  • Takeaways & Limitations

    DVD-GAN shows that large GANs can model complex natural video with high complexity and diversity, providing reference results for generative modeling on Kinetics-600.

  • Takeaways & Limitations

    GANs can exhibit limited diversity through mode collapse and are difficult to evaluate quantitatively because they lack an explicit likelihood measure.

Abstract

from arXiv · show

Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

1 INTRODUCTION

DVD-GAN extends high-fidelity generative modeling to complex natural video by scaling GANs to Kinetics-600 with an efficient spatio-temporal discriminator decomposition. It targets video synthesis and prediction, producing high-resolution, temporally coherent samples and state-of-the-art results on UCF-101 synthesis and Kinetics-600 prediction.

  • Motivation: Video generation remains challenging because natural videos have greater data complexity and computational requirements than natural images.Prior work therefore often used simpler datasets or strong temporal conditioning information.
  • Approach: DVD-GAN uses an efficient spatio-temporal discriminator decomposition to train on the complex Kinetics-600 video dataset.The model builds on BigGAN and targets video synthesis and prediction while scaling to a dataset an order of magnitude larger than commonly used alternatives.
  • Approach: DVD-GAN produces high-quality natural videos at resolutions up to 256 × 256 and lengths up to 48 frames.The model is described as generating temporally coherent, relatively high-fidelity videos on Kinetics-600.
  • Results: DVD-GAN achieves state of the art for video synthesis on UCF-101 and video prediction on Kinetics-600.These results are presented as the paper’s main benchmark achievements.
  • Results: The paper establishes class-conditional synthesis on Kinetics-600 as a new generative-video benchmark and reports DVD-GAN as a strong baseline.The benchmark addresses a complex dataset previously less represented in generative video modeling.

2 BACKGROUND

The background frames video generation as a spectrum from unconditional synthesis to strongly conditioned generation, while introducing GANs, Kinetics-600, and video evaluation metrics. It also notes GAN limitations and motivates Kinetics-600 through its scale and diversity.

  • Video generation tasks: Video generation ranges from unconditional synthesis to strongly conditioned generation using signals such as another video, segmentation masks, or pose information.The paper positions intermediate conditioning tasks between these extremes.
  • Generative adversarial networks: GANs train a generator and discriminator through a minimax game, and this work uses the hinge objective optimized by gradient descent.The background identifies GAN training stability as a major focus of subsequent improvements.
  • Generative adversarial networks: GANs can suffer from mode collapse and lack an explicit likelihood measure for quantitative evaluation.Despite these limitations, GANs have produced high-fidelity samples across visual domains.
  • Kinetics-600: Kinetics-600 contains around 500,000 videos across 600 classes, making it substantially larger and more diverse than UCF-101.The dataset consists of unconstrained, high-resolution YouTube clips and includes artifacts such as cuts, title screens, and visual effects.
  • Kinetics-600: Prior work on Kinetics often used a stabilized 4,000-video subset, whereas this paper considers the full Kinetics-600 dataset.The authors regard the preprocessed subset as incomparable because of its heavy stabilization and much smaller size.
  • Evaluation metrics: Video evaluation uses Inception Score and Fréchet Inception Distance computed with an Inflated 3D Convnet trained on Kinetics-600.The resulting FID is similar to Fréchet Video Distance, though implemented differently and aligned with the original FID.

3 DUAL VIDEO DISCRIMINATOR GAN

DVD-GAN scales video generation by pairing a full-resolution spatial discriminator with a downsampled temporal discriminator, preserving high-resolution detail while modeling motion across the video.

  • 3 DUAL VIDEO DISCRIMINATOR GAN: DVD-GAN uses a scalable video-specific architecture built on BigGAN, with sequential components but no autoregressive pixel dependence.The model relies on learned data-driven structure rather than explicit foreground, background, or optical-flow priors.
  • 3.1 DUAL DISCRIMINATORS: The spatial discriminator samples k = 8 full-resolution frames and judges them individually, supplying feedback for single-frame content and structure.Its final score is the sum of the per-frame scores.
  • 3.1 DUAL DISCRIMINATORS: The temporal discriminator receives a 2 × 2 average-pooled version of the whole video, allowing it to assess movement and temporal discrepancies across the full sequence.The decomposition cannot reflect temporal evolution within a 2 × 2 window, but the authors report no observed effect on generated samples.
  • 3.1 DUAL DISCRIMINATORS: A 58% pixel-processing reduction enables DVD-GAN to scale discrimination for 48-frame 128 × 128 videos while retaining temporal and high-resolution feedback.The decomposition processes k × H × W pixels spatially and T × W^2 temporally, rather than the entire video at full resolution.
  • 3.1 DUAL DISCRIMINATORS: Unlike TGANv2, which evaluates full-length videos only at very low resolution after adaptive batch reduction, DVD-GAN retains a full-resolution spatial learning signal.This makes the spatial discriminator essential when temporal downsampling is used.

4 EXPERIMENTS AND ANALYSIS

Experiments evaluate DVD-GAN on class-conditional synthesis and future prediction across Kinetics-600, UCF-101, and BAIR. The model produces high-quality videos at multiple resolutions and lengths, achieves strong comparative results, and exposes trade-offs from temporal compression and resolution.

  • 4.1.1 KINETICS-600 RESULTS: DVD-GAN establishes strong Kinetics-600 synthesis benchmarks across resolutions and video lengths, although FID and IS must not be compared across different lengths.Table 1 reports FID and IS with and without truncation, evaluated at the training point with best FID.
  • 4.1.1 KINETICS-600 RESULTS: DVD-GAN generates plausible videos from 64 × 64 to 256 × 256 and up to 48 frames, retaining coherent scene details while higher resolution makes object coherence harder.The 48-frame models learn high-resolution images despite seeing no more high-resolution frames than the 12-frame model because k = 8 is fixed.
  • 4.2.1 FRAME-CONDITIONAL KINETICS: DVD-GAN-FP outperforms prior frame-conditional prediction work on Kinetics-600, while the unconditioned synthesis model has notably better FVD under the reported setup.The comparison generates 11 frames at 64 × 64 conditioned on 5 frames without frame skipping; the synthesis baseline generates 16 frames.
  • 4.3 ABLATIONS: Increasing the spatial discriminator’s sampled-frame count k improves both metrics with diminishing returns, whereas downsampling in the temporal discriminator lowers IS while leaving FID similar.The analysis seeks compressive discriminator inputs that reduce computation without sacrificing generator quality.
  • 4.2.2 BAIR ROBOT PUSHING: On BAIR, DVD-GAN-FP outperforms prior adversarial models but performs slightly worse than the concurrently developed Video Transformer.The task generates 15 frames conditioned on one starting frame, with FVD computed as in Unterthiner et al. (2018).

5 CONCLUSION

The paper concludes that DVD-GAN captures the complexity of large natural-video datasets and provides strong benchmarks for future work. It also describes separable attention as a computationally cheaper way to propagate information across spatio-temporal axes.

  • 5 CONCLUSION: DVD-GAN achieves state-of-the-art results on UCF-101 synthesis and frame-conditional Kinetics-600 prediction while producing complex and diverse samples.The authors position these results and Kinetics-600 baselines as reference points for future generative video modeling.
  • A SEPARABLE ATTENTION: Separable Attention attends over temporal, height, and width axes separately, reducing memory complexity while retaining information from all spatio-temporal locations.The decomposition replaces full attention over H × W × T positions with sequential axis-wise attention.

B.1 DATASET PROCESSING

The model processes varied video datasets through dataset-specific resizing and cropping, then combines class-conditioned latent inputs with recurrent and residual generator components and dual discriminators.

  • Dataset processing: UCF-101 clips are resized to 128 × 128 while preserving aspect ratio, then randomly cropped; BAIR videos remain at native 64 × 64 resolution.
  • Generator inputs: The generator maps Gaussian noise and a learned class embedding into a spatial tensor, using their concatenation throughout class-conditional Batch Normalization layers.
  • Temporal generation: A convolutional gated recurrent unit is unrolled once per frame, followed by two residual blocks that process each generated frame.
  • Dual discrimination: The spatial discriminator scores uniformly sampled frames independently, while the temporal discriminator downsamples videos and applies 3-D convolutions in its first two residual blocks.The default number of sampled frames is k = 8.

B.3 TRAINING DETAILS

DVD-GAN combines efficient feed-forward sampling with a dual-discriminator training setup and a residual-network modification for conditioned future-video prediction.

  • Video prediction: Figure 8 presents the architecture changes used to add conditioning for future video prediction.
  • Training details: The model uses an exponential moving average of generator weights, spectral normalization, orthogonal initialization, and Adam optimization during training.The moving average uses decay γ = 0.9999 after 20,000 training steps.
  • Video prediction: DVD-GAN-FP supports future prediction by passing each conditioning frame through a deep residual network modeled after the spatial discriminator.Features corresponding to generator intermediate tensors are stacked across conditioning frames, then processed with a 3 × 3 convolution and ReLU.

C.1 UCF-101

On UCF-101, DVD-GAN achieves a high Inception Score, but the authors caution that memorization and limited dataset diversity undermine the metric’s interpretation.

  • Interpolation analysis: Figure 9 shows the first frames of latent interpolations between UCF-101 samples, with each row representing a separate interpolation.
  • UCF-101 results: 32.97 Inception Score significantly outperforms the state of the art on UCF-101.The model is trained on 16-frame 128 × 128 clips using the same architecture as on Kinetics.
  • Metric caveat: The authors attribute at least part of the improved score to memorization, with generated samples highly correlating with training examples.Interpolation samples show abrupt latent-space jumps and little intra-video diversity within groups.
  • Metric caveat: UCF-101’s Inception Score can fail to reflect meaningful generation because its small classes and reused source clips restrict intra-class diversity.

D MISCELLANEOUS EXPERIMENTS

Additional experiments examine normalization and residual-block variants, revealing unstable or inconclusive alternatives and a critical dependence on the final generator normalization layer.

  • Experimental scope: The appendix reports these modifications as experiments that did not produce a conclusive result.
  • Normalization: Group Normalization performed nearly as well as, but worse than, Batch Normalization, while Layer Normalization, Instance Normalization, and no normalization significantly underperformed.
  • Normalization: Removing the generator’s final Batch Normalization caused catastrophic learning failure, whereas removing residual-block normalization still produced slightly worse models.Some residual-block variants nevertheless reached Inception Scores up to 110.05 for 64 × 64 12-frame samples.
  • Temporal normalization: Normalizing independently over timesteps worked as well as early all-frame normalization without requiring batch statistics to generate good samples.
  • Residual blocks: BigGAN-deep residual blocks trained faster in wall-clock time but more slowly by metrics and failed to match the accuracy of BigGAN residual blocks.

E.1 SYNTHESIS SAMPLES

DVD-GAN produces samples across Kinetics-600 video lengths and resolutions, including 12 or 48 frames at 64 × 64 and 128 × 128, and 12 frames at 256 × 256. A degenerate sample is occasionally observed in the 48-frame, 128 × 128 setting.

  • Synthesis samples: DVD-GAN generates samples at 12 or 48 frames and resolutions from 64 × 64 to 256 × 256 on Kinetics-600.The supplied figures cover 12-frame 64 × 64, 48-frame 64 × 64, 12-frame 128 × 128, 48-frame 128 × 128, and 12-frame 256 × 256 samples.
  • Synthesis samples: A degenerate sample occasionally appears in the 48-frame, 128 × 128 Kinetics-600 samples.The caption identifies the example as row 1, column 5.

E.2 INTERPOLATION SAMPLES

DVD-GAN supports truncation and produces smooth video changes when interpolating either latent vectors or class embeddings. These interpolations indicate that the generator learned a relatively smooth mapping from latent inputs to real videos.

  • Truncation and smoothness: DVD-GAN is amenable to truncation, which is motivated by expecting higher-quality samples from latents near the distribution mean.The paper also uses latent and class interpolations as evidence of a relatively smooth generator mapping.
  • Intra-class interpolation: Latent-space interpolations show smooth transitions between generated videos across all six displayed timesteps.The interpolation holds the class fixed while varying between two randomly sampled latent vectors.
  • Class interpolation: Class-embedding interpolation varies the generated class while keeping the latent vector fixed.The endpoint columns use two different classes, with intermediate columns generated across the class-embedding interpolation.
Loading 1907.06571v2…