PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN

Masaki Saito, Shunta Saito, Masanori Koyama, Sosuke Kobayashi

arXiv:1811.09245v2cs.CV

TL;DR

High-resolution video GAN training is constrained by complex observations, large datasets, and high computational and memory demands. The paper trains a stacked multi-scale generator with subsampling layers that reduce frame rate during training while retaining dense high-resolution outputs for inference. Its results indicate that combining the multiscale model with subsampling layers is important for substantial video-generation quality improvement.

  • Problem

    High-resolution video GAN training must handle complex temporal-spatial observations and large video datasets under rapidly increasing computational and GPU-memory demands.

  • Method

    The method stacks sub-generators and trains them with intermediate frame-rate subsampling layers, using sparse training samples and dense high-resolution outputs during inference.

  • Results

    Combining the multiscale model with subsampling layers produces the substantial quality improvement, whereas either component alone does not dramatically improve quality.

  • Takeaways & Limitations

    The approach enables efficient training of high-resolution video generators while preserving or improving generated-video quality relative to baseline architectures.

  • Takeaways & Limitations

    Applications beyond video generation, including audio generation and video prediction, remain possibilities for future exploration rather than demonstrated uses here.

Abstract

from arXiv · show

Training of Generative Adversarial Network (GAN) on a video dataset is a challenge because of the sheer size of the dataset and the complexity of each observation. In general, the computational cost of training GAN scales exponentially with the resolution. In this study, we present a novel memory efficient method of unsupervised learning of high-resolution video dataset whose computational cost scales only linearly with the resolution. We achieve this by designing the generator model as a stack of small sub-generators and training the model in a specific way. We train each sub-generator with its own specific discriminator. At the time of the training, we introduce between each pair of consecutive sub-generators an auxiliary subsampling layer that reduces the frame-rate by a certain ratio. This procedure can allow each sub-generator to learn the distribution of the video at different levels of resolution. We also need only a few GPUs to train a highly complex generator that far outperforms the predecessor in terms of inception scores.

1 Introduction

High-resolution video GAN training is difficult because videos combine complex temporal-spatial structure with large data and rapidly increasing computational demands. The paper addresses these constraints with stacked sub-generators and training-time frame subsampling, producing higher-quality video efficiently.

  • Video generation requires modeling complex temporal dynamics while making every generated frame photo-realistic.
  • Large video observations make dataset training difficult, while computational resource grows exponentially with resolution.
  • Prior work often improved photo-realism using low-resolution videos, leaving high-resolution training with limited GPU memory and batch size.
  • The generator stacks sub-generators and inserts training-time subsampling layers that reduce frame rate between consecutive stages.Lower stages process high-frame-rate, low-resolution abstract information, while higher stages process low-frame-rate, high-resolution local information.
  • The method efficiently produces videos with a significantly higher inception score of ∼26 than predecessors.

2 Related work

Prior video-generation GANs established several architectures, including spatiotemporal, motion-content, and domain-translation models, but high-resolution generation remains computationally demanding. This work extends that line with multiple subsampling layers to reduce cost and memory for video generation.

  • RecycleGAN and Vid2Vid address video translation, with Vid2Vid learning high-resolution outputs from paired videos.
  • Multi-scale image GANs such as LAPGAN, StackGAN, and HDGAN motivate the paper’s multi-scale generator design.
  • VGAN combines a 2D background generator with 3D convolutional networks for foreground video generation.
  • TGAN separates temporal latent-vector generation from spatial image transformation, while MoCoGAN decomposes motion and content representations.
  • Existing video GANs require enormous computational cost and GPU memory, especially for high-resolution video.The proposed model is described as an extension of MoCoGAN that uses multiple discriminators with lower computational and memory requirements.

3 Method

The method replaces a conventional video GAN's single high-cost discriminator path with stacked sub-generators, training-time subsampling layers, and multiple sub-discriminators. This divides temporal-flow and visual-fidelity roles across resolutions and frame rates while reducing computational and memory demands.

  • Conventional video GANs: Conventional temporal GANs use a temporal generator for T latent vectors, an image generator for frames, and a 3D-convolutional discriminator on the synthesized video.The temporal generator is typically recurrent; the proposed implementation uses a convolutional LSTM for this component.
  • Single subsampling layer: The training architecture inserts a subsampling layer between the abstract and rendering blocks, producing a lower-frame-rate video for the discriminator while inference retains the conventional generator.The real videos are downscaled and frame-rate reduced with a corresponding subsampling operation so their tensor dimensions match the synthetic samples.
  • Single subsampling layer: A single subsampling layer reduces discriminator cost, but the resulting samples receive significantly lower scores than samples from the naive architecture.The method therefore extends the design with multiple sub-generators and subsampling layers rather than relying on one subsampling operation.
  • Multiple subsampling layers: The full generator stacks L sub-generators and uses L−1 training-time subsampling layers, with resolution increases countered by frame-rate decreases.At inference, abstract blocks are applied sequentially and a rendering block produces the video; during training, corresponding rendering blocks and subsampling layers are used.
  • Multiple discriminators: Multiple sub-discriminators evaluate outputs from the corresponding sub-generators, while real videos are transformed to matching resolution and frame-rate scales.The overall discriminator aggregates scalar outputs from the sub-discriminators through a sigmoid-based score.
  • Role of each discriminator: The role division assigns low-index sub-generators to low-resolution temporal flow and high-index sub-generators to high-resolution visual quality.This lets different sub-generators learn selected parts of the video distribution while reducing computational cost and memory consumption.

4 Network architecture

The model uses stacked sub-generators and sub-discriminators, with training-time subsampling layers that reduce frame rates across levels. This design targets the discriminator’s rapidly increasing cost and memory while preserving multi-scale video generation.

  • Generator: The generator consists of four sub-generators that progressively transform latent feature maps into video frames at increasing resolutions.Each sub-generator produces intermediate features, while the inference path ultimately renders frames at W × H resolution.
  • Training-time subsampling: Training inserts subsampling layers between sub-generators to reduce the number of frames in intermediate feature maps.A reduction rate st changes a tensor with Th frames into one with ⌈Th/st⌉ frames, using a probabilistic temporal subsampling operation.
  • Discriminator: The sub-discriminator with the lowest index evaluates global video flow, whereas the highest-index sub-discriminator evaluates photorealism in selected frames.Although the sub-discriminators share the same general structure, they serve different roles across temporal and spatial scales.
  • Computational cost and memory: At st = 2, subsampling improves total memory usage and computational cost by four∼five times over the vanilla model without subsampling layers.The overall gain is driven mainly by reducing discriminator cost, which is substantially heavier than generator cost.
  • Computational cost and memory: The model with st = 4 uses about 60% less memory than the model with st = 2.Larger st generally increases computational savings, but the number of original frames required for constant-order cost grows exponentially with st.

5 Experiments

Experiments on FaceForensics and UCF101 evaluate video quality, diversity, stability, and memory efficiency against high-resolution baselines. The proposed models generate high-fidelity videos while avoiding the mode collapse and memory growth observed in baselines, although larger subsampling can destabilize initial frames.

  • Datasets and setup: Experiments use UCF101 and FaceForensics, with 16-frame training clips and normalized video inputs.UCF101 provides 13,320 videos across 101 categories, while the FaceForensics setup uses 854 facial videos resized to 256 × 256 pixels.
  • FaceForensics: At st = 2, the model generates high-resolution FaceForensics videos with high-fidelity faces, smoothly moving facial parts, and diverse identities.The authors report no mode collapse across sampled faces and describe eyes and mouths as moving smoothly.
  • Memory efficiency: The proposed model can train videos that consume 16 times more memory than conventional small samples while avoiding extreme discriminator-memory growth.The authors contrast this with MoCoGAN, which requires about 10Gb to generate 64 × 64 px videos at batch size 8.
  • Effect of subsampling rate: With st = 4, FaceForensics videos show inconsistent initial frames, whereas UCF101 shows milder gradual changes and quantitative performance can favor st = 4.Both settings were reported as stable after the initial frames, and the instability depends on the dataset.
  • UCF101: On UCF101, proposed models produce more interpretable content, clearer backgrounds, and sharper shadows than the baseline models.The baselines often produce meaningless videos through mode collapse, whereas the proposed models do not show that tendency.
  • Conditional generation: Conditional videos tend to be more stable and quantitatively better than unconditional videos, especially for distant views.The quality advantage is reported as a tendency, while videos showing well-moving people remain comparable between settings.

5.5 Linear interpolation of the noise vector

The paper tests whether interpolating between noise vectors yields smoothly changing videos rather than memorized training examples. Intermediate noise produces smooth content changes in the FaceForensics model.

  • Interpolation setup: The experiment interpolates linearly between two noise vectors and evaluates the resulting video content.The model uses st = 2 and is trained on FaceForensics.
  • Interpolation result: The generated video content changes smoothly as the noise vector moves between two endpoints, indicating that the model is not merely memorizing the dataset.Changing the noise can alter the speaker or smoothly modify attributes such as face orientation.

5.6 Consistency of generated samples

The consistency analysis examines whether rendering blocks preserve content and where st = 4 instability arises. Rendering blocks generally agree on content, but higher-level blocks can introduce unstable initial frames when aggressive subsampling is used.

  • Content consistency: Rendering blocks tend to generate the same content under the same noise vector without color-consistency regularization.This observation is made from visualizing outputs at each rendering block.
  • Instability source: For st = 2, rendering blocks produce videos with almost identical content, while st = 4 becomes increasingly unstable in the first frames as level increases.The authors attribute the instability to later rendering blocks no longer outputting the same content.
  • Comparison setup: The analysis compares videos produced by rendering blocks at different levels and across time.The figures organize rendering-block levels and temporal frames to inspect content consistency and frame differences.
  • Quantitative consistency: PSNR and SSIM comparisons between level 1 and level 4 show rapid initial instability for FaceForensics but not for UCF101.FaceForensics later approaches level-1 content, although st = 4 remains worse in identity than st = 2; UCF101 shows slightly worse post-transition similarity than FaceForensics.

5.7 Comparison with other existing methods

The proposed model achieves higher video-generation quality than existing methods, while experiments examine whether this advantage reflects resolution, batch size, or the subsampling design.

  • Evaluation setup: The evaluation uses Inception Score (IS) and Fréchet Inception Distance (FID) on UCF101 videos, with generated videos resized for the classifier.IS and FID were computed using a pretrained C3D classifier; each snapshot was evaluated with 2,048 samples and repeated ten times.
  • Comparison scope: The existing methods mostly generate 64 × 64 videos, whereas the proposed method is evaluated at higher resolution.ProgressiveGAN is the exception among existing methods, generating 128 × 128 videos.
  • Comparison results: Simply increasing resolution does not substantially improve baseline scores, while the proposed model with subsampling layers obtains much higher IS and FID performance.The no-subsampling model performs much worse than the proposed model, indicating that resolution alone does not explain the improvement.
  • Comparison results: The proposed models with st = 2 and st = 4 significantly outperform existing methods using four GPUs, and st = 4 is slightly better than st = 2.The paper notes that larger st can save more computation and GPU memory but may destabilize early frames on some datasets.

5.8 Quantitative results under a single GPU

Under a single-GPU constraint, the proposed model maintains higher video-generation quality than existing and baseline models, whereas the no-subsampling model struggles with high-resolution training.

  • Experimental setting: The single-GPU experiment compares two baselines and the proposed models under the same constrained hardware setting.The experiment tests whether the proposed method retains its advantage without the multi-GPU setting used elsewhere.
  • Baseline behavior: The high-resolution model without subsampling layers performs much worse than existing methods because its batch size is only one.This result illustrates the difficulty of training high-resolution video models with limited computational resources.
  • Results: Even with one GPU, the proposed model achieves significantly higher IS than existing methods, including ProgressiveGAN.The comparison supports the method’s effectiveness under constrained computational resources.
  • Results: The proposed models outperform existing and baseline models under both one- and four-GPU environments.The broader comparison is reported alongside the single-GPU results and same-batch-size experiments.

5.10 Effectiveness of frame sub-sampling layers

Frame subsampling improves the proposed model’s IS and FID across batch sizes, with st = 4 generally outperforming st = 2 and reaching a reported IS of 28.87.

  • Experimental coverage: The experiments evaluate subsampling-layer effects across batch sizes, including conditional and unconditional settings.The reported comparisons use IS and FID, with st values of 2 and 4 and batch sizes including 16, 32, 64, and 128.
  • Subsampling effectiveness: Subsampling layers improve IS and FID over the model without subsampling layers across tested batch sizes.The later discriminator judges lower-frame-rate videos using global motion and still-image quality.
  • Hyperparameter comparison: The st = 4 configuration produces significantly higher IS and FID than st = 2 in almost all batch sizes.At batch size 128, st = 4 achieves an inception score of 28.87, compared with 14.56 for the existing state-of-the-art method.
  • Conditional model: The conditional model substantially improves scores across batch sizes and hyperparameters.For st = 2 and batch size 64, it reaches an inception score of 54.93; increasing batch size does not consistently improve FID.

5.12 Effectiveness of each subsampling layer

Sequential activation experiments show that individual subsampling layers contribute incrementally, with S1 providing the largest improvement; applying subsampling alone to baseline models instead harms quality.

  • Layer-wise contribution: Gradually enabling subsampling layers improves both IS and FID for st = 2 and st = 4.The experiments activate S1, S2, and S3 sequentially to measure each layer’s contribution.
  • Layer-wise contribution: S1 contributes most to the quality improvement in both st settings.The largest score change occurs when S1 is enabled, while later layers provide additional improvements in the proposed architecture.
  • Layer-wise contribution: For st = 4, changing frame counts from to [1] improves both scores, whereas the intermediate change does not.This result distinguishes the effects associated with later subsampling layers.
  • Baseline comparison: Applying a single subsampling layer to baseline models makes IS and FID significantly worse than the naive implementation.With st = 4, training failed for both baseline models; the authors attribute the contrast to combining subsampling with a multiscale model.

6 Discussion

The experiments identify the combination of multiscale modeling and subsampling as the main source of quality improvement, while also examining training settings, discriminator choices, and longer-video generation. The method sometimes generates stable videos beyond the 16-frame training length, but this remains dataset- and hyperparameter-dependent.

  • The multiscale model and subsampling layer jointly produced the largest quality improvement; either component alone was insufficient.Increasing GPU count without subsampling yielded only a small score improvement, whereas subsampling helped substantially only in the multiscale model.
  • Subsampling improved quality while significantly reducing GPU computation and memory consumption, enabling high-resolution generation with a single GPU.The resulting inception scores exceeded existing methods, and simply increasing resolution in two baselines did not reproduce the increase.
  • Increasing batch size improved video quality, while increasing st improved quality regardless of batch size but could destabilize the first frames.The authors recommend starting training with st = 2, then increasing it while monitoring IS and FID.
  • The first subsampling layer, S1, contributed most to score improvement, while enabling later layers did not reduce scores or add computation and memory cost.The authors suggest enabling all layers initially and fine-tuning by activating them gradually when resources and time permit.
  • Longer-video generation beyond the 16-frame training length was sometimes stable and sometimes broken, with the governing conditions remaining unclear.The text states that stability depends on hyperparameters and the dataset, while generation up to 16 frames was stable regardless of dataset.
  • The final 3D ResNet discriminator achieved higher inception scores than a simpler 3D-convolutional discriminator, although the simpler version trained faster.The simple discriminator was used for faster trial-and-error searches, while 3D ResNet was used for final evaluation.

7 Conclusion

The paper introduces a training method that uses sparse samples during training and dense high-resolution videos during inference. Experiments report higher-quality high-resolution videos than conventional and baseline methods, with subsampling layers contributing to the improvement.

  • The method trains a multiscale video generator using sparse samples at low computational cost, then produces dense high-resolution videos during inference.This training–inference distinction is the paper’s central design.
  • The method saves computational cost and GPU memory while training a high-resolution video-generation model.
  • Both qualitative and quantitative experiments found higher-quality high-resolution videos than conventional methods, with subsampling layers contributing to the improvement.
  • The authors suggest that the core idea may extend to video prediction and other time-series domains such as audio generation.The paper presents these as possible future applications rather than demonstrated results.
Loading 1811.09245v2…