PaperlayerView exact revision on arXiv

Source-linked AI summary

A Good Image Generator Is What You Need for High-Resolution Video Synthesis

Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, Sergey Tulyakov

arXiv:2104.15069v1cs.CV

TL;DR

High-resolution video synthesis remains difficult because video models are computationally demanding and produce lower-resolution outputs than image generators. MoCoGAN-HD searches for motion trajectories in a fixed pre-trained image generator’s latent space, disentangling motion from content. The framework reports superior benchmark results, resolutions up to 1024 × 1024, and an order of magnitude less training time than prior video-based works.

  • Problem

    Video synthesis has progressed more slowly than image synthesis, producing relatively low-resolution videos while requiring longer training times and larger computational budgets.

  • Method

    MoCoGAN-HD uses a motion generator to discover trajectories in the latent space of a fixed pre-trained image generator, with motion and content disentangled.

  • Results

    The framework achieves superior results across benchmarks and reaches resolutions unattainable by prior state-of-the-art techniques.

  • Takeaways & Limitations

    The disentangled representation supports content and motion manipulation and cross-domain video synthesis using disjoint image and video datasets.

  • Takeaways & Limitations

    Training is limited to 16 consecutive frames because of computational-resource constraints, although longer sequences can be generated during inference.

Abstract

from arXiv · show

Image and video synthesis are closely related areas aiming at generating content from noise. While rapid progress has been demonstrated in improving image-based models to handle large resolutions, high-quality renderings, and wide variations in image content, achieving comparable video generation results remains problematic. We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. Not only does such a framework render high-resolution videos, but it also is an order of magnitude more computationally efficient. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled. With such a representation, our framework allows for a broad range of applications, including content and motion manipulation. Furthermore, we introduce a new task, which we call cross-domain video synthesis, in which the image and motion generators are trained on disjoint datasets belonging to different domains. This allows for generating moving objects for which the desired video data is not available. Extensive experiments on various datasets demonstrate the advantages of our methods over existing video generation techniques. Code will be released at https://github.com/snap-research/MoCoGAN-HD.

1 INTRODUCTION

MoCoGAN-HD addresses the difficulty of generating high-resolution, temporally consistent videos by searching for trajectories in a pre-trained image generator’s latent space. Its motion generator separates motion from content, enabling efficient high-resolution and cross-domain video synthesis.

  • 1 INTRODUCTION: Video synthesis remains comparatively limited because video statistics and model sizes make high-resolution generation computationally demanding.Existing video methods produce relatively low-resolution videos and require longer training times than recent image-generation systems.
  • 1 INTRODUCTION: Generated videos must combine high-quality individual frames with temporal consistency depicting the same content under plausible motion.
  • 1 INTRODUCTION: The framework discovers a latent trajectory with a motion generator that models motion as residuals between continuous image-generator codes.The residual representation supports disentangling motion and content.
  • 1 INTRODUCTION: 1024 × 1024 is the maximum reported resolution, while training requires an order of magnitude less time than previous video-based works.The image generator remains fixed and can be trained on a separate high-quality image dataset.
  • 1 INTRODUCTION: Cross-domain synthesis transfers motion learned from videos to image domains lacking corresponding high-quality video data.The approach can generate videos of objects not seen moving during training.

2 RELATED WORK

The related work situates MoCoGAN-HD at the intersection of high-quality image generation, interpretable latent directions, contrastive representation learning, and video synthesis. Its distinctive connection is transferring disentangled motion trajectories from video data to image domains without temporal data.

  • 2 RELATED WORK: Prior GAN-based image methods established high-quality image synthesis, while video-to-video translation targets high-resolution outputs in another domain.
  • 2 RELATED WORK: GAN latent spaces contain semantically meaningful directions that support image manipulation such as rotation and background removal.
  • 2 RELATED WORK: MoCoGAN-HD extends latent-space analysis to motion vectors and disentangles motion trajectories without supervision.
  • 2 RELATED WORK: The disentangled motion representation transfers motion information from a video dataset to an image dataset with no temporal information.
  • 2 RELATED WORK: Contrastive representation learning brings related augmented inputs closer in representation space during unsupervised training.

3 METHOD

The method fixes a pre-trained image generator and trains a motion generator to produce latent trajectories whose frames preserve quality, content, and realistic temporal dynamics. Video, image, contrastive, feature-matching, and mutual-information objectives jointly constrain the generated sequences.

  • 3 METHOD: The fixed pre-trained image generator synthesizes high-quality frames, while motion learning can use video data from a different domain.
  • 3 METHOD: The motion generator uses LSTM encoder and decoder networks to predict consecutive latent codes for the synthesized frames.
  • 3 METHOD: Each latent code is updated from the previous code by a residual motion direction scaled by λ and projected through the PCA basis V.
  • 3 METHOD: The mutual-information loss prevents motion mode collapse by linking decoder hidden vectors to sampled noise vectors.
  • 3 METHOD: A multi-scale 3D video discriminator constrains generated motion patterns to resemble training videos, while an image discriminator matches frame quality and content.
  • 3 METHOD: The contrastive image discriminator uses shared-weight image features and InfoNCE-style pairs so same-content frames are close and different-content frames are separated.Positive pairs can be formed from augmented frames sharing content, including in cross-domain settings.
  • 3 METHOD: Feature matching additionally compares the generated first frame with later frames using cosine similarity instead of L1 regularization.

4 EXPERIMENTS

Experiments evaluate MoCoGAN-HD for unconditional video synthesis, video prediction, cross-domain generation, disentanglement, and ablations. Results cover multiple datasets and show high-quality generation, strong benchmark performance, separable motion and content, and longer-sequence synthesis beyond the training length.

  • Video Generation: MoCoGAN-HD achieves state-of-the-art IS and FVD on UCF-101, outperforming existing works by a large margin.The authors interpret this as evidence that the image generator can represent rich motion patterns when paired with a trained motion generator.
  • Video Generation: On FaceForensics, MoCoGAN-HD has identity consistency close to ground truth and is preferred over TGANv2 in 73.6% of comparisons.The method also achieves better results than TGANv2 on the reported FVD evaluation.
  • Video Generation: On Sky Time-lapse, MoCoGAN-HD significantly outperforms MDGAN and DTVNet on video synthesis, and is evaluated for video prediction using PSNR and SSIM.The evaluation uses 128 × 128 frames and 16-frame generated sequences.
  • Cross-Domain Video Generation: Cross-domain experiments combine image content from FFHQ, AFHQ-Dog, AnimeFaces, or LSUN-Church with motion from VoxCeleb or TLVDB, producing high-quality, temporally consistent videos.Generated resolutions include 256 × 256, 512 × 512, and 1024 × 1024.
  • Cross-Domain Video Generation: Disentanglement experiments show diverse motion for the same identity and the same motion applied to different identities.Figures 5 and 6 report these controls for AFHQ-Dog with VoxCeleb motion and AnimeFaces with VoxCeleb motion.
  • Ablation Analysis: The full model outperforms ablations that remove residual motion modeling, the contrastive image discriminator, or the video discriminator on UCF-101.Additional cross-domain ablations find contrastive loss beneficial for coherent content and mutual information loss beneficial for video quality and motion diversity.
  • Long-Sequence Generation: Although trained on 16 consecutive frames because of computational limitations, the model generates longer sequences through motion-generator unrolling or motion-trajectory interpolation.Examples include 64 frames from a FaceForensics model and 32 frames for the AFHQ-Dog–VoxCeleb setting.

5 CONCLUSION

MoCoGAN-HD uses a motion generator to find latent trajectories in a fixed image generator, enabling high-resolution, efficient, disentangled video synthesis and broader applications.

  • 5 CONCLUSION: The framework generates motion residuals that define a meaningful trajectory through an image generator’s latent space.The motion generator uses recurrent networks, while contrastive image and video discriminators support the learned trajectory.
  • 5 CONCLUSION: MoCoGAN-HD supports content and motion manipulation and cross-domain video synthesis through disentangled representations.The image and motion components can be trained on disjoint domains, enabling motion generation when matching video data is unavailable.
  • 5 CONCLUSION: The framework achieves high-resolution video synthesis, including resolutions unattainable by prior state-of-the-art techniques.The implementation uses StyleGAN2 latent codes and principal components to model motion residuals.

A.2.1 VIDEO DISCRIMINATOR

The video discriminator processes generated sequences at two spatial scales and compares the first frame with each subsequent frame using six-channel inputs.

  • A.2.1 VIDEO DISCRIMINATOR: The discriminator receives video inputs at 128 × 128 and 64 × 64 resolutions.The generated images are downsampled to both scales before being processed.
  • A.2.1 VIDEO DISCRIMINATOR: Each discriminator input has 6 × (n −1) channels-by-time-and-space structure for a sequence of n frames.For every subsequent frame, the RGB channels of the first and subsequent frames are concatenated.
  • A.2.1 VIDEO DISCRIMINATOR: The section provides the network architecture for the video discriminator.The architecture is summarized in Table 6.

A.2.2 IMAGE DISCRIMINATOR

The image discriminator and contrastive feature extractor support temporally consistent content learning through shared architecture choices, augmentation, and a memory bank of negative examples.

  • A.2.2 IMAGE DISCRIMINATOR: The image discriminator follows BigGAN’s discriminator architecture without self-attention.The contrastive feature extractor shares this architecture, removes its last layer, and adds two 256-unit projection layers.
  • A.2.2 IMAGE DISCRIMINATOR: The section introduces the image augmentation and memory-bank techniques used for contrastive learning.These techniques are presented as supporting details for the contrastive training procedure.
  • A.2.2 IMAGE DISCRIMINATOR: Contrastive learning creates positive and negative image pairs using normalization and affine, brightness, color, cutout, and flipping augmentations.The augmentations perturb image geometry, intensity, color, visibility, and orientation.
  • A.2.2 IMAGE DISCRIMINATOR: A memory bank increases the number of contrastive negative pairs by buffering fake video-frame outputs from an exponential-moving-average image discriminator.The memory mechanism follows MoCo and uses a dictionary of 4

B MORE DETAILS FOR EXPERIMENTS

The experiments combine trained or released image generators with video-generation and prediction evaluations across multiple image and video datasets, including cross-domain settings.

  • B MORE DETAILS FOR EXPERIMENTS: StyleGAN2 image generators are trained on UCF-101, FaceForensics, Sky Time-lapse, and AFHQ-Dog, while released models are used for FFHQ, AnimeFaces, and LSUN-Church.Selected checkpoints use the best FID scores; BigGAN is additionally trained on FFHQ at 128 × 128 resolution.
  • B MORE DETAILS FOR EXPERIMENTS: Training each listed image generator takes less than 2 days, while video-generator training takes 1.5–3 days depending on the dataset.Generating 1024 × 1024 videos required 5 days on 8 Quadro RTX 8000 GPUs.
  • B MORE DETAILS FOR EXPERIMENTS: Video prediction inverts the first test frame into the image generator’s latent space, then uses the motion generator to synthesize subsequent frames.The inverse latent code is optimized with perceptual loss, and the resulting trajectory is interpolated to produce 32 latent codes.
  • B MORE DETAILS FOR EXPERIMENTS: Evaluation uses PSNR and SSIM for video prediction, with AMT studies averaging scores across 5 iterations.The AMT comparisons include FaceForensics versus TGANv2 and Sky Time-lapse versus DTVNet.
  • B MORE DETAILS FOR EXPERIMENTS: On Sky Time-Lapse, the method significantly outperforms MDGAN and DTVNet on FVD and shows a clear human-evaluation advantage over DTVNet.The dataset contains dynamic sky scenes and uses 16-frame training sequences.
  • B MORE DETAILS FOR EXPERIMENTS: Cross-domain datasets pair image collections such as FFHQ, AFHQ-Dog, AnimeFaces, and LSUN-Church with video collections including VoxCeleb and TLVDB.Video training sequences are formed by selecting every other frame from 32 consecutive frames.

C MORE VIDEO RESULTS

The framework generates videos across multiple datasets and domains, including long sequences, diverse motion patterns, content-motion recombination, and resolutions up to 1024 × 1024.

  • The approach generates videos on UCF-101, FaceForensics, and Sky Time-lapse datasets.
  • 32- and 64-frame FaceForensics videos are generated even though training uses 16-frame clips.
  • The same content code produces diverse motion patterns, while the same motion codes produce videos with different content.
  • Cross-domain synthesis combines VoxCeleb motion with FFHQ content at 128 × 128, 256 × 256, and 1024 × 1024 resolutions.
  • Cross-domain results combine VoxCeleb motion with AFHQ-Dog or AnimeFaces content at 512 × 512 resolution.
  • Additional cross-domain synthesis combines TLVDB time-lapse motion with LSUN-Church content at 256 × 256 resolution.

D MORE ABLATION ANALYSIS FOR MUTUAL INFORMATION LOSS Lm

The ablation evaluates whether mutual information loss increases motion diversity by comparing the Full model with a version that omits Lm.

  • The experiment synthesizes 100 sixteen-frame videos from one sampled content code and analyzes their mean video and per-pixel variation.
  • 0.233 ± 0.036 versus 0.126 ± 0.025: the Full model has higher per-pixel standard deviation on the 16th frame than w/o Lm.
  • The Full model’s blurrier averaged frames and higher per-pixel standard deviation indicate more diverse motion than w/o Lm.
  • The comparison runs across 50 trials and disables StyleGAN2 noise inputs to isolate the ablation.

E LIMITATIONS

The framework depends on an image generator that provides both high-quality frames and sufficiently diverse image content; weaknesses in either property constrain video quality or motion modeling.

  • The image generator must synthesize high-quality images so frame quality does not reveal generated videos to the video discriminator.
  • It must also generate diverse image contents to provide enough motion modes for sequence modeling.
  • On UCF-101, StyleGAN2 achieves FID 45.63, and the generated video quality remains below that of other datasets and not close to real videos.
  • On BAIR, the method models shadow changes and robot-arm motion but struggles to decouple the arm from small background objects.

E.1 ANALYSIS OF THE INFORMATION CONTAINED IN PCA COMPONENTS.

PCA analysis shows that the BAIR image-generator latent space concentrates variation in a few entangled components, limiting motion disentanglement compared with FFHQ.

  • The top 20 PCA components capture 85% of the variance in the BAIR image-generator latent space.
  • Moving along PCA directions is used to visualize how latent components affect synthesized five-frame video clips.
  • Later BAIR components produce only small changes, while its first seven components have entangled semantic meanings.
  • FFHQ components exhibit more disentangled meanings, such as rotation and smile, making foreground-background disentanglement easier.
Loading 2104.15069v1…