PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, Jinwoo Shin

arXiv:2202.10571v1cs.CVcs.LG

TL;DR

Video generation is difficult because discrete 3D-grid representations scale poorly and neglect continuous spatio-temporal dynamics. DIGAN uses an INR-based GAN with separate space-time motion handling and an efficient motion discriminator, improving benchmark results while supporting long, interpolated, extrapolated, and non-autoregressive videos.

  • Problem

    Discrete 3D-grid video modeling has cubic complexity and ignores continuous temporal dynamics, limiting scalable video generation.

  • Method

    DIGAN is an INR-based video GAN with a dynamics-aware generator and a motion discriminator operating on image pairs and time differences.

  • Results

    30.7%: DIGAN improves UCF-101 FVD from 833 to 577 and demonstrates long-video, extrapolation, and non-autoregressive generation.

  • Takeaways & Limitations

    DIGAN shows that continuous INR video representations can support diverse video-generation capabilities, including 128-frame 128×128 training.

  • Takeaways & Limitations

    The paper notes that separating image and motion discriminators is an empirical design choice, although a video discriminator alone can learn the video distribution.

Abstract

from arXiv · show

In the deep learning era, long video generation of high-quality still remains challenging due to the spatio-temporal complexity and continuity of videos. Existing prior works have attempted to model video distribution by representing videos as 3D grids of RGB values, which impedes the scale of generated videos and neglects continuous dynamics. In this paper, we found that the recent emerging paradigm of implicit neural representations (INRs) that encodes a continuous signal into a parameterized neural network effectively mitigates the issue. By utilizing INRs of video, we propose dynamics-aware implicit generative adversarial network (DIGAN), a novel generative adversarial network for video generation. Specifically, we introduce (a) an INR-based video generator that improves the motion dynamics by manipulating the space and time coordinates differently and (b) a motion discriminator that efficiently identifies the unnatural motions without observing the entire long frame sequences. We demonstrate the superiority of DIGAN under various datasets, along with multiple intriguing properties, e.g., long video synthesis, video extrapolation, and non-autoregressive video generation. For example, DIGAN improves the previous state-of-the-art FVD score on UCF-101 by 30.7% and can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.

1 INTRODUCTION

Video generation remains difficult because videos contain complex, continuous spatio-temporal dynamics that discrete 3D-grid modeling scales poorly and does not capture. DIGAN addresses this with an INR-based GAN combining separate space-time handling and motion-aware discrimination.

  • Motivation: Prior video methods represent videos as 3D RGB grids, causing cubic complexity and overlooking continuous temporal dynamics.This limits scalability for generated videos.
  • Motivation: INRs encode continuous signals as compact neural networks mapping coordinates to signal values rather than storing discrete grid values.Their compactness supports reduced memory use and arbitrary-resolution upsampling.
  • DIGAN: DIGAN introduces an INR-based video GAN whose generator decomposes content and motion while incorporating temporal dynamics into motion features.The generator regulates temporal variation, adds nonlinear motion modeling, and conditions motion on a random vector for diverse motions sharing an initial frame.
  • DIGAN: DIGAN uses a motion discriminator that detects unnatural motion from image pairs and their time difference instead of observing long frame sequences.This design uses a 2D convolutional network rather than an expensive full-sequence motion discriminator.
  • Results: 30.7%: DIGAN improves UCF-101 FVD from 833 to 577.FVD is lower-is-better.
  • Results: DIGAN supports long-video synthesis, time interpolation and extrapolation, non-autoregressive generation, diverse motion sampling, and space interpolation and extrapolation.The paper reports 128-frame, 128×128 video training and parallel generation of arbitrary time frames.

2 RELATED WORK

Prior work spans GANs, autoregressive models, and implicit neural representations for image and video generation. INR-based generative models synthesize continuous signal representations and offer interpolation, flexible inference cost, and parallel computation.

  • Image generation: Image generation has advanced through GANs, autoregressive models, and diffusion models, with GANs supporting fast high-resolution inference.The related work emphasizes GAN-based image synthesis as an inspiration for DIGAN.
  • Video generation: Video-generation methods extend image GANs temporally or train autoregressive models over pixels and discretized embeddings.These are the two major prior approaches described in the paper.
  • Implicit neural representations: INRs represent continuous signals with coordinate-to-value mappings and have been applied to static and dynamic 3D scenes.High-frequency sinusoidal activations improve continuous signal modeling.
  • Generative models with INRs: INR-based generative models synthesize signal-specific INR weights or control signals through latent inputs and coordinates.Reported advantages include natural interpolation and extrapolation, anycost inference, and parallel computation.

3 DYNAMICS-AWARE IMPLICIT GENERATIVE ADVERSARIAL NETWORK

DIGAN models videos as continuous space-time signals with INRs instead of discrete RGB grids. Its generator incorporates temporal dynamics into motion features, while its discriminator evaluates image pairs and time differences to detect motion efficiently.

  • Generative modeling with INRs: Discrete video grids scale poorly because their cubic complexity grows with spatial and temporal resolution and they ignore continuous dynamics.This motivates replacing grid-wise outputs with an implicit representation.
  • Generative modeling with INRs: DIGAN generates a fixed-size INR parameter from latent input, synthesizing a continuous video function rather than directly predicting a 3D RGB grid.This reduces the complexity associated with directly generating grid outputs.
  • Generative modeling with INRs: A video is modeled as a continuous function v(x, y, t) mapping space-time coordinates to RGB values.An INR parameterizes this coordinate-to-signal mapping with a neural network.
  • Incorporating temporal dynamics: The DIGAN generator separately handles spatial and temporal structure, using lower temporal frequency to encourage coherent frame changes and an additional motion latent for diversity.The design also applies an extra nonlinear mapping to increase motion expressiveness.
  • Experiments: Table 1 compares IS, FVD, and KVD across UCF-101, Sky, TaiChi, and Kinetics-food, with arrows indicating whether higher or lower values are better.The table also distinguishes train-split from train+test-split evaluation.
  • Incorporating temporal dynamics: The motion discriminator classifies a pair of generated images together with their time difference using an efficient 2D convolutional architecture.INRs can generate frames at arbitrary times without producing the intervening sequence.

4 EXPERIMENTS

DIGAN is evaluated across video-generation benchmarks and extended capabilities, including long-video synthesis, temporal and spatial extrapolation, non-autoregressive prediction, and diverse motion sampling.

  • Main results: DIGAN significantly outperforms prior work across all evaluated datasets, improving UCF-101 FVD from 833 to 577 (+30.7%) over MoCoGAN-HD.Its FID is similar to MoCoGAN-HD, so the reported FVD gains come from better dynamics modeling.
  • Long video generation: DIGAN generates 128-frame videos at 128×128 resolution and trains efficiently because its motion discriminator processes image pairs instead of long sequences.The paper reports this as the first result of this quality at that length.
  • Time interpolation and extrapolation: DIGAN supports temporal interpolation and extrapolation by controlling input coordinates, outperforming MoCoGAN-HD across the considered scenarios.The evaluated videos are interpolated to 64 frames and extrapolated by 16 additional frames from models trained on 16-frame videos.
  • Non-autoregressive generation: DIGAN enables arbitrary-time sampling, including forward and backward prediction, and parallel computation of an entire video rather than autoregressive next-frame generation.Figure 5 demonstrates prediction conditioned on a given frame.
  • Diverse motion sampling: DIGAN produces diverse motions from shared initial frames by changing motion vectors, while preserving some content such as the tree in the Sky examples.The amount of variation depends on the dataset.
  • Spatial and INR interpolation: DIGAN supports spatial upsampling and extrapolation, producing 4× higher-resolution videos without ad-hoc training tricks, and INR-weight interpolation yields semantically meaningful video changes.Interpolated TaiChi weights produce smooth variations such as cloth color changing from white to blue.
  • Ablation studies: Ablations show that smaller temporal frequency, motion vectors, and nonlinear motion mapping each contribute to UCF-101 FVD performance, while their combination saturates.The motion vector and nonlinear mapping have especially large effects when applied individually.

5 CONCLUSION

The paper concludes that DIGAN is an INR-based GAN incorporating video temporal dynamics, with experiments supporting its performance and multiple video-generation properties.

  • 5 CONCLUSION: DIGAN is an INR-based GAN for video generation that incorporates temporal dynamics, and extensive experiments verify its superiority and multiple properties.The authors position the work as guidance for future video-generation and INR research.

ETHICS STATEMENT

The ethics statement highlights misuse risks from generated videos and presents joint generation-and-detection development as a response.

  • ETHICS STATEMENT: Generated videos can enable unethical uses such as fake political propaganda or sexual videos, creating a social problem associated with DeepFakes.The statement cites efforts to detect fake videos as part of the response.
  • ETHICS STATEMENT: DIGAN uses a GAN discriminator that identifies fake videos without observing long sequences, which the authors view as a step toward efficient DeepFake detection.The statement argues that generation and detection techniques should develop in parallel.

A.1 MODEL DETAILS

The implementation uses separate spatial and temporal frequency settings, diverse temporal offsets, factorized INR modulation, and augmentation choices inherited from related architectures.

  • MODEL DETAILS: DIGAN uses spatial frequencies σx = σy = 10 and a smaller temporal frequency σt = 0.25 to encourage temporal coherence.For 128-frame training, σt = 0.5 is used to improve longer-video performance.
  • MODEL DETAILS: The INR uses a progressive MLP with factorized multiplicative modulation, while the motion mapping is a two-layer leaky-ReLU MLP without bias for motion vectors.DiffAug is applied to mitigate overfitting on limited video data.
  • MODEL DETAILS: Implicit GANs sample network parameters whereas standard GANs sample input latents, though the implicit generator resembles StyleGAN2 in practice.The similarity arises because StyleGAN2 injects latents into intermediate layers through mapping and weight modulation.

A.2 DATASET AND EVALUATION DETAILS

The evaluation uses multiple video datasets, standardized preprocessing and metrics, and comparisons against diverse video-generation baselines. Additional procedures cover latent projection, timing measurements, spatial extrapolation, and related prediction or decomposition methods.

  • Datasets and evaluation: UCF-101 contains 13,320 videos, Tai-Chi-HD contains 280 long videos, Sky Time-lapse contains 5,000 videos, and Kinetics-600 contains 495,547 videos.The experiments use the UCF-101, Tai-Chi-HD, Sky Time-lapse, and a food subclass of Kinetics-600.
  • Datasets and evaluation: Inception score uses a C3D network and 10,000 generated videos, while FVD and KVD use I3D features averaged over 10 runs.FVD and KVD are computed from 2,048 sampled real and generated videos per run.
  • Evaluation procedures: Forward and backward prediction projects the given frame into latent space using the StyleGAN2 procedure and 20,000 optimization iterations.The setup is used for the prediction experiments.
  • Evaluation procedures: Generation-time comparisons use the same machine with an Intel Xeon E5-2630 v4 CPU and a Titan XP GPU.Other processes are stopped during measurement.
  • Baselines and related methods: The baseline set includes 3D-convolutional, spatial-temporal, motion-content, autoregressive, progressive, and computationally efficient video-generation models.Examples include VGAN, TGAN, MoCoGAN, MoCoGAN-HD, VideoGPT, DVD-GAN, and TGANv2.

D LATENT DYNAMICS OF MOTION FEATURES

DIGAN represents motion features as latent trajectories whose variation depends on dataset and feature resolution. PCA visualizations show how sampled motion vectors evolve from t = 0 to t = 1.

  • Latent dynamics: Latent motion trajectories are visualized for UCF-101, Sky, and TaiChi using five random motion vectors projected into 2D by PCA.Dot colors progress from blue at t = 0 to red at t = 1.
  • Latent dynamics: Lower-resolution motion features control high-level semantics, whereas higher-resolution features control low-level variations.Resolution identifies where motion features are injected into the progressive generator.
  • Dataset-dependent variation: UCF-101 permits both high-level and low-level motion variation, Sky mainly permits low-level variation, and TaiChi permits little variation.TaiChi motion is largely determined by prior frames and predefined gestures.
  • Prediction: DIGAN additionally provides forward and backward prediction results for video dynamics.The figure marks the given frame with a yellow box.

E.2 INTERMEDIATE SCENE PREDICTION

DIGAN predicts intermediate frames between distant endpoint frames, while additional figures examine forward, backward, extrapolated, and zoomed-out video behavior. The intermediate predictions are quantitatively compared with StyleGAN2 latent interpolation.

  • Additional results: Additional figures show forward and backward prediction, intermediate prediction, 128-frame TaiChi videos, time extrapolation, and zoomed-out samples.Yellow boxes mark given frames or extrapolated frames, while red boxes mark original frames where specified.
  • Intermediate scene prediction: DIGAN predicts intermediate frames between two frames from different time steps, even when the endpoints are somewhat far apart.The authors interpret this as evidence that DIGAN learned ground-truth distribution dynamics.
  • Quantitative comparison: StyleGAN2 and DIGAN intermediate-scene predictions are compared using SSIM in Table 6.StyleGAN2 linearly interpolates between projected initial and final frame latent vectors.

I EFFICIENCY OF DIGAN

DIGAN is evaluated for computation, energy, memory, and inference-time efficiency against MoCoGAN-HD under matched machine settings. Its INR generator and 2D-convolutional discriminator support lower resource use and parallel frame generation.

  • Computational efficiency: DIGAN requires 147.9 GFLOPS for a 16-frame 128×128 video, versus 682.3 GFLOPS for MoCoGAN-HD.The DIGAN generator uses 4.6 times fewer FLOPS.
  • Energy efficiency: 689±24 FVD after 8 GPU days on UCF-101 is reported for DIGAN, versus 838 for MoCoGAN-HD after 16 GPU days.The authors interpret this as at least 2 times greater training energy efficiency.
  • Memory efficiency: DIGAN uses 9.7GB per GPU for batch-size-4 training on 16-frame 128×128 videos, versus 28.0GB for MoCoGAN-HD.This is reported as 2.9 times lower memory use.
  • Inference-time efficiency: DIGAN discriminators process 195.6 video clips per second, versus 60.3 for MoCoGAN-HD under 128-frame, 128×128, batch-size-16 inference.The measured discriminator throughput is 3.2 times larger.
  • Overall efficiency: DIGAN is reported as more computation-efficient (> 4.6 times), training-energy-efficient (> 2 times), memory-efficient (> 2.9 times), and inference-time-efficient (> 3.2 times).The authors attribute these gaps to an INR generator without autoregressive modeling and a 2D rather than 3D convolutional discriminator.
  • Ablation and conditional evaluation: Content vectors improve UCF-101 and TaiChi FVD results because video motions often depend on the content or initial frame.The comparison removes the content vector and uses only the motion vector.
  • Ablation and conditional evaluation: DIGAN is reported to outperform existing baselines in class-conditional UCF-101 video synthesis.The comparison includes DVD-GAN and TSB using IS, FVD, and KVD results.
Loading 2202.10571v1…