Source-linked AI summary
MoCoGAN: Decomposing Motion and Content for Video Generation
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz
TL;DR
Video generation must model both object appearance and physically plausible, varied motion, a challenge made harder by temporal variation. MoCoGAN decomposes latent representations into content and motion, generates frames sequentially with fixed content and recurrent motion, and uses image and video discriminators; experiments report stronger results than comparison methods and controllable synthesis.
Problem
Video generation is difficult because models must learn object appearance, plausible motion, and large temporal variation, while motion artifacts are especially perceptible.
Method
MoCoGAN decomposes an image latent space into content and motion, keeps content fixed across frames, generates motion recurrently, and trains with image and video discriminators.
Results
MoCoGAN achieved an inception score of 12.42 on UCF101 versus 11.85 for TGAN, and an ACD score of 0.201 versus 0.322 for VGAN and 0.305 for TGAN on facial-expression videos.
Takeaways & Limitations
The learned decomposition supports generating videos with the same content and different motion, or different content and the same motion.
Takeaways & Limitations
The framework assumes an image latent space in which video clips are paths, and decomposes that space into content and motion subspaces.
Abstract
from arXiv · showhide
Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-the-art approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion.
1. Introduction
Video generation is challenging because models must capture physically plausible motion, large temporal variation, and perceptible motion quality, not just object appearance. MoCoGAN addresses this by decomposing image latents into content and motion subspaces for more controlled unconditional video generation.
- Video generation requires learning plausible object motion in addition to appearance, making it substantially harder than image generation.
- Incorrect motion models can produce physically impossible movements, while varied execution speeds create many videos with identical appearance.
- Representing videos as trajectories through an image latent space handles variable lengths and different execution speeds more naturally than treating each video as one latent-space point.
- Decomposing image latents into content and motion subspaces separates appearance changes from temporal motion and supports controlled generation.
- MoCoGAN maps noise vectors to videos by sequentially generating frames, using fixed content across a short clip and learned recurrent motion dynamics.
- The framework proposes unconditional video generation and control over content and motion, followed by quantitative and subjective comparisons with existing methods.
2. Generative Adversarial Networks
GANs train a generator to produce images resembling real data and a discriminator to distinguish real from generated images. The video extension replaces image models with spatio-temporal networks that generate fixed-length clips from random vectors.
- A GAN consists of a generator that produces images resembling real images and a discriminator that distinguishes real images from generated ones.
- The image generator maps a random vector z to an image, while the discriminator estimates whether an input image comes from the real or generated distribution.
- Training the image generator and discriminator is formulated as a minimax problem and solved in practice through alternating gradient updates.
- With sufficient model capacity and training, the generated distribution can converge to the real image distribution.
- VGAN extends GANs to video by using spatio-temporal CNNs that map a random vector to a fixed-length video clip.
3. Motion and Content Decomposed GAN
MoCoGAN represents videos as paths through a latent image space decomposed into fixed content and time-varying motion, enabling variable lengths and speed variations. It combines recurrent motion generation, image synthesis, and separate image and video discriminators to learn realistic frames and dynamics.
- Latent-space decomposition: Videos of different lengths use paths of different lengths, while different execution speeds traverse the same latent path at different speeds.This path-based representation avoids requiring every generated video to have a fixed length.
- Latent-space decomposition: MoCoGAN decomposes the image latent space into content and motion subspaces, with content modeling motion-independent appearance and motion modeling motion-dependent appearance.For example, identity can remain fixed while facial muscle configurations change.
- Video generation: The content vector is sampled once and reused across frames, while motion codes vary over time and are combined with content vectors to generate frames.The image generator sequentially maps the resulting latent vectors to images.
- Motion generation: A recurrent neural network maps i.i.d. noise inputs to correlated motion codes, injecting noise at each step to model uncertainty in future motion.The motion codes form paths in the motion subspace that the image generator uses to produce consecutive frames.
- Adversarial learning: MoCoGAN uses image and video discriminators: the former critiques individual-frame appearance, while the latter evaluates fixed-length clips and motion.Image-discriminator feedback improves adversarial-training convergence, while video-discriminator motion feedback reaches the recurrent network.
- Categorical dynamics: Categorical action variables are held fixed for a short video and linked to generated clips through an InfoGAN mutual-information objective.An auxiliary distribution is implemented with a softmax layer on the video discriminator’s final feature layer.
4. Experiments
MoCoGAN is evaluated across synthetic shapes, facial expressions, human actions, UCF101, and Tai-Chi using quantitative comparisons, user studies, qualitative controls, and conditioning experiments. Results indicate improved content consistency, temporal realism, controllable motion/content generation, and image-to-video translation quality.
- Datasets and evaluation: Experiments span shape motion, facial expression, Tai-Chi, human actions, and UCF101 datasets, with quantitative, qualitative, and user-study evaluations.The study also compares MoCoGAN with VGAN, TGAN, MCNET, and C-VGAN in relevant settings.
- Video generation performance: MoCoGAN achieved an ACD score of 0.201 on facial-expression generation, compared with 0.322 for VGAN and 0.305 for TGAN.The lower ACD indicates more consistent generated content across frames.
- Video generation performance: MoCoGAN reached an inception score of 12.42 on UCF101, versus 11.85 for TGAN with WGAN training and Singular Value Clipping.The comparison used the same MoCoGAN model as the other experiments, while the TGAN score used additional training procedures.
- Categorical video generation: Fixing content while varying motion generated the same object with different motions, while fixing motion and varying content generated different objects performing the same motion.A qualitative experiment sampled two content codes and seven motion codes, producing 14 videos arranged by shared codes.
- Categorical video generation: Models with image discriminators performed better across categorical-generation metrics, and the zA →RM conditioning scheme outperformed direct conditioning of the image generator.The experiment also evaluated a MoCoGAN variant trained without the image discriminator.
- Categorical video generation: With dC + dM = 60, larger dC produced smaller ACD, while larger dM caused poor facial quality and lower motion-control scores; experiments therefore used dC = 50 and dM = 10.The motion-control classifier could only guess expressions reliably when generated faces remained recognizable.
- Image-to-video translation: On Tai-Chi image-to-video translation, a MoCoGAN variant generated more realistic videos than C-VGAN and MCNET in a user study.MoCoGAN and C-VGAN required one input frame, whereas MCNET used four frames to predict a video.
5. Conclusion
MoCoGAN decomposes video generation into content and motion, learning this separation unsupervisedly while generating frames sequentially. Experiments support its effectiveness and enable controlled synthesis across identities and expressions.
- MoCoGAN generates video clips sequentially, using random vectors whose content and motion parts control distinct aspects of each frame.The content subspace uses a Gaussian distribution, while the motion subspace uses a recurrent neural network.
- Given sufficient video data, MoCoGAN automatically learns to disentangle motion from content without supervision.
- For facial-expression videos, the framework can synthesize one person performing different expressions or fix an expression while varying identity.
- Experimental evaluation supports MoCoGAN’s superiority over contemporary video-generation and next-frame-prediction methods.
A. Network Architecture
The experiments use specified convolutional, 3D convolutional, normalization, activation, and recurrent layers across the image and video networks. Table 6 organizes the architectures for the generators and discriminators.
- Table 6 specifies architectures for the image generator G_I, image discriminator D_I, and video discriminator D_V used in the experiments.
- The architecture notation records output channels, kernel size, stride, and padding for convolutional and 3D convolutional layers.CONV and CONV3D layers use equal kernel size, padding, and stride across dimensions.
- Batch-normalization layers are followed by LeakyReLU nonlinearities in the implementation.
- The recurrent motion component consists of a single GRU.
B. Additional Qualitative Results
Additional qualitative results demonstrate controlled generation by fixing content while varying action and motion variables. The generated videos preserve identity and can extend beyond the training clips’ lengths.
- Figures 7 and 8 present categorical facial-expression and human-action video-generation results, respectively.
- Each three-row group fixes the content vector z_C while randomizing the action vector z_A and motion vectors z(t).
- The generated videos keep identity fixed throughout each video.
- Generated videos are longer than the training examples, indicating generalization along the time dimension.