Source-linked AI summary
Deep Video Generation, Prediction and Completion of Human Action Sequences
Haoye Cai, Chunyan Bai, Yu-Wing Tai, Chi-Keung Tang
TL;DR
Deep video generation, prediction, and completion are ill-posed and have limited prior results, especially for completion. The paper uses a two-stage human-pose-to-image framework with constrained latent generation and reports plausible videos that surpass current methods quantitatively and visually, while focusing on human action videos.
Problem
Video generation has limited deep-learning results, prediction results are few, and representative deep-learning video completion work is absent because all three problems are severely ill-posed.
Method
A two-stage framework first generates human pose sequences, then transforms them into pixel-level videos; prediction and completion use constrained latent-space generation.
Results
The model generates plausible human action videos from scratch and under constraints, surpassing current methods quantitatively and visually across generation, prediction, and completion.
Takeaways & Limitations
Human pose as an intermediate representation enables one generative model to uniformly address human video generation, prediction, and completion.
Takeaways & Limitations
The authors identify the need for more training data to improve the motion generator and skeleton-to-image transformer's robustness and generality.
Abstract
from arXiv · showhide
Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a general, two-stage deep framework to generate human action videos with no constraints or arbitrary number of constraints, which uniformly address the three problems: video generation given no input frames, video prediction given the first few frames, and video completion given the first and last frames. To make the problem tractable, in the first stage we train a deep generative model that generates a human pose sequence from random noise. In the second stage, a skeleton-to-image network is trained, which is used to generate a human action video given the complete human pose sequence generated in the first stage. By introducing the two-stage strategy, we sidestep the original ill-posed problems while producing for the first time high-quality video generation/prediction/completion results of much longer duration. We present quantitative and qualitative evaluation to show that our two-stage approach outperforms state-of-the-art methods in video generation, prediction and video completion. Our video result demonstration can be viewed at https://iamacewhite.github.io/supp/index.html
1. Introduction
The paper frames human video generation, prediction, and completion as related synthesis problems and proposes one two-stage framework for all three. It models human pose sequences first, then generates pixels from those poses.
- Video generation remains difficult, while prior prediction methods produce only short futures and representative deep-learning video completion results are absent.
- The framework targets generation from scratch, prediction from initial frames, and completion under input-frame constraints within one approach.
- The first stage generates plausible human pose sequences for a specified action category using a conditional generative adversarial network.
- The second stage transfers generated pose sequences into pixel-space videos with a supervised reconstruction network and feature-matching loss.
- Latent-space optimization adapts the framework to prediction and completion by generating videos that satisfy arbitrary available input-frame constraints.
2. Related Work
Related work spans pose-space motion forecasting, predictive and feature-based video representations, direct pixel-space generation, and human pose estimation. This paper instead seeks one framework for constrained completion alongside generation and prediction.
- Prior video forecasting models uncertain human motion in pose space, while other approaches use predictive coding or learned feature representations.
- Existing work generates videos from scratch directly in pixel space, whereas this paper targets generation and completion within the same framework.
- Human pose estimation provides the motion representation used to complete smooth, natural pose sequences before video synthesis.
3. Methodology
The methodology decomposes human video synthesis into pose-sequence generation and skeleton-to-image rendering, then adapts the same framework to constrained prediction and completion. A two-step pose generator and supervised image network make longer, class-conditional human action sequences tractable.
- Overall framework: The framework first generates human skeleton sequences from random noise, then transforms them into pixel-level images.This two-stage decomposition is the paper’s central generation strategy.
- Pose sequence generation: A Single Pose Generator maps latent vectors to individual poses, while a Pose Sequence Generator produces latent sequences for frame-by-frame pose generation.The sequence generator’s outputs are transformed through the single-pose generator.
- Pose sequence generation: Training single-frame poses enforces per-frame human-pose constraints, reducing the difficulty of end-to-end GAN training and enabling longer sequences.A one-hot action-class vector conditions the generators so they can produce different motion types.
- Pose sequence generation: The pose pipeline uses a conditional improved WGAN for single poses and a conditional normal GAN with recurrent sequence modeling for pose trajectories.The sequence discriminator uses a bidirectional LSTM, and latent-vector-shift regularization encourages temporal smoothness.
- Skeleton-to-image transformation: The skeleton-to-image network converts joint locations into heat maps and uses a supervised feed-forward reconstruction model with binary cross-entropy plus feature matching loss.Feature matching is added because binary cross-entropy alone tends to produce blurry images.
- Constrained generation: For prediction and completion, the method optimizes latent variables so generated pose sequences best fit arbitrary constrained frames, then obtains the full sequence from the optimized generator.The constrained-frame set can contain frames at arbitrary indices.
4. Experiments
The experiments use Human3.6m and evaluate synthesized videos with recognition-based realism measures because pixel MSE cannot capture multiple plausible outputs, temporal smoothness, or human-likeness.
- Dataset: Human3.6m contains 896 high-resolution motion videos from seven subjects, with 2D joint-location ground truth.The selected actions are Direction, Greeting, Sitting, Sitting Down, and Walking.
- Dataset: Frames are subsampled to 16 fps to reduce redundancy and encourage larger motion variations.
- Protocol: The skeleton generator trains on five subjects and tests on two, while the skeleton-to-image task tests five selected action classes unseen during training.Pose annotations are normalized by hip centering and limb-length scaling.
- Evaluation: Pixel MSE is unsuitable because multiple visually plausible videos can exist and it does not measure temporal smoothness or human-likeness.
- Baselines: The baselines include VGAN for generation, PredNet, MS-GAN, and PoseVAE for prediction, and conditional VGAN for completion.
5. Results
The proposed model generates plausible, visually compelling human-action videos across generation, prediction, and completion, outperforming competing methods qualitatively and quantitatively.
- Qualitative results: Qualitative comparisons cover generation, prediction, and completion across Direction, Greeting, Sitting, Sitting Down, and Walking.The figure shows the 10th and 40th frames for each method and generated poses for the proposed method.
- Qualitative results: The proposed model produces plausible human motion videos and the most compelling human images, whereas competing methods tend to be noisy or blurry.The paper attributes these competing results to their structural limitations.
- Quantitative results: The proposed videos achieve the highest frame-IS and video-IS among competing methods, although real videos score highest overall.These scores indicate visual features closer to real videos in both spatial and temporal domains.
- Quantitative results: Frame-by-frame Inception Scores for PredNet and MS-GAN tend to decline over time, while PoseVAE remains lower overall than the proposed method.The decline indicates deteriorating image quality over the long run for the compared prediction models.
6. Conclusion and Future Work
The framework uniformly addresses human video generation, prediction, and completion by using constrained generation over human pose sequences. Future work targets robustness, generality, image sharpness, and variable-length output.
- Conclusion: The two-step pose-to-image strategy enables longer human motion videos from scratch and constrained generation for prediction and completion.
- Future work: More training data could improve the motion generator's and skeleton-to-image transformer's robustness and generality.
- Future work: Multi-scale progressive training may increase generated-image sharpness, while an RNN could support variable-length videos.