Source-linked AI summary
The Pose Knows: Video Forecasting by Generating Pose Futures
Jacob Walker, Kenneth Marino, Abhinav Gupta, Martial Hebert
TL;DR
Video forecasting is difficult because direct pixel-space models must capture scene structure, dynamics, and rendering simultaneously, often producing uninterpretable results in unconstrained settings. The paper forecasts human pose futures with a VAE, then conditions a GAN on those poses to generate video pixels, and reports plausible futures and improved performance over contemporary baselines.
Problem
Direct pixel-space forecasting must simultaneously model scene structure, dynamics, and low-level rendering, making unconstrained video forecasting difficult.
Method
The method uses human pose as an intermediate representation, forecasting future pose distributions with a VAE and conditioning a GAN to generate pixel-level videos.
Results
The model generates plausible video futures and outperforms contemporary baselines in quantitative and qualitative evaluation.
Takeaways & Limitations
Structured pose forecasting provides an interpretable intermediate space for modeling plausible human-motion futures before rendering complete videos.
Abstract
from arXiv · showhide
Current approaches in video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is to model the forecasting problem at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene---humans---and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.
1. Introduction
The paper frames video forecasting as uncertain prediction of human activity and proposes separating object structure from pixel appearance. It uses human pose as an intermediate representation, combining a VAE for future motions with a GAN for pixel-level video generation.
- Forecasting challenge: Video forecasting requires inferring active objects, their future deformation and motion, and the resulting pixel changes, all under uncertainty.The paper identifies future human motion as especially uncertain relative to the other forecasting steps.
- Forecasting challenge: Explicitly separating object structure from low-level pixel appearance may benefit forecasting models in unconstrained scenes.End-to-end pixel forecasting must learn structure, dynamics, and rendering factors simultaneously.
- Proposed approach: Human pose provides a tractable intermediate representation for forecasting plausible future human motions without explicit class labels or human-labeled data.The approach uses pose detectors as a free source of supervision and models a distribution over future poses with a VAE.
- Proposed approach: The generated future pose structure conditions a GAN that fills in details and produces the final forecast video in pixel space.This combines VAE-based pose forecasting with GAN-based pixel generation.
2. Related Work
Prior video-forecasting methods model semantic actions, pixels, or pixel motion, but each has limitations in unconstrained or long-horizon settings. The paper instead uses human pose as a low-dimensional, interpretable intermediate representation.
- Activity forecasting: Activity-forecasting research has predicted semantic action classes or other semantic information, while motion forecasting has often targeted specific domains or labeled motion-capture data.This paper seeks approaches requiring few semantic assumptions and usable with unlabeled video.
- Pixel forecasting: Direct pixel-prediction methods can work in constrained domains but often blur on realistic datasets, despite adversarial losses improving unconstrained video generation.Prior approaches include LSTMs, VAEs, PixelCNN, and adversarial models.
- Pixel-motion forecasting: Pixel-motion methods produce interpretable short-term forecasts but cannot model occluded pixels entering the frame or changes in pixel appearance.Their reliance on warping existing pixels limits longer-horizon video prediction.
- Intermediate representations: Human pose is interpretable, low dimensional, and easier to model probabilistically than pixels while remaining a proxy for rendering final video details.Many 2D pose models use only 18 joints, and a pose video can condition pixel-level generation.
3. Methodology
The method separates video forecasting into pose-space motion prediction and pixel-level video generation. Pose-VAE models stochastic future human motion, while Pose-GAN renders predicted skeleton sequences into future frames.
- The pipeline first predicts high-level human movement in pose space, then uses that structure to generate a pixel-level future video.Pose-VAE forecasts pose motion, and Pose-GAN conditions video generation on the resulting skeleton sequence.
- Pose-VAE: The Past Encoder summarizes image features, past poses, and pose velocities into a hidden representation Ht.Its Past Decoder reconstructs past information during training but is discarded during testing.
- Pose-VAE: Because future motion is nondeterministic, the Future Decoder uses a conditional VAE with latent variables to model multiple plausible pose-velocity sequences.During testing, latent variables are sampled from a standard normal distribution and supplied to the Future Decoder.
- Pose-VAE: The predicted pose velocities are integrated with the current pose to reconstruct the future pose sequence step by step.At training time the decoder uses ground-truth poses; at test time each predicted motion updates the next pose.
- Training uses Adam with learning rate 0.001, β1 = 0.9, and staged KL-divergence weights across 80,000 iterations.The KL weight is 0.00025 for 60,000 iterations and 0.0005 for an additional 20,000 iterations.
- Pose-GAN: Pose-GAN takes an input image and generated skeleton video, using volumetric convolutions to produce realistic future videos at the pixel level.Its generator processes 32 input frames of size 80x64x6 and outputs 32 frames of 80x64 pixels; adversarial training uses real and generated videos.
4. Experiments
The experiments evaluate pose and pixel-level forecasting on unconstrained UCF-101, comparing the proposed model with established baselines using Euclidean distance, Inception score, and MMD.
- Evaluation setup: The model is evaluated on UCF-101 in both pose space and video space, using approximately 1,500 one-second clips for testing.Pose detections are obtained with a human pose detector, filtered by confidence, and temporally smoothed.
- Pose evaluation: Pose forecasting is compared against deterministic recurrent, structured recurrent, and feed-forward VAE baselines using Euclidean distance from ground-truth pose velocities.The evaluation uses nearest-sample error because averaging distances across all stochastic predictions can be uninformative.
- Pixel-level evaluation: Inception score evaluates the confidence and diversity of action-class predictions from generated videos using a two-stream RGB-and-flow classifier.The score is computed from the classifier’s conditional label distribution for generated videos.
- Pixel-level evaluation: MMD evaluates how closely generated and real video distributions match using semantic fc7 features rather than raw pixels or optical flow.The implementation uses an unbiased estimator and Gaussian kernels across multiple bandwidths.
- Metrics: MMD is presented alongside Inception score as a more statistically justifiable distribution-comparison metric, with lower scores indicating closer distributions.The paper defines MMD through a supremum over a function class and reports scores in Table 2.
- Metrics: The paper reports Inception scores in Table 1, where higher values are better.The metric is used to compare generated videos with the baseline and real videos.
5. Results
The results show that the pose-based forecasting pipeline produces plausible future motions and improves quantitative pose and video prediction outcomes relative to the reported baselines.
- 5.1. Qualitative Results: The Pose-GAN generates plausible motions conditioned on forecast poses, including forward movement, arm motion, body displacement, and recovery actions.Qualitative examples include skateboarders, jump-ropers, drummers, bowlers, and people performing pull-ups or household actions.
- 5.1. Qualitative Results: Pose-GAN extrapolates pixels while preserving general person shading and color as the body deforms.The model also shows limited ability to inpaint occluded background revealed by human movement.
- 5.2. Quantitative Results: The proposed method outperforms the baselines on Euclidean pose distance even with a small number of samples.Pose-VAE error decreases quickly as samples increase, reflecting stochastic future pose motion.
- 5.2. Quantitative Results: The recurrent Pose-VAE outperforms the feed-forward VAE for pose forecasting.The paper attributes this difference to recurrent models predicting more refined motions sequentially rather than all timesteps simultaneously.
- 5.2. Quantitative Results: Videos generated by the proposed model have a higher Inception score than the compared method, although real videos achieve the highest score.The score reflects how confidently the action classifier detects meaningful classes in generated videos.
- 5.2. Quantitative Results: The generated-video distribution is closer to the real-video distribution under MMD than the distribution generated by the compared method.MMD measures distributional difference, with lower values indicating greater similarity.
6. Conclusion and Future Work
The paper advances pixel-level video prediction by forecasting human pose movements before generating future pixels. It produces plausible futures and outperforms contemporary baselines, while suggesting extensions to structured recurrent models and unlabeled-video applications.
- 6. Conclusion and Future Work: The method predicts human pose movements with a VAE, then generates each future pixel with a GAN.This separates high-level scene dynamics from pixel-level video synthesis.
- 6. Conclusion and Future Work: The model generates a distribution of plausible futures and outperforms contemporary baselines.
- 6. Conclusion and Future Work: A proposed future direction is combining VAEs with structured RNNs to improve performance.
- 6. Conclusion and Future Work: Because the method is unsupervised, the authors suggest applying it to representation learning for action recognition and early action detection using unlabeled video.