PaperlayerView exact revision on arXiv

Source-linked AI summary

Pose Guided Human Video Generation

Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, Dahua Lin

arXiv:1807.11152v1cs.CV

TL;DR

Existing human video-generation methods lack an interpretable representation for explicitly controlling video dynamics. This paper uses pose-guided two-stage generation to separate motion prediction from appearance generation, with experiments on human action and face datasets demonstrating superiority over prior methods.

  • Problem

    Existing human video-generation methods lack an interpretable representation for explicitly controlling video dynamics.

  • Method

    A two-stage pose-guided method separately predicts pose dynamics with PSGAN and generates coherent appearances with SCGAN under semantic consistency.

  • Results

    Experiments on human action and face datasets demonstrate superiority over prior methods and explicit manipulation of human pose and appearance.

  • Takeaways & Limitations

    Pose provides an interpretable basis for generating human videos while separately controlling motion and appearance.

  • Takeaways & Limitations

    At inference, ground-truth poses are unavailable, requiring training to handle abnormal predicted poses for generalization to testing scenarios.

Abstract

from arXiv · show

Due to the emergence of Generative Adversarial Networks, video synthesis has witnessed exceptional breakthroughs. However, existing methods lack a proper representation to explicitly control the dynamics in videos. Human pose, on the other hand, can represent motion patterns intrinsically and interpretably, and impose the geometric constraints regardless of appearance. In this paper, we propose a pose guided method to synthesize human videos in a disentangled way: plausible motion prediction and coherent appearance generation. In the first stage, a Pose Sequence Generative Adversarial Network (PSGAN) learns in an adversarial manner to yield pose sequences conditioned on the class label. In the second stage, a Semantic Consistent Generative Adversarial Network (SCGAN) generates video frames from the poses while preserving coherent appearances in the input image. By enforcing semantic consistency between the generated and ground-truth poses at a high feature level, our SCGAN is robust to noisy or abnormal poses. Extensive experiments on both human action and human face datasets manifest the superiority of the proposed method over other state-of-the-arts.

1 Introduction

Human video generation must handle temporal smoothness and uncertain human motion, motivating interpretable pose-based control. The proposed two-stage method separates plausible motion prediction from coherent appearance generation using PSGAN and SCGAN.

  • Human video generation is challenging because it must satisfy temporal smoothness constraints while accounting for uncertain human motions.
  • Human skeletons provide an interpretable motion representation that captures geometric body configurations independently of appearance.
  • The method separates human video synthesis into plausible motion prediction and coherent appearance generation.
  • PSGAN predicts plausible pose sequences conditioned on action labels, explicitly modeling diverse human motion patterns.
  • SCGAN synthesizes coherent video frames from generated poses and an input image while handling abnormal poses.
  • Qualitative and quantitative experiments on human action and facial expression datasets demonstrate the method’s superiority over existing state-of-the-art methods.

2 Related work

Related work progresses from deep generative models for natural-image synthesis to conditional generation, while video generation additionally requires modeling foreground dynamics and enforcing temporal smoothness. Prior video methods use spatio-temporal architectures, disentangle motion from content, or impose flow-based consistency.

  • Image generation: Deep generative models synthesize natural images using Variational Auto-Encoders and Generative Adversarial Networks, with follow-up work improving GAN training and image quality.Convolutional networks have also been proposed to stabilize GAN training against model collapse caused by uncertainty in data distributions.
  • Conditional generation: Conditional image-generation methods control synthesis using class labels, attributes, poses, text descriptions, or mappings between domains.Examples include conditional GANs, pose-guided person generation, StackGAN, and domain-translation approaches.
  • Video generation: Video generation is more challenging than image synthesis because it must model foreground dynamics while satisfying temporal smoothness constraints.The field gained momentum with powerful GPUs and deep convolutional networks.
  • Video generation: Prior video-generation methods model foreground scene dynamics with spatio-temporal convolutions, decompose motion and content, or enforce consistency with pixel-wise video flows.These approaches include GAN-based spatio-temporal architectures, motion-content decomposition, and dual learning for future-frame prediction.

3 Methodology · 3.1 Framework Overview · 3.2 Plausible Motion Prediction

The framework disentangles human video synthesis into plausible motion prediction and coherent appearance generation. Its first stage uses PSGAN to generate temporally smooth, class-conditioned pose sequences that model rich motion patterns from an input pose.

  • 3.1 Framework Overview: The method synthesizes human action or facial-expression videos from an input image while explicitly controlling motion and maintaining appearance coherence.The target action class may include Skip, TaiChi, or Jump, and the generated video starts with the input image.
  • 3.1 Framework Overview: The two-stage framework uses human skeletons or poses to represent motion dynamics, then generates appearance-conditioned frames from the predicted poses.PSGAN performs pose-sequence generation, while SCGAN focuses on appearance modeling and semantic consistency.
  • 3.1 Framework Overview: PSGAN generates a temporally smooth pose sequence conditioned on the input pose and target action class.The input pose is extracted from the input image before sequence generation.
  • 3.2 Plausible Motion Prediction: Because pose-sequence prediction is an ill-posed one-to-many problem, PSGAN learns plausible motion patterns rather than precise pose coordinates.The model learns from example pose sequences in the training set to represent rich motion possibilities.
  • 3.2 Plausible Motion Prediction: PSGAN receives the input pose and target action label, then synthesizes pose sequences through an encoder-decoder architecture.After the last residual block, the feature map is extended with a time dimension and processed by fractionally-strided spatio-temporal convolution layers.
  • 3.2 Plausible Motion Prediction: The initial pose is estimated as 18 key-point coordinates and encoded as 18 heatmaps, each marking a radius of 4 pixels around its key point.This representation forms a C = 18 channel tensor and avoids learning a mapping from key points to body-part locations.

Pose extraction. · Pose Sequence GAN. · LSTM embedding.

The method encodes an initial pose and target action label into shared latent features, decodes a pose sequence, and models its temporal structure with an LSTM. PSGAN’s objective functions are formulated explicitly, while experiments assess whether LSTM embedding stabilizes training and improves generated pose quality.

  • Pose extraction.: PSGAN synthesizes a meaningful pose sequence from an initial pose and target action label.The method conditions generation jointly on the starting pose and action category.
  • Pose extraction.: The input pose is represented as a C × W × H tensor and processed through several convolutional layers.This convolutional encoding forms the pose representation used by the encoder-decoder architecture.
  • Pose extraction.: The target action label is represented as an n-dimensional one-hot vector, where n is the number of action types.The pose and label signals are combined after residual processing.
  • Pose extraction.: After residual blocks, the pose and action-label signals are embedded into common feature maps in latent space.This shared embedding supports conditional pose-sequence generation.
  • Pose Sequence GAN.: The decoder outputs a C×T×W×H tensor, interpreted as T pose tensors of size C × W × H for temporal modeling.Each tensor is passed to a one-layer LSTM module.
  • Pose Sequence GAN.: The one-layer LSTM performs temporal pose modeling and is reported to stabilize training and improve generated pose-sequence quality.These effects are evaluated in the experiments.
  • LSTM embedding.: PSGAN’s objective functions are formulated explicitly as part of the model specification.The passage introduces the objective-function formulation without giving its equations here.

Objective function.

The objective combines adversarial and reconstruction losses to train pose generation, using key-point masks to address heatmap sparsity and imbalance. This design also targets abnormal poses caused by weak or missing key-point responses.

  • Loss design: The objective combines adversarial losses for the discriminator and generator with a reconstruction loss that stabilizes training.The discriminator distinguishes generated pose sequences G(p,a) from ground-truth P.
  • Masked reconstruction: The reconstruction term uses a ground-truth pose mask to focus weighted L1 computation on small regions around key points.M addresses heatmap sparsity and key-point imbalance; when α = 0, the term becomes unweighted L1 loss.
  • Abnormal poses: The objective is motivated by abnormal poses in which key points appear too large, too small, or missing because of weak responses.Such cases require the network to capture the semantic implication of human pose rather than rely only on small numerical responses.

Abnormal poses. · 3.3 Coherent Appearance Generation

The second stage uses SCGAN to generate coherent video frames from an input image and pose sequence while remaining robust to noisy or abnormal poses. Unlike prior two-stage pose-to-image generation, SCGAN produces all video frames over time in one step and enforces high-level semantic consistency.

  • Abnormal poses.: The abnormal-pose examples vary key-point responses and include missing key points alongside the ground-truth pose.Figure 3 illustrates larger or smaller key-point responses and missing key points as abnormal-pose conditions.
  • 3.3 Coherent Appearance Generation: SCGAN uses an encoder-decoder generator conditioned on human poses and the original input image.Its discriminators distinguish generated images from real images and identify which pose generated a frame.
  • 3.3 Coherent Appearance Generation: SCGAN synthesizes coherent video frames conditioned on the input image and the pose sequence predicted in stage one.The method targets appearance generation after pose-sequence prediction.
  • Abnormal poses.: Noisy or abnormal poses can destabilize or even break methods that directly generate images from pose inputs.The paper identifies this as the motivation for introducing SCGAN.
  • 3.3 Coherent Appearance Generation: SCGAN imposes semantic consistency between generated and ground-truth poses at a high feature level.This design is intended to make image generation robust to pose noise and abnormalities.
  • 3.3 Coherent Appearance Generation: Unlike prior work that synthesizes a coarse image before refining it, SCGAN generates video results in one step for all frames over time.The process uses the input image at time t0 and a target pose at time ti.

Conditional generation. · Semantic consistency.

The method addresses noisy pose predictions during inference by training the second-stage generator with ground-truth pose guidance. Semantic consistency is enforced between predicted- and ground-truth-pose streams through shared high-level features, adversarial discrimination, and reconstruction.

  • Conditional generation.: During inference, noisy or abnormal pose predictions from PSGAN can affect second-stage image generation.
  • Semantic consistency.: Because ground-truth poses are unavailable at inference, training must teach the generator to handle abnormal poses for testing generalization.
  • Semantic consistency.: Abnormal poses often arise from small differences in corresponding keypoint responses that produce small losses and back-propagation gradients.
  • Semantic consistency.: The method does not require maximizing PSGAN pose accuracy because small prediction errors should not change global pose interpretation.
  • Semantic consistency.: SCGAN shares the final convolutional-layer weights of two pose encoders to impose semantic consistency in high-level feature space.
  • Semantic consistency.: SCGAN generates frames from both predicted and ground-truth poses, improving tolerance to pose noise.
  • Semantic consistency.: The final objective generates frames from two pose streams while maintaining their semantic consistency adversarially.
  • Semantic consistency.: G1 conditions on input image It0 and PSGAN pose ˆPti, whereas G2 uses the corresponding ground-truth pose to generate Iti|gt.

Full objective function.

The full training objective uses three discriminators to distinguish real and fake images under different pose conditions and to identify the conditioning pose. For ground-truth-pose images, gradients from Dwhich do not propagate to G2.

  • Full objective function.: Three discriminators distinguish real and fake images using predicted or ground-truth poses, while Dwhich judges which pose conditions the generated image.D1 and D2 operate with predicted and ground-truth poses, respectively; Dwhich identifies the conditioning pose.
  • Full objective function.: The Dwhich gradient is not propagated to G2 when the generated image is guided by a ground-truth pose.Ground-truth-pose-guided images are real for Dwhich.

3.4 Implementation Details

The models use convolutional and interpolation-based architectures implemented in PyTorch, with stage-specific upsampling and batch sizes. Training uses ADAM with fixed reconstruction-loss weights and dataset-specific scaling factors.

  • Network architecture: All generators use four stride-2 convolutional layers with kernel size 4 for downsampling, while stage-specific layers perform upsampling.Stage one uses stride-2 transposed convolutions; stage two replaces them with normal convolutions and interpolation.
  • Optimization: The models are implemented in PyTorch and trained with ADAM at a learning rate of 0.001, using batch sizes of 64 and 128 in stages one and two, respectively.All reconstruction loss weights are set to 10.
  • Optimization: The scaling factor α is set to 10 for the human action dataset and 20 for the facial expression dataset.The scaling factor is selected from 0 to 100 and affects convergence speed only.

4 Experiments

Experiments on human action and facial datasets show that the proposed pose generator produces smooth, plausible motions and that the pose-guided video generator improves visual realism, user preference, and controllable separation of pose and appearance. Ablations further show that adversarial and temporal pose modeling, together with semantic consistency, are important for generation quality.

  • Pose generation: Generated pose sequences change smoothly and typically under each action scenario.Pose heatmap post-processing is used only for visualization.
  • Pose generation: Adversarial training and GRU or LSTM temporal modeling improve the quality of generated pose sequences over the PSGAN variants.The compared variants use L1 loss instead of adversarial loss, omit LSTM, or use the full PSGAN with GRU.
  • Pose generation: The full PSGAN is most often ranked first by users, while PSGAN without LSTM and PSGAN-L1 rank lower.The user study shows the importance of temporal and adversarial pose modeling.
  • Video generation: SCGAN generates sharper and more realistic video frames than VGAN and MoCoGAN, especially for the complex-motion TaiChi action.The visual comparison also finds the proposed method better or equally good for Wave.
  • Video generation: Users choose the proposed method as the winner most of the time over MoCoGAN and VGAN in paired video comparisons.Each method generates 50 videos, which users compare in random pairs based on realism.
  • Controllability and ablation: Controlled tests successfully generate different actions for fixed appearances and different humans for fixed actions across body and facial-expression cases.These results demonstrate explicit pose modeling, generalization, and separate modeling of pose and appearance.
  • Controllability and ablation: Semantic consistency makes SCGAN sharper and more photo-realistic than SCGAN-gen and helps it remain close to SCGAN-gt while mitigating abnormal-pose effects.The full model improves facial mouth and human waist regions, whereas generated-pose-only guidance can be noisy; it also produces varied motion beyond the static baseline.

5 Conclusion and Future Work

The paper presents a two-stage pose-guided method for disentangled human video generation, separating motion representation from appearance generation. Results on human action and face datasets support its effectiveness, while the current method remains limited to cropped human or face images.

  • Conclusion: The proposed method generates human videos in a disentangled way through a two-stage, pose-guided framework.The paper emphasizes human pose as important for video generation.
  • Conclusion: The method supports explicit manipulation of human pose and appearance.
  • Conclusion: Quantitative and qualitative results on human action and face datasets demonstrate the method’s superiority.
  • Future Work: The current method is limited to cropped human or face images.
Loading 1807.11152v1…