PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning 3D Human Dynamics from Video

Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, Jitendra Malik

arXiv:1812.01601v4cs.CV

TL;DR

Recovering 3D human dynamics is difficult because 3D supervision is scarce and single-image motion is ambiguous. HMMR learns temporal dynamics from video, transfers them to static images, and reports improved performance from Internet-scale pseudo-2D supervision, including state-of-the-art 3D prediction without fine-tuning.

  • Problem

    Large-scale 3D supervision is costly and unavailable for most Internet videos, while predicting motion from a single image remains inherently ambiguous.

  • Method

    HMMR uses a temporal encoder over image features to predict 3D pose, shape, and nearby motion, then trains a hallucinator for single-image inference.

  • Results

    Adding more pseudo-ground-truth 2D-labeled videos monotonically improves performance, while HMMR achieves state-of-the-art 3D error on 3DPW without fine-tuning.

  • Takeaways & Limitations

    Temporal representations enable smooth 3D video predictions and transfer learned dynamics to static images, while unlabeled video improves both 2D and 3D prediction.

  • Takeaways & Limitations

    Static-image dynamics prediction works well for ballistic motion but predicts no change when the motion direction is ambiguous.

Abstract

from arXiv · show

From an image of a person in action, we can easily guess the 3D motion of the person in the immediate past and future. This is because we have a mental model of 3D human dynamics that we have acquired from observing visual sequences of humans in motion. We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features. At test time, from video, the learned temporal representation give rise to smooth 3D mesh predictions. From a single image, our model can recover the current 3D mesh as well as its 3D past and future motion. Our approach is designed so it can learn from videos with 2D pose annotations in a semi-supervised manner. Though annotated data is always limited, there are millions of videos uploaded daily on the Internet. In this work, we harvest this Internet-scale source of unlabeled data by training our model on unlabeled video with pseudo-ground truth 2D pose obtained from an off-the-shelf 2D pose detector. Our experiments show that adding more videos with pseudo-ground truth 2D pose monotonically improves 3D prediction performance. We evaluate our model, Human Mesh and Motion Recovery (HMMR), on the recent challenging dataset of 3D Poses in the Wild and obtain state-of-the-art performance on the 3D prediction task without any fine-tuning. The project website with video, code, and data can be found at https://akanazawa.github.io/human_dynamics/.

1. Introduction

HMMR learns 3D human dynamics from video and transfers that representation to single images. It produces smooth video-based 3D predictions, predicts nearby past and future motion from one image, and improves with pseudo-labeled Internet video.

  • Method: HMMR encodes temporal image features to learn 3D human dynamics and predicts current pose, shape, and nearby pose changes.A hallucinator transfers the learned temporal representation to a single image using self-supervised training.
  • Results: 56% acceleration error reduction versus a comparable single-view approach demonstrates smoother video predictions.The method also achieves state-of-the-art 3D error on 3DPW without fine-tuning.
  • Motivation: Limited large-scale 3D supervision motivates training with 2D pose annotations, reprojection loss, and an adversarial pose prior.Controlled 3D datasets do not capture the complexity of real-world video.
  • Motivation: Millions of unlabeled Internet videos provide a scalable training source through pseudo-ground-truth 2D poses from an off-the-shelf detector.The approach is designed for semi-supervised learning with varied supervision.
  • Contribution: The learned representation supports smooth 3D mesh prediction from video and predicts current mesh plus nearby past and future pose changes from a static image.The training set contains no ground-truth 3D batting-motion sequences.

2. Related Work

Prior work addresses 3D pose and shape from single images and video, while HMMR focuses on monocular video and predicts 3D motion from 2D inputs. It also differs from single-image future-prediction methods by operating in the 3D domain.

  • 3D pose and shape from a single image: Single-image methods fit statistical body models using silhouettes, 2D joints, or learned image cues to estimate 3D pose and shape.Recent approaches integrate the SMPL model into deep-learning frameworks.
  • 3D pose and shape from video: Most video methods use multi-view capture or smooth single-view reconstructions afterward through constrained optimization.Some approaches pre-capture actors to recover clothing shape and texture from silhouettes.

3. Approach

The framework learns 3D human dynamics from temporal image features, predicts current and nearby-future or past body states, and transfers this representation to static images.

  • Temporal representation: The framework encodes a temporal window of image features into a representation of 3D human dynamics.A 1D fully convolutional temporal encoder produces the representation from per-frame visual features.
  • Body representation and supervision: The body is represented with SMPL shape, pose, and camera parameters, with projected joints used for image-plane supervision.SMPL produces a mesh from shape and pose, while a weak-perspective camera projects 3D joints into the image.
  • Body representation and supervision: Training combines available 2D and 3D losses with adversarial and shape priors, plus a constant-shape constraint across each sequence.The temporal objective includes frame losses and a shape-consistency term.
  • Temporal representation: The model predicts the current 3D body and pose changes at nearby past and future time steps.Delta predictors estimate changes in 3D pose while the predicted current shape is reused across neighboring frames.
  • Static-image transfer: A hallucinator maps a single-image feature to the temporal representation and trains it through shared 3D and dynamics predictors.The hallucinated representation is matched to the temporal encoder’s representation and optimized to predict both current body state and dynamics.

4. Learning from Unlabeled Video

The framework expands supervision with Internet videos carrying detector-generated 2D pseudo-labels, alongside smaller fully or manually annotated datasets.

  • Data sources: Table 1 organizes training data into full 2D and 3D supervision, ground-truth 2D supervision, and pseudo-ground-truth 2D supervision.The pseudo-ground-truth datasets differ substantially in scale from the annotated datasets.
  • Data sources: Millions of Internet videos provide a large potential source of unlabeled human-motion data beyond limited manual annotations.The paper notes that 300 hours of video are uploaded to YouTube every minute.
  • Pseudo-labeled datasets: The authors curate VLOGpeople and InstaVariety using OpenPose-generated pseudo-ground-truth 2D poses from Internet videos.InstaVariety targets diverse activities through videos collected with hashtags including instruction, swimming, and dancing.

5. Experimental Setup

Experiments use pretrained image features, temporal convolutional layers, multiple annotated video sources, and 3DPW as an unseen in-the-wild test set.

  • Architecture: The image feature extractor is a ResNet-50 pretrained for single-view 3D human pose and shape prediction.Average-pooled features have dimension 2048.
  • Architecture: The temporal encoder uses three residual 1D convolutional layers with an effective receptive field of 13 frames.Features are precomputed to reduce memory use during video training.
  • Training datasets: Human3.6M supplies full 3D supervision, while Penn Action and NBA provide in-the-wild videos with 2D pose annotations.Human3.6M training and testing follow the stated subject split; Penn Action and NBA have separate training, validation, and test partitions.
  • Evaluation: The model is evaluated on 3DPW without training on that dataset, using its indoor and outdoor sequences with IMU-based 3D annotations.Frames with fewer than six visible keypoints are skipped.
  • Evaluation: The experimental setup assumes a temporal tube of person detections and rescales all image crops to 224x224.Ground-truth boxes are used when available; otherwise OpenPose supplies temporally smooth detection tubes.

6. Experiments

Experiments show that temporal context and auxiliary dynamics prediction improve smoothness and 2D estimation, while pseudo-labeled Internet video improves 3D and 2D performance. HMMR achieves state-of-the-art 3DPW performance and predicts dynamics from single images, though static-image dynamics remain ambiguous.

  • Local vs Temporal Context: Temporal context substantially improves smoothness, while auxiliary dynamics prediction is necessary to learn human dynamics effectively.The ablation finds temporal context alone insufficient; the dynamics loss is important for forcing the network to learn human dynamics.
  • Comparison to state-of-the-art approaches: HMMR achieves state-of-the-art 3D performance on 3DPW without fine-tuning and outperforms TP-Net on in-the-wild video.The comparison includes feed-forward methods and excludes post-optimization smoothing, which is described as complementary.
  • Qualitative video results: Figure 3 presents input crops, predicted meshes, and alternate mesh views for Penn Action, NBA, and VLOG sequences.The caption reports smooth, temporally consistent predictions.
  • Predicting 3D dynamics: Figure 4 evaluates single-image prediction by comparing ground-truth past and future frames with predicted meshes across past, present, and future.The predictions are shown from two mesh viewpoints.
  • Training on pseudo-ground truth 2D pose: Adding more pseudo-ground-truth 2D-labeled videos monotonically improves performance, including a 9% reduction in 3D pose error and an 8% increase in 2D pose accuracy on 3DPW.The largest dataset, InstaVariety, produces these improvements; 2D-dataset gains are small or unchanged.
  • Predicting 3D dynamics: Static-image dynamics prediction is challenging because pose ambiguities and motion stochasticity make direction uncertain.The model performs well for ballistic motions but predicts no change when upward versus downward motion is unclear.

7. Discussion

The discussion presents HMMR as an end-to-end model for smooth video-based 3D prediction and single-image dynamics hallucination. It also identifies occlusions and multi-person interactions as important unresolved challenges.

  • Discussion: HMMR jointly supports smooth 3D prediction from video and 3D dynamics hallucination from single images.The model learns current 3D body state and nearby pose changes, and can use semi-supervised 2D pose annotations.
  • Discussion: Occlusions and interactions between multiple people remain upcoming challenges in recovering 3D human bodies from video.The paper describes its results as promising while stating that much remains to be done.

8. Appendix

The appendix details HMMR’s temporal encoder, regressors, evaluations, and failure modes. It also examines how temporal context interacts with pseudo-ground-truth 2D supervision and resolves image ambiguities.

  • Architecture: The temporal encoder uses 1D convolutions and residual blocks, with a 13-frame effective receptive field for modeling motion context.Each convolution has temporal kernel size 3; the encoder output retains the image-feature dimension.
  • Architecture: The hallucinator maps a static image feature to the learned temporal representation using two fully connected layers with a skip connection.This transfers temporal dynamics knowledge to single-image prediction.
  • Architecture: The 3D regressor iteratively updates an 85D mesh parameter vector three times, while dynamics predictors estimate 72D pose changes from the current pose.The initial mesh estimate is the learned mean parameter vector; dynamics predictors initialize from the current-frame pose and use separate weights.
  • Additional Ablations and Evaluations: Table 6 evaluates baselines and ablations on 2D and 3D keypoints and 3D mesh, including a variant without the constant shape loss.The evaluation compares models trained on Human3.6M, with only the proposed models additionally trained on Penn Action.
  • Failure Modes: Temporal context helps resolve static-image ambiguities, but predictions remain difficult under person-to-person occlusion, fast motion, motion blur, and rare poses.When the current-frame pose is poor, dynamics predictions also become incorrect because they are initialized from that pose.
  • Additional Ablations and Evaluations: Additional pseudo-ground-truth 2D video benefits both local and temporal models, with the relative improvement significantly higher for the temporal model.The appendix suggests temporal context acts as a regularizer against outlier pseudo-labels.
Loading 1812.01601v4…