PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting 3D Human Dynamics from Video

Jason Y. Zhang, Panna Felsen, Angjoo Kanazawa, Jitendra Malik

arXiv:1908.04781v2cs.CV

TL;DR

The paper addresses predicting future 3D human motion from past video, a setting distinct from predicting 2D futures or using past 3D motion. It predicts future motion autoregressively in a latent representation and reports demonstrations on Human3.6M and Penn Action. The approach is limited by short training videos, which constrain prediction length.

  • Problem

    The paper studies how to predict future 3D human motion from past image sequences, unlike prior approaches focused on 2D futures or past 3D motion.

    Source: §1 · p. 1

  • Method

    A causal encoder maps past video into a latent representation, where an autoregressive model predicts future representations that can be read out as 3D meshes.

    Source: §1 · p. 1, §3.3 · p. 5

  • Results

    The approach is demonstrated on Human3.6M and the in-the-wild Penn Action dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method can be trained on in-the-wild videos without 3D annotations, using 2D reprojection loss and an adversarial prior.

    Source: §1 · p. 1, §3.2 · p. 4

    Short in-the-wild training videos constrain the past context and prediction length; Penn Action videos have a median length of 56 frames.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Given a video of a person in action, we can easily guess the 3D future motion of the person. In this work, we present perhaps the first approach for predicting a future 3D mesh model sequence of a person from past video input. We do this for periodic motions such as walking and also actions like bowling and squatting seen in sports or workout videos. While there has been a surge of future prediction problems in computer vision, most approaches predict 3D future from 3D past or 2D future from 2D past inputs. In this work, we focus on the problem of predicting 3D future motion from past image sequences, which has a plethora of practical applications in autonomous systems that must operate safely around people from visual inputs. Inspired by the success of autoregressive models in language modeling tasks, we learn an intermediate latent space on which we predict the future. This effectively facilitates autoregressive predictions when the input differs from the output domain. Our approach can be trained on video sequences obtained in-the-wild without 3D ground truth labels. The project website with videos can be found at https://jasonyzhang.com/phd.

1. Introduction

The paper introduces autoregressive prediction of future 3D human meshes from past video, addressing a setting distinct from predicting 2D futures or 3D motion from 3D inputs. Its framework predicts in a shared latent space and can be trained on in-the-wild video without 3D annotations.

1. Introduction

  • The model can be trained on in-the-wild videos without ground-truth 3D annotations.The introduction describes this as a more abundant and natural information source than motion sequences recorded in a motion-capture studio.Source: §1 · pp. 1–2
  • The paper proposes predicting future 3D human motion from video, a problem prior approaches had not explored to the authors’ knowledge.Earlier future-prediction work generally predicted 2D outputs from video, while 3D human-motion prediction used past 3D motion-capture sequences.Source: §1 · p. 1
  • The framework learns a shared latent representation so an autoregressive model can predict when the input is video and the output is a 3D mesh.It uses a causal temporal encoder to capture past context and predicts future latent representations successively.Source: §1 · p. 2
  • The approach is demonstrated on Human3.6M and the in-the-wild Penn Action dataset.The stated applications include autonomous systems that must operate safely around people using visual inputs.Source: §1 · pp. 1–2

2. Related Work

Prior work predicts future visual information from images or video, and predicts 3D human motion from past 3D sequences. This paper differs by predicting future 3D human motion from video inputs.

Visual Prediction

  • Visual prediction from video has covered activity classes, trajectories, pixels, flow, and 2D human pose, while the cited approaches predict in 2D or categories.Some cited methods start from a single image and predict object dynamics, trajectories, flow, difference images, action categories, or image representations.Source: §2 · pp. 2–3

3D to 3D Human Motion Prediction

  • Deep-learning approaches to 3D human-motion prediction use past 3D skeleton sequences, whereas this work uses past video to predict future 3D motion.Earlier methods also modeled motion with approaches including hidden Markov models, linear dynamical systems, and Gaussian process latent variable models.Source: §2 · p. 3

3D Future Prediction from Single Image

  • A related single-image approach predicts 2D future poses from which 3D poses can be estimated, while earlier mesh prediction produced one future mesh per image.The earlier mesh approach required a separate future predictor for every time step; this work proposes one autoregressive model for successive predictions.Source: §2 · p. 3

3. Approach

The approach predicts future human 3D meshes from past video by learning a causal motion representation and forecasting in that latent space. It uses SMPL pose and shape parameters, with training procedures designed for autoregressive prediction from limited in-the-wild videos.

3.1. 3D Mesh Representation

  • The output is an SMPL mesh represented by 82 pose and shape parameters, with 6,890 vertices.Pose has 72 parameters and shape has 10; SMPL also supports extracting joints and projecting them into 2D with camera parameters.Source: §3.1 · p. 3

3.2. Causal Model of Human Motion

  • The causal encoder represents motion using only current and past frames, supporting video-based 3D mesh estimation with 3D or 2D supervision.When 3D annotations are unavailable, training can use visible-keypoint reprojection loss alongside pose and shape priors.Source: §3.2 · p. 4
  • The causal temporal encoder performs comparably to a non-causal model despite using only past context.Both use a 13-frame receptive field; the causal model uses edge padding, while [27] uses zero padding.Source: §3.2 · p. 4, Table 1 · p. 4

3.3. Autoregressive prediction

  • An autoregressive predictor forecasts future motion in the learned latent representation, transitioning from past image conditioning to previously generated predictions.A distillation loss trains predicted movie strips to approximate the encoder’s strips; readout losses also apply to the predicted meshes.Source: §3.3 · pp. 4–5
  • A pose-space autoregressive variant predicts directly in pose space to compare with methods that predict 3D motion from 3D input.Source: §3.3 · p. 5

3.4. Training Procedures

  • Training proceeds in two stages, then uses a curriculum that gradually increases consecutive self-generated inputs from 1 to 25 steps.The setting is constrained by short in-the-wild training videos; Penn Action videos have a median length of 56 frames, and 25 predictions approached the upper bound given 13-frame receptive fields.Source: §3.4 · p. 5

4. Evaluation

The evaluation combines quantitative comparisons on Human3.6M and Penn Action with qualitative tests across Human3.6M, Penn Action, and NBA. Latent-space autoregression outperforms pose-space prediction and baselines, while qualitative analyses examine motion-type predictions and frames associated with accuracy gains.

4.1. Experimental Setup

  • Training combines four datasets with different supervision levels, then evaluates on Human3.6M and Penn Action; the temporal encoder sees 13 frames (0.5 seconds).The datasets are combined in an action- and dataset-agnostic manner.Source: §4.1 · p. 5

4.2. Quantitative Evaluation

  • Latent-space autoregression significantly outperforms pose-space prediction and both Constant and Nearest Neighbors baselines under DTW.Table 2 compares these prediction variants and baselines.Source: Table 2 · p. 7, §4.2 · p. 7
  • Removing the distillation loss slightly reduces Penn Action performance but changes Human3.6M performance negligibly.The authors suggest the learned latent representation may be more useful without 3D ground truth.Source: §4.2 · p. 7
  • On Penn Action, sequence-conditioned autoregression gains significantly over single-image future predictors at five frames ahead.The comparison evaluates all methods five frames into the future without DTW.Source: Table 3 · p. 7, §4.2 · p. 8

4.3. Qualitative Evaluation

  • Across Human3.6M, Penn Action, and NBA, latent predictions usually capture motion type despite tempo errors; pose-space forecasts often miss action type unless pose is obvious.Figure 3 shows predictions over 35 steps from 13 input frames.Source: Figure 3 · p. 6, §4.3 · p. 8
  • The analysis selects the conditioning window with the largest per-frame accuracy improvement; its following frame captures a visually suggestive action moment.The selected frames are presented as resembling classical athlete sculptures.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Conclusion

The paper presents autoregressive prediction of 3D human mesh motion from video and trains on motion-capture or in-the-wild video with only 2D annotations. It identifies multimodal futures, occlusion, and 3D-environment affordances as remaining challenges.

5. Conclusion

  • The paper presents a new approach for predicting 3D human mesh motion from video and trains an autoregressive model on a latent video representation.The latent representation lets conditioning transition from past video input to previously predicted futures.Source: §5 · p. 8
  • The approach can be trained on motion-capture video or in-the-wild video with only 2D supervision and could predict arbitrarily long sequences in principle.Source: §5 · p. 8
  • Remaining challenges include multiple possible futures, significant occlusions, and constraints imposed by the 3D environment's affordances.Source: §5 · p. 8

6. Appendix

The appendix details how short training tracks are used for long-range prediction and adds per-action evaluations, baseline comparisons, and qualitative examples. These analyses show that results vary with action timing, motion complexity, viewpoint, and conditioning quality.

6. Appendix

  • The appendix shows randomly sampled discovered “Statue” poses, selected from the conditioning window where prediction accuracy improves most.Each visualization shows the first frame after that window.Source: Figure 7 · p. 11

6.1. Implementation Details of Sequence Length

  • With training tracks of at least 40 frames, the method edge-pads the first image features and conditions on 15 images to train predictions up to 25 frames ahead.The track-length constraint reflects detections being truncated by people leaving the frame or becoming occluded; padding leaves enough training samples with previous predictions in the input.Source: §6.1 · pp. 11–12, Figure 6 · p. 11
  • Dynamic Time Warping matches frames across sequences and reduces the penalty when the predicted action begins too early, as in the illustrated golf swing.The figure describes each ground-truth frame as matched with at least one predicted frame, and vice versa.Source: Figure 8 · p. 12
  • With Dynamic Time Warping, Human3.6M predictions are strongest for periodic motion and weaker for intricate poses; Penn Action favors fast linear motions over direction changes and clearer views over side views.Jumping Jacks is omitted from the Penn Action evaluation because its videos are too short.Source: Table 4 · p. 12, Table 5 · p. 12
  • Without Dynamic Time Warping, errors accumulate over time; fast Penn Action sequences are especially sensitive to speed and action-start timing, while viewpoint quality affects slower actions.On Human3.6M, intricate poses remain challenging, and periodic motion is better evaluated with Dynamic Time Warping.Source: Table 7 · p. 13, Table 6 · p. 13
  • Without Dynamic Time Warping, autoregressive prediction in latent space still significantly outperforms the Constant, Nearest Neighbor, and pose-space baselines.The comparison also includes a latent-space ablation without the movie-strip distillation loss.Source: Table 8 · p. 13
  • For a clean, simple motion, latent- and pose-space autoregression both perform reasonably; on a complex tennis serve, latent-space predictions fare better while pose-space predictions struggle with the action and motion.The figures show ground-truth conditioning and use blue for latent-space predictions and pink for pose-space predictions.Source: Figure 11 · p. 15, Figure 12 · p. 16
  • Failure cases include plausible but incorrect 3D poses from ambiguous 2D keypoints, errors inherited from unusual-view conditioning, missed action onset after still frames, and drift beyond about 35 future frames.The paper notes that longer-sequence training may alleviate the drift issue.Source: Figure 13 · p. 17, Figure 14 · p. 17, Figure 15 · p. 18

Related papers

Loading 1908.04781v2…