PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting temporal context for 3D human pose estimation in the wild

Anurag Arnab, Carl Doersch, Andrew Zisserman

arXiv:1905.04266v1cs.CV

TL;DR

Single-frame monocular 3D pose estimation is ambiguous, while existing supervision is concentrated in constrained mocap settings. The paper applies temporal bundle adjustment to monocular videos, using consistent body shape and motion to recover poses and meshes, then generates weak supervision from YouTube videos. The resulting data and fits improve both temporal reconstruction and retrained per-frame estimators across mocap and real-world datasets.

  • Problem

    Single-image monocular 3D pose estimation is under-constrained, and leading methods discard temporal information while relying largely on constrained mocap data.

  • Method

    The method jointly fits SMPL and camera parameters across video frames using bundle adjustment with a shared sequence-level body-shape parameter and temporal consistency.

  • Results

    The approach improves Human 3.6M over single-frame estimation, and retraining on automatically generated Kinetics data improves performance on 3DPW and HumanEVA.

  • Takeaways & Limitations

    Large amounts of unlabeled real-world video can provide weak supervision that improves existing 3D pose estimation models.

  • Takeaways & Limitations

    The formulation assumes the subject’s body shape remains constant throughout the video by using one shared β parameter.

Abstract

from arXiv · show

We present a bundle-adjustment-based algorithm for recovering accurate 3D human pose and meshes from monocular videos. Unlike previous algorithms which operate on single frames, we show that reconstructing a person over an entire sequence gives extra constraints that can resolve ambiguities. This is because videos often give multiple views of a person, yet the overall body shape does not change and 3D positions vary slowly. Our method improves not only on standard mocap-based datasets like Human 3.6M -- where we show quantitative improvements -- but also on challenging in-the-wild datasets such as Kinetics. Building upon our algorithm, we present a new dataset of more than 3 million frames of YouTube videos from Kinetics with automatically generated 3D poses and meshes. We show that retraining a single-frame 3D pose estimator on this data improves accuracy on both real-world and mocap data by evaluating on the 3DPW and HumanEVA datasets.

1. Introduction

Monocular 3D pose estimation is ambiguous from single images and is often trained on constrained mocap data, limiting performance in the wild. The paper uses temporal bundle adjustment and YouTube videos to improve reconstruction and per-frame estimation.

  • Motivation: Single-image 3D pose is under-constrained because multiple 3D poses can project to the same 2D image.Temporal consistency helps narrow the space of possible poses.
  • Motivation: Temporal video cues constrain pose estimates because body shape and bone lengths stay constant while joint positions change slowly.The method exploits multiple views of a person across frames.
  • Contributions: Multi-frame bundle adjustment improves Human 3.6M performance over single-frame estimation and achieves state-of-the-art results for SMPL models.The method jointly optimizes information across video frames.
  • Contributions: The method processes 107 000 Kinetics YouTube videos to generate a diverse dataset of 3D human poses aligned with video frames and covering 400 actions.SMPL fitting also enables automatic extraction of 2D keypoints and body-part segmentations.
  • Contributions: Retraining a single-frame estimator on the generated data improves performance on both 3DPW and HumanEVA datasets.The results use automatically generated predictions as weak supervision from real-world videos.

2. Related Work

Prior work lifts 2D keypoints or fits body models to images, but commonly relies on constrained mocap data, synthetic imagery, or limited temporal modeling. This paper instead uses real-world YouTube videos and bundle-adjustment results to improve a per-frame model.

  • Pose representations: 3D pose methods represent humans as joint point clouds or parametric body-model parameters, using neural networks, dictionaries, or priors to resolve ambiguity.Image-to-3D regression and 2D-keypoint lifting are both established approaches.
  • Body-model fitting: Human body models such as SMPL reduce ambiguity by encoding prior knowledge, including constant bone-length ratios and symmetric limbs.Earlier methods fitted SCAPE or SMPL models manually, by optimization, or through learned regressors.
  • Temporal methods: The per-frame HMR model is used to initialize optimization, after which bundle adjustment corrects failures on challenging real-world videos.The resulting fits are also used to finetune the per-frame network.
  • Training data: Synthetic rendering adds labeled diversity but is not photorealistic, whereas this approach leverages unlabeled real-world YouTube videos.The two data-generation strategies are complementary.

3. Bundle Adjustment using the SMPL Model

The method jointly fits SMPL body and camera parameters across an entire video, combining reprojection, temporal-consistency, and pose-prior terms to resolve ambiguities in monocular 3D estimation.

  • The method jointly optimises SMPL and camera parameters for every frame, using per-frame 2D keypoints and initial SMPL estimates as input.It outputs mutually consistent parameters that reproject to the detected 2D keypoints.
  • Its objective combines reprojection error, temporal consistency, and a prior over SMPL pose and shape parameters.The temporal term operates on 3D joints, 2D projections, and camera parameters, while the prior encourages realistic poses.
  • The robust Huber reprojection penalty helps handle noisy 2D detections in in-the-wild sequences.The method also applies temporal smoothness to compensate for spurious detector errors at individual frames.
  • A single body-shape parameter β is shared across the sequence, while pose and camera parameters vary by frame.SMPL maps these parameters to the 3D mesh and joint positions.
  • The 3D prior prevents fits that match 2D keypoints and have low temporal error but do not form valid human poses.The prior includes a joint-angle model learned from CMU Mocap and a term keeping the solution near the HMR initialisation.
  • The approach jointly optimises all sequence parameters with L-BFGS and TensorFlow, taking about 8 minutes for a typical 250-frame Kinetics clip.This supports batch, offline video processing.
  • Compared with earlier bundle-adjustment approaches, it adds temporal consistency on 2D projections and cameras, uses robust penalties, and uses fitted results to improve a per-frame model.It directly optimises SMPL and camera parameters rather than HMR feature representations.

4. Leveraging Kinetics for weak supervision

The method adapts bundle adjustment to challenging YouTube videos by handling multiple people, outliers, missing detections, and unsuitable low-motion sequences during training-data selection.

  • Kinetics contains diverse YouTube videos with multiple people, camera shake, occlusions, and off-camera motion that make bundle adjustment difficult.These conditions also increase failures of the multi-person 2D pose detector and HMR.
  • The formulation uses the best-matching person detection per frame while treating sufficiently distant detections or initial estimates as outliers.This allows the optimization to accommodate multiple or absent people without letting unreliable detections affect the result.
  • Tracking initializes the optimization by selecting a person path through the video and filling skipped frames with the nearest available pose.The path permits missing detections by assigning a fixed penalty for skipped frames.
  • Normalizing total loss by 3D trajectory length reduces the preference for videos in which people remain still.Without normalization, low-motion people tend to produce low loss but unsuitable training data.
  • The pipeline excludes videos with more than six detected people, discards high-normalized-loss videos, and retains roughly 10% of the original videos.It then keeps frames whose 2D reprojections are inliers relative to detected keypoints.

5. Experiments

Experiments evaluate bundle adjustment on Human 3.6M through ablations and comparisons with SMPL-based methods, including settings with detected or ground-truth 2D keypoints.

  • Experimental setup: The experiments use Human 3.6M because its metric 3D ground truth supports ablations and comparisons with SMPL-based pose-estimation methods.The broader evaluation also applies the method to Kinetics and uses its predictions as weak supervision.
  • Ablation study: The optimization starts from HMR and combines reprojection, temporal consistency, and prior terms to improve pose estimates.Using reprojection alone can produce impossible poses, while the prior substantially improves the initialization.
  • Experimental setup: Human 3.6M videos are downsampled from 50fps to 10fps, with validation evaluated using MPJPE and rigidly aligned PA-MPJPE.Some sequences contain up to 1175 frames that are jointly optimized.
  • Ablation study: 9.4% MPJPE and 5.6% PA-MPJPE improvements over HMR are achieved by the final model with smooth joints, camera parameters, and a shared shape parameter.The prior alone reduces MPJPE by 6.2mm relative to HMR initialization.
  • Ablation study: 26.2% MPJPE and 27.2% PA-MPJPE reductions over HMR are obtained when ground-truth 2D keypoints, including occluded joints, are used.The result indicates that occluded-keypoint information further reduces pose ambiguity.
  • Comparison with prior work: The method achieves the best Human 3.6M results among approaches fitting SMPL, while direct CNN results are not directly comparable because they use different representations.The comparison excludes Mehta et al.'s method from direct comparison because it does not use SMPL.

5.3. Results on Kinetics

The Kinetics results show that bundle adjustment can produce diverse weakly supervised data from real-world action videos, including activities absent from motion-capture datasets.

  • Dataset construction: The pipeline processes Kinetics-400, comprising videos from 400 action classes, to create training data for single-frame 3D pose estimation.About 16.7K pre-selected videos yield 4.1M frames before filtering.
  • Dataset construction: 3.4M of 4.1M frames match the 2D pose detector under a 50-pixel total-error threshold.Visual inspection attributed most outlier frames to occlusion or people leaving the frame.
  • Action diversity: Selected action classes include tap dancing, salsa dancing, roller skating, and spinning poi, which are absent from existing motion-capture datasets.The method performs well on both multi-person or outdoor activities and diverse action categories.
  • Action diversity: Some poorly visible activities, including knitting and tying a tie, contribute only one selected video each.No action class has zero selected videos, but coverage varies with person visibility.

5.4. Weak supervision from Kinetics

The authors use bundle-adjustment outputs from Kinetics videos as weak supervision to retrain HMR, improving performance across real-world, mocap, and ordinal-depth evaluations. Their ablations show that training modifications are beneficial specifically with Kinetics data.

  • Training setup: HMR retraining uses original data alongside Kinetics data, with training-procedure modifications evaluated through dataset-specific experiments.The experiments evaluate 3DPW, HumanEVA, and Ordinal Depth.
  • 3D Poses in the Wild: 300K Kinetics frames improve 3DPW PA-MPJPE by 3.4mm, while 3M frames improve it by 5mm over baseline.The Kinetics-trained model also outperforms public HMR on 3DPW.
  • HumanEVA: 300K Kinetics frames improve HumanEVA PA-MPJPE by 2.2mm, while 3M frames improve it by 3.6mm over baseline.HumanEVA is an indoor motion-capture validation dataset not used to train the original HMR model.
  • Ordinal Depth: Kinetics training improves ordinal-depth accuracy, with additional gains from using the full dataset.The evaluation covers real-world images from MPII and LSP, and no ordinal-depth annotations are used during training.
  • Cross-dataset results: The retrained model improves per-frame HMR across multiple datasets, with larger gains on real-world 3DPW than on mocap HumanEVA.This supports using automatically generated Kinetics supervision to improve a single-frame estimator.
  • Ablations: PA-only 3D training discards SMPL-joint and absolute-location losses, while No 2D disables HMR datasets containing only 2D data and its adversarial prior.These schemes are defined in the ablation-table description.
  • Training modifications: The authors train only on Procrustes-aligned 3D keypoints, allowing HMR to learn camera orientation through 2D reprojection error.They suspect this helps because Kinetics spans a broad range of camera orientations.
  • Ablations: Training modifications help only with additional Kinetics data, whereas removing original 2D data substantially harms performance.The authors attribute the latter effect to the limited amount of 3D supervision in the original training data.

6. Conclusion and Future Work

The paper uses temporal bundle adjustment to improve monocular 3D pose estimates and applies those results to large-scale, unlabeled YouTube data. The resulting supervision improves per-frame pose estimators beyond motion-capture-restricted settings.

  • Conclusion: The bundle-adjustment algorithm leverages temporal context in video to improve estimates of a person’s 3D pose.The method exploits constraints across frames rather than relying only on isolated images.
  • Conclusion: The automatically generated Kinetics dataset is used to improve per-frame 3D pose estimators with large amounts of unlabeled real-world data.The conclusion links temporal reconstruction, dataset generation, and estimator improvement.
  • Future work: Kinetics videos provide slowly changing 3D positions and consistent person size and appearance, creating constraints that strengthen supervision.These constraints help move estimation beyond environments restricted by motion-capture devices.

Appendix

The appendix identifies the Kinetics source material and directs readers to supplementary details about dataset generation and bundle-adjustment settings.

  • Appendix: The Kinetics data include at least one selected video for every action class.The passage refers to the action-class coverage of the dataset used by the paper.
  • Appendix: The appendix provides bundle-adjustment hyperparameters and additional details about the automatically generated Kinetics dataset.These materials are organized in Sections A and B.

A. Experimental Details

The appendix documents the hyperparameters used in bundle adjustment and explains how parameter scales and initialization quality affect selected weights.

  • Experimental Details: Table 7 lists the bundle-adjustment hyperparameter values used in the experiments.The table provides the experimental settings for the optimization procedure.
  • Hyperparameter scales: Small λR and λ2 values reflect the larger pixel-scale range of 2D joints compared with 3D joints and camera parameters.2D positions typically reach about 450, while 3D positions and camera parameters are typically in [−1, 1].
  • Initialization priors: λI and λβ are higher on Human 3.6M because HMR initialization is more accurate there than on Kinetics.These weights encourage bundle-adjustment results to remain close to initialization.

B. Dataset statistics

The dataset contains a varied distribution of Kinetics action classes with a fairly long tail, including both common and sparsely represented activities.

  • The action-class distribution has a fairly long tail, with many classes represented by only a handful of automatically selected videos.
  • The bundle adjustment method works across varied classes, including many dancing and outdoor activities where the whole body is usually visible.
  • Figure 5 shows the overall distribution of selected video clips across Kinetics action classes, including the most- and least-common classes.Panels (a), (b), and (c) present these three views of the distribution.
Loading 1905.04266v1…