PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision

Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, Christian Theobalt

arXiv:1611.09813v5cs.CV

TL;DR

The paper addresses poor generalization in monocular 3D pose estimation caused by limited and insufficiently diverse 3D pose data. It combines CNN supervision, transfer learning from in-the-wild 2D pose data, and a diverse real-human dataset, achieving state-of-the-art benchmark performance and improved generalization to in-the-wild images.

  • Problem

    Monocular 3D pose estimation lacks suitably large and diverse annotated 3D pose corpora, limiting evidence for performance in uncontrolled environments.

  • Method

    The approach uses a feedforward CNN pipeline with enhanced supervision, transfer learning from 2D pose data, and the MPI-INF-3DHP real-human dataset.

  • Results

    The method attains state-of-the-art results on established benchmarks and quantitatively outperforms existing methods on the introduced in-the-wild benchmark.

  • Takeaways & Limitations

    Transfer learning from in-the-wild 2D data combined with more varied real and augmented human appearances, activities, and camera views significantly improves generalization to in-the-wild images.

  • Takeaways & Limitations

    Training data and evaluation primarily use chest-height cameras, so estimating 3D pose from substantially different camera elevations remains challenging.

Abstract

from arXiv · show

We propose a CNN-based approach for 3D human body pose estimation from single RGB images that addresses the issue of limited generalizability of models trained solely on the starkly limited publicly available 3D pose data. Using only the existing 3D pose data and 2D pose data, we show state-of-the-art performance on established benchmarks through transfer of learned features, while also generalizing to in-the-wild scenes. We further introduce a new training set for human body pose estimation from monocular images of real humans that has the ground truth captured with a multi-camera marker-less motion capture system. It complements existing corpora with greater diversity in pose, human appearance, clothing, occlusion, and viewpoints, and enables an increased scope of augmentation. We also contribute a new benchmark that covers outdoor and indoor scenes, and demonstrate that our 3D pose dataset shows better in-the-wild performance than existing annotated data, which is further improved in conjunction with transfer learning from 2D pose data. All in all, we argue that the use of transfer learning of representations in tandem with algorithmic and data contributions is crucial for general 3D body pose estimation.

1. Introduction

The paper targets monocular 3D pose estimation in uncontrolled scenes, where limited and insufficiently diverse 3D annotations hinder generalization. It combines CNN supervision, transfer learning from in-the-wild 2D pose data, and a new real-human dataset to improve performance on controlled benchmarks and in-the-wild sequences.

  • Research gap: Monocular 3D pose estimation remains difficult in the wild because suitably large and diverse annotated 3D pose corpora are scarce.Crowdsourced 2D annotations are more available, but lifting 2D keypoints to 3D remains vulnerable to depth ambiguity and can require expensive iterative optimization.
  • Research gap: Existing datasets trade clothing diversity against capture cost: marker-based systems require skin-tight clothing, while markerless dome systems require expensive studios.Synthetic avatar data offers another option but can lack realism and generalize poorly to real images.
  • Method contributions: The method transfers mid- and high-level features from in-the-wild 2D pose datasets to existing 3D pose data, improving accuracy and generalizability over naive initialization and domain adaptation.The authors report previously unseen accuracy levels on established benchmarks and generalization to in-the-wild scenes using existing 3D datasets.
  • Data contributions: The MPI-INF-3DHP dataset provides real-human 3D annotations from markerless motion capture, with varied clothing, motions, object interactions, and camera viewpoints.Its capture approach also supports appearance augmentation and enhanced foreground texture variation.
  • Method contributions: CNN architectural supervision complements the data-side contributions and is presented as orthogonal to improvements in in-the-wild performance.
  • Evaluation: The authors introduce an accurately annotated outdoor-inclusive test set and report improved in-the-wild performance over existing solutions.The components are also evaluated on existing datasets, where they demonstrate state-of-the-art results in controlled settings.

2. Related Work

The paper situates its CNN-based monocular 3D pose estimator among approaches that lift 2D predictions or regress 3D pose directly, then combines multi-stage prediction, auxiliary supervision, multimodal representations, and global-pose reconstruction.

  • Related work: Prior methods either lift 2D keypoints to 3D using geometric or learned relations, or regress 3D pose directly from image information.The paper positions its method in the direct-regression family while using predicted 2D joints for global pose estimation.
  • CNN-based 3D Pose Estimation: The proposed pipeline detects 2D joints and a bounding box, regresses root-centered 3D pose from the crop, then computes global coordinates using 2D joints and camera calibration.The final global-pose computation includes perspective correction in closed form.
  • CNN-based 3D Pose Estimation: 3DPoseNet uses auxiliary 2D heatmap prediction, intermediate pose supervision, corrective skip connections, and multiple pose-relative representations.The representations include root-relative, first-order-parent-relative, and second-order-parent-relative joint locations.
  • Global Pose Computation: Bounding-box cropping simplifies pose regression by normalizing subject scale and position, while global reconstruction restores information lost through cropping.The reconstruction uses the fused pelvis-centered pose, camera intrinsics, and predicted 2D joints.

4. Transfer Learning

The transfer-learning study initializes 3DPoseNet from features learned by 2DPoseNet and evaluates learning-rate strategies for preserving useful representations while adapting to 3D pose.

  • Transfer Learning: 2DPoseNet and 3DPoseNet are initialized with ResNet-101 features learned on ImageNet before task-specific training.The paper additionally studies transfer from in-the-wild 2D pose datasets to 3D pose estimation.
  • Transfer Learning: Table 1 compares learning-rate multiplier combinations across network parts using Human3.6m training and testing splits.Random initialization is marked with an asterisk in the table.
  • Transfer Learning: Transfer from 2DPoseNet requires balancing preservation of transferred features against learning new features relevant to 3D pose.The selected scheme scales down the learning rate through res4b22 by a factor of 1000.
  • Transfer Learning: The proposed transfer mechanism outperforms domain adaptation and is more effective than fine-tuning only fully connected layers or the complete network.The authors report that these comparisons validate the learning-rate discrepancy scheme.

5. MPI-INF-3DHP: Human Pose Dataset

MPI-INF-3DHP is a markerless multi-camera dataset of real humans designed to broaden 3D pose diversity and support appearance augmentation. Its test set spans varied subjects, clothing, poses, viewpoints, and indoor/outdoor settings.

  • Dataset capture: MPI-INF-3DHP captures real humans with ground-truth 3D annotations from a commercial markerless multi-camera motion-capture system.The setup avoids special suits and markers, allowing everyday and loose clothing.
  • Dataset diversity: Eight actors perform eight activity sets covering walking, sitting, complex exercises, and dynamic actions, with more pose classes than Human3.6m.Each activity set lasts roughly one minute, and actors use two clothing sets.
  • Viewpoints and scale: Fourteen cameras provide chest-height, elevated, overhead, and low-angle viewpoints, yielding more than 1.3M captured frames.The dataset includes true 3D annotations and a compatible skeleton.
  • Augmentation: Chroma-key masks separately support augmentation of backgrounds, chairs, and upper- and lower-body regions using independently composited textures.The capture design enables foreground and background variation while retaining photorealistic intensity changes.
  • Pose coverage: The Human3.6m test poses are clustered into 20 centroids and three broad classes, with upright poses dominant and sitting and crouching representing 25% and 8%.Multimodal fusion improves Sit by 3.5mm and Crouch by 5.5mm.
  • Test set: The MPI-INF-3DHP test set includes diverse subjects, clothing, and poses across green-screen studio, ordinary studio, and outdoor settings.This expands evaluation beyond restricted indoor test environments.

6. Experiments and Evaluation

Experiments evaluate the proposed supervision, transfer learning, dataset, and augmentation choices on standard benchmarks and the new MPI-INF-3DHP test set. The combined approach achieves state-of-the-art controlled-setting accuracy and stronger in-the-wild generalization.

  • Evaluation protocol: The evaluation uses Human3.6m, HumanEva, MPI-INF-3DHP, and qualitative tests on LSP and CMU Panoptic.The latter qualitative evaluations assess robustness to general scenes.
  • Supervision methods: Corrective skip connections improve Human3.6m accuracy by more than 5mm for seven activity classes, while vanilla skip connections perform worse than the baseline.Multimodal fusion yields additional improvements across the tested datasets.
  • Transfer learning: 64.7% 3DPCK on MPI-INF-3DHP is obtained using 2DPoseNet transfer with only Human3.6m training, versus 63.7% without transfer on augmented MPI-INF-3DHP training.Combining the dataset and transfer learning reaches approximately 72.5% 3DPCK in the reported comparison.
  • Transfer learning: Transfer learning exceeds 28.8% 3DPCK for synthetic data with domain adaptation and 21.7% 3DPCK for the reported synthetic-data base model.The comparison is made on the MPI-INF-3DHP test set after the stated alignment conditions.
  • Dataset benefit: Without transfer learning, MPI-INF-3DHP improves outdoor-scene performance over Human3.6m by approximately 9% 3DPCK, while augmentation raises overall performance to 57.3% 3DPCK.The reported gains occur across scene settings.
  • Combined results: 76.5% 3DPCK on MPI-INF-3DHP and 72.88mm on Human3.6m are achieved when the datasets are combined with transfer learning.Using computed bounding boxes lowers MPI-INF-3DHP performance to 74.4% 3DPCK, and omitting perspective correction lowers it to 73% 3DPCK.
  • Benchmark results: The method attains 74.11mm on Human3.6m without scaling and further reduces the error to approximately 72mm when Human3.6m is complemented with augmented MPI-INF-3DHP.HumanEva results are 30.8 and 33.5 MPJPE on the reported aligned S1 Box and Walk sequences.

7. Discussion

The discussion identifies viewpoint bias and temporal jitter as remaining limitations while emphasizing transfer learning together with algorithmic and data contributions as a route to broader accuracy and generalizability.

  • Limitations: Training sets remain biased toward chest-height cameras, so estimating 3D pose from substantially different viewpoints remains challenging.The diverse viewpoints in MPI-INF-3DHP are intended to support future viewpoint-invariant methods.
  • Limitations: Per-frame estimation exhibits temporal jitter on video sequences, motivating future integration with model-based temporal tracking.The paper reports processing below 250ms per frame but notes further scope for real-time improvement.
  • Implication: The paper argues that transfer learning, algorithmic supervision, and data contributions jointly support progress in overall accuracy and generalizability.This conclusion is stated across multiple directions of 3D pose estimation.

8. Conclusion

The paper presents a feedforward CNN for monocular 3D human pose estimation that reaches state-of-the-art on established benchmarks and outperforms existing methods on a new in-the-wild benchmark. Transfer learning and a diverse real-and-augmented dataset improve generalization, while the method also extracts global 3D position from non-cropped images without iterative optimization.

  • Conclusion: The fully feedforward CNN attains state-of-the-art performance on established benchmarks and quantitatively outperforms existing methods on the introduced in-the-wild benchmark.Its supervision techniques and improved kinematic-chain parent relationships contribute to the reported benchmark performance.
  • Conclusion: Transfer learning from in-the-wild 2D pose data combined with MPI-INF-3DHP improves generalization across human appearances, activities, augmentations, and camera views.The conclusion links the improved generalization to the combination of representation transfer and broader dataset variation.
  • Conclusion: The method efficiently extracts global 3D position in non-cropped images without time-consuming iterative optimization.This distinguishes the reported feedforward global-pose capability from optimization-dependent approaches.

Supplemental Document: Monocular 3D

The fusion scheme combines predictions from P, O1, and O2, with distinct prediction stubs encouraging different misprediction modes.

  • 74.49mm MPJPE is achieved on Human3.6m using P, O1, and O2 together.
  • Removing O2 increases MPJPE from 74.49mm to 74.77mm, while removing both O1 and O2 increases it to 75.27mm.
  • Distinct prediction stubs decorrelate the feature sets for P, O1, and O2, supporting different modes of misprediction.

2. Further Discussion of Multi-level Corrective Skip

The multi-level corrective skip scheme adds a final-stage loss that increases the core network’s effective learning rate, so a matched-loss-weight baseline tests whether this explains the gains.

  • A Base network with loss weights matched to the combined Xdeep and Xsum weights performs worse than the proposed multi-level corrective skip scheme.

3. Global Pose Computation

Global 3D pose is recovered by estimating translation from noisy 2D and 3D joint predictions under a weak-perspective assumption, then correcting crop-induced viewpoint changes.

  • The method estimates global translation T for a noisy 3D point set P using its 2D projection K and known camera intrinsics.
  • Weak perspective approximates the pinhole model when depth spread is negligible relative to the distance z0 from the camera.
  • Translation is obtained by minimizing a least-squares energy over x, y, and z, with P_i and K_i denoting corresponding 3D and 2D joints.
  • The depth solution uses an approximation valid when the 3D and 2D pose rotations are assumed to match.
  • 58.6mm is obtained on HumanEva’s Box sequence, improving over the reported 82.1mm result without using HumanEva training data.
  • Rotating the crop-coordinate reconstruction by R maps it from the virtual camera to the original camera, correcting inaccuracies away from the camera center.

4. CNN Architecture and Training Specifics

The system combines a ResNet-101-derived 2D network, intermediate supervision, corrective skips, and a 3D prediction architecture, with transfer learning and dataset-specific training protocols.

  • 2DPoseNet: The 2DPoseNet architecture derives from ResNet-101 and uses intermediate joint-heatmap supervision plus multi-level corrective skip connections.
  • Evaluation: Table 1 evaluates HumanEva-I under different alignment strategies, while Table 2 reports 2DPoseNet results on MPII and LSP.
  • 3DPoseNet: The 3D network attaches prediction stubs to the core network and sums corrective-skip predictions with the deep prediction Pdeep.
  • Multi-modal Fusion: Multi-modal fusion uses distinct stubs for P, O1, and O2, followed by fully connected layers producing the final prediction.
  • Supervision and training: Intermediate supervision predicts P, O1, and O2 together while auxiliary heatmap and part-label tasks support the network; part-label maps are omitted for MPI-INF-3DHP training.
  • Datasets: Approximately 37.5k frames per dataset produce approximately 75k samples after two-scale augmentation, with Human3.6m subjects S1,5,6,7,8 for training and S9,11 for testing.
  • Datasets: MPI-INF-3DHP training uses five chest-high cameras across eight subjects, while its augmented version mixes unaugmented frames with background and chair augmentation.
  • Domain adaptation: The domain-adaptation baseline uses a gradient-inversion domain-classification stub attached after res4b22, with 2D heatmap and part-label prediction as auxiliary tasks.

5. MPI-INF-3DHP Dataset

The MPI-INF-3DHP dataset covers diverse activities, subjects, and augmentations, while the paper’s evaluation uses chest-height cameras to remain compatible with existing datasets.

  • The training and test sets cover a wide range of poses grouped into activity classes, with varied subjects represented in both sets.Figure 3 samples training activities, augmentation, and subjects; Figure 4 samples test activities and subjects.
  • The paper considers only chest-height cameras in training, partly to remain compatible with existing datasets.Viewpoint elevation invariance is identified as a significantly more challenging problem, and an expanded test set with multiple elevations is planned for release.
  • Training data includes varied activities, clothing, subjects, and augmentation, as illustrated in Figure 3.
  • The test set samples diverse activities and subjects, as illustrated in Figure 4.
Loading 1611.09813v5…