PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach

Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, Yichen Wei

arXiv:1704.02447v2cs.CV

TL;DR

The paper targets 3D human pose estimation in the wild, where available datasets separate in-the-wild 2D labels from controlled-environment 3D labels. It introduces unified weakly-supervised transfer learning with shared 2D/3D representations and geometric regularization, achieving competitive cross-task benchmark performance, including 69.2 PCK and 32.5 AUC on MPI-INF-3DHP.

  • Problem

    3D human pose estimation in the wild lacks comprehensive 3D-labeled datasets, while existing 3D data come mainly from controlled laboratory environments.

  • Method

    A unified end-to-end network combines in-the-wild 2D labels and indoor 3D labels, shares intermediate representations between 2D and depth modules, and uses a bone-length geometric constraint.

  • Results

    The method achieves state-of-the-art performance on several benchmarks and obtains 69.2 PCK and 32.5 AUC on MPI-INF-3DHP without using that dataset's training data.

  • Takeaways & Limitations

    The framework transfers 3D pose information from controlled indoor data to in-the-wild images while using 2D annotations and geometric structure to regularize depth prediction.

  • Takeaways & Limitations

    The approach is constrained by the sub-optimality of sequential pipelines and by the assumption that relative bone-length ratios remain approximately fixed.

Abstract

from arXiv · show

In this paper, we study the task of 3D human pose estimation in the wild. This task is challenging due to lack of training data, as existing datasets are either in the wild images with 2D pose or in the lab images with 3D pose. We propose a weakly-supervised transfer learning method that uses mixed 2D and 3D labels in a unified deep neutral network that presents two-stage cascaded structure. Our network augments a state-of-the-art 2D pose estimation sub-network with a 3D depth regression sub-network. Unlike previous two stage approaches that train the two sub-networks sequentially and separately, our training is end-to-end and fully exploits the correlation between the 2D pose and depth estimation sub-tasks. The deep features are better learnt through shared representations. In doing so, the 3D pose labels in controlled lab environments are transferred to in the wild images. In addition, we introduce a 3D geometric constraint to regularize the 3D pose prediction, which is effective in the absence of ground truth depth labels. Our method achieves competitive results on both 2D and 3D benchmarks.

1. Introduction

The paper addresses limited generalization in 3D human pose estimation by transferring 3D supervision from controlled indoor images to in-the-wild images using mixed 2D and 3D labels. Its unified network shares representations between 2D and 3D tasks, trains end-to-end, and adds a geometric constraint for weak supervision.

  • Motivation: 3D pose estimation in the wild is limited by ambiguous single-image depth and the lack of comprehensive in-the-wild 3D pose datasets.Existing 3D datasets primarily use mocap systems in controlled laboratories, and models trained on them generalize poorly to other environments.
  • Motivation: Sequential pipelines separately estimate 2D joints and recover 3D pose, discarding original in-the-wild image cues during 3D recovery.The paper identifies this loss of image information as a reason such pipelines are sub-optimal.
  • Approach: The method formulates weakly-supervised transfer learning from fully annotated indoor images to weakly labeled in-the-wild images.In-the-wild 2D annotations serve as weak labels for 3D pose estimation.
  • Approach: A unified network connects 2D and 3D modules through intermediate representations and trains them end-to-end with 2D and 3D data simultaneously.This differs from approaches that merely pass 2D predictions into a separately trained 3D module.
  • Approach: The geometric constraint regularizes weakly-supervised 3D pose learning using approximately fixed relative bone lengths.The constraint is experimentally verified for adapting 3D pose information from indoor labeled images to in-the-wild images.

2. Related Work

Prior work addresses in-the-wild 3D pose estimation through sequential 2D-to-3D pipelines, synthetic data, or mixed 2D and 3D training. The paper instead emphasizes unified weak supervision and geometry-guided regularization to use image information and constrain predictions.

  • 3D Human Pose Estimation: Standard methods split in-the-wild 3D pose estimation into separate 2D joint detection and 3D recovery tasks.The 2D stage uses in-the-wild annotations, while the 3D stage can train on benchmark datasets because it receives only 2D locations.
  • 3D Human Pose Estimation: These pipelines can produce ambiguous 3D results because they estimate pose only from 2D joints, whereas this method also uses intermediate image features.The additional image representations preserve cues discarded by joint-only recovery.
  • 3D Human Pose Estimation: Synthetic datasets provide 3D ground truth, but their effectiveness depends on modeling image distributions that match natural images.The passage reports that state-of-the-art synthetic-data methods are less competitive on natural images.
  • Mixed 2D and 3D Data: Other mixed-data approaches use 2D-to-3D fine-tuning or multitask learning, while this work integrates 2D and 3D data with a weakly-supervised loss.The distinction is the unified treatment of both annotation types.
  • Weakly-/Un-supervised Constraints: Weakly supervised constraints have been used in classification, segmentation, and domain adaptation to impose relationships on predictions or features.Examples include label-probability constraints, domain confusion, and adversarial domain alignment.
  • Weakly-/Un-supervised Constraints: The paper presents geometry-guided constraint regularization for pose estimation in the wild as a novel use of weak supervision.The constraint is intended to regularize pose predictions when direct 3D supervision is insufficient.

3. Approach

The approach jointly predicts 2D joints and depths from RGB images, combining intermediate 2D features with depth regression and training on mixed 2D/3D supervision. A geometric loss regularizes depth prediction when depth labels are unavailable.

  • The network represents each 3D joint with image-plane coordinates and metric depth, producing the final pose by concatenating 2D locations and depth values.
  • The unified network is trained end-to-end with both 2D-labeled wild images and 3D-labeled lab images, integrating the two modules instead of training them separately.Mini-batches contain both 2D and 3D training examples, while the final training stage fine-tunes the whole network.
  • The architecture combines 2D heat-maps and intermediate 2D-module features as inputs to the depth regression module, rather than using only predicted 2D joints.This supplies image features alongside the 2D skeleton for depth prediction.
  • For weakly labeled images, a geometric loss regularizes predicted depths using approximately fixed relative bone-length ratios within human skeleton groups.The loss measures variance among normalized bone lengths and is differentiable with respect to predicted depths.
  • The geometric constraint excludes torso bones because their lengths vary relatively strongly across human shapes, making the constraint less valid there.
  • Although direct end-to-end training from scratch performed poorly, the authors use three stages and activate the geometric constraint only during final fine-tuning.

4. Experimental Evaluation

The evaluation trains one model on Human3.6M and MPII and tests it on three datasets, covering supervised 3D estimation and transferred 3D estimation in the wild. Qualitative results are reported in Table 5.

  • A single model is trained using Human3.6M and MPII data and evaluated on three different testing datasets.
  • The evaluation covers supervised 3D human pose estimation and transferred 3D human pose estimation in the wild.
  • Qualitative results are summarized in Table 5, with additional MPII validation results provided in the supplementary material.

4.1. Experimental Setup

The experiments combine in-the-wild 2D data and controlled 3D data, evaluate across multiple datasets and metrics, and compare the proposed method with three baselines.

  • Implementation: The implementation uses a two-stack shallow hourglass and a depth module with four sequential residual-and-pooling modules.All experiments use the same network architecture and training iterations.
  • Datasets: MPII provides 25k training images and 2D joint annotations for training the 2D module and weakly supervising depth regression.The dataset contains 16 annotated joints per image and is collected from online videos.
  • Datasets: Human3.6M supplies projected 2D locations and depth annotations for training, with five subjects used for training and two for testing.Evaluation uses MPJPE in millimeters after aligning root-joint depths.
  • Datasets: MPI-INF-3DHP is used only for evaluation, with 2,929 valid frames from six subjects and PCK and AUC as metrics.The test images include indoor and outdoor scenes, and evaluation assumes the global scale is known.
  • Baselines: The proposed method is compared with 3D/wo geo, 3D/w geo, and 3D+2D/wo geo baselines that differ in data usage and geometric-constraint application.The proposed model is denoted 3D+2D/w geo.

4.2. Supervised 3D Human Pose Estimation

On Human3.6M, combining 2D and 3D training data substantially improves 3D pose accuracy, while the geometric constraint further improves the proposed approach.

  • Baseline comparison: 82.44mm average MPJPE is obtained by the 3D/wo geo baseline, already comparable to several state-of-the-art methods.This baseline uses only 3D-labeled data after pretraining the 2D module.
  • Baseline comparison: Adding the geometric constraint to the 3D-only baseline provides a decent performance gain.This comparison isolates the contribution of the geometric constraint.
  • Baseline comparison: 64.90mm average MPJPE is achieved by 3D+2D/wo geo, outperforming previous work on Human3.6M.The result supports combining 2D and 3D data in unified training.
  • Baseline comparison: The proposed 3D+2D/w geo approach achieves the best Human3.6M results, including when constraints are applied to the disjoint 2D dataset.Applying the constraint to fully supervised 3D data produces similar results.
  • In-the-wild comparisons: 64.90mm MPJPE is reported for the proposed method, compared with 114.18mm, 79.9mm, and 78.1mm for three other in-the-wild methods.The cited comparison reports the proposed method as significantly better.
  • Analysis: Adding 2D training data mainly benefits depth regression through shared deep representations rather than improving 2D pose accuracy.The 2D pose is reported as very accurate across all baselines and the proposed method.

4.3. Transferred Human Pose In the Wild

The method generalizes to a domain-shifted 3D benchmark and in-the-wild MPII images, while geometric constraints improve pose validity and can also improve 2D accuracy.

  • MPI-INF-3DHP Dataset: The geometric constraint remains effective under domain shift, with 3D/wo geo performing worse than 3D/w geo.The 3D-only baselines perform poorly because their training images are indoor only.
  • MPI-INF-3DHP Dataset: 65.8 PCK and 32.1 AUC are achieved by 3D+2D/wo geo, exceeding the corresponding 64.7 PCK and 31.7 AUC from prior work.The comparison uses Human3.6M training data for the prior method.
  • MPI-INF-3DHP Dataset: 69.2 PCK and 32.5 AUC are obtained on MPI-INF-3DHP without using that dataset’s training data.These results are close to 72.5 PCK and 36.5 AUC derived from the original training data.
  • Geometric validity: Using the geometric constraint considerably improves the geometric validity measured by left-right symmetric bone lengths.The evaluation is reported on MPI-INF-3DHP and the MPII validation set.
  • MPII Validation Dataset: The geometric constraint also improves 2D joint accuracy on MPII validation, indicating propagation from the 3D module to the 2D module.This supports the network’s shared end-to-end design.

5. Future Work and Conclusions

The paper introduces an end-to-end system combining 2D pose labels from in-the-wild images with 3D pose labels from restricted environments.

  • The system combines 2D pose labels in the wild with 3D pose labels from restricted environments.
  • Its target challenge is 3D human pose estimation in the wild.
  • The passage indicates that future work was planned, but does not specify the planned direction.
Loading 1704.02447v2…