PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Human Pose Estimation from a Single Image via Distance Matrix Regression

Francesc Moreno-Noguer

arXiv:1611.09010v1cs.CV

TL;DR

Single-image 3D human pose estimation is ambiguous and typically requires substantial annotated 3D training data. This paper instead regresses a 3D distance matrix from detected 2D joints, achieving strong benchmark performance and in-the-wild generalization. The method also handles detector noise and missing joints, including limb hypotheses with its Fully Convolutional model.

  • Problem

    The problem is inferring 3D human pose from a single image despite ambiguous 2D projections and limited representative in-the-wild 3D training data.

  • Method

    The method detects noisy or missing 2D joints, represents 2D and 3D poses as EDMs, regresses between them with simple neural networks, and reconstructs 3D joints using MDS.

  • Results

    The approach achieves state-of-the-art results on Humaneva-I and Human3.6M, with robustness to detector errors and qualitative generalization to LSP images in the wild.

  • Takeaways & Limitations

    EDMs incorporate structural information and joint correlations directly, while the Fully Convolutional model can hypothesize unobserved body parts.

  • Takeaways & Limitations

    The LSP evaluation covers detections, poses, or camera viewpoints that differ substantially from Human3.6M, limiting direct comparability to that training setting.

Abstract

from arXiv · show

This paper addresses the problem of 3D human pose estimation from a single image. We follow a standard two-step pipeline by first detecting the 2D position of the $N$ body joints, and then using these observations to infer 3D pose. For the first step, we use a recent CNN-based detector. For the second step, most existing approaches perform 2$N$-to-3$N$ regression of the Cartesian joint coordinates. We show that more precise pose estimates can be obtained by representing both the 2D and 3D human poses using $N\times N$ distance matrices, and formulating the problem as a 2D-to-3D distance matrix regression. For learning such a regressor we leverage on simple Neural Network architectures, which by construction, enforce positivity and symmetry of the predicted matrices. The approach has also the advantage to naturally handle missing observations and allowing to hypothesize the position of non-observed joints. Quantitative results on Humaneva and Human3.6M datasets demonstrate consistent performance gains over state-of-the-art. Qualitative evaluation on the images in-the-wild of the LSP dataset, using the regressor learned on Human3.6M, reveals very promising generalization results.

1. Introduction

The paper uses a two-stage pipeline that detects 2D joints and regresses a 3D pose representation from those observations. It replaces Cartesian regression with EDM regression to encode pose structure and improve robustness to ambiguity, noise, and occlusion.

  • Motivation: Single-image 3D pose estimation is ill-posed because different body configurations can share nearly identical 2D projections.Existing image-to-3D methods also require large datasets with ground-truth 3D annotations, while available in-the-wild datasets mainly provide 2D poses.
  • Approach: The proposed pipeline detects 2D joints with a CNN, regresses a 3D EDM from a 2D EDM, and recovers joint positions using Multidimensional Scaling.The 2D detector supplies image observations, while MDS converts the predicted 3D distance matrix into joint coordinates.
  • Representation: EDMs encode pose structure, provide invariance to in-plane translation, rotation, and normalized scale, and capture dependencies among all joints.These properties are presented as advantages over vector representations for 2D-to-3D inference.
  • Learning: Shallow Fully Connected and Fully Convolutional networks regress N × N distance matrices, with N = 14 joints in the model.The architectures use two hidden layers for FConn and four convolutional layers for FConv.
  • Results: The approach achieves state-of-the-art results on Humaneva-I and Human3.6M and shows robustness to large 2D detector errors.On LSP images in the wild, a network trained on Human3.6M demonstrates good generalization.
  • Results: The Fully Convolutional network can hypothesize occluded body limbs, while predicted 3D EDMs are converted to joint positions with MDS.The method is designed to handle missing observations in the 2D input.

2. Related Work

Prior work includes generative and discriminative approaches, as well as pipelines that infer 3D pose from detected 2D joints. The paper positions EDMs as a representation that captures joint dependencies while enabling simple neural-network inference.

  • Existing approaches: 3D pose methods broadly use generative models to constrain possible shapes or discriminative models to predict pose from image evidence.Generative approaches include linear modes, spectral embeddings, mixture models, and Gaussian processes.
  • Existing approaches: Direct discriminative methods map image descriptors or full images to 3D pose, but their success depends on large datasets with ground-truth 3D annotations.Humaneva and Human3.6M are identified as popular motion-capture datasets for this purpose.
  • Two-stage methods: Two-stage methods first estimate 2D joints and then infer 3D pose using regression forests, Expectation Maximization, evolutionary algorithms, or iterative refinement.This decomposition allows use of large-scale in-the-wild datasets for CNN-based 2D detectors.
  • Pose representation: Most methods represent skeletons as 3N-dimensional Cartesian vectors, while some enforce joint dependencies through latent representations or generative volumetric models.The paper instead proposes N × N EDMs to capture dependencies between joints.
  • Pose representation: EDMs are chosen over GDMs because a specific EDM uniquely determines the shape up to translation, whereas isometric deformations can share the same GDM.The paper notes prior EDM applications in shape, protein, sensor-localization, and kinematic-constraint problems.
  • Paper positioning: Distance-matrix representations encode joint dependencies directly, allowing inference with simple neural networks without explicitly modifying architectures for those dependencies.Figure 2 reports a Pearson correlation coefficient of 0.60 for EDMs versus 0.09 for Cartesian representations.

3. Method

The method detects noisy or missing 2D joints, converts normalized 2D and 3D poses into Euclidean Distance Matrices, and learns a 2D-to-3D matrix regression. Neural architectures enforce matrix structure, after which 3D joints are recovered with Multidimensional Scaling.

  • Problem Formulation: The pipeline detects 2D joints, represents their normalized pairwise distances as an EDM, regresses a 3D EDM, and recovers joint positions with reflection-aware MDS.The detector output may contain noise, occlusions, or mis-detections; unobserved joints are set to zero.
  • Representing Human Pose with EDMs: 2D and 3D poses are mapped to N × N matrices, with the 3D matrix entry defined by the Euclidean distance between joints m and n.The model uses N = 14 joints; normalized 2D vertical coordinates lie in [−1, 1], while 3D coordinates remain in meters.
  • Representing Human Pose with EDMs: EDMs encode pose structure, provide rotation, translation, reflection, and scaling invariance, and capture pairwise dependencies among all joints.These properties reduce ambiguities in the 2D-to-3D estimation problem relative to Cartesian representations.
  • Representing Human Pose with EDMs: EDM representations produce more correlated 2D and 3D pairwise differences than Cartesian representations, especially for ambiguous projected poses.This correlation makes the subsequent regression task easier for critical samples.
  • 2D-to-3D Distance Matrix Regression: The proposed regressors use shallow Fully Connected or Fully Convolutional networks to map 14 × 14 2D EDMs to 14 × 14 3D EDMs.The Fully Connected model predicts the 91 entries above the diagonal, while the convolutional model processes entire matrices.
  • 2D-to-3D Distance Matrix Regression: Symmetrization and a final ReLU enforce symmetry and positivity in the Fully Convolutional output, while the Fully Connected design guarantees symmetry by construction.Symmetry and positivity are necessary structural properties of Euclidean Distance Matrices.

4. Experiments

The experiments evaluate the distance-matrix approach on Humaneva-I, Human3.6M, and LSP, including robustness to occlusions and noisy 2D detections. Results show strong benchmark performance, reasonable occlusion and noise tolerance, and promising but imperfect in-the-wild generalization.

  • The approach is evaluated on Humaneva-I and Human3.6M, with additional qualitative results on the LSP dataset.The evaluation includes quantitative comparisons, robustness tests, and qualitative assessment on images in the wild.
  • 4.1. Evaluation on Humaneva-I: All proposed Humaneva-I configurations significantly outperform state-of-the-art, while FConn and FConv perform similarly without occlusions.Training configurations vary by regressor architecture and 2D training source; FConv performs better under occlusions.
  • 4.1. Evaluation on Humaneva-I: FConv yields a clear and consistent advantage over FConn under occlusions, with hypothesized-joint errors generally within reasonable bounds.The networks are retrained using randomly occluded joint pairs and tested with random or structured limb occlusions.
  • 4.2. Evaluation on Human3.6M: On Human3.6M, the approach improves state-of-the-art considerably under Protocols #1 and #3 and performs similarly to [10] under Protocol #2.The experiments consider three evaluation protocols, with training and testing details differing by protocol.
  • 4.2. Evaluation on Human3.6M: Under Human3.6M occlusions, average body error remains within reasonable bounds, although some rare limb configurations are poorly hypothesized.The weaker cases include ‘Sit’ and ‘Photo’, where the relevant limb configurations have few training examples; other methods also perform poorly on these actions.
  • 4.2. Evaluation on Human3.6M: 3D error increases gradually with 2D noise but remains reasonably supported up to 20 pixels standard deviation; more precise 2D detections could improve accuracy.The mean 2D error of the CPM detections used in related evaluations is 10.91 pixels.
  • 4.3. Evaluation on Leeds Sports Pose Dataset: On LSP, reprojection errors provide indicative evidence of proper operation, while visual results generalize promisingly to poses absent from Human3.6M.Quantitative 3D-shape accuracy cannot be evaluated because LSP lacks 3D annotations, and failure cases remain.

5. Conclusion

The paper formulates 3D pose estimation as regression between 2D and 3D joint-distance matrices, using simple neural networks to improve robustness and generalization.

  • Distance matrices incorporate human-body structure and joint correlations, reducing the pose problem’s inherent ambiguity.
  • Simple neural network architectures regress 3D distance matrices from 2D distance matrices.
  • The approach is robust to noise and occlusions in detected 2D joints.
  • A Fully Convolutional network can hypothesize unobserved body parts.
  • Quantitative benchmarks show remarkable improvement over the state of the art, while in-the-wild results show good generalization to untrained data.
  • New training data from novel viewpoints and shape configurations can be readily synthesized from joint positions.
Loading 1611.09010v1…