Source-linked AI summary
3D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Information
Sungheon Park, Jihye Hwang, Nojun Kwak
TL;DR
3D human pose estimation from a single image remains challenging because pose varies dynamically and images contain occlusion and viewpoint variation, while prior CNN methods did not use 2D pose information. This paper uses CNN regression with 2D joint classification results and multiple-root relative 3D regression, achieving performance comparable to state-of-the-art methods without temporal video information.
Problem
3D human pose estimation from single images is challenging, and prior CNN methods did not use 2D pose information that could help discard unnatural 3D joint positions.
Method
The framework uses CNN regression that combines image features with 2D joint classification results and estimates relative 3D positions with respect to multiple root joints.
Results
The combined framework achieves comparable performance to state-of-the-art methods on Human 3.6m without exploiting temporal information from video sequences.
Takeaways & Limitations
Reusing 2D joint classifications and combining multiple-root regression results significantly improves CNN-based 3D pose estimation from single images.
Takeaways & Limitations
The authors suggest that incorporating temporal information and more efficient alignment of multiple regression results could further improve accuracy.
Abstract
from arXiv · showhide
While there has been a success in 2D human pose estimation with convolutional neural networks (CNNs), 3D human pose estimation has not been thoroughly studied. In this paper, we tackle the 3D human pose estimation task with end-to-end learning using CNNs. Relative 3D positions between one joint and the other joints are learned via CNNs. The proposed method improves the performance of CNN with two novel ideas. First, we added 2D pose information to estimate a 3D pose from an image by concatenating 2D pose estimation result with the features from an image. Second, we have found that more accurate 3D poses are obtained by combining information on relative positions with respect to multiple joints, instead of just one root joint. Experimental results show that the proposed method achieves comparable performance to the state-of-the-art methods on Human 3.6m dataset.
1 Introduction
3D human pose estimation from a single image remains less studied and more difficult than 2D estimation because depth is unavailable and poses vary dynamically. The paper proposes a CNN framework that combines 2D pose information with multiple relative 3D position estimates.
- Motivation: Single-image human pose estimation must handle occlusion, viewpoint variation, body dynamics, skin colors, and clothing differences.These challenges make reliable pose recovery difficult even though the resulting information supports applications such as action recognition and markerless motion capture.
- Research gap: 3D human pose estimation with CNNs has been studied less thoroughly than 2D estimation and is harder because single images lack depth information.
- Research gap: Existing CNN-based 3D methods do not use 2D pose information, which can help discard 3D joint positions that produce unnatural poses.
- Approach: The proposed framework regresses joint positions with CNNs while adding 2D joint classification results to image features for 3D pose estimation.
- Approach: Relative 3D positions are estimated with respect to multiple joints rather than only one root joint, reducing errors for joints far from the root.
- Evaluation: The framework is evaluated on Human 3.6m without temporal information and achieves performance comparable to state-of-the-art methods.
2 Related Work
Prior work moved from hand-crafted features toward CNNs for 2D and 3D pose estimation, with 3D methods commonly formulated as regression. This paper extends CNN-based 3D estimation by propagating 2D classification results and using relative distances from multiple joints.
- 2D pose estimation: Early 2D pose methods used hand-crafted features, whereas CNN-based approaches substantially improved performance through learned representations.
- 2D pose estimation: Later 2D CNN systems improved predictions through iterative refinement, confidence-map feedback, self-correction, and joint feature relationships.
- 3D pose estimation: Early 3D pose methods relied on low-level features and formulated estimation as regression using relevance vector machines, structured SVMs, or random forests.
- 3D pose estimation: Because 3D search spaces are larger than 2D image spaces, CNN-based 3D human pose estimation is often treated as regression rather than classification.
- This paper: This paper jointly learns 2D and 3D pose in one CNN and directly propagates 2D classification results to 3D pose regressors.
- This paper: The method combines 2D classification results with relative distances from multiple joints to improve 3D pose estimation over a baseline.
3 3D-2D Joint Estimation of Human Body Using CNN
The proposed CNN jointly estimates 2D joint locations and relative 3D positions, then improves 3D regression by using 2D classification features and multiple root joints. Its final pose averages translated predictions from the multiple regression branches.
- Joint 2D-3D estimation: The CNN jointly trains 2D joint classification and 3D joint regression using shared image features.The network accepts 225 × 225 images and uses separate fully connected branches for the two tasks.
- 2D pose classification: 2D pose classification divides each input image into Ng × Ng grids, with each grid treated as a class for each joint.The ground-truth position is represented with a soft label over nearby grids rather than a single hard class.
- 2D pose classification: The soft-label target assigns nonzero probability to the four nearest grids, weighted inversely by distance and normalized before cross-entropy training.The indicator function selects the four neighboring grids, while the CNN softmax supplies the predicted class probabilities.
- 2D features for 3D regression: The 3D branch concatenates 2D softmax outputs with image features so joint-location information contributes directly to 3D pose regression.The design exploits the relationship between 2D joint locations and 3D pose while retaining shared convolutional features.
- End-to-end training: The framework is trained end-to-end, and the authors report successful 3D-loss convergence with improved 3D pose-estimation performance.This improvement was observed despite potentially inaccurate 2D predictions early in training.
- Multiple-root regression: Instead of one hip-area root, the method estimates relative 3D positions from six selected root joints and uses separate fully connected layers for the regression losses.The multiple roots are chosen so most joints are roots or neighboring joints, and the resulting pose estimates are translated and averaged at test time.
4 Implementation Details
The implementation preprocesses and augments input images, trains the CNN with regularization and scheduled optimization, and normalizes 3D poses to reduce scale ambiguity.
- Network regularization: The CNN uses batch normalization throughout and dropout with probability 0.3 in every fully connected layer.
- Optimization: Training uses stochastic gradient descent with batch size 128, initial learning rate 0.01, halving every four epochs, and termination after 28 epochs.
- Input preprocessing: Input images are centered through segmentation-based cropping, resized to 250 × 250, randomly cropped to 225 × 225 for training, and center-cropped at test time.PCA-based data augmentation is also applied.
- Pose normalization: Training poses are translated to zero mean and scaled to unit Frobenius norm, with scale recovered during testing for evaluation.The normalization is intended to reduce scale ambiguity and produce scale-invariant poses.
5 Experimental Results
Experiments on Human 3.6m evaluate the method against prior approaches and a baseline, showing strong single-image performance and benefits from both proposed modifications. Integrating 2D classification information particularly improves test behavior and qualitative 3D pose estimates.
- Dataset and protocol: The method is evaluated on Human 3.6m using synchronized RGB images and motion-capture 3D poses across 15 action sequences and seven subjects.Five subjects are used for training and two for testing, with MPJPE as the evaluation metric.
- Comparison with prior work: The method achieves the best performance in 3 sequences and the second-best performance in 9 sequences on Human 3.6m.The comparison includes methods using temporal information from multiple frames, whereas this method estimates a 3D pose from a single image.
- Comparison with prior work: The method outperforms CNN-based methods that predict 3D pose from a single image.Its estimation uses a forward pass of the CNN followed by simple averaging.
- Ablation analysis: Both 2D classification integration and multiple-root regression improve results over the baseline CNN in all tested sequences.The 2D classification integration produces a larger error reduction than multiple regression, while multiple regression acts as an ensemble of estimation results.
- Loss analysis: On the Walking sequence, test loss is much lower when 2D classification information is used, although training loss is slightly smaller without it.The authors interpret this as improved generalization and reduced overfitting, with 2D joint probabilities providing more abstract and subject-independent information than image features.
- Qualitative analysis: Qualitative results show that adding 2D classification information reduces 3D pose error compared with the CNN without that information.The comparison visualizes input images, ground-truth poses, and estimates with and without 2D classification information.
6 Conclusions
The method improves CNN-based 3D human pose estimation by reusing 2D joint classifications and combining regressions from multiple root nodes. Combined, these strategies achieve comparable performance to state-of-the-art methods without temporal video information, while temporal modeling and more efficient alignment remain potential improvements.
- Reusing 2D joint classification results implicitly learns the relationship between 2D and 3D poses during training.
- Multiple regression results with different root nodes provide an ensemble-learning effect for 3D pose estimation.
- Combined strategies significantly improve 3D pose estimation and achieve comparable performance to state-of-the-art methods without temporal video information.
- Temporal information and more efficient alignment of multiple regression results are identified as potential routes for further accuracy improvements.