Source-linked AI summary
Structured Prediction of 3D Human Pose with Deep Neural Networks
Bugra Tekin, Isinsu Katircioglu, Mathieu Salzmann, Vincent Lepetit, Pascal Fua
TL;DR
Monocular 3D pose estimation is challenged by depth ambiguity and by methods that either overlook joint dependencies or require expensive inference. The paper combines CNN regression with an overcomplete auto-encoder that learns a high-dimensional structured pose representation, and reports better structure preservation and prediction accuracy than state-of-the-art methods.
Problem
Monocular 3D pose estimation is difficult because projection loses depth information, while direct CNN regression can ignore dependencies between human joints and max-margin modeling is costly at inference.
Method
The method pretrains an overcomplete auto-encoder on human poses, maps CNN image features to its high-dimensional latent representation, and decodes the representation into 3D joint locations.
Results
The approach outperforms state-of-the-art methods on the Human3.6m dataset in prediction accuracy and structure preservation.
Takeaways & Limitations
Combining CNNs with auto-encoders accounts for dependencies between human body parts while retaining efficient regression-based prediction.
Abstract
from arXiv · showhide
Most recent approaches to monocular 3D pose estimation rely on Deep Learning. They either train a Convolutional Neural Network to directly regress from image to 3D pose, which ignores the dependencies between human joints, or model these dependencies via a max-margin structured learning framework, which involves a high computational cost at inference time. In this paper, we introduce a Deep Learning regression architecture for structured prediction of 3D human pose from monocular images that relies on an overcomplete auto-encoder to learn a high-dimensional latent pose representation and account for joint dependencies. We demonstrate that our approach outperforms state-of-the-art ones both in terms of structure preservation and prediction accuracy.
1 Introduction
Monocular 3D pose estimation remains ambiguous, while existing structured methods either ignore joint dependencies or require costly inference. The paper proposes combining CNN regression with an overcomplete auto-encoder to model pose structure efficiently.
- Monocular images make 3D pose estimation difficult because occlusions, complex backgrounds, and lost depth information create reconstruction ambiguities.
- Max-margin structured methods model joint dependencies effectively but require solving a computationally expensive optimization problem at test time.
- The proposed model pretrains an overcomplete auto-encoder, maps CNN image features into its high-dimensional latent pose representation, and decodes them into pose space.
- The latent representation encodes dependencies between body parts, implicitly enforces pose constraints, preserves human body statistics, and supports end-to-end fine-tuning.
- Combining CNNs for supervised learning with auto-encoders for structured learning preserves CNN power while accounting for dependencies and increasing performance.
2 Related Work
Prior 3D pose methods use deep regression or structured representations to address the ambiguities caused by projecting 3D poses into 2D images. Auto-encoder and max-margin approaches add structure but retain important data, feature, or inference limitations.
- Deep learning methods for 3D pose estimation typically regress continuous 3D joint locations because discretizing 3D space is more difficult than discretizing 2D space.
- Projection from 3D to 2D introduces ambiguities that motivate modeling dependencies between body joints within deep learning approaches.
- An earlier auto-encoder method learned a shared representation for silhouettes and 3D poses but required accurate foreground masks and handcrafted features.
- The strongest prior deep structured method extended structured SVMs by learning similarity scores between image and pose embeddings, but required costly test-time pose search.
- Denoising and contractive auto-encoders had shown useful generalization or robustness, but prior work used auto-encoders primarily for feature learning.
3 Method
The method encodes 3D human-pose joint dependencies in an overcomplete auto-encoder’s latent representation, then trains a CNN to regress images into that space and fine-tunes the complete network for pose prediction.
- 3 Method: 3D human pose is represented as J joint locations relative to a root joint, rather than joint angles and limb lengths.The pose vector is y ∈ R3J.
- 3 Method: An overcomplete denoising auto-encoder learns a high-dimensional latent pose representation that encodes dependencies between human joints.Its middle hidden layer has higher dimension than the input and output and is denoted hL.
- 3 Method: The auto-encoder is trained on noisy pose vectors to reconstruct the original denoised poses using reconstruction loss and Jacobian regularization.The Jacobian penalty is used to improve robustness to small pose changes.
- 3 Method: A CNN maps input images to the auto-encoder’s learned latent pose representation using a squared loss over training samples.The CNN uses a standard convolutional architecture and linearly maps its image representation to the latent pose embedding.
- 3 Method: The auto-encoder’s decoding layers are appended to the CNN to reproject latent estimates into pose space, after which the complete network is fine-tuned.Fine-tuning minimizes squared differences between predicted and ground-truth 3D poses.
4 Results
The method was evaluated on Human3.6m using monocular training and testing, and consistently outperformed competing approaches while preserving pose structure. Auto-encoder variants outperformed direct CNN regression, with fine-tuning further improving results.
- Evaluation protocol: The dataset contains 3.6 million image frames with corresponding 2D and 3D poses across complex activities and 4 camera viewpoints.Images were cropped around subjects and resized to 128×128, with outputs represented by 17 3D joint coordinates.
- Evaluation protocol: Human3.6m evaluation used 5 subjects for training, 2 different subjects for testing, and average Euclidean joint distance in millimeters.Training and testing were performed monocularly across all camera views for each separate action.
- Comparison with prior methods: Our method consistently outperforms KDE regression, joint detector-regressor networks, and max-margin structured prediction on Human3.6m.The max-margin baseline used either the highest-scoring pose or an average of the 500 highest-scoring training poses.
- Ablation studies: Both one- and two-layer auto-encoders perform similarly and better than CNN-Direct, especially for Taking Photo and Walking Dog.These actions involve environmental interactions and are therefore more physically constrained.
- Ablation studies: Fine-tuning consistently improves the results, while the overcomplete auto-encoder outperforms added fully connected layers and PCA-based latent representations.For PCA, the 40-dimensional configuration performed best, but the 40-dimensional auto-encoder performed better.
- Structure preservation: Predictions from the proposed method better preserve ground-truth limb ratios than KDE and CNN-Direct, indicating better modeling of joint dependencies.The comparison uses differences between ground-truth and predicted limb ratios.
5 Conclusion
The paper presents a generic auto-encoder/CNN architecture for structured monocular 3D human-pose prediction, improving dependency modeling and prediction accuracy. Limb-ratio evaluations and configuration validation support its reported effectiveness and reuse across actions.
- Conclusion: The approach combines auto-encoders with CNNs to account for dependencies between human body parts efficiently while improving prediction accuracy over state-of-the-art methods.The framework is presented as generic and intended for structured prediction beyond human pose.
- Conclusion: Our method outperforms other methods on upper-body and full-body limb-ratio errors, although all methods perform well on the lower body.Figure 4 compares KDE, CNN-Direct, and the proposed method with and without fine-tuning.