Source-linked AI summary
DeepPose: Human Pose Estimation via Deep Neural Networks
Alexander Toshev, Christian Szegedy
TL;DR
Human pose estimation must localize joints despite articulations, small or occluded joints, and contextual dependencies, while prior holistic methods had limited real-world success. DeepPose uses joint-coordinate DNN regression with a cascade, achieving state-of-art or better results across challenging datasets, including 0.78 PCP for upper legs versus 0.74 for the next-best method.
Problem
Human pose estimation requires localizing joints under articulations, small or occluded joints, and contextual dependencies, while holistic methods have had limited real-world success.
Method
DeepPose formulates pose estimation as joint-coordinate regression using a generic convolutional DNN and cascades pose predictors for refinement.
Results
0.78 PCP for upper legs versus 0.74 for the next-best method, with state-of-art or better performance across several challenging academic datasets.
Takeaways & Limitations
The approach captures contextual and holistic pose information while performing strongly across challenging limbs and diverse academic datasets.
Takeaways & Limitations
When some joints are unlabeled, the objective omits their corresponding terms rather than using them for training.
Abstract
from arXiv · showhide
We propose a method for human pose estimation based on Deep Neural Networks (DNNs). The pose estimation is formulated as a DNN-based regression problem towards body joints. We present a cascade of such DNN regressors which results in high precision pose estimates. The approach has the advantage of reasoning about pose in a holistic fashion and has a simple but yet powerful formulation which capitalizes on recent advances in Deep Learning. We present a detailed empirical analysis with state-of-art or better performance on four academic benchmarks of diverse real-world images.
1. Introduction
The paper frames human pose estimation as joint localization under articulation, occlusion, visibility, and context challenges. It proposes holistic DNN regression with a refinement cascade and reports state-of-the-art or better results across four benchmarks.
- Human pose estimation requires localizing joints despite strong articulations, small or barely visible joints, occlusions, and contextual dependencies.
- Part-based models efficiently handle articulations but sacrifice expressiveness by using local detectors and modeling only limited body-part interactions.
- The paper proposes a holistic pose-estimation algorithm based on a Deep Neural Network to address precise localization of articulated objects.
- Pose estimation is formulated as joint regression from the full image using a 7-layered generic convolutional DNN, enabling each regressor to capture complete joint context.
- A cascade refines initial full-image predictions with higher-resolution sub-images, improving joint-localization precision.
- The approach achieves state-of-art or better than state-of-art performance on four widely used benchmarks and generalizes across datasets and substantial appearance and articulation variation.
2. Related Work
Prior human-pose work commonly modeled articulated objects as graphs of parts, with pictorial-structure methods made tractable through the distance transform. Subsequent research enriched model expressiveness and explored holistic pose reasoning, while neural-network approaches had also been applied to facial-point localization and face-pose estimation.
- Part-based models: Pictorial-structure models represent articulated objects as graphs of parts and became practical through the distance transform trick.This enabled a wide variety of practically significant pictorial-structure models.
- Richer pose models: Tree-based pose models with simple binary potentials independent of image data motivated richer part detectors and models expressing complex joint relationships.Examples include mixture models of parts, although the passage truncates the broader list of proposed approaches.
- Holistic reasoning: Holistic pose-reasoning approaches included exemplar retrieval with joint transfer, locality-sensitive-hashing nearest neighbors, and a semi-global classifier for part configuration.The passage characterizes these approaches as having limited practicality.
- Neural-network approaches: DNN regressors had been used to localize facial points, while face-pose estimation used a neural-network pose embedding trained with a contrastive loss.These methods address facial-point localization and the related problem of face pose estimation.
3. Deep Learning Model for Pose Estimation
The model formulates pose estimation as regression to normalized joint coordinates using a convolutional DNN, then improves localization precision through a cascade of refinements on joint-centered sub-images.
- DNN-based pose regression: Pose estimation is formulated as regression from an image to a normalized 2k-dimensional pose vector of joint coordinates.Joint locations are normalized relative to a bounding box, and predictions can be transformed back to absolute image coordinates.
- DNN-based pose regression: The convolutional DNN learns pose features directly from data, with shared internal features implicitly capturing interactions among joints.The network uses seven hidden layers, and all internal features are shared across joint regressors.
- Training: Training minimizes L2 distance between predicted and true normalized pose vectors, using adaptive updates with mini-batches of 128 and learning rate 0.0005.The method also supports images with unlabeled joints by omitting the corresponding loss terms.
- Cascaded pose regression: The fixed 220 × 220 input captures context but limits fine detail, motivating a cascade of regressors that progressively predicts joint displacements toward the true locations.Each subsequent stage refines the pose estimated by the preceding stage.
- Cascaded pose regression: Each cascade stage crops a higher-resolution sub-image around each previously predicted joint and applies a joint-specific displacement regressor for finer-scale localization.All stages use the same architecture but learn different network parameters.
4. Empirical Evaluation
DeepPose is evaluated on challenging FLIC and LSP benchmarks using PCP and PDJ, outperforming prior methods while generalizing across movie and sports imagery. Cascade refinement substantially improves high-precision joint localization, especially after the first refinement stage.
- Datasets: FLIC contains 4,000 training and 1,000 test images with 10 upper-body joints, while LSP contains 11,000 training and 1,000 testing images with 14 full-body joints.LSP images primarily depict sports activities, challenging articulation and appearance, with most people approximately 150 pixels tall.
- Metrics: The evaluation uses PCP for limb detection and PDJ for joint detection across varying localization thresholds relative to torso diameter.PDJ applies a common distance threshold across joints, addressing PCP’s differing limb-based detection criteria.
- Comparisons: DeepPose performs well on both appearance-heavy movie images and strongly articulated sports images, indicating good generalization across the two datasets.Qualitative examples show correct poses under upside-down views and severe foreshortening, although imprecise estimates often preserve the overall pose shape.
- Effects of cascade-based refinement: Cascade refinement produces its largest gains at normalized distances of 0.15–0.2, with most improvement achieved after one refinement stage.The initial regressor generally estimates a rough pose, while the second stage often resolves scale, translation, and joint-alignment errors; later stages trade context for resolution.
5. Conclusion
The paper presents DNN-based regression and cascaded regressors for holistic human pose estimation, achieving state-of-art or better results on several challenging academic datasets. It also demonstrates that a generic classification CNN can be applied to localization and identifies tailored architectures as future work.
- 5. Conclusion: The method formulates human pose estimation as DNN-based regression to joint coordinates and uses a cascade of regressors for holistic pose reasoning.The formulation captures context while reasoning about pose in a holistic manner.
- 5. Conclusion: State-of-art or better results are achieved on several challenging academic datasets.
- 5. Conclusion: A generic convolutional neural network originally designed for classification is shown to apply to localization.
- 5. Conclusion: Future work will investigate architectures better tailored to localization problems generally and pose estimation specifically.