Source-linked AI summary
Human pose estimation via Convolutional Part Heatmap Regression
Adrian Bulat, Georgios Tzimiropoulos
TL;DR
Accurate localization of occluded body parts remains difficult in human pose estimation despite strong CNN performance on visible parts. The paper introduces a detection-followed-by-regression CNN cascade that stacks part heatmaps with image features to model context and part relationships. Its VGG-based and residual variants achieve top performance on the MPII and LSP datasets.
Problem
Accurately estimating occluded body-part locations remains a difficult open problem in human pose estimation.
Method
A two-stage CNN cascade detects body parts as heatmaps, then jointly regresses stacked heatmaps and image features into confidence maps for visible and occluded parts.
Results
Both VGG-based and residual cascade variants achieve top performance on the MPII and LSP datasets.
Takeaways & Limitations
Low-confidence heatmaps for occluded parts guide regression toward contextual information from the remaining parts, supporting pose inference under severe occlusion.
Abstract
from arXiv · showhide
This paper is on human pose estimation using Convolutional Neural Networks. Our main contribution is a CNN cascaded architecture specifically designed for learning part relationships and spatial context, and robustly inferring pose even for the case of severe part occlusions. To this end, we propose a detection-followed-by-regression CNN cascade. The first part of our cascade outputs part detection heatmaps and the second part performs regression on these heatmaps. The benefits of the proposed architecture are multi-fold: It guides the network where to focus in the image and effectively encodes part constraints and context. More importantly, it can effectively cope with occlusions because part detection heatmaps for occluded parts provide low confidence scores which subsequently guide the regression part of our network to rely on contextual information in order to predict the location of these parts. Additionally, we show that the proposed cascade is flexible enough to readily allow the integration of various CNN architectures for both detection and regression, including recent ones based on residual learning. Finally, we illustrate that our cascade achieves top performance on the MPII and LSP data sets. Code can be downloaded from http://www.cs.nott.ac.uk/~psxab5/
1 Introduction
Human pose estimation must handle diverse appearances, occlusions, many feasible poses, and nearby people, while modeling discriminative features and complex part relationships. The paper addresses difficult occluded-part localization with an end-to-end CNN cascade that combines part heatmaps, contextual information, and flexible CNN backbones.
- 1 Introduction: Human pose estimation is difficult because poses, appearances, occlusions, and multi-person configurations vary substantially.The problem includes foreshortening, clothing changes, self-occlusion, and people in close proximity.
- 1 Introduction: Accurately localizing occluded body parts remains an open problem despite CNNs' strong localization performance on visible parts.The paper’s main contribution is designed specifically to alleviate this difficulty.
- 1 Introduction: The proposed cascade first detects individual body parts as heatmaps, then jointly regresses stacked heatmaps and image features into body-part confidence maps.The detection network produces N heatmaps, while the regression subnetwork predicts part locations.
- 1 Introduction: Stacked part heatmaps guide where the regression network focuses and encode structural relationships among body parts.For occluded parts, low-confidence heatmaps encourage reliance on contextual information from remaining parts.
- 1 Introduction: The cascade is end-to-end trainable and supports both VGG-FCN and residual CNN architectures, with both achieving top performance on MPII and LSP.This flexibility allows the detection and regression subnetworks to use different CNN architectures.
2 Closely Related Work
Earlier pose-estimation methods used pictorial structures, graphical models, sequential prediction, or CNN-based detection and regression. The paper distinguishes its heatmap cascade from prior approaches by combining detection heatmaps with subsequent joint regression and broader spatial context.
- 2 Closely Related Work: CNN pose-estimation methods are commonly detection-based or regression-based, with confidence-map regression replacing direct L2 regression in notable approaches.Detection methods combine or refine part detections, whereas regression methods learn mappings from images and CNN features to part locations.
- 2 Closely Related Work: Compared with a prior regression-followed-by-regression method, this paper learns its first-stage part heatmaps with pixelwise sigmoid cross entropy and provides larger spatial context.The authors report that prior pre-confidence maps were too localized for their intended use.
- 2 Closely Related Work: Unlike related detection-and-refinement methods, the proposed cascade includes a subsequent part heatmap regression network rather than only local refinement around crude estimates.This design retains contextual information for refinement instead of restricting processing to cropped local regions.
- 2 Closely Related Work: Residual learning had recently been applied to pose estimation, including part detection and stacked hourglass architectures, before its integration into this cascade.The paper presents residual architectures as one option within its detection-and-regression framework.
3 Method
The method uses a two-stage CNN cascade: part detection produces heatmaps, and regression combines them with image features to predict visible and occluded part locations. The paper implements both VGG-FCN and residual variants, with training and architecture choices tailored to spatial context and localization.
- Cascade design: The cascade first detects body parts as N heatmaps, then jointly regresses stacked heatmaps and image/CNN features into confidence maps for all parts.The detection stage is trained with a per-pixel classification loss, while the regression stage predicts confidence maps rather than single coordinates.
- Part detection: Binary part maps mark pixels within a selected radius around each ground-truth location, separating nearby or overlapping body parts.The radius is chosen to include the target part; 10px was empirically selected for a 200px upright person.
- Regression: The regression input combines detection heatmaps with the image, guiding attention and encoding structural relationships between parts.Low-confidence heatmaps for occluded parts encourage reliance on contextual information from the remaining parts.
- VGG-FCN implementation: The VGG-FCN detector reduces localization stride from 32px to 8px by combining earlier CNN features, while its regression network uses seven convolutional layers with large kernels for context.The first four regression layers use kernels from 7 to 15, and the final three use 1x1 kernels.
- Training: Training uses staged optimization: detectors are trained first, then the regression subnetwork is trained with detector layers frozen to accelerate training and avoid early divergence.MPII training uses augmentation by flipping, scaling between 0.7 and 1.3, and rotation between -40 and 40 degrees; LSP fine-tuning uses 1,000 training images.
- Residual implementation: The residual detector is based on ResNet-152, with modified stride and deconvolution for pixel-level predictions; the residual regression network uses a modified hourglass architecture.The residual detector removes fully connected and average-pooling layers, adds an N-output scoring layer, and upsamples its output.
4 Results
Experiments on MPII and LSP evaluate cascade components, architectural choices, and comparisons with prior state-of-the-art methods. The results support detection-followed-by-regression, stacked part heatmaps, and residual architectures.
- Overview: The experiments evaluate cascade variants and compare the method with published state-of-the-art approaches on MPII and LSP.Table 5 reports component analyses on the MPII validation set using PCKh; Tables 6 and 7 summarize state-of-the-art comparisons.
- Overview: The detection-followed-by-regression cascade outperforms a comparable two-step regression approach with both VGG-FCN and residual architectures.The paper reports this benefit as part of its experiments on the two challenging datasets.
- Analysis: Stacking part heatmaps as regression input is necessary for high performance and significantly improves over the part detection network alone.Residual detection alone works well, but the regression subnetwork provides a large performance boost.
- Analysis: Residual part heatmap regression has a large performance advantage over the same cascade implemented with VGG-FCN.The analysis also reports a benefit from using residual architectures over VGG-FCN.
- Analysis: Direct regression alone performs better than detection alone, but the complete cascade significantly outperforms the two-step regression approach.The proposed part heatmap regression was also considerably easier to train than the two-step regression approach.
- Comparison with state-of-the-art: Both VGG-based and residual part heatmap regression are competitive, with the residual version and other residual architectures among the top performers on both datasets.A non-residual method using six sequential CNN regressions is reported as very close in performance but more challenging to train.
6 Conclusions
The paper concludes that its CNN cascade learns part relationships and spatial context for pose inference under severe occlusion. It is simple, end-to-end trainable, and achieves top performance on MPII and LSP, although examples include failures under extreme occlusion and rare poses.
- Conclusions: The CNN cascade is designed to learn part relationships and spatial context while inferring pose under severe part occlusions.Its key feature is joint regression of part detection heatmaps.
- Conclusions: Both cascade versions achieve top performance on the MPII and LSP datasets.The conclusion reports this outcome for the VGG-FCN and residual architectures.
- Conclusions: Examples show good handling of occlusions and difficult poses, but combinations of extreme occlusion and rare poses produce failure cases.The failures are shown in the last row of the figure examples.