Source-linked AI summary
Human Pose Regression by Combining Indirect Part Detection and Contextual Information
Diogo C. Luvizon, Hedi Tabia, David Picard
TL;DR
Still-image human pose estimation must handle articulated, occluded, and visually variable bodies, while detection methods require extra heat-map processing and regression methods can underperform them. The paper introduces Soft-argmax regression, which directly converts feature maps to joint coordinates in an end-to-end differentiable network and indirectly learns part-based maps with contextual information. Evaluations on LSP and MPII report performance surpassing regression methods and competitive with state-of-the-art detection approaches.
Problem
Still-image pose estimation is difficult, and existing detection methods require nondifferentiable heat-map processing while regression methods can underperform detection approaches.
Method
The method uses Soft-argmax to convert feature maps into joint coordinates within an end-to-end network that outputs joint probabilities and coordinates while aggregating part-based and contextual maps.
Results
The method surpasses existing regression methods and achieves results comparable to state-of-the-art detection approaches on LSP and MPII.
Takeaways & Limitations
Soft-argmax enables indirect part-based learning and seamless contextual integration while retaining the flexibility of regression-based pose estimation.
Takeaways & Limitations
Detection-based alternatives require high-resolution heat maps for precision, increasing computational cost and memory usage quadratically.
Abstract
from arXiv · showhide
In this paper, we propose an end-to-end trainable regression approach for human pose estimation from still images. We use the proposed Soft-argmax function to convert feature maps directly to joint coordinates, resulting in a fully differentiable framework. Our method is able to learn heat maps representations indirectly, without additional steps of artificial ground truth generation. Consequently, contextual information can be included to the pose predictions in a seamless way. We evaluated our method on two very challenging datasets, the Leeds Sports Poses (LSP) and the MPII Human Pose datasets, reaching the best performance among all the existing regression methods and comparable results to the state-of-the-art detection based approaches.
1. Introduction
Human pose estimation is difficult because bodies are articulated, occluded, and visually variable, while existing detection and regression approaches face complementary limitations. The paper proposes Soft-argmax regression to retain part-detector benefits in a fully differentiable, end-to-end framework.
- Articulation, occlusion, image quality, and changing body-part appearance make still-image pose estimation challenging.
- Detection methods require nondifferentiable heat-map conversion and high-resolution maps, increasing computational and memory demands.
- Regression methods map images directly to joint positions but can learn the regression function sub-optimally compared with detection approaches.
- Soft-argmax replaces argmax to preserve specialized part-detector properties while keeping the pipeline fully differentiable.
- The method avoids artificial heat-map generation, uses a regression loss linked to joint-position error, and aggregates contextual information into predictions.
- The authors report accuracy surpassing regression methods and approaching state-of-the-art detection methods despite using a much smaller network.
2. Related work
Prior work divides CNN-based pose estimation into regression and detection approaches, with detection methods relying on additional heat-map processing and computationally costly spatial resolution. The paper presents its framework as an alternative to these requirements.
- CNN-based human pose methods are commonly organized into regression-based and detection-based approaches.
- Regression based approaches: Regression methods directly estimate joint positions, using strategies such as cascaded refinement, temporal tracking, or iterative error feedback.
- Detection based approaches: Detection methods build pose predictions from detected body parts and use architectures including residual networks and stacked hourglass models.
- Detection based approaches: Detection-based methods require hand-defined artificial ground-truth heat maps during training and argmax conversion during evaluation.
- Detection based approaches: High spatial heat-map resolution improves precision but increases computational cost and memory usage quadratically.
3. Proposed method
The proposed method regresses joint coordinates end to end by applying Soft-argmax to part-based and contextual heat maps. It aggregates specialized and contextual predictions differentiably, while intermediate supervision supports training across stacked prediction blocks.
- Network architecture: The network takes RGB images and outputs joint-presence probabilities and regressed coordinates for each body joint.
- Network architecture: The architecture combines a Stem, refined-feature Block-A, and body-part/contextual-map Block-B across K prediction blocks with intermediate supervision.The final pose is produced by the Kth prediction block.
- Detection and context aggregation: Block-B produces joint-specific part maps and contextual maps, whose locations and probabilities are combined into final joint predictions.The contextual weighted contribution enables filters to respond to particular patterns while remaining differentiable inside the network.
- Soft-argmax layer: Soft-argmax converts spatially normalized heat maps into joint coordinates as a weighted average over a uniform coordinate grid.Unlike argmax, it provides subpixel accuracy and can learn discriminative heat maps directly from joint coordinates.
- Soft-argmax layer: The Soft-argmax derivative enables backpropagation, and its implementation uses spatial softmax followed by a fixed two-filter convolution.The gradient increases exponentially for higher heat-map values, producing a discriminative response at the joint position.
4. Experiments
Experiments on MPII and LSP evaluate the method’s training setup, robustness, and accuracy. The method achieves strong results across LSP protocols and MPII while indirectly learning localized part maps and using contextual maps to refine predictions.
- Datasets: The method was evaluated on the challenging MPII Human Pose and Leeds Sports Poses datasets.MPII contains around 28K annotated training poses and 15K test poses; LSP contains 2000 annotated sports poses.
- Training: Training jointly optimizes joint coordinates with elastic-net regression loss and joint probabilities with binary cross-entropy loss.Coordinates are normalized to [0, 1], with image corners mapped to (0, 0) and (1, 1).
- LSP results: On LSP observer-centric evaluations, the method exceeds the best scores by significant margins, including 6.3% for lower legs and 4.6% for ankles over Pishchulin et al.The evaluation uses PCK and PCP under observer-centric and person-centric protocols.
- LSP results: 18.0%: the method outperforms Carreira et al. on LSP PCK with person-centric annotations, achieving the best regression result and the second overall score.Carreira et al. is identified as the only reported regression method for this setup.
- MPII results: 91.2%: the MPII test score is 0.7% below the best detection-based result and 4.8% above the second regression score.The score was computed by the dataset providers because test labels are not publicly available.
- Discussion: Soft-argmax encourages indirectly learned part-based detectors, while contextual maps provide complementary information that can refine predicted joint positions.Strong contextual responses can shift predictions, whereas weak responses produce few changes; examples show increased precision for right elbows.
5. Conclusion
The proposed Soft-argmax regression method learns part-based detection maps indirectly and improves regression-based pose estimation while remaining competitive with detection-based approaches. Context maps and joint probabilities can be integrated into the framework to refine predictions.
- Soft-argmax enables differentiable learning of part-based detection maps within the regression method.The operation integrates into a deep convolutional network without requiring artificial heat-map generation for training.
- The method achieves a significant improvement over state-of-the-art regression scores and very competitive results compared to detection-based approaches.
- MPII testing results are compared using PCKh at a 0.5 head-segment-length threshold, with detection-based methods listed above regression-based methods.
- Context maps and joint probabilities can be seamlessly integrated to refine final pose predictions.Figure 6 depicts context-map aggregation from input and part-based detection maps through the final pose.