Source-linked AI summary
Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation
Zhengxiong Luo, Zhicheng Wang, Yan Huang, Tieniu Tan, Erjin Zhou
TL;DR
The paper addresses the mismatch between fixed Gaussian heatmap deviations and bottom-up pose estimation with varying human scales and ambiguous labels. It introduces SAHR to adapt deviations per keypoint and WAHR to rebalance foreground and background samples, achieving a reported +1.5AP over the state of the art and 72.0AP on COCO test-dev2017.
Problem
Fixed Gaussian standard deviations in heatmap regression are unreasonable for bottom-up pose estimation because human scales vary and keypoint labels have inherent ambiguities.
Method
SAHR adaptively adjusts each keypoint’s Gaussian-kernel standard deviation, while WAHR alleviates foreground-background imbalance by focusing more on harder examples.
Results
+1.5AP over the state-of-the-art model and 72.0AP on COCO test-dev2017 are reported for the proposed approach.
Takeaways & Limitations
SAHR together with WAHR largely improves the performance of the original bottom-up pose-estimation model and reaches performance comparable with most top-down methods.
Abstract
from arXiv · showhide
Heatmap regression has become the most prevalent choice for nowadays human pose estimation methods. The ground-truth heatmaps are usually constructed via covering all skeletal keypoints by 2D gaussian kernels. The standard deviations of these kernels are fixed. However, for bottom-up methods, which need to handle a large variance of human scales and labeling ambiguities, the current practice seems unreasonable. To better cope with these problems, we propose the scale-adaptive heatmap regression (SAHR) method, which can adaptively adjust the standard deviation for each keypoint. In this way, SAHR is more tolerant of various human scales and labeling ambiguities. However, SAHR may aggravate the imbalance between fore-background samples, which potentially hurts the improvement of SAHR. Thus, we further introduce the weight-adaptive heatmap regression (WAHR) to help balance the fore-background samples. Extensive experiments show that SAHR together with WAHR largely improves the accuracy of bottom-up human pose estimation. As a result, we finally outperform the state-of-the-art model by +1.5AP and achieve 72.0AP on COCO test-dev2017, which is com-arable with the performances of most top-down methods. Source codes are available at https://github.com/greatlog/SWAHR-HumanPose.
1. Introduction
Bottom-up pose estimation must handle varying human scales and ambiguous keypoint labels, but standard heatmap regression uses fixed Gaussian-kernel deviations. The paper proposes SAHR and WAHR to adapt kernel scales and rebalance samples, reporting improved COCO performance.
- Motivation: Bottom-up HPE detects all persons’ keypoints simultaneously but must handle large scale variation and labeling ambiguities.Unlike top-down methods, bottom-up methods do not resize each person to a common scale before keypoint detection.
- Motivation: Standard heatmap regression places identical Gaussian kernels on all keypoints, which can create semantic confusion across human scales.The same deviation may cover only a nose for one person but a face or whole head for another.
- Proposed methods: SAHR adaptively adjusts each keypoint’s Gaussian-kernel standard deviation to address scale variation and labeling ambiguities.The approach is motivated by relating each keypoint’s deviation to its scale and uncertainty rather than manually specifying deviations.
- Proposed methods: WAHR addresses SAHR’s foreground-background imbalance by emphasizing relatively harder examples.The paper presents WAHR as a way to help exploit SAHR’s benefits despite this imbalance.
- Results: +1.5AP over the state-of-the-art model and 72.0AP on COCO test-dev2017 are reported for the model using the proposed methods.The result is described as comparable with the performances of most top-down methods.
2. Related Works
Related work covers bottom-up grouping methods and heatmap regression. It highlights that fixed Gaussian deviations are less suitable when bottom-up methods process people at different scales, while uncertainty prediction addresses labeling ambiguity.
- Bottom-up HPE: Bottom-up HPE detects identity-free keypoints and groups them into people, avoiding human detectors and decoupling runtime from the number of persons.These methods may offer speed advantages but must handle grouping and large human-scale variance.
- Bottom-up HPE: Recent bottom-up HPE work primarily develops grouping methods, including intensity and association fields, heatmaps, and center-point offsets.This paper uses associative embedding for grouping.
- Heatmap regression: Heatmap regression constructs ground-truth maps with 2D Gaussian kernels and has become dominant for HPE.The method is easy to implement and can potentially attain pixel-wise accuracy.
- Heatmap regression: Using the same Gaussian standard deviation for every keypoint may suit resized top-down inputs but is less desirable for bottom-up inputs with varying person scales.The related-work discussion motivates adjusting deviations according to the corresponding person’s scale.
- Uncertainty prediction: Uncertainty prediction explicitly models inevitable labeling ambiguities in training datasets.Prior work represents labeled positions as distributions or predicts coordinates together with objecting scores.
3. Proposed Method
The proposed method adapts heatmap standard deviations to keypoint scale and uncertainty, then uses adaptive loss weighting to address foreground-background imbalance. These components are implemented with heatmap aggregation for bottom-up pose estimation.
- Scale-Adaptive Heatmap Regression: SAHR predicts scale maps and modifies a base standard deviation σ0 into σ0 · s for each keypoint.The scale maps have the same shape as the ground-truth heatmaps, and the modification is performed point-wise.
- Scale-Adaptive Heatmap Regression: Adaptive standard deviations enlarge or shrink Gaussian-covered regions according to predicted scale factors, which can reflect person scale and keypoint uncertainty.Factors above 1 produce larger deviations and covered regions; factors below 1 produce the reverse.
- Scale-Adaptive Heatmap Regression: SAHR adds a regularizer for predicted scale maps to stabilize training while retaining L2 heatmap regression.The regularizer constrains the predicted scale maps over regions covered by Gaussian kernels.
- Weight-Adaptive Heatmap Regression: WAHR defines adaptive loss weights for continuous heatmap values, using a soft boundary controlled by γ to distinguish relatively harder samples.For heatmap values above the threshold p, weights approach 1 − P; otherwise, they approach P, with γ = 0.01 in practice.
- Implementation Details: SWAHR combines SAHR and WAHR, while training scales ground-truth heatmaps and testing uses predicted heatmaps with associative embeddings for person grouping.The method is implemented on HrHRNet, which aggregates multi-scale heatmaps during testing.
4. Experiments
Experiments evaluate SWAHR on COCO across standard settings, ablations, scale analyses, visualizations, and comparisons with bottom-up and top-down methods. The results show consistent gains, with improvements linked to adaptive heatmap scales and foreground-background balancing.
- COCO keypoint detection: SWAHR achieves 72.0 AP on COCO test-dev2017 with multi-scale testing.
- Ablation study: +1.8AP is obtained when SAHR and WAHR are combined in the COCO val2017 ablation, compared with the HrHRNet-W32 baseline.
- Scale analysis: WAHR benefits medium and large persons, while SAHR mainly improves medium-person performance.The analysis attributes WAHR's broad gains to foreground-background imbalance at both scales and SAHR's gains to adjusting deviations for smaller persons.
- Scale-map visualization: Learned scale maps assign smaller scale factors to smaller persons and larger scale factors to relatively larger persons.The observed ordering of scale factors matches the ordering of person scales without extra supervision.
- Scale analysis: Increasing the Gaussian standard deviation worsens medium-person performance but improves large-person performance.This supports using larger σ for larger persons.
- COCO keypoint detection: SWAHR brings +5.7AP without multi-scale testing and +6.2AP with multi-scale testing on HrHRNet.
5. Conclusion
The paper addresses fixed-standard-deviation heatmap regression in bottom-up pose estimation by introducing adaptive scale and loss weighting. Together, SAHR and WAHR improve performance to 72.0AP on COCO test-dev2017 and exceed the state-of-the-art model by +1.5AP.
- SAHR learns per-keypoint Gaussian standard deviations to address varying human scales and labeling ambiguities.
- WAHR down-weights well-classified samples and focuses more on relatively harder, usually foreground, samples.
- +1.5AP over the state-of-the-art model and 72.0AP on COCO test-dev2017 are reported for the combined methods.