Source-linked AI summary
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation
Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S. Huang, Lei Zhang
TL;DR
Bottom-up pose estimation struggles with scale variation, especially for small persons, while top-down methods remain computationally intensive and not truly end-to-end. HigherHRNet addresses this with high-resolution feature pyramids, multi-resolution supervision, and heatmap aggregation, outperforming existing bottom-up methods on COCO and CrowdPose.
Problem
Bottom-up multi-person pose estimation has difficulty handling person-scale variation, particularly for small persons, while top-down alternatives require separate detection and per-person pose estimation.
Method
HigherHRNet builds an even higher-resolution feature pyramid on HRNet and uses multi-resolution supervision during training with multi-resolution heatmap aggregation during inference.
Results
HigherHRNet outperforms all existing bottom-up methods on COCO and achieves 67.6% AP on the CrowdPose test set.
Takeaways & Limitations
The results support HigherHRNet as an effective approach for scale variation in bottom-up pose estimation, with particular gains for medium persons and performance in crowded scenes.
Takeaways & Limitations
The default COCO design uses one deconvolution module because adding a second reduced performance from 66.9 AP to 66.5 AP, including a −0.8 AP drop for large persons.
Abstract
from arXiv · showhide
Bottom-up human pose estimation methods have difficulties in predicting the correct pose for small persons due to challenges in scale variation. In this paper, we present HigherHRNet: a novel bottom-up human pose estimation method for learning scale-aware representations using high-resolution feature pyramids. Equipped with multi-resolution supervision for training and multi-resolution aggregation for inference, the proposed approach is able to solve the scale variation challenge in bottom-up multi-person pose estimation and localize keypoints more precisely, especially for small person. The feature pyramid in HigherHRNet consists of feature map outputs from HRNet and upsampled higher-resolution outputs through a transposed convolution. HigherHRNet outperforms the previous best bottom-up method by 2.5% AP for medium person on COCO test-dev, showing its effectiveness in handling scale variation. Furthermore, HigherHRNet achieves new state-of-the-art result on COCO test-dev (70.5% AP) without using refinement or other post-processing techniques, surpassing all existing bottom-up methods. HigherHRNet even surpasses all top-down methods on CrowdPose test (67.6% AP), suggesting its robustness in crowded scene. The code and models are available at https://github.com/HRNet/Higher-HRNet-Human-Pose-Estimation.
1. Introduction
Bottom-up pose estimation struggles with scale variation, especially for small persons. HigherHRNet addresses this with high-resolution feature pyramids and achieves strong results on COCO and CrowdPose.
- Motivation: Bottom-up methods must handle scale variation directly, unlike top-down methods that normalize detected person regions.This scale difference contributes to a performance gap, especially for small persons.
- Motivation: Small-person pose estimation requires both scale-robust representations and high-resolution heatmaps for precise keypoint localization.Earlier bottom-up methods commonly used a single feature-map resolution and neglected scale variation.
- Approach: HigherHRNet combines a high-resolution feature pyramid, multi-resolution supervision, and multi-resolution heatmap aggregation to produce scale-aware predictions.The method builds its pyramid on HRNet’s 1/4-resolution path and adds higher-resolution maps through deconvolution.
- Results: 67.6% AP is achieved on CrowdPose test, surpassing existing methods and suggesting robustness in crowded scenes.The authors characterize this result as a new state-of-the-art result for the dataset.
2. Related works
Related work covers top-down and bottom-up pose estimation, feature pyramids for scale variation, and architectures for producing high-resolution feature maps. HigherHRNet combines HRNet with deconvolution for efficient high-resolution heatmap prediction.
- Pose estimation methods: Top-down methods detect person boxes first and estimate pose within each box, while bottom-up methods detect identity-free joints and group them into individuals.Top-down systems generally reduce scale sensitivity but require person detection and per-person pose estimation.
- Pose estimation methods: Bottom-up systems use grouping mechanisms such as part affinity fields, associative embedding, and intensity or association fields.These methods differ in how they link detected joints into complete human poses.
- Feature pyramids: Feature pyramids predict instances at multiple resolutions so features at different levels can handle different object scales.FPN recovers resolution through a top-down pathway with bilinear upsampling and lateral connections.
- High-resolution features: High-resolution feature maps can be generated using encoder-decoder architectures, dilated convolutions, and related resolution-preserving designs.These approaches trade off context aggregation, feature resolution, and computational structure in different ways.
- HigherHRNet: HigherHRNet adopts HRNet as its backbone and adds deconvolution to generate higher-resolution feature maps for heatmap prediction.The authors describe both components as efficient for producing high-resolution features.
3. Higher-Resolution Network
HigherHRNet extends HRNet with deconvolution-based higher-resolution feature maps, multi-resolution supervision, and heatmap aggregation. These components target precise localization and scale variation in bottom-up pose estimation.
- HigherHRNet architecture: HigherHRNet uses HRNet as its backbone and preserves parallel feature branches at multiple resolutions.HRNet begins with a high-resolution branch and adds lower-resolution branches in later stages.
- HigherHRNet architecture: Higher-resolution heatmaps address localization confusion caused by Gaussian-smoothed targets, especially for small-person keypoints.Reducing the Gaussian standard deviation instead makes optimization harder and worsens results.
- HigherHRNet architecture: The deconvolution module takes HRNet features and predicted heatmaps as input and produces feature maps twice the input resolution.Its outputs form a feature pyramid and also predict heatmaps through an additional 1 × 1 convolution.
- Design choices: A single deconvolution module performs best on COCO, while the required number of modules depends on the dataset’s person-scale distribution.Datasets containing smaller persons generally require larger-resolution feature maps.
- Multi-resolution supervision: Multi-resolution supervision transforms keypoint targets to every pyramid resolution and applies the same Gaussian standard deviation across resolutions.The heatmap loss sums mean squared errors between each predicted heatmap and its corresponding target.
- Multi-resolution supervision: The model trains tagmaps only at 1/4 input resolution because tagmap learning requires global reasoning and higher resolutions do not converge well.Heatmaps, unlike tagmaps, are trained across multiple resolutions.
- Multi-resolution aggregation: During inference, predicted heatmaps from all resolutions are upsampled to input resolution and averaged for final prediction.This aggregation is intended to support scale-aware pose estimation across varying person sizes.
4. Experiments
Experiments evaluate HigherHRNet on COCO2017 through benchmark comparisons and component ablations. Results show that higher-resolution, scale-aware prediction improves medium-person performance while preserving large-person accuracy, with one deconvolution module preferred for COCO.
- COCO Keypoint Detection: 70.5 AP on COCO2017 test-dev with multi-scale test surpasses all existing bottom-up methods without refinement or post-processing.Single-scale HigherHRNet reaches 66.4 AP and improves over HRNet by +2.3 AP with only marginal parameter and FLOPs increases.
- COCO Keypoint Detection: HigherHRNet further closes the performance gap between bottom-up and top-down methods on COCO2017 test-dev.
- HRNet vs. HigherHRNet: 66.9 AP with one deconvolution module and heatmap aggregation improves over HRNet by +2.5 AP, mainly through medium-person gains.APM increases from 57.1 for HRNet to 61.0 for HigherHRNet, while large-person AP does not drop.
- Ablation Experiments: One deconvolution module is preferred for COCO: two modules reduce performance to 66.5 AP from 66.9 AP, while larger training images mainly benefit medium persons.Increasing training size to 640 produces a 1.4 AP gain; increasing it further to 768 does not change overall AP.
- HigherHRNet Gain Breakdown: 66.0 AP after adding one deconvolution module improves the 64.4 AP HRNet baseline by 1.6 AP through higher-resolution heatmaps.The ablation attributes the improvement to prediction on larger feature maps with higher quality.
- HigherHRNet Gain Breakdown: 66.3 AP after feature concatenation yields a +3.7 APM gain for medium persons but a −1.6 AP decrease for large persons.The result supports sensitivity of different feature-map resolutions to different person scales.
5. Conclusion
HigherHRNet addresses scale variation in bottom-up multi-person pose estimation with an efficient high-resolution feature pyramid, multi-resolution supervision, and multi-resolution heatmap aggregation. It improves pose estimation accuracy, especially for small persons, on COCO.
- HigherHRNet addresses scale variation in bottom-up multi-person pose estimation, especially precise keypoint localization for small persons.
- The method uses an efficient high-resolution feature pyramid based on HRNet, trained with multi-resolution supervision.
- During inference, multi-resolution heatmap aggregation efficiently generates multi- and higher-resolution heatmaps for more accurate pose estimation.
- HigherHRNet outperforms existing bottom-up methods by a large margin on the challenging COCO dataset, especially for small persons.