Source-linked AI summary
Distribution-Aware Coordinate Representation for Human Pose Estimation
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, Ce Zhu
TL;DR
Human pose estimation commonly uses heatmaps, but the coordinate encoding and decoding processes had not been systematically investigated. The paper proposes DARK, a distribution-aware decoding and unbiased encoding plug-in, and reports improved performance across state-of-the-art models on MPII and COCO, with best single-model accuracy on both benchmarks.
Problem
Heatmap coordinate representation, especially decoding predicted heatmaps into original-image coordinates, was largely ignored despite its important role in pose estimation performance.
Method
DARK combines Taylor-expansion-based distribution-aware coordinate decoding with unbiased sub-pixel-centred heatmap encoding as a model-agnostic plug-in.
Results
DARK significantly improves existing state-of-the-art models and achieves the best single-model accuracy on the MPII and COCO benchmarks.
Takeaways & Limitations
DARK enables smaller input resolutions with much smaller performance degradation while boosting inference efficiency for low-latency and low-energy applications.
Takeaways & Limitations
The distribution-aware decoding assumes that predicted heatmaps follow the same 2D Gaussian form as ground-truth heatmaps.
Abstract
from arXiv · showhide
While being the de facto standard coordinate representation in human pose estimation, heatmap is never systematically investigated in the literature, to our best knowledge. This work fills this gap by studying the coordinate representation with a particular focus on the heatmap. Interestingly, we found that the process of decoding the predicted heatmaps into the final joint coordinates in the original image space is surprisingly significant for human pose estimation performance, which nevertheless was not recognised before. In light of the discovered importance, we further probe the design limitations of the standard coordinate decoding method widely used by existing methods, and propose a more principled distribution-aware decoding method. Meanwhile, we improve the standard coordinate encoding process (i.e. transforming ground-truth coordinates to heatmaps) by generating accurate heatmap distributions for unbiased model training. Taking the two together, we formulate a novel Distribution-Aware coordinate Representation of Keypoint (DARK) method. Serving as a model-agnostic plug-in, DARK significantly improves the performance of a variety of state-of-the-art human pose estimation models. Extensive experiments show that DARK yields the best results on two common benchmarks, MPII and COCO, consistently validating the usefulness and effectiveness of our novel coordinate representation idea.
Introduction
The paper argues that heatmap coordinate representation, including encoding and decoding, is an underexamined but important part of human pose estimation. It introduces DARK, combining distribution-aware decoding with unbiased sub-pixel encoding to improve localization and efficiency.
- Motivation: Human pose estimation detects body-joint coordinates despite clothing variation, occlusion, and unconstrained backgrounds.Existing work typically emphasizes CNN architecture design for pose inference.
- Heatmap Representation: Heatmaps encode joint labels as 2D Gaussian distributions, providing spatial support and representing target-position ambiguity.This representation is widely used by state-of-the-art pose models.
- Coordinate Decoding: Resolution reduction makes heatmap computation affordable but requires recovery from low-resolution predictions to original-image coordinates, introducing quantisation error.Standard decoding selects maximal activation and commonly applies a hand-crafted shift toward the second-highest activation.
- Importance of Coordinate Representation: 5.7% AP is gained on COCO validation with HRNet-W32 from the standard coordinate-shifting operation, showing that coordinate representation materially affects performance.The paper notes this gain exceeds that of most individual architectural methods.
- Efficiency Challenge: 74.4% to 66.9% is the HRNet-W32 performance drop on COCO validation when input resolution decreases from 256×192 to 128×96, while FLOPs fall from 7.1×10^9 to 1.8×10^9.The result illustrates the accuracy–inference-cost trade-off associated with resolution reduction.
- Proposed Method: DARK combines Taylor-expansion-based coordinate decoding with unbiased sub-pixel-centred encoding and operates as a model-agnostic plug-in.The encoding improvement places Gaussian kernels at sub-pixel ground-truth locations to avoid quantisation-imprecise supervision.
Related Work
Prior pose-estimation research largely focused on model architectures and related learning or representation techniques rather than heatmap coordinate representation. This work instead studies heatmap issues and proposes a plug-in method that improves existing models without changing their designs.
- Prior Approaches: Existing work explored coordinate regression, knowledge-guided and active learning, adversarial learning, deconvolution upsampling, multi-scale supervision, attention, and high-resolution representations.These approaches represent major prior directions in human pose estimation.
- Heatmap Representation: Heatmap representation was a largely ignored perspective, despite resolution reduction affecting its use and coordinate methods improving existing-model performance.The proposed method integrates without model-design changes.
Methodology
DARK investigates heatmap coordinate encoding and decoding, replacing empirical sub-pixel shifting with distribution-aware localisation and correcting quantisation-biased supervision. Its decoding pipeline modulates heatmaps, estimates joint locations by Taylor expansion, and maps them back to original coordinates.
- Coordinate Decoding: Coordinate decoding translates each predicted joint heatmap into a coordinate in the original image space after resolution reduction.Because heatmaps are commonly lower resolution than input images, decoding requires upsampling by a sample-specific factor λ and sub-pixel localisation.
- Coordinate Decoding: The standard decoder finds the maximal and second-maximal activations, then shifts the maximum 0.25 pixels toward the second maximum.This empirical shift compensates the quantisation effect of downsampling, but lacks a principled interpretation.
- Distribution-Aware Localisation: DARK assumes the predicted heatmap follows a 2D Gaussian and estimates its mean, the sub-pixel joint location, from distribution statistics.Taylor expansion around the maximal activation uses derivative and Hessian information estimated from the heatmap.
- Heatmap Distribution Modulation: Heatmap modulation smooths multiple peaks with a Gaussian kernel before distribution-aware decoding, then rescales the result to preserve the original heatmap magnitude.The authors report that this modulation further improves decoding performance.
- DARK Pipeline: The decoding pipeline consists of heatmap modulation, Taylor-expansion localisation at sub-pixel accuracy, and resolution recovery to original coordinates.The authors state that none of these steps incurs high computational cost, enabling DARK as a plug-in for existing models.
- Coordinate Encoding: Standard coordinate encoding quantises downsampled ground-truth coordinates before Gaussian heatmap generation, introducing biased heatmaps and sub-optimal supervision.The issue occurs with common floor, ceil, and round quantisation choices; DARK instead centres the Gaussian kernel at sub-pixel locations.
Experiments
Experiments show that coordinate encoding and decoding materially affect pose accuracy, while DARK preserves more accuracy as input resolution decreases and improves state-of-the-art models on COCO and MPII.
- Coordinate decoding: 5.7% AP is gained by standard coordinate shifting, while distribution-aware decoding adds 1.5% AP beyond standard decoding on COCO validation.Distribution modulation contributes 0.3% of the additional gain.
- Coordinate encoding: Over 1% AP is contributed consistently by unbiased coordinate encoding in both tested cases.The result supports the importance of encoding choices in addition to decoding.
- Input resolution: DARK mitigates accuracy loss at reduced input resolutions, especially for very small inputs, supporting deployment on low-resource devices.The comparison uses HRNet-W32 and evaluates varying input image sizes on COCO validation.
- State-of-the-art comparison: 76.2 AP is achieved by DARK with HRNet-W48 at 384×288, improving 0.7% AP over HRNet-W48 at the same size without extra parameters.DARK also improves over the efficient Integral Pose Regression comparison by 2.2% AP at 16.4% of its execution cost.
- State-of-the-art comparison: DARK consistently outperforms the best competitor on MPII, with a larger margin under PCKh@0.1 and evidence of generalization across training-data sizes.The comparison uses single-scale HRNet-W32 on the MPII validation set.
Conclusion
The paper systematically investigates heatmap coordinate representation and introduces DARK as a plug-in for more discriminative pose training and inference. Across contemporary models and MPII and COCO, extensive experiments validate its performance advantages.
- Conclusion: DARK systematically investigates coordinate representation and combines distribution-aware decoding with unbiased sub-pixel-centred encoding.Its two components target coordinate decoding and coordinate encoding in human pose estimation.
- Conclusion: Existing state-of-the-art models can use DARK without algorithmic adaptation and at negligible cost.The paper evaluates the plug-in across a wide spectrum of contemporary models and two challenging datasets.
- Conclusion: Experiments on MPII and COCO validate DARK’s performance advantages and provide component analyses of its design rationale.The conclusion describes both empirical validation and in-depth component analysis.