Source-linked AI summary
Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression
Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, Jingdong Wang
TL;DR
Dense bottom-up keypoint regression has weaker spatial localization than keypoint detection and grouping. DEKR uses adaptive convolutions and separate branches to learn keypoint-focused representations, and it achieves superior bottom-up pose estimation results on COCO and CrowdPose. The paper reports a limitation in its multi-scale testing scheme.
Problem
Dense keypoint regression produces spatially inaccurate predictions and has been inferior to keypoint detection and grouping.
Method
DEKR combines adaptive convolutions through a pixel-wise spatial transformer with a multi-branch structure that separately represents and regresses each keypoint.
Results
DEKR improves keypoint localization quality, outperforms keypoint detection and grouping methods, and achieves superior bottom-up results on COCO and CrowdPose.
Takeaways & Limitations
Disentangling representations lets each keypoint representation focus on its corresponding region, supporting spatially accurate direct regression.
Takeaways & Limitations
The absorbing scheme does not improve single-scale testing, and the paper leaves a better multi-scale testing scheme for future work.
Abstract
from arXiv · showhide
In this paper, we are interested in the bottom-up paradigm of estimating human poses from an image. We study the dense keypoint regression framework that is previously inferior to the keypoint detection and grouping framework. Our motivation is that regressing keypoint positions accurately needs to learn representations that focus on the keypoint regions. We present a simple yet effective approach, named disentangled keypoint regression (DEKR). We adopt adaptive convolutions through pixel-wise spatial transformer to activate the pixels in the keypoint regions and accordingly learn representations from them. We use a multi-branch structure for separate regression: each branch learns a representation with dedicated adaptive convolutions and regresses one keypoint. The resulting disentangled representations are able to attend to the keypoint regions, respectively, and thus the keypoint regression is spatially more accurate. We empirically show that the proposed direct regression method outperforms keypoint detection and grouping methods and achieves superior bottom-up pose estimation results on two benchmark datasets, COCO and CrowdPose. The code and models are available at https://github.com/HRNet/DEKR.
1. Introduction
The paper targets efficient bottom-up human pose estimation, where dense keypoint regression has been less spatially accurate than detection-and-grouping methods. DEKR addresses this by learning keypoint-focused, disentangled representations and reports superior benchmark performance.
- Motivation: Bottom-up pose estimation directly regresses keypoints or detects and groups them, offering greater efficiency than top-down estimation because it avoids person detection.Human pose estimation also requires identifying which keypoints belong to each person.
- Motivation: Dense regression methods such as CenterNet estimate all keypoint positions together, but their predictions are spatially inaccurate and underperform detection-and-grouping schemes.Their salient regression regions can spread broadly, reducing regression quality.
- Approach: DEKR uses adaptive convolutions through a pixel-wise spatial transformer to activate pixels in each keypoint region and learn focused representations.The approach is motivated by the need for representations that focus on keypoint regions.
- Approach: A multi-branch structure separately learns one representation and regresses one keypoint per branch, producing disentangled representations focused on corresponding keypoint regions.This decouples representation learning for one keypoint from the others.
- Results: DEKR improves localization quality and outperforms keypoint detection-and-grouping methods, achieving superior bottom-up results on COCO and CrowdPose.The method performs direct regression without matching regression outputs to heatmap-detected keypoints.
2. Related Work
Prior work includes top-down estimation, bottom-up association and grouping, dense regression, and disentangled representation learning. DEKR instead improves direct regression by dedicating adaptively activated representations to individual keypoints.
- Top-down paradigm: Top-down methods detect each person before single-person pose estimation and exploit person detections and bounding-box annotations.Representative methods include HRNet, PoseNet, RMPE, Mask R-CNN, and others.
- Bottom-up paradigm: Bottom-up methods commonly associate detected keypoints using affinity fields, associative embedding, voting, graph clustering, or optimization-based formulations.DeepCut, DeeperCut, and L-JPA use integer linear programming, which can take hours.
- Bottom-up paradigm: Dense regression methods produce pose candidates but often have weak localization, so post-processing matches regressions to nearby heatmap detections.This matching improves spatial accuracy but adds a separate processing step.
- Disentangled regression: DEKR improves direct regression by learning K representations, each dedicated to one keypoint and formed from adaptively activated pixels in its corresponding area.Its approach differs from mixture-density methods that handle uncertainty.
- Disentangled representation learning: Disentangled representation learning has been studied for separating factors such as content and pose, motion and content, or pose and appearance.PBN applies related disentanglement ideas to top-down pose estimation but learns high-quality heatmaps for part groups.
3. Approach
DEKR estimates keypoint offsets with adaptive, keypoint-specific representations and separate regression branches, while auxiliary heatmaps score and rank candidate poses.
- 3.1. Disentangled Keypoint Regression: The framework predicts a 2K-dimensional offset vector from each center pixel for the K keypoints.The offset maps are produced by a keypoint position regression head from backbone features.
- 3.1. Disentangled Keypoint Regression: DEKR applies adaptive convolutions to learn representations that focus on keypoint regions.The adaptive convolutions use pixel-wise spatial transformations to activate pixels in the relevant keypoint areas.
- 3.1. Disentangled Keypoint Regression: The regression head divides backbone features into K partitions and uses one independently parameterized branch for each keypoint.Each branch predicts the 2D offset map for its corresponding keypoint.
- 3.1. Disentangled Keypoint Regression: Separate branches focus activation on their corresponding keypoint regions, unlike a single branch that activates pixels around all keypoints.The multi-branch structure explicitly decouples representation learning and reduces regression loss.
- 3.2. Loss Function: A separate heatmap branch estimates keypoint and center heatmaps, which score and rank the regressed poses.Inference applies center NMS and pose NMS before ranking remaining candidates using center, keypoint, and shape features.
- 3.3. Inference: Separate regression reduces COCO training regression loss from 5.9e-5 to 5.5e-5 and increases validation AP from 65.4 to 68.0.These results use the HRNet-W32 backbone.
4. Experiments
Experiments evaluate DEKR on COCO and CrowdPose using standard pose metrics, comparisons, complexity analyses, ablations, and inference variants. DEKR achieves strong bottom-up pose-estimation results, with gains linked to adaptive activation and separate regression.
- COCO setup: COCO evaluation uses train2017, val2017, and test-dev2017, with 17-keypoint annotations and OKS-based AP and AR metrics.Training uses augmented crops with HRNet-W32 or HRNet-W48; testing includes single-scale and multi-scale variants.
- COCO results: COCO validation DEKR with HRNet-W32 achieves 68.0 AP, over 9.0 higher than CenterNet-DLA and PersonLab at similar GFLOPs.It gains 4.0 over the larger CenterNet-HG model, while the regression baseline scores 61.9.
- COCO results: COCO validation DEKR with HRNet-W48 and 640 input obtains 71.0 AP, gaining 3.0 over HRNet-W32 and 1.1 over HrHRNet-W48.The reported gains also include 7.0 over CenterNet-HG and 4.5 over PersonLab.
- COCO results: On COCO test-dev, HRNet-W32 reaches 67.3 AP and HRNet-W48 reaches 70.0, with gains over PersonLab, PifPaf, and HrHRNet.With multi-scale testing, HRNet-W32 reaches 69.8 and HRNet-W48 reaches 71.0 AP.
- Ablation analysis: Ablations show adaptive activation adds 3.5 AP over the 61.9 regression baseline, separate regression adds 2.6, and the combined gain is 6.1.Adaptive activation and separate regression mainly reduce Jitter and Miss localization errors.
- Empirical analysis: The absorbing scheme does not improve single-scale testing, improves multi-scale results, and motivates future work on better multi-scale testing schemes.The authors state that multi-scale testing has lower practical value than single-scale testing for their current focus.
- CrowdPose results: On CrowdPose, HRNet-W48 achieves 67.3 AP, exceeding HrHRNet-W48 at 65.9 on the challenging crowded-scene benchmark.CrowdPose reports AP, threshold-specific AP, and performance across easy, medium, and hard crowding levels.
5. Conclusions
DEKR improves keypoint localization and achieves state-of-the-art bottom-up pose estimation by disentangling representations so each focuses on its corresponding keypoint region.
- DEKR improves keypoint localization quality and achieves state-of-the-art bottom-up pose estimation results.