Source-linked AI summary
Integral Human Pose Regression
Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, Yichen Wei
TL;DR
Heat-map pose estimation performs well but is limited by nondifferentiable maximum selection and quantization error. The paper replaces maximum selection with differentiable integral regression, then validates it across settings, especially 3D pose estimation, with improvements including state-of-the-art Human3.6M results.
Problem
Heat-map representations have nondifferentiable maximum selection and quantization error, while regression methods have not matched their 2D effectiveness.
Method
Integral regression estimates joint coordinates as the expectation of heat-map locations weighted by normalized probabilities.
Results
Integral regression significantly improves 3D pose estimation, enables mixed 3D and 2D training, and achieves state-of-the-art results on Human3.6M.
Takeaways & Limitations
The differentiable integral representation unifies heat-map and regression approaches while remaining compatible with heat-map methods and adding little computation and storage.
Takeaways & Limitations
On MPII, the approach is slightly below the state of the art, and combining it with leading complex approaches is left for future work.
Abstract
from arXiv · showhide
State-of-the-art human pose estimation methods are based on heat map representation. In spite of the good performance, the representation has a few issues in nature, such as not differentiable and quantization error. This work shows that a simple integral operation relates and unifies the heat map representation and joint regression, thus avoiding the above issues. It is differentiable, efficient, and compatible with any heat map based methods. Its effectiveness is convincingly validated via comprehensive ablation experiments under various settings, specifically on 3D pose estimation, for the first time.
1 Introduction
Heat maps perform strongly but suffer from nondifferentiable coordinate extraction, quantization, and costly high-resolution representations. Integral regression replaces maximum selection with expectation, unifying heat-map and regression approaches while enabling broad evaluation across settings.
- Motivation: Heat-map pose estimators locate joints by taking the maximum-likelihood point, but this operation is nondifferentiable and blocks end-to-end training.Lower-resolution heat maps also introduce quantization error, while higher resolutions increase computation and storage costs.
- Motivation: Regression offers end-to-end learning and continuous outputs, yet remains less effective than detection-based methods for 2D pose estimation.Regression performance is also described as unsatisfactory for 3D pose estimation.
- Contribution: Integral regression replaces taking the heat-map maximum with taking the expectation of all locations weighted by normalized probabilities.The operation is differentiable, adds little computation and storage, and can combine with heat-map methods.
- Contribution: The approach addresses an underexplored relation between heat-map detection and joint regression through comprehensive experiments beyond the nearly saturated MPII 2D benchmark.The paper specifically investigates settings such as 3D pose estimation.
- Results: Integral regression significantly improves 3D pose estimation, supports mixed 3D and 2D training, and achieves state-of-the-art results on Human3.6M.Results on MPII and COCO are reported as competitive.
2 Integral Pose Regression
Integral pose regression converts a learned heat map into continuous joint coordinates by integrating locations weighted by normalized probabilities. The differentiable, non-parametric operation can augment heat-map methods and supports mixed 2D–3D training through coordinate-wise decomposition.
- Integral representation: Each location in a learned heat map represents the probability that the corresponding joint is there.The final joint coordinate is obtained from the heat-map representation.
- Integral representation: Integral regression estimates a joint by taking the expectation of all locations in the heat-map domain using normalized heat-map probabilities.The approach modifies maximum selection into expectation and uses softmax normalization.
- Properties: The integral function is differentiable, continuous, non-parametric, and compatible with heat-map methods with negligible computation and memory overhead.Its heat-map representation also makes training easy.
- Joint 3D and 2D training: Differentiability allows integral regression to adopt mixed-training techniques that supervise 2D joint coordinates using abundant 2D data.The paper reports significant improvement from this technique in its experiments.
- Joint 3D and 2D training: Integral regression separates x, y, and z targets through two-step integration, enabling mixed 2D and 3D training.The 3D heat map is first integrated into one-dimensional heat vectors and then into joint coordinates.
3 Methodology for Comprehensive Experiment
The methodology evaluates integral regression across tasks, architectures, losses, supervision choices, and resolutions. Experiments report consistent improvements, robustness to resolution changes, and flexibility across network designs.
- Methodology for Comprehensive Experiment: The ablation methodology evaluates integral regression across tasks, network architectures, heat map losses, supervision signals, and image or heat map resolutions.The study compares 2D and 3D tasks, multiple backbone designs, several heat map losses, optional intermediate heat map supervision, and resolution variations.
- Network Architecture: Integral regression can be embedded into various backbone networks, and its results are less affected by network capacity than heat map methods.ResNet, HourGlass, multiple ResNet depths, and multi-stage designs are investigated.
- Heat Map Losses: Integral versions of the tested heat map losses produce prominent and consistent improvements across tasks.The experiments include multiple manually designed heat map losses, whose non-integral versions may require task-specific tuning.
- Heat Map and Joint Loss Combination: Integral regression works with or without intermediate heat map supervision, with I1 best for 2D and I* best for 3D tasks.I* removes the intermediate heat map loss and supervises only the joint output.
- Image and Heat Map Resolutions: The approach is more robust to image and heat map resolution variation, making it useful when computational resources are restricted.Higher resolutions otherwise increase memory and computation demands, especially for 3D heat maps.
4 Datasets and Evaluation Metrics
The evaluation uses three pose-estimation benchmarks spanning controlled 3D estimation, single-person 2D estimation, and challenging multi-person in-the-wild conditions. Each benchmark uses its corresponding pose metric.
- Datasets: The experiments validate the approach on three benchmark datasets.The supplied dataset descriptions cover Human3.6M, MPII, and COCO.
- Datasets: Human3.6M is a large controlled-environment 3D pose benchmark with 3.6 million frames, 11 subjects, four viewpoints, and 15 activities.It provides accurate 3D joint locations from motion-capture devices.
- Metrics: Human3.6M evaluation commonly reports MPJPE, with PA MPJPE computed after rigid Procrustes alignment.The two metrics differ in whether the predicted and ground-truth poses are aligned before error computation.
- Datasets and Metrics: MPII evaluates single-person 2D pose estimation on daily-activity images using PCKh@0.5 as the common benchmark metric.The dataset contains about 25k images, with approximately 29k training poses and 7k testing poses.
- Datasets and Metrics: COCO evaluates multi-person pose estimation in challenging uncontrolled conditions using mean average precision over 10 OKS thresholds.Its train, validation, and test sets contain more than 200k images and 250k labeled person instances.
5 Experiments
Across MPII, COCO, and Human3.6M, integral regression consistently improves heat-map and direct-regression baselines, including under reduced resolution, smaller networks, multi-stage architectures, and mixed 2D/3D training. The method achieves competitive or state-of-the-art results, with especially strong gains on 3D pose estimation.
- MPII: Integral regression improves all three heat-map baselines, with I1 exceeding H1 by +0.5 PCKh@0.5, +12.1 PCKh@0.1, and +5.4 AUC.I3 also raises AUC from 46.3 to 57.7 (+11.4), supporting the benefit of joint training.
- MPII: I* outperforms direct regression R1 despite using the same supervision and nearly the same network, while joint heat-map and coordinate training performs best overall.The comparisons are summarized as I*>H, I*>R, and I>H,R,I*.
- MPII: Integral regression is less sensitive to resolution: halving the heat-map size saves 1.1 G FLOPs while AUC drops 0.6 for I1 versus 4.8 for H1.At the lower image size, the corresponding drops are 3.5 for I1 and 12.5 for H1 with 0.3 G FLOPs saved.
- MPII: Integral regression improves every multi-stage comparison: MS-I1 at stage 2 reaches 87.7, exceeding MS-H1 at stage 4 with 87.4.Performance also improves as the number of stages increases.
- COCO: On COCO, integral regression raises keypoint AP by 1.5 points and achieves the state-of-the-art result.The baseline uses a one-stage ResNet-101 trained on COCO data without extra data.
- Human3.6M: On Human3.6M, integral regression improves baselines with and without extra 2D data, enabling mixed-data training and new state-of-the-art results.Without extra 2D data, I1, I2, and I* improve their baselines by 6.0%, 13.2%, and 17.7%; with extra 2D data, improvements are 11.7%, 17.1%, and 11.6%.
6 Conclusions
Integral regression unifies heat map representation and joint regression, combining their merits. Experiments on 2D and 3D pose estimation achieve strong benchmark performance with simple, inexpensive baseline networks.
- Integral regression unifies heat map representation and joint regression approaches while sharing the merits of both.