Source-linked AI summary
RMPE: Regional Multi-person Pose Estimation
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, Cewu Lu
TL;DR
Multi-person pose estimation can fail when detector boxes are slightly inaccurate or redundant, even when those boxes meet conventional detection criteria. RMPE adapts the two-step pipeline with SSTN, parametric Pose NMS, and PGPG, achieving 76.7 mAP on MPII and reporting higher scores with a stronger detector and pose estimator.
Problem
Two-step pose estimation is vulnerable to inaccurate human bounding boxes, while redundant detections can produce multiple poses for one person.
Method
RMPE combines symmetric STN with parallel SPPE, parametric Pose NMS, and PGPG to refine regions, eliminate redundant poses, and generate detector-like training proposals.
Results
76.7 mAP is achieved on the MPII multi-person dataset, outperforming the state-of-the-art methods.
Takeaways & Limitations
The framework is applicable to different human detectors and single-person pose estimators and supports the two-step framework when SPPE is adapted to detector outputs.
Takeaways & Limitations
The system remains vulnerable to rare poses, highly overlapping people, missed detections, and objects that resemble humans.
Abstract
from arXiv · showhide
Multi-person pose estimation in the wild is challenging. Although state-of-the-art human detectors have demonstrated good performance, small errors in localization and recognition are inevitable. These errors can cause failures for a single-person pose estimator (SPPE), especially for methods that solely depend on human detection results. In this paper, we propose a novel regional multi-person pose estimation (RMPE) framework to facilitate pose estimation in the presence of inaccurate human bounding boxes. Our framework consists of three components: Symmetric Spatial Transformer Network (SSTN), Parametric Pose Non-Maximum-Suppression (NMS), and Pose-Guided Proposals Generator (PGPG). Our method is able to handle inaccurate bounding boxes and redundant detections, allowing it to achieve a 17% increase in mAP over the state-of-the-art methods on the MPII (multi person) dataset.Our model and source codes are publicly available.
1. Introduction
Multi-person pose estimation is difficult because two-step methods depend on imperfect human boxes, while nearby people create ambiguity for part-based methods. RMPE addresses inaccurate boxes and redundant detections with SSTN, parametric Pose NMS, and PGPG, achieving 76.7 mAP on MPII.
- Two-step methods estimate poses independently inside detected boxes, making pose accuracy highly dependent on box quality.
- Part-based methods can produce ambiguous assemblies when people are close and lose global pose information by relying on second-order body-part dependence.
- RMPE combines SSTN, parametric Pose NMS, and PGPG to handle inaccurate boxes, redundant poses, and training-data augmentation.SSTN extracts a high-quality single-person region, Pose NMS compares pose similarity, and PGPG simulates detector-generated boxes.
- 76.7 mAP is achieved on the MPII multi-person dataset, outperforming state-of-the-art methods.
2. Related Work
Prior work includes single-person, part-based, and two-step pose-estimation frameworks, each with distinct assumptions and limitations. RMPE continues the two-step approach while adapting single-person estimation to detector outputs.
- Single Person Pose Estimation: Single-person methods perform well when a person is correctly located, but that assumption is not always satisfied.
- Part-based Framework: Part-based frameworks detect body parts independently and assemble them into human poses using models such as graphical structures, poselets, or optimization.
- Two-step Framework: Two-step frameworks first detect humans and then estimate each pose independently, but their performance depends on detector output quality.
- Two-step Framework: RMPE follows the two-step framework and adapts it with a pipeline connecting human proposals, single-person pose estimation, pose proposals, and Pose NMS.
3. Regional Multi-person Pose Estimation
RMPE combines a symmetric spatial transformer with SPPE, pose-based suppression, and detector-like proposal augmentation to address inaccurate boxes and redundant detections. Its training strategy centers transformed single-person regions, while its pose-guided generator models pose-dependent box-offset distributions.
- Framework overview: RMPE feeds detector bounding boxes into Symmetric STN + SPPE, automatically generates pose proposals, then refines them with parametric Pose NMS.Training adds Parallel SPPE to avoid local minimums, while PGPG augments training samples.
- Symmetric STN and Parallel SPPE: The SSTN applies a 2D affine transformation to extract a high-quality dominant-person region before SPPE estimates the pose.An SDTN then remaps the estimated pose to the original proposal coordinates.
- Symmetric STN and Parallel SPPE: Frozen Parallel SPPE compares its output with center-located pose labels, back-propagating errors that encourage the SSTN to center the extracted person.The parallel branch shares the STN with the original SPPE and does not use the spatial de-transformer.
- Parametric Pose NMS: Parametric Pose NMS eliminates redundant poses using a pose distance metric and threshold, selecting the most confident pose and iteratively suppressing similar ones.The metric combines soft joint matching with spatial distance, while its parameters are learned in a data-driven manner.
- Pose-guided Proposals Generator: PGPG models pose-dependent bounding-box offset distributions to generate training proposals with the same distribution as detector outputs.These proposals provide a large training sample and further boost system performance.
4. Experiments
The framework was evaluated on MPII and MSCOCO, with experiments testing performance, generality, and the contribution of its components. Results show strong benchmark performance and measurable benefits from the proposed modules.
- The method was evaluated on MPII and MSCOCO, two standard multi-person pose datasets with substantial occlusion.
- Results: 82.1 mAP was achieved on MPII using a stronger human detector and pose estimator, exceeding the previous best result by 4.6 mAP.
- Results: The method achieved state-of-the-art performance on the MSCOCO Keypoints test-dev set without a pose-estimation-specific network design.
- Symmetric STN and Parallel SPPE: Removing parallel SPPE degraded performance, indicating that its center-located pose supervision encourages the STN to extract single-person regions.
- Pose-guided Proposals Generator: 73.0% mAP resulted when pose-guided proposal augmentation was removed, while the proposed augmentation outperformed jittered-box proposals.
- Parametric Pose NMS: Removing parametric pose NMS significantly reduced mAP, while the proposed algorithm processed 1,300 validation images in 1.8 seconds versus 62.2 seconds for a compared implementation.
5. Conclusion
RMPE combines three components to improve multi-person pose estimation accuracy and efficiency, addressing localization errors and redundant detections. The framework also supports the broader use of detector-plus-SPPE pipelines when SPPE is adapted to human detectors.
- Framework components: RMPE combines symmetric STN with parallel SPPE, parametric pose NMS, and PGPG.PGPG augments training data by learning bounding-box proposal distributions conditioned on human poses.
- Localization robustness: Symmetric STN and parallel SPPE improve SPPE robustness to human localization errors.
- Detection refinement: Parametric pose NMS reduces redundant detections using pose-based suppression.
- Overall outcome: RMPE significantly outperforms state-of-the-art methods for multi-person human pose estimation in accuracy and efficiency.
- Implication: The framework validates the potential of two-step human detector plus SPPE pipelines when SPPE is adapted to the detector.