Source-linked AI summary
Towards Accurate Multi-person Pose Estimation in the Wild
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, Kevin Murphy
TL;DR
Multi-person pose estimation must detect people and localize their keypoints without known boxes, especially under crowded-scene association challenges. The paper uses a top-down two-stage system with keypoint-based prediction and postprocessing, achieving state-of-the-art COCO results. Its strongest reported result is 0.673 AP on test-standard with COCO+int training.
Problem
Multi-person pose estimation in the wild lacks ground-truth person locations and scales, while crowded scenes complicate assigning body parts to individuals.
Method
A two-stage top-down pipeline detects person boxes, then predicts keypoint heatmaps and offsets with a fully convolutional network, using aggregation, OKS-NMS, and keypoint-based scoring.
Results
0.673 AP on COCO test-standard with COCO+int training, a 6.2% absolute improvement over CMU-Pose's 0.611 test-standard score.
Takeaways & Limitations
The results show that a carefully adapted top-down system can achieve state-of-the-art performance on the challenging COCO multi-person keypoints benchmark.
Takeaways & Limitations
Further detector enhancements can improve results but require significantly increased computation time.
Abstract
from arXiv · showhide
We propose a method for multi-person detection and 2-D pose estimation that achieves state-of-art results on the challenging COCO keypoints task. It is a simple, yet powerful, top-down approach consisting of two stages. In the first stage, we predict the location and scale of boxes which are likely to contain people; for this we use the Faster RCNN detector. In the second stage, we estimate the keypoints of the person potentially contained in each proposed bounding box. For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet. To combine these outputs we introduce a novel aggregation procedure to obtain highly localized keypoint predictions. We also use a novel form of keypoint-based Non-Maximum-Suppression (NMS), instead of the cruder box-level NMS, and a novel form of keypoint-based confidence score estimation, instead of box-level scoring. Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set and the 0.643 test-standard sets, outperforming the winner of the 2016 COCO keypoints challenge and other recent state-of-art. Further, by using additional in-house labeled data we obtain an even higher average precision of 0.685 on the test-dev set and 0.673 on the test-standard set, more than 5% absolute improvement compared to the previous best performing method on the same dataset.
1. Introduction
Multi-person pose estimation in the wild combines person detection with 2-D keypoint localization under crowded-scene association challenges. The paper revisits a top-down two-stage design with keypoint-based prediction, suppression, and scoring.
- Human pose estimation localizes 2-D joints on the arms, legs, torso, and face for broader visual understanding of people.
- Most prior work addressed single-person pose assuming a ground-truth bounding box or torso keypoint location and scale.
- Multi-person pose estimation in the wild must jointly detect people and estimate poses without provided instance locations or scales.
- Crowded scenes make association difficult because body parts must be assigned to the correct person.
- COCO keypoints provides 17-keypoint annotations and uses object keypoint similarity to measure predicted-pose agreement with ground truth.
- The paper revisits top-down estimation with person-box proposals, fully convolutional keypoint heatmaps and offsets, heatmap-offset aggregation, OKS-NMS, and keypoint-based confidence scoring.
2. Related Work
Prior pose-estimation research developed part-based, CNN, bottom-up, and top-down approaches, with different strategies for modeling relationships and associating parts into people.
- Part-based methods model articulated human bodies while seeking tractable inference and rich dependencies among body parts.
- Single-Person Pose: CNN-based single-person methods use regression, convolutional heatmaps, cascaded refinement, image-dependent priors, or sequential part prediction.
- Single-Person Pose: Newell et al. report state-of-the-art single-person pose results on MPII and FLIC using skip connections and repeated pooling-upsampling for multi-scale refinement.
- Top-Down Multi-Person Pose: Top-down multi-person methods first detect people and then estimate poses, addressing multi-person association through person-level proposals.
- Bottom-Up Multi-Person Pose: Bottom-up multi-person methods detect body parts and associate them into person instances through differing association representations and inference procedures.
- Bottom-Up Multi-Person Pose: The 2016 COCO challenge winner combined joint detection, part affinity fields, greedy instance proposals, and top-down refinement.
3. Methods
The method is a two-stage top-down cascade: Faster-RCNN proposes person boxes, then a fully convolutional pose estimator localizes and scores keypoints using heatmaps, offsets, voting, rescoring, and OKS-based suppression.
- Cascade architecture: The two-stage cascade first detects candidate person boxes, then estimates keypoints for each proposed person instance.The second stage also refines proposal confidence using pose information.
- Person box detection: Faster-RCNN with a ResNet-101 backbone generates person proposals, while multi-scale evaluation and ensembling are not used for reproducibility.The detector uses atrous convolution for denser feature maps; additional enhancements could improve results at higher computational cost.
- Keypoint prediction: The pose network predicts 17 keypoints with disk-shaped heatmaps and 2-D offsets, replacing direct fine-grid localization with classification and regression.Heatmaps identify pixels near each keypoint, while offsets point from each pixel to the exact keypoint location.
- Keypoint aggregation: A weighted voting aggregation combines heatmaps and offsets into highly localized activation maps that pinpoint keypoint positions.Each spatial point casts a vote toward the estimated keypoint location, weighted by its heatmap probability.
- Scoring and suppression: At test time, pose-based rescoring averages keypoint confidence, and OKS-NMS uses keypoint overlap to distinguish duplicate detections from nearby people.The pose score replaces reliance on the person detector score, while OKS-NMS follows box-level filtering at the detector output.
4. Experimental Evaluation
The evaluation studies training-data variants, detector and pose-estimator configurations, and suppression settings on COCO splits. The system achieves state-of-the-art results, while larger and more accurate pose-estimation configurations improve accuracy at increased computational cost.
- COCO+int combines COCO-only training with 73,024 Flickr images containing 227,029 additional annotated person instances.
- The system achieves state-of-the-art results on COCO test-dev and test-standard, outperforming CMU-Pose and Mask-RCNN even with COCO-only training.
- 0.673 AP on COCO test-standard is achieved with COCO+int pose training, a 6.2% absolute improvement over CMU-Pose's 0.611.
- The box-detection ablation compares fast 600x900 and accurate 800x1200 Faster-RCNN variants, whose test-dev person-detection AP is 0.456 and 0.487, respectively.
- ResNet-101 pose estimation performs about 2% better than ResNet-50, while the 353x257 crop is more accurate than the faster 257x185 crop.
- The selected pose estimator uses ResNet-101 with a 353x257 crop and disk radius R = 25 pixels; OKS-NMS is fixed at threshold 0.5.
5. Conclusion
The paper addresses person detection and pose estimation in cluttered images in the wild with a simple two-stage system. Despite its simplicity, the system achieves state-of-the-art results on the challenging COCO benchmark.
- The system combines a person detection stage with a keypoint estimation stage for each detected person.
- Despite its simplicity, the system achieves state-of-the-art results on the challenging COCO benchmark.