Source-linked AI summary
Cascaded Pyramid Network for Multi-Person Pose Estimation
Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, Jian Sun
TL;DR
Multi-person pose estimation still struggles with occluded, invisible, and context-dependent keypoints. The paper introduces a two-stage CPN within a top-down detection pipeline, combining pyramid features with online hard-keypoint mining. It achieves 73.0 AP on COCO test-dev and 72.1 AP on COCO test-challenge.
Problem
Occluded, invisible, and crowded-background keypoints remain difficult to localize in multi-person pose estimation.
Method
CPN combines GlobalNet’s feature-pyramid representation and RefineNet’s multi-level feature integration with online hard keypoints mining in a top-down pipeline.
Results
73.0 AP on COCO test-dev and 72.1 AP on COCO test-challenge are reported as state-of-the-art results.
Takeaways & Limitations
The proposed CPN addresses hard keypoints using pyramid context and explicit hard-keypoint refinement while achieving state-of-the-art COCO benchmark performance.
Takeaways & Limitations
The pipeline assumes each detected proposal contains only one main person in its cropped region.
Abstract
from arXiv · showhide
The topic of multi-person pose estimation has been largely improved recently, especially with the development of convolutional neural network. However, there still exist a lot of challenging cases, such as occluded keypoints, invisible keypoints and complex background, which cannot be well addressed. In this paper, we present a novel network structure called Cascaded Pyramid Network (CPN) which targets to relieve the problem from these "hard" keypoints. More specifically, our algorithm includes two stages: GlobalNet and RefineNet. GlobalNet is a feature pyramid network which can successfully localize the "simple" keypoints like eyes and hands but may fail to precisely recognize the occluded or invisible keypoints. Our RefineNet tries explicitly handling the "hard" keypoints by integrating all levels of feature representations from the GlobalNet together with an online hard keypoint mining loss. In general, to address the multi-person pose estimation problem, a top-down pipeline is adopted to first generate a set of human bounding boxes based on a detector, followed by our CPN for keypoint localization in each human bounding box. Based on the proposed algorithm, we achieve state-of-art results on the COCO keypoint benchmark, with average precision at 73.0 on the COCO test-dev dataset and 72.1 on the COCO test-challenge dataset, which is a 19% relative improvement compared with 60.5 from the COCO 2016 keypoint challenge.Code (https://github.com/chenyilun95/tf-cpn.git) and the detection results are publicly available for further research.
1. Introduction
Multi-person pose estimation remains difficult for occluded, invisible, and context-dependent keypoints. The paper proposes CPN, combines GlobalNet and RefineNet, and reports state-of-the-art COCO results.
- Multi-person pose estimation supports applications including human action recognition and human-computer interaction.
- Occluded, invisible, and crowded-background keypoints remain difficult to localize because some joints cannot be recognized from appearance alone.
- CPN combines GlobalNet’s feature-pyramid representation with RefineNet and online hard keypoints mining to address difficult joints.
- The method uses a top-down pipeline that detects human bounding boxes before localizing keypoints within each box.
- 73.0 AP on COCO test-dev and 72.1 AP on COCO test-challenge establish state-of-the-art benchmark results.
2. Related Work
Prior work spans classical graphical models and CNN-based single- and multi-person methods. Multi-person systems are commonly organized as bottom-up assembly or top-down person detection followed by pose estimation.
- Human pose estimation evolved from pictorial structures and graphical models with hand-crafted features toward convolutional neural networks.
- Multi-person pose estimation is challenging because of occlusion, varied gestures, and unpredictable interactions between people.
- Bottom-Up Approaches: Bottom-up approaches predict all keypoints first and then assemble them into complete poses for each person.
- Top-Down Approaches: Top-down approaches first locate and crop people, then estimate poses in the resulting person patches.
- Single Person Pose Estimation: Single-person methods include heatmap prediction, stacked refinement, hourglass architectures, iterative feedback, voting, and recurrent refinement.
3. Our Approach for Multi-perosn Keypoints Estimation
The approach applies a top-down detector-plus-estimator pipeline and introduces CPN for single-person keypoint localization. CPN combines pyramid features with targeted refinement of hard keypoints.
- The top-down pipeline detects human bounding boxes and predicts detailed keypoints for each person with a single-person skeleton estimator.
- Cascaded Pyramid Network (CPN): CPN contains GlobalNet and RefineNet and is motivated as an efficient alternative to extensively stacked hourglass modules.
- GlobalNet: GlobalNet combines features from ResNet levels C2 through C5 to balance spatial localization and semantic information in keypoint heatmaps.
- GlobalNet: GlobalNet can localize simple keypoints such as eyes but may miss hips that require more contextual information.
- RefineNet: RefineNet integrates pyramid features across levels through upsampling and concatenation rather than using only final upsampled features.
- RefineNet: Online hard keypoints mining selects difficult keypoints according to training loss and backpropagates gradients from those selected points.
4. Experiment
The experiments evaluate CPN in a top-down COCO pose-estimation pipeline and examine detector, architecture, refinement, and hard-keypoint-mining choices. CPN improves the speed–accuracy trade-off and reaches strong COCO test performance.
- Experimental Setup: The pipeline generates human proposals with a state-of-the-art detector, then applies CPN to localize keypoints within each proposal.Models are trained on MS COCO trainval and primarily evaluated using OKS-based mAP.
- Person Detector: Soft-NMS surpasses hard NMS for both detection and keypoint-detection performance under the same model.Higher hard-NMS thresholds also improve performance, largely through better bounding-box AP and AR.
- Person Detector: Increasing detection AP from 44.3 to 49.3 does not improve keypoint-detection accuracy, indicating that hard-keypoint accuracy is more important once boxes cover most medium and large persons.The corresponding human detection AP increases by 3.0 points while keypoint accuracy remains unchanged.
- Cascaded Pyramid Network: CPN achieves a better speed–accuracy trade-off than the hourglass and dilated ResNet-50 baselines; RefineNet adds 2.0 AP over GlobalNet and reaches 68.6 AP.Online hard keypoint mining further raises performance to 69.4 AP.
- Cascaded Pyramid Network: RefineNet improves by integrating pyramid features through concatenation and bottleneck blocks, outperforming alternative design implementations.The reported gain exceeds 2 points compared with GlobalNet alone.
- Online Hard Keypoints Mining: Online hard keypoint mining improves the overall network by 0.8 AP to 69.4 AP, while removing intermediate supervision causes a 0.9 AP drop.The method focuses on the top M hard keypoint losses during RefineNet training.
- Results: On COCO, CPN obtains 73.0 AP on test-dev with ensembled models and 72.1 AP on test-challenge2017 without extra training data.A single CPN model reaches 72.1 AP on test-dev.
5. Conclusion
The paper presents a top-down CPN pipeline for addressing hard keypoints in multi-person pose estimation and reports state-of-the-art COCO benchmark results.
- CPN combines a GlobalNet feature pyramid with a RefineNet that integrates pyramid features and online hard keypoint mining.The design targets hard keypoints such as occluded or invisible joints.
- The method follows a top-down pipeline for multi-person pose estimation.A detector first generates human bounding boxes, then CPN localizes keypoints within each box.
- 73.0 AP on COCO test-dev and 72.1 AP on COCO test-challenge are reported as state-of-the-art results.
- The reported performance outperforms the COCO 2016 keypoint challenge winner by a 19% relative improvement.