Source-linked AI summary
OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh
TL;DR
Realtime multi-person 2D pose estimation must handle unknown numbers of people, occlusion, part association, and realtime constraints. OpenPose uses Part Affinity Fields with greedy parsing, and PAF-only refinement increases speed by approximately 200% and accuracy by 7%.
Problem
Multi-person 2D pose estimation must associate parts amid varying people, scale, occlusion, interactions, and runtime demands that grow with crowd size.
Method
OpenPose represents limb position and orientation with Part Affinity Fields and uses efficient greedy parsing to assemble multi-person poses.
Results
PAF-only refinement increases runtime performance by approximately 200% and accuracy by 7%, while combined body-foot estimation preserves accuracy and reduces sequential inference time.
Takeaways & Limitations
The work delivers OpenPose, an open-source realtime system for multi-person body, foot, hand, and facial keypoint detection.
Takeaways & Limitations
The approach can fail on atypical or upside-down poses, occlusions, crowded overlaps, and animal or statue-like figures.
Abstract
from arXiv · showhide
Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.
1 INTRODUCTION
The paper addresses the challenges of realtime multi-person 2D pose estimation with a bottom-up method based on Part Affinity Fields (PAFs). It reports improved speed and accuracy, a combined body-and-foot detector, vehicle keypoint estimation, and the release of OpenPose.
- Problem and motivation: Multi-person pose estimation must handle an unknown number of people at varying positions and scales, while interactions between people create additional challenges.Top-down approaches can fail when person detectors miss closely spaced people, and their runtime grows with the number of detections.
- Method: The method introduces Part Affinity Fields, 2D vector fields encoding limb location and orientation, and jointly predicts body-part confidence maps and PAFs from the entire image.Bipartite matching associates body-part candidates before assembling complete poses for all people.
- Efficiency and performance: The proposed method achieves competitive performance on multiple public benchmarks with quality results at a fraction of the computational cost.Unlike initial bottom-up methods requiring costly global inference, the approach supports a greedy parse using jointly inferred detection and association representations.
- Network refinement: Approximately 200% faster and 7% more accurate, the refined network increases depth, retains PAF refinement, and removes body-part prediction refinement stages.The paper identifies PAF refinement as crucial for maximizing accuracy, while body-part prediction refinement is less important.
- Extensions and release: The work releases a publicly available foot dataset with 15K human foot instances, introduces a combined body-and-foot model, applies the method to vehicle keypoint estimation, and documents OpenPose’s release.The combined model preserves the speed of the body-only model while maintaining its accuracy, and OpenPose is described as the first available realtime open-source system for multi-person 2D pose estimation.
2 RELATED WORK
Prior pose-estimation work modeled articulated-body relationships with graphical models and progressively refined part predictions using multi-stage architectures. Earlier PAF work introduced flow fields for associating body parts across multiple people, while this paper extends that approach through PAF-only refinement and combined body-foot detection.
- Single Person Pose Estimation: Traditional single-person pose estimation combines local body-part observations with spatial dependencies modeled by tree-structured or non-tree graphical models.Tree models encode relationships between adjacent parts along a kinematic chain.
- Multi-stage architectures: Wei et al. used a multi-stage convolutional architecture that iteratively incorporated global context to refine part confidence maps while preserving multimodal uncertainty.Intermediate supervision was applied at the end of each stage to address vanishing gradients during training.
- Part Affinity Fields: Earlier PAF work represented unstructured pairwise relationships between body parts of a variable number of people as a set of flow fields.PAFs provide a nonparametric representation for associating body parts across multiple people.
- Multi-person association: PAFs can yield pairwise scores without an additional training step, enabling greedy parsing for high-quality multi-person results with realtime performance.Concurrent work simplified body-part relationship graphs for faster single-frame inference and formulated articulated human tracking as spatiotemporal grouping of part proposals.
- Extensions to earlier work: This work extends earlier PAF methods by removing body-part confidence-map refinement, increasing network depth, and introducing a combined body-and-foot keypoint detector.The authors state that PAF refinement is critical and sufficient for high accuracy, producing a faster and more accurate model.
3 METHOD
OpenPose predicts body-part confidence maps and Part Affinity Fields from image features, then parses them to assemble 2D keypoints into poses for all people. Its staged architecture refines PAFs before confidence maps and uses efficient matching over a tree-structured skeleton.
- Overall pipeline: The pipeline transforms a color image into body-part confidence maps and PAF vector fields, then greedily parses them into 2D keypoints for all people.PAFs encode the degree of association between body parts, with one 2D vector field per limb or part pair.
- Staged architecture: A CNN initialized with VGG-19 features predicts PAFs at the first stage and concatenates previous predictions with original features for subsequent refinement.The architecture applies intermediate supervision at each successive stage.
- Staged architecture: Unlike the prior approach, OpenPose refines PAFs before confidence maps, reducing computation per stage by half while using refined affinity fields to improve confidence-map results.Confidence maps begin from the most updated PAF prediction, whereas the opposite refinement direction does not improve results.
- Training objective: The network applies spatially masked L2 losses between predictions and ground-truth maps or fields at each stage, with intermediate supervision replenishing gradients.The binary mask prevents missing annotations from penalizing true-positive predictions.
- Confidence maps: Ground-truth confidence maps place peaks at visible body-part locations for each person and aggregate individual maps with a maximum operator to preserve nearby-peak precision.At test time, body-part candidates are obtained using non-maximum suppression.
- Pose assembly: For each limb, PAF-weighted bipartite matching links detected parts; a spanning-tree skeleton and decomposed subproblems avoid the NP-Hard full matching formulation.The tree-based optimization is orders of magnitude faster than optimization over the fully connected graph.
4 OPENPOSE
OpenPose is an open-source real-time multi-person system that jointly detects body, foot, hand, and facial keypoints. It integrates these detectors across platforms and input types while achieving about 22 FPS with high accuracy on a GTX 1080 Ti.
- Release: OpenPose publicly releases the first real-time multi-person system jointly detecting body, foot, hand, and facial keypoints, totaling 135 keypoints.The system is intended to support computer vision and machine learning applications requiring 2D human pose estimation.
- System integration: OpenPose addresses fragmented pose-estimation tooling by combining facial and body detectors and providing input, visualization, and output handling.Users can select images, video, webcams, or IP camera streams, and choose whether to display or save results.
- Architecture: The system comprises body+foot, hand, and face detection blocks, with facial and hand proposals estimated from body-detector outputs.The core block is the combined body+foot keypoint detector, with an alternative to use original body-only models trained on COCO and MPII.
- Performance: About 22 FPS is achieved on a Nvidia GTX 1080 Ti while preserving high accuracy, and the inference time outperforms all state-of-the-art methods.OpenPose has also been used in person re-identification, video retargeting, human-computer interaction, and 3D pose estimation.
- Foot detection: Foot keypoints beyond ankles are needed for avatar retargeting and 3D human shape reconstruction because missing foot information causes candy wrapper effects, floor penetration, and foot skate.Existing MPII and COCO annotations provide only limited foot information, primarily ankle positions.
- Foot detection: OpenPose trains the previously described body-estimation architecture to predict body and foot locations jointly rather than using a separate top-down foot detector.This approach uses the proposed dataset for foot keypoint detection.
5 DATASETS AND EVALUATIONS
The method is evaluated on MPII, COCO, and a COCO-derived foot dataset, using accuracy and inference speed across body, foot, and broader keypoint tasks. Experiments show strong efficiency and accuracy, while identifying limitations on scale, occlusion, crowded scenes, and atypical poses.
- Datasets: Evaluation covers MPII, COCO, and a COCO-derived foot dataset spanning 14 body parts, 17 body and facial keypoints, and foot annotations.MPII contains 3844 training and 1758 testing groups; COCO includes over 100K labeled person instances and over 1 million keypoints.
- MPII evaluation: On MPII, the method outperforms previous state-of-the-art bottom-up methods by 8.5% mAP on the 288-image subset while reporting inference and optimization time.Performance is measured with mean Average Precision under the PCKh metric.
- MPII evaluation: Minimal tree-edge structures achieve results comparable to fully connected graphs, while the separated minimal-edge model outperforms alternative structures with greater efficiency.The tree uses 13 part-association channels versus 91 for a graph, facilitating training convergence.
- COCO evaluation: COCO evaluation uses Object Keypoint Similarity and mean average precision across 10 OKS thresholds, while the body+foot model improves over the original work by 6.9%.The combined model shows almost no accuracy difference from a body-only model on COCO test-dev despite adding foot outputs.
- Efficiency: The architecture without CPM refinement is approximately 7% more accurate than the original approach while increasing speed ×2.Parsing takes 0.58 ms for 9 people, compared with 36 ms for CNN processing.
- Limitations and generalization: Failures arise from atypical poses, occlusion, overlapping people, and foot localization, while the same architecture also generalizes to vehicle keypoint detection.Rotation augmentation partially addresses atypical poses but reduces COCO validation accuracy by about 5%.
6 CONCLUSION
The paper presents a nonparametric limb-association representation and shows that efficient greedy parsing, PAF-focused refinement, and joint body-foot estimation enable accurate realtime multi-person pose estimation.
- The method represents keypoint associations nonparametrically, encoding both the position and orientation of human limbs.
- A greedy parsing algorithm produces high-quality body-pose parses while preserving efficiency regardless of the number of people.
- PAF refinement is more important than combined PAF and body-part-location refinement, increasing runtime performance and accuracy.
- Combining body and foot estimation in one model improves each component’s accuracy and reduces inference time compared with running them sequentially.
- The authors created a foot keypoint dataset to support combined body and foot estimation.