Source-linked AI summary
CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark
Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, Cewu Lu
TL;DR
Crowded-scene pose estimation remains poorly evaluated because existing benchmarks contain relatively few overlaps and current methods degrade as crowding increases. The paper proposes a joint-candidate SPPE, global graph-based joint association, and CrowdPose dataset. It surpasses state-of-the-art methods by 5.2 mAP on CrowdPose and improves MSCOCO performance by 0.8 mAP, while conventional SPPE remains dependent on high-quality detections.
Problem
Existing benchmarks contain few overlapping people, while current pose-estimation methods degrade in crowded scenes and no public benchmark targets this setting.
Method
The method predicts multiple joint candidates with joint-candidate SPPE, then globally associates them with human proposals using a person-joint graph.
Results
5.2 mAP higher than state-of-the-art methods on CrowdPose, with 0.8 mAP improvement over AlphaPose on MSCOCO using the same detector and SPPE network.
Takeaways & Limitations
CrowdPose evaluates performance across uncrowded and crowded scenes, while experiments show the method generalizes to MSCOCO and remains efficient in inference.
Takeaways & Limitations
Conventional SPPE depends on high-quality human detection, and crowded scenes produce redundant, truncated, or incompact human proposals.
Abstract
from arXiv · showhide
Multi-person pose estimation is fundamental to many computer vision tasks and has made significant progress in recent years. However, few previous methods explored the problem of pose estimation in crowded scenes while it remains challenging and inevitable in many scenarios. Moreover, current benchmarks cannot provide an appropriate evaluation for such cases. In this paper, we propose a novel and efficient method to tackle the problem of pose estimation in the crowd and a new dataset to better evaluate algorithms. Our model consists of two key components: joint-candidate single person pose estimation (SPPE) and global maximum joints association. With multi-peak prediction for each joint and global association using graph model, our method is robust to inevitable interference in crowded scenes and very efficient in inference. The proposed method surpasses the state-of-the-art methods on CrowdPose dataset by 5.2 mAP and results on MSCOCO dataset demonstrate the generalization ability of our method. Source code and dataset will be made publicly available.
1. Introduction
Multi-person pose estimation performs well on existing benchmarks but degrades in crowded scenes, which lack an appropriate public benchmark. The paper introduces a global-association method and CrowdPose dataset to address both problems.
- Top-down methods detect people before estimating poses, whereas bottom-up methods detect joints before associating them into people.
- 67.01% of MSCOCO persons-subset images have no overlapped person, limiting evaluation of crowded-scene pose estimation.
- Crowded scenes cause state-of-the-art methods to degrade, with wrong joint assembly and redundant pose predictions among the main errors.
- The method predicts multiple joint candidates, builds a person-joint graph, and globally solves joint association for detected human proposals.
- CrowdPose uniformly distributes images across Crowd Index [0, 1], requiring strong performance in both uncrowded and crowded scenes for a high score.
- Using the same ResNet-101 backbone, the method surpasses state-of-the-art methods by 5.2 mAP on CrowdPose and improves MSCOCO by 0.8 mAP.
2. Related Work
Prior multi-person pose datasets expanded evaluation beyond single-person settings, while related methods use part-based or two-step frameworks. Two-step methods score highly but depend strongly on human detection and fail in crowds.
- Early RGB pose datasets evaluated single-person estimation, while MPII, MSCOCO, and AI Challenger support multi-person pose estimation.
- Part-Based Framework: Part-based methods detect joints and associate them into complete people using affinity fields, displacements, or pose residual networks.
- Two-Step Framework: Two-step methods first detect human proposals and then perform single-person pose estimation, achieving higher scores than part-based methods.
- Two-Step Framework: Two-step approaches depend heavily on human detection because close people make it unlikely that a crop contains only one person.
3. Our Method
The proposed pipeline combines joint-candidate single-person pose estimation with graph-based global joint association. It converts candidate outputs into a person-joint matching problem and selects the best association.
- Human detector bounding-box proposals are fed into joint-candidate SPPE, which locates joint candidates with different heatmap response scores.
- The association algorithm builds a person-joint connection graph from human proposals and candidate joints.
- A global maximum joints association algorithm solves the graph-matching problem to find the best joint assignments.
3.1. Joint-Candidates SPPE
Joint-candidate SPPE preserves both target and interference joints with different response intensities, allowing later global association to resolve crowded-scene ambiguity. This addresses the brittleness of conventional SPPE, whose mistakes and missing joints cannot be recovered by pose-NMS.
- Joint-candidate SPPE receives proposal images and outputs heatmaps containing joints from the target person and other human instances.
- The method uses a global-view loss because conventional SPPE’s limited receptive fields cannot reliably suppress interference joints in crowded scenes.
- Interference joints are retained as candidates because they may be target joints for other proposals, so the network outputs target and interference candidates with different intensities.
- Loss Design: The heatmap design represents target joints with 2D Gaussian responses and interference joints with Gaussian-mixture responses.
- Loss Design: µ = 0.5 attenuates interference joints without over-suppressing them, while conventional heatmap loss is the special case µ = 0.
- Discussion: Conventional SPPE errors are unrecoverable because missing joints cannot be restored during post-processing, whereas candidate loss encourages multi-peak predictions with high recall.
3.2. Person-Joint Graph
The method represents candidate joints and detected human proposals as nodes in a person-joint graph, connecting proposals to candidate-joint groups for globally optimized pose construction.
- Joint Node Building: The person-joint graph reduces redundant joints by grouping candidates representing the same actual joint into one joint node.This addresses cases where overlapped human proposals predict the same joint.
- Joint Node Building: Joint candidates are grouped when their locations satisfy a deviation-based criterion controlled by the joint-specific parameter δ(k).The criterion uses the Gaussian response sizes of the two joints and requires mutual inclusion within their control domains.
- Joint Node Building: The joint node set contains nodes indexed by body part, with Nk denoting the number of joint nodes for body part k.Each node represents a cluster of candidate joints for one body part.
- Person Node Building: Person nodes represent the human proposals produced by the detector, including potentially truncated, incompact, or redundant bounding boxes.Low-quality person nodes are eliminated during global person-joint matching.
- Person-Joint Edge: An edge connects person node hi to joint node vk_j when that joint node contains a candidate from the proposal.The edge weight is the candidate joint’s heatmap response score wk_i,j.
3.3. Globally Optimizing Association
Global association formulates pose estimation as maximizing weighted person-joint assignments, solves independent body-part subgraphs, and achieves quadratic complexity comparable to greedy NMS.
- Global Association: Pose estimation is transformed into solving the person-joint graph while maximizing the total edge weights.The objective selects assignments based on candidate response scores.
- Global Association: The global assignment is mathematically equivalent to solving each body-part subgraph separately with an updated Kuhn-Munkres algorithm.Each subgraph is bipartite and connects person nodes with the joint subset for one body-part type.
- Global Association: The matched joint nodes are converted into pose joints using response-weighted centers, while person nodes without matches are removed.This constructs the final pose for each retained human proposal.
- Computational Complexity: O(|H|^2) is the resulting association complexity, matching the complexity of conventional greedy NMS algorithms.Sparse linear assignment is solved in O(n^2), and the expected number of joint nodes equals the number of human proposals.
3.4. Discussion
The graph-based association globally matches joints to proposals, allowing redundant proposals to be rejected and addressing missing-joint and wrong-assembly cases more effectively than instance-based NMS.
- Discussion: Global graph matching lets human proposals compete for joint nodes and rejects proposals whose joint response scores are relatively low.This removes many redundant and poor human proposals without relying on instance-level elimination.
- Discussion: Unlike conventional and pose-based NMS, the method reduces redundancy at the joint level to handle missing joints and wrong assembling.The cited NMS alternatives are described as instance-based approaches.
4. CrowdPose Dataset
CrowdPose is designed to evaluate pose estimation across crowding levels using a Crowd Index based on occlusion, with uniformly distributed, re-annotated data.
- 4.1. Crowding Level Definition: The Crowd Index measures crowding through occlusion between joints belonging to different people within human bounding boxes.It averages each person’s crowd ratio across all persons in an image.
- 4.1. Crowding Level Definition: Three public benchmarks are dominated by uncrowded images, where mutual occlusion is limited.In MSCOCO’s persons subset, 67.01% of images contain no overlapping person.
- 4.3. Annotation: The images are re-annotated with 14 keypoints, full-body boxes, and interference keypoints, followed by cross-annotation quality checks.Images with large annotation deviations are re-annotated, and keypoint locations are averaged.
- 4.4. Dataset Statistics: 20,000 images containing about 80,000 persons remain in the dataset, split into training, validation, and testing subsets in a 5:1:4 ratio.CrowdPose has a uniform Crowd Index distribution rather than concentrating only on crowded scenes.
- 4.4. Dataset Statistics: CrowdPose has an average human bounding-box IoU of 0.27, compared with 0.06 for MSCOCO, 0.11 for MPII, and 0.12 for AI Challenger.This statistic reflects greater box overlap in CrowdPose than in the comparison datasets.
5. Experiments
Experiments evaluate the proposed method on CrowdPose and MSCOCO using standard keypoint metrics, comparisons, speed measurements, and component ablations. The method improves performance on crowded scenes while retaining generalization and efficient inference.
- 5.2. Implementation Details: The proposed two-step system uses YOLOv3 for human detection, AlphaPose’s pose-estimation network, and joint-candidate SPPE during inference.Detected boxes are expanded by 30% before joint-candidate estimation.
- 5.4. Results: 5.2 mAP separates the proposed method from state-of-the-art methods on the CrowdPose test set.The gains are 4.1 mAP on uncrowded, 4.9 mAP on medium-crowded, and 6.2 mAP on extremely crowded scenes.
- 5.4. Results: 70.9 mAP is achieved on the MSCOCO test-dev set, improving 0.8 mAP over AlphaPose with the same human detector and SPPE network.The comparison uses the same detection backbone for fairness.
- 5.4. Results: 10.1 FPS is achieved on the test set, slightly slower than AlphaPose but faster than the other compared methods.The authors report this as the most accurate yet efficient configuration for crowded cases.
- 5.5. Ablation Studies: Replacing joint-candidate loss with mean square loss reduces final mAP from 66.0% to 61.7%.The ablation supports the role of joint-candidate loss in predicting possible joints and resisting interference.
6. Conclusion
The paper addresses occluded pose estimation with a global graph-based method and introduces CrowdPose for evaluating crowded scenes. Experiments report strong CrowdPose performance and generalization to different scenarios.
- The method addresses occlusion in human pose estimation using a person-joint graph and global optimization.
- CrowdPose is established with a normal distribution of Crowd Index to evaluate performance in crowded scenes.
- Experiments show the proposed method significantly outperforms state-of-the-art methods on CrowdPose.
- Results on MSCOCO demonstrate that the method can generalize to different scenarios.