Source-linked AI summary
Multi-Person Pose Estimation with Local Joint-to-Person Associations
Umar Iqbal, Juergen Gall
TL;DR
Multi-person pose estimation must handle occlusion, truncation, and assigning detected joints to the correct individuals. The paper uses local integer-linear-programming association over joint candidates, achieving state-of-the-art accuracy with substantially lower runtime. On MPII, it is reported as 6,000 to 19,000 times faster than the prior approach.
Problem
Existing pose estimators assume a single visible person, while multi-person images require handling unknown counts, occlusions, truncations, and joint-to-person association.
Method
The method detects people, generates joint candidates, constructs a fully connected graph for each person, and solves local association and outlier removal with integer linear programming.
Results
6,000 to 19,000 times faster runtime accompanies accuracy matching or slightly improving on a state-of-the-art method.
Takeaways & Limitations
Independent local association problems provide an efficient alternative to globally modeling all persons while remaining effective under severe occlusions and truncations.
Takeaways & Limitations
Accuracy can be further improved with a better person detector, and the method’s scope is evaluated on the MPII Multi-Person Pose Dataset.
Abstract
from arXiv · showhide
Despite of the recent success of neural networks for human pose estimation, current approaches are limited to pose estimation of a single person and cannot handle humans in groups or crowds. In this work, we propose a method that estimates the poses of multiple persons in an image in which a person can be occluded by another person or might be truncated. To this end, we consider multi-person pose estimation as a joint-to-person association problem. We construct a fully connected graph from a set of detected joint candidates in an image and resolve the joint-to-person association and outlier detection using integer linear programming. Since solving joint-to-person association jointly for all persons in an image is an NP-hard problem and even approximations are expensive, we solve the problem locally for each person. On the challenging MPII Human Pose Dataset for multiple persons, our approach achieves the accuracy of a state-of-the-art method, but it is 6,000 to 19,000 times faster.
1 Introduction
Multi-person pose estimation must handle unknown person counts, occlusions, truncations, and joint-to-person association. The proposed local approach addresses these challenges while matching state-of-the-art accuracy and greatly reducing runtime.
- Multi-person images require estimating the number of people, handling occlusions and truncations, and associating joints with the correct person.
- Independent pose estimation after person detection does not resolve joint association for nearby people or truncated bodies.
- 72 hours per image made the prior globally optimized approach impractical despite state-of-the-art results on a small MPII subset.
- The proposed method solves joint-to-person association locally with integer linear programming after detecting people and cropping contextual regions.
- 6,000 to 19,000 times faster runtime accompanies slightly improved accuracy over the prior approach on MPII multi-person evaluation.
2 Related Work
Earlier multi-person methods struggle with nearby or occluded people, whereas global graph optimization handles broader cases at prohibitive computational cost. The paper motivates a locally solved alternative to that trade-off.
- Some graphical-model approaches generate multiple pose hypotheses but require people to be distant and fully visible.
- Person-detector pipelines estimate poses independently but do not fully address joint association in crowded or truncated cases.
- DeepCut jointly labels proposals, suppresses incompatible parts, and assigns joints to unique individuals without requiring a person detector or known person count.
- DeepCut’s NP-hard optimization limits proposals and requires roughly 72 hours for one image, restricting practical use.
- A later approach reduces optimization time with stronger detectors, spatial models, and incremental optimization, but builds on the same global formulation.
3 Overview
The method detects people, generates joint candidates within person-centered regions, and uses local graph inference to produce final poses. Integer linear programming resolves associations and removes erroneous candidates.
- A person detector first identifies people and provides image regions centered on each detected person.
- A single-person pose model generates joint candidates for every detected person, although occlusion and truncation can make these candidates erroneous.
- For each person, a fully connected graph is solved locally with integer linear programming to associate joints and remove outliers.
4 Convolutional Pose Machines
Convolutional Pose Machines use staged confidence-map prediction: the first stage captures all visible people, while later stages use context to focus on the primary person. Candidate sampling and ILP then address invisible or misassociated joints.
- The CPM represents a pose as 14 two-dimensional body-joint locations and predicts confidence maps through multiple CNN stages.
- The first stage uses local image evidence, while subsequent stages incorporate preceding score maps and expanded receptive fields for spatial context.
- Each stage is locally supervised with an L2 loss against Gaussian target score maps centered on ground-truth joints.
- For multi-person training, stage 1 includes joints from all people, whereas later stages target only the primary person and suppress others.
- Taking the maximum score-map response can produce erroneous invisible-joint estimates or associate nearby people’s joints incorrectly.
- The method samples multiple candidates per joint and uses integer linear programming to resolve associations and remove outliers.
5 Joint-to-Person Association
The method reformulates joint-to-person association as a local integer-linear-programming problem around each detected person, reducing the cost of global association while retaining appearance and spatial compatibility cues.
- Local association: The proposed model solves joint-to-person association locally for each person instead of globally across all image regions.It uses detected joint candidates with known joint types and assigns selected detections to the primary person.
- Computational reduction: The local formulation drastically reduces the number of variables and constraints compared with the global formulation.This reduction follows from restricting association to a primary person rather than jointly modeling all persons.
- ILP formulation: Integer linear programming selects unsuppressed detections and connects detections that belong to the same person.The constraints enforce valid connections, transitivity, and membership in the primary person's pose.
- Detection confidence: Joint-detection confidence comes from CPM score maps, with thresholding used to suppress low-confidence detections.The resulting confidence is used as a probability in the local association objective.
- Pairwise compatibility: Pairwise association probabilities model whether detections belong to the same joint or to different joint types of the same person.The probabilities are learned from appearance and spatial features of detection candidates.
6 Experiments
Experiments on the MPII Multi-Person Pose Dataset evaluate parameter choices, detector sensitivity, validation performance, and comparisons with prior approaches. The proposed method achieves strong accuracy while substantially reducing runtime.
- Dataset and setup: The MPII Multi-Person subset contains 3,844 training and 1,758 testing images with multiple persons, including substantial occlusions and truncations.Intermediate experiments use a validation set of 1,200 images because the original test data is withheld.
- Parameter study: Increasing τ improves mean AP until τ = 0.3, while values above 0.4 reduce accuracy by discarding correct detections; τ = 0.2 is used thereafter.The parameter study is conducted on the 1,200-image validation set.
- Validation results: 49.2% mAP with L-JPA and N = 1 improves over 45.2% mAP with CPM alone, with only a slight runtime increase.Increasing the number of candidates per joint yields only slight additional accuracy improvements, so N = 5 is used subsequently.
- Detector sensitivity: 76.9% mAP with GT Torso detections versus 49.3% mAP with the off-the-shelf detector shows that detector quality strongly affects accuracy.The detector is not fine-tuned on MPII, and small detections are discarded for quantitative evaluation.
- Comparison with prior approaches: 54.7% versus 53.5% mAP against DeepCut, with runtimes of 10 seconds versus 57,995 seconds, shows comparable accuracy with much lower computation.With N = 1, the proposed approach requires 3 seconds per image and is more than 19,000 times faster than DeepCut on 288 test images.
- Full test set: On all MPII Multi-Person test images, the method achieves 43.1% mAP in 10 seconds per image and is around 50 times faster than the concurrent approach.Using GT Torso detections raises accuracy to 62.2% mAP; qualitative examples are shown in Figure 7.
7 Conclusion
The paper presents local joint-to-person association for multi-person pose estimation under occlusions and truncations. It reports state-of-the-art accuracy with substantially lower runtime than global modeling approaches.
- Conclusion: The method formulates multi-person pose estimation as independent local joint-to-person association problems rather than one global model.This decomposition targets cases involving occlusions and truncations.
- Conclusion: Local association problems can be solved efficiently while remaining effective for severe occlusions or truncations.The conclusion contrasts this with the impracticality of globally modeling poses for all persons.
- Conclusion: The proposed method achieves the accuracy of a state-of-the-art method while being 6,000 to 19,000 times faster.The authors note that a better person detector could further improve accuracy.