Source-linked AI summary
DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, Bernt Schiele
TL;DR
Multi-person pose estimation must jointly handle occlusion, close proximity, and an unknown number of people rather than separating detection from pose estimation. DeepCut formulates this as an ILP that partitions and labels CNN-generated body-part candidates, achieving improvements across single- and multi-person benchmarks, including a 30% PCP gain on WAF over a traditional two-stage approach.
Problem
Multi-person pose estimation must infer person count, part assignments, occlusions, and poses in overlapping real-world scenes where two-stage detection and independent pose estimation are unsuitable.
Method
DeepCut jointly selects, labels, and partitions CNN-generated body-part candidates in an ILP using geometric and appearance constraints.
Results
30% PCP improvement on the multi-person WAF dataset over the traditional two-stage approach, with significant improvements reported for both multi-person and single-person pose estimation.
Takeaways & Limitations
A joint formulation is crucial for disambiguating multiple people in close proximity and supports state-of-the-art pose estimation across single- and multi-person tasks.
Takeaways & Limitations
For images containing at most one person, the feasible solutions require every pair of nonsuppressed distinct body parts to be clustered together.
Abstract
from arXiv · showhide
This paper considers the task of articulated human pose estimation of multiple people in real world images. We propose an approach that jointly solves the tasks of detection and pose estimation: it infers the number of persons in a scene, identifies occluded body parts, and disambiguates body parts between people in close proximity of each other. This joint formulation is in contrast to previous strategies, that address the problem by first detecting people and subsequently estimating their body pose. We propose a partitioning and labeling formulation of a set of body-part hypotheses generated with CNN-based part detectors. Our formulation, an instance of an integer linear program, implicitly performs non-maximum suppression on the set of part candidates and groups them to form configurations of body parts respecting geometric and appearance constraints. Experiments on four different datasets demonstrate state-of-the-art results for both single person and multi person pose estimation. Models and code available at http://pose.mpi-inf.mpg.de.
1. Introduction
Multi-person pose estimation must handle occlusion, overlapping people, and an unknown number of persons, challenges that two-stage detect-then-estimate strategies do not address well. DeepCut therefore jointly partitions and labels body-part candidates to infer poses and person instances.
- Motivation: Real-world multi-person images contain partial visibility, overlapping people, and an unknown number of persons, requiring joint inference of identities and poses.The task assigns part detections to person instances while respecting geometric and appearance constraints.
- Contribution: DeepCut formulates joint detection and pose estimation as partitioning and labeling body-part candidates into mutually consistent subsets.The model can infer the number of people by linking part hypotheses and can deactivate or merge candidates, effectively performing non-maximum suppression.
- Method overview: DeepCut combines CNN-based body-part candidates with geometric and appearance constraints to assemble valid configurations.The approach jointly clusters detections into people and labels each part class.
- Related work: The approach targets full-body multi-person pose estimation, unlike related methods focused on upper-body poses or simplified frontal views.Related joint models also differ in their use of person detectors and assumptions about occlusion states.
- Novelty: The formulation uses a fully connected model rather than a fixed-part tree, supporting an unknown number of people and variable body parts per person.This contrasts with earlier integer-linear-programming pose estimation designed for a single person with a fixed number of parts.
- Related work: Correlation clustering, also called the Minimum Cost Multicut Problem, had not previously been used for pose estimation and is NP-hard.DeepCut connects this partitioning perspective with pose-estimation inference.
2. Problem Formulation
DeepCut casts multi-person pose estimation as an ILP that jointly selects, labels, and partitions body-part candidates. Its objective combines unary and pairwise probabilities under feasibility constraints, with branch-and-cut used to obtain bounded solutions.
- 2. Problem Formulation: The formulation jointly selects a subset of body-part candidates, assigns body-part classes, and partitions selected parts by person.These three decisions are represented as an optimization problem for an unknown number of people.
- 2.1. Feasible Solutions: Binary variables x, y, and z encode part-class assignments, same-person relationships, and their joint consistency.zdd′cc′ links two candidate labels with the relation that both candidates belong to the same person.
- 2.1. Feasible Solutions: Feasibility constraints suppress unlabeled candidates, prevent suppressed parts from being grouped, and enforce consistent same-person relationships.The constraints restrict binary labelings to well-defined articulated poses.
- 2.1. Feasible Solutions: Auxiliary variables make the objective linear in z instead of cubic in x and y.The feasible set XDC contains all triples satisfying the stated inequalities.
- 2.1. Feasible Solutions: When at most one person is present, additional inequalities force every pair of nonsuppressed distinct parts into the same cluster.This is an explicit single-person constraint on feasible solutions.
- 2.2. Objective Function: Unary probabilities pdc score candidate-class assignments, while pairwise probabilities pdd′cc′ model same-person relationships between candidate parts.Pairwise terms form a fully connected graph and also support clustering duplicate same-part candidates while repelling nearby candidates of the same type.
- 2.2. Objective Function: The subset partition and labeling problem is an ILP that minimizes an objective over the feasible solution set XDC.The objective is the MAP estimate of joint detections and clusterings.
- 2.3. Optimization: Branch-and-cut separates feasibility inequalities during successive ILP relaxations, adding violated constraints until an integer solution satisfies them.The procedure uses Gurobi and provides guaranteed bounds for feasible solutions.
3. Pairwise Probabilities
Pairwise probabilities encode whether detections represent compatible body parts from the same person, using geometric, appearance, and class-specific relations. These probabilities supply coefficients for the joint optimization model.
- Pairwise features capture proximity, kinematic relations, and appearance similarity between body-part detections.
- For detections of the same body-part class, features encode normalized spatial differences, box overlap, and nonlinear mappings of these quantities.The normalized differences use the average detection scale, while overlap includes intersection-over-union, minimum, and maximum variants.
- For different body-part classes, posterior probabilities over distance and angle encode geometric relations between the two classes.The posterior is estimated from normalized two-dimensional histograms of positive and negative training examples.
- Appearance scores from the CNN detectors can be concatenated with geometric features to form the final pairwise representation.
- The model estimates unary probabilities for each detection-class assignment and pairwise probabilities for two detections belonging to the same person with specified classes.Parameters are learned with logistic regression under a Gaussian prior, using |C| × (|C| + 1) / 2 parameter sets.
- During inference, unary and pairwise probabilities are converted into objective coefficients for the integer-linear optimization.
4. Body Part Detectors
The paper develops two CNN-based body-part detectors: an adapted region-proposal architecture and a fully convolutional dense architecture. Detector design choices improve localization and close-part modeling, supporting strong pose-estimation results.
- The detector evaluation compares adapted Fast R-CNN and Dense-CNN models on the LSP, LSP Extended, and MPII benchmarks.The experiments use PCK and area under the PCK curve as evaluation measures.
- AFR-CNN adapts Fast R-CNN for body parts by changing proposal generation and detection-region size.Fast R-CNN receives an image and class-independent region proposals and outputs class probabilities with refined bounding boxes.
- Dense-CNN uses fully convolutional scoremaps because region proposals may be suboptimal for body-part detection.The architecture reduces VGG’s stride from 32 pixels to 8 pixels using a hole algorithm and selects a 340-pixel standing height for scale.
- Multi-label sigmoid outputs allow multiple nearby body parts to receive high probabilities, outperforming softmax and converging faster than MSE.Location refinement and regression to other parts are added as auxiliary objectives to improve localization and performance.
- 87.1% PCK is achieved after finetuning to LSP, while successive training and architecture changes raise PCK from 83.0% to 86.1%.The reported progression includes larger-data training, sigmoid cross-entropy, location refinement, and regression to other parts.
- The Dense-CNN detector uses a representative detection set for optimization and sigmoid unary scores, while the SPLP problem remains NP-hard.The default detection-set size is |D| = 100.
5. DeepCut Results
DeepCut models perform strongly on both single-person and multi-person pose estimation, with the joint multi-person formulation improving robustness to occlusion, overlap, and unknown person counts.
- Single Person Pose Estimation: DeepCut models significantly outperform state-of-the-art methods on LSP single-person pose estimation.DeepCut SP Dense-CNN improves by 13.7% PCK over the best known result on LSP.
- Single Person Pose Estimation: DeepCut SP AFR-CNN improves over AFR-CNN alone on MPII, reaching 79.8% versus 78.8% PCK and 51.1% versus 49.0% AUC.
- Single Person Pose Estimation: 82.4% PCK is achieved by DeepCut SP Dense-CNN on MPII Single Person, exceeding competing results of 79.6% and 82.0%.
- Multi Person Pose Estimation: DeepCuts outperform prior methods on WAF, including Chen&Yuille at 84.7 versus 80.7% mPCP and 86.5 versus 84.9% AOP.
- Multi Person Pose Estimation: DeepCut MP AFR-CNN improves over the detection-ROI baseline by 4.3%, achieving 51.4% AP on MPII Multi-Person.The largest gains occur for ankle, knee, elbow, and wrist parts.
6. Conclusion
The paper formulates multi-person pose estimation as joint subset partitioning and labeling rather than separate detection and pose-estimation stages.
- DeepCut jointly infers the number of people, their poses, spatial proximity, and part-level occlusions.
- The formulation improves results over previous methods on both multi-person and single-person pose estimation across four datasets.
A. Additional Results on LSP dataset
Additional LSP experiments report person-centric and observer-centric evaluation results using PCK and PCP measures.
- The additional LSP evaluation uses both person-centric and observer-centric settings.
A.1. LSP Person-Centric (PC)
Additional person-centric LSP results show strong performance from the proposed detectors and DeepCut models across PCK, AUC, and PCP evaluations.
- AFR-CNN Performance: 82.8% PCK and 57.0% AUC are achieved by VGG AFR-CNN fine-tuned on LSP.This improves over the shallower AlexNet configuration.
- PCP Evaluation: 78.3% PCP is achieved by AFR-CNN under the strict PCP measure.DeepCut SP AFR-CNN slightly improves over unary-only inference by enforcing more consistent body-part-stick predictions.
- Comparison to State of the Art: DeepCut models outperform prior methods by large margins on LSP PCP, with DeepCut SP Dense-CNN exceeding Fan et al. by 14.2% PCP.
- Comparison to State of the Art: DeepCut SP Dense-CNN outperforms Tompson et al. by 17.7% PCP.
A.2. LSP Observer-Centric (OC)
On LSP observer-centric annotations, both proposed CNN part detectors outperform prior methods across PCK and PCP evaluation, with Dense-CNN consistently strongest. The results also show that performance differences widen at smaller PCK thresholds.
- Observer-centric annotations count body parts by their side relative to the pelvis–neck line, without penalizing right/left prediction flips.
- 84.2% PCK and 58.1% AUC are achieved by AFR-CNN on LSP (OC).
- 88.2 vs. 84.2% PCK and 65.0 vs. 58.1% AUC show that Dense-CNN outperforms AFR-CNN.The differences are more pronounced at smaller PCK distance thresholds.
- +10.7 and +18.2% PCK improvements over Chen&Yuille and Ouyang et al. are reported for Dense-CNN.Both proposed approaches significantly outperform other methods.
- 85.0% PCP is achieved by Dense-CNN, improving by +10% PCP over the best published result.
B. Additional Results on WAF dataset
Figure 5 qualitatively compares DeepCut MP Dense-CNN with a traditional two-stage detector-based approach and Chen&Yuille on the WAF dataset. The passage directs readers to the figure caption for visual performance analysis.
- Figure 5 presents a qualitative comparison on the WAF dataset.
- The comparison includes DeepCut MP Dense-CNN and the traditional two-stage Dense-CNN det ROI approach.
- Chen&Yuille is included as an additional comparison method.
C. Additional Results on MPII Multi-Person
On MPII Multi-Person, the joint formulation is compared with a detector-based two-stage approach and is designed to partition candidates despite proximity, occlusion, and truncation. Its pairwise terms cluster compatible parts, repel conflicting assignments, and permit suppression or deactivation of hypotheses.
- Dense-CNN det ROI works well for fully visible, sufficiently separated people whose parts can be partitioned by detection bounding boxes.
- Dense-CNN det ROI often links body parts across nearby or occluding individuals and predicts every body part despite occlusions and truncations.
- Pairwise terms cluster multiple candidates of the same body part belonging to the same person and repel nearby conflicting candidates.
- DeepCut MP Dense-CNN can merge or deactivate part hypotheses, effectively performing non-maximum suppression and handling occlusions and truncations.