PaperlayerView exact revision on arXiv

Source-linked AI summary

DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model

Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, Bernt Schiele

arXiv:1605.03170v3cs.CV

TL;DR

Multi-person pose estimation requires assembling body-part detections into consistent configurations for an unknown number of people. DeeperCut strengthens detectors and pairwise reasoning, then uses incremental optimization to search more efficiently, advancing multi-person accuracy while reducing runtime and retaining strong single-person performance.

  • Problem

    Multi-person pose estimation is a challenging structured prediction problem involving multiple potentially overlapping people and a variable number of outputs.

  • Method

    DeeperCut combines strong body-part detectors, image-conditioned pairwise terms, and incremental optimization within the DeepCut integer-linear-programming framework.

  • Results

    The approach significantly advances multi-person pose estimation while demonstrating competitive single-person performance, with runtime reductions of 2–3 orders of magnitude and a further 4x reduction from incremental optimization.

  • Takeaways & Limitations

    Strong detectors and image-conditioned pairwise terms improve multi-person pose estimation, while incremental optimization improves accuracy and reduces inference runtime.

  • Takeaways & Limitations

    The underlying DeepCut ILP is APX-hard, so direct branch-and-cut optimization is not necessarily practical.

Abstract

from arXiv · show

The goal of this paper is to advance the state-of-the-art of articulated pose estimation in scenes with multiple people. To that end we contribute on three fronts. We propose (1) improved body part detectors that generate effective bottom-up proposals for body parts; (2) novel image-conditioned pairwise terms that allow to assemble the proposals into a variable number of consistent body part configurations; and (3) an incremental optimization strategy that explores the search space more efficiently thus leading both to better performance and significant speed-up factors. Evaluation is done on two single-person and two multi-person pose estimation benchmarks. The proposed approach significantly outperforms best known multi-person pose estimation results while demonstrating competitive performance on the task of single person pose estimation. Models and code available at http://pose.mpi-inf.mpg.de

1 Introduction

DeeperCut advances multi-person pose estimation through stronger body-part detectors, image-conditioned pairwise terms, and faster incremental optimization. The approach is evaluated across single-person and multi-person benchmarks, with reported improvements in both accuracy and runtime.

  • Contributions: 2–3 orders of magnitude runtime reduction results from combining image-conditioned pairwise terms with very good part-detection candidates in a fully-connected model.The paper presents this as the “faster” contribution, alongside a further 4x reduction from incremental optimization.
  • Contributions: A novel incremental optimization method further reduces runtime by 4x while improving human pose estimation accuracy.The approach explores the optimization problem incrementally rather than fully exploring the search space at once.
  • Evaluation: The approach is evaluated on two single-person and two multi-person pose estimation benchmarks, reporting the best results in each case.Sample multi-person predictions are shown in Fig. 1.
  • Motivation: Multi-person pose estimation remains more complex than single-person estimation because images contain multiple potentially overlapping people and a variable number of outputs.The paper frames this as a more complex structured prediction problem requiring joint reasoning over multiple people.
  • Motivation: Strong body-part detectors provide effective bottom-up proposals and can incorporate much contextual information directly into detection.The paper argues that spatial reasoning offers diminishing returns with strong detectors, while multi-person inference still benefits substantially from them.
  • Contributions: Image-conditioned pairwise terms use local image evidence to group body-part proposals, improving on purely geometric terms when people are closely positioned.Geometric terms only approximately guide articulated body parts and are less discriminative in crowded scenes.

2 DeepCut Recap

DeepCut formulates multi-person pose estimation as an integer linear program that jointly selects, labels, and clusters body-part candidates into person configurations. Because the resulting problem is computationally hard, the paper motivates incremental optimization.

  • DeepCut formulation: DeepCut jointly estimates all people in an image by partitioning and labeling body-part candidates into consistent configurations.It serves as a general optimization framework in which different components can be replaced and combined.
  • DeepCut formulation: Body-part candidates receive unary costs or rewards for assigning each candidate to a body-part class.Candidates are drawn from putative detections, while classes include examples such as head, shoulder, and knee.
  • DeepCut formulation: Pairwise terms assign costs or rewards when two classified body-part candidates belong to the same person.These terms connect candidate pairs across body-part classes within the joint pose-estimation objective.
  • DeepCut formulation: Binary x variables indicate candidate-class assignments, while binary y variables indicate whether candidate pairs belong to the same person.Additional variables and constraints ensure selection, classification, and clustering define feasible person configurations.
  • Optimization: The DeepCut ILP generalizes minimum-cost multicut or correlation clustering and is APX-hard, making direct branch-and-cut optimization potentially impractical.The paper therefore introduces incremental optimization, reporting 4–5x runtime reduction with increased pose-estimation accuracy.

3 Part Detectors

The paper develops deep fully-convolutional body-part detectors based on ResNet, intermediate supervision, and learned spatial representations, then evaluates them on single-person benchmarks.

  • Architecture: The detector uses a 152-layer ResNet adapted for fully-convolutional human body-part detection.The architecture draws on advances in semantic segmentation, object classification, and pose estimation.
  • Evaluation: The evaluation uses PCK and AUC on LSP, and reports PCKh and AUC for MPII Single Person.The LSP evaluation uses standard PCK at a fixed threshold and AUC across the full range of thresholds.
  • Architecture: Intermediate supervision improves early representations for later spatial disambiguation of body parts.On LSP, it raises performance to 90.1% PCK and 66.1% AUC.
  • Results: On LSP, the best model achieves 90.1% PCK and 66.1% AUC, outperforming DeepCut’s 87.1% PCK.Its 90.1% PCK is close to Wei et al.’s 90.5%, while its 66.1% AUC exceeds 65.4%.
  • Results: On MPII Single Person, intermediate supervision improves ResNet-152 to 88.5% PCKh and 60.8% AUC.This improves over DeepCut by +5.9% PCKh and +4.2% AUC, while matching Wei et al. at 88.5% PCKh.

4 Image-Conditioned Pairwise Terms

The paper uses deep networks to predict relative positions between body parts, then converts bidirectional agreement and geometric features into image-conditioned pairwise costs for grouping detections. These terms improve multi-person pose estimation while supporting efficient inference through refined detections and incremental search.

  • 4.1 Model: Deep networks predict relative positions from each detected joint to every other joint, extending body-part detection with image-conditioned spatial information.The model jointly trains body-part detection, location regression, and pairwise regression losses.
  • 4.1 Model: Long-range regression predictions may be less accurate for difficult poses, but they still constrain the fully connected spatial-model search space.The figure contrasts nearby and longer-range shoulder and limb predictions.
  • 4.1 Model: Pairwise costs compare observed detection offsets with CNN-predicted offsets in both directions, measuring agreement through distances and angles.The resulting features include forward and backward distances, angles, and exponential transformations before logistic scoring.
  • 4.2 Inference: Location refinement moves detections toward predicted joint locations before non-maximum suppression, increasing candidate density around true locations.The refined candidates can be distributed more effectively for subsequent grouping.
  • 4.3 Evaluation of Pairwise: 52.6% AP and 578 s/frame are achieved when bidirectional predictions include angles, outperforming variants that omit angles or use one direction.Single-joint regression reaches 51.1% AP with 4× the runtime, while removing angles gives 51.3% AP and doubles runtime.

5 Incremental Optimization

The paper replaces one difficult whole-image DeepCut optimization with incremental ILP instances, prioritizing reliable body-part classes and enabling more candidates. This improves accuracy while substantially reducing runtime on multi-person benchmarks.

  • Motivation: The original whole-image optimization can miss body parts or entire people because branch-and-cut runtime limits the candidate pool.Its worst-case computation time can be exponential in the number of body-part candidates.
  • Method: Incremental optimization solves several ILP instances, beginning with reliably detected body-part classes and adding less reliable classes later.Clusters from one stage become fixed-class candidates in the next stage, reducing the remaining optimization problem.
  • Evaluation: 54.5 vs. 52.6% AP results from stronger candidate distribution through more aggressive NMS with a 24 px radius.Increasing candidates from |D| = 100 to 150 adds only +0.6% AP while increasing runtime from 596 to 1041 s/frame.
  • Evaluation: 57.6% AP is achieved by three-stage incremental inference, reducing runtime to 271 s/frame while improving performance for nearly all body parts.Two-stage inference already halves median runtime from 1041 to 483 s/frame and achieves 56.5 vs. 55.1% AP.
  • Comparison: 58.7 vs. 33.3% AP compares DeeperCut with DeepCut, while runtime falls from 259220 to 270 s/frame.The comparison attributes the overall improvement to better unary and pairwise terms together with incremental optimization.
  • Comparison: 69.7% AP improves 16.5% AP over the baseline on the MPII Multi-Person subset, while runtime drops from 57995 to 230 s/frame.On the full testing set, three-stage optimization improves 59.4 vs. 54.7% AP, although performance is over 10% AP lower than on the subset.
  • Comparison: DeeperCut achieves 86.3 vs. 84.7% mPCP and 88.1 vs. 86.5% AOP against DeepCut on WAF.Upper- and lower-arm improvements are +2.3% and +2.4% mPCP, respectively.

6 Conclusion

The paper advances articulated multi-person pose estimation by redesigning detectors, pairwise terms, and optimization. Together, these improvements nearly double pose accuracy while reducing runtime by three orders of magnitude.

  • Conclusion: The authors propose strong deep body-part detectors that achieve state-of-the-art performance on standard pose-estimation benchmarks independently.These detectors constitute the paper’s “deeper” contribution.
  • Conclusion: Novel image-conditioned pairwise terms improve multi-person pose estimation and reduce fully-connected inference runtime.They constitute the paper’s “stronger” contribution.
  • Conclusion: Incremental optimization further reduces runtime and improves pose-estimation accuracy.It constitutes the paper’s “faster” contribution.
  • Conclusion: Pose accuracy nearly doubles in the challenging multi-person setting while runtime decreases by 3 orders of magnitude.This overall result combines the proposed detector, pairwise, and optimization improvements.

Appendix A Qualitative Evaluation on MPII Multi-Person Dataset

The qualitative evaluation examines DeeperCut on MPII Multi-Person through visualizations of image-conditioned pairwise terms and examples of successful and failed predictions.

  • Qualitative Evaluation: The qualitative analysis evaluates DeeperCut on the MPII Multi-Person dataset.The analysis includes both visual inspection of pairwise terms and prediction examples.
  • Qualitative Evaluation: The evaluation visualizes novel image-conditioned pairwise terms before demonstrating successful and failure cases of the proposed approach.

A.1 Image-Conditioned Pairwise

The pairwise analysis fixes other body-part locations and predicts a target body part across the image. Individual predictions are fuzzy, while combined scoremaps show their interaction in the fully-connected model.

  • Image-Conditioned Pairwise: For a selected person, the analysis fixes every other body part at its ground-truth location and predicts target part c using a learned regressor.This controls for possible misdetections when examining the pairwise terms.
  • Image-Conditioned Pairwise: The method computes pairwise probability p(cc′) for every possible target location, visualizing right-knee probabilities conditioned on other body parts.
  • Image-Conditioned Pairwise: Individual pairwise scoremaps form fuzzy cones extending toward the correct target location.
  • Image-Conditioned Pairwise: Combined pairwise scoremaps aggregate predictions from all other body parts for a randomly selected individual and overlay them for visualization.Rows 1, 3, and 5 show combined pairwise scoremaps, while rows 2, 4, and 6 show unary scoremaps.

A.2 Examples of Successful and Failure Cases

DeeperCut succeeds under challenging visibility, articulation, and occlusion conditions, but still fails when nearby people create ambiguous groupings or spurious configurations.

  • Successful cases: DeeperCut correctly estimates only visible body parts when portions of a person are hidden.The model outputs only the visible parts for a partially visible person.
  • Successful cases: The model assembles body parts correctly for rare articulations and strong partial occlusions.
  • Failure cases: Close proximity can cause DeeperCut to merge body parts from several people into one configuration.The pairwise terms may fail to disambiguate people, even when the resulting limbs are geometrically consistent.
  • Failure cases: DeeperCut also produces spurious body configurations that cannot be assigned to ground-truth annotations.

people

The supplied material identifies a failure-case figure and lists related human-pose-estimation references, but provides no substantive description of the section’s content.

  • Figure 7 presents failure cases of the DeeperCut model on the MPII Multi-Person dataset.
  • The supplied numbered entries include prior work on human pose estimation and image-dependent pairwise relations.
  • The supplied bibliography continues with references on deep neural networks and human pose estimation.
Loading 1605.03170v3…