Source-linked AI summary
Human Pose Estimation using Deep Consensus Voting
Ita Lifshitz, Ethan Fetaya, Shimon Ullman
TL;DR
The paper targets single-image human pose estimation, where existing methods rely on sparse keypoint evidence and image-independent pose relations. It uses dense convolutional keypoint voting and consensus-derived image-dependent joint probabilities, achieving competitive results on MPII and Leeds sports pose datasets.
Problem
Single-image pose estimation remains challenging, while prior methods limit evidence to keypoint locations and use image-independent binary relations.
Method
A convolutional network predicts every keypoint’s relative location from each image patch, then combines dense votes and consensus-derived joint probabilities to estimate the pose.
Results
The method achieves competitive results on the MPII human pose and Leeds sports pose datasets.
Takeaways & Limitations
Dense multi-target voting uses information from the whole person and supports expressive image-dependent relationships between keypoints.
Takeaways & Limitations
The voting network is trained with separate per-keypoint losses under a naive independence assumption.
Abstract
from arXiv · showhide
In this paper we consider the problem of human pose estimation from a single still image. We propose a novel approach where each location in the image votes for the position of each keypoint using a convolutional neural net. The voting scheme allows us to utilize information from the whole image, rather than rely on a sparse set of keypoint locations. Using dense, multi-target votes, not only produces good keypoint predictions, but also enables us to compute image-dependent joint keypoint probabilities by looking at consensus voting. This differs from most previous methods where joint probabilities are learned from relative keypoint locations and are independent of the image. We finally combine the keypoints votes and joint probabilities in order to identify the optimal pose configuration. We show our competitive performance on the MPII Human Pose and Leeds Sports Pose datasets.
1 Introduction
Human pose estimation from a single image remains challenging despite deep-learning progress. The paper addresses limitations of sparse keypoint detection and image-independent pose consistency by using dense votes and consensus-based image-dependent relationships.
- Single-image human pose estimation remains challenging, with state-of-the-art results still below human performance.
- Previous methods classify local patches as keypoints, produce heat maps, and combine them with graphical-model structure prediction.
- Sparse keypoint detection limits the use of information from the entire person, especially when keypoints are occluded.
- Graphical-model binary terms typically use training-set relative keypoint statistics and remain independent of the input image.
- The proposed approach predicts all keypoint locations relative to every patch center, aggregates votes, and derives image-dependent binary terms from vote agreements.
- The method produces competitive results on the MPII human-pose and Leeds sports pose datasets.
2 Related Work
Related work combines keypoint detection with structural models to handle body-part relationships, while convolutional networks and iterative schemes improve detection and pose refinement.
- Most human-pose methods combine keypoint detectors with pictorial structures to capture relations between body parts.
- Poselet-based features incorporate higher-order part dependencies, while chains-models replace pictorial structures with voting chains from head to hand.
- Convolutional neural networks improved part-detector reliability, with prior work using multi-scale representations and simultaneous multi-person detection.
- Recent methods also use iterative schemes to refine pose estimates, and some learn image-dependent binary terms explicitly.
3 Overview of the Method
The method aggregates dense keypoint votes from image patches, converts their agreements into image-dependent joint terms, and sequentially optimizes a pose using unary and binary evidence.
- At inference, a neural network predicts each keypoint’s location relative to every image patch, producing patch-level vote distributions.
- Aggregating patch votes yields a probability distribution over possible locations for each keypoint.
- The voting network uses separate keypoint losses, initially treating keypoints as conditionally independent at each patch.
- Averaging products of keypoint vote probabilities creates joint distributions in which keypoints are no longer independent.
- The final pose is estimated by minimizing an energy over voting-derived unary and binary terms, proceeding sequentially from reliable keypoints to the full pose.
4 Keypoint Voting
The method predicts every keypoint’s relative location from each image patch, discretizes these predictions into log-polar bins, and aggregates the resulting votes in image space. It uses dense keypoint coverage and a fixed deconvolution layer to produce probability distributions over keypoint locations.
- Voting representation: A fully convolutional deep neural net predicts the relative location of every keypoint from each image patch center.This differs from methods that classify whether the patch center itself is a keypoint.
- Voting representation: Relative locations are classified into 50 log-polar bins, providing finer resolution near the patch center and coarser estimates farther away.The classes include a central location, a background class, and four rings divided into 12 angular bins each.
- Voting representation: The model uses 30 keypoints: 16 annotated joints, 12 synthetic midsection points, and estimated hand locations.Synthetic points are generated between neighboring skeleton joints, while hands are estimated by extrapolating the elbow-wrist vector by 30%.
- Net architecture: The VGG-16-based network produces a 50-class probability distribution for each keypoint, with deconvolution increasing the output resolution to every 4 pixels.The architecture uses pretrained convolutional layers, convolution with holes, and a learned upsampling deconvolution layer.
- The voting scheme: A fixed deconvolution kernel maps each log-polar-bin channel back to image locations, after which votes from all patch centers are aggregated into final keypoint distributions.The kernel has size 65 × 65 × 50, and the aggregated distribution can generate votes outside the visible image.
5 Consensus Voting
Consensus voting converts dense per-location keypoint votes into image-dependent joint probabilities, helping distinguish plausible keypoint combinations in cluttered or ambiguous images. These probabilities are combined with unary and conventional binary terms for pose inference.
- Consensus voting: Unlike image-independent relative-location priors, consensus voting favors keypoint pairs supported strongly by many common voters.Separate per-keypoint predictions become dependent after averaging joint votes over patch centers.
- Consensus voting: Figure 4 conditions the left-elbow distribution on a selected left-shoulder location to suppress a nearby misleading response and retain the correct elbow.The conditional response becomes strong only at the correct location for the selected person.
- Pose objective: The pose objective combines unary scores from individual keypoint probabilities with a weighted binary term containing consensus joint probabilities and relative-location statistics.The unary term is −log(P(Ki = xi)); the binary term blends the learned joint distribution with the conventional prior.
- Efficiency: Naively computing consensus probabilities costs O(N^6), so the implementation exploits their convolution structure and optimized GPU computation.The expensive computation sums over all voters and all pairs of possible keypoint locations.
6 Pose Prediction
Pose prediction adds geometric constraints and staged inference to improve optimization over augmented keypoints with unequal detection reliability. Midpoint constraints are folded into pairwise terms, while inference proceeds from reliable to less certain parts.
- Pose Prediction: The method adds geometric constraints on synthetic keypoints and performs inference in stages from reliable parts to less certain ones.These are additional performance-improvement steps beyond the basic unary and binary terms.
- Local Geometric Constraints: A synthetic midpoint keypoint is eliminated from the optimization by substituting its midpoint location into the associated unary and binary terms.The resulting pairwise term incorporates the geometric relationship without retaining the synthetic variable.
- Local Geometric Constraints: The substituted formulation is equivalent to enforcing the midpoint constraint while optimizing faster.The resulting linking feature also uses appearance between the two original keypoints.
- Pose Prediction: Staged prediction lets highly reliable keypoints influence uncertain ones without allowing occluded or unclear keypoints to distort more visible parts.The motivation is unequal detection accuracy across body parts, such as the head versus the wrist.
7 Results
The method was evaluated on MPII and LSP under their standard pose-estimation metrics, achieving competitive benchmark performance and strong MPII head-keypoint results.
- MPII: The MPII evaluation uses 19,185 training images, 7,247 test images, and returns 16 annotated keypoints per person.Testing focuses on the single-person subset, with a 504 × 504 crop around the target person.
- MPII: MPII performance is measured by PCKh, which counts a keypoint as correct within half the head segment length.
- MPII: 85.0% mean PCKh was achieved on MPII, with state-of-the-art performance on head keypoints.
- Leeds Sports Pose: The LSP evaluation contains 2,000 sports images and uses the person-centric setting with the strict PCP limb-detection metric.LSP contains 1,000 training and 1,000 testing images, with 14 returned keypoints.
- Leeds Sports Pose: For LSP, the MPII-trained model was fine-tuned on 1,000 training images and evaluated in both original and flipped orientations.The two orientations address the greater prevalence of upside-down people in LSP.
- Leeds Sports Pose: On LSP, performance was comparable to Pishchulin et al. and superior to other compared methods, despite not using the 10,000 additional poses in LSP-Extended.
8 Discussion
The discussion presents dense keypoint voting as a way to aggregate evidence across the person and derive image-dependent joint probabilities, with competitive results across MPII and LSP.
- The proposed method replaces keypoint detection with dense voting, allowing all evaluated image regions to contribute to keypoint predictions.
- Consensus voting supports image-dependent joint keypoint probabilities by aggregating agreements between votes for multiple keypoints.
- Empirical results on MPII and LSP were competitive, improving the state of the art on a subset of evaluated keypoints.
- Future work targets occlusions from closely interacting people and iterative refinement using previous predictions as network input.