Source-linked AI summary
Learning Human Pose Estimation Features with Convolutional Networks
Arjun Jain, Jonathan Tompson, Mykhaylo Andriluka, Graham W. Taylor, Christoph Bregler
TL;DR
Unconstrained human pose estimation from monocular RGB images is difficult because body parts can be tiny, occluded, and varied, while precision and nonrigid structure challenge end-to-end learning. The paper develops a two-stage convolutional approach with part detectors and a weak spatial model, and reports improved state-of-the-art performance over traditional architectures.
Problem
Unconstrained monocular human pose estimation remains difficult because body parts may occupy few pixels amid occlusion, clutter, nonrigid structure, and varied appearance.
Method
The paper trains separate convolutional body-part classifiers as sliding-window detectors, then denoises their response maps using a second process informed by part hierarchy.
Results
The architecture outperforms or matches MODEC and DPM across wrist, elbow, and shoulder detection, with spatial modeling improving wrist and elbow localization by approximately 5% at a 5-pixel threshold.
Takeaways & Limitations
Convnets can provide strong low-level pose features, while a generic higher-level spatial model achieves the best results and contributes less than expected relative to those features.
Takeaways & Limitations
The authors identify enlarging the training set and adding multi-resolution input representations as avenues for further improvement.
Abstract
from arXiv · showhide
This paper introduces a new architecture for human pose estimation using a multi- layer convolutional network architecture and a modified learning technique that learns low-level features and higher-level weak spatial models. Unconstrained human pose estimation is one of the hardest problems in computer vision, and our new architecture and learning schema shows significant improvement over the current state-of-the-art results. The main contribution of this paper is showing, for the first time, that a specific variation of deep learning is able to outperform all existing traditional architectures on this task. The paper also discusses several lessons learned while researching alternatives, most notably, that it is possible to learn strong low-level feature detectors on features that might even just cover a few pixels in the image. Higher-level spatial models improve somewhat the overall result, but to a much lesser extent then expected. Many researchers previously argued that the kinematic structure and top-down information is crucial for this domain, but with our purely bottom up, and weak spatial model, we could improve other more complicated architectures that currently produce the best results. This mirrors what many other researchers, like those in the speech recognition, object recognition, and other domains have experienced.
1 Introduction
Unconstrained monocular human pose estimation requires locating articulated body parts despite occlusion, clutter, and substantial visual variation. The paper addresses this challenge with an end-to-end convolutional approach that combines learned features with a weak spatial model.
- Monocular pose estimation is difficult because human bodies have many degrees of freedom, self-occlusion, self-similar parts, and variation in clothing, body type, and lighting.
- The hardest setting avoids prior motion, pose, and background models while estimating full-body configuration from a monocular RGB image.
- Hands, elbows, shoulders, hips, knees, and feet may occupy only a few pixels against arbitrary cluttered backgrounds.
- End-to-end learning is challenging because pose is nonrigid and multi-modal, while pooling can discard the precise location information required for accurate estimation.
- The paper presents an end-to-end convolutional architecture whose part-detector response maps are denoised by a second process informed by part hierarchy.
2 Related Work
Earlier pose-estimation work used handcrafted features, structural models, specialized detectors, or restricted end-to-end systems. The paper situates its approach within this progression and highlights the growing availability of labeled data for full-pose learning.
- Earlier unconstrained pose methods used bag-of-features representations with regression, nearest-neighbor, or SVM-based architectures.
- Many approaches combine local body-part detectors with structural reasoning through pictorial structures, poselets, and other part models.
- HumanEva and higher-resolution shape-model approaches differ from this setting because they use higher-quality, less cluttered images or video sequences.
- Kinect-based body-part detection uses random forests trained on synthetic depth data, whereas the proposed work uses a different learning strategy with simple part-based detectors.
- Earlier end-to-end methods addressed limited subsets such as a single hand or face pose parameters rather than full-body pose estimation.
- Limited labeled data constrained deep full-pose learning, motivating the use of larger datasets such as FLIC.
3 Model
The model replaces direct pose regression with independent convolutional body-part detectors and a weak spatial model that filters their responses. It combines learned local evidence with non-parametric pose priors while limiting pooling to preserve spatial precision.
- Convolutional Network Architecture: Directly mapping image input to full-body pose coefficients worked poorly because pooling loses spatial precision and valid poses occupy a low-dimensional manifold.The body’s nonrigid structure, required localization precision, and multimodal poses further complicate end-to-end learning.
- Convolutional Network Architecture: Independent convnets classify each body part in sliding windows, producing confidence response-maps while retaining the benefits of smaller networks and pooling.Each network maps a pixel window to a binary presence-or-absence output, with separate parameters for each body part.
- Enforcing Global Pose Consistency with a Spatial Model: Independent detectors cannot enforce full-body pose consistency, motivating the higher-level spatial model despite its weak structural constraints.At test time, the convnet and spatial model are run across multiple image scales because their learned features and priors are not explicitly scale invariant.
- Convolutional Network Architecture: The convolutional pipeline uses local connectivity, weight sharing, pooling, and fully connected layers to produce a logistic body-part-presence probability.It processes 64×64 RGB patches through local contrast normalization, convolution and subsampling stages, then fully connected layers.
- Convolutional Network Architecture: Only two 2 × 2 pooling stages are used because pooling reduces computation and translation sensitivity but sacrifices spatial precision.The offline application and sufficient training examples make learned translation invariance preferable to more pooling.
- Enforcing Global Pose Consistency with a Spatial Model: A kinematic chain of neighboring joints combines convnet unary distributions with conditional spatial priors using belief-propagation-like filtering.The model uses non-parametric histograms for adjacent-part relationships and a global face-position prior; λ controls unary confidence and was set to 1.
4 Results
The architecture was evaluated on the challenging FLIC dataset using joint localization accuracy and comparisons with DPM and MODEC detectors. It matched or exceeded those detectors across the three evaluated body parts, while the spatial model mainly helped wrist and elbow outliers.
- Dataset and evaluation: The evaluation used 5003 still RGB movie images with labeled upper-body joints, including 3987 training images and a 351-image single-person test subset.Training images were horizontally mirrored, yielding 7974 examples.
- Dataset and evaluation: Accuracy was measured as the percentage of joints within a specified 2D pixel-radius threshold, using the same 351 test images for all detectors.The model was compared with DPM and MODEC architectures.
- Detector comparison: The architecture outperformed or equaled MODEC and DPM for the wrist, elbow, and shoulder joints.This comparison is reported across all three evaluated body parts.
- Spatial-model impact: Approximately 5% of test cases improved at a 5-pixel threshold for wrist and elbow localization after applying the spatial model.The improvement enabled the method to outperform the other detectors for those joints.
- Spatial-model impact: The spatial model removed strong false positives and outliers for wrist and elbow localization, but decreased shoulder accuracy for large thresholds.The reported shoulder degradation was attributed to poor elbow-convnet performance.
5 Conclusion
The conclusion reports improved unconstrained human pose estimation from convolutional features combined with a weak spatial model. Among the tested structural alternatives, the most generic higher-level spatial model performed best, while larger training sets and broader context remained directions for improvement.
- Conclusion: Convolutional networks combined with a global position prior outperformed more complex and popular models for unconstrained human pose estimation.The conclusion characterizes convolutional networks as effective low-level feature detectors.
- Conclusion: Figure 7 contrasts MODEC with predictions before and after applying the spatial model, using red, blue, and green crosses respectively.The figure also labels the input RGB image and the distributions before and after spatial modeling.
- Conclusion: The most generic higher-level spatial model achieved the best results among the structural models explored.The authors describe this finding as counter-intuitive relative to common beliefs about human kinematic structure.
- Future work: The authors expected further improvement from enlarging the training set with a pose-based warping technique and from multi-resolution inputs with larger spatial context.Both techniques were described as ongoing investigations.