Source-linked AI summary
Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network
Sijin Li, Zhi-Qiang Liu, Antoni B. Chan
TL;DR
Monocular 2D human pose estimation is challenging because of appearance ambiguity, self-occlusion, and limited data for training high-capacity models. The paper addresses this with a heterogeneous deep-CNN framework that jointly learns pose regression and sliding-window body-part detection. The authors report better convergence and generalization, competitive results, and middle-layer neurons selective to localized body-part shapes.
Problem
Monocular 2D pose estimation remains difficult because images contain appearance ambiguities and self-occlusion, while deep networks can generalize poorly with limited training data.
Method
The framework jointly trains a pose-joint regression task with sliding-window body-part detection tasks in a deep convolutional neural network.
Results
The detection tasks help the regression network converge to better local minima, generalize on testing data, and learn neurons selective to localized body-part shapes.
Takeaways & Limitations
Heterogeneous auxiliary detection tasks provide meaningful shared features for human pose estimation, including localized body-part shape patterns in higher network layers.
Takeaways & Limitations
The experiments remove images with occluded body parts, although the framework is intended for future extension to occluded poses.
Abstract
from arXiv · showhide
We propose an heterogeneous multi-task learning framework for human pose estimation from monocular image with deep convolutional neural network. In particular, we simultaneously learn a pose-joint regressor and a sliding-window body-part detector in a deep network architecture. We show that including the body-part detection task helps to regularize the network, directing it to converge to a good solution. We report competitive and state-of-art results on several data sets. We also empirically show that the learned neurons in the middle layer of our network are tuned to localized body parts.
1. Introduction
Monocular 2D pose estimation is valuable for common visual media and devices but remains difficult because of appearance ambiguity, self-occlusion, and costly inference or optimization. The paper proposes heterogeneous multi-task learning that combines pose regression with body-part detection to guide deep-network feature learning and convergence.
- 2D pose estimation is useful because most visual media and many mobile devices provide 2D images rather than depth data.
- Appearance ambiguity and self-occlusion make 2D image pose estimation harder than estimation from depth maps.
- Part-based graphical models encode body structure but can require expensive inference, while regression methods rely on pose-informative features and costly optimization.
- Deep CNNs efficiently learn translation-invariant features but are difficult to train to generalize well when limited data meets high model capacity.
- The proposed framework jointly trains pose regression and sliding-window body-part detectors, using the accessory task to guide feature learning and convergence.
2. Related work
The framework differs from prior approaches by sharing feature layers across heterogeneous regression and classification tasks, using sliding-window body-part detection as an accessory task. It also contrasts with multi-stage refinement, pose-sensitive embeddings, semi-supervised representations, and neuron-visualization methods.
- Heterogeneous multi-task learning: The framework shares feature layers between regression and classification tasks rather than enforcing a common sparsity pattern, learning representations useful for both.Prior heterogeneous multi-task learning encouraged matching sparsity patterns; this approach instead shares the learned feature layers.
- Body-part detection: Body-part detection is formulated over sliding windows, with each window performing binary classification while potentially containing multiple body parts.Figure 2 illustrates joint and body-part annotations, indicator maps, and detection windows for left-upper-arm detection.
- Pose representations: Compared with pose-sensitive embedding and nearest-neighbor prediction, the framework uses accessory tasks to learn shared pose features and directly regresses joint locations.The contrasting embedding approach predicts head and hand locations by nearest-neighbor search in learned embedding space.
- Other related approaches: Other related methods include region-focused networks for facial landmarks, semi-supervised shared representations, and visualization of neuron-preferred inputs or high-response image patches.The framework differs from semi-supervised representation learning by using multiple body-part classification tasks sharing one feature space, and from prior visualization by averaging associated patches.
- Pose-estimation architectures: Unlike cascaded pose-estimation systems that add refinement stages, this strategy improves a single regression network through accessory tasks and can also be combined with multi-stage methods.The cited cascaded approach uses multiple stages, whereas this framework introduces accessory tasks without increasing stages.
3. Heterogeneous Multi-task Learning
The framework jointly learns pose-joint regression and sliding-window body-part detection from shared CNN features. Its architecture preserves local spatial information, integrates context, and combines both task losses during training.
- Task formulation: The framework predicts human-joint locations while classifying whether image windows contain specific body parts.A human bounding box is assumed available, and coordinates are normalized relative to it.
- Joint optimization: The global objective linearly combines squared-error joint regression and cross-entropy detection losses across joints, parts, windows, and training images.λr and λd weight the regression and detection terms, respectively.
- Task formulation: Each body-part detector is location-specific, allowing it to learn both part appearance and contextual information from neighboring parts.Separate classifiers are trained for each body part and overlapping window location.
- Body-part detection: Body-part labels are created from stick annotations using a fixed overlap threshold of β = 0.3.A window is positive when the portion of the annotated part inside it reaches the specified fraction of the part’s total length; multiple parts and windows may overlap.
- Network structure: The design shares low-level features, preserves location information, and incorporates whole-person context for local part detection.Shared representations reduce parameters and support generalization, while local connectivity keeps neurons sensitive to input sub-regions.
- Network structure: The shared CNN uses convolution and max-pooling layers before separate fully connected regression and detection subnetworks.The input is a 112×112 RGB image; the shared CNN has three convolutional layers, each followed by max-pooling, and each subnetwork has three fully connected layers.
- Training: Both networks are trained jointly by back-propagation, with summed gradients updating shared layers and dropout used in the first task-specific fully connected layers.The dropout probability is 0.5 in the experiments.
4. Experiments
Experiments evaluate HMLPE on Buffy and FLIC, compare it with prior methods, and examine how detection-task weighting affects multi-task training. Results show competitive pose accuracy and improved convergence and generalization when regression and detection are trained together.
- 4.1. Training data: HMLPE uses 8427 training images assembled from six datasets, with upper-body joints and parts defined for pose regression and detection.The training set combines Buffy, ETHZ, LSP, SA, FLIC, and WAF; images are augmented by a factor of 32.
- 4.1. Training data: Occluded body-part images are excluded from the current experiments, although the framework could be extended to handle occlusion.
- 4.3. Evaluation on Buffy Set: On the whole Buffy test set, HMLPE improves lower-arm PCP by 4.8% over but is 1.1% lower on upper arms than.The evaluation covers 276 images and reports PCP for lower and upper arms.
- 4.3. Evaluation on Buffy Set: With correctly detected upper-body bounding boxes, HMLPE exceeds by 0.7% on lower arms and 0.5% on upper arms.This isolates pose-estimation performance from upper-body detection errors.
- 4.4. Evaluation on FLIC Data set: On FLIC, HMLPE is about 6% higher than MODEC at r = 20 but about 7% and 5% lower on wrists and elbows at r = 6.The results suggest robust general-pose estimation but lower accuracy for exact joint localization under strict criteria.
- 4.5. Effect of multi-task training: Adding even tiny detection-task weights substantially improves regression convergence, while excessively large weights reduce performance because detection dominates the gradient.Training and testing errors are compared after epoch 100, with all parameters except task weights held fixed.
- 4.5. Effect of multi-task training: Shared features induced by detection tasks help regression reach better training minima and improve generalization despite learning 7*8*8 detectors from limited data.
5. Visualization of features
The network’s learned features progress from low-level edge and gradient detectors to mid- and high-level patterns resembling localized body parts. Backtracked activations also reveal responses to contextual background structures.
- First-layer filters resemble edge or gradient detectors at different orientations.
- Backtracked patches identify third-layer feature maps whose maximal activations frequently occur over heads, shoulders, and arms.
- Averaged backtracked patches show mid-level detectors for heads, necks, arms, and shoulders.
- High-level features correspond to localized body parts, including heads in different positions, left and right shoulders, and arms.
- Some high-level features respond to horizontal background structures and may provide contextual information about a door’s position relative to the head.
6. Conclusion
The paper proposes a heterogeneous multitask CNN that jointly learns pose regression and sliding-window body-part detection. Joint training guides meaningful feature learning and improves generalization, while visualization shows mid- and high-level neurons responding to localized body-part shapes; future work targets occlusion, pre-training, video, and other structured objects.
- 6. Conclusion: The framework combines pose regression with body-part detection through sliding-window classifiers.
- 6. Conclusion: Joint training guides the network toward meaningful pose features and improves generalization on testing data.
- 6. Conclusion: Average backtracked patches show mid- and high-level neurons selective to shape patterns resembling localized human body parts.
- 6. Conclusion: Future work will address poses with occlusion and unsupervised pre-training.
- 6. Conclusion: The framework is also intended for future extension to video sequences and other structured objects.