Source-linked AI summary
Combining Local Appearance and Holistic View: Dual-Source Deep Neural Networks for Human Pose Estimation
Xiaochuan Fan, Kang Zheng, Yuewei Lin, Song Wang
TL;DR
Single-image human pose estimation must handle substantial variation in pose, appearance, viewpoint, and occlusion. The paper introduces DS-CNN, which combines local part patches with full-body context for joint detection and localization, and reports superior performance to several existing methods on FLIC and LSP. The method performs better than Tompson et al. at large normalized distances but worse at small normalized distances.
Problem
Single-image 2D human pose estimation is difficult because pose, appearance, viewpoint, and occlusion vary, while existing part-based and holistic approaches provide limited modeling alternatives.
Method
DS-CNN jointly learns from local part patches and full-body patches to combine local appearance and holistic context for joint detection and localization.
Results
The proposed method produces superior performance to several existing methods on FLIC and LSP, with over 15% average PCP improvement over comparison methods on LSP.
Takeaways & Limitations
Combining local appearance with holistic view yields more reliable pose estimation within the evaluated FLIC and LSP settings.
Takeaways & Limitations
The method focuses on CNN-based local-part detection without a high-level graphical model, and its PDJ performance is worse than Tompson et al. at small normalized distances.
Abstract
from arXiv · showhide
We propose a new learning-based method for estimating 2D human pose from a single image, using Dual-Source Deep Convolutional Neural Networks (DS-CNN). Recently, many methods have been developed to estimate human pose by using pose priors that are estimated from physiologically inspired graphical models or learned from a holistic perspective. In this paper, we propose to integrate both the local (body) part appearance and the holistic view of each local part for more accurate human pose estimation. Specifically, the proposed DS-CNN takes a set of image patches (category-independent object proposals for training and multi-scale sliding windows for testing) as the input and then learns the appearance of each local part by considering their holistic views in the full body. Using DS-CNN, we achieve both joint detection, which determines whether an image patch contains a body joint, and joint localization, which finds the exact location of the joint in the image patch. Finally, we develop an algorithm to combine these joint detection/localization results from all the image patches for estimating the human pose. The experimental results show the effectiveness of the proposed method by comparing to the state-of-the-art human-pose estimation methods based on pose priors that are estimated from physiologically inspired graphical models or learned from a holistic perspective.
1. Introduction
Human pose estimation from a single 2D image is important but difficult because pose, appearance, viewpoint, and occlusion vary widely. The paper proposes DS-CNN to combine local part appearance with holistic body context and jointly detect and localize joints.
- Single-image 2D pose estimation supports surveillance, action recognition, and human-computer interaction but remains challenging under varied poses, appearances, viewpoints, and occlusions.
- Earlier part-based models combine local appearance with contextual relations, but graphical models can be noise-sensitive and insufficiently expressive for complex poses.
- DeepPose illustrates holistic CNN-based estimation, using an initial whole-body pose estimate followed by local joint refinement.
- DS-CNN integrates local image-patch appearance with each part’s holistic view in the full body, using object proposals as semantically meaningful multi-scale inputs.
- The unified DS-CNN output performs joint detection and joint localization, whose results are combined to estimate the human pose.
- Experiments on FLIC and LSP compare DS-CNN with several reported methods, including DeepPose, and show effectiveness of combining local appearance with holistic view.
2. Related Work
Related work progresses from part-based graphical models to CNNs that learn pose representations directly. DS-CNN extends CNN-based part localization by combining object proposals with full-body input to capture relations between local parts.
- Part-based pose models represent the body as physiologically inspired parts connected through deformable configurations.
- Pictorial structures model pairwise geometric relations between adjacent body parts with a tree, while later methods capture broader contextual relations.
- CNNs learn multiple representation levels and complex nonlinear relations, supporting applications including image classification, localization, and detection.
- R-CNN object proposals improve localization of smaller parts compared with sliding windows, but the cited method does not model complex relations between parts.
- Unlike DeepPose and other CNN approaches, DS-CNN uses both object proposals and full-body input during training to capture local parts with broader context.
3. Problem Description and Notations
The paper represents a pose as 2D coordinates for all joints and normalizes joint locations relative to image patches. Joint visibility records whether each joint lies inside a patch.
- A human pose is represented as J = {j_i} from i=1 to L, where each joint j_i has 2D coordinates (x_i, y_i) and L is the number of joints.
- The task is to estimate all 2D joint locations from a single image, using rectangular image patches for detection and regression.
- Each patch is described by width, height, and center, enabling joint coordinates to be converted from absolute image positions to patch-relative normalized locations.
- Joint visibility is binary: v_i(p)=1 when joint i lies inside patch p and v_i(p)=0 when it lies outside.
4. Model Inputs
DS-CNN uses paired local part patches and full-body patches to combine appearance with context. Training selects object proposals with controlled size and balanced joint coverage, while testing uses multi-scale sliding windows.
- Model inputs: DS-CNN has two inputs: part patches containing local image regions and body patches providing the full-body context for those regions.
- Model inputs: The paired inputs help distinguish visually similar joints, such as left and right wrists or wrists and legs, that local appearance alone may confuse.
- Patch normalization: Patches preserve aspect ratio by extending the shorter side before uniform scaling, adding context while causing only a minor resolution reduction.
- Part-patch selection: Training selects category-independent object proposals within a size range defined by µ1 and µ2 relative to torso-joint distance.
- Coverage: The selection algorithm provides relatively balanced joint coverage on both FLIC and LSP across the evaluated joints and µ2 settings.
- Testing: Testing uses multi-scale sliding windows rather than object proposals because uneven proposal density can distort the resulting heatmap.
- Input representation: Training samples combine a part patch, a body patch, and a binary mask locating the part patch within the body patch.
5. Multi-Task Learning
DS-CNN combines joint detection and joint localization in one model, assigning each task its own loss before balancing them in a combined objective.
- Multi-Task Learning: DS-CNN jointly determines whether a part patch contains a body joint and localizes that joint within the patch.Each task is associated with a loss function.
- Joint detection: The detection output contains one likelihood for no visible joint and one likelihood for each joint being visible and closest to the patch center.A softmax classifier is used for joint detection.
- Joint localization: Joint localization is formulated as regression, with the network predicting coordinates for all L joints in the patch.The predicted location of joint i is represented by z_i(pp,b) = (b_xi, b_yi)^T.
- Combined objective: A combined DS-CNN loss balances the detection and localization losses with a positive factor λ_d.The summation is taken over all training patch pairs.
6. DS-CNN Structure
The DS-CNN structure uses separate convolutional streams for local part appearance and holistic body information before combining their learned representations.
- DS-CNN Structure: One five-layer convolutional sequence processes the part patch to extract local-appearance features.The part-patch stream is one of two separate convolutional sequences.
- DS-CNN Structure: A second five-layer convolutional sequence processes the body patch to extract holistic features.The two streams provide the local and full-body inputs to DS-CNN.
- CNN architecture: The underlying CNN comprises five convolutional layers, three fully connected layers, and a final 1000-way softmax, with max pooling after convolutional layers 1, 2, and 5.In the proposed architecture, the convolutional processing is duplicated into separate part and body streams.
- Architecture notation: In the architecture diagram, convolutional layers followed by pooling are labeled C_i, while fully connected layers are labeled F_i.The index i identifies the layer position.
7. Human Pose Estimation
At test time, DS-CNN combines detection and localization outputs from multi-scale patch pairs. Heatmaps provide coarse joint evidence, while selected localization outputs refine each joint position.
- Testing pipeline: Testing uses multi-scale sliding-window patch pairs, and DS-CNN produces joint detection and localization outputs for each pair.An algorithm combines these outputs to estimate the final pose.
- Heatmap construction: For each joint, a heatmap matches the original image size and assigns each pixel a value reflecting the likelihood that the joint is located there.Patch-level detection likelihoods are distributed across pixels and then summed over patch pairs.
- Heatmap construction: The left-wrist heatmap resolves limb ambiguity when body patches are incorporated but remains insufficient for accurate joint localization alone.The heatmap therefore provides a rough rather than precise joint position.
- Joint refinement: Refinement selects patch pairs where the target joint is more likely than no joint and ranks among the k most likely joints.For k=1, the target joint must have the largest joint likelihood.
- Evaluation: DS-CNN, DeepPose, and Johnson et al. are trained with LSP and its extension, whereas the other listed methods use only LSP.This dataset-training distinction accompanies the PCP comparison in Table 1.
- Joint refinement: The final joint location is computed as a weighted average of selected localization outputs represented in body-patch coordinates.In the experiments, the whole testing image serves as the body patch, and coordinates are transformed from the localization output.
8. Experiments
Experiments on LSP and FLIC evaluate DS-CNN with PCP, PDJ, and joint-detection AP, showing strong performance while exposing threshold-dependent differences from Tompson et al. The experiments also illustrate learned local–holistic features and improved handling of occluded joints.
- Datasets and metrics: The evaluation uses LSP, extended LSP, and FLIC datasets containing challenging articulated human poses and annotated body joints.LSP contains 11,000 training and 1,000 testing images with 14 full-body joints; FLIC contains 3,987 training and 1,016 testing images with 10 upper-body joints.
- Datasets and metrics: PCP and PDJ measure limb and joint detection, respectively, with PDJ addressing PCP’s penalty for short limbs.PCP counts limbs whose detected joints are within half the limb length, while PDJ measures joint detection using normalized distance.
- Quantitative comparisons: Over 15% average PCP improvement on LSP versus the best comparison results accompanies better performance than comparison methods at nearly every body part except the head.The method outperforms DeepPose and six other methods for the evaluated LSP body parts, except head.
- Quantitative comparisons: On FLIC and LSP PDJ evaluations, DS-CNN generally outperforms comparison methods except Tompson et al., with the advantage depending on normalized-distance threshold.Tompson et al. performs better below the relevant threshold, whereas DS-CNN performs better above it; on FLIC, thresholds are 0.15 for elbows and 0.18 for wrists, while LSP shows a more significant large-distance gain.
- Limitations: The authors identify graphical-model integration as a route for further improving DS-CNN’s PDJ performance.Tompson et al.’s MRF-based graphical model substantially improves CNN-based part detection, whereas DS-CNN does not use a high-level graphical model.
- Ablation and analysis: Dual-source patch pairs achieve the best joint-detection AP and mAP across all joints on both LSP and FLIC.Body patches alone outperform part patches partly because they include a low-resolution binary mask, but combining part and body patches remains necessary.
- Ablation and analysis: F7 activations respond to pose patterns such as a bent right elbow and right hip, while holistic views support estimation when joints are occluded.LSP’s complex full-body poses are less accurate than FLIC’s upper-body poses in the shown examples.
9. Conclusion
The paper concludes that DS-CNN combines part and body patches to integrate local and contextual information, jointly supporting joint detection and localization. Tests on FLIC and LSP report superior performance against several existing methods, with threshold-dependent results relative to Tompson et al.
- Conclusion: DS-CNN combines part patches and body patches to use local and contextual information for human pose estimation.The method is described as producing more reliable pose estimates from these two patch types.
- Conclusion: DS-CNN combines joint detection and joint localization outputs to estimate human pose.The conclusion identifies both outputs as components of the pose-estimation procedure.
- Conclusion: Testing on FLIC and LSP shows superior performance against several existing methods, while comparison with Tompson et al. varies with normalized distance.DS-CNN performs better at large normalized distances and worse at small normalized distances.
- Conclusion: The implementation uses the open-source Caffe CNN library and is described as having good expandability.This implementation detail is presented as part of the paper’s concluding assessment.