Source-linked AI summary
Joint Multi-Person Pose Estimation and Semantic Part Segmentation
Fangting Xia, Peng Wang, Xianjie Chen, Alan Yuille
TL;DR
The paper addresses the problem that pose estimation and semantic part segmentation are usually solved independently despite their complementarity in natural multi-person images. It jointly refines both tasks by combining FCN potentials, FCRF-based pose inference, and pose-guided segmentation, and reports superior performance for both tasks with substantially faster inference.
Problem
Independent task models can mislocalize joints or produce erroneous part segments because they lack complementary part coherence and human-shape context.
Method
The method combines Pose and Part FCNs with an FCRF that refines joints using part consistency, then feeds refined pose features into a second-stage Part FCN.
Results
The approach outperforms state-of-the-art methods on both tasks and reduces CRF inference time by a factor of 40.
Takeaways & Limitations
Joint pose and segmentation modeling improves joint localization and produces clearer part details, particularly for arms and legs.
Abstract
from arXiv · showhide
Human pose estimation and semantic part segmentation are two complementary tasks in computer vision. In this paper, we propose to solve the two tasks jointly for natural multi-person images, in which the estimated pose provides object-level shape prior to regularize part segments while the part-level segments constrain the variation of pose locations. Specifically, we first train two fully convolutional neural networks (FCNs), namely Pose FCN and Part FCN, to provide initial estimation of pose joint potential and semantic part potential. Then, to refine pose joint location, the two types of potentials are fused with a fully-connected conditional random field (FCRF), where a novel segment-joint smoothness term is used to encourage semantic and spatial consistency between parts and joints. To refine part segments, the refined pose and the original part potential are integrated through a Part FCN, where the skeleton feature from pose serves as additional regularization cues for part segments. Finally, to reduce the complexity of the FCRF, we induce human detection boxes and infer the graph inside each box, making the inference forty times faster. Since there's no dataset that contains both part segments and pose labels, we extend the PASCAL VOC part dataset with human pose joints and perform extensive experiments to compare our method against several most recent strategies. We show that on this dataset our algorithm surpasses competing methods by a large margin in both tasks.
1. Introduction
Pose estimation and semantic part segmentation are correlated but commonly solved independently, causing errors when appearance or spatial context is ambiguous. The paper jointly models them so part segments refine joints and pose provides shape regularization for segmentation.
- Motivation: Independent pose models can place joints outside human instances or confuse nearby people, while segmentation models can miss or mislabel parts when appearance cues are weak.These errors reflect the absence of dense part coherence in pose estimation and overall human-shape regularization in segmentation.
- Motivation: Joint inference uses semantic part segments to correct ambiguous joint locations and pose skeletons to align segments with human instances, especially for arms and legs.The two tasks provide complementary spatial and shape information.
- Approach: The refined pose is used to refine semantic part segmentation, completing an iterative exchange of information between the two tasks.The paper’s framework first improves pose using part information and then feeds pose information back into segmentation.
- Approach: The framework trains Pose FCN and Part FCN, then fuses joint and part potentials with an FCRF to refine pose estimation.The initial networks produce joint score, neighboring-joint score, and semantic-part score maps before fusion.
- Evaluation: The authors augment PASCAL-Person-Part with 14 human pose joints across 3533 images and report improvements over recent methods for both tasks.The dataset includes substantial variation in poses, scales, and occlusion.
2. Related Works
Prior methods use graphical models, FCNs, or manually designed structures, but joint-task approaches struggle with large pose variation and overlapping people. This paper combines deep FCN potentials with graphical models and task-specific consistency terms.
- Human pose estimation: Recent pose methods use deep joint detectors with graphical models, yet joint localization can remain inaccurate and sometimes fall outside the human body.The related work contrasts these methods with the paper’s use of richer task interaction.
- Semantic part segmentation: FCN-based part segmentation is fast but can produce coarse details and local confusion under unusual poses or similar nearby appearances.The cited examples include arms mislabeled as legs and background regions labeled as arms.
- Proposed framework: The framework fuses initial joint and part scores for pose refinement, then uses estimated poses to refine part segmentation.This ordering is summarized in the framework description.
- Joint methods: Earlier joint pose-and-segmentation methods demonstrate complementarity mainly on simpler datasets and struggle with large pose variation or multi-person overlap.The text attributes this limitation to weaker features or poor part-region proposals.
- Joint methods: The proposed model combines FCNs with graphical models to increase representation power for large pose variation.It also introduces part-segment consistency for pose estimation and pose consistency for part segmentation.
3. Our Approach
The framework refines multi-person pose and semantic part segmentation in detection boxes by combining FCN outputs with an FCRF and pose-conditioned part refinement. Pose-derived joint and skeleton features regularize segmentation, while segment-based cues improve pose assembly and localization.
- Initial estimation: The pipeline detects and resizes each human box, then obtains initial pose and part estimates from separate Pose FCN and Part FCN networks.Pose FCN uses ResNet-101, while Part FCN uses DeepLab-LargeFOV.
- Initial estimation: Pose FCN outputs 14-channel joint potentials and 364-channel neighbor-location potentials, while Part FCN outputs seven semantic classes.The 364 dimensions encode 14×13×2 offsets for neighboring joint locations.
- Pose refinement: The FCRF assembles joint proposals into people by assigning joint types and same-person links over a fully connected proposal graph.Six candidate locations are generated per joint from the joint score map, and background is represented as label 0.
- Pose refinement: Unary terms use joint score-map likelihoods, whereas pairwise terms combine neighbor-location and segmentation features through logistic regression.The combined pairwise features encode proposal compatibility and whether candidate joints belong to the same person.
- Pose refinement: Segment-joint features encode whether joints lie within part regions or near boundaries, whether connecting bones remain inside parts, and rectangle-region overlap.These cues are extracted for neighboring joints, with reduced features for joints that are far apart.
- Part refinement: The second-stage Part FCN concatenates the original part scores with joint and skeleton maps derived from refined pose configurations to produce final segmentation.Joint maps draw radius-3 circles, skeleton maps draw width-7 sticks, and the final labels use output-map argmax.
- Training and inference: The method is trained stage-wise and performs one refinement pass because iterative learning and inference were marginally beneficial and practically inefficient.Inference takes roughly 6 seconds per image, while training takes roughly 3 days.
4. Experiments
Experiments evaluate joint pose estimation and semantic part segmentation on the manually annotated PASCAL-Person-Part dataset, using quantitative metrics and qualitative comparisons across methods and instance scales.
- Dataset and metrics: The augmented dataset contains 14 joint types and six semantic part types, with annotations spanning substantial pose and scale variation.The experiments use manually labeled PASCAL-Person-Part images with joint and part-segment annotations.
- Dataset and metrics: Pose estimation is evaluated with mAP, while ADK measures relative localization distance and penalizes imprecise joint placement more directly.The authors motivate ADK because PCK, PCP, and mAP treat estimates within a distance threshold as correct.
- Human pose estimation: The proposed model reduces keypoint distance significantly for wrists and lower-body joints through consistency with semantic part segmentation.This result is reported for the ADK evaluation in Table 2.
- Human semantic part segmentation: For semantic segmentation, the model surpasses previous methods in mIOU, and ResNet-101 raises its performance to 64.39%.The comparison includes Attention, HAZN, and a first-stage model without pose information; the authors report that pose information improves segmentation.
- Human semantic part segmentation: For small-scale people, the model surpasses other state-of-the-art methods by over 5% in part-segmentation mIOU.Performance is also evaluated across different human-instance sizes.
- Qualitative evaluation: Qualitative comparisons show more accurate pose predictions and overall part configurations, including clearer arm and leg details, especially for small-scale people.Pose results are compared with DeeperCut and a version without segment consistency; segmentation is compared with Attention and HAZN.
5. Conclusion
The paper jointly and iteratively refines pose estimation and semantic part segmentation, using each task to improve the other in complex multi-person images. An auto-zoom strategy also reduces CRF inference time by a factor of 40, while experiments show gains over state-of-the-art methods on both tasks.
- The framework jointly performs human instance clustering and joint labeling with a fully-connected CRF using deep-learned and part-consistency features.
- The approach improves joint localization, especially for arms and legs, and produces clearer semantic-part details.
- 40× faster CRF inference is achieved through an auto-zoom strategy that handles object-scale variation for both tasks.
- Experiments on PASCAL-Person-Part show that the method outperforms state-of-the-art methods for both pose estimation and semantic segmentation.