PaperlayerView exact revision on arXiv

Source-linked AI summary

Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmark

Xiaodan Liang, Ke Gong, Xiaohui Shen, Liang Lin

arXiv:1804.01984v1cs.CV

TL;DR

Existing datasets lack sufficient scale, diversity, challenging cases, and unified annotations for human parsing and pose estimation. The paper introduces the LIP benchmark and joint networks that incorporate structural context, achieving state-of-the-art performance across both tasks while also improving parsing without pose annotations.

  • Problem

    Existing human parsing and pose datasets have limited scale and coverage of varied appearances, unconstrained challenges, and unified pixel-wise and joint annotations.

  • Method

    The paper constructs LIP and proposes JPPNet for joint parsing and pose estimation, plus SS-JPPNet for self-supervised structure-sensitive parsing without pose annotations.

  • Results

    JPPNet achieves state-of-the-art performance for both human parsing and pose estimation, while SS-JPPNet significantly surpasses previous methods on PASCAL-Person-Part and LIP.

  • Takeaways & Limitations

    LIP supplies a large, richly annotated benchmark for studying correlated human parsing and pose estimation under diverse appearances, viewpoints, and occlusions.

Abstract

from arXiv · show

Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named "Look into Person (LIP)" that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The dataset, code and models are available at http://www.sysu-hcp.net/lip/.

1 INTRODUCTION

Existing human-centric datasets lack the scale, diversity, challenging unconstrained cases, unified parsing-and-pose annotations, and robust benchmark infrastructure needed for holistic analysis. LIP addresses these gaps with a large annotated benchmark and joint networks that use structural context to improve both tasks.

  • Motivation: CNN-based human analysis requires large-scale datasets covering varied clothing, articulation, occlusion, truncation, viewpoints, and cluttered backgrounds.Existing datasets often target special scenarios and have limited coverage or scalability.
  • Dataset gap: Existing datasets separate human parsing and pose annotations, leaving no unified resource for holistic human understanding.The largest cited parsing dataset has 17,000 fashion images, while MPII has approximately 25K images and over 40K people with joint annotations.
  • LIP benchmark: LIP provides 50,462 images with pixel-wise annotations for 19 semantic parts and 16 body joints, alongside a public evaluation server.The benchmark is designed to increase appearance variability and complexity while reducing potential dataset over-fitting through a secret test set.
  • Proposed networks: JPPNet jointly predicts parsing and pose using shared feature extraction, multi-scale feature combinations, iterative location refinement, and efficient context modeling.The framework is intended to make the two correlated tasks mutually beneficial in an end-to-end model.
  • Proposed networks: SS-JPPNet applies self-supervised structure-sensitive learning to parsing without requiring manually annotated pose joints.It derives approximated joints from parsing annotations and evaluates predictions from a joint-structure perspective.

2 RELATED WORK

Prior human parsing and pose datasets are narrow in scale, categories, appearances, or annotation type, while existing methods separately model structure, context, and multi-scale features. LIP unifies the two tasks in a larger, more diverse benchmark and motivates joint modeling of their complementary information.

  • Datasets: Previous parsing datasets contain limited images or categories, including a largest cited dataset with 17,000 mostly upright fashion images.Such datasets are described as unsuitable for training models requiring complex appearance representations and multiple components.
  • Datasets: Existing pose datasets also have restricted coverage, such as sports-only images in LSP, limiting representation of real-life human poses.The cited dataset discussion contrasts specialized pose collections with the broader scope sought by LIP.
  • LIP benchmark: LIP is presented as the first benchmark focused on both human parsing and pose estimation, with 50,462 images, 20 parsing categories, and 16 body joints.The paper distinguishes LIP from datasets dedicated to clothes recognition, retrieval, or fashion modeling.
  • Human parsing: Human parsing methods commonly combine CNNs or recurrent networks with CRFs and multi-scale representations to capture structure and context.Attention mechanisms can learn pixel-level weights over multi-scale features.
  • Pose estimation: Pose estimation methods evolved from pictorial and graph models toward deep networks that encode relationships among body parts and joints.Examples include sequential convolutional architectures and repeated bottom-up, top-down networks.
  • Joint modeling: Prior work used pose to guide parsing or body-part attention to guide pose, whereas this paper focuses on jointly predicting both tasks.The motivation is to exploit their correlated structure and complementary information within one network.

3 LOOK INTO PERSON BENCHMARK

LIP is a large-scale benchmark for human parsing and pose estimation, combining detailed annotations with diverse, challenging real-world appearances and a secret test set.

  • Dataset properties: 50,462 images provide pixel-wise labels for 19 semantic parts and locations for 16 body joints.The dataset includes a background label alongside the 19 human-part or clothing categories.
  • Dataset properties: LIP covers varied poses, viewpoints, resolutions, clothing appearances, occlusions, and complex backgrounds from real-world scenarios.More than half of the images contain occlusions, and back views introduce ambiguity in left-right spatial layouts.
  • Evaluation protocol: The benchmark provides separate training, validation, and test splits, with annotations publicly available for training and validation images and a secret test set.The reported split contains 30,462 training, 10,000 validation, and 10,000 test images.
  • Annotation: The annotation process used a dedicated tool, trained annotators, supervision, periodic checks, and a second-round review to select usable images.More than 100 students annotated images over five months, with 50,000 usable images selected from over 60,000 submissions.
  • Evaluation protocol: LIP also establishes a multiple-human parsing and pose estimation benchmark containing 5,147 multiple-person images.Its split includes 4,192 training, 497 validation, and 458 test images.

4 EMPIRICAL STUDY OF STATE-OF-THE-ART

The empirical study evaluates leading human parsing and pose estimation methods on LIP, using its annotations to examine robustness under varied appearances, viewpoints, and other challenges.

  • Study scope: The study benchmarks leading human parsing and pose estimation approaches on LIP and analyzes factors influencing their results.The stated goal is to evaluate robustness across challenges and identify limitations that can stimulate further research.
  • Study scope: Human parsing is compared on both the LIP validation and test sets using the benchmark tables.Table 2 covers validation performance, while Table 3 covers test performance.

4.1 Human Parsing

The human parsing study finds that detailed fine-grained parsing remains difficult, especially under missing heads, occlusion, and back views, with performance varying strongly by label size and appearance.

  • Overall performance: 44.80% mean IoU is the best LIP validation result, achieved by DeepLab (ResNet-101); Attention reaches 42.92% mean IoU.FCN-8s and SegNet obtain 28.29% and 18.17% mean IoU, respectively.
  • Overall performance: The reported parsing scores are substantially lower than leading results on other segmentation benchmarks, indicating that fine-grained human parsing is especially challenging.The paper attributes this difficulty to small parts and diverse fine-grained labels.
  • Challenge analysis: Back view is the most challenging parsing condition: Attention’s IoU falls from 42.92% to 33.50%.Head-missing images and occlusions also substantially reduce performance, while upper-body images are relatively easiest.
  • Per-class analysis: Parsing accuracy is higher for large-region labels such as face, upper clothes, coats, and pants than for small labels such as sunglasses, scarf, and skirt.DeepLab (ResNet-101) and Attention perform better on small labels because they use additional feature/context mechanisms, as described in the passage.
  • Qualitative failures: Qualitative results show left-right arm errors in back views, severe failures on head-missing images, and substantial degradation under severe occlusion.SegNet and FCN-8s fail to recognize some arms and legs in head-missing images, while stronger methods still confuse limbs and shoes.

4.2 Pose Estimation

The pose study evaluates established methods with PCKh and shows that performance degrades most for back views and occlusions, motivating joint modeling of parsing and pose.

  • Evaluation setup: PCKh counts a keypoint as correct when it lies within 50% of the person’s head-segment length.The study evaluates CPM, Hourglass, and ResNet-101 with atrous convolutions on LIP.
  • Overall performance: 77.5% total PCKh is achieved by Hourglass on the LIP validation set, the best reported overall pose result.The paper links this performance to multiple hourglass modules and intermediate supervision; CPM obtains comparable performance.
  • Challenge analysis: Back view reduces all methods’ scores by nearly 10% relative to the average, while occlusion lowers Hourglass PCKh by 4.60%.Head-missing images are excluded because PCKh depends on head size.
  • Qualitative failures: Qualitative failures include missed occluded joints, swapped left-right limbs in back-view or head-missing images, and irrational predictions when limbs are close together.The reported examples also show unsatisfactory performance on gray images and unusual appearances.
  • Joint modeling: The paper argues that parsing and pose can benefit from their intrinsic consistency and presents a unified framework to model their correlation.Its context modeling uses body parts and joints together, improving parsing across the challenging appearance types shown in Fig. 5.

5 METHODS

The paper develops JPPNet, a unified network that jointly models pixel-wise human parsing and joint-wise pose estimation through shared features, contextual modules, and coarse-to-fine refinement. SS-JPPNet further derives structure-sensitive supervision from parsing annotations to inject pose structure without extra manual supervision.

  • Joint network design: JPPNet jointly predicts human parsing and pose estimation by exploiting their complementary pixel-wise and joint-wise representations.Parsing supplies detailed semantic information, while pose provides higher-level body structure.
  • Coarse-to-fine modeling: JPPNet combines multi-scale features with iterative refinement as two complementary coarse-to-fine schemes.Multi-scale processing and refinement are incorporated into one end-to-end trainable framework.
  • Joint network design: The framework uses a shared residual network for efficient feature extraction, followed by separate parsing and pose subnetworks and a refinement network.The refinement stage reintegrates intermediate predictions into feature space to improve segmentation and joint localization.
  • Context modeling: The parsing and pose subnetworks use atrous convolution and ASPP to capture contextual information across multiple scales.ASPP applies filters at multiple sampling rates and effective fields of view.
  • Self-supervised structure-sensitive learning: The structure-sensitive loss combines joint-structure and parsing-segmentation losses, enabling pose guidance without extra information.The strategy is called self-supervised because its structural supervision is generated from existing parsing results.
  • Self-supervised structure-sensitive learning: SS-JPPNet generates approximated joints from parsing annotations and uses their structure to weight the pixel-wise segmentation loss.The approach defines joints from semantic-region centers and evaluates their alignment with corresponding ground-truth structures.

6 EXPERIMENTS

Experiments evaluate JPPNet and SS-JPPNet on LIP and PASCAL-Person-Part, including quantitative, qualitative, cross-condition, and ablation analyses. The results show strong performance across parsing and pose estimation, while multi-scale features and iterative refinement benefit the two tasks differently.

  • Experimental setup: The experiments compare the proposed models with strong baselines on LIP and evaluate SS-JPPNet on PASCAL-Person-Part.The study reports validation and test comparisons, qualitative examples, challenging-factor analyses, and ablations.
  • Human parsing: 51.37% mean IoU on LIP validation improves the previous best performance from 44.80%, while SS-JPPNet exceeds DeepLab and Attention on average IoU.The simplified model is 3.09% better than DeepLab (VGG-16) and 1.81% better than Attention.
  • Human parsing: JPPNet improves parsing across challenging appearances, object sizes, semantic classes, and confusing left-right body-part labels.Reported gains particularly include arms, legs, and shoes, where joint structure helps resolve ambiguity.
  • Human parsing: 59.36% IoU on PASCAL-Person-Part surpasses DeepLab-LargeFOV by 7.58% and Attention by 2.97%.This result is reported as evidence that the self-supervised strategy benefits human parsing.
  • Pose estimation: 82.7% PCKh on the LIP test set establishes a state-of-the-art pose result, with 5.0% and 5.5% gains on hip and ankle over the closest competitor.The method also reports improvements on the validation set and across different appearance conditions.
  • Ablation studies: Multi-scale features greatly improve human parsing but only slightly improve pose estimation, whereas iterative refinement substantially benefits pose estimation and can reduce parsing performance.Two-stage refinement decreases parsing performance in the reported ablation.

7 CONCLUSION

The paper presents LIP as a large-scale benchmark for human parsing and pose estimation and introduces a joint network exploring the connection between the two tasks. Extensive results demonstrate the effectiveness of the proposed approaches.

  • Benchmark: LIP contains 50,462 images with 19 semantic part labels and 16 body joints, exceeding existing datasets in scale and annotation richness.The benchmark is designed to support progress in human-centric tasks.
  • Joint modeling: The proposed joint human parsing and pose estimation network explores the intrinsic connection between the two tasks.The paper reports extensive results demonstrating the effectiveness of the proposed approaches.
Loading 1804.01984v1…