Source-linked AI summary
Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, Liang Lin
TL;DR
Human parsing lacks large, diverse benchmarks and methods that adequately encode human-body structure. This paper introduces LIP and a self-supervised structure-sensitive learning framework, which improves parsing on LIP and PASCAL-Person-Part. The results support using structure information for more effective fine-grained human parsing.
Problem
Existing human parsing datasets provide limited scale and diversity, while appearance-based methods can produce unreasonable outputs without human-body structure priors.
Method
The paper introduces the LIP benchmark and a self-supervised structure-sensitive framework that derives nine human joints from parsing annotations to enforce structural consistency.
Results
The method significantly surpasses previous methods on LIP and PASCAL-Person-Part, including 59.36% IoU on PASCAL-Person-Part, 7.58% above DeepLab-LargeFOV.
Takeaways & Limitations
Human joint structure can improve fine-grained parsing, especially ambiguous left-right labels and small regions such as sunglasses, socks, and gloves.
Takeaways & Limitations
The approach relies on structure-sensitive supervision derived from parsing annotations because no other extensive information is provided for human parsing.
Abstract
from arXiv · showhide
Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.
1. Introduction
Human parsing needs larger, more representative benchmarks because existing CNN-based methods depend heavily on annotated training data and can produce structurally unreasonable outputs. LIP addresses these gaps with a challenging benchmark and self-supervised structure-sensitive learning framework.
- Human parsing supports detailed semantic understanding and higher-level applications such as person re-identification and human behavior analysis.
- Existing CNN-based human parsing methods rely heavily on annotated images, motivating datasets with varied clothing, articulation, and partial occlusion.
- Bottom-up appearance-based approaches can produce unreasonable configurations, such as connecting the right arm to the left shoulder, without human-body structure priors.
- The proposed self-supervised structure-sensitive framework explicitly enforces consistency between parsing results and human joint structures, surpassing previous methods on LIP and PASCAL-Person-Part.
2. Look into Person Benchmark
LIP is a large-scale human parsing benchmark designed for detailed semantic labeling under varied real-world appearances, viewpoints, occlusions, and backgrounds. Its annotations and evaluation setup quantify both dataset diversity and parsing performance across challenging conditions.
- LIP contains 50,462 annotated images and is described as an order of magnitude larger and more challenging than previous similar datasets.
- Each image has pixel-wise labels for 19 semantic human parts plus background, covering body parts, clothing, and fine-grained left-right distinctions.
- The benchmark includes challenging real-world poses, viewpoints, heavy occlusions, varied appearances, broad resolutions, and complex backgrounds.
- A standard evaluation server keeps the test set secret to reduce benchmark overfitting.
- The dataset includes 19,081 full-body, 13,672 upper-body, 403 lower-body, 3,386 head-missed, 2,778 back-view, and 21,028 occluded images.
- LIP validation analyses compare methods across occlusion, full-body, upper-body, head-missed, and back-view conditions.
3. Empirical study of state-of-the-arts
The study evaluates four human parsing and semantic segmentation approaches on LIP, examining overall performance, per-class behavior, and robustness across challenging conditions. Results show that detailed human parsing remains difficult, especially for small parts, missing heads, occlusion, and back views.
- Overall evaluation: The analysis evaluates FCN-8s, SegNet, DeepLabV2, and Attention on LIP using mean accuracy, mean IoU, and pixel-wise accuracy.Each method is trained on the LIP training set and evaluated on validation and test sets.
- Overall evaluation: 54.39% mean accuracy and 42.92% mean IoU make Attention the strongest method on the LIP validation set.FCN-8s and SegNet obtain 28.29% and 18.17% mean IoU, respectively.
- Overall evaluation: LIP performance is substantially lower than results on object-segmentation benchmarks such as PASCAL VOC, indicating greater difficulty for detailed human parsing.The paper attributes this challenge to small parts and diverse fine-grained labels.
- Challenging conditions: Back-view is the most challenging factor, while upper-body images are relatively easiest; head-missed images and occlusion also substantially reduce performance.Lower-body parts are harder than upper-body parts, partly because of small labels such as shoes and socks.
- Per-class analysis: Large-region labels such as face, upperclothes, coats, and pants achieve higher per-class IoU than small-region labels including sunglasses, scarf, and skirt.Attention and DeepLabV2 perform better on small labels through multi-scale features.
- Qualitative analysis: Without human body structure priors, bottom-up approaches can produce unreasonable configurations such as connecting the right arm to the left shoulder.Qualitative comparisons also show frequent right-arm/left-arm confusion in back-view images and failures when the head is missing.
4. Self-supervised Structure-sensitive Learning
The paper introduces self-supervised structure-sensitive learning, deriving human-joint supervision from parsing annotations to enforce structurally consistent pixel-wise predictions without extra joint labels.
- Overview: The method addresses existing parsing models’ limited consideration of human body configuration by adding high-level structural cues to pixel-wise prediction.Pose-estimation joints are unsuitable because their predictions may be erroneous and their annotations do not align with parsing labels.
- Self-supervised Structure-sensitive Loss: The framework computes joint-structure supervision directly from existing parsing annotations, avoiding additional human-joint labeling.This makes the supervision self-generated from the same annotations used for parsing.
- Self-supervised Structure-sensitive Loss: The structure-sensitive loss weights pixel-wise segmentation loss using the joint-structure loss, combining semantic segmentation with structural consistency.The parsing term is a pixel-wise softmax loss, while the joint-structure term uses Euclidean distance between predicted and ground-truth joint heatmaps.
- Self-supervised Structure-sensitive Loss: Nine joints are defined as region centers for the head, upper and lower body, arms, legs, and shoes, with merged semantic labels forming composite regions.For each parsing result and ground truth, these centers are represented as heatmaps; missing joints use zero-filled maps.
- Self-supervised Structure-sensitive Loss: The learning framework can be injected into advanced neural networks to incorporate global human-joint knowledge.The paper describes this adaptability and extensibility as consequences of generating the loss without extra information.
5. Experiments
Experiments evaluate the self-supervised structure-sensitive method on PASCAL-Person-Part and LIP using established network architectures, finding consistent gains across benchmarks, challenges, object sizes, and input scales.
- Experimental Settings: The method is evaluated on PASCAL-Person-Part and the challenging LIP dataset, which includes severe pose complexity, occlusions, and body truncation.Experiments use 1,716 training and 1,817 testing images from PASCAL-Person-Part, alongside LIP.
- Experimental Settings: The experiments use Attention as the basic architecture and also train a DeepLabV2-based network.Attention is selected for its leading accuracy and competitive efficiency; DeepLabV2 uses atrous convolution, multi-scale inputs, and atrous spatial pyramid pooling.
- Results and Comparisons: 59.36% mean IoU on PASCAL-Person-Part exceeds DeepLab-LargeFOV by 7.58% and Attention by 2.97%.The comparison uses the standard intersection over union criterion and four state-of-the-art baselines.
- Results and Comparisons: On LIP validation, the proposed architecture improves average IoU by 3.09% over DeepLabV2 and 1.81% over Attention, while also outperforming baselines on the test set.The reported gains are attributed to incorporating body-joint structure into pixel-wise prediction.
- Results and Comparisons: Structure-sensitive loss improves performance across LIP challenge types and substantially benefits different object sizes and input scales.Per-class results also improve for ambiguous arms, legs, and shoes and for small labels such as sunglasses, socks, and gloves.
- Results and Comparisons: Qualitative comparisons show more semantically meaningful and precise predictions under appearance and position variation, including successful segmentation of small shoes and confusing left-right labels.The comparison covers upper-body, back-view, head-missed, occluded, and full-body images.
6. Conclusions
The paper introduces LIP as a large-scale human parsing benchmark and proposes self-supervised structure-sensitive learning to enforce semantic consistency with human joint structures.
- Conclusions: LIP contains 50,462 images with 19 semantic part labels and supports detailed analysis of leading human parsing approaches.The benchmark is designed to spark progress in human parsing.
- Conclusions: Self-supervised structure-sensitive learning explicitly enforces consistency between parsing results and human joint structures.The strategy is presented as a novel learning approach within the paper.