Source-linked AI summary
Devil in the Details: Towards Accurate Single and Multiple Human Parsing
Tao Ruan, Ting Liu, Zilong Huang, Yunchao Wei, Shikui Wei, Yao Zhao, Thomas Huang
TL;DR
Human parsing needs accurate and efficient handling of fine-grained regions, but it remains unclear how to exploit relevant semantic-segmentation properties. The paper analyzes feature resolution, global context, and edge details, then develops CE2P and extends it to multiple-person parsing. CE2P achieves state-of-the-art results across three human-parsing benchmarks, while the paper notes failure cases in complex multiple-person scenes.
Problem
It remains unclear how to develop an accurate and efficient human-parsing system by exploiting feature resolution, global context information, and edge details.
Method
The paper develops an end-to-end CE2P framework combining high-resolution embedding, global context embedding, and edge perceiving, and extends it with global-to-local branches for multiple human parsing.
Results
CE2P achieves state-of-the-art performance on all three human-parsing benchmarks, including 45.31% mean AP^r on CIHP and 33.34% AP^p_0.5 on MHP v2.0.
Takeaways & Limitations
The analysis identifies feature resolution, global context, and edge details as useful properties for improving single and multiple human parsing.
Takeaways & Limitations
The multiple-person system still produces failure cases, including limitations associated with its fusion strategy.
Abstract
from arXiv · showhide
Human parsing has received considerable interest due to its wide application potentials. Nevertheless, it is still unclear how to develop an accurate human parsing system in an efficient and elegant way. In this paper, we identify several useful properties, including feature resolution, global context information and edge details, and perform rigorous analyses to reveal how to leverage them to benefit the human parsing task. The advantages of these useful properties finally result in a simple yet effective Context Embedding with Edge Perceiving (CE2P) framework for single human parsing. Our CE2P is end-to-end trainable and can be easily adopted for conducting multiple human parsing. Benefiting the superiority of CE2P, we achieved the 1st places on all three human parsing benchmarks. Without any bells and whistles, we achieved 56.50\% (mIoU), 45.31\% (mean $AP^r$) and 33.34\% ($AP^p_{0.5}$) in LIP, CIHP and MHP v2.0, which outperform the state-of-the-arts more than 2.06\%, 3.81\% and 1.87\%, respectively. We hope our CE2P will serve as a solid baseline and help ease future research in single/multiple human parsing. Code has been made available at \url{https://github.com/liutinglt/CE2P}.
Introduction
Human parsing requires accurate pixel-level separation of fine-grained body and clothing regions, yet it remains unclear how to exploit semantic-segmentation advances efficiently. CE2P addresses this gap by analyzing feature resolution, context, and edge details in a unified framework.
- Human parsing identifies body parts and clothing items at pixel level, supporting applications such as ecommerce, human-machine interaction, image editing, and virtual reality.
- Fine-grained parsing is difficult because left-right body-part distinctions, especially in crossed-leg and back-view cases, are easily confused.
- The paper analyzes feature resolution, global context information, and edge details to determine how they affect human-parsing performance.
- CE2P combines high-resolution embedding, global context embedding, and edge perceiving modules in an end-to-end framework.
- CE2P reports significant performance gains over current state-of-the-art methods on all three human-parsing benchmarks.
Related Work
Human parsing builds on semantic-segmentation methods, especially fully convolutional, encoder-decoder, dilated-convolution, and context-integration approaches. Prior work also includes CRF- and pose-assisted methods and architectures that combine local and global information.
- Early human-parsing methods used conditional random fields and human pose estimation to assist parsing.
- Co-CNN hierarchically integrates local and global context information and improves human-parsing performance.
- Many recent human-parsing methods are based on fully convolutional networks following their improvements in semantic segmentation.
- Encoder-decoder architectures with skip connections recover dense feature responses, while dilated convolutions produce higher-resolution outputs.
- Other approaches integrate local and global information or use multi-scale features to generate more reliable predictions.
Context Embedding with Edge Perceiving
CE2P unifies high-resolution details, global context, and semantic edge information for single human parsing, then extends the framework to multiple-person images through global-to-local prediction and instance refinement.
- Key Modules of CE2P: CE2P integrates local fine details, global context, and semantic edge context into an end-to-end unified network.
- Context Embedding: The context embedding module uses pyramid pooling at four scales to incorporate global representations and provide context priors.
- High-resolution Embedding: The high-resolution embedding module combines intermediate low-level visual features with high-level semantic features to recover details needed for small objects.
- Training: CE2P jointly produces two parsing results and an edge prediction, with losses for parsing, edge detection, and edge-guided parsing.
- Multiple Human Parsing: M-CE2P extends CE2P to multiple human parsing with global and local branches, using global context for occlusions and Mask R-CNN masks for instance assignment.
Experimental Results
Experiments evaluate CE2P on LIP, CIHP, and MHP v2.0, showing gains from its modules and strong results in single and multiple human parsing. The analyses also examine branch fusion, label refinement, and remaining failure cases.
- Single Human Parsing: LIP experiments evaluate CE2P module variants against a baseline using validation-set performance and visualized errors.The baseline predicts directly from a ResNet-101 final feature map at 1/16 input resolution and reaches 47.97% accuracy.
- Single Human Parsing: About 1.5% mIoU improvement from global context embedding shows that multi-scale context assists fine-grained parsing.Small categories such as socks and sunglasses also improve with high-resolution embedding and edge perceiving.
- Single Human Parsing: CE2P outperforms JPPNet’s 51.37% result with a simpler network structure and no extra pose annotation.The reported advantage is associated with recovering details and capturing characteristics of small objects for pixel-level classification.
- Multiple Human Parsing: M-CE2P’s all-branch fusion achieves the best performance on most MHP v2.0 metrics, especially AP^p and PCP, because branches complement one another.The global branch supplies context while local branches capture precise local views and small-scale persons.
- Multiple Human Parsing: 9.64% and 9.40% validation-set gains over Zhao et al. (2018) are reported for MHP v2.0 AP^p_0.5 and PCP_0.5, respectively.On the MHP v2.0 test set, the reported gains are 8.20%, 9.57%, and 0.47% for AP^p_0.5, PCP_0.5, and mean AP^p.
- Challenge and Benchmark Results: 45.31% mean AP^r and 33.34% AP^p_0.5 are reported for CIHP and MHP v2.0, respectively, with improvements over second place of more than 3.81% and 1.87%.The paper reports first places across all three human parsing tracks and CIHP test-set results of 63.77% mIoU, 45.31% mean AP^r, and 50.94% AP^r_0.5.
Conclusion
The paper investigates feature resolution, global context information, and edge details, then integrates them into the CE2P system for human parsing.
- CE2P unifies high-resolution embedding, global context embedding, and edge perceiving modules to incorporate detailed and contextual information.