Source-linked AI summary
Parsing R-CNN for Instance-Level Human Analysis
Lu Yang, Qing Song, Zhihui Wang, Ming Jiang
TL;DR
Human part segmentation, dense pose estimation, and related tasks require models to distinguish human instances and represent their details. Parsing R-CNN provides a region-based pipeline with a parsing branch for instance-level human analysis. It reports state-of-the-art results on human-parsing benchmarks and first place in the COCO 2018 DensePose Estimation challenge.
Problem
Human part segmentation, dense pose estimation, and human-object interactions require instance-level detail, while existing class-agnostic mask branches lack detailed features and relations between human parts or dense points.
Source: §1 · p. 1Method
Parsing R-CNN uses a region-based pipeline with a parsing branch, proposals separation sampling, geometric and context encoding, and decoupled parsing operations.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors present Parsing R-CNN as general beyond part segmentation and dense pose, naming human-object interactions as a possible application.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Instance-level human analysis is common in real-life scenarios and has multiple manifestations, such as human part segmentation, dense pose estimation, human-object interactions, etc. Models need to distinguish different human instances in the image panel and learn rich features to represent the details of each instance. In this paper, we present an end-to-end pipeline for solving the instance-level human analysis, named Parsing R-CNN. It processes a set of human instances simultaneously through comprehensive considering the characteristics of region-based approach and the appearance of a human, thus allowing representing the details of instances. Parsing R-CNN is very flexible and efficient, which is applicable to many issues in human instance analysis. Our approach outperforms all state-of-the-art methods on CIHP (Crowd Instance-level Human Parsing), MHP v2.0 (Multi-Human Parsing) and DensePose-COCO datasets. Based on the proposed Parsing R-CNN, we reach the 1st place in the COCO 2018 Challenge DensePose Estimation task. Code and models are public available.
1. Introduction
The paper frames human part segmentation, dense pose estimation, and human-object interactions as related tasks that need instance-level detail. It proposes Parsing R-CNN to address gaps in existing methods and reports strong results across human-parsing and dense-pose benchmarks.
1. Introduction
- Existing class-agnostic mask branches lack the detailed features and geometric and semantic relations needed for instance-level human analysis.The paper identifies these gaps as motivation for Parsing R-CNN.Source: §1 · p. 1
- Proposals separation sampling is adopted to enhance feature semantic information while maintaining feature resolution.The pipeline figure identifies an FPN backbone, RoIAlign, and a parsing branch for instance-level human analysis.Source: §1 · p. 1, Figure 2 · p. 2
- The parsing branch enlarges RoI resolution to distinguish the many human-part or dense-point categories.Source: §1 · p. 2
- The geometric and context encoding module enlarges receptive field and captures relationships between body parts, producing better-quality class-aware masks.It comprises multi-level receptive-field and context information, followed by geometric-correlation learning.Source: §1 · p. 2
- The parsing branch is decoupled into semantic space transformation, geometric and context encoding, and semantic feature representation, with a composition scheme aimed at high accuracy and low computational overhead.Source: §1 · p. 2
- Parsing R-CNN outperforms all known top-down or bottom-up methods on CIHP and MHP v2.0 for human part segmentation, and achieves 64.1% mAP on COCO DensePose test, winning 1st place in COCO 2018 Challenge.Source: §1 · p. 2
- Parsing R-CNN is presented as applicable beyond part segmentation and dense pose, with human-object interactions named as a possible further task.Source: §1 · p. 2
2. Related Work
Related work covers region-based detection and segmentation, human-part datasets and methods, and dense-pose estimation. The authors treat part segmentation and dense pose as related instance-level tasks and propose Parsing R-CNN as a unified solution.
2. Related Work
- Region-based methods generate candidate regions and perform classification and bounding-box regression within each region; Mask R-CNN extended this approach to instance segmentation and pose estimation.Source: §2 · p. 2
- MHP v2.0 contains 25,403 annotated images with 58 fine-grained categories, while CIHP has 38,280 images with pixel-wise labels for 20 categories and instance identification.These datasets have promoted research in human part segmentation.Source: §2 · pp. 2–3
- Prior part-segmentation work includes Nested Adversarial Network and Part Grouping Network, while the authors identify room for better results and a unified end-to-end pipeline.Source: §2 · p. 3
- DensePose-COCO provides manually annotated image-to-surface correspondences on 50k COCO images, and DensePose-RCNN combines dense regression with Mask R-CNN.The cited prior work leaves instance scale and feature resolution among the issues not discussed.Source: §2 · p. 3
- The authors argue that human part segmentation and dense pose share commonalities and present Parsing R-CNN as a unified region-based solution for instance-level human analysis.Source: §2 · p. 3
3. Parsing R-CNN
Parsing R-CNN adds a parsing branch to a region-based pipeline for instance-level human analysis. Its design adapts feature sampling, RoI resolution, context encoding, and branch structure to represent human-instance details.
3.1. Proposals Separation Sampling
- 74% of CIHP and 86% of MHP v2.0 instances occupy >10% of the image, versus <20% of COCO instances, motivating PSS's finest-level parsing features.PSS retains FPN scale assignment for the bbox branch but performs parsing-branch RoIPool/RoIAlign only on P2.Source: §3.1 · p. 3, Figure 3 · p. 3
3.2. Enlarging RoI Resolution
- ERR uses 32×32 RoIs in the parsing branch to retain detail, while decoupling its batch size from detection to a fixed value such as 32 reduces training time and memory overhead without accuracy degradation.The higher RoI resolution increases computational cost in the branch.Source: §3.2 · p. 4
3.3. Geometric and Context Encoding
- GCE replaces the parsing-branch FCN with ASPP and non-local components to encode multi-scale context and geometric relations between human parts.The module's ASPP uses one 1×1 and three 3×3 convolutions with rates (6, 12, 18), alongside image-level features and an embedded-Gaussian non-local operation.Source: §3.3 · p. 4, Figure 5 · p. 4
3.4. Parsing Branch Decoupling
- Parsing branch decoupling separates semantic-space transformation, GCE, and task-specific semantic feature representation so their speed/accuracy trade-offs can be analyzed.The paper identifies these components as before GCE, GCE module, and after GCE, respectively.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate Parsing R-CNN on human part segmentation and dense pose estimation, with FPN and RoIAlign architectures trained end-to-end. The reported results include leading benchmark comparisons and gains from the proposed components, longer training, and COCO pretraining.
4.2. Experiments on Human Part Segmentation
- On CIHP, PSS improves mIoU by about 1.0; ERR at 32×32 adds 2.8 mIoU and instance-level gains of 5.0, 1.7, and 4.4, with a 12% speed reduction.The 64×64 RoI improves performance further, but the authors select 32×32 for its speed/accuracy trade-off.Source: §4.2 · p. 6
- GCE improves mIoU by about 2.0 over eight stacked convolutions; PBD's after-GCE layers add +0.8, +5.3, +2.0, and +3.9 across semantic and instance metrics.Without ASPP, Non-local alone degrades performance versus baseline; the authors infer that GCE can perform semantic space transformation, making the before-GCE part unnecessary.Source: Table 4 · p. 5, §4.2 · p. 6
- On CIHP, longer training improves results; COCO keypoint pretraining adds about 1.1–2.4 mIoU, and combining both strategies yields 4.0 mIoU and instance-metric gains of 6.9, 2.9, and 6.1.Table 5 reports results for different pretrained models and iteration counts.Source: Table 5 · p. 5, §4.2 · p. 7
- On CIHP, the full Parsing R-CNN improves over baseline by 10.3 mIoU, 24.0 APp50, 9.2 APp vol, and 18.3 PCP50.The results progressively add PSS, ERR, GCE, and PBD; Table 6 summarizes them.Source: Table 6 · p. 6, §4.2 · p. 7
- On MHP v2.0, Parsing R-CNN improves over baseline by 10.3 mIoU, 16.5 APp50, 7.1 APp vol, and 18.2 PCP50.The dataset has 59 semantic categories, including some small-scale categories; Table 7 reports the results.Source: Table 7 · p. 6, §4.2 · p. 7
- On CIHP, ResNet-50-FPN beats PGN by 1.7 mIoU; on MHP v2.0, Parsing R-CNN exceeds the prior best by 7.4 APp50, 1.0 APp vol, and 15.7 PCP50.PGN uses multi-scale inputs and horizontal flips; Parsing R-CNN's CIHP result uses no test-time augmentation. Table 8 reports the comparisons.Source: Table 8 · p. 6, §4.2 · p. 7
4.3. Experiments on Dense Pose Estimation
- On DensePose-COCO, Parsing R-CNN reaches 55.0% AP, 6.1 points above DensePose-RCNN; COCO pretraining adds 3.3 AP, and AP75 improves from 50.8% to 66.9%.Table 9 compares the backbones and components on the validation set.Source: Table 9 · p. 7, §4.3 · p. 8
- A single ResNeXt101-32x8d model placed first in the COCO 2018 DensePose challenge, reaching 64.1% AP and surpassing second place by 6 points.The leaderboard results are summarized in Table 10.Source: §4.3 · p. 8, Table 10 · p. 8
5. Conclusion
Parsing R-CNN is a region-based approach for instance-level human analysis that achieves state-of-the-art results on several challenging benchmarks. Its proposed aspects are evaluated on human part segmentation and dense pose estimation, and the method placed first in the COCO 2018 DensePose Estimation Challenge.
5. Conclusion
- Parsing R-CNN achieves state-of-the-art results on several challenging benchmarks for instance-level human analysis.The conclusion identifies it as a novel region-based approach.Source: §5 · p. 8
- The paper verifies the effectiveness of its four proposed aspects on human part segmentation and dense pose estimation.Source: §5 · p. 8
- Parsing R-CNN placed first in the COCO 2018 DensePose Estimation Challenge; extending it to more human-analysis applications is future work.Source: §5 · p. 8