Source-linked AI summary
Pose2Seg: Detection Free Human Instance Segmentation
Song-Hai Zhang, Ruilong Li, Xin Dong, Paul L. Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Hao-Zhi Huang, Shi-Min Hu
TL;DR
Detection-based instance segmentation struggles to separate heavily intertwined people, and public datasets provide limited severe-occlusion coverage with comprehensive annotations. Pose2Seg instead segments humans using pose-guided alignment and skeleton features, while introducing OCHuman; the paper reports better performance than detection-based frameworks and provides an occlusion-focused benchmark.
Problem
Bounding-box-based instance segmentation struggles with heavily intertwined people, while existing human datasets often lack severe occlusion or comprehensive annotations.
Method
Pose2Seg uses human pose to select and align instance regions, concatenates skeleton features with image features, and introduces the OCHuman benchmark.
Results
The pose-based system achieves better performance than detection-based instance segmentation, especially for occlusion, and OCHuman contains 8110 instances in 4731 images with average MaxIoU 0.67.
Takeaways & Limitations
Pose-based human instance segmentation and OCHuman emphasize human pose as a basis for separating occluded instances and studying occlusion robustness.
Takeaways & Limitations
OCHuman is designed for validation and testing, with general datasets such as COCO considered preferable for training.
Abstract
from arXiv · showhide
The standard approach to image instance segmentation is to perform the object detection first, and then segment the object from the detection bounding-box. More recently, deep learning methods like Mask R-CNN perform them jointly. However, little research takes into account the uniqueness of the "human" category, which can be well defined by the pose skeleton. Moreover, the human pose skeleton can be used to better distinguish instances with heavy occlusion than using bounding-boxes. In this paper, we present a brand new pose-based instance segmentation framework for humans which separates instances based on human pose, rather than proposal region detection. We demonstrate that our pose-based framework can achieve better accuracy than the state-of-art detection-based approach on the human instance segmentation problem, and can moreover better handle occlusion. Furthermore, there are few public datasets containing many heavily occluded humans along with comprehensive annotations, which makes this a challenging problem seldom noticed by researchers. Therefore, in this paper we introduce a new benchmark "Occluded Human (OCHuman)", which focuses on occluded humans with comprehensive annotations including bounding-box, human pose and instance masks. This dataset contains 8110 detailed annotated human instances within 4731 images. With an average 0.67 MaxIoU for each person, OCHuman is the most complex and challenging dataset related to human instance segmentation. Through this dataset, we want to emphasize occlusion as a challenging problem for researchers to study.
1 Tsinghua University 2 BNRist 3 Cardiff University 4 Tencent AI Lab
The paper identifies its authors’ affiliations and provides links to the released code and OCHuman dataset.
- Code and dataset resources are available through the Pose2Seg and OCHumanApi repositories.
- Figure 1 states that human pose better separates heavily occluded people than bounding-boxes.
1. Introduction
The introduction motivates pose-based human instance segmentation because bounding-box methods struggle with intertwined people and severe occlusion. Pose2Seg addresses this with pose-guided alignment, skeleton features, and the OCHuman benchmark.
- Motivation: Bounding-box-based instance segmentation struggles when intertwined people occupy the same RoI, especially under heavy occlusion.Overlapping proposals can be removed by NMS, while shared bounding-boxes make target-instance identification difficult.
- Motivation: Human pose skeletons provide body-part location and visibility information that can distinguish heavily intertwined people.
- Framework: Pose2Seg introduces Affine-Align, which uses scale, translation, rotation, and left-right flip to normalize pose-based image windows.The alignment can correct objects with strange poses to a standard pose.
- Framework: Skeleton features are concatenated with aligned instance features to guide segmentation and distinguish intertwined instances within the same RoI.
- Benchmark: OCHuman is a benchmark for heavily occluded humans with comprehensive bounding-box, pose, and instance-mask annotations.It contains 8110 annotated human instances in 4731 images, with over 67% average bounding-box occlusion.
- Contributions: The paper claims its pose-based framework works better than detection-based frameworks, particularly under occlusion.
2. Related Work
Related work covers top-down and bottom-up pose estimation alongside detection-based instance segmentation. Existing approaches commonly rely on detection, while Pose2Seg relates pose estimation more directly to instance segmentation.
- Multi-Person Pose Estimation: Top-down pose methods detect and crop each person before single-person pose estimation, inheriting detection weaknesses under heavy occlusion.
- Multi-Person Pose Estimation: Bottom-up methods detect body-part keypoints for all people and cluster them into human pose instances.
- Instance Segmentation: Detection-based instance segmentation pipelines either apply segmentation after bounding-box detection or integrate detection and segmentation jointly.
- Instance Segmentation: Mask R-CNN is described as a state-of-the-art detection-based framework on the COCO competition.
- Pose and Instance Segmentation: Prior work combines human pose estimation and instance segmentation, including simultaneous prediction and cascade-based pose integration.
3. Occluded Human Benchmark
OCHuman is a benchmark designed to evaluate human detection, pose estimation, and instance segmentation under severe occlusion, with comprehensive annotations and defined validation/testing splits.
- OCHuman contains 8110 human instances in 4731 images, each heavily occluded by other people.Instances are selected using MaxIoU > 0.5; the dataset average is 0.67 MaxIoU per person.
- Each OCHuman instance includes a bounding-box, binary instance mask, and 17 body-joint locations.The annotations support detection, instance segmentation, and pose estimation.
- The dataset is split into validation and test sets containing 2500 and 2231 images, respectively.These splits contain 4313 validation instances and 3797 test instances.
- OCHuman further divides instances into Moderate cases with MaxIoU from 0.5 to 0.75 and Hard cases with MaxIoU above 0.75.
- OCHuman addresses a limitation of COCO, which has comprehensive annotations but few heavily occluded human cases.The benchmark is intended to test robustness to occlusion rather than rely solely on standard training data.
4. Approach
The approach segments people using pose-guided alignment and skeleton features instead of relying solely on detection bounding-boxes. It combines pose templates, Affine-Align, and a residual-based SegModule to process aligned instance features.
- 4.1. Overall Structure: The network takes an image and human pose as input, extracts image features, aligns RoIs to 64×64 using pose, and concatenates Skeleton features.
- 4.2. Affine-Align: Affine-Align estimates a transformation between each detected pose and clustered pose templates, then applies the best transformation to the image or features.The transformation encodes rotation, scale, translation, and left-right flipping.
- 4.2.1. Human Pose Representation: Each pose is represented by m body-part vectors containing joint coordinates and visibility, with m = 17 for COCO.Visibility handling includes distinct encodings for visible, invisible, and out-of-image joints.
- 4.2.2. Pose Templates: Pose templates are learned with K-means after cropping, centering, normalizing, and retaining poses with more than 8 valid points.Poses with few valid points are excluded because they would act as outliers during clustering.
- 4.2.2. Pose Templates: K = 3 is selected for clustering pose templates on the COCO training set.
- 4.3. Skeleton Features: Skeleton features combine 38-channel PAF maps with 17-channel part-confidence maps for each COCO human pose.These features encode skeleton structure and emphasize regions around body-part keypoints.
- 4.4. SegModule: SegModule is built from residual units and sized to provide sufficient receptive fields for combining aligned image features with artificial Skeleton features.
5. Experiments
Experiments evaluate the pose-based framework on occluded and general human images, comparing it with detection-based alignment and segmentation methods. Results show stronger performance under occlusion and benefits from pose-guided alignment, Skeleton features, and suitable receptive fields.
- Evaluation setup: The framework is evaluated on OCHuman for heavily occluded humans and COCOPersons for common daily-life scenarios.Models are trained end-to-end on COCOPersons using pose-keypoint and segmentation-mask annotations.
- 5.1. Performance on occlusion: Nearly 50% higher performance than Mask R-CNN is achieved on OCHuman using predicted keypoints.Using ground-truth keypoints more than doubles the accuracy, indicating sensitivity to keypoint quality.
- 5.2. Performance on general cases: 0.555 AP is achieved on COCOPersons with predicted keypoints, compared with 0.532 AP for Mask R-CNN.With ground-truth keypoints, the framework reaches 0.582 AP on the instance segmentation task.
- 5.3.1 Affine-Align v.s. RoI-Align: 0.544 AP with ground-truth pose exceeds 0.476 AP from ground-truth box-based alignment on OCHuman validation.The comparison keeps other components unchanged and removes NMS-related deficiencies by using ground-truth boxes.
- 5.3.1 Affine-Align v.s. RoI-Align: Pose-derived box alignment cannot match Affine-Align regardless of the expansion factor α.Affine-Align permits rotation, producing more discriminative RoIs for heavily intertwined people.
- 5.3.2 Skeleton features: Skeleton features improve accuracy across alignment strategies and help the network focus on a target when multiple people occupy one RoI.Combining RoI-Align and Skeleton features with ground-truth boxes and keypoints achieves 0.648 AP on COCOPersons.
- 5.3.3 SegModule: Ten residual units provide an approximately 50-pixel receptive field, enough for the 64 × 64 alignment size; deeper architectures add little benefit.Fewer units reduce learning capability, while additional units provide little further help.
6. Conclusion
The paper concludes that human instance segmentation benefits from selecting regions using pose rather than bounding boxes and from explicitly adding pose Skeleton features. It also introduces OCHuman as a benchmark centered on heavily occluded humans.
- 6. Conclusion: The proposed framework selects regions using pose-based Affine-Align rather than bounding boxes.Affine-Align combines scale, translation, rotation, and left-right flip for pose-based RoI selection.
- 6. Conclusion: Explicitly concatenating human pose Skeleton features further improves segmentation performance.The conclusion identifies Skeleton features as an additional component beyond pose-based region selection.
- 6. Conclusion: The pose-based system performs better in general cases and handles occlusion better than traditional detection-based instance segmentation.The paper presents OCHuman as a challenging benchmark focused on heavily occluded humans.