Source-linked AI summary
PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model
George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, Kevin Murphy
TL;DR
PersonLab addresses box-free multi-person pose estimation and instance segmentation in crowded images, where semantic detections must be associated with individual people. It uses a fully convolutional bottom-up model that detects keypoints, predicts relative displacements, and assigns person pixels through part-induced geometric embeddings. On COCO, it reports 0.687 keypoint AP with multi-scale inference and 0.417 person mask AP, while requiring keypoint-level annotations for segmentation training.
Problem
Multi-person images require jointly identifying people, localizing their keypoints, and estimating separate instance masks without relying on bounding-box-first processing.
Method
A fully convolutional bottom-up model detects keypoints, predicts pairwise displacements, and uses per-pixel keypoint offsets as geometric embeddings to group poses and segment instances.
Results
0.687 keypoint AP with multi-scale inference and 0.417 person mask AP are reported on COCO, with the mask result outperforming FCIS at 0.386.
Takeaways & Limitations
The unified bottom-up model delivers competitive pose and person-segmentation results with efficient inference that avoids second-stage box refinement and clustering.
Takeaways & Limitations
Training the instance segmentation task relies on keypoint-level annotations, motivating future weakly supervised part discovery.
Abstract
from arXiv · showhide
We present a box-free bottom-up approach for the tasks of pose estimation and instance segmentation of people in multi-person images using an efficient single-shot model. The proposed PersonLab model tackles both semantic-level reasoning and object-part associations using part-based modeling. Our model employs a convolutional network which learns to detect individual keypoints and predict their relative displacements, allowing us to group keypoints into person pose instances. Further, we propose a part-induced geometric embedding descriptor which allows us to associate semantic person pixels with their corresponding person instance, delivering instance-level person segmentations. Our system is based on a fully-convolutional architecture and allows for efficient inference, with runtime essentially independent of the number of people present in the scene. Trained on COCO data alone, our system achieves COCO test-dev keypoint average precision of 0.665 using single-scale inference and 0.687 using multi-scale inference, significantly outperforming all previous bottom-up pose estimation systems. We are also the first bottom-up method to report competitive results for the person class in the COCO instance segmentation task, achieving a person category average precision of 0.417.
1 Introduction
PersonLab unifies bottom-up multi-person detection, pose estimation, and instance segmentation by detecting keypoints, grouping them with learned displacements, and associating person pixels geometrically. It achieves strong COCO results while remaining efficient and box-free.
- 1 Introduction: PersonLab jointly targets multi-person detection, 2-D pose estimation, and instance segmentation in cluttered, crowded images.The unified goal is to identify every person, localize facial and body keypoints, and estimate an instance mask.
- 1 Introduction: The model predicts all person keypoints fully convolutionally, learns pairwise relative displacements, and greedily groups detections starting from the most confident keypoint.Starting from the most confident detection helps grouping in clutter instead of relying on a distinguished landmark.
- 1 Introduction: Dense person segmentation combines semantic person maps with per-pixel offsets to all keypoints, creating geometric embeddings for instance association.Pixels are assigned by comparing predicted keypoint locations with candidate person instances using weighted average keypoint distances.
- 1 Introduction: 0.417 person mask AP outperforms the top-down FCIS result of 0.386 on COCO instance segmentation.The authors report this as the first competitive bottom-up result for the person class.
- 1 Introduction: 0.687 keypoint AP improves on the best previous bottom-up result of 0.655.The model is trained on the standard COCO keypoint dataset and avoids second-stage box refinement and clustering.
2 Related work
Prior work organizes pose estimation and instance segmentation around top-down detection-first and bottom-up parts-first paradigms. PersonLab follows the bottom-up direction, associating pixel-level predictions with object instances through efficient geometric or voting-based grouping.
- 2 Related work: Human pose estimation research moved from part-based graphical models toward large convolutional networks for single- and multi-person settings.Earlier models represented people as configurable parts and used tractable inference for dependencies among body parts.
- 2 Related work: Top-down pose methods detect or localize person bounding boxes before predicting keypoints within each person region.Examples include G-RMI, CFN, RMPE, Mask R-CNN, and CPN.
- 2 Related work: Bottom-up pose methods detect body parts first and group them into human instances using optimization, affinity fields, or associative embeddings.Representative approaches include Linear Programming, greedy part-affinity grouping, and associative embedding.
- 2 Related work: Instance segmentation likewise divides methods into top-down approaches that classify or refine mask proposals and bottom-up approaches that associate pixel predictions with object instances.PersonLab belongs to the bottom-up family and uses multiple predictions in a Hough-voting-like clustering process.
- 2 Related work: Bottom-up instance segmentation research includes semantic clustering, center-direction prediction, depth-order encoding, Hough-like transforms, graph optimization, and boundary prediction.These methods differ in how they separate pixels and aggregate them into complete object instances.
3 Methods
PersonLab uses a box-free, bottom-up CNN to detect and group human keypoints, then associates person pixels with detected instances through long-range geometric embeddings. Recurrent offset refinement improves long-range keypoint grouping, while greedy decoding and scale-normalized association support efficient instance-level segmentation.
- Keypoint detection: The method detects all visible keypoints instance-agnostically using one heatmap and two displacement channels per keypoint type.Heatmaps mark disks around keypoints, while short-range offsets point from nearby image positions to the corresponding keypoints.
- System overview: PersonLab predicts keypoint heatmaps, short-range offsets, mid-range pairwise offsets, person segmentation maps, and long-range offsets in one CNN.The first three outputs support pose estimation; segmentation maps and long-range offsets, together with pose detections, support instance segmentation.
- Instance segmentation: The segmentation module combines semantic person maps with pose detections and long-range offsets to predict person instance masks.The vector fields partition the image into basins of attraction around person instances, supporting efficient pixel-to-instance assignment.
- Keypoint grouping: Mid-range pairwise offsets connect candidate keypoints across a person’s kinematic graph, enabling association when multiple people are present.The method recurrently refines these offsets with short-range offsets; repeating refinement twice drastically decreases regression errors and facilitates grouping.
- Keypoint grouping: Greedy decoding groups keypoints by processing Hough-map maxima from a shared priority queue, without privileging any keypoint type as the seed.Candidates above a 0.01 threshold initialize instances, and the type-agnostic strategy handles cases where substantial portions of a person are occluded.
- Instance segmentation: Long-range offsets point each foreground person pixel toward the corresponding person’s keypoints, forming a geometric embedding for instance association.The association metric weights discrepancies by keypoint presence probability and normalizes them by detected-instance scale λ_j.
4 Experimental evaluation
PersonLab is evaluated on COCO keypoints and person instance segmentation using COCO-only training data, with strong results across both tasks. The evaluation also examines inference settings, proposal limits, and small-instance handling.
- Experimental setup: 64,115 COCO training images and 5,000 validation images define the main evaluation setup, with test-dev evaluation on 20,288 images.
- Keypoint evaluation: 0.687 keypoint AP is achieved on COCO test-dev with the best ResNet-152 model and multi-scale inference.The single-scale result is already stronger than earlier bottom-up methods and several top-down methods.
- Instance segmentation evaluation: Small-instance missing-keypoint imputation significantly improves the reported instance segmentation performance.The imputed keypoints are used for the reported segmentation experiments, while results without imputation appear in the appendix.
- Instance segmentation evaluation: On COCO validation, performance is similar to Mask R-CNN for medium and large people but worse for small people.
- Instance segmentation evaluation: The method evaluates only the person class and cannot be applied to the other COCO classes because it is keypoint-based.
5 Conclusions
The paper presents a unified bottom-up, part-based model for person detection, pose estimation, and instance segmentation. Its main limitation is reliance on keypoint annotations for training the segmentation task.
- PersonLab jointly addresses person detection, pose estimation, and instance segmentation through unified part-based modeling.
- The segmentation task relies on keypoint-level annotations during training, limiting the current approach.The authors propose weakly supervised part discovery as a future direction for overcoming this limitation.
A Ablation Experiments
The ablation experiments examine how different model choices affect performance under a fixed ResNet-101, single-scale setup.
- All ablation experiments use a ResNet-101 model with single-scale inference.Boldface identifies the model variant used for the main reported results.
A.1 Ablation: Input image size and activation output stride
PersonLab balances accuracy and inference cost through input resolution and activation-stride choices. Denser activations improve performance, while higher input resolution is especially important for small people.
- Activation output stride: Reducing output stride from 32 to 16 pixels significantly increases model performance.Reducing it further to 8 pixels yields only a small additional improvement but significantly raises computation cost.
- Input image size: Higher-resolution input images are required to accurately capture small person instances.Large person instances retain reasonably good keypoint AP with input sizes as small as 601 or 801 pixels.
- Speed-accuracy tradeoff: 341 msec, 355 msec, and 464 msec are the inference times for output strides 32, 16, and 8, respectively, on a Titan X.These timings cover both keypoint and instance-segmentation final outputs for an 801x529 image.
- Speed-accuracy tradeoff: Output stride 16 pixels strikes an excellent speed-accuracy tradeoff.It improves over stride 32 without the substantially higher computation cost associated with stride 8.
A.2 Ablation: Keypoint scoring and non-maximum suppression
The ablation compares keypoint scoring and suppression strategies in four model configurations. Both proposed components improve AP over their alternatives and work well together.
- Non-maximum suppression: The proposed soft-NMS is compared against hard OKS-based NMS using a threshold of 0.5.Both suppression choices are evaluated within the four alternative model configurations.
- Scoring and NMS: Expected-OKS keypoint scoring and soft-NMS both bring significant AP improvements over the alternatives from [33].The four configurations combine the two scoring mechanisms with hard-NMS and soft-NMS for comparison.
A.3 Ablation: Mid- and long-range offset refinement
Offset refinement improves both keypoint and person-segmentation performance, with the largest gains occurring for large objects. The evaluation compares models without and with refinement on COCO validation splits.
- Keypoint and segmentation effects: 3.3% keypoint AP and 2.2% segmentation AP are the improvements from offset refinement.The comparison disables refinement during both training and evaluation for the baseline model.
- Object-size effects: +5.4% keypoint AP and +9.1% segmentation AP are the largest improvements, observed for large object instances.Large objects span more of the image, making accurate regression without refinement particularly challenging.
- Refinement mechanism: Offset refinement reduces mid-range regression errors through repeated refinement at CNN output-activation resolution.The method repeats the refinement step twice and uses bilinear interpolation, keeping the process fast.
A.4 Ablation: Small instance keypoint imputation in model training
Keypoint imputation has opposite effects across the two tasks: it slightly lowers keypoint performance but substantially improves segmentation, especially for small objects.
- Training strategy: Keypoint imputation is used during model training to address small COCO person instances.Its benefit is apparent in segmentation evaluation, where small-object performance more than doubles.
- Keypoint evaluation: 0.8% is the decrease in COCO keypoint performance from keypoint imputation.The keypoints evaluation protocol excludes small person instances, which explains why imputation does not help this task.
- Segmentation evaluation: 4.4% is the improvement in COCO segmentation performance from keypoint imputation.Most of the gain comes from small objects, whose AP increases from 7.6% to 16.4%.