Source-linked AI summary
PifPaf: Composite Fields for Human Pose Estimation
Sven Kreiss, Lorenzo Bertoni, Alexandre Alahi
TL;DR
The paper addresses multi-person 2D pose estimation for small, crowded, and occluded pedestrians in transportation settings. PifPaf uses PIFs for body-part localization and composite PAFs for association in a bottom-up, box-free architecture. It outperforms prior methods at low resolution and performs on par at high resolution, including strong results on transportation-focused evaluations.
Problem
Existing pose-estimation methods perform poorly for limited-resolution humans and dense crowds with partial occlusion, although these settings matter for autonomous navigation.
Method
PifPaf is a bottom-up method using PIFs to localize body parts and composite PAFs to associate them into poses with two neural-network heads.
Results
PifPaf outperforms previous state-of-the-art methods in the low-resolution regime and performs on par at high resolution.
Takeaways & Limitations
PifPaf is particularly suited to urban-mobility scenes involving small, crowded, and partially occluded pedestrians.
Abstract
from arXiv · showhide
We propose a new bottom-up method for multi-person 2D human pose estimation that is particularly well suited for urban mobility such as self-driving cars and delivery robots. The new method, PifPaf, uses a Part Intensity Field (PIF) to localize body parts and a Part Association Field (PAF) to associate body parts with each other to form full human poses. Our method outperforms previous methods at low resolution and in crowded, cluttered and occluded scenes thanks to (i) our new composite field PAF encoding fine-grained information and (ii) the choice of Laplace loss for regressions which incorporates a notion of uncertainty. Our architecture is based on a fully convolutional, single-shot, box-free design. We perform on par with the existing state-of-the-art bottom-up method on the standard COCO keypoint task and produce state-of-the-art results on a modified COCO keypoint task for the transportation domain.
1. Introduction
PifPaf targets multi-person 2D pose estimation in transportation settings, where pedestrians are small, crowded, and partially occluded. It introduces composite Part Intensity and Part Association Fields to preserve localization and association detail at low resolution.
- Motivation: Transportation-domain pose estimation remains less accurate than “in the wild” estimation, despite its relevance to early detection of pedestrian actions.The paper connects farther-away pose detection with safer autonomous navigation and lower required perception resolution.
- Challenges: The target setting combines wide viewing angles, 30-90-pixel human heights, and dense crowds with pedestrian occlusion.The method aims for high recall and precision in this regime.
- Prior limitations: OpenPose and Mask R-CNN perform poorly on limited-resolution images and dense crowds despite strong results at sufficiently high resolution.OpenPose is bottom-up, whereas Mask R-CNN is top-down and begins with person detection.
- Contribution: PifPaf extends pose-estimation fields from scalar and vector fields to composite fields.Its PIF localizes body parts, while its composite PAF associates them into poses.
- Contribution: PifPaf uses two head networks, fine-grained PAF encoding, and Laplace-based L1 regression loss to support precise localization on low-resolution activation maps.The Laplace-based loss incorporates uncertainty for regression, while the PAF stores fine-grained association information.
2. Related Work
Prior human-pose methods divide into bottom-up approaches that group detected joints and top-down approaches that detect people before estimating joints. Their computational strategies and failure modes differ across resolution and crowding conditions.
- Method families: Bottom-up methods estimate body joints first and group them into individual poses, whereas top-down methods detect people first and estimate joints within bounding boxes.This distinction describes the two main organizational paradigms for multi-person pose estimation.
- Top-down methods: Top-down methods benefit from person detectors and labeled bounding boxes but generally struggle when person bounding boxes overlap.Mask R-CNN treats keypoint detection as an instance-segmentation task with one-pixel foreground masks.
- Bottom-up methods: Early bottom-up methods such as DeepCut and DeeperCut used integer linear programming for part association, requiring hours to process a single image.Later methods accelerated prediction with greedy decoders and additional association tools.
- Downstream tasks: Improved 2D pose estimates would benefit downstream representations including 3D pose, video pose, and dense pose estimation.These intermediate representations build on 2D pose estimates in the image plane.
3. Method
PifPaf is a bottom-up, box-free pose estimator designed for low-resolution and crowded scenes. It uses composite PIF and PAF fields with grid-free localization, adaptive regression losses, and greedy decoding to form multi-person poses.
- Overall model: PifPaf uses a shared encoder with separate PIF and PAF heads to localize joints and associate them into poses.The decoder converts the predicted fields into pose estimates containing 17 joints.
- Part Intensity Fields: PIFs predict joint confidence, precise location, and size, then fuse these components into a high-resolution confidence map.The resulting localized map seeds pose generation and scores newly proposed joints.
- Part Association Fields: PAFs predict association confidence, two endpoint vectors, and two regression widths at every output location.For COCO, 19 connections link pairs of joint types, and the second endpoint may be selected by the ground-truth association rather than proximity alone.
- Part Association Fields: PAF endpoint regressions avoid grid discretization, helping distinguish nearby joints and separate annotations for close-by people.Floating-point origins provide spatial precision beyond feature-map cell centers.
- Adaptive Regression Loss: The method uses L1-type regression losses with scale dependence, including SmoothL1 and a Laplace loss attenuated by predicted spread.The Laplace loss is independent of person-instance area and keypoint-size estimates and is used for all vectorial components.
- Greedy Decoding: Greedy decoding seeds poses from high-confidence PIF locations and adds joints through PAF associations, making each accepted connection final.Association scores combine connection confidence, Laplace-calibrated distance, and high-resolution confidence at the proposed target.
4. Experiments
The experiments evaluate PifPaf on low-resolution COCO, nuScenes, Market-1501, and standard high-resolution COCO settings, including crowded and occluded scenes. PifPaf generally improves pose estimation quality over the compared methods while also showing an accuracy–runtime advantage.
- Baselines: PifPaf is compared with OpenPose and Mask R-CNN under matched low-resolution conditions without upscaling small images.Mask R-CNN was retrained for low resolution, and PifPaf uses a ResNet50 backbone in Table 1.
- COCO results: PifPaf outperforms OpenPose and Mask R-CNN on all reported low-resolution COCO metrics, leading by over 18% in AP.The qualitative results also show fewer false positives and detections of partially occluded pedestrians.
- Crowded-scene results: In crowded scenes, PifPaf estimates poses that Mask R-CNN misses when nearby pedestrians’ bounding boxes occlude people farther away.Figure 7 reports that PifPaf estimates all poses found by Mask R-CNN plus the additionally highlighted poses.
- Re-identification results: On Market-1501 pedestrian crops, the simplified accuracy is 96% for PifPaf versus 43% for Mask R-CNN.The same model trained on COCO produces qualitatively improved poses for the person re-identification setting.
- Additional comparisons: On high-resolution COCO test-dev, PifPaf performs on par with the best existing bottom-up method; adding the PIF field raises ResNet101 AP from 64.5% to 65.7%.For the same backbone, PifPaf also outperforms PersonLab by 9.5% in AP while achieving a 32% speedup.
5. Conclusions
PifPaf addresses transportation-domain pose-estimation failure modes, outperforming previous state-of-the-art methods at low resolution while matching them at high resolution. Its Part Association Fields may also transfer to other structured image tasks.
- PifPaf outperforms previous state-of-the-art methods in the low resolution regime and performs on par at high resolution.
- The method addresses failure modes prevalent in transportation settings such as self-driving cars and social robots.
- The proposed Part Association Fields can be applied to other tasks, including structured image concepts.