PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network

Muhammed Kocabas, Salih Karagoz, Emre Akbas

arXiv:1807.04067v1cs.CV

TL;DR

Multi-person 2D pose estimation requires accurate joint grouping while balancing the speed of bottom-up methods against the accuracy of top-down methods. MultiPoseNet addresses this with a multi-task architecture and a Pose Residual Network that assigns keypoints to detected people. On COCO, it achieves state-of-the-art bottom-up performance, comparable top-down performance, and the fastest inference time reported by the paper.

  • Problem

    Multi-person pose estimation must detect joints and assign them to person instances, while bottom-up methods have typically been less accurate than top-down methods.

  • Method

    MultiPoseNet combines shared feature extraction, person and keypoint detection and segmentation subnetworks, and a Pose Residual Network that learns keypoint-to-person assignment.

  • Results

    On COCO, the method outperforms previous bottom-up methods, achieves a 4-point mAP increase over the previous best result, and performs on par with the best top-down methods while being an order of magnitude faster.

  • Takeaways & Limitations

    The paper reports state-of-the-art bottom-up performance, comparable top-down performance, fastest inference time, and approximately 23 frames/sec on typical COCO images.

  • Takeaways & Limitations

    Comparisons using PRN with keypoints from other methods are considered unfair because PRN was trained with MultiPoseNet's detection architecture.

Abstract

from arXiv · show

In this paper, we present MultiPoseNet, a novel bottom-up multi-person pose estimation architecture that combines a multi-task model with a novel assignment method. MultiPoseNet can jointly handle person detection, keypoint detection, person segmentation and pose estimation problems. The novel assignment method is implemented by the Pose Residual Network (PRN) which receives keypoint and person detections, and produces accurate poses by assigning keypoints to person instances. On the COCO keypoints dataset, our pose estimation method outperforms all previous bottom-up methods both in accuracy (+4-point mAP over previous best result) and speed; it also performs on par with the best top-down methods while being at least 4x faster. Our method is the fastest real time system with 23 frames/sec. Source code is available at: https://github.com/mkocabas/pose-residual-network

1 Introduction

MultiPoseNet is a bottom-up, multi-task architecture for multi-person 2D pose estimation that uses a learned assignment method to construct poses from detections. On COCO, it improves bottom-up accuracy, matches leading top-down methods, and operates in real time.

  • Problem: Multi-person 2D pose estimation requires detecting joints, grouping them into person instances, or detecting people and assigning joints.Bottom-up methods detect and group joints, whereas top-down methods detect people before estimating each pose.
  • Multi-task architecture: MultiPoseNet jointly performs person detection, keypoint detection, person segmentation, and pose estimation through a shared backbone and task-specific subnetworks.Its final network assigns detected keypoints to person instances.
  • Pose assignment: The Pose Residual Network assigns detected keypoints to person boxes using a learned function that considers all joints together in one shot.This assignment step addresses ambiguity when overlapping people place unrelated keypoints inside one bounding box.
  • Results: 4-point mAP increase over the previous best result is reported on COCO, where MultiPoseNet outperforms previous bottom-up methods without external data.The result is reported for the pose estimation method on the COCO dataset.
  • Results: An order of magnitude faster than the best performing top-down methods, MultiPoseNet performs on par with them.The paper also reports approximately 23 FPS for a typical COCO image containing about three people.
  • Results: The architecture reaches approximately 23 frames/sec in real time and can be extended to related tasks such as person segmentation.The reported speed varies from 27 FPS for one person detection to 15 FPS for 20 detections.

2 Related Work

Prior multi-person pose estimation methods divide into bottom-up and top-down approaches, with different speed, localization, and assignment trade-offs. Related work uses CNNs, graphical models, affinity fields, embeddings, and clustering to detect and group body parts.

  • Single-person estimation: Single-person pose estimation predicts individual body parts from a cropped person image or known location and scale.Earlier methods used HOG features and probabilistic graphical models to represent pose structure.
  • Single-person estimation: Deep neural networks became dominant in pose estimation after direct coordinate regression, with later methods combining deep features, graphical models, and iterative refinement.Hourglass architectures use convolution-deconvolution structures with residual connections and stacked refinement blocks.
  • Multi-person approaches: Bottom-up methods detect body joints and assign them to people, making them faster and smaller than top-down methods but less able to zoom into individual instances.This trade-off creates an accuracy gap between bottom-up and top-down approaches.
  • Related multi-task work: Earlier work also jointly predicted human part segmentations and locations or jointly detected people and keypoints.These approaches illustrate efforts to combine related person and pose predictions.
  • Bottom-up methods: Bottom-up systems use assignment mechanisms including correlation clustering, part affinity fields with Hungarian matching, and associative embeddings.These methods detect parts or relationships and then form individual skeletons.
  • Top-down methods: Top-down methods detect people first and run a single-person pose estimator for each instance, enabling fine localization but making inference extremely slow.Each pose estimator can focus on one detected person instance.

3 The Method and Models

MultiPoseNet combines shared feature extraction, detection subnets, and a Pose Residual Network to perform multi-person pose estimation with keypoint grouping. Its PRN resolves overlapping-person ambiguities by learning pose structures, while the architecture jointly supports detection, segmentation, and keypoint estimation.

  • Architecture: MultiPoseNet uses a ResNet backbone with separate feature pyramids for keypoint and person detection subnets.The backbone extracts features for both tasks, while the FPNs provide multi-scale representations.
  • Keypoint Estimation Subnet: The keypoint subnet outputs Gaussian keypoint heatmaps and a person segmentation mask from hierarchical CNN features.Each keypoint layer can contain multiple peaks corresponding to different person instances.
  • Pose Residual Network: Overlapping person boxes create assignment ambiguity because a box may contain keypoints belonging to multiple people; PRN learns pose structures to resolve it.The method assigns detected keypoints to person boxes using a learned mapping from input heatmaps X to corrected outputs Y.
  • Pose Residual Network: PRN was compared with Max, Unary Conditional Relationship, and a non-residual model as alternative grouping strategies.Max selects the peak in each keypoint channel, while UCR multiplies each channel by a learned joint-location prior.
  • Pose Residual Network: The PRN generalizes lower-order grouping models by learning richer relations than methods modeling only pairwise or unary channels.The paper characterizes prior bottom-up methods as lower-order grouping models relative to PRN.
  • Pose Residual Network: PRN crops and resizes keypoint heatmaps within each detected person box, then outputs heatmaps containing one selected keypoint per channel.The fixed RoI size enables PRN to process person detections with arbitrary sizes and shapes.

4 Experiments

Experiments evaluate MultiPoseNet on COCO for keypoint and person detection, and on PASCAL VOC for person segmentation, with ablations on COCO validation data. Test-dev results use augmentation and ensembling, while the method achieves a 4-point mAP gain over the previous best bottom-up method and averages 23 FPS.

  • COCO keypoint and person detection experiments used no external data, while person segmentation was evaluated on PASCAL VOC 2012 because COCO lacked semantic-segmentation annotations.
  • COCO ablations used val2017, comparisons used the 20K-image test-dev2017 split, and evaluation followed the official COCO average-precision metric.
  • Precision-recall curves were reported on the COCO validation set for all, large, and medium person scales.
  • 4-point mAP increase over the previous best bottom-up method, with performance exceeding all bottom-up and most top-down methods on COCO test-dev.Results used multi-scale and multi-crop test-time augmentation, two backbones, a refinement network, ensembling, and OKS-based NMS.
  • 23 FPS average runtime was reported alongside the accuracy results, supporting a reasonable speed-accuracy tradeoff.

Different Backbones

The study compares shared-backbone choices and reports accuracy gains from deeper or dilated features, while noting that dilation increases computational complexity and can hinder real-time performance.

  • Different Backbones: The backbone comparison is part of COCO test-dev evaluation excluding systems trained with external data.
  • Different Backbones: Keypoint models use 480 × 480 inputs and output 120 × 120 × (K + 1) heatmaps, with K = 17 for COCO.Lower resolutions harmed mAP, while higher resolutions increased training and inference complexity.

Different Keypoint Architectures

Keypoint architecture ablations show that intermediate supervision and multi-level feature concatenation improve precision and mAP within the dense prediction model.

  • Different Keypoint Architectures: Intermediate supervision enhanced precision significantly by refining representations across feature hierarchies.The intermediate loss is appended to the outputs of K blocks.
  • Different Keypoint Architectures: 2 mAP gain came from applying a final loss to concatenated D features downsized from K features.This stage combines multi-level features, compresses them into a uniform space, and extracts more semantic features.
  • Different Keypoint Architectures: PRN is a simple assignment strategy designed for faster inference while providing reasonable accuracy.

Pose Residual Network Design

PRN design experiments compare model configurations and grouping methods, showing strong performance with a residual MLP and improved handling of ambiguities and non-grouped keypoints.

  • Pose Residual Network Design: The PRN ablation compares Max, UCR, and PRN implementations, including variants using ground-truth detections.
  • Pose Residual Network Design: 89.4 mAP was achieved on ground-truth inputs using a 1024-node MLP, 0.5 dropout, and a residual input-output connection.
  • Pose Residual Network Design: PRN improved results by approximately 20 mAP over naive assignment techniques by learning pose structures that overcome ambiguities.Lower-order grouping methods performed poorly on overlapping detections.
  • Pose Residual Network Design: With ground-truth keypoints and detected person boxes, the system reached 75.3 mAP; with detected keypoints and ground-truth person boxes, it reached 65.1 mAP.Both ground-truth detections produced 89.4 mAP, indicating substantial room for keypoint-localization improvement.
  • Pose Residual Network Design: PRN improved detection results by approximately 1 mAP when assigning keypoints from previous bottom-up methods.The comparison used MultiPoseNet person boxes and keypoints from the official implementations of those methods.

4.3 Person Detection

The person detection subnet was trained on COCO person instances and evaluated with different backbones against the original adopted methods.

  • The person detection subnet was trained only on COCO person instances while the backbone was frozen with keypoint detection parameters.
  • Both ResNet-50 and ResNet-101 backbones outperformed the original implementations used in the architecture.
  • Table 5 reports person detection results on the COCO dataset.

4.4 Person Segmentation

Person segmentation was jointly trained with keypoint prediction as a lightweight added layer and achieved competitive results on PASCAL VOC 2012.

  • The segmentation output is an additional layer appended to the keypoint outputs and jointly trained with the keypoint task.Person masks were combined into a single binary ground-truth mask layer.
  • The added segmentation layer introduces very small model complexity.
  • The system outperformed most successful baselines, including SegNet and Deeplab-v2, and was comparable to Deeplab v3.Evaluation used the PASCAL VOC 2012 test set and PASCAL IoU metric.
  • The multi-task model handled person segmentation alongside other tasks with competitive performance.

4.5 Runtime Analysis

Runtime is dominated by backbone feature extraction, while the detection subnets and PRN remain lightweight. A shallow ResNet-50 backbone enables real-time inference.

  • Figures 6, 7, and 8 present qualitative results, parameter counts by block, and runtime with respect to the number of people.
  • Most parameters are concentrated in the backbone, while the subnets and PRN are relatively lightweight.The system comprises a backbone, keypoint and person detection subnets, and the pose residual network.
  • Most computation time is spent on feature extraction rather than the subnets or PRN.
  • ResNet-50 with 384×576 inputs was evaluated on images containing 1 to 20 people by averaging inference time over 1000 images.
  • A shallow feature extractor such as ResNet-50 provides real-time performance.

5 Conclusion

The paper concludes that PRN accurately assigns keypoints within a multi-task MultiPoseNet architecture while supporting strong pose performance, fast inference, and related outputs.

  • Pose Residual Network accurately assigns keypoints to person detections produced by MultiPoseNet.
  • The pose estimation method achieved state-of-the-art performance among bottom-up methods and comparable results with top-down methods.
  • The method had the fastest inference time compared with previous methods.
  • The multi-task model jointly produced keypoints, person bounding boxes, and person segmentation results.
Loading 1807.04067v1…