PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking on Multi-Stage Networks for Human Pose Estimation

Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, Jian Sun

arXiv:1901.00148v4cs.CV

TL;DR

Multi-stage pose estimation appears well suited to the task but has underperformed single-stage methods, leaving the architecture's value unclear. The paper addresses this issue with MSPN, improving its single-stage modules, cross-stage feature flow, and supervision; it reports state-of-the-art results on COCO and MPII. These results support the effectiveness of a carefully designed multi-stage architecture.

  • Problem

    Multi-stage methods seem suited to pose estimation but underperform single-stage methods in practice, especially on COCO, leaving their relative value unclear.

  • Method

    MSPN combines a stronger single-stage module with cross-stage feature aggregation and coarse-to-fine supervision.

  • Results

    The method achieves state-of-the-art results on COCO and MPII, including 78.1 AP on COCO test-dev and 92.6 PCKh@0.5 on MPII test.

  • Takeaways & Limitations

    The reported results support the effectiveness of multi-stage pose estimation when its architectural and training design choices are improved.

Abstract

from arXiv · show

Existing pose estimation approaches fall into two categories: single-stage and multi-stage methods. While multi-stage methods are seemingly more suited for the task, their performance in current practice is not as good as single-stage methods. This work studies this issue. We argue that the current multi-stage methods' unsatisfactory performance comes from the insufficiency in various design choices. We propose several improvements, including the single-stage module design, cross stage feature aggregation, and coarse-to-fine supervision. The resulting method establishes the new state-of-the-art on both MS COCO and MPII Human Pose dataset, justifying the effectiveness of a multi-stage architecture. The source code is publicly available for further research.

1. Introduction

Human pose estimation uses single-stage and multi-stage architectures, but multi-stage methods have underperformed on COCO despite their apparent suitability. This work attributes that gap to design choices and introduces MSPN improvements that achieve state-of-the-art results on COCO and MPII.

  • Motivation: Single-stage pose methods currently perform best in practice, while multi-stage methods remain competitive on MPII but lag on the more challenging COCO dataset.The comparison leaves the relative advantage of multi-stage architecture unresolved before this work.
  • Motivation: The work argues that multi-stage underperformance primarily reflects insufficient architectural and training design choices rather than the architecture itself.The study aims to exploit the potential advantage of multi-stage networks through targeted improvements.
  • Contributions: MSPN improves the single-stage module, aggregates features across adjacent stages, and applies coarse-to-fine supervision to refine localization.The supervision differs from commonly used multi-scale supervision by matching progressively refined pose localization.
  • Results: With similar FLOPs, MSPN improves from 74.5 AP at 2 stages to 75.9 AP at 4 stages, whereas Hourglass improves only from 71.3 AP at 4 stages to 71.6 at 8 stages.The comparison supports a better accuracy-FLOPs tradeoff for MSPN than for the representative multi-stage baseline.
  • Results: 76.1 AP is achieved by the proposed single model on COCO test-dev, while MPII reaches 92.6 PCKh@0.5, reported as the best performance on that test dataset.The work also reports 78.1 AP on COCO test-dev and 76.4 AP on the test-challenge dataset.

2. Related Work

Prior human pose estimation work is organized around single-stage and multi-stage approaches, with deep convolutional networks replacing earlier handcrafted-feature methods. Multi-stage systems progressively refine pose estimates, while this work modifies existing multi-stage designs to argue for their effectiveness.

  • Background: Deep convolutional neural networks have substantially improved pose estimation, whose methods are commonly divided into single-stage and multi-stage categories.Earlier approaches widely used handcrafted features.
  • Single-Stage Approach: Single-stage methods use image-classification backbones such as VGG or ResNet and include systems for heatmaps, person proposals, keypoint refinement, and hard-keypoint mining.Examples include Mask R-CNN, Cascade Pyramid Network, and a simple baseline.
  • Multi-Stage Approach: Multi-stage methods aim to produce increasingly refined estimates and may be bottom-up or top-down.Bottom-up systems associate predicted joints into human instances, whereas top-down systems detect people before estimating each person's pose.
  • Multi-Stage Approach: Hourglass uses repeated downsampling and upsampling with intermediate supervision, and later work builds variants around this multi-stage design.These methods are representative examples of multi-stage pose estimation.
  • This Work: This work modifies existing multi-stage architecture and reports that the resulting multi-stage approach is better than prior single-stage-based methods.The paper positions this result against COCO 2017 winner and simple-baseline systems, which are single-stage based.

3. Multi-Stage Pose Network

MSPN improves multi-stage pose estimation through a stronger single-stage module, cross-stage feature aggregation, and coarse-to-fine supervision. These changes address information loss and progressively refine localization.

  • 3.1. Analysis of a Single-Stage Module: The network doubles feature maps after each spatial downsampling and allocates more computation to downsampling, obtaining 74.5 AP with 9.6G FLOPs at two stages.The design retains 256 channels during upsampling while increasing capacity during downsampling.
  • 3.1. Analysis of a Single-Stage Module: MSPN uses a ResNet-based GlobalNet of CPN as its single-stage module within a multi-stage network.The U-shaped module uses features from multiple scales for predictions, and its use in a multi-stage setting is presented as effective.
  • 3.2. Cross Stage Feature Aggregation: Cross-stage feature aggregation propagates multi-scale features from earlier stages to the current stage to reduce information loss.For each scale, separate flows from previous-stage downsampling and upsampling units are transformed with 1 × 1 convolutions before fusion.
  • 3.3. Coarse-to-fine Supervision: Coarse-to-fine supervision uses larger ground-truth kernels in early stages and smaller kernels later to progressively refine localization accuracy.The supervision matches the observed coarse-to-fine behavior of estimated heat maps across stages.
  • 3.3. Coarse-to-fine Supervision: MSPN applies multi-scale intermediate supervision at four scales in each stage, with OHKM applied to the largest-scale supervision.The multi-scale supervision provides contextual information at different levels for locating challenging poses.

4.1. Dataset and Evaluation Protocol

The experiments evaluate the framework on MS COCO and MPII Human Pose using their standard pose-estimation splits and metrics.

  • MS COCO: MS COCO evaluation combines the train and validation data into trainval and uses a separate 5k-image minival split for evaluation.The trainval set contains nearly 57K images and 150K person instances.
  • MS COCO: MS COCO performance is measured with OKS-based mean average precision, abbreviated as AP.
  • MPII Human Pose: MPII Human Pose contains around 25k images and over 40k annotated person instances, with 12k instances reserved for testing.Single-person pose estimation is evaluated using PCKh@0.5.

4.2. Implementation Details

Implementation uses detected human proposals followed by MSPN pose estimation, with specified optimization, augmentation, and test-time post-processing procedures.

  • Human Detector: MegDet generates human proposals, and the top 100 boxes across categories are filtered to retain human boxes for pose estimation.COCO boxes are expanded to a fixed 4:3 aspect ratio.
  • Training: Training uses eight Nvidia GTX 1080Ti GPUs, batch size 32 per GPU, 90k iterations, Adam, and a learning rate decayed from 5e-4 to 0.Weight decay is set to 1e-5.
  • Training: Training augmentation randomly applies cropping, flipping, rotation, and scaling, with rotation from −45° to 45° and scaling from 0.7 to 1.35.Input sizes are 256×192 or 384×288 for COCO and 256×256 for MPII.
  • Testing: Testing filters predicted heat maps, averages original and flipped-image predictions, applies a quarter offset, and scores poses by box and keypoint scores.

4.3. Ablation Study

The ablation study evaluates backbone capacity, multi-stage depth, smaller modules, feature aggregation, supervision, and detector settings on COCO minival. Results show that MSPN benefits from improved module design and complementary training and aggregation strategies.

  • Backbone Capacity: Single-stage performance saturates as backbone capacity grows, making Res-152 or larger backbones ineffective.Res-101 improves over Res-50 by 1.6 AP, but Res-152 adds only 0.5 AP and Res-254 adds 0.4 AP over Res-152.
  • Multi-Stage Architecture: MSPN gains 3.0 AP when expanding from one to two stages and another 1.4 AP from adding the third and fourth stages.MSPN reaches 74.5 AP at two stages, while adding the third and fourth stages maintains a strong upward trend.
  • Smaller Modules: A 2-stage Res-18 MSPN matches Res-50 with fewer FLOPs, while 4-stage S-XCP exceeds the large single-stage model by 1.0 AP at similar complexity.These comparisons support using smaller single-stage modules within deeper multi-stage networks.
  • Cross Stage Feature Aggregation: Cross-stage feature aggregation improves MSPN from 74.2 to 74.5 AP and improves Hourglass by 0.5 AP.The strategy fuses different-level features across adjacent stages to mitigate information loss from repeated up- and down-sampling.
  • Coarse-to-Fine Supervision: Coarse-to-fine supervision raises MSPN from 73.3 to 74.2 AP and improves a comparable 4-stage Hourglass by 1.2 AP.The strategy uses progressively finer supervision to improve keypoint localization and transfers to another multi-stage architecture.

4.4. Influence of Human Detector

The study tests MSPN with detectors of different quality and finds that detector performance has only a limited effect on pose-estimation accuracy. Most of the observed performance is attributed to MSPN.

  • Detector Settings: A 49.4-AP human detector is compared with a 41.1-AP detector and an oracle detector using ground-truth boxes.The comparisons are reported for pose estimation on COCO minival.
  • Influence on Accuracy: Only 0.5 AP separates the 41.1-AP and 49.4-AP detectors when used with 4-stage MSPN.The authors state that detector influence is quite limited and that good performance mostly comes from MSPN.

4.5. Comparison with State-of-the-art Methods

MSPN achieves state-of-the-art results on both COCO and MPII, including strong single-model performance and further gains from external data and ensembling. The method also handles crowded, occluded, and challenging poses effectively.

  • COCO Test-Dev: 76.1 AP is achieved by a single COCO-only model on test-dev, outperforming other methods across all reported metrics.External data raises performance to 77.1 AP, and an ensemble reaches 78.1 AP.
  • COCO Test-Challenge: 76.4 AP is obtained on the COCO test-challenge dataset, exceeding the COCO 2017 Challenge winner by 4.3 AP.The method also exceeds Sample Baseline by 1.9 AP on this benchmark.
  • Qualitative Results: MSPN handles crowd, occlusion, and challenging poses effectively in qualitative results.These examples are visualized on the COCO minival dataset.
  • Conclusion: The study shows that a multi-stage architecture can be competitive on the challenging COCO dataset.This conclusion follows the reported state-of-the-art comparisons and extensive experiments.

5. Conclusion

The conclusion introduces MSPN for multi-person pose estimation and frames its approach around a multi-stage pipeline with improved single-stage modules.

  • Conclusion: MSPN is proposed as a multi-stage network for multi-person pose estimation.The conclusion states that the work first verifies the effectiveness of the multi-stage pipeline.
Loading 1901.00148v4…