PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Delicate Local Representations for Multi-Person Pose Estimation

Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xiangyu Zhang, Xinyu Zhou, Erjin Zhou, Jian Sun

arXiv:2003.04030v3cs.CV

TL;DR

Human pose estimation needs precise keypoint localization, but prior methods mainly emphasize inter-level feature aggregation while intra-level fusion remains less explored. The paper proposes RSN for efficient intra-level fusion and PRM to balance local and global representations; it reports state-of-the-art results on COCO and MPII without extra training data or pretrained models.

  • Problem

    Prior pose-estimation methods mainly aggregate inter-level features, leaving same-resolution intra-level feature fusion less explored despite its relevance to precise keypoint localization.

  • Method

    RSN uses efficient intra-level feature fusion to learn delicate local representations, while PRM reweights output features to trade off local and global representations.

  • Results

    The approach outperforms state-of-the-art methods on both COCO and MPII without extra training data or pretrained models.

  • Takeaways & Limitations

    The method achieves a better trade-off between pose-estimation accuracy and inference speed than prior work, including HRNet comparisons under matched settings.

Abstract

from arXiv · show

In this paper, we propose a novel method called Residual Steps Network (RSN). RSN aggregates features with the same spatial size (Intra-level features) efficiently to obtain delicate local representations, which retain rich low-level spatial information and result in precise keypoint localization. Additionally, we observe the output features contribute differently to final performance. To tackle this problem, we propose an efficient attention mechanism - Pose Refine Machine (PRM) to make a trade-off between local and global representations in output features and further refine the keypoint locations. Our approach won the 1st place of COCO Keypoint Challenge 2019 and achieves state-of-the-art results on both COCO and MPII benchmarks, without using extra training data and pretrained model. Our single model achieves 78.6 on COCO test-dev, 93.0 on MPII test dataset. Ensembled models achieve 79.2 on COCO test-dev, 77.1 on COCO test-challenge dataset. The source code is publicly available for further research at https://github.com/caiyuanhao1998/RSN/

1 Introduction

Human pose estimation requires precise keypoint localization while balancing spatial and semantic information. The paper addresses underexplored intra-level feature fusion with RSN and PRM, and reports improved performance without extra data or pretrained models.

  • Multi-person pose estimation locates all persons’ keypoints in one image and requires both spatial information for localization and semantic information for classification.
  • Existing methods mainly aggregate inter-level features, while feature fusion among same-resolution features remains less explored in human pose estimation.
  • Intra-level fusion produces delicate local representations with smaller receptive-field gaps and more precise spatial information than inter-level fusion.
  • RSN uses Residual Steps Blocks with dense element-wise sum connections to efficiently fuse features within each level and retain low-level information.
  • PRM reweights output features to trade off local and global representations, further refining keypoint locations.
  • The approach outperforms state-of-the-art methods on COCO and MPII without extra training data or pretrained models.

2 Related Work

Human pose estimation methods are commonly categorized as top-down or bottom-up, while related feature-fusion work has emphasized inter-level aggregation. The paper situates intra-level fusion among approaches spanning multi-scale, dense, and attention-based representations.

  • Top-down methods detect each person before estimating poses, whereas bottom-up methods detect keypoints first and assemble them into individuals.
  • Hourglass, CPN, and HRNet aggregate features across spatial levels to combine spatial and semantic information.
  • Intra-level feature fusion is less explored in pose estimation despite applications in semantic segmentation and image classification.
  • Inception adds separately processed features, DenseNet uses continuous concatenation, and Res2Net and OSNet target multi-scale representations.
  • Dense connections can provide gradients and supervision for low-level features, supporting precise spatial information for keypoint localization.
  • Attention mechanisms are broadly divided into channel attention and spatial attention, with prior modules using pooling, convolutions, and activation functions.

3 Proposed Method

RSN learns delicate local representations through intra-level feature fusion, while PRM reweights mixed local and global features to refine pose predictions.

  • Residual Steps Network: RSN uses Residual Steps Blocks as its constituent units, replacing ResNet bottleneck blocks in a cascaded multi-stage architecture.
  • Residual Steps Network: Each RSB splits features into four branches, applies progressively more 3×3 convolutions, concatenates outputs, and adds an identity connection.The incremental convolutional depths motivate the name Residual Steps Network.
  • Delicate Local Representations: RSB combines deeply connected branches with receptive fields spanning multiple scales, enabling fusion of features with small receptive-field gaps.The maximum receptive field in RSB is 15, compared with a single receptive field in a ResNet bottleneck.
  • Receptive Field Analysis: RSN has a wider receptive-field scale range than ResNet, Res2Net, and OSNet, supporting features for joints with substantially different scales.The paper relates wider receptive-field ranges to both joint-specific feature extraction and keypoint classification.
  • Pose Refine Machine: PRM reweights final-stage RSN features through identity, channel-attention, and spatial-attention paths.The channel path produces weight vector α using global pooling and 1×1 convolutions; the spatial path produces attention map β using 1×1 and depth-wise separable 9×9 convolutions.
  • Pose Refine Machine: PRM addresses the unequal contributions of mixed low-level spatial and high-level semantic features by trading off local and global representations.

4 Experiments

Experiments evaluate the method on COCO using standard OKS-based AP, with specified training, augmentation, and heatmap-postprocessing procedures.

  • Datasets and Evaluation: COCO evaluation uses train2017 for training, minival for validation, and test-dev and test-challenge for testing.COCO train2017 contains about 57K images and 150K person instances; the evaluation sets include 5K, 20K, and 20K images respectively.
  • Datasets and Evaluation: The evaluation metric is standard OKS-based AP, with MegDet and MegDet-v2 used as human detectors on validation and test sets.
  • Training Details: Training uses Adam for 200 epochs with linear learning-rate decay from 5e-4 to 0 and weight decay of 1e-5.Training uses 8 V100 GPUs, batch size 48 per GPU, and augmentation including cropping, flipping, rotation, and scaling.
  • Testing Details: Testing applies Gaussian filtering, averages predictions from original and flipped images, and uses a quarter offset for keypoint localization.Pose scores multiply the average keypoint score by the bounding-box score.

Ablation Study of RSN Improvement.

Ablations compare RSN with alternative backbones and controlled baselines, showing stronger accuracy–complexity trade-offs from intra-level feature fusion.

  • Complexity Analysis: Figure 4 compares ResNet, Res2Net, DenseNet, and RSN as GFLOPs vary on COCO minival, isolating backbone efficiency without PRM.
  • Ablation Setup: The study tests whether intra-level fusion matters by removing vertical fusion connections and comparing against a conventional branching baseline at matched GFLOPs.
  • Ablation Results: 2.9 AP higher: RSN-18 versus ResNet-18, with only 0.2 GFLOPs added.RSN-18 is also 2.3 AP higher than Res2Net-18 with only 0.3 GFLOPs added.
  • Ablation Results: 1 AP higher: RSN-50 versus Baseline1-50 at the same GFLOPs; RSN-50 is 2 AP higher than Baseline2-50.
  • Ablation Results: RSN-18 reaches similar AP to ResNet-101 and Res2Net-101 with only a third of their computation cost.

Ablation Study of RSN Efficiency.

RSN improves feature utilization and spatial precision by efficiently fusing intra-level features, outperforming DenseNet and Res2Net in the reported analyses.

  • DenseNet retains delicate spatial texture through frequent intra-level aggregation, but concatenation becomes redundant as computation increases.
  • RSN uses most channels more effectively than Res2Net and DenseNet, whose filter weights are generally smaller or near zero.
  • RSN’s dense element-wise sum connections reduce redundant low-utilization features while preserving efficient intra-level fusion.
  • Compared with Res2Net, RSN combines denser connections and wider receptive fields, producing more fully used weights and considerable improvement.
  • RSN produces more complete feature responses, clearer heatmaps, and more accurate keypoint predictions than Res2Net and DenseNet.
  • Four branches provide the best trade-off between branch representation capacity and intra-level fusion in the reported RSN ablation.

Ablation Study of RSN Architecture.

The architecture study examines branch count in RSN and compares PRM with alternative attention mechanisms through ablation experiments.

  • Pose Refine Machine: PRM is evaluated against SE-block and CBAM on single-stage and multi-stage architectures using COCO minival ablations.

Ablation Study of Pose Refine Machine.

RSN achieves strong COCO results without extra training data or pretrained backbones, while offering a favorable accuracy–speed trade-off against prior methods.

  • COCO results: 78.0 AP on COCO test-dev surpasses HRNet by 2.5 AP and SimpleBaseline by 4.3 AP at comparable GFLOPs.
  • COCO results: 0.7 AP improvement over MSPN is reported on the COCO test-challenge set without using a pretrained model.
  • Efficiency: RSN is reported to be much faster than HRNet while achieving comparable performance on GPU and CPU platforms.

Inference Speed.

The detector choice affects measured COCO performance, with MegDet-v2 yielding higher AP than MegDet at both tested input sizes.

  • 77.3 AP with MegDet versus 78.0 AP with MegDet-v2 is reported for 4×RSN-50 at 256×192 input size.
  • 77.9 AP with MegDet versus 78.6 AP with MegDet-v2 is reported for 4×RSN-50 at 384×288 input size.

4.2 MPII Human Pose Estimation

On MPII, RSN improves state-of-the-art single-person pose estimation performance, supporting its generalization beyond the paper’s primary multi-person setting.

  • 0.7 PCKh@0.5 improvement over state of the art demonstrates RSN’s superiority and generalization ability on MPII.
  • MPII provides a single-person pose estimation benchmark for evaluating RSN.

5 Conclusion

RSN combines efficient intra-level feature fusion with PRM to balance local and global output representations. The method achieves the best results on COCO and MPII, with prediction examples visualized in Figure 7.

  • RSN learns delicate local representations through efficient intra-level feature fusion.
  • Pose Refine Machine balances local and global representations in output features.
  • RSN achieves the best results on both COCO and MPII benchmarks.
  • Figure 7 visualizes prediction results on the COCO and MPII validation sets.
Loading 2003.04030v3…