PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Delicate Local Representations for Multi-Person Pose Estimation

Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xiangyu Zhang, Xinyu Zhou, Erjin Zhou, Jian Sun

arXiv:2003.04030v3cs.CV

TL;DR

Human pose estimation needs precise keypoint localization, but prior methods mainly emphasize inter-level feature aggregation while intra-level fusion remains less explored. The paper proposes RSN for efficient intra-level fusion and PRM to balance local and global representations; it reports state-of-the-art results on COCO and MPII without extra training data or pretrained models.

  • Problem

    Prior pose-estimation methods mainly aggregate inter-level features, leaving same-resolution intra-level feature fusion less explored despite its relevance to precise keypoint localization.

    Source: §1 · p. 2

  • Method

    RSN uses efficient intra-level feature fusion to learn delicate local representations, while PRM reweights output features to trade off local and global representations.

    Source: §1 · pp. 2–3, §3.1 · p. 5, §3.3 · p. 6

  • Results

    The approach outperforms state-of-the-art methods on both COCO and MPII without extra training data or pretrained models.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The method achieves a better trade-off between pose-estimation accuracy and inference speed than prior work, including HRNet comparisons under matched settings.

    Source: Inference Speed · p. 13

Abstract

from arXiv · show

In this paper, we propose a novel method called Residual Steps Network (RSN). RSN aggregates features with the same spatial size (Intra-level features) efficiently to obtain delicate local representations, which retain rich low-level spatial information and result in precise keypoint localization. Additionally, we observe the output features contribute differently to final performance. To tackle this problem, we propose an efficient attention mechanism - Pose Refine Machine (PRM) to make a trade-off between local and global representations in output features and further refine the keypoint locations. Our approach won the 1st place of COCO Keypoint Challenge 2019 and achieves state-of-the-art results on both COCO and MPII benchmarks, without using extra training data and pretrained model. Our single model achieves 78.6 on COCO test-dev, 93.0 on MPII test dataset. Ensembled models achieve 79.2 on COCO test-dev, 77.1 on COCO test-challenge dataset. The source code is publicly available for further research at https://github.com/caiyuanhao1998/RSN/

1 Introduction

Human pose estimation requires precise keypoint localization while balancing spatial and semantic information. The paper addresses underexplored intra-level feature fusion with RSN and PRM, and reports improved performance without extra data or pretrained models.

  • Multi-person pose estimation locates all persons’ keypoints in one image and requires both spatial information for localization and semantic information for classification.Source: §1 · p. 1
  • Existing methods mainly aggregate inter-level features, while feature fusion among same-resolution features remains less explored in human pose estimation.Source: §1 · p. 2
  • Intra-level fusion produces delicate local representations with smaller receptive-field gaps and more precise spatial information than inter-level fusion.Source: §1 · p. 2
  • RSN uses Residual Steps Blocks with dense element-wise sum connections to efficiently fuse features within each level and retain low-level information.Source: §1 · pp. 2–3
  • PRM reweights output features to trade off local and global representations, further refining keypoint locations.Source: §1 · p. 3
  • The approach outperforms state-of-the-art methods on COCO and MPII without extra training data or pretrained models.Source: §1 · p. 3

2 Related Work

Human pose estimation methods are commonly categorized as top-down or bottom-up, while related feature-fusion work has emphasized inter-level aggregation. The paper situates intra-level fusion among approaches spanning multi-scale, dense, and attention-based representations.

  • Top-down methods detect each person before estimating poses, whereas bottom-up methods detect keypoints first and assemble them into individuals.Source: §2.1 · p. 3
  • Hourglass, CPN, and HRNet aggregate features across spatial levels to combine spatial and semantic information.Source: §2.2 · pp. 3–4
  • Intra-level feature fusion is less explored in pose estimation despite applications in semantic segmentation and image classification.Source: §2.2 · p. 4
  • Inception adds separately processed features, DenseNet uses continuous concatenation, and Res2Net and OSNet target multi-scale representations.Source: §2.2 · p. 4
  • Dense connections can provide gradients and supervision for low-level features, supporting precise spatial information for keypoint localization.Source: §2.2 · p. 4
  • Attention mechanisms are broadly divided into channel attention and spatial attention, with prior modules using pooling, convolutions, and activation functions.Source: §2.3 · p. 5

3 Proposed Method

RSN learns delicate local representations through intra-level feature fusion, while PRM reweights mixed local and global features to refine pose predictions.

Residual Steps Network

  • RSN uses Residual Steps Blocks as its constituent units, replacing ResNet bottleneck blocks in a cascaded multi-stage architecture.Source: §3 · p. 5
  • Each RSB splits features into four branches, applies progressively more 3×3 convolutions, concatenates outputs, and adds an identity connection.The incremental convolutional depths motivate the name Residual Steps Network.Source: §3.1 · p. 5

Delicate Local Representations

  • RSB combines deeply connected branches with receptive fields spanning multiple scales, enabling fusion of features with small receptive-field gaps.The maximum receptive field in RSB is 15, compared with a single receptive field in a ResNet bottleneck.Source: §3.1 · p. 5

Receptive Field Analysis

  • RSN has a wider receptive-field scale range than ResNet, Res2Net, and OSNet, supporting features for joints with substantially different scales.The paper relates wider receptive-field ranges to both joint-specific feature extraction and keypoint classification.Source: §3.2 · p. 6

Pose Refine Machine

  • PRM reweights final-stage RSN features through identity, channel-attention, and spatial-attention paths.The channel path produces weight vector α using global pooling and 1×1 convolutions; the spatial path produces attention map β using 1×1 and depth-wise separable 9×9 convolutions.Source: §3.3 · p. 6, Figure 3 · p. 7
  • PRM addresses the unequal contributions of mixed low-level spatial and high-level semantic features by trading off local and global representations.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate the method on COCO using standard OKS-based AP, with specified training, augmentation, and heatmap-postprocessing procedures.

Datasets and Evaluation

  • COCO evaluation uses train2017 for training, minival for validation, and test-dev and test-challenge for testing.COCO train2017 contains about 57K images and 150K person instances; the evaluation sets include 5K, 20K, and 20K images respectively.Source: §4.1 · pp. 7–8
  • The evaluation metric is standard OKS-based AP, with MegDet and MegDet-v2 used as human detectors on validation and test sets.Source: §4.1 · p. 8

Training Details

  • Training uses Adam for 200 epochs with linear learning-rate decay from 5e-4 to 0 and weight decay of 1e-5.Training uses 8 V100 GPUs, batch size 48 per GPU, and augmentation including cropping, flipping, rotation, and scaling.Source: §4.1 · p. 8

Testing Details

  • Testing applies Gaussian filtering, averages predictions from original and flipped images, and uses a quarter offset for keypoint localization.Pose scores multiply the average keypoint score by the bounding-box score.Source: §4.1 · p. 8

Ablation Study of RSN Improvement.

Ablations compare RSN with alternative backbones and controlled baselines, showing stronger accuracy–complexity trade-offs from intra-level feature fusion.

Complexity Analysis

Ablation Setup

  • The study tests whether intra-level fusion matters by removing vertical fusion connections and comparing against a conventional branching baseline at matched GFLOPs.Source: Ablation Study of RSN Improvement · p. 9

Ablation Results

Ablation Study of RSN Efficiency.

RSN improves feature utilization and spatial precision by efficiently fusing intra-level features, outperforming DenseNet and Res2Net in the reported analyses.

Ablation Study of RSN Architecture.

The architecture study examines branch count in RSN and compares PRM with alternative attention mechanisms through ablation experiments.

Pose Refine Machine

Ablation Study of Pose Refine Machine.

RSN achieves strong COCO results without extra training data or pretrained backbones, while offering a favorable accuracy–speed trade-off against prior methods.

COCO results

Efficiency

  • RSN is reported to be much faster than HRNet while achieving comparable performance on GPU and CPU platforms.Source: Inference Speed · p. 13

Inference Speed.

The detector choice affects measured COCO performance, with MegDet-v2 yielding higher AP than MegDet at both tested input sizes.

4.2 MPII Human Pose Estimation

On MPII, RSN improves state-of-the-art single-person pose estimation performance, supporting its generalization beyond the paper’s primary multi-person setting.

  • 0.7 PCKh@0.5 improvement over state of the art demonstrates RSN’s superiority and generalization ability on MPII.Source: §4.2 · p. 13
  • MPII provides a single-person pose estimation benchmark for evaluating RSN.Source: §4.2 · p. 13

5 Conclusion

RSN combines efficient intra-level feature fusion with PRM to balance local and global output representations. The method achieves the best results on COCO and MPII, with prediction examples visualized in Figure 7.

  • RSN learns delicate local representations through efficient intra-level feature fusion.Source: §5 · p. 14
  • Pose Refine Machine balances local and global representations in output features.Source: §5 · p. 14
  • RSN achieves the best results on both COCO and MPII benchmarks.Source: §5 · p. 14
  • Figure 7 visualizes prediction results on the COCO and MPII validation sets.Source: Figure 7 · p. 14

Related papers

Loading 2003.04030v3…