PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep High-Resolution Representation Learning for Human Pose Estimation

Ke Sun, Bin Xiao, Dong Liu, Jingdong Wang

arXiv:1902.09212v1cs.CV

TL;DR

Single-person human pose estimation requires accurate localization of anatomical keypoints, but existing approaches often recover high-resolution representations from low-resolution ones. HRNet maintains high-resolution representations throughout with repeated multi-scale fusion, achieving superior keypoint detection on COCO and MPII and superior video pose tracking on PoseTrack.

  • Problem

    Single-person human pose estimation seeks accurate localization of anatomical keypoints and underpins related pose-estimation problems.

  • Method

    HRNet maintains high-resolution representations throughout by connecting parallel multi-resolution subnetworks and repeatedly fusing information across scales.

  • Results

    The network achieves superior keypoint detection performance on COCO and MPII and superior video pose tracking performance on PoseTrack.

  • Takeaways & Limitations

    Maintaining resolution and repeatedly fusing multi-resolution representations yields accurate, spatially precise keypoint heatmaps.

  • Takeaways & Limitations

    Applications to other dense prediction tasks and less lightweight multi-resolution aggregation remain future work.

Abstract

from arXiv · show

This is an official pytorch implementation of Deep High-Resolution Representation Learning for Human Pose Estimation. In this work, we are interested in the human pose estimation problem with a focus on learning reliable high-resolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutli-resolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich high-resolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection dataset and the MPII Human Pose dataset. The code and models have been publicly available at \url{https://github.com/leoxiaobin/deep-high-resolution-net.pytorch}.

1. Introduction

The paper introduces High-Resolution Net (HRNet), which maintains high-resolution representations throughout pose estimation by connecting parallel multi-resolution subnetworks with repeated information exchange. It reports superior performance on COCO and MPII keypoint detection and PoseTrack video pose tracking.

  • Architecture: HRNet maintains high-resolution representations throughout the network by progressively adding high-to-low resolution subnetworks and connecting them in parallel.This replaces the usual series connection followed by resolution recovery.
  • Architecture: Repeated multi-scale fusions exchange information across parallel multi-resolution subnetworks over and over.The architecture uses repeated information exchange across different feature-map resolutions.
  • Advantages: Compared with existing pose-estimation networks, HRNet’s parallel design avoids recovering resolution through a low-to-high process, potentially producing more spatially precise predicted heatmaps.The comparison is made against widely used networks that connect high-to-low subnetworks in series.
  • Empirical evaluation: The paper demonstrates superior keypoint detection performance on the COCO keypoint detection dataset and the MPII Human Pose dataset.These are the two benchmark datasets used for the reported keypoint-detection evaluation.
  • Empirical evaluation: The paper also shows HRNet’s superiority for video pose tracking on the PoseTrack dataset.PoseTrack provides an additional evaluation beyond the two keypoint-detection benchmarks.

2. Related Work

Related work is dominated by convolutional approaches that either regress keypoint positions or estimate heatmaps, typically using high-to-low and low-to-high representation processing. The proposed approach instead maintains high-resolution representations throughout parallel high-to-low subnetworks with repeated multi-scale fusion.

  • Pose estimation methods: Deep convolutional neural networks provide dominant solutions for single-person pose estimation, using either keypoint-position regression or heatmap estimation followed by peak selection.Traditional alternatives include probabilistic graphical and pictorial structure models, later enhanced with deep learning or iterative inference.
  • Network designs: Most heatmap-estimation networks reduce resolution in a stem, process representations in a main body, and recover heatmaps through a regressor and full-resolution transformation.Their main bodies mainly use high-to-low and low-to-high processing, sometimes with multi-scale fusion.
  • Network designs: Representative designs use symmetric high-to-low and low-to-high processes, heavy high-to-low processing with lightweight upsampling, or dilated convolutions followed by resolution increase.These patterns include hourglass-style mirroring, classification-network backbones with bilinear upsampling or transpose convolutions, and dilation-based resolution preservation.
  • Multi-scale fusion and supervision: Prior multi-scale methods aggregate separately processed resolutions, progressively connect features through skip connections or global pathways, and use intermediate supervision to improve training and heatmap quality.Intermediate heatmaps may become inputs or partial inputs to the remaining subnetwork.
  • Our approach: Our approach connects high-to-low subnetworks in parallel, maintains high-resolution representations throughout, and repeatedly fuses their outputs instead of using a separate low-to-high upsampling process.The stated goal is spatially precise heatmap estimation from reliable high-resolution representations.

3. Approach

The approach introduces HRNet, which maintains high-resolution representations through parallel multi-resolution subnetworks and repeatedly fuses information across them. It predicts keypoint heatmaps from the final high-resolution representation using mean squared error against Gaussian ground-truth heatmaps.

  • Parallel multi-resolution subnetworks: HRNet replaces sequential high-to-low processing with parallel subnetworks that retain the original high-resolution branch while progressively adding lower-resolution branches.Each later stage contains the previous resolutions plus one additional lower resolution.
  • Repeated multi-scale fusion: Repeated exchange units aggregate information across parallel resolutions, so each subnetwork repeatedly receives information from the others.The third stage is divided into multiple exchange blocks, each combining parallel convolution units with an exchange unit.
  • Exchange unit: Each exchange output is formed by aggregating transformed inputs from all resolutions while preserving the input resolutions and widths.Downsampling uses strided 3×3 convolutions, whereas upsampling uses nearest-neighbor sampling after a 1×1 convolution for channel alignment.
  • Network instantiation: The instantiated HRNet has four stages and four parallel subnetworks, with resolution halved and channel width doubled across resolutions; stages 2–4 contain 1, 4, and 3 exchange blocks.The experiments use HRNet-W32 and HRNet-W48, whose high-resolution widths in the last three stages are 32 and 48, respectively.

4. Experiments

Experiments on COCO, MPII, and PoseTrack show that HRNet maintains accurate high-resolution pose representations and achieves leading results across keypoint detection and tracking benchmarks. Ablations further support the benefits of resolution maintenance, repeated multi-scale fusion, pretraining, wider networks, and larger inputs.

  • COCO validation results: On COCO validation, HRNet-W32 trained from scratch at 256 × 192 achieves 73.4 AP, outperforming methods using the same input size.It gains 6.5 AP over Hourglass and 4.8 and 4.0 points over CPN without and with OHKM, respectively.
  • COCO validation results: Pretraining, increased width, and larger input resolution improve COCO validation performance.ImageNet pretraining adds 1.0 points for HRNet-W32; HRNet-W48 adds 0.7 and 0.5 points at 256 × 192 and 384 × 288, while HRNet-W32 and HRNet-W48 reach 75.8 and 76.3 AP at 384 × 288.
  • MPII results: HRNet-W32 achieves 92.3 PCKh@0.5 on MPII, matching the best previously published leaderboard result and surpassing stacked hourglass methods and extensions.MPII evaluation uses cropped 256 × 256 inputs and reports the head-normalized PCKh@0.5 metric.
  • PoseTrack results: HRNet-W48 achieves 74.9 mAP and 57.9 MOTA on PoseTrack2017, exceeding FlowTrack in SimpleBaseline by 0.3 mAP and 0.1 MOTA.PoseTrack evaluates human pose estimation and articulated tracking in video.
  • Ablation studies: Multi-scale fusion helps COCO validation performance, with more fusions producing better results.Resolution also affects keypoint prediction quality, and the lowest-resolution response map produces an AP below 10 points.
  • Ablation studies: Maintaining high resolution throughout the network improves COCO validation AP: HRNet-W32 reaches 73.4 AP versus 72.5 for a similar-cost variant that adds all four subnetworks initially.The variant has similar parameter count and GFLOPs but does not preserve the staged high-resolution design.
  • Ablation studies: Compared with SimpleBaseline using ResNet-50, HRNet’s improvement is 4.0 points at 256 × 192 and 6.3 points at 128 × 96.The larger gains at smaller inputs are attributed to maintaining high resolution throughout the process.

5. Conclusion and Future Works

The paper presents a high-resolution pose-estimation network that maintains high resolution throughout processing and repeatedly fuses multi-resolution representations to produce accurate, spatially precise keypoint heatmaps. Future work targets other dense prediction tasks and less lightweight aggregation of multi-resolution representations.

  • Conclusion: The network maintains high resolution throughout processing and repeatedly fuses multi-resolution representations, producing reliable, accurate, and spatially precise keypoint heatmaps.Its success stems from avoiding high-resolution recovery and repeatedly exchanging information across resolutions.
  • Future Works: Future applications include semantic segmentation, object detection, face alignment, and image translation.These applications extend the approach beyond human pose estimation to other dense prediction tasks.
  • Future Works: Future work will investigate aggregating multi-resolution representations in a less light way.The paper identifies aggregation design as an additional direction for improvement.

Appendix · Results on the MPII Validation Set

The appendix reports MPII validation results from models trained on a subset of the MPII training set and evaluated on 2,975 held-out images. Testing averages heatmaps from original and flipped images and uses six-scale pyramid testing, with results presented in Table 7.

  • Results on the MPII Validation Set: Models are trained on a subset of MPII training data and evaluated on a held-out validation set of 2,975 images.The training procedure matches that used for the full MPII training set.
  • Results on the MPII Validation Set: Testing averages heatmaps from the original and horizontally flipped images.This procedure is used to produce the test heatmap.
  • Results on the MPII Validation Set: Following prior work, evaluation also applies a six-scale pyramid procedure for multi-scale testing.The resulting validation-set measurements are reported in Table 7.

More Results on the PoseTrack Dataset

On PoseTrack2017, HRNet-W48 achieves strong multi-person pose estimation performance, surpassing the previous state-of-the-art on both validation and test sets. The section also reports detailed multi-person pose tracking results on the test set.

  • PoseTrack2017 results: The section provides results for all keypoints and reports multi-person pose estimation performance on the PoseTrack2017 dataset.Table 8 presents the multi-person pose estimation performance.
  • Pose estimation results: HRNet-W48 achieves 77.3 points mAP on the PoseTrack2017 validation set.The result concerns multi-person pose estimation performance.
  • Pose estimation results: 74.9 points mAP on the PoseTrack2017 test set places HRNet-W48 above the previous state-of-the-art method by 0.3 points.The cited comparison is for the test set.
  • Pose estimation results: HRNet-W48 outperforms the previous state-of-the-art method by 0.6 points on the PoseTrack2017 validation set.This comparison accompanies the validation mAP result.
  • Pose tracking results: Detailed multi-person pose tracking performance on the PoseTrack2017 test set is provided as a supplement in Table 9.The tracking results supplement those reported in the paper.

Results on the ImageNet Validation Set

On ImageNet 2013, the networks were trained for 100 epochs under a stepwise learning-rate schedule and achieved classification performance comparable to purpose-built models such as ResNet. HRNet-W32 reached a single-model top-5 validation error of 6.5%.

  • The models were trained and evaluated on the ImageNet 2013 classification dataset.
  • Training used 100 epochs with batch size 256 and a learning rate reduced tenfold at epochs 30, 60, and 90.The initial learning rate was 0.1.
  • The networks achieved performance comparable to image-classification models such as ResNet.
  • 6.5% was HRNet-W32’s single-model top-5 validation error.
Loading 1902.09212v1…