PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Feature Pyramids for Human Pose Estimation

Wei Yang, Shuang Li, Wanli Ouyang, Hongsheng Li, Xiaogang Wang

arXiv:1708.01101v1cs.CV

TL;DR

Human pose estimation must handle body-part scale variation caused by viewpoint changes and foreshortening. The paper introduces Pyramid Residual Modules and branch-aware initialization, achieving state-of-the-art results on two pose benchmarks, including 92.0% PCKh on MPII.

  • Problem

    Human pose estimation remains difficult because viewpoint changes, foreshortening, articulation, and occlusion create inconsistent body-part scales that challenge localization.

  • Method

    The paper learns multi-scale feature pyramids with Pyramid Residual Modules and derives weight initialization for multi-branch networks.

  • Results

    92.0% PCKh on MPII establishes a new state-of-the-art result, while experiments report state-of-the-art performance across two pose benchmarks.

  • Takeaways & Limitations

    The proposed modules and initialization scheme are presented as general components that can support pose estimation and other CNN tasks.

  • Takeaways & Limitations

    The initialization derivation assumes weights and inputs are independent and identically distributed, with additional independence assumptions for activation derivatives and inputs.

Abstract

from arXiv · show

Articulated human pose estimation is a fundamental yet challenging task in computer vision. The difficulty is particularly pronounced in scale variations of human body parts when camera view changes or severe foreshortening happens. Although pyramid methods are widely used to handle scale changes at inference time, learning feature pyramids in deep convolutional neural networks (DCNNs) is still not well explored. In this work, we design a Pyramid Residual Module (PRMs) to enhance the invariance in scales of DCNNs. Given input features, the PRMs learn convolutional filters on various scales of input features, which are obtained with different subsampling ratios in a multi-branch network. Moreover, we observe that it is inappropriate to adopt existing methods to initialize the weights of multi-branch networks, which achieve superior performance than plain networks in many tasks recently. Therefore, we provide theoretic derivation to extend the current weight initialization scheme to multi-branch network structures. We investigate our method on two standard benchmarks for human pose estimation. Our approach obtains state-of-the-art results on both benchmarks. Code is available at https://github.com/bearpaw/PyraNet.

1. Introduction

Human pose estimation remains difficult because articulation, occlusion, viewpoint changes, and foreshortening create inconsistent body-part scales even after whole-body scale normalization. The paper addresses this with a Pyramid Residual Module, a multi-branch initialization scheme, and a solution to activation-variance accumulation, reporting state-of-the-art results on two benchmarks.

  • 1. Introduction: Body-part localization is difficult because articulation, occlusion, viewpoint changes, and foreshortening cause inconsistent part scales despite whole-body scale normalization.These challenges undermine body-part detectors, such as when hand and foot scales differ across views after images are warped to the same human-body size.
  • 1. Introduction: The Pyramid Residual Module learns multi-scale feature pyramids by subsampling input features at different ratios and learning convolutional filters at multiple network levels.It is designed to improve scale invariance in DCNNs with only a small increase in complexity.
  • 1. Introduction: The paper derives a weight-initialization scheme for DCNN layers with multiple input or output branches, extending applicability to structures such as Inception models [30] [46] and ResNets [25] [26].The need arises because the Pyramid Residual Module uses branches and existing initialization schemes are inappropriate for such multi-branch networks.
  • 1. Introduction: The authors address activation-variance accumulation when identity-mapped residual outputs are added in Hourglass networks, where summation can approximately double output variance.They introduce a simple, effective solution with negligible additional parameters.
  • 1. Introduction: The method achieves state-of-the-art results on two human pose estimation benchmarks, while ablations test the pyramid module, initialization scheme, and activation-variance solution.The paper also evaluates generalization on a standard image-classification task.

2. Related Work

Prior human pose estimation used handcrafted graph-based features and image pyramids, while deep and multi-branch networks improved performance and scale modeling. The proposed approach builds on these directions by efficiently learning multi-scale features within an Hourglass-based architecture and motivating suitable initialization for multi-branch networks.

  • Human pose estimation: Graph-based pose estimators modeled spatial relationships among body parts with handcrafted features and relied heavily on image pyramids.These methods included pictorial and loopy structures and HOG features.
  • Human pose estimation: Deep models achieved state-of-the-art results in human pose estimation, including DeepPose and other recent approaches.The passage identifies DeepPose as one of the first deep models in this line of work.
  • Human pose estimation: Training a multi-branch network on three image-pyramid scales strengthened scale invariance, but computation and memory increased linearly with the number of scales.The proposed pyramid residual module is presented as a lower-cost alternative for learning multi-scale features.
  • DCNNs combining multiple layers: Multi-branch networks generally outperform plain networks, using parallel transformations to capture varied context information; the proposed module applies this principle to Hourglass residual units.The approach is complementary to multi-layer prediction methods [37] [6] [23] [4] [9] and replaces single-scale residual units with Pyramid Residual Modules.
  • Weight initialization: Weight initialization is essential for training deep models, motivating theoretically grounded variance estimates such as Xavier initialization.Earlier Gaussian initialization could make very deep networks difficult to train because of gradient instability [45].

3. Framework

The framework embeds Pyramid Residual Modules (PRMs) in a stacked Hourglass Network to learn multi-scale feature pyramids for human pose estimation. PRMs transform features at multiple resolutions, combine the resulting representations, and serve as general CNN building blocks.

  • Hourglass Network: PRMs replace standard residual units in the stacked Hourglass Network to capture multi-scale visual patterns and semantics rather than features at only one scale.The Hourglass Network repeatedly performs bottom-up subsampling and top-down upsampling with intermediate supervision at each stack.
  • Pyramid Residual Module: The PRM learns feature pyramids across DCNN levels, spanning primitive visual patterns through high-level semantics by applying filters to input features at different resolutions.Outputs from the pyramid transformations are summed and then convolved by subsequent filters.
  • Pyramid Residual Module: Each PRM branch uses a bottleneck design that reduces feature dimensions with 1 × 1 convolution, processes subsampled inputs with 3 × 3 convolutions, and combines the resulting features.The module is designed to reduce computational and space complexity.
  • Pyramid Residual Module: With M = 1 and C = 4, the implemented pyramid ranges from the original input resolution to a lowest scale at half the input resolution.The c = 0 output retains the input resolution, while the lowest-scale branch has half resolution.
  • PRM Variants and Generality: The PRM is presented as a general CNN module applicable beyond stacked Hourglass pose networks, including Wide Residual Nets and ResNeXt for image classification.The framework also considers variants using fractional max-pooling, convolution and upsampling, shared weights, concatenation, or dilated convolutions to construct pyramids.

4. Training and Inference

Training represents each joint with a Gaussian score map and optimizes squared error across hourglass stacks, while inference selects maximum-score locations. The method also derives multi-branch initialization rules and replaces selected residual identity mappings to prevent variance explosion.

  • Training objective: The network predicts Gaussian score maps for K body joints and applies a squared-error loss at the end of each hourglass stack.Ground-truth maps are generated from Gaussians centered at the joint locations, and each stack predicts one score map per joint.
  • Inference: During inference, each joint location is recovered by selecting the maximum-score position in the score map produced by the last hourglass stack.
  • Multi-branch initialization: Existing initialization methods [24] assume branchless networks, so the paper derives variance-based initialization constraints that account for input and output branch counts.The derivation assumes independently and identically distributed weights and activations, with activation-dependent α equal to 0.5 for ReLU and 1 for Tanh and Sigmoid.
  • Multi-branch initialization: Stacking multi-branch layers without intervening operations can increase output variance approximately Q^i times under Xavier or MSR [24] initialization.The number of input and output branches must therefore be considered when initializing parameters.
  • Residual variance control: Replacing summed residual identity mappings with BN-ReLU-1 × 1 convolution stops variance explosion and improves performance in experiments.The replacement is used when outputs of two residual units are summed; the paper reports that breaking variance explosion provides better performance in Section 5.1.3.

5. Experiments

Experiments evaluate the proposed approach on MPII and LSP human-pose benchmarks using PCKh and PCK metrics, alongside complexity and ablation analyses. The study also tests pyramid branches in Wide ResNet and ResNeXt on CIFAR-10.

  • Human-pose benchmarks: The approach is evaluated on MPII and LSP using PCKh@0.5 and PCK@0.2, respectively, with results reported in Tables 1 and 2.MPII contains about 25k images and over 40k people, while LSP and its extension contain 12k challenging sports-pose images.
  • Complexity: The eight-stack model increases parameters from 23.7M to 26.9M and computation from 41.2 to 45.9 GFLOPs, while deeper hourglass networks reportedly provide little improvement.These correspond to increases of 13.5% in parameters and 11.4% in computation for a 256×256 RGB image.
  • Architectures of PRM: All tested pyramid residual modules outperform the baseline, and PRM-B* is selected because it has similar accuracy with fewer parameters and lower computational complexity than several alternatives.PRM-A/C have higher parameter counts than PRM-B/B*/D, while PRM-A/C/D require more GFLOPs than PRM-B/B*.
  • Ablation studies: Increasing the number of pyramid scales generally improves performance, while the proposed initialization scheme outperforms Xavier and MSR initialization.The scale study varies pyramid scales from 3 to 5 at fixed model size; training and validation curves compare the initialization methods.
  • Ablation studies: Controlling variance explosion raises validation performance from the baseline’s 87.6 to 88.0 PCKh, and the pyramid residual module further raises it to 88.5.The ablation uses a 2-stack hourglass network on the MPII validation set.
  • CIFAR-10 experiments: On CIFAR-10, the proposed pyramid branches are incorporated into Wide ResNet and ResNeXt, using four branches with scales from 0.5 to 1.The dataset contains 50k training images and 10k test images across 10 classes.

6. Conclusion

The paper proposes a Pyramid Residual Module for scale-invariant DCNN features and derives an initialization scheme for multi-branch networks. It also introduces a method to prevent response variances from exploding when combining multiple identity mappings.

  • The Pyramid Residual Module enhances scale invariance in deep convolutional neural networks.
  • The paper derives an initialization scheme for multi-branch networks and evaluates its theoretical soundness and efficiency experimentally.
  • A simple method prevents response variances from exploding when outputs of multiple identity mappings are added.
Loading 1708.01101v1…