PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Context Attention for Human Pose Estimation

Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L. Yuille, Xiaogang Wang

arXiv:1702.07432v1cs.CV

TL;DR

Human pose estimation is difficult in cluttered and occluded scenes, and manually defined context lacks flexibility. The paper introduces an end-to-end CNN framework combining multi-context attention, CRF spatial modeling, holistic-part attention, and Hourglass Residual Units. It outperforms previous methods across body parts on both evaluated benchmarks.

  • Problem

    Human pose estimation is challenging under articulation, self-occlusion, cluttered backgrounds, and ambiguous body-part appearances, while manually defined contextual regions lack flexibility and diversity.

  • Method

    The framework uses stacked hourglass networks with multi-resolution and multi-semantic attention, CRFs for neighboring-region correlations, hierarchical holistic-part attention, and Hourglass Residual Units for larger receptive fields.

  • Results

    The approach outperforms previous methods across all body parts on the MPII Human Pose and extended Leeds Sports Poses benchmarks.

  • Takeaways & Limitations

    The model can focus on different granularities, from local salient regions to globally semantic-consistent spaces, while combining holistic and body-part context.

Abstract

from arXiv · show

In this paper, we propose to incorporate convolutional neural networks with a multi-context attention mechanism into an end-to-end framework for human pose estimation. We adopt stacked hourglass networks to generate attention maps from features at multiple resolutions with various semantics. The Conditional Random Field (CRF) is utilized to model the correlations among neighboring regions in the attention map. We further combine the holistic attention model, which focuses on the global consistency of the full human body, and the body part attention model, which focuses on the detailed description for different body parts. Hence our model has the ability to focus on different granularity from local salient regions to global semantic-consistent spaces. Additionally, we design novel Hourglass Residual Units (HRUs) to increase the receptive field of the network. These units are extensions of residual units with a side branch incorporating filters with larger receptive fields, hence features with various scales are learned and combined within the HRUs. The effectiveness of the proposed multi-context attention mechanism and the hourglass residual units is evaluated on two widely used human pose estimation benchmarks. Our approach outperforms all existing methods on both benchmarks over all the body parts.

1. Introduction

Human pose estimation remains difficult under articulation, occlusion, clutter, and foreshortening. The paper addresses these challenges with multi-context attention, CRF-based spatial modeling, hierarchical holistic-part attention, and hourglass residual units, achieving strong benchmark results.

  • Human pose estimation is challenging because articulated limbs, self-occlusion, varied clothing, and foreshortening complicate body-part localization.
  • Manually defined contextual regions lack flexibility and diversity, motivating image-dependent attention maps with variable shapes.
  • The proposed attention mechanism uses CRFs to model spatial correlations among neighboring regions instead of conventional spatial Softmax normalization.
  • Hourglass Residual Units combine multi-scale features, enlarge receptive fields, and support nested hourglass networks.
  • The framework is evaluated on two human pose benchmarks and outperforms previous methods across all body parts.
  • Multi-context attention combines multi-resolution, multi-semantic, and hierarchical holistic-to-part attention to capture both global consistency and precise body-part detail.

2. Related Work

Prior human-pose methods used graphical models and ConvNets, while contextual representations were often manually defined. This paper applies image-dependent visual attention to construct adaptive multi-context representations for pose estimation.

  • Earlier human-pose methods modeled articulated poses with unary terms and graph structures such as mixtures of body parts or pictorial structures.
  • ConvNet-based approaches improved feature representation and incorporated contextual information through multi-scale concatenation or gated functions.
  • Earlier contextual regions were manually defined as rectangles without considering object appearance, limiting adaptive multi-context modeling.
  • The paper uses visual attention to focus on image-dependent regions and reports the first investigation of attention models for human pose estimation.

3. Framework

The framework combines an 8-stack hourglass baseline with multi-context attention, including multi-semantics, multi-resolution, and hierarchical holistic-part attention. Attention maps reweight features to infer regions of interest, while nested hourglass components enrich multiscale information.

  • The framework uses an 8-stack hourglass network as its baseline, with repeated bottom-up and top-down inference across scales.
  • Multi-Resolution Attention: Multi-resolution attention maps are generated from features at different scales, combined, and used to produce progressively refined features and attention maps.
  • Hierarchical Holistic-Part Attention: Higher hourglass stacks encode whole-body configurations and then use coarse-to-fine attention to focus on local body parts.

4. Nested Hourglass Networks

The nested hourglass architecture replaces residual units with hourglass residual units that preserve residual learning while expanding receptive fields through a micro hourglass branch. This design combines local and larger-scale information within stacked hourglass networks.

  • Residual networks pass inputs through identity mappings and stacked convolutional transformations, forming the basis for the proposed residual units.
  • Hourglass residual units replace residual units in side branches, creating a nested hourglass architecture that enriches information across multiple resolutions.
  • The micro hourglass branch uses pooling, convolutions, and upsampling to expand receptive fields while retaining multiscale features.
  • Each hourglass residual unit contains identity, conventional residual, and micro hourglass branches.

B. Residual branch

The hourglass residual unit combines three branches with different receptive fields and resolutions. Its hourglass branch provides broader context, while the other branches preserve high-resolution information.

  • Figure 3 shows identity, conventional residual, and hourglass residual branches within the hourglass residual unit.
  • The conventional residual branch has a 3×3 receptive field, whereas the hourglass residual branch has a 10×10 receptive field.
  • Adding the hourglass branch increases receptive-field size while the identity and conventional residual branches preserve high-resolution information.

5. Attention Mechanism

The attention mechanism generates spatial attention from convolutional features and uses CRFs, multiple resolutions, and hierarchical stacks to refine human-pose representations. It progresses from global body configuration to detailed body-part localization.

  • Conventional Attention: Convolutional features are summarized into a spatial map, whose attention weights reweight the original features channel-wise.
  • Spatial CRF Model: The model replaces global Softmax with a CRF to incorporate local spatial correlations among neighboring attention locations.
  • Multi-Resolution Attention: Attention maps from feature resolutions 8, 16, 32, and 64 are upsampled, summed, and applied to features to create refined representations.
  • Multi-Resolution Attention: Lower-resolution maps emphasize whole-body configuration, while higher-resolution maps focus on local body parts.
  • Hierarchical Holistic-Part Attention: Across hourglass stacks, shallow maps capture local information and deeper maps capture global person-level information that is more robust to occlusion.
  • Hierarchical Holistic-Part Attention: Part-specific attention maps refine features for individual joints and support heatmap prediction through joint-specific classifiers.

6. Training the model

The model trains stacked hourglass outputs with per-stack mean squared error against Gaussian ground-truth heatmaps. Learned attention focuses classifiers on human-body regions and helps exclude easy background.

  • Each hourglass stack predicts body-joint heatmaps and is trained with a mean squared error loss.The loss compares predicted heatmaps with ground-truth heatmaps generated by 2-D Gaussians centered on body-part locations.
  • Attention maps progressively highlight human-body regions, allowing refined features to support classification after easy background regions are removed.For part attention, classifiers focus on individual joints using defined human-body regions rather than background.

7. Experiments

Experiments on MPII and LSP evaluate accuracy, component contributions, training behavior, and qualitative robustness. The method achieves strong benchmark results, while attention and HRUs improve validation performance and address clutter, occlusion, and part confusion.

  • Datasets and setup: The evaluation uses MPII Human Pose and extended LSP, with augmentation including rotation, flipping, rescaling, and color jittering.MPII contains about 25k images and 40k annotated poses; LSP contains 11k training and 1k testing images.
  • Quantitative results: 91.5% PCKh at 0.5 is achieved on MPII, with 1.0% and 1.4% improvements over the closest competitor for wrists and ankles.PCKh normalizes error tolerance by head size.
  • Quantitative results: 1.9% improvement in average PCK at 0.2 is obtained on LSP, outperforming the state of the art across all body joints.Training combines the MPII training set with the extended LSP training set using person-centric annotations.
  • Component analysis: Adding multi-semantics attention reaches 87.2% PCKh, a 1.2% baseline improvement, while HRUs add a further 1% improvement.The ablation evaluates multi-semantics attention, hourglass residual units, multi-resolution attention, and hierarchical part attention on MPII validation.
  • Component and qualitative analysis: The CRF attention model converges faster and reaches higher validation accuracy than Softmax, producing cleaner attention maps through spatial-correlation modeling.Qualitative results also associate holistic attention with reduced clutter and part attention with more precise localization and reduced double counting.
  • Limitations: Failure cases include heavy occlusion, twisted limbs, illumination changes, and left/right confusion caused by clothing or lighting.These cases are described as difficult even for human observers.

8. Conclusion

The conclusion presents an end-to-end ConvNet framework with multi-context attention and CRF-based spatial correlation modeling. It combines multi-resolution, multi-semantics, and hierarchical holistic-part attention with HRUs, which the authors describe as general for other vision tasks.

  • The framework combines ConvNets with multi-context attention in an end-to-end human-pose-estimation model.
  • Multi-context attention uses multi-resolution, multi-semantics, and hierarchical holistic-part components, while CRFs model spatial correlations.
  • Hourglass residual units enrich residual representations, and the authors state that both HRUs and multi-context attention are general for other vision tasks.
Loading 1702.07432v1…