PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Person Pose Estimation with Enhanced Channel-wise and Spatial Information

Kai Su, Dongdong Yu, Zhenqi Xu, Xin Geng, Changhu Wang

arXiv:1905.03466v1cs.CV

TL;DR

Multi-person pose estimation still faces accurate-localization challenges, while spatial and channel-wise feature enhancement has received limited attention. The paper introduces CSM for cross-scale channel communication and SCARB for spatial and channel-wise attention, and reports state-of-the-art performance on the COCO keypoint benchmark.

  • Problem

    Accurate multi-person pose localization remains difficult under close interactions, occlusions, and varying human scales, while spatial and channel-wise feature enhancement has been little used.

  • Method

    The method combines CSM for cross-channel communication across feature-map scales with SCARB for adaptive spatial and channel-wise enhancement of fused pyramid features.

  • Results

    The proposed method achieves state-of-the-art performance on the COCO keypoint benchmark.

  • Takeaways & Limitations

    Enhancing cross-scale channel communication and fused feature responses provides the paper's supported approach to improving multi-person pose estimation.

Abstract

from arXiv · show

Multi-person pose estimation is an important but challenging problem in computer vision. Although current approaches have achieved significant progress by fusing the multi-scale feature maps, they pay little attention to enhancing the channel-wise and spatial information of the feature maps. In this paper, we propose two novel modules to perform the enhancement of the information for the multi-person pose estimation. First, a Channel Shuffle Module (CSM) is proposed to adopt the channel shuffle operation on the feature maps with different levels, promoting cross-channel information communication among the pyramid feature maps. Second, a Spatial, Channel-wise Attention Residual Bottleneck (SCARB) is designed to boost the original residual unit with attention mechanism, adaptively highlighting the information of the feature maps both in the spatial and channel-wise context. The effectiveness of our proposed modules is evaluated on the COCO keypoint benchmark, and experimental results show that our approach achieves the state-of-the-art results.

1. Introduction

Multi-person pose estimation remains difficult because accurate localization must handle close interactions, occlusions, and varying human scales. This paper addresses limited spatial and channel-wise feature enhancement with CSM and SCARB, achieving state-of-the-art COCO keypoint results.

  • Multi-person pose estimation locates body parts for all people but remains difficult under close interactions, occlusions, and different human scales.
  • High-level features help infer invisible or occluded keypoints, while high-resolution low-level features support detailed keypoint refinement.
  • Spatial and channel-wise attention can highlight informative feature-map components, but has been little used in multi-person pose estimation.
  • The Channel Shuffle Module enhances cross-channel communication among feature maps across scales.
  • SCARB integrates spatial and channel-wise attention into residual units to adaptively enhance fused pyramid feature responses.
  • The proposed method achieves state-of-the-art results on the COCO keypoint benchmark.

2. Related Work

Related work covers multi-scale feature fusion and visual attention mechanisms used to represent spatial relationships and emphasize informative feature channels or locations.

  • Multi-scale fusion: Multi-scale pose-estimation methods use sequential, stacked, or pyramid architectures to capture spatial relationships across resolutions.
  • Visual attention: Visual attention has been applied across network design, image captioning, and pose estimation to emphasize informative feature components.
  • Visual attention: SE-Net models channel-wise statistics but does not account for spatial attention in feature maps.

3. Method

The method uses CPN as its base network and investigates two modules that enhance multi-scale feature communication and fused feature responses.

  • The framework adopts Cascaded Pyramid Network as the basic structure for evaluating the proposed modules.
  • The Channel Shuffle Module operates on pyramid features from different scales to improve information communication across feature-map channels.
  • The Spatial, Channel-wise Attention Residual Bottleneck adaptively enhances fused pyramid feature responses in spatial and channel-wise contexts.

3.1. Revisiting Cascaded Pyramid Network

CPN is a two-step pose-estimation network that first locates simpler keypoints and then refines difficult ones using online hard-keypoint mining. In this method, its multi-scale GlobalNet features are enhanced through channel shuffling before use.

  • CPN first uses GlobalNet with an FPN architecture to locate relatively simple keypoints from a human box.
  • CPN then uses RefineNet with Online Hard Keypoints Mining to address difficult keypoints.
  • ResNet feature maps R-Conv-2∼5 are reduced to 256 channels, shuffled by CSM, and concatenated with the original pyramid features.

3.2. CSM: Channel Shuffle Module

The Channel Shuffle Module addresses complementary information across pyramid levels by shuffling concatenated feature channels, enabling cross-level channel communication.

  • 3.2. CSM: Channel Shuffle Module: CSM targets limitations in the trade-off between low-level detail and high-level context by combining complementary channel information across feature-map levels.Low-level maps provide higher resolution for refinement, while high-level maps provide larger receptive fields for challenging or occluded keypoints.
  • 3.2. CSM: Channel Shuffle Module: Conv-3∼5 are upsampled to Conv-2 resolution, concatenated with Conv-2, channel-shuffled, then split and downsampled to their original levels.
  • 3.2. CSM: Channel Shuffle Module: The shuffle operation reshapes 1024 channels into (g,c), transposes them to (c,g), and flattens them back, making the concatenated features fully related in the channel context.Here, 1024 = 256 × 4, and g is the number of groups discussed in ablation studies.

3.3. ARB: Attention Residual Bottleneck

The Attention Residual Bottleneck augments the residual branch with spatial and channel-wise attention, adaptively enhancing fused pyramid responses before combining them with the identity mapping.

  • 3.3.1 Spatial Attention: Spatial attention generates a spatial weight map from the input and uses element-wise multiplication to emphasize task-related feature-map regions.The weights are produced by a 1×1 convolution followed by a sigmoid activation.
  • 3.3.2 Channel-wise Attention: Channel-wise attention treats feature channels as pattern detectors and adaptively selects important detectors using channel statistics and learned weights.Global average pooling produces channel statistics, followed by a sigmoid-based gating mechanism and channel-wise rescaling.
  • 3.3. ARB: Attention Residual Bottleneck: The bottleneck applies attention on the non-identity residual branch before summation with the identity branch.Figure 4 contrasts the original residual bottleneck with the attention-enhanced design.
  • 3.3.3 SCARB: Spatial, Channel-wise Attention Residual Bottleneck: SCARB applies spatial attention before channel-wise attention, producing enhanced spatial and channel-wise information in the residual mapping.The attention output is combined with the identity branch in the residual bottleneck.
  • 3.3.4 CSARB: Channel-wise, Spatial Attention Residual Bottleneck: CSARB reverses the order by applying channel-wise attention before spatial attention, and the choice between SCARB and CSARB is evaluated in ablation studies.

4. Experiments

The experiments evaluate the proposed pose estimation system on COCO using controlled ablations and comparisons with established methods. Training and testing follow specified top-down settings, with results assessed using OKS-based AP.

  • The system uses a top-down pipeline that detects human boxes before predicting each person’s pose.
  • Experiments use COCO trainval for training, minival for ablations, and test-dev for final comparisons under the official OKS-based AP metric.Training uses 57K images and 150K person instances without extra data; ablations use 5K minival images and final results use 20K test-dev images.
  • The ablation tables isolate the Channel Shuffle Module and Attention Residual Bottleneck, then analyze their combined contribution over the CPN baseline.
  • The default backbone is ResNet-50, while ResNet-50, ResNet-101, and ResNet-152 are evaluated experimentally.
  • The comparison includes 8-stage Hourglass, CPN, and Simple Baselines on COCO minival, with Online Hard Keypoints Mining marked separately.
  • Testing uses shared detection settings and averages heatmaps from original and flipped images before applying a quarter-pixel keypoint offset.

4.2. Component Ablation Studies

Component ablations show that both proposed modules improve the CPN baseline on COCO minival, with the strongest result obtained when CSM-4 and SCARB are combined.

  • Channel Shuffle Module: 71.7 AP is achieved by CSM-4 alone, improving 2.3 AP over the 69.4-AP CPN baseline.Four groups are selected because they provide the best CSM result in this ablation.
  • Attention Residual Bottleneck: 70.8 AP is achieved by SCARB alone, improving 1.4 AP over the CPN baseline.SCARB outperforms the alternative CSARB ordering and is selected by default.
  • Component Analysis: 72.1 AP is achieved when CSM-4 and SCARB are combined, yielding a 2.7 AP improvement over baseline CPN.

4.3. Comparisons on COCO minival dataset

On COCO minival, the proposed model outperforms 8-stage Hourglass, CPN, and Simple Baselines under the reported input-size comparisons. Qualitative results also cover close interactions and occlusions where CPN performs less well.

  • The model improves 8-stage Hourglass by 5.2 AP at the 256 × 192 input size.
  • The model outperforms CPN by 2.7 AP at 256 × 192 and 2.2 AP at 384 × 288.
  • The model outperforms Simple Baselines by 1.5 AP at 256 × 192 and 1.6 AP at 384 × 288.
  • Qualitative comparisons indicate that the model handles close interactions and occlusions where CPN does not perform well.

4.4. Experiments on COCO test-dev dataset

On COCO test-dev, the proposed single model achieves higher pose-estimation AP than the reported CPN and Simple Baselines results without extra training data. Pose accuracy also varies with human-detection performance, but the comparison favors improving the pose estimator in the reported setting.

  • State-of-the-art Comparison: 74.3 AP is achieved by the single ResNet-152 model without extra training data, exceeding CPN’s single-model 72.1 AP, ensembled 73.0 AP, and Simple Baselines’ 73.8 AP.
  • State-of-the-art Comparison: 74.6 AP is achieved with the ResNet-152 model using original, flipped, and rotated-image heatmaps.
  • Human Detection Performance: Pose-estimation AP rises from 73.8 to 74.3 when human-detection AP increases from 57.2 to 58.1 for the proposed model.
  • Human Detection Performance: Despite lower human-detection AP, the proposed model reaches 74.3 pose AP versus 73.8 for Simple Baselines with higher detection AP.

5. Conclusions

The top-down model combines CSM for cross-scale channel communication with SCARB for spatial and channel-wise enhancement, achieving state-of-the-art performance on COCO keypoints.

  • The top-down model uses CSM to promote cross-channel communication across feature maps at all scales.
  • SCARB adaptively highlights fused pyramid feature maps in spatial and channel-wise contexts.
  • The overall model achieves state-of-the-art performance on the COCO keypoint benchmark.
Loading 1905.03466v1…