PaperlayerView exact revision on arXiv

Source-linked AI summary

Poseur: Direct Human Pose Regression with Transformers

Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, Anton van den Hengel

arXiv:2201.07412v2cs.CV

TL;DR

Pose estimation regressors avoid heatmaps but historically lagged because prior designs could lose spatial structure, misalign features, or omit keypoint dependencies. Poseur casts single-person pose estimation as transformer-based sequence prediction and reports state-of-the-art performance among both regression- and heatmap-based methods on MS-COCO and MPII.

  • Problem

    Regression-based pose estimation has received less attention because its performance has lagged, with prior methods losing spatial structure, misaligning features and predictions, or neglecting dependencies between joints.

    Source: §1 · p. 3

  • Method

    Poseur treats single-person pose estimation as predicting a sequence of K joint coordinates with a Transformer, using attention to select relevant image features.

    Source: §1 · p. 3

  • Results

    Poseur achieves state-of-the-art performance among regression- and heatmap-based methods on MS-COCO and MPII; with 49% fewer FLOPs, ResNet-50 Poseur exceeds SimpleBaseline by 5.0 AP on COCO val.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    Poseur can be deployed with fast, low-resolution backbones without a large performance drop, a result the paper says is difficult for heatmap-based methods to achieve.

    Source: §1 · p. 3

    As decoder layers increase, Poseur’s parameters grow rapidly, which may limit deployment in real-time applications on mobile devices.

    Source: §3 · p. 19

Abstract

from arXiv · show

We propose a direct, regression-based approach to 2D human pose estimation from single images. We formulate the problem as a sequence prediction task, which we solve using a Transformer network. This network directly learns a regression mapping from images to the keypoint coordinates, without resorting to intermediate representations such as heatmaps. This approach avoids much of the complexity associated with heatmap-based approaches. To overcome the feature misalignment issues of previous regression-based methods, we propose an attention mechanism that adaptively attends to the features that are most relevant to the target keypoints, considerably improving the accuracy. Importantly, our framework is end-to-end differentiable, and naturally learns to exploit the dependencies between keypoints. Experiments on MS-COCO and MPII, two predominant pose-estimation datasets, demonstrate that our method significantly improves upon the state-of-the-art in regression-based pose estimation. More notably, ours is the first regression-based approach to perform favorably compared to the best heatmap-based pose estimation methods.

1 Introduction

Poseur frames single-person pose estimation as transformer-based sequence prediction, directly regressing keypoint coordinates while addressing spatial-information loss and feature misalignment in prior regression methods. The paper reports strong comparisons with both regression- and heatmap-based methods, including in low-resolution-backbone settings.

1 Introduction

  • Heatmap methods require manually designed and tuned targets and heuristic peak-finding, while lower-resolution maps introduce quantization error.Interpolation may mitigate quantization but adds complexity and hyperparameters and makes the framework less differentiable.Source: §1 · p. 2
  • Prior regression methods may lose spatial structure through global average pooling, misalign features and predictions, and fail to exploit dependencies among joints.These issues are described as contributing to regression methods’ performance deficit relative to heatmap methods.Source: §1 · p. 3
  • Poseur predicts a length-K sequence of joint coordinates from CNN features; keypoint queries attend to relevant features, and self-attention captures dependencies among keypoints.The paper reports more significant improvements with low-resolution backbones such as MobileNet V2 and ResNet, which it says can be used without a large performance drop.Source: §1 · p. 3
  • With 49% fewer FLOPs, ResNet-50 Poseur exceeds SimpleBaseline by 5.0 AP on COCO val and RLE by 4.9 AP; it beats UDP-Pose by 1.0 AP on COCO test-dev.The comparisons are against SimpleBaseline, the previously best regression-based method RLE, and the previously best heatmap-based method UDP-Pose, respectively.Source: §1 · pp. 3–4
  • In an end-to-end Mask R-CNN setting, Poseur surpasses PointSet Anchor by 3.8 AP with HRNet-W48 on COCO val.The integration avoids a manual crop operation.Source: §1 · p. 4

2 Related Work

Related work contrasts heatmap-based pose estimation with direct regression, then places Poseur among regression and Transformer approaches.

Heatmap-based pose estimation

  • Heatmap methods estimate per-pixel keypoint likelihoods but involve non-differentiable decoding and downsampling quantization errors, while RLE brought regression performance to a level comparable with heatmap methods.RLE is trained on a backbone pretrained with heatmap loss.Source: §2 · p. 4

Transformer-based architectures

  • Poseur uses a lightweight Transformer decoder for regression, unlike heatmap-based Transformer methods that use heavy Transformer encoders to improve model capacity.TransPose and HRFormer enhance the backbone with Transformer encoders, while TokenPose uses image patches and class tokens.Source: §2 · p. 4

Regression-based pose estimation

  • PRTR uses DETR 's encoder-decoder structure and Hungarian matching, whereas Poseur is described as alleviating PRTR's computational cost and redundant queries while achieving much higher performance.The architecture figure depicts Poseur combining a backbone, keypoint encoder, and query decoder to predict keypoint coordinates in parallel.Source: §2 · pp. 4–5, Figure 3 · p. 5

3 Method

Poseur predicts keypoint coordinates with keypoint-specific queries that attend to image features, and trains these predictions with likelihood-based supervision. At inference, it derives keypoint confidence scores from predicted distributions; adding those scores improves AP in the reported comparison.

3.1 Poseur Architecture

  • The backbone produces dense feature maps and coarse keypoint proposals that initialize location- and category-specific queries, which the decoder updates into final coordinates.Noisy reference-point sampling during training uses random-coordinate queries to improve robustness to coarse-proposal errors at inference; these queries are not used at inference.Source: §3.1 · pp. 5–7
  • EMSDA samples features around query reference points and aggregates them with attention; this cross-attention addresses feature misalignment in fully connected regression.The method applies its linear layer to sampled features after bilinear interpolation rather than to entire feature maps, which is reported as more efficient with similar performance.Source: §3.1 · p. 7

3.2 Training Targets and Loss Functions

  • Poseur models a conditional distribution over keypoint locations using a flow model, then shifts and rescales it with predicted location and scale parameters.Both coarse proposals and query-decoder predictions receive maximum-likelihood supervision, and their losses are summed with λ = 1 by default.Source: §3.2 · pp. 7–8

3.3 Inference

  • At inference, the predicted keypoint coordinate is the distribution center, while its scale parameter supports a confidence score based on probability within a nearby coordinate interval.The method reports comparable performance when calculating that probability by shifting and rescaling a zero-mean Laplace distribution instead of running the normalization flow.Source: §3.3 · pp. 8–9
  • 4.7 AP improvement (74.7 AP vs. 70.0 AP) follows adding keypoint scores, with almost the same average recall for both models.The passage attributes the improvement to a significantly reduced number of false positives.Source: §3.3 · p. 9

4 Experiments

Experiments evaluate Poseur’s implementation, component ablations, end-to-end extension, and benchmark performance. Poseur is reported to outperform competing methods across several backbones, input resolutions, and evaluation settings.

4.1 Implementation Details

  • Experiments use COCO keypoint detection with AP based on OKS and MPII with PCK, alongside specified backbone, training, and inference configurations.Unless specified, ablation studies use ResNet-50 at 256 × 192 input resolution.Source: §4.1 · pp. 9–10

4.2 Ablation Study

  • Ablations report gains from coarse-proposal query initialization, noisy reference points, more feature-map levels, and the proposed confidence score; decoder performance grows through three layers and saturates at six.The proposed confidence score improves over no uncertainty estimation by 4.7 AP and over the RLE score by 1.0 AP; at 64×64 input, Poseur exceeds SimpleBaseline by 14.6 AP.Source: §4.2 · pp. 10–11

4.3 Extensions: End-to-End Pose Estimation

  • On COCO val, Poseur outperforms heatmap-based Mask R-CNN with ResNet-101 by 1.9 AP and regression-based PointSet Anchor with HRNet-W48 by 3.8 AP in end-to-end top-down estimation.Source: §4.3 · p. 12, Table 3 · p. 12

4.4 Main Results

  • With ResNet-50, Poseur outperforms SimpleBaseline and HRNet-W32 at lower computational cost, while MobileNet-V2 Poseur is 6.0 AP above its SimpleBaseline counterpart at about one-ninth the computational cost.The reported low-resolution comparison uses 256 × 192 input images.Source: §4.4 · pp. 13–14
  • Poseur outperforms same-backbone regression and heatmap methods, reaching 79.6 AP on COCO val and 78.3 AP on COCO test-dev with HRFormer-B.It exceeds UDP-Pose by 1.1 AP on COCO val at 256×192 input resolution; on MPII val, HRNet-W32 Poseur is 0.4 PCKh higher than heatmap HRNet-W32.Source: Table 4 · p. 12, Table 5 · p. 13, §4.4 · p. 14

5 Conclusion

The conclusion presents Poseur as a Transformer-based pose-estimation framework that improves regression-based pose estimation and avoids heatmap-related drawbacks. Experiments on MS-COCO and MPII support state-of-the-art performance among both regression- and heatmap-based methods.

  • Across MS-COCO and MPII, Poseur achieves state-of-the-art performance among regression-based and heatmap-based methods while bypassing heatmap post-processing and quantization drawbacks.Source: §5 · p. 14

1 The Effect of Training Schedules

The training-schedule study tests whether Poseur’s longer training explains its reported advantage. Shorter schedules remain close to the fully trained model.

  • Poseur trained for 275 or 250 epochs performs only slightly below the 325-epoch model, indicating that the longer schedule is not the main reason for its superior performance.Source: §1 · p. 18

2 The Effect of Self-attention

The ablation study finds a performance drop when decoder self-attention is removed. The attention visualization shows the left-shoulder query attending to related joints.

  • Removing decoder self-attention lowers performance from 75.5 AP to 74.0 AP.Source: §2 · p. 18, Table 2 · p. 19
  • The left-shoulder query attends to the left elbow, left wrist, and left ear; thicker visualization lines indicate larger attention weights.Source: §2 · p. 18, Figure 2 · p. 19
  • On truncations, Poseur can predict keypoints outside the bounding box, unlike heatmap methods that predict only within it.Source: Figure 1 · p. 19

3 Reducing the Number of Parameters

Sharing weights across Poseur’s decoder layers substantially reduces its parameters, with a 0.5 AP performance drop; increasing decoder depth otherwise grows parameters rapidly and may constrain real-time mobile deployment.

3 Reducing the Number of Parameters

  • As decoder layers increase, Poseur’s parameters grow rapidly, potentially limiting deployment in real-time mobile applications.Source: §3 · p. 19
  • Weight sharing significantly reduces Poseur’s parameter count while reducing performance by only 0.5 AP.The shared-weight Poseur introduces 2.7 M parameters beyond the 23.5 M ResNet-50 backbone.Source: §3 · p. 19
  • With weight sharing, Poseur introduces 2.7 M parameters beyond its 23.5 M ResNet-50 backbone.Source: §3 · p. 19

4 Computational Cost of EMSDA

EMSDA substantially reduces computational cost compared with MSDA while maintaining almost the same performance, replacing a complexity term that scales with image-feature pixels with one that scales with keypoint queries.

4 Computational Cost of EMSDA

  • EMSDA reduces computational cost from 1.25 GFLOPs to 0.44 GFLOPs while achieving almost the same performance as MSDA.Source: §4 · p. 20
  • MSDA’s complexity includes O(PC^2), while EMSDA replaces this with O(KC^2), with K much smaller than P.For 256 × 192 input and feature maps from Res2 to Res5, P = 4080 and K = 17.Source: §4 · p. 20
  • The reported MSDA complexity is O(KC^2 + PC^2 + 5KSC), while EMSDA’s is O(KC^2 + KC^2 + 5KSC).Source: §4 · p. 20

5 Comparing the Performance of Poseur and RLE

With a ResNet-50 backbone, Poseur outperforms RLE with an HRNet-w32 backbone on AP and inference speed; training memory is lower, while whole-system test memory is reported as similar.

5 Comparing the Performance of Poseur and RLE

  • 75.4 AP vs. 74.3 AP: Poseur with ResNet-50 achieves higher performance than RLE with HRNet-w32.Source: §5 · p. 20
  • 94 FPS vs. 62 FPS: Poseur has faster inference speed than RLE.Source: §5 · p. 20
  • Poseur uses less training memory than RLE (1386 M vs. 1456 M); whole-system test memory is approximately 2000 M for both.Poseur’s pose-estimator test memory is slightly higher: 86.25 M vs. 68.12 M.Source: §5 · p. 20

6 Verifying the Effect of Keypoint Encoder and Query Decoder in Poseur

The keypoint encoder and query decoder improve COCO performance over RLE, the positional encoding converts coordinate proposals into axis-specific embeddings, and queries can infer truncated joints outside predicted boxes.

6 Verifying the Effect of Keypoint Encoder and Query Decoder in Poseur

  • 3.8 AP: the proposed keypoint encoder and query decoder boost performance over RLE on COCO with ResNet-50, without uncertainty estimation.All settings are strictly aligned.Source: §6 · p. 20

7 The Explanation of the Positional Encoding in Keypoint Encoder

  • The keypoint encoder maps each normalized x-y coordinate proposal to separate sine-cosine positional embeddings for the two axes, then concatenates them.The encoding uses d = 128.Source: §7 · p. 21

8 Robustness to Truncation

  • Coordinate queries can infer joints outside predicted boxes by attending to the whole input image, unlike heatmap-based Mask R-CNN in the visualization.Source: §8 · p. 21

Related papers

Loading 2201.07412v2…