Source-linked AI summary
Pose Recognition with Cascade Transformers
Ke Li, Shijie Wang, Xiang Zhang, Yifan Xu, Weijian Xu, Zhuowen Tu
TL;DR
Multi-person pose recognition must localize keypoints despite substantial variation and occlusion, while heatmap methods often require heuristic processing. PRTR uses cascade encoder-decoder Transformers for regression-based person and keypoint detection, reports competitive regression-based results, and visualizes query refinement; its person evaluation excludes annotations without visible keypoints.
Problem
Multi-person pose recognition requires localizing keypoints under pose, appearance, occlusion, and background variation, while heatmap approaches rely on heuristic processing.
Method
PRTR uses cascade Transformers for person and keypoint regression, with two-stage and end-to-end variants and query refinement across decoder attention layers.
Results
PRTR achieves state-of-the-art performance among regression-based methods on COCO, with its best model reaching 72.1 AP.
Takeaways & Limitations
PRTR offers a general-purpose regression approach requiring fewer heuristic designs and reveals Transformer decoder refinement through keypoint-query visualization.
Takeaways & Limitations
Person annotations without visible keypoints were skipped during person-detector training and evaluation; the detector reached 50.2 mAP on standard val2017.
Abstract
from arXiv · showhide
In this paper, we present a regression-based pose recognition method using cascade Transformers. One way to categorize the existing approaches in this domain is to separate them into 1). heatmap-based and 2). regression-based. In general, heatmap-based methods achieve higher accuracy but are subject to various heuristic designs (not end-to-end mostly), whereas regression-based approaches attain relatively lower accuracy but they have less intermediate non-differentiable steps. Here we utilize the encoder-decoder structure in Transformers to perform regression-based person and keypoint detection that is general-purpose and requires less heuristic design compared with the existing approaches. We demonstrate the keypoint hypothesis (query) refinement process across different self-attention layers to reveal the recursive self-attention mechanism in Transformers. In the experiments, we report competitive results for pose recognition when compared with the competing regression-based methods.
1. Introduction
2D multi-person pose recognition requires localizing keypoints under substantial pose, appearance, occlusion, and background variation. PRTR addresses this with cascade Transformers for regression-based detection, query refinement, and competitive results against regression-based methods.
- 2D human pose recognition localizes keypoints for multiple people in RGB images and supports tracking, security, animation, interaction, games, and robotics.
- Large pose and shape variation, inter-person and self occlusion, appearance variation, and background clutter make pose recognition challenging.
- Heatmap methods achieve strong performance but rely on heuristic, mostly non-end-to-end clustering, grouping, encoding, and decoding procedures.
- PRTR is a regression-based pose recognition method built from cascade Transformers and based on the end-to-end object-detection Transformer DETR.
- PRTR provides two cascade designs: a sequentially learned two-stage model and a jointly learned end-to-end model using a spatial Transformer network.
- The authors visualize keypoint-query distributions across attention layers to expose gradual detection refinement, and report competitive COCO results against regression-based methods.
2. Related Work
Pose recognition methods differ in how they represent and optimize keypoint predictions, with heatmaps offering dense localization and regression offering more direct end-to-end learning. PRTR applies cascade Transformers to formulate step-by-step regression while visualizing decoder refinement.
- Pose recognition predicts a possibly empty set of persons, each with a bounding box and skeleton coordinates for dataset-defined joints.
- Heatmap methods densely classify keypoints and then cluster and group them, providing fine-grained localization but reducing end-to-end integration.
- Regression methods directly minimize coordinate loss and integrate more easily into end-to-end learning, but holistic high-precision optimization is difficult.
- PRTR implicitly formulates step-by-step regression through layered Transformer processing, following the regression-based pose-estimation line.
- PRTR builds cascade Transformers on DETR to pursue general-purpose keypoint regression without specific heuristic-driven designs.
- The paper reports limited prior visualization of Transformer decoding in computer vision and presents dynamic decoder visualization for design insight.
3. Method
PRTR uses cascaded encoder-decoder Transformers for top-down person and keypoint detection, with separate two-stage and jointly trained end-to-end variants. Its keypoint Transformer uses query matching and multi-layer feature cropping to support regression-based prediction.
- PRTR frames pose recognition as Transformer-based regression for person and keypoint detection.The approach uses attention to model relations between image features and prediction hypotheses.
- The first Transformer detects people, while a second Transformer predicts joint classes and 2-channel keypoint coordinates.The keypoint detector operates on matched person queries and predicts in parallel.
- The architecture includes multi-layer feature cropping to provide zoomed-in, multi-level features to the keypoint-detection Transformer.The end-to-end variant applies differentiable bilinear sampling instead of cropping the RGB image directly.
- Training matches fixed prediction queries to ground-truth keypoints with Hungarian optimal bipartite matching.The matching cost combines classification probabilities and coordinate deviation; unmatched queries receive classification loss only.
- At inference, J prototype keypoints are matched to queries using classification probabilities because ground-truth coordinates are unavailable.The matched J keypoints are returned as the prediction.
- The end-to-end variant uses differentiable STN cropping from backbone feature maps, allowing gradients through both features and predicted bounding-box coordinates.Multi-scale feature maps are sampled with a bilinear kernel, resized to a common spatial size, and concatenated for the keypoint Transformer.
4. Experiment
Experiments on COCO and MPII evaluate PRTR’s setup, variants, ablations, performance, and query-refinement behavior. PRTR achieves competitive regression-based pose-recognition results, with performance affected by input size, backbone strength, detection boxes, query design, and inference choices.
- Experiment Setup: PRTR is evaluated on COCO and MPII using OKS and PCK, with COCO experiments covering val2017 and test-dev2017 splits.COCO contains 17 labeled joints per person; MPII is a single-person dataset.
- COCO Results: The end-to-end PRTR variant surpasses CenterNet and DirectPose, while the two-stage variant outperforms competing regression-based baselines on COCO.The ResNet-101 model is comparable to PointSetNet despite PointSetNet using HRNet-W48.
- COCO Results: 72.1 AP is achieved by the best PRTR model, which is reported as able to emulate heatmap-based HigherHRNet; increasing input size improves ResNet-50 and ResNet-101 by 2.2 and 1.9 AP, respectively.The reported gains come from enlarging input size from 384×288 to 512×384.
- MPII Results: 89.5 PCKh@0.5 is achieved by PRTR’s best model on MPII, comparable to SimpleBaseline.MPII is not evaluated with the end-to-end variant because it does not require a person-detection stage.
- Ablation Studies: Class-specific queries reduce AP by 11.4, whereas using 100 queries provides only a small advantage over 50 or 17 queries.The results support dynamically assigning queries to preferred keypoint types and selecting estimates through Hungarian matching.
- Ablation Studies: Including background logits lowers AP by 0.9−1.5, flip testing consistently improves performance, and replacing predicted boxes with ground-truth boxes improves AP by 2−2.5.These findings identify inference configuration and person-detection quality as important contributors to pose-estimation performance.
- Visualization: Queries initially produce equivocal or randomly located predictions, then progressively refine toward keypoint locations across decoder layers.Queries biased toward particular keypoints can also show location-specific behavior; predictions already near ground truth change little in later layers.
5. Conclusion
PRTR is a Transformer-based regression method for multi-person pose recognition that reduces heuristic pre/post-processing through two-stage and end-to-end alternatives. It achieves state-of-the-art performance among regression-based methods on COCO and visualizes query refinement in the decoder.
- Conclusion: PRTR treats pose recognition as regression, removing complex pre/post-processing and requiring fewer heuristic designs than heatmap-based approaches.The method includes both two-stage and end-to-end alternatives.
- Conclusion: PRTR achieves state-of-the-art performance compared with existing regression-based methods on COCO.The conclusion also identifies query-distribution and refinement visualizations as a way to reveal Transformer decoder mechanisms.