Source-linked AI summary
TokenPose: Learning Keypoint Tokens for Human Pose Estimation
Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, Erjin Zhou
TL;DR
Human pose estimation requires visual evidence and anatomical relationships, but CNN-based methods have limited ability to model keypoint constraints explicitly. TokenPose embeds each keypoint as a token alongside visual tokens so self-attention learns appearance and constraint cues together. The models achieve competitive performance with substantially fewer parameters and lower computation than CNN-based counterparts.
Problem
CNN-based pose estimators model visual representations well but lack concrete variables for explicitly learning constraint relationships between keypoints.
Method
TokenPose represents image patches as visual tokens and each keypoint type as a keypoint token, using Transformer self-attention to learn appearance cues and anatomical constraint relations.
Results
TokenPose achieves competitive state-of-the-art performance on COCO and MPII with much fewer parameters and lower computation than existing CNN-based counterparts.
Takeaways & Limitations
The paper shows that a low-capacity pure Transformer without pre-training can work well, while hybrid architectures offer competitive CNN-level results at lower computational cost.
Abstract
from arXiv · showhide
Human pose estimation deeply relies on visual clues and anatomical constraints between parts to locate keypoints. Most existing CNN-based methods do well in visual representation, however, lacking in the ability to explicitly learn the constraint relationships between keypoints. In this paper, we propose a novel approach based on Token representation for human Pose estimation~(TokenPose). In detail, each keypoint is explicitly embedded as a token to simultaneously learn constraint relationships and appearance cues from images. Extensive experiments show that the small and large TokenPose models are on par with state-of-the-art CNN-based counterparts while being more lightweight. Specifically, our TokenPose-S and TokenPose-L achieve $72.5$ AP and $75.8$ AP on COCO validation dataset respectively, with significant reduction in parameters ($\downarrow80.6\%$; $\downarrow$ $56.8\%$) and GFLOPs ($\downarrow$ $75.3\%$; $\downarrow$ $24.7\%$). Code is publicly available.
1. Introduction
TokenPose addresses the difficulty CNN-based pose estimators have in explicitly modeling relationships between keypoints by representing keypoints as tokens alongside visual tokens. The resulting framework jointly learns appearance and anatomical constraint cues, and reports competitive performance with reduced model size and computation.
- 2D human pose estimation must use both visual cues and anatomical constraint relationships to localize keypoints.
- CNN-based methods represent keypoints through heatmaps but lack concrete variables that explicitly model constraint relationships between parts.
- Visual tokens encode image patches, while keypoint tokens represent specific keypoint types and learn visual clues and constraint relations through token interactions.
- TokenPose represents each keypoint entity with a token, unifying visual-cue and constraint-cue learning.
- TokenPose explores both hybrid and pure Transformer-based architectures, including TokenPose-T as the first reported pure Transformer-based model for 2D human pose estimation.
- Experiments on COCO and MPII report competitive state-of-the-art performance with fewer parameters and lower computation than existing CNN-based counterparts.
2. Related Work
Related work establishes CNN heatmap methods as strong visual estimators but identifies explicit keypoint-constraint modeling as difficult. Transformer-based vision methods offer dependency modeling, while prior pose approaches do not directly represent keypoints as constraint-aware tokens.
- Deep CNNs have substantially improved human pose estimation, especially through heatmap-based architectures such as Hourglass, SimpleBaseline, and HRNet.
- CNNs preserve spatial feature structure effectively but remain limited in explicitly capturing constraint relationships between keypoints.
- Transformers use self-attention to model global dependencies and have become promising architectures for vision tasks.
- Prior Transformer pose methods include 3D sequence modeling and long-range dependency modeling, but TransPose does not directly model keypoint constraint relationships.
- TokenPose responds by explicitly representing keypoints as token embeddings so visual clues and constraint relations can be learned together.
3. Method
TokenPose converts image regions and target keypoints into a shared token sequence processed by a Transformer encoder. The output keypoint tokens are linearly mapped to heatmaps, while hybrid variants can use CNN features for efficient low-level extraction.
- The method starts from image patches or CNN feature maps and represents them as visual tokens for Transformer processing.Patches are flattened and linearly projected into d-dimensional embeddings; hybrid variants instead use CNN-extracted feature maps.
- 2D positional embeddings are added to visual tokens so each token represents a specific area of the original image.
- N learnable d-dimensional keypoint tokens are prepended to the visual-token sequence, with each token representing one target keypoint.The output states of these tokens serve as the representations of the N keypoints.
- The Transformer encoder stacks M blocks containing multi-head self-attention and MLP modules to learn keypoint representations.Layer normalization is applied before each module.
- The final keypoint-token outputs are linearly projected into heatmap-sized vectors, reshaped into 2D heatmaps, and trained with mean squared error.
- Hybrid TokenPose variants use CNNs to extract low-level image features more efficiently, while pure variants process image patches directly.
4. Experiments
TokenPose experiments evaluate hybrid and pure Transformer variants on COCO and MPII, emphasizing competitive accuracy with reduced model cost. Ablations and visualizations examine scaling, positional encoding, token fusion, and learned keypoint interactions.
- Model configurations: TokenPose provides hybrid and pure Transformer variants, with TokenPose-T using no convolutional layer.Hybrid variants use CNN backbones for image feature extraction, while TokenPose-T is the pure Transformer variant.
- Datasets and metrics: Experiments use COCO and MPII benchmarks with standard AP and PCKh@0.5 evaluation settings.COCO evaluation uses average precision based on Object Keypoint Similarity; MPII results use PCKh@0.5.
- Benchmark results: TokenPose-S-v1 improves AP by 2.1 points over ResNet-50 SimpleBaseline while reducing parameters by 80.6% and GFLOPs by 75.3%.TokenPose-L/D24 reaches 75.9 AP on COCO test-dev with 384 × 288 input.
- Scaling and visualization: Increasing model depth and width improves results, while attention evolves from global context to reliance on several relevant parts.Visualizations show early whole-body or global interactions and later local, adjacent, or symmetric keypoint relationships.
5. Conclusion
TokenPose represents image patches as visual tokens and keypoint entities as token embeddings, allowing self-attention to capture both appearance and constraint cues. The approach performs competitively with state-of-the-art CNN methods at lower computational cost.
- TokenPose splits images into patches for visual tokens and represents keypoint entities with token embeddings.
- Self-attention interactions enable TokenPose to explicitly capture appearance cues and constraint cues.
- A low-capacity pure Transformer architecture without pre-training can also work well for human pose estimation.
- Hybrid TokenPose architectures achieve competitive results against state-of-the-art CNN-based methods at much lower computational cost.