PaperlayerView exact revision on arXiv

Source-linked AI summary

TransPose: Keypoint Localization via Transformer

Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang

arXiv:2012.14214v5cs.CV

TL;DR

CNN-based pose estimators have achieved strong performance, but the spatial dependencies they use to localize keypoints remain unclear. TransPose combines convolutional low-level features with Transformer-based global dependency modeling and attention-based heatmap localization. It reports competitive COCO results with a lighter, faster model and strong transfer to MPII.

  • Problem

    Existing CNN pose estimators leave their internal spatial relationships and image-specific keypoint dependencies unclear.

  • Method

    TransPose uses convolutions for low-level features, Transformer attention for global dependencies, and a final attention layer that aggregates image contributions into keypoint heatmaps.

  • Results

    75.8 AP on COCO validation and 75.0 AP on COCO test-dev were achieved, while TransPose used 73% fewer parameters and ran 1.4× faster than HRNet-W48.

  • Takeaways & Limitations

    Attention maps provide image-specific, fine-grained evidence of keypoint dependencies, including how the model handles occlusion, while the lightweight model transfers strongly to MPII.

  • Takeaways & Limitations

    Higher input resolutions incur prohibitively expensive self-attention computation because of quadratic complexity.

Abstract

from arXiv · show

While CNN-based models have made remarkable progress on human pose estimation, what spatial dependencies they capture to localize keypoints remains unclear. In this work, we propose a model called \textbf{TransPose}, which introduces Transformer for human pose estimation. The attention layers built in Transformer enable our model to capture long-range relationships efficiently and also can reveal what dependencies the predicted keypoints rely on. To predict keypoint heatmaps, the last attention layer acts as an aggregator, which collects contributions from image clues and forms maximum positions of keypoints. Such a heatmap-based localization approach via Transformer conforms to the principle of Activation Maximization~\cite{erhan2009visualizing}. And the revealed dependencies are image-specific and fine-grained, which also can provide evidence of how the model handles special cases, e.g., occlusion. The experiments show that TransPose achieves 75.8 AP and 75.0 AP on COCO validation and test-dev sets, while being more lightweight and faster than mainstream CNN architectures. The TransPose model also transfers very well on MPII benchmark, achieving superior performance on the test set when fine-tuned with small training costs. Code and pre-trained models are publicly available\footnote{\url{https://github.com/yangsenius/TransPose}}.

1. Introduction

TransPose addresses the unclear spatial dependencies inside CNN-based pose estimators by combining convolutional feature extraction with Transformer-based global dependency modeling. Its attention-based heatmap localization exposes image-specific evidence while achieving strong accuracy with fewer parameters and faster inference.

  • Motivation: CNN pose estimators implicitly encode spatial relationships, leaving their internal decision processes and keypoint dependencies difficult to interpret.Deepness, implicit relationships, limited working memory, and a lack of suitable visualization tools complicate analysis.
  • Motivation: Transformer attention can capture pairwise interactions between arbitrary image locations more directly than deeply stacked convolutions.The authors motivate this design by noting convolution’s poor scaling for global dependencies and its locality bias.
  • TransPose: TransPose uses convolutions for low-level features and Transformer layers for high-level global dependencies, then recovers Transformer outputs as 2D keypoint heatmaps.Feature maps are flattened before Transformer processing and converted back into heatmaps afterward.
  • Interpretability: The final attention layer aggregates contributions from image locations to form heatmap maxima, linking keypoint localization to Activation Maximization and exposing concrete image clues.These attention scores support fine-grained, image-specific analysis of predicted keypoints, including occluded joints.
  • Results: 75.8 AP on COCO validation and 75.0 AP on COCO test-dev were achieved with 73% fewer parameters and 1.4× faster speed than HRNet-W48.The reported results position TransPose as competitive with CNN-based state-of-the-art models while being more lightweight and faster.

2. Related Work

The related work contrasts CNN locality and implicit spatial reasoning with interpretability methods and emerging Transformer applications. TransPose differs by using Transformer attention to predict spatially distributed 2D heatmaps for human pose estimation.

  • CNN-based pose estimation: CNN locality and translation equivariance make low-level feature extraction efficient, but global dependencies in human pose estimation remain difficult to capture.Prior approaches commonly enlarge receptive fields through deeper networks or downsampling.
  • Explainability: Interpretability research identifies input relevance through gradient-based optimization, saliency maps, and feature-activity visualization.These approaches motivate examining which inputs contribute to predictions.
  • Position information: Proxy-task methods such as CoordConv and Zero Padding investigate how networks predict positions and store positional information.TransPose likewise studies the importance of position embedding and its effect on generalization across input scales.
  • Transformer-based methods: Prior Transformer applications include machine translation, vision tasks, and 3D pose estimation using attention for multi-view fusion or coordinate-sequence prediction.These approaches differ from TransPose’s 2D heatmap representation of keypoint spatial distributions.

3. Method

TransPose combines a CNN backbone with a Transformer encoder and heatmap head to capture global spatial dependencies and expose which image locations support localized keypoints. Its last attention layer aggregates location contributions into heatmap maxima, connecting Transformer localization with Activation Maximization.

  • Architecture: TransPose uses a CNN backbone for low-level features, a Transformer Encoder for long-range spatial interactions, and a head for keypoint heatmaps.Feature maps are flattened into a sequence before Transformer processing and reshaped for heatmap prediction.
  • Efficiency: TransPose targets efficient global interaction modeling by using reduced-resolution attention, with r = 8 for ResNet-S and r = 4 for HRNet-S.These settings trade off attention memory footprint against spatial resolution.
  • Self-Attention: The Transformer processes flattened feature sequences with query-key-value self-attention, whose attention scores dynamically reweight information from image locations.Each query compares with keys across locations, then combines values using the resulting weights.
  • Keypoint Localization: The last attention layer acts as an aggregator, collecting contributions from all image locations to form maximum activations in predicted keypoint heatmaps.The position-wise FFN and head approximately preserve relative contribution proportions through shared linear transformations.
  • Interpretability: The method extends Activation Maximization to heatmap-based localization without extra optimization costs because image-specific attention patterns are learned during training.The resulting dependency areas expose keypoint-specific image clues supporting predicted locations.
  • Dependency Analysis: For a predicted keypoint at location i, locations j with attention score A_i,j ≥ δ are identified as dependencies that significantly contribute to the prediction.The attention map is image- and parameter-dependent, so each query row reveals a location-specific dependency area.

4. Experiments

Experiments evaluate TransPose on COCO and MPII, including efficiency, transfer learning, position embeddings, scaling, and qualitative dependency behavior. Results show competitive COCO performance, strong MPII transfer, improved unseen-resolution generalization with position embeddings, and image- and keypoint-specific dependencies.

  • COCO keypoint detection: TransPose-R-A4 and TransPose-H-A6 outperform SimpleBaseline-Res152, HRNet-W48, and DARK-HRNet at 256×192 with fewer parameters and faster speeds.The reported gains are +0.6 AP, +0.7 AP, and +0.2 AP, respectively.
  • Transfer to MPII benchmark: Pre-training and fine-tuning improve MPII performance while reducing training costs compared with full training.Fine-tuned models outperform longer full-training runs; pre-training improves DARK-HRNet by +1.4 AP and TransPose by +2.0 AP.
  • Transfer to MPII benchmark: TransPose-H-A6 reaches 93.5% accuracy on the MPII test set after fine-tuning on MPII train and validation sets at 256×256 resolution.The model is transferred from pre-training and fine-tuned with the described learning-rate setup.
  • Ablations: Increasing Transformer depth improves TransPose-H performance, while TransPose-R improvements tend to saturate or degrade by six layers.The encoder-size study varies the number of layers, feature dimensions, and FFN hidden units.
  • Ablations: Position embeddings improve generalization to unseen input resolutions, with 2D sine embedding performing especially well.Without position embedding, SimpleBaseline and TransPose-R show performance collapses, particularly at 128×96; 2D sine embedding generalizes better.
  • Ablations: Self-attention at higher input resolutions has quadratic computational complexity, making resolutions such as 384×288 prohibitively expensive for current models.The models are mainly trained at 256×192, corresponding to sequence lengths of 768 and 3072 for TP-R and TP-H.
  • Qualitative analysis: Attention dependencies vary by extractor, keypoint type, layer depth, and image context, revealing global and fine-grained clues for localization.HRNet-S-based models use more long-range joint clues, deeper layers focus on local details, and image-specific dependencies help with occlusion.

5. Conclusion

TransPose introduces Transformer attention into human pose estimation to capture and expose global spatial dependencies. It matches CNN-based counterparts with lightweight architectures, improves MPII performance through fine-tuning, and reveals behavior across model, keypoint, layer, and image conditions.

  • Conclusion: TransPose uses Transformer attention to capture global spatial dependencies explicitly and connects heatmap localization with Activation Maximization.The analysis examines how dependencies vary across layer depths, keypoint types, trained models, and input images.
  • Conclusion: TransPose matches state-of-the-art CNN-based models on COCO with lightweight architectures and gains significant MPII improvements when fine-tuned with small training costs.The conclusion also highlights position embedding and qualitative evidence about handling occlusion.

A. 2D Sine Position Embedding

The Transformer needs position information to preserve spatial structure because sequence processing alone is permutation-equivariant. TransPose therefore encodes horizontal and vertical positions separately with 2D sine embeddings before self-attention.

  • Without position information, the Transformer Encoder is permutation-equivariant over pixel locations or sequence order.
  • TransPose separates position information into independent horizontal and vertical directions.The original 2D structure is retained with d/2 channels for each direction.
  • Sine and cosine functions encode position indices along both spatial directions using the embedding dimensions.The encoding uses p_x and p_y position indices with dimensions indexed by i.
  • The resulting embeddings are stacked and flattened into an R^L×d sequence before being added to the inputs for self-attention.2D sine position embedding is the default for all TransPose models.

B. What position information has been learned in the TransPose model with learnable position embedding?

Learnable position embeddings acquire coarse 2D structure despite being represented as vectors. Neighboring locations become more similar than distant locations, while sine embeddings are reported to generalize better across unseen resolutions.

  • Learnable position embeddings have unique vectors whose cosine similarities are higher for neighboring grid locations and lower for distant ones.This indicates that coarse 2D position information is implicitly learned.
  • Figure 8 visualizes pairwise cosine similarities between learnable position-embedding vectors after reshaping them into 2D grid-like maps.Each map corresponds to one position and compares it with all other locations.
  • Position embedding improves generalization to unseen input resolutions, particularly with 2D sine position embedding.The paper conjectures that fixed receptive fields struggle with scale changes and sine encodings may help associate position across sizes.

C. Transformer Encoder Layer

Each Transformer Encoder layer applies multi-head self-attention and a feed-forward network, with residual connections and layer normalization. Position embeddings modify queries and keys while values come from the original input sequence.

  • The Transformer Encoder layer first applies multi-head self-attention with a residual connection and layer normalization.
  • It then applies a feed-forward network with another residual connection and layer normalization to produce the next layer’s input sequence.The output is denoted X* and is passed to the next Transformer Encoder layer.
  • Position embeddings are added to X when computing queries and keys, while values are computed from X without position embeddings.

D. Gradient Analysis

The gradient analysis models how input tokens contribute to predicted heatmap activations. It argues that last-layer attention scores provide image-specific weights whose contributions are approximately linear, while later position-wise operators apply shared transformations.

  • Gradient Analysis: A heatmap activation at one location can be viewed as judging whether a keypoint is present there using all image tokens.
  • Gradient Analysis: The mapping from the last self-attention feature z_i to heatmap scores h_i is locally approximated as a linear function.The approximation uses learned weights W_f and bias b around a local point z_0.
  • Gradient Analysis: The analysis computes the derivative of a heatmap score at location i with respect to an input token at location j.
  • Gradient Analysis: Attention score A_i,j is computed from the dot product of query and key vectors and is treated as an observed variable for contribution analysis.
  • Gradient Analysis: The contribution function is approximately linear in A_i,j, so each location’s contribution to h_i directly depends on its attention score.The decomposition separates image-specific dynamic weights from learned static weights K and B.
  • Gradient Analysis: The last attention layer aggregates contributions from all image locations according to attention scores and forms maximum activations in the output heatmaps.The FFN and head are position-wise operators that almost linearly transform these scores with the same transformation.

E. Architecture Details

The section documents compact TransPose configurations and visualizes how attention layers identify dependency and affected areas across models and inputs. It also states assumptions about where global interactions occur and how normalization affects them.

  • Architecture configurations: The architecture details for ResNet-S and HRNet-S-W32(48) are reported in Tables 8 and 9, with additional HRNet transition and stage-block details deferred to prior work.ResNet-S* differs from ResNet-S in its bottleneck configuration, while HRNet details are referenced externally within the paper.
  • Assumptions: The analysis assumes that convolutions extract features from limited patches, so global interactions mostly occur in attention layers.It also assumes layer normalization does not alter interactions between locations.
  • Attention visualizations: TransPose-R-A4 and TransPose-H-A4 are used to visualize predicted keypoint locations and their dependency areas.The visualizations use the last attention layer and annotate predicted keypoint locations with white pentagrams.
  • Attention visualizations: Dependency areas are shown for different keypoint types and input images, while affected areas are presented across attention layers.Figure 9 covers two model variants; Figure 10 compares dependency and affected areas across layers and images.

F. More Attention Maps Visualizations

The section presents additional attention-map visualizations for TransPose-R-A4 and TransPose-H-A4. It compares maps from the models’ last attention layers with maps from different attention layers.

  • Models: Additional attention-map visualizations are provided for TransPose-R-A4 and TransPose-H-A4.The section explicitly identifies both model variants as the subjects of the visualizations.
  • Last-layer maps: Figure 9 shows attention maps from the last attention layers of the two models.The figure focuses on the final attention layer for each model.
  • Layer-wise maps: Figure 10 shows attention maps from different attention layers of the two models.This extends the visualization beyond the final attention layer.
Loading 2012.14214v5…