PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Human Pose Estimation with Spatial and Temporal Transformers

Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, Zhengming Ding

arXiv:2103.10455v3cs.CVcs.AIcs.HC

TL;DR

3D human pose estimation from video must resolve depth ambiguity, while existing CNN and recurrent methods have limited temporal connectivity. PoseFormer uses spatial and temporal transformers without convolutions and achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP.

  • Problem

    Video-based 2D-to-3D pose lifting faces depth ambiguity, while CNN and recurrent approaches have limited temporal connectivity for modeling frame correlations.

  • Method

    PoseFormer uses distinct spatial and temporal transformer modules to model within-frame joint relationships and global dependencies across video frames without convolutions.

  • Results

    PoseFormer achieves state-of-the-art performance on both the Human3.6M and MPI-INF-3DHP 3D pose-estimation benchmarks.

  • Takeaways & Limitations

    Pure transformer architectures can provide strong 3D human-pose estimation performance while modeling local joint relationships and global temporal dependencies.

Abstract

from arXiv · show

Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at \url{https://github.com/zczcwh/PoseFormer}

1. Introduction

PoseFormer is introduced as a pure transformer for video-based 2D-to-3D human pose lifting, explicitly modeling spatial joint relations and temporal frame dependencies. It addresses depth ambiguity and token growth while achieving state-of-the-art results on Human3.6M and MPI-INF-3DHP.

  • Motivation: Temporal information helps address depth ambiguity and occlusion in mapping 2D poses to 3D, motivating video-based human pose estimation methods.Prior approaches use temporal CNNs or recurrent architectures to capture dependencies across frames.
  • Motivation: Transformers are well suited to 3D human pose estimation because self-attention captures global correlations across long input sequences.The introduction connects transformer scalability and modeling capability to sequence-based 3D HPE.
  • Challenges: Treating each 2D joint coordinate as a token captures spatial relations but creates 243×17=4131 tokens for a common 243-frame, 17-joint sequence.A frame-level token baseline instead ignores natural joint-to-joint spatial distinctions.
  • PoseFormer: PoseFormer is the first pure transformer network for video-based 2D-to-3D lifting, using distinct spatial and temporal transformer modules without enormous token counts.The spatial module models human joint relations, while the temporal module captures dependencies across the sequence.
  • Results: PoseFormer achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP and produces smoother, more reliable 3D pose results in visual comparisons.The paper also analyzes PoseFormer’s attention maps.

2. Related Works

Related work on 3D human pose estimation distinguishes direct estimation from 2D-to-3D lifting, with the latter more common among state-of-the-art methods. Prior approaches include residual networks, graph neural networks, and vision transformers, motivating PoseFormer’s transformer-based lifting formulation.

  • 2D-to-3D Lifting HPE: 3D single-person, single-view HPE methods include direct estimation from images and 2D-to-3D lifting from intermediate 2D poses.PoseFormer follows the 2D-to-3D lifting approach, which can use off-the-shelf 2D pose estimators.
  • 2D-to-3D Lifting HPE: 2D-to-3D lifting methods commonly use detectors such as OpenPose, CPN, AlphaPose, and HR-Net before generating the corresponding 3D pose.Martinez et al. regress 3D joint locations from single-frame 2D joint locations with a fully connected residual network.
  • GNNs in 3D HPE: GNN-based lifting represents joints as graph nodes and bones as edges, with LCNs and semantic graph convolutions modeling local neighborhoods or channel-wise edge weights.These approaches were reported to provide promising performance for 2D-to-3D pose lifting.
  • GNNs in 3D HPE: PoseFormer views a transformer encoder as a fully connected graph whose edge weights come from input-conditioned multi-headed self-attention.The operation also uses node-feature normalization, feedforward aggregation, and residual connections.
  • Vision Transformers: Vision-transformer work includes DETR for detection and panoptic segmentation, ViT for image classification, DeiT for data-efficient training, and HPE methods such as Transpose and METRO.ViT required large-scale ImageNet-21k and JFT-300M training data, while METRO combined CNNs with transformers for 3D pose and mesh reconstruction.

3. Method

PoseFormer lifts sequences of detected 2D poses to a 3D center-frame pose using separate spatial and temporal transformer modules, avoiding convolutional architectures. Spatial attention models joint relationships within frames, temporal attention models dependencies across frames, and a weighted regression head produces the final pose.

  • Overall Pipeline: The pipeline uses off-the-shelf 2D pose detections from consecutive frames to estimate the 3D pose of the center frame with a convolution-free transformer network.This follows the established 2D-to-3D lifting setup while replacing CNN-based models with a transformer architecture.
  • Temporal Transformer Baseline: The temporal-transformer baseline treats each complete 2D pose as a token, embeds tokens linearly with positional information, and applies temporal self-attention.The input sequence has f frame tokens, each representing J joints with 2D coordinates, and the embedded sequence is processed by a temporal transformer encoder.
  • PoseFormer Architecture: Because linear patch embedding weakly represents local kinematic relationships, PoseFormer separates spatial and temporal transformers into three modules: spatial encoder, temporal encoder, and regression head.The spatial module focuses on joint correlations, while the temporal module models frame-to-frame dependencies.
  • Spatial and Temporal Modules: Each joint becomes a spatial patch whose projected features are integrated by self-attention, then flattened per frame, concatenated across frames, and augmented with learnable temporal positional embeddings.The temporal transformer uses the same multihead self-attention and MLP-block architecture as the spatial transformer encoder.
  • Regression Head: The regression head applies a learned weighted mean across frames and an MLP with LayerNorm and one linear layer to output the center-frame 3D pose.The output dimension is y ∈R1×(J·3), corresponding to the predicted 3D coordinates of J joints.
  • Loss Function: Training minimizes standard MPJPE between the predicted and ground-truth 3D joint locations.The loss compares the estimated and ground-truth positions of each joint.

4. Experiments

PoseFormer is evaluated on Human3.6M and MPI-INF-3DHP using standard 3D pose metrics, where it outperforms baselines and achieves the best reported performance. Ablations further show benefits from spatial-temporal design choices, longer sequences, and positional embeddings, while attention analyses examine learned joint and temporal relations.

  • Human3.6M: PoseFormer outperforms the temporal transformer baseline by 6.1% under Protocol 1 and 6.4% under Protocol 2 on Human3.6M.Evaluation uses CPN-detected 2D poses as input and reports all 15 test-set actions.
  • Human3.6M: With clean 2D poses, MPJPE decreases from 44.3mm to 31.3mm, approximately 29.7%, and PoseFormer ranks first in 9 actions and second in 6.Its average score improves by approximately 2% compared with SRNet.
  • MPI-INF-3DHP: PoseFormer achieves the best performance on PCK, AUC, and MPJPE across MPI-INF-3DHP.The model uses 2D poses from 9 frames because this dataset typically has shorter sequences.
  • Ablation Studies: Applying positional embeddings to both spatial and temporal transformers produces the best overall result.The ablation compares spatial-transformer use and positional embeddings while fixing the other architecture parameters.
  • Ablation Studies: The optimal PoseFormer configuration uses c = 32, LS = 4, and LT = 4.Here, c denotes spatial-transformer embedding dimension, while LS and LT denote spatial and temporal encoder layers.
  • Computational Complexity Analysis: Increasing input sequence length improves accuracy without substantially increasing total parameters, while inference speed remains acceptable for real-time use.The computational analysis compares performance, parameters, FLOPs per frame, and FPS across sequence lengths and models.

5. Conclusion

PoseFormer is a pure transformer-based method for estimating 3D human pose from 2D videos. It combines spatial modeling of joint relationships with temporal modeling of frame dependencies and achieves state-of-the-art performance on two 3D pose datasets.

  • PoseFormer estimates 3D human pose from 2D videos using a purely transformer-based architecture.The method does not involve convolutional architectures.
  • Its spatial transformer encodes local relationships between 2D joints, while its temporal transformer captures global dependencies across arbitrarily distant frames.The spatial-temporal design models both within-frame joint relations and across-frame temporal correlations.
  • PoseFormer achieves state-of-the-art performance on two popular 3D pose datasets.The conclusion attributes this result to extensive experiments.

Appendix

The appendix supplements the main paper with attention visualizations, frame-wise MPJPE comparisons, qualitative pose analyses, in-the-wild YouTube-video results, and demo videos.

  • Appendix: The appendix provides spatial and temporal attention-map visualizations, frame-wise average MPJPE comparisons, and additional qualitative comparisons of estimated 3D poses.It also presents PoseFormer’s estimated 3D poses on in-the-wild YouTube videos and demo videos showcasing the results.

A. Attention Visualization

The paper visualizes PoseFormer’s spatial and temporal attention maps across all eight heads on a Human3.6M test sequence. Spatial heads exhibit different attention intensities, reflecting distinct learned local relations among input joints.

  • A. Attention Visualization: Visualization examples cover spatial and temporal attention maps for all 8 heads on Human3.6M test set S11 during SittingDown.The examples are presented for the evaluated PoseFormer model.
  • A. Attention Visualization: In spatial self-attention maps, the x-axis represents queries over 17 joints, while the y-axis indicates attention output.These axes define how joint queries and attention responses are visualized in Fig. 7.
  • A. Attention Visualization: Different attention heads produce different intensities, representing varied local relations learned among the input joints.The visualization shows that heads capture distinct spatial interaction patterns.

B. Frame-wise Analysis

Frame-wise analysis on Human3.6M shows that PoseFormer generally produces lower per-frame joint-position error than both the temporal-transformer-only baseline and a state-of-the-art method across Eating and Photo actions.

  • B. Frame-wise Analysis: PoseFormer yields lower frame-wise MPJPE in most frames of the Eating and Photo actions than the temporal-transformer-only baseline and state-of-the-art method [5].The analysis averages MPJPE across all estimated joints per frame using Human3.6M test subject S11.

C. More Qualitative Results

PoseFormer is qualitatively evaluated against ground truth, a state-of-the-art method, and the baseline on Human3.6M S11 for Greeting and WalkDog, achieving more accurate estimations.

  • C. More Qualitative Results: On Human3.6M test set S11, PoseFormer achieves more accurate 3D pose estimations than the state-of-the-art method [5] and the baseline for Greeting and WalkDog.The visual comparison is shown in Fig. 10.

D. Performance on Videos in-the-wild

PoseFormer is evaluated on online YouTube videos to examine 3D pose estimation under the dynamic environments, unknown camera settings, appearance variation, and severe occlusions of in-the-wild footage. Figure 11 presents qualitative results with detected 2D joints and recovered 3D poses.

  • Challenges of in-the-wild videos: In-the-wild 3D pose estimation is more challenging than Human3.6M because environments are dynamic and camera settings are unknown.The paper also notes large foreground/background appearance variations and severe occlusions in unconstrained settings.
  • Qualitative evaluation: PoseFormer is evaluated qualitatively on online videos from YouTube.Figure 11 shows original frame sequences alongside detected 2D joints and recovered 3D poses.
  • Qualitative evaluation: Figure 11 visualizes original frame sequences, detected 2D joints, and recovered 3D poses using PoseFormer.
Loading 2103.10455v3…