Source-linked AI summary
P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation
Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Wen Gao
TL;DR
Monocular 2D-to-3D pose estimation is difficult because of depth ambiguity and challenging spatial-temporal correlations. P-STMO uses masked pose pre-training followed by fine-tuning with modular many-to-one modeling, achieving superior benchmark performance with lower computational cost. The method also reports 42.1mm MPJPE on Human3.6M with CPN inputs.
Problem
Monocular 3D pose estimation is ill-posed because 2D keypoints leave joint depth ambiguous, while Transformer methods make direct 2D-to-3D correlation learning difficult and require more training data.
Method
P-STMO pre-trains an encoder with spatial-temporal masked pose modeling to reconstruct corrupted 2D poses, then fine-tunes it with STMO and a many-to-one aggregator for current-frame 3D prediction.
Results
P-STMO achieves superior performance over state-of-the-art methods on two datasets with smaller computational budgets.
Takeaways & Limitations
The two-stage strategy reduces STMO optimization difficulty, while modular spatial-temporal processing and temporal downsampling support efficient pose estimation.
Takeaways & Limitations
Transformer-based components require more training data than convolutional neural networks, and the input sequence typically assumes an odd number of frames centered on the current frame.
Abstract
from arXiv · showhide
This paper introduces a novel Pre-trained Spatial Temporal Many-to-One (P-STMO) model for 2D-to-3D human pose estimation task. To reduce the difficulty of capturing spatial and temporal information, we divide this task into two stages: pre-training (Stage I) and fine-tuning (Stage II). In Stage I, a self-supervised pre-training sub-task, termed masked pose modeling, is proposed. The human joints in the input sequence are randomly masked in both spatial and temporal domains. A general form of denoising auto-encoder is exploited to recover the original 2D poses and the encoder is capable of capturing spatial and temporal dependencies in this way. In Stage II, the pre-trained encoder is loaded to STMO model and fine-tuned. The encoder is followed by a many-to-one frame aggregator to predict the 3D pose in the current frame. Especially, an MLP block is utilized as the spatial feature extractor in STMO, which yields better performance than other methods. In addition, a temporal downsampling strategy is proposed to diminish data redundancy. Extensive experiments on two benchmarks show that our method outperforms state-of-the-art methods with fewer parameters and less computational overhead. For example, our P-STMO model achieves 42.1mm MPJPE on Human3.6M dataset when using 2D poses from CPN as inputs. Meanwhile, it brings a 1.5-7.1 times speedup to state-of-the-art methods. Code is available at https://github.com/paTRICK-swk/P-STMO.
1 Introduction
P-STMO addresses the depth ambiguity and optimization challenges of monocular 2D-to-3D pose estimation through self-supervised pre-training and modular spatial-temporal modeling. Its two-stage design combines masked pose modeling, specialized STMO modules, and temporal downsampling, achieving strong benchmark performance with reduced computational demands.
- Motivation: Monocular 3D pose estimation remains ill-posed because 2D keypoints leave joint depth ambiguous.
- Motivation: Transformer-based methods directly learn 2D-to-3D spatial-temporal correlations, making optimization difficult and requiring more training data than convolutional networks.
- P-STMO: P-STMO introduces two-stage training: masked pose modeling reconstructs corrupted 2D poses, then fine-tuning predicts the current 3D pose from a 2D sequence.
- P-STMO: MPM randomly masks frames and joints, encouraging the encoder to learn spatial and temporal dependencies while reconstructing the original 2D poses.
- STMO: STMO separates spatial encoding, temporal encoding, and many-to-one aggregation, using an MLP for spatial extraction and temporal downsampling to reduce redundancy.
- Results: P-STMO achieves state-of-the-art performance on two benchmarks with fewer parameters and smaller computational budgets.
2 Related Work
Related work integrates spatial and temporal information using convolutions, graph operations, or Transformers, while many-to-one models aggregate multiple frames for a target pose. P-STMO distinguishes these functions through dedicated modules and adapts masked pre-training to pose estimation.
- Spatiotemporal modeling: Prior methods use fully connected layers, graph convolutions, 1D convolutions, or Transformers to integrate spatial and temporal pose information.
- Spatiotemporal modeling: Some approaches estimate 3D pose from a single frame, whereas others investigate spatiotemporal integration across video sequences.
- Many-to-one aggregation: Temporal convolutional networks progressively reduce the temporal dimension to aggregate multiple frames for predicting one frame, but do not explicitly extract spatial and temporal features.
- Many-to-one aggregation: STMO assigns spatial encoding, temporal encoding, and many-to-one aggregation to separate modules for modeling their intrinsic properties.
- Pre-training: MPM masks joints in spatial and temporal domains to recover original 2D poses, unlike METRO, which masks inputs to directly regress 3D joints.
3 Method
P-STMO separates 2D-to-3D pose estimation into self-supervised pre-training and 3D fine-tuning. Its masked pose modeling learns spatial-temporal dependencies, while STMO combines spatial encoding, temporal encoding, temporal downsampling, and many-to-one aggregation.
- Overview: The method first pre-trains an encoder with masked pose modeling, then fine-tunes STMO to predict the 3D pose in the current frame.Stage I reconstructs corrupted 2D poses; Stage II loads the encoder and estimates the middle-frame 3D pose.
- Pre-Training: Masked pose modeling randomly removes frames and joints, requiring the encoder-decoder to reconstruct the original 2D pose sequence.Temporal masking replaces frames with a shared temporal padding embedding, while spatial masking replaces joints with a shared spatial padding joint.
- STMO Model: STMO uses SEM for within-frame spatial features, TEM for cross-frame temporal dependencies, and MOFA to aggregate multiple frames for middle-frame prediction.The modules are arranged in series after the pre-trained encoder is loaded for fine-tuning.
- Pre-Training: Spatial-temporal masking applies temporal masking before spatial masking on the remaining frames, with total ratio qST = qT + (1 − qT) · qS.This combined strategy is the one used in P-STMO.
- STMO Model: An MLP block provides spatial feature extraction, while temporal downsampling reduces redundant input data and enlarges the temporal receptive field.The paper motivates downsampling because adjacent video frames can change only slightly and larger temporal inputs increase computational demands.
- STMO Model: MOFA stacks strided 1D temporal convolutions to aggregate adjacent frames without overlap, ultimately mapping multiple input frames to the current frame.The convolution stride equals the kernel size, and successive layers reduce the temporal sequence to one frame.
4 Experiments
Experiments on Human3.6M and MPI-INF-3DHP evaluate P-STMO against existing methods, its components, masking strategies, SEM designs, temporal downsampling, and computational complexity.
- Comparison with State-of-the-Art Methods: 42.1mm MPJPE is achieved on Human3.6M with CPN inputs and the refining module, surpassing all other methods.With 243 input frames without the refining module, P-STMO obtains 42.8mm MPJPE and 34.4mm P-MPJPE.
- Comparison with State-of-the-Art Methods: 24.2% MPJPE improvement and 9.1% AUC improvement are reported on MPI-INF-3DHP, while PCK remains competitive with.The evaluation uses ground-truth 2D poses and 81 input frames.
- Comparison with State-of-the-Art Methods: P-STMO produces good qualitative results in both indoor and complex outdoor scenes on Human3.6M and MPI-INF-3DHP.The qualitative comparison is against Poseformer.
- Ablation Study: MPJPE falls from 51.0mm with SEM alone to 44.2mm with SEM, TEM, and MOFA, while MPM and TDS provide additional 1.0mm and 0.2mm gains.The component analysis uses P-STMO-S with CPN 2D keypoints on Human3.6M.
- Ablation Study: Spatial-temporal masking performs best at qT = 80% and mS = 2, improving over spatial-only and temporal-only masking by 0.33mm and 0.3mm.The corresponding optimal spatial-temporal masking ratio is qST = 82.3%.
- Ablation Study: MLP outperforms fc and Transformer for SEM by 3.7mm and 2.9mm, while TDS improves performance by up to 2.1mm and can reduce overhead without much sacrifice.TDS gains decrease as the number of input frames increases; the reported gains are 2.1mm, 1.5mm, and 0.2mm for N = 27, 81, and 243.
5 Conclusion
P-STMO combines masked pose modeling, an MLP spatial encoder, and temporal downsampling for 3D human pose estimation. Experiments on two datasets report superior performance over state-of-the-art methods with smaller computational budgets.
- P-STMO uses masked pose modeling to enhance the representation capability of its spatial and temporal encoding modules.
- An MLP block serves as the spatial encoding backbone and is reported as more effective than Transformer and fully connected alternatives.
- Temporal downsampling reduces input redundancy while increasing the temporal receptive field.
- Experiments on two datasets show superior performance over state-of-the-art methods with smaller computational budgets.
A.1 Other Related Work
The related work contrasts P-STMO's spatial-temporal masking and learnable mask tokens with prior approaches in sign language recognition and 2D pose embedding.
- P-STMO replaces masked joints with learnable vectors, avoiding confusion between masked joints and unmasked joints assigned coordinates (0,0).
- Unlike metric-learning approaches for view-invariant embeddings, P-STMO learns spatial-temporal embeddings by solving the masked pose modeling task.
- P-STMO uses masking to increase pre-training difficulty, whereas related work uses it to simulate real-world occlusion.
- P-STMO explores spatial-temporal masking, while the compared 2D pose embedding method does not.
A.2 Implementation Details
The implementation trains both stages with Adam for 80 epochs and uses an MLP for spatial encoding alongside Transformer-based temporal encoding and decoding.
- Both training stages use Adam for 80 epochs, with initial learning rates of 1e−4 for Stage I and 7e−4 for Stage II.
- Training uses batch size 160 and horizontal flipping for data augmentation during training and testing.
- The spatial encoding module uses an MLP with one sub-block and 256-dimensional latent features.
- The temporal encoding module and Stage I decoder use vanilla Transformers, with depths varying by the P-STMO-S or P-STMO configuration.
A.3 Analysis on Model Hyperparameters
The hyperparameter analysis examines temporal encoder depth, Stage I decoder depth, and Transformer latent-feature dimension within an asymmetric encoder-decoder design.
- Table 7 investigates the depths of TEM and the Stage I decoder, along with the latent-feature dimension of all Transformers.
- The design is asymmetric because the decoder depth may differ from the encoder depth.
- Pre-training parameters and FLOPs are excluded from reported inference cost because the analysis focuses on inference speed.
A.4 Reconstruction Results in the Pre-Training Stage
The pre-training network reconstructs masked 2D pose sequences even when only a small fraction of joints remains visible. This demonstrates qualitative recovery under substantial spatial and temporal masking.
- 18.7% of joints remain visible under qT = 0.8 and mS = 2 masking.The input sequence is masked across frames and joints before reconstruction.
- The network roughly recovers the original 2D poses from the sparsely visible input.The qualitative reconstruction is obtained using only a small number of visible joints.
- The reconstruction result supports masked pose modeling as a pre-training task for corrupted 2D pose sequences.The task evaluates whether the model can restore poses after spatial-temporal masking.
A.5 Visualization of Multi-Head Self-Attention
The attention visualizations characterize how Transformer modules use spatial and temporal information across pre-training and fine-tuning. Stage I attention supports recovery from masked frames, while Stage II combines local and global temporal patterns for 3D prediction.
- Stage I: Stage I uses N = 243 frames, with 48 unmasked and 195 masked frames under qT = 0.8.The 48 unmasked frames are passed to the encoder, while the masked frames are reconstructed.
- Stage I: The decoder restores masked-frame positions by appending temporal padding embeddings and adding positional embeddings.This implementation produces different attention behavior before and after the unmasked-frame portion.
- Stage I: Temporal attention in Stage I links masked frames to nearby unmasked frames to support sequence recovery.Most heads give little attention from unmasked frames to masked ones, while some heads interpret whole-sequence continuity.
- Stage II: Stage II TEM learns both local patterns around the query frame and global patterns spanning a longer temporal range.Different attention heads specialize in short- and long-term information for 3D pose sequence prediction.
- Stage II: MOFA attention concentrates on the current frame and its neighbours because it predicts the 3D pose in the current frame.MOFA and TEM exhibit different attention patterns, attributed to the presence of multi-frame loss.
A.6 Qualitative Results on in-the-wild Videos
The proposed method generalizes qualitatively to in-the-wild videos containing rare or unseen poses. Additional experiments show benefits from pre-training across camera views and robustness under noisy or shuffled inputs.
- The method generalizes well to in-the-wild videos with rare or unseen poses.The model is trained on Human3.6M and evaluated on in-the-wild videos using AlphaPose 2D keypoints.
- 44.4mm MPJPE with pre-training compares with 45.2mm MPJPE without pre-training on a different camera view.Stage I uses camera views 0, 1, and 2, while Stage II uses camera view 3.
- Pre-training remains effective when ground-truth 2D keypoints receive Gaussian noise or when input frames are randomly shuffled.The robustness analysis evaluates multiple noise levels and the random-shuffle strategy.