Source-linked AI summary
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video
Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, Junsong Yuan
TL;DR
Single-frame and prior transformer approaches do not adequately capture joint-specific motion and sequence-wide coherence in monocular 3D pose estimation. MixSTE alternates joint-separated temporal and inter-joint spatial transformer blocks in a seq2seq architecture, achieving state-of-the-art results across three benchmarks.
Problem
Single-frame and prior transformer methods inadequately model differing joint motions and tightly coupled input-output sequence coherence.
Method
MixSTE separates each joint’s temporal trajectory and alternates temporal and spatial transformer blocks within a seq2seq pose-estimation architecture.
Results
MixSTE achieves state-of-the-art performance on Human3.6M, MPI-INF-3DHP, and HumanEva, with reported generalization across benchmarks.
Takeaways & Limitations
Predicting complete pose sequences while modeling joint-specific motion improves sequence coherence and reduces redundant computation relative to central-frame pipelines.
Takeaways & Limitations
The method remains limited by inaccurate 2D detections, including missing and noisy keypoints.
Abstract
from arXiv · showhide
Recent transformer-based solutions have been introduced to estimate 3D human pose from 2D keypoint sequence by considering body joints among all frames globally to learn spatio-temporal correlation. We observe that the motions of different joints differ significantly. However, the previous methods cannot efficiently model the solid inter-frame correspondence of each joint, leading to insufficient learning of spatial-temporal correlation. We propose MixSTE (Mixed Spatio-Temporal Encoder), which has a temporal transformer block to separately model the temporal motion of each joint and a spatial transformer block to learn inter-joint spatial correlation. These two blocks are utilized alternately to obtain better spatio-temporal feature encoding. In addition, the network output is extended from the central frame to entire frames of the input video, thereby improving the coherence between the input and output sequences. Extensive experiments are conducted on three benchmarks (Human3.6M, MPI-INF-3DHP, and HumanEva). The results show that our model outperforms the state-of-the-art approach by 10.9% P-MPJPE and 7.6% MPJPE. The code is available at https://github.com/JinluZhang1126/MixSTE.
1. Introduction
MixSTE addresses limitations in transformer-based 3D pose lifting by separately modeling each joint’s temporal motion and alternating temporal and spatial encoding. Its seq2seq design predicts complete pose sequences, targeting stronger coherence, efficiency, and benchmark performance.
- Monocular 3D pose estimation lifts detected 2D keypoints to 3D, but depth ambiguity makes single-frame reconstruction difficult.
- PoseFormer models spatial and temporal correlations but ignores joint-specific motion differences and limits longer sequences through increased temporal-transformer dimensionality.
- Seq2frame methods estimate only the central pose, weakening sequence coherence and requiring overlapping repeated inputs to reconstruct all frames.
- MixSTE separates each joint’s temporal motion, alternates spatial and temporal transformer processing, and uses a seq2seq architecture to predict coherent pose sequences.
- The method is presented as a transformer encoder for seq2seq 3D pose estimation, improving inference speed and adapting to different input lengths.
- MixSTE achieves state-of-the-art performance across Human3.6M, MPI-INF-3DHP, and HumanEva, with reported generalization across benchmarks.
2. Related Work
Related work spans end-to-end, 2D-to-3D lifting, seq2frame, seq2seq, and transformer-based approaches. Existing methods face computation, fixed-receptive-field, efficiency, or incomplete global-modeling limitations, motivating transformer-based seq2seq exploration.
- End-to-end methods regress 3D coordinates directly from images, while 2D-to-3D lifting methods first detect 2D keypoints and then reconstruct 3D poses.
- Seq2frame methods use temporal context to predict a central-frame pose, but convolutional variants require fixed receptive fields and commonly incur redundant calculation.
- Seq2seq methods reconstruct all input frames and can improve coherence and efficiency, but prior LSTM and GCN approaches have computing-efficiency or global-modeling limitations.
- Transformer architectures have been applied to visual and pose tasks, but earlier pose applications often omit video temporal information or address different settings.
- The related-work analysis identifies transformer-based 3D pose estimation combined with a seq2seq framework as an open direction.
3. Our Approach
MixSTE alternates spatial and temporal transformer blocks to encode inter-joint relations and separate joint motions, then predicts full 3D pose sequences with temporal-aware losses.
- Alternating design in spatio-temporal correlation: MixSTE alternates Spatial Transformer Blocks and Temporal Transformer Blocks to encode spatial dependencies and separate temporal motion.The spatial block attends across joints within each frame, while the temporal block attends across frames for each joint.
- Separate Temporal Correlation Learning: Each temporal token represents one joint trajectory across T frames, reducing the temporal model dimension from N × dm to dm.This separate processing models different joint motion trajectories in parallel and enables longer input sequences.
- Alternating design in spatio-temporal correlation: The model stacks the spatial and temporal blocks for dl loops while preserving feature dimension dm and applying spatial-temporal positional embeddings in the first encoder.The alternating architecture is intended to improve coherence and spatio-temporal feature encoding.
- Seq2seq framework: The seq2seq pipeline predicts all 3D poses for the input sequence at once, preserving input-output sequence coherence and reducing redundant inference.For input length t < T, the inference-time gap relative to seq2frame methods grows approximately as (1+2δ) · t, where δ is padding length.
- Transformer Block in MixSTE: MixSTE uses scaled dot-product multi-head attention, with Q, K, and V formed by linear projections of token features.The attention output combines multiple heads through concatenation and an output projection.
- Loss Function: The end-to-end loss combines WMPJPE, temporal coherence loss, and MPJVE using coefficients λt and λm to avoid excessive smoothness.WMPJPE assigns different weights to different body joints when computing per-joint position error, while T-Loss merges TCLoss and MPJVE.
4. Experiment
Experiments across Human3.6M, MPI-INF-3DHP, and HumanEva evaluate MixSTE under varied detectors, input lengths, metrics, and fine-tuning settings. MixSTE achieves strong accuracy, difficult-pose performance, joint-error stability, component efficiency, and temporal smoothness.
- Human3.6M: MixSTE achieves 40.9mm MPJPE and 32.6mm P-MPJPE on Human3.6M with CPN keypoints, outperforming PoseFormer by 7.6% MPJPE.It performs best with T=243 and second-best with T=81 across all actions.
- Human3.6M: 10.2% improvement under Protocol 1 is obtained with HRNet, while ground-truth 2D keypoints yield approximately 31.0% improvement over PoseFormer.The ground-truth comparison reports MixSTE as significantly outperforming all other methods.
- Human3.6M: MixSTE produces fewer high-error poses, more poses below 30mm MPJPE, and better performance on difficult actions in Human3.6M.The comparison examines MPJPE distributions on test subjects S9 and S11.
- MPI-INF-3DHP: MixSTE obtains the best results on all three MPI-INF-3DHP metrics with T=27 and second-best accuracy with a single input frame.The experiment uses ground-truth 2D keypoints and evaluates both multi-frame and single-frame settings.
- HumanEva: On HumanEva, fine-tuning from Human3.6M and smaller sample strides improve performance, supporting better generalization than previous methods.Without fine-tuning, MixSTE is slightly worse than its baseline because of the seq2seq setting and transformer limitations on the small dataset.
- Ablation Study: Ablations reduce MPJPE from 51.7mm to 40.9mm while joint separation lowers per-frame FLOPs from 186405 to 645.Alternating design, joint separation, and the proposed losses contribute to the final result.
- Ablation Study: The loss design reaches 40.9mm MPJPE and 2.3 MPJVE, while temporal and velocity losses improve temporal smoothness and coherence.MPJVE decreases from 4.6 to 3.6 with temporal consistency loss and to 2.6 with MPJVE loss before the final configuration.
- Visualization: Spatial attention learns different joint dependencies, while temporal attention visualizations examine frame correlations for the SittingDown action.Attention heads are averaged and normalized to [0, 1] for visualization.
5. Conclusion
MixSTE is presented as a transformer-based seq2seq approach that captures global sequence coherence and joint-specific temporal motion, while improving estimation efficiency and benchmark performance. Its main limitation is dependence on inaccurate 2D detections, including missing and noisy keypoints.
- MixSTE captures global sequence coherence and temporal motion trajectories of different body joints.
- The transformer-based seq2seq model improves the efficiency of 3D human pose estimation from monocular video.
- Comprehensive evaluation results show that MixSTE obtains the best performance.
- MixSTE remains limited by inaccurate 2D detection results, such as missing and noisy keypoints.The paper suggests better 2D detectors and modeling input-noise distributions as possible avenues for mitigation.