PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-stream Spatio-Temporal GCN-Transformer Network for 3D Human Pose Estimation

Jiawen Duan, Jian Xiang, Zhiqiang Li, Linlin Xue, Wan Xiang

arXiv:2604.17688v1cs.CV

TL;DR

Monocular 2D-to-3D pose estimation remains challenging because depth uncertainty and flattened Transformer inputs can limit local skeletal modeling. MixTGFormer addresses this with dual-stream spatial-temporal fusion of Transformer and GCN features plus channel-dependency modeling, achieving state-of-the-art results on two benchmarks.

  • Problem

    Monocular 2D-to-3D pose estimation is challenging because depth uncertainty and Transformer-based modeling can neglect local skeletal relationships.

  • Method

    MixTGFormer uses dual-stream spatial and temporal Mixformer Blocks that fuse Transformer and GCN features, followed by an SE Layer that models channel dependencies.

  • Results

    MixTGFormer achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP, including 37.6mm P1 error on Human3.6M and 85.4% AUC with 16.5mm MPJPE on MPI-INF-3DHP.

  • Takeaways & Limitations

    The method provides a unified way to model global and local spatio-temporal relationships in 2D-to-3D human pose estimation.

  • Takeaways & Limitations

    The authors identify reducing computational complexity and expanding applications such as real-time tracking and multi-view estimation as future directions.

Abstract

from arXiv · show

3D human pose estimation is a classic and important research direction in the field of computer vision. In recent years, Transformer-based methods have made significant progress in lifting 2D to 3D human pose estimation. However, these methods primarily focus on modeling global temporal and spatial relationships, neglecting local skeletal relationships and the information interaction between different channels. Therefore, we have proposed a novel method,the Dual-stream Spatio-temporal GCN-Transformer Network (MixTGFormer). This method models the spatial and temporal relationships of human skeletons simultaneously through two parallel channels, achieving effective fusion of global and local features. The core of MixTGFormer is composed of stacked Mixformers. Specifically, the Mixformer includes the Mixformer Block and the Squeeze-and-Excitation Layer ( SE Layer). It first extracts and fuses various information of human skeletons through two parallel Mixformer Blocks with different modes. Then, it further supplements the fused information through the SE Layer. The Mixformer Block integrates Graph Convolutional Networks (GCN) into the Transformer, enhancing both local and global information utilization. Additionally, we further implement its temporal and spatial forms to extract both spatial and temporal relationships. We extensively evaluated our model on two benchmark datasets (Human3.6M and MPI-INF-3DHP). The experimental results showed that, compared to other methods, our MixTGFormer achieved state-of-the-art results, with P1 errors of 37.6mm and 15.7mm on these datasets, respectively.

1. Introduction

3D human pose estimation is important for recovering 3D keypoints, but Transformer methods can underrepresent local skeletal relationships. MixTGFormer combines Transformer and GCN processing across spatial and temporal dimensions, with channel interaction modeling.

  • 3D human pose estimation recovers 3D keypoint coordinates from images or videos and supports applications including motion analysis, virtual reality, augmented reality, and activity recognition.
  • Monocular 2D-to-3D lifting is mainstream because RGB cameras are practical, but depth uncertainty makes estimation challenging.
  • Transformer models capture long-range dependencies but can neglect local spatial node relationships after 2D pose sequences are flattened.
  • MixTGFormer combines Transformer and GCN features to integrate local skeletal relationships with global spatial relationships.
  • Its spatial and temporal Mixformer Block forms aggregate Transformer and GCN features across spatio-temporal dimensions.
  • The SE module learns dependencies between feature channels, and MixTGFormer achieves best performance on Human3.6M and MPI-INF-3DHP.

2. Related Work

Related work frames 3D pose estimation around input type and estimation strategy, with monocular 2D-to-3D lifting becoming mainstream. Transformer, GCN, and hybrid models address temporal, spatial, and skeletal relationships in different ways.

  • 3D Human Pose Estimation: 3D human pose estimation evolved from handcrafted features and geometric constraints toward deep learning methods.
  • 3D Human Pose Estimation: Monocular methods reduce camera and hardware requirements compared with multi-view methods, despite lacking direct depth information.
  • 3D Human Pose Estimation: Direct estimation predicts 3D poses from images, whereas 2D-to-3D lifting first detects 2D poses and then estimates their 3D counterparts.
  • Transformer-based Methods: Poseformer established a Transformer backbone for 3D pose estimation by modeling spatial and temporal information.
  • Transformer-based Methods: Later Transformer methods explored frequency-domain representations, masked pose modeling, parallel spatial-temporal attention, and general motion representations.
  • GCN-based Methods: GCNs are used in 3D pose estimation because their efficient relational modeling suits skeleton-based tasks and can learn connections between joints.
  • Hybrid Methods: Hybrid Transformer-GCN models combine global attention with graph-based joint relationships and have produced state-of-the-art results on public datasets.

3. Method

MixTGFormer lifts 2D keypoint sequences to 3D poses using stacked dual-stream Mixformers that fuse spatial and temporal information. Its Mixformer Blocks combine GCN and multi-head attention, while the SE Layer adaptively reweights channels after stream fusion.

  • Overall Architecture: MixTGFormer projects 2D keypoint sequences into features, processes them with stacked Mixformers, and regresses the resulting representation into 3D poses.The input contains frame and joint dimensions, and the final regression head estimates a 3D pose sequence.
  • Training Objective: Training combines 3D position loss, acceleration loss, and the 2D detector loss, with a coefficient balancing position accuracy and motion smoothness.The acceleration term uses differences between consecutive predicted and target poses.
  • Overall Architecture: The dual-stream Mixformer uses parallel branches with different temporal-spatial construction orders, then adaptively fuses their features.Both branches combine spatial and temporal information but emphasize these dimensions differently because their module orders differ.
  • Mixformer: The combined Mixformer Block and SE Layer model local, global, spatial, temporal, and channel dependencies within MixTGFormer.The paper describes this combination as enhancing the model’s representational capability and 3D pose estimation accuracy.
  • Mixformer Block: Mixformer Blocks combine multi-head self-attention and GCN to model global and local dependencies in spatial and temporal dimensions.The spatial form treats joints as tokens within frames, whereas the temporal form treats frames as tokens to model relationships across consecutive frames.
  • Squeeze-and-Excitation Layer: The SE Layer pools temporal and joint dimensions, generates channel weights with a two-layer MLP, and rescales the input channels.Its excitation stage reduces and restores channel dimensionality, while Sigmoid constrains the weights to [0,1].

4. Experiments

MixTGFormer was evaluated on Human3.6M and MPI-INF-3DHP using standard quantitative, qualitative, and ablation experiments. It achieved strong benchmark performance, while ablations examined module composition, connection order, encoding embeddings, and SE Layer placement.

  • Datasets and Evaluation Metrics: Experiments evaluated MixTGFormer on the Human3.6M and MPI-INF-3DHP 3D human pose estimation datasets.Human3.6M evaluation used MPJPE and P-MPJPE, while MPI-INF-3DHP evaluation used MPJPE, PCK, and AUC.
  • Results on Human3.6M: 37.6mm P1 and 31.8mm P2 errors were achieved on Human3.6M, both reported as the best results to date.The comparison excluded models pretrained on additional data and also considered parameter counts and computational complexity.
  • Results on Human3.6M: MixTGFormer produced more comprehensive and superior human body reconstruction results than the compared methods in qualitative visualizations.Figure 4 compares MixTGFormer with STCFormer and MotionAGFormer, highlighting significant differences with arrows.
  • Results on MPI-INF-3DHP: On MPI-INF-3DHP, the standard model achieved 85.4% AUC and 16.5mm MPJPE, improving the prior best results by 1.2% and 1.7mm.Its PCK was 0.2% lower than the current best result, while the small model also achieved competitive performance.
  • Ablation Studies: Using different modules in the two Mixformer Block branches reduced P1 error by 1.0mm and 0.4mm relative to alternative compositions.The ablation attributes this to combining GCN local spatial relationship capture with MHSA global information extraction.
  • Ablation Studies: Ablations also evaluated spatial-temporal connection order, encoding embeddings, and SE Layer placement, with the latter reducing P1 error by 0.3mm at its best position.The encoding study reported 38.9mm with temporal embeddings and a 1.0mm reduction after switching to spatial embeddings; using both produced the largest decrease.

5. Conclusion

MixTGFormer combines Transformer and GCN-based dual-stream modeling with channel enhancement to capture global and local spatio-temporal relationships. Experiments on Human3.6M and MPI-INF-3DHP surpassed current state-of-the-art methods, while computational complexity and broader applications remain improvement directions.

  • MixTGFormer uses dual-stream Transformer-GCN fusion and a Squeeze-and-Excitation Layer to model global and local spatio-temporal relationships.The architecture targets feature interactions across both the backbone and core components.
  • Further work could reduce computational complexity and expand applications to real-time pose tracking and multi-view pose estimation.

CRediT authorship contribution statement

The authors contributed across writing, validation, visualization, investigation, supervision, methodology, data curation, and formal analysis.

  • Contributions span manuscript preparation, supervision, methodology, validation, visualization, investigation, data curation, and formal analysis.
Loading 2604.17688v1…