PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network

Soroush Mehraban, Vida Adeli, Babak Taati

arXiv:2310.16288v1cs.CV

TL;DR

3D pose estimation models need to capture both global and local skeleton dependencies while remaining computationally practical. MotionAGFormer addresses this with an adaptive transformer-GCNFormer architecture and multiple variants, achieving state-of-the-art results on Human3.6M and MPI-INF-3DHP.

  • Problem

    Existing transformer-based pose estimators capture global joint relationships but do not precisely model local dependencies, while high-accuracy models can be computationally impractical.

  • Method

    MotionAGFormer uses parallel transformer and GCNFormer streams to model global and local relationships, adaptively fusing their representations within stacked AGFormer blocks.

  • Results

    MotionAGFormer achieves state-of-the-art performance on both Human3.6M and MPI-INF-3DHP.

  • Takeaways & Limitations

    Its variants let users select a balance between speed and accuracy, while the fused representations target the underlying 3D structure of 2D pose sequences.

Abstract

from arXiv · show

Recent transformer-based approaches have demonstrated excellent performance in 3D human pose estimation. However, they have a holistic view and by encoding global relationships between all the joints, they do not capture the local dependencies precisely. In this paper, we present a novel Attention-GCNFormer (AGFormer) block that divides the number of channels by using two parallel transformer and GCNFormer streams. Our proposed GCNFormer module exploits the local relationship between adjacent joints, outputting a new representation that is complementary to the transformer output. By fusing these two representation in an adaptive way, AGFormer exhibits the ability to better learn the underlying 3D structure. By stacking multiple AGFormer blocks, we propose MotionAGFormer in four different variants, which can be chosen based on the speed-accuracy trade-off. We evaluate our model on two popular benchmark datasets: Human3.6M and MPI-INF-3DHP. MotionAGFormer-B achieves state-of-the-art results, with P1 errors of 38.4mm and 16.2mm, respectively. Remarkably, it uses a quarter of the parameters and is three times more computationally efficient than the previous leading model on Human3.6M dataset. Code and models are available at https://github.com/TaatiTeam/MotionAGFormer.

1. Introduction

3D human pose estimation matters across applications, but current models must better balance global and local motion structure with accuracy and computational efficiency. MotionAGFormer addresses this need through a transformer-graph hybrid with adaptive fusion and multiple speed-accuracy variants.

  • 3D human pose estimation supports applications including augmented and virtual reality, autonomous vehicles, and human-computer interaction.
  • 2D-to-3D lifting is pivotal because real-world pose sequences are primarily captured using standard RGB cameras.
  • Global transformer attention captures long-range dependencies, whereas graph models provide stronger local dependency modeling for structured human motion.
  • Complex high-accuracy models can be impractical, while central-frame prediction reprocesses overlapping sequences and introduces computational redundancy.
  • MotionAGFormer combines transformer global modeling, GCN local spatial-temporal modeling, and adaptive feature fusion for 3D pose estimation.
  • MotionAGFormer offers multiple variants that trade speed against accuracy and achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP.

2. Related works

Prior 3D human pose estimation methods include transformer, graph-convolutional, and hybrid architectures. These approaches differ in how they model global and local dependencies, temporal scope, and input structure.

  • 3D pose methods can use multi-view or monocular video and can directly estimate 3D poses or lift 2D poses into 3D.
  • Transformer-based methods: Transformer-based methods capture long-range dependencies and have been extended with frequency-domain representations, multiple pose hypotheses, and masked pose modeling.
  • Graph Convolutional Network: GCN-based methods efficiently model local joints but generally report less competitive error than transformer-based counterparts.
  • Hybrid methods: Hybrid methods combine local GCN and global self-attention modules, but prior designs may fail to distinguish temporal and spatial dimensions.

3. Method

MotionAGFormer lifts 2D pose sequences into 3D using stacked dual-stream AGFormer blocks that separately model global attention-based and local graph-based spatial-temporal relationships. Its GCNFormer stream uses skeleton topology spatially and feature-derived K-nearest-neighbor connectivity temporally, with adaptive fusion combining both streams.

  • AGFormer block: Each AGFormer block contains spatial and temporal MetaFormers, treating joints as spatial tokens and frames as temporal tokens.The two streams differ in their token mixers: one uses Transformers and the other uses GCNFormers.
  • Transformer stream: The Transformer stream applies spatial and temporal multi-head self-attention to capture relationships across joints and frames.Spatial attention is followed by an MLP, residual connection, and LayerNorm before temporal processing.
  • GCNFormer stream: The GCNFormer stream focuses on local spatial and temporal relationships, complementing the Transformer stream’s global information.The customized GCN adds self-connections, normalized adjacency aggregation, trainable transformations, activation, batch normalization, an MLP, and residual normalization.
  • GCNFormer topology: Spatial GCNFormer uses human-skeleton topology, whereas Temporal GCNFormer connects each joint to K nearest neighbors selected from feature similarity across time.Temporal graph topology is determined by learned node features.
  • Adaptive fusion: Adaptive fusion combines Transformer and GCNFormer features using learnable stream weights and element-wise multiplication.The fusion operates on features extracted at corresponding network depths.

4. Experiments

MotionAGFormer is evaluated on Human3.6M and MPI-INF-3DHP, with experiments covering model variants, quantitative comparisons, qualitative reconstructions, and AGFormer design choices. The results show strong accuracy, efficiency, and complementary benefits from the hybrid graph-transformer design.

  • Evaluation setup: MotionAGFormer is evaluated on Human3.6M and MPI-INF-3DHP using multiple model variants selected according to accuracy and computational requirements.The variants differ in parameter size and computational demands.
  • Human3.6M results: 38.4 mm and 17.3 mm are MotionAGFormer-L’s P1 errors for estimated and ground-truth 2D poses on Human3.6M.These results use comparisons without extra pre-training; MotionAGFormer-L requires approximately half the computation and parameters of MotionBERT while improving the corresponding errors by 0.8 mm and 0.5 mm.
  • Human3.6M results: MotionAGFormer-S uses one third of the baseline frames, while MotionAGFormer-XS matches PoseTransformerV2 comparably with 27 versus 243 frames and approximately seven times fewer parameters.These variants target different speed-accuracy trade-offs.
  • MPI-INF-3DHP results: 85.3% AUC and 16.2 mm P1 are achieved by MotionAGFormer-L on MPI-INF-3DHP, exceeding the best models by 1.4% AUC and 6.9 mm P1.Its 98.2% PCK is 0.5% lower than the compared models.
  • Ablation studies: A 16-block, width-128 configuration performs similarly to a 12-block, width-256 configuration while using approximately three times fewer memory and computational resources.The ablation generally favors deeper but narrower models.
  • Ablation studies: Replacing GCNFormer or Temporal GCNFormer with alternative graph modules worsens P1 error, while CTR-GCN increases it by 2.5 mm.The authors hypothesize that CTR-GCN reduces complementary information for the transformer stream by connecting joints beyond immediate neighbors.

5. Conclusion

MotionAGFormer combines GCNFormer’s local joint modeling with the Transformer’s global interdependencies, offering variants that balance speed and accuracy and outperform alternatives on two benchmarks.

  • 5. Conclusion: MotionAGFormer combines GCNFormer for local joint relationships with Transformer modeling of global joint interdependencies.The architecture fuses the two representations to better capture the 3D structure of input 2D sequences.
  • 5. Conclusion: The model provides adaptable variants for selecting a speed–accuracy trade-off.
  • 5. Conclusion: MotionAGFormer surpasses alternative methods on Human3.6M and MPI-INF-3DHP.

A.1. Per-action Result

MotionAGFormer achieves strong per-action performance on Human3.6M, leading on most listed actions and ranking second on the remaining actions.

  • A.1. Per-action Result: MotionAGFormer achieves superior P1 error across Direction, Discuss, Eating, Greet, Purchase, Sitting, Sitting Down, Smoke, Wait, and Walk Dog.
  • A.1. Per-action Result: For the remaining Human3.6M actions, MotionAGFormer delivers the second-best results.

A.2. Per-joint Error Comparison

Per-joint comparisons show that MotionAGFormer’s relative performance varies by joint, with advantages on shoulders but weaker results than MotionBERT on knees and elbows.

  • A.2. Per-joint Error Comparison: MotionAGFormer slightly outperforms MotionBERT and significantly outperforms other models on shoulder joints.
  • A.2. Per-joint Error Comparison: MotionBERT generally outperforms MotionAGFormer on two-degree-of-freedom joints, including knees and elbows.
  • A.2. Per-joint Error Comparison: STCFormer has a slight performance advantage over MotionAGFormer on hips.

B. Ablation on Temporal GCNFormer

The temporal GCNFormer ablation evaluates acceleration error alongside P1 error to compare alternatives for modeling local temporal dependencies.

  • B. Ablation on Temporal GCNFormer: The ablation compares alternative temporal GCNFormer modules using acceleration error in addition to previously reported P1 error.
  • B. Ablation on Temporal GCNFormer: Acceleration error is computed from second-order temporal differences between ground-truth and estimated 3D poses.
  • B. Ablation on Temporal GCNFormer: CTR-GCN and TCN converge faster in acceleration error despite having higher P1 error.
  • B. Ablation on Temporal GCNFormer: CTR-GCN and TCN ultimately improve acceleration error by approximately 0.1 mm.

C. Qualitative comparison for a sequence

The sequence-level qualitative analysis compares MotionAGFormer-B with three models across consecutive Human3.6M frames. MotionAGFormer and MotionBERT show better alignment with ground truth than the other compared models.

  • MotionAGFormer-B is compared with STCFormer, PoseFormerV2, and MotionBERT across consecutive frames from a Human3.6M sequence.The analysis extends beyond single-frame qualitative evaluation to examine model behavior over a sequence.
  • MotionAGFormer and MotionBERT demonstrate superior alignment with the ground truth compared to the other models.

D. In-the-wild Evaluation

The paper evaluates MotionAGFormer qualitatively on unseen in-the-wild videos using its base variant, alongside failure-case and comparison visualizations on Human3.6M. The in-the-wild estimates remain reasonable under noisy 2D poses and sudden noisy movements.

  • The base MotionAGFormer variant is evaluated on unseen in-the-wild videos to assess generalization.
  • 3D pose estimates remain reasonable and resilient when inferred 2D poses contain noise and sudden noisy movements.The passage specifically notes noise in the final frame of the second example.
  • Figure 9 provides qualitative comparisons with STCFormer, PoseFormerV2, and MotionBERT on a random Human3.6M sequence.
  • Figure 10 presents a failure case of the models on a random Human3.6M sequence.
Loading 2310.16288v1…