PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation

Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, Qiang Xu

arXiv:2108.07181v2cs.CV

TL;DR

Single-view 2D-to-3D pose estimation remains inaccurate on hard poses involving ambiguity, occlusion, and complex or rare configurations. The paper introduces hop-aware hierarchical channel-squeezing fusion and temporal-aware dynamic graph learning, achieving 10.3% average improvement on Human3.6M and state-of-the-art action-recognition performance.

  • Problem

    Hard poses remain difficult for single-view 2D-to-3D estimation despite improved average prediction accuracy.

  • Method

    The method combines hierarchical channel-squeezing fusion with temporal-aware dynamic graph learning for skeletal GNNs.

  • Results

    10.3% average improvement over state-of-the-art techniques was achieved on Human3.6M, with greatly improved hard-pose performance and state-of-the-art action recognition.

  • Takeaways & Limitations

    The approach provides a skeletal representation-learning solution that addresses both long-range aggregation noise and pose-dependent joint relationships.

  • Takeaways & Limitations

    Dynamic graph learning is sensitive to input outliers such as jitter and missing joints, motivating temporal-aware refinement.

Abstract

from arXiv · show

Various deep learning techniques have been proposed to solve the single-view 2D-to-3D pose estimation problem. While the average prediction accuracy has been improved significantly over the years, the performance on hard poses with depth ambiguity, self-occlusion, and complex or rare poses is still far from satisfactory. In this work, we target these hard poses and present a novel skeletal GNN learning solution. To be specific, we propose a hop-aware hierarchical channel-squeezing fusion layer to effectively extract relevant information from neighboring nodes while suppressing undesired noises in GNN learning. In addition, we propose a temporal-aware dynamic graph construction procedure that is robust and effective for 3D pose estimation. Experimental results on the Human3.6M dataset show that our solution achieves 10.3\% average prediction accuracy improvement and greatly improves on hard poses over state-of-the-art techniques. We further apply the proposed technique on the skeleton-based action recognition task and also achieve state-of-the-art performance. Our code is available at https://github.com/ailingzengzzz/Skeletal-GNN.

1. Introduction

Single-view 2D-to-3D pose estimation remains difficult on high-error poses despite improving average accuracy. The paper addresses these cases with hop-aware feature fusion and temporal-aware dynamic skeletal graphs, reporting gains on Human3.6M.

  • Challenge: High-error poses include depth ambiguity, self-occlusion, complex configurations, and poses rarely represented in training data.These cases occur in actions such as “Sit” and “Sit Down.”
  • Motivation: GNN message passing mixes useful distant-node information with undesired noise, creating a signal-to-noise challenge.The problem becomes more pronounced as aggregation reaches farther neighbors.
  • Motivation: Body-joint relationships vary across poses, so a static skeleton graph cannot reliably capture action-specific dependencies.For example, hand-foot relationships differ between running and sitting.
  • Challenge: Existing fully connected and graph-based methods improve general accuracy but still yield large errors on difficult poses.FCNs can overfit, while geometric and GNN-based methods improve prediction accuracy without resolving all hard cases.
  • Contributions: The proposed hop-aware hierarchical channel-squeezing fusion layer preserves relevant neighboring information while suppressing noise.It squeezes long-range context features and fuses them hierarchically with short-range features.
  • Results: 10.3% average improvement was achieved over state-of-the-art techniques on Human3.6M, with greater gains on hard poses.The method also surpasses a state-of-the-art GNN-based solution by 16.3%.

2. Preliminaries and Motivation

The paper motivates skeletal GNNs by contrasting fixed physical connectivity with pose-dependent relationships and by identifying noise in long-range aggregation. Its framework combines hierarchical context fusion with temporal-aware dynamic graph learning.

  • Skeletal Graph Neural Network: A skeletal graph represents joints as nodes and physical bone connections as edges, with hop-k describing shortest-path distance.Hop-1 and hop-2 neighbors provide progressively longer-range context during message passing.
  • Related Work: Existing GNN methods include locally connected and high-order approaches, but high-order aggregation can introduce noise from less-related nodes.Learning edge weights without changing topology is also limited for rare poses.
  • Motivation: Hard actions such as “Sit,” “Sit Down,” and “Take Photos” require both short-range and long-range context, which existing methods do not fuse effectively.Distant neighbors carry valuable semantic information alongside irrelevant variation.
  • Motivation: Fixed physical graphs miss strong hidden relationships that vary across actions, motivating action-specific dynamic edges.Dynamic graphs can represent motion-specific relations more directly than physical topology alone.
  • Method Motivation: Frame-level dynamic graphs are vulnerable to single-frame outliers, so the method introduces temporal information to stabilize graph learning.The framework’s dynamic graph component updates graph structure using temporal context.

3. Method

The method targets hard 2D-to-3D pose estimation with a Dynamic Hierarchical Channel-Squeezing Fusion layer that combines multi-hop spatial context and temporal-aware dynamic graphs. Its design preserves useful short-range information, compresses less informative long-range context, and adapts graph relations to input poses.

  • Dynamic Hierarchical Channel-Squeezing Fusion: The D-HCSF layer updates node features through hierarchical channel-squeezing fusion and temporal-aware dynamic graph learning.The framework takes 2D keypoints X ∈ R^N×2 and outputs 3D joint positions Y ∈ R^N×3.
  • Hierarchical Channel-Squeezing Fusion: The HCSF layer extends node-wise aggregation from direct neighbors to hop-k neighbors and distinguishes short-range from long-range context during fusion.The method is described as adaptable to other graph frameworks, with differences in weight-sharing schemes treated as orthogonal to the fusion approach.
  • Hierarchical Channel-Squeezing Fusion: Hierarchical fusion first aggregates short-range features and long-range features separately, then fuses the resulting representations in two stages.F_k produces fused long-range features, F_a combines them with short-range features, and W_a maps the result to the final dimension.
  • Channel-Squeezing Block: Long-range context is channel-squeezed to suppress irrelevant information, while short-range context retains all channels to preserve essential features.The output size for long-range context decreases with hop distance, reflecting the reported decay of information gain as k increases.
  • Temporal-aware Dynamic Graph Learning: Dynamic graph learning combines a trainable graph based on physical skeleton topology with feature-derived relations that adapt to the input pose.The physical-topology graph is updated during training and fixed during inference, while the feature-based stream produces input-adaptive node relations.
  • Temporal-aware Dynamic Graph Learning: Temporal-aware graph learning adds a time dimension to the input because frame-level dynamic graphs can be unstable as node connections vary across space and time.The temporally extended input is represented as X ∈ R^Cin×T_in×N.

4. Experiment

Experiments on Human3.6M and MPI-INF-3DHP show that the proposed method improves overall 3D pose estimation, especially for hard poses, while also transferring effectively to skeleton-based action recognition.

  • Hard Poses: 16.1% average improvement is obtained on the hardest poses, with only 3.6% of cases exceeding 60mm MPJPE.The method also yields lower counts above 40mm and higher counts below 30mm than competing methods.
  • Hard Poses: 70.7mm mean error on the top 5% hardest poses is 13.8% lower than LCN and 17.1% lower than SRNet.The qualitative comparison likewise reports benefits on hard poses.
  • Cross-Dataset Evaluation: 5.5% improvement across metrics is reported when the Human3.6M-trained model is evaluated on MPI-INF-3DHP.This cross-dataset result is presented as evidence of generalization capability.
  • Ablation Study: Hierarchical fusion outperforms non-hierarchical fusion by 2mm–3.7mm under different hop settings.The ablation study indicates that short- and long-range information should not be fused in a single stage.
  • Ablation Study: Temporal context improves the baseline from 30.8mm to 29.7mm by making dynamic graph learning more robust to single-frame outliers.Adding weighted graph and dynamic offsets alone provides a 0.3mm gain over baseline.
  • Action Recognition: 89.2% and 87.5% top-1 accuracy are achieved on the X-Set and X-Sub settings of NTU RGB+D 120, surpassing state-of-the-art methods.The proposed HCSF and temporal-aware scheme are integrated into a skeleton-based action recognition framework.

5. Conclusion

The paper presents a skeletal representation learning method combining HCSF with adaptive dynamic graphs to address hard 3D pose estimation. It reports improvements over non-graph and graph-based methods while extending the approach to action recognition.

  • Conclusion: The method combines HCSF for encoding short- and long-range context with dynamic graphs that adapt to input poses rather than fixed topology.The HCSF module retains essential information while reducing irrelevant noise.
  • Conclusion: 10.3% improvement over non-graph methods is reported, alongside enhanced effectiveness of graph-based methods.The paper also applies the proposed representation learning method to skeleton-based action recognition.

Supplementary Material: Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation

The supplementary material provides additional experimental and implementation information supporting the main paper.

  • Supplementary Material: The supplement contains data preprocessing, implementation details, additional experimental results, and ablation studies.

1. 3D Human Pose Estimation

Supplementary experiments analyze hard-pose behavior, PA-MPJPE performance, dynamic graph design, and temporal consistency for 3D pose estimation.

  • Hard Poses: Hard poses are defined as high-error poses, and the hardest 10% differ across methods, showing that hard-pose difficulty is model-specific.As pose difficulty increases, all methods’ errors rise, while the proposed method performs best on the hardest poses selected by other methods.
  • PA-MPJPE Comparison: 10.8% improvement is achieved over method [46] on PA-MPJPE with 2D ground-truth inputs.With detected 2D keypoints, the method improves over from 41.2mm to 39.0mm, a 5.3% improvement.
  • Dynamic Graph Ablation: The dynamic graph uses learned connectivity and offsets to represent action-related relations, while temporal consistency improves robustness to single-frame outliers.The ablation reports a 0.5mm gain from combining the weighted graph with dynamic offsets.
  • Dynamic Graph Ablation: Temporal-aware graph learning decreases MPJPE from 34.0mm to 33.5mm in the dynamic-graph ablation.The temporal scheme is introduced because single-frame uncertainty can make dynamic graph learning unstable and misleading.

2. Skeleton-based Human Action Recognition

The action-recognition experiments evaluate the proposed skeletal graph components on NTU RGB+D, including dynamic graphs, hierarchical channel squeezing, multiple hops, and temporal aggregation. Ablations show gains from HCSF and temporal-aware graph learning, with performance depending on hop structure and temporal context.

  • Experimental Setup: NTU RGB+D 60 evaluation uses 25-joint skeleton sequences under Cross-Subject and Cross-View settings.The dataset contains 60 actions, 40 subjects, and three cameras.
  • Experimental Setup: The network uses nine cascaded HCSF–temporal-convolution blocks with residual connections, followed by global average pooling and a fully connected layer.Block output dimensions progress from 64 to 256 channels.
  • Ablation Study: 0.7% accuracy improvement is obtained from the hierarchical channel-squeezing fusion block across static and dynamic graph settings.The best result uses HCSF with dynamic graph Ak, and d=1/8 performs best among the tested squeezing ratios.
  • Ablation Study: MixHop and MS-Hop over-mix useful and noisy information because they do not distinguish distant from close neighbors.The proposed hierarchical channel-squeezing approaches outperform the other multiple-hop baselines.
  • Ablation Study: 95.1% accuracy remains stable within approximately 0.2% across tested S and L settings, indicating robustness to graph noise.Performance first improves and then stabilizes as the number of hops increases; the final setting is S=5, L=7, d=1/8.
  • Ablation Study: 0.6% improvement over the single-frame setting is achieved using three frames for temporal convolution in dynamic graph learning.Longer temporal contexts degrade performance, making three frames the optimal tested setting.
Loading 2108.07181v2…