PaperlayerView exact revision on arXiv

Source-linked AI summary

Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition

Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, Qi Tian

arXiv:1904.12659v1cs.CVcs.AI

TL;DR

Fixed skeleton graphs mainly encode local physical dependencies and can miss implicit joint correlations. AS-GCN learns actional and higher-order structural links, combines them with spatial-temporal convolutions, and adds future-pose self-supervision; it improves recognition across two datasets and also predicts future poses.

  • Problem

    Fixed skeleton graphs mainly capture local physical dependencies and may miss implicit correlations between joints important for action recognition.

  • Method

    AS-GCN combines AIM-inferred actional links, higher-order structural links, spatial-temporal graph convolutions, and a parallel future-pose prediction head.

  • Results

    AS-GCN achieves large recognition improvements over previous methods on NTU-RGB+D and Kinetics and shows promising future-pose prediction.

  • Takeaways & Limitations

    Adaptive generalized graphs capture non-local action information, while future-pose prediction preserves detailed patterns for recognition.

  • Takeaways & Limitations

    The actional-link prior is designed to promote sparsity because too many links may capture useless dependencies and confuse action-pattern learning.

Abstract

from arXiv · show

Action recognition with skeleton data has recently attracted much attention in computer vision. Previous studies are mostly based on fixed skeleton graphs, only capturing local physical dependencies among joints, which may miss implicit joint correlations. To capture richer dependencies, we introduce an encoder-decoder structure, called A-link inference module, to capture action-specific latent dependencies, i.e. actional links, directly from actions. We also extend the existing skeleton graphs to represent higher-order dependencies, i.e. structural links. Combing the two types of links into a generalized skeleton graph, we further propose the actional-structural graph convolution network (AS-GCN), which stacks actional-structural graph convolution and temporal convolution as a basic building block, to learn both spatial and temporal features for action recognition. A future pose prediction head is added in parallel to the recognition head to help capture more detailed action patterns through self-supervision. We validate AS-GCN in action recognition using two skeleton data sets, NTU-RGB+D and Kinetics. The proposed AS-GCN achieves consistently large improvement compared to the state-of-the-art methods. As a side product, AS-GCN also shows promising results for future pose prediction.

1. Introduction

Skeleton-based methods often rely on fixed, local physical graphs that miss long-range joint dependencies. AS-GCN addresses this with action-specific and higher-order links, combining spatial-temporal learning with future-pose self-supervision.

  • Motivation: Skeleton data provide efficient, noise-robust 3D joint representations for human action recognition.They can come from depth sensors or video-based pose estimation.
  • Proposed approach: AS-GCN stacks actional-structural graph convolutions with temporal convolutions to learn spatial and temporal features.The architecture adapts to action recognition and future pose prediction.
  • Limitations of prior graphs: ST-GCN captures directly connected joints, but structurally distant joints such as hands and feet during walking may be ignored.Hierarchical diffusion can weaken node features over long ranges.
  • Proposed approach: AS-GCN infers actional links with an encoder-decoder AIM and extends skeleton graphs with higher-order structural links.These links form generalized skeleton graphs for richer joint dependencies.
  • Self-supervision: A parallel future-pose prediction head provides self-supervision intended to preserve detailed action patterns and improve recognition.Recognition and prediction are trained simultaneously.
  • Evaluation: AS-GCN outperforms state-of-the-art approaches on NTU-RGB+D and Kinetics, while also accurately predicting future frames.The paper reports consistently large recognition improvements and promising prediction results.

2. Related Works

Prior skeleton-action methods include hand-crafted and deep-learning approaches. The paper distinguishes its adaptive graph learning by capturing useful non-local action information directly from data.

  • Existing approaches: Skeleton-action recognition has used hand-crafted descriptors or deep models such as recurrent networks to learn action features.Examples include local occupancy, temporal joint covariances, Lie group curves, and temporal dependencies across frames.
  • Adaptive graph learning: Unlike previous methods, the paper learns graphs adaptively from data to capture useful non-local information about actions.This complements fixed physical graph structure with data-driven dependencies.

3. Background

The background represents skeletons as joint-and-bone graphs and partitions neighboring joints for graph convolution. ST-GCN alternates spatial graph convolution with temporal convolution, while higher-order structural links enlarge the receptive field.

  • Skeleton graph notation: A skeleton graph G(V, E) represents body joints as vertices and bones as edges, with adjacency and degree matrices encoding connectivity.The adjacency matrix records whether joint pairs are connected.
  • Skeleton graph notation: Neighboring joints are partitioned into root, centripetal, and centrifugal groups to encode refined location information.The adjacency matrix is correspondingly divided into partition-specific matrices.
  • Input representation: Skeleton data are represented as 3D joint positions X across T frames, with X_t denoting one frame and x_i the position of joint i.These tensors provide the spatial-temporal input for graph-based models.
  • ST-GCN: ST-GCN blocks apply spatial graph convolution followed by temporal convolution to extract joint-level spatial and temporal features.A final fully connected classifier generates predictions.
  • ST-GCN: Partition-specific normalized adjacency matrices and trainable weights control neighborhood aggregation and feature importance.Normalization helps define the graph convolution operation for each partition group.
  • Generalized links: Figure 3 contrasts limited skeleton links with S-links that reach higher-order neighbors and A-links that capture long-range action-specific relations.For walking, the example focuses on links from the left hand to neighboring or correlated joints.

4. Actional-Structural GCN

AS-GCN combines action-specific A-links inferred by an encoder-decoder with higher-order structural links, then integrates both in graph convolutions for spatial-temporal modeling. A parallel future-pose head provides self-supervision during joint training.

  • Generalized Skeleton Graph: The generalized skeleton graph contains structural links derived from body structure and actional links inferred directly from skeleton data.A-links model latent dependencies between arbitrary joints, while S-links represent higher-order structural relationships.
  • A-Link Inference: The AIM encoder estimates A-link states from joint positions across time by alternating information propagation between joints and links.The encoder converts propagated link features into linking probabilities, using Gumbel softmax for approximately categorical link types.
  • A-Link Inference: The AIM decoder predicts future 3D joint positions conditioned on inferred A-links and previous poses, providing the future-prediction pathway.The decoder extracts joint features from A-links and converts them into predicted future positions.
  • Structural Links: Structural graph convolution uses higher-order powers of a degree-normalized transition matrix to reach long-range neighbors beyond ST-GCN’s local diffusion.When L = 1, SGC reduces to the original spatial graph convolution; larger polynomial orders capture longer-range dependencies.
  • Actional-Structural Graph Convolution: ASGC combines actional and structural graph-convolution responses through a convex combination, with λ trading off their relative importance.ASGC extracts spatial features, while a temporal convolution extracts inter-frame features; together they form the AS-GCN block.
  • Multitasking: The future-prediction head decodes backbone features into future 3D joint positions, and joint training improves recognition by promoting self-supervision and avoiding overfitting.The recognition and prediction heads are trained together as a multitask model.

5. Experiments

Experiments evaluate AS-GCN on NTU-RGB+D and Kinetics, including link-type, A-link, prediction-head, feature-visualization, recognition, and future-pose analyses. AS-GCN outperforms competing methods and shows that combining actional and structural links, prediction self-supervision, and adaptive dependencies improve the reported outcomes.

  • 5.1. Data sets and Model Configuration: NTU-RGB+D contains 56,880 skeleton action sequences across 60 classes, while Kinetics contains over 240,000 clips across 400 actions and uses OpenPose-derived 18-joint [x, y, c] features.NTU-RGB+D provides 25-joint 3D coordinates; Kinetics skeletons are estimated from RGB videos.
  • 5.2. Ablation Study: Combining S-links and A-links achieves the best recognition performance, while A-links with the original skeleton reach 83.2% versus 81.5% for first-order S-links.Higher-order S-links or A-links alone also improve recognition performance.
  • 5.2. Ablation Study: A-links vary by action: upper-limb actions connect arms with other body parts, whereas kicking links the active leg strongly to other joints for balancing.The visualized A-links use probabilities larger than 0.9, with wider lines denoting larger probabilities.
  • 5.2. Ablation Study: Three A-link types with ghost-link prior P0 = 0.95 produce the highest recognition accuracy, whereas too few types miss relationships and too many cause overfitting.The model configuration also sets λ = 0.5.
  • 5.2. Ablation Study: Adding the future pose prediction head consistently improves recognition by around 1% by preserving detailed information and introducing self-supervision.The paper notes that sparse actions may depend on detailed motions rather than easily confused coarse profiles.

6. Conclusions

The paper concludes that AS-GCN models skeleton actions with actional dependencies, higher-order structural relationships, and future-pose self-supervision. It reports large recognition improvements and promising future-pose prediction results.

  • 6. Conclusions: AS-GCN combines actional links inferred by an A-link inference module with higher-order structural links in generalized skeleton graphs.The generalized graphs are processed by AS-GCN blocks for action representation.
  • 6. Conclusions: AS-GCN achieves large improvement over previous methods on NTU-RGB+D and Kinetics action recognition.The conclusion also reports promising future-pose prediction results.
  • 6. Conclusions: The future pose prediction head captures more detailed action patterns through self-supervision.

Appendix A: Theorem Proof

The appendix proves that actional-structural graph convolution is linear in its input data. The proof derives this from the linearity of its actional and structural components and their weighted sum.

  • Appendix A: Theorem Proof: The actional-structural graph convolution satisfies aY1 + bY2 = ASGC(aX1 + bX2) for all real a and b.
  • Appendix A: Theorem Proof: The proof establishes linearity by showing that both actional graph convolution and structural graph convolution are linear operations.
  • Appendix A: Theorem Proof: ASGC is a linear summation of AGC and SGC, with their combination expressed through the structural-actional weighting parameter λ.

Appendix B: Model Architectures

This appendix introduces the detailed architectures of the proposed AS-GCN model.

  • Appendix B: Model Architectures: The appendix presents the detailed architectures of the proposed AS-GCN model.

A-links Inference Module (AIM)

The A-link inference module uses an encoder-decoder design to infer action-specific dependencies and predict future joint positions from surrounding links and previous frames.

  • The input skeleton sequence is downsampled or padded to 50 frames, with each joint represented by a 150-dimensional feature vector.The representation concatenates 3D positions across 50 frames.
  • AIM models four types of actional dependencies, including a ghost link, and aggregates link features by averaging.
  • The decoder predicts each joint’s future position conditioned on its surrounding A-links.
  • A GRU-based decoder predicts the future positions of all joints using A-links and previous frames.The GRU hidden feature dimension is 64.

Backbone

The AS-GCN backbone combines actional-structural graph convolution with temporal convolution to extract spatial and temporal action features, while a parallel head predicts future poses.

  • Backbone: The backbone extracts spatial and temporal action features using ASGC and temporal CNN operations.
  • Backbone: The AS-GCN backbone contains nine ASGC blocks operating on joint, feature, and frame dimensions.For the NTU-RGB+D Cross-Subject benchmark, actions use 25 joints, 3D positions, and 300 padded frames.
  • Backbone: The backbone uses A-links and S-links to represent actional and structural dependencies in its graph convolutions.
  • Future action prediction head: The future action prediction head progressively aggregates action features, then regenerates actions while concatenating the last input frame to each later feature map.
  • Future action prediction head: A residual connection produces a [30,1,25] tensor containing predicted joint positions for 10 future frames.

Appendix C: More Future Pose Predictions

The appendix presents additional future-pose predictions for several NTU-RGB+D actions, reporting very low prediction error for the illustrated examples.

  • The illustrated actions include “wipe face,” “throw,” and “nausea or vomiting condition.”
  • The reported predictions for these actions have very low error.
Loading 1904.12659v1…