PaperlayerView exact revision on arXiv

Source-linked AI summary

Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition

Yi-Fan Song, Zhang Zhang, Caifeng Shan, Liang Wang

arXiv:2106.15125v2cs.CV

TL;DR

Skeleton action-recognition models often extract rich features at prohibitive computational cost. This paper builds EfficientGCN from early-fused input branches, efficient convolutional layers, attention, and compound scaling. On NTU RGB+D 60 and 120, EfficientGCN-B4 achieves SOTA performance with substantially lower model size and computation.

  • Problem

    Current SOTA skeleton action-recognition models are sophisticated and over-parameterized, making training, inference, and large-scale architecture validation costly.

  • Method

    EfficientGCN early-fuses joint, velocity, and bone branches, uses separable convolutional layers and spatial-temporal joint attention, and scales width and depth with a compound coefficient.

  • Results

    EfficientGCN-B4 achieves SOTA performance on NTU RGB+D 60 and 120 while using fewer FLOPs and parameters than other models.

  • Takeaways & Limitations

    EfficientGCN provides efficient, strong GCN baselines for developing more complicated skeleton-based action-recognition models.

  • Takeaways & Limitations

    Cross-view recognition is weaker than several four-stream SOTA models, and noisy frames remain challenging because the approach is not specifically designed for noise.

Abstract

from arXiv · show

One essential problem in skeleton-based action recognition is how to extract discriminative features over all skeleton joints. However, the complexity of the recent State-Of-The-Art (SOTA) models for this task tends to be exceedingly sophisticated and over-parameterized. The low efficiency in model training and inference has increased the validation costs of model architectures in large-scale datasets. To address the above issue, recent advanced separable convolutional layers are embedded into an early fused Multiple Input Branches (MIB) network, constructing an efficient Graph Convolutional Network (GCN) baseline for skeleton-based action recognition. In addition, based on such the baseline, we design a compound scaling strategy to expand the model's width and depth synchronously, and eventually obtain a family of efficient GCN baselines with high accuracies and small amounts of trainable parameters, termed EfficientGCN-Bx, where "x" denotes the scaling coefficient. On two large-scale datasets, i.e., NTU RGB+D 60 and 120, the proposed EfficientGCN-B4 baseline outperforms other SOTA methods, e.g., achieving 91.7% accuracy on the cross-subject benchmark of NTU 60 dataset, while being 3.15x smaller and 3.21x faster than MS-G3D, which is one of the best SOTA methods. The source code in PyTorch version and the pretrained models are available at https://github.com/yfsong0709/EfficientGCNv1.

1 INTRODUCTION

Skeleton-based action recognition needs discriminative features across spatial configurations and temporal dynamics, but leading models are often over-parameterized and costly. EfficientGCN addresses this with early branch fusion, efficient convolutional layers, attention, and compound scaling.

  • Motivation: Skeleton representations compactly encode body motion, but discriminative recognition requires modeling both spatial configurations and temporal dynamics.Skeleton data uses 3D coordinates of multiple joints and is more robust than RGB representations to illumination and viewpoint variation.
  • Motivation: 6.94 million parameters and nearly 4 GPU-days make 2s-AGCN costly, while DGNN exceeds 26 million parameters.These examples illustrate the training and tuning burden of sophisticated multi-stream GCN models.
  • EfficientGCN design: Early-fused MIB combines joint positions, motion velocities, and bone features before the main stream to reduce multi-stream redundancy.The fusion location is selected through exhaustive search, contrasting with conventional late score-layer fusion.
  • EfficientGCN design: BottleLayer, SepLayer, EpSepLayer, and SGLayer extend efficient CNN convolutional structures to GCNs for temporal modeling and lower computational cost.The layers reduce parameter-tuning costs during training and accelerate inference during testing.
  • Results: EfficientGCN-B4 is 5.82× smaller and 5.85× faster than MS-G3D, while EfficientGCN-B4 reaches 92.1% on NTU RGB+D 60 cross-subject recognition.EfficientGCN-B0 also improves relative performance by over 1% versus 2s-AGCN on both NTU RGB+D 60 and 120.
  • EfficientGCN design: Compound scaling configures model width and depth with a scaling coefficient, while omitting resolution scaling because skeleton graph resolution is predefined.The width and depth multipliers are controlled by fixed coefficients and resource assignments.

2 RELATED WORK

Skeleton action recognition evolved from RNN- and CNN-based sequence models toward graph-based methods that represent joint structure. Related efficiency work uses separable convolutions or preprocessing, while attention methods refine spatial and temporal feature weighting.

  • Skeleton action recognition: Early skeleton action-recognition methods used hierarchical RNNs or CNNs to analyze dependencies and classify trimmed skeleton sequences.These approaches later became less prominent because they inadequately represented spatial configurations.
  • Graph-based methods: ST-GCN introduced graph convolution to model skeleton spatial configurations and temporal dynamics synchronously, motivating subsequent multi-stream and multi-scale GCNs.Later work addressed occlusion, multi-adjacency graphs, and dilated temporal convolutions.
  • Efficient architectures: MobileNet and related architectures reduce model size through separable convolutions, establishing an efficiency-oriented design direction.These studies provide precedents for reducing parameters and FLOPs in neural networks.
  • Efficient architectures: Skeleton-specific efficiency methods trade accuracy or preprocessing complexity against speed: CNN blocks can be less accurate, while multi-feature preprocessing reaches 188 sequences/(second*GPU).The cited fast method uses positions, velocities, frame indexes, and joint types as inputs.
  • Attention models: Attention mechanisms include channel-wise and spatial-wise image attention, while action-recognition methods extend attention to discriminative joints across spatial and temporal dimensions.Skeleton-based attention examples include spatial-temporal LSTM modeling and related joint-focused mechanisms.

3 PRELIMINARY TECHNIQUES

This section introduces preprocessing for joint, motion, and bone features, reviews graph convolution, and compares standard with separable convolution for efficient computation.

  • Data preprocessing: EfficientGCN preprocessing uses joint positions, motion velocities, and bone features as three input feature classes.Joint positions combine original and normalized relative positions; motion uses fast and slow velocities; bone features use lengths and angles.
  • Data preprocessing: Relative position features are normalized against a center spine joint and concatenated with the original coordinates.
  • Data preprocessing: Motion features concatenate fast and slow velocity differences for each joint at each time step.Fast velocity uses a two-frame temporal difference, while slow velocity uses a one-frame difference.
  • Data preprocessing: Bone features comprise each bone’s length and angle, calculated using adjacent joints and the three spatial coordinates.
  • Graph convolution: Graph convolution aggregates neighboring joint features using normalized contributions and learnable weights assigned by graph-distance labels.The model uses distance-based partitioning, grouping joints with the same graph distance into subsets sharing a learnable weight function.
  • Separable convolution: Separable convolution factorizes standard convolution into depthwise and point-wise convolutions, reducing computational cost.For feature-map size Df × Df, kernel size Dk × Dk, and channels Cin and Cout, its cost is Dk × Dk × Cin × Df × Df + Cin × Cout × Df × Df.

4 EFFICIENTGCN

EfficientGCN combines early fusion, efficient graph-convolutional layers, compound scaling, and spatiotemporal joint attention to build compact skeleton-action-recognition baselines. The architecture fuses Joint, Velocity, and Bone inputs before extracting features in a main stream, while scaling and attention target efficiency and discrimination.

  • 4.1 Model Architecture: EfficientGCN first constructs an early-fused MIB architecture that combines Joint, Velocity, and Bone branches before a single main stream.The three input branches are concatenated before the main stream, retaining rich input features while reducing parameters and computational complexity.
  • 4.1 Model Architecture: The model's GCN blocks stack spatial graph convolution, temporal convolution, attention, and residual links for feature extraction and optimization.Block depth is defined by the number of temporal-convolution layers, with the first temporal layer using stride 2.
  • 4.2 Convolutional Layers: EfficientGCN extends BottleLayer, SepLayer, EpSepLayer, and SGLayer from CNNs to graph convolution, reducing parameter-tuning costs and accelerating inference.These layers complement the original BasicLayer for temporal dynamics extraction and model compression.
  • 4.3 Scaling Strategy: The compound scaling strategy synchronously expands network width and depth using fixed coefficients, while omitting resolution scaling because skeleton structure is predefined.Width and depth multipliers are controlled by a compound coefficient; α and β are set to 1.2 and 1.35 by grid search.
  • 4.4 Spatial Temporal Joint Attention: ST-JointAtt jointly models spatial and temporal attention to identify informative joints in particular frames rather than weighting joints or frames independently.Frame- and joint-level pooled features produce separate scores whose channel-wise outer product forms sequence-level attention scores.
  • 4.6 Discussion: EfficientGCN's efficiency is attributed mainly to separable temporal convolutions, while its accuracy is associated with compound scaling and ST-JointAtt.The separable convolutions reduce parameters and FLOPs, and the attention module focuses GCN learning on informative joints and frames.

5 EXPERIMENTAL RESULTS

Experiments evaluate EfficientGCN on NTU RGB+D 60 and 120, using ablations to assess temporal layers, attention, inputs, and fusion stages. The studies identify configurations that balance accuracy with computational cost.

  • Experimental Setup: EfficientGCN experiments cover NTU RGB+D 60 and 120, with ablations assessing each major model component.The ablations use EfficientGCN-B0 with SGBlock (rrd = 2) as the baseline.
  • TC Layers: 90.0% accuracy, 2.73G FLOPs, and 0.29M parameters make SGLayer with rrd = 2 the selected trade-off for the baseline.EpSepLayer reaches 90.1% accuracy, but SGLayer has lower standard deviation and better efficiency.
  • Input Branches: The three-input model performs better than models using fewer branches, supporting joint-position, velocity, and bone inputs.Performance improves as the number of input branches increases.
  • Fusion Architecture: Fusing after the second stage provides the best accuracy-parameter balance, whereas score-layer fusion greatly increases model size and computational cost.The early fused MIB architecture is therefore selected for the baseline.

5.4 Comparisons of Compound Scaling Strategies

The study searches compound-scaling coefficients for EfficientGCN-B2 and B4 by varying width and depth together. EfficientGCN-B4 performs best with α = 1.2 and β = 1.35.

  • Scaling Search: The experiments construct five B2 and five B4 models from EfficientGCN-B0 using different width and depth scaling hyper-parameters.α is tested from 1.0 to 1.4, while β is calculated from Eq. 6.
  • Scaling Search: EfficientGCN-B4 obtains its best performance with width coefficient α = 1.2 and depth coefficient β = 1.35.The coefficients are evaluated under the constraint conditions in Eq. 6.

5.5 Comparisons with SOTA Methods

Across NTU RGB+D 60 and 120, EfficientGCN achieves competitive or leading accuracy while emphasizing lower model complexity. Its cross-view weakness is associated with stronger multi-stream and multi-scale alternatives.

  • NTU RGB+D 60: 92.1% X-sub accuracy makes EfficientGCN-B4 outperform other SOTA methods on NTU RGB+D 60.EfficientGCN-B0 reaches 90.2% on X-sub and 94.9% on X-view, while accuracy improves with the scaling coefficient.
  • Attention-Based Comparisons: EfficientGCN-B4 exceeds STA-LSTM by over 10% on both benchmarks and surpasses AGC-LSTM by 2.9% on X-sub and 1.1% on X-view.DC-GCN+ADG is better on X-view but significantly worse on X-sub.
  • Limitations: Four SOTA models achieve slightly higher X-view accuracy than EfficientGCN, while using four-stream fusion or multi-scale graph convolution.Those design choices increase model complexity and computational cost.
  • Discussion: EfficientGCN is presented as a strong baseline whose compound scaling balances accuracy and complexity, with ST-JointAtt contributing to accuracy.The supplied passage attributes the result to scaling and attention design.
  • NTU RGB+D 120: EfficientGCN-B4 outperforms the current SOTA method MST-GCN on NTU RGB+D 120.The supplied passage reports the highest performance among the compared models.
  • Model Complexity: EfficientGCN-B0 beats ST-GCN by 8.7% accuracy while using 5.98× fewer FLOPs and 10.68× fewer parameters.DGNN reaches the same accuracy as EfficientGCN-B0 with about 90× more trainable parameters.
  • Model Complexity: EfficientGCN-B2 achieves 91.4% accuracy with 4.05×10^9 FLOPs and 0.51×10^6 parameters.It is reported as about 4.57× faster and 7.14× smaller than PA-ResGCN-B19.

5.6 Discussion and Visualization

Visualization shows that EfficientGCN-B4 focuses on informative joints and temporally selective movements. Similar hand actions remain difficult because hand-joint information is insufficient and skeleton data can contain noise.

  • Failure Cases: EfficientGCN-B4 improves recognition of wearing-a-shoe and taking-off-a-shoe actions, but similar hand actions remain difficult to distinguish.The difficult hand-action group includes reading, writing, playing with a phone, and typing on a keyboard.
  • Failure Cases: Insufficient representation of the two hands is identified as a main cause of difficulty for subtle hand actions.The paper states that the available hand-joint information is generally insufficient.
  • Failure Cases: Noise in NTU RGB+D 60 skeleton sequences can strongly affect discriminative-feature capture, and EfficientGCN is not specifically designed to handle it.The paper cites noisy frames in the throwing action as an example.
  • Attention Visualization: Attention maps become more selective across stages, adding temporal selectivity at higher-level GCN stages.Darker squares indicate higher attention weights for spatial-temporal joints.
  • Activation Visualization: Activation maps highlight action-relevant body regions, including the left arm for drinking water, throwing, and waving hand.The upper body is emphasized for taking off a jacket, and the left leg for kicking.
  • Attention Visualization: Compared with PartAtt, ST-JointAtt assigns more temporally reasonable weights by selecting informative moving joints in specific frames.For kicking, no joints are activated in the first two frames.

5.7 Generalization of EfficientGCN

EfficientGCN-B4 is evaluated on four open skeleton-based person re-identification benchmarks using established splits and task-specific frame settings, showing potential beyond action recognition.

  • Four open skeleton-based person ReID benchmarks are used: BIWI, IAS-A, IAS-B, and KGBD.
  • The experiments follow the training/testing splits from prior work and use 80 input frames for KGBD and 10 for the other datasets.Longer sequences are split into multiple samples with the same person ID.
  • EfficientGCN-B4 further demonstrates potential for other skeleton-based motion analysis tasks.

6 CONCLUSION

The paper presents efficient GCN baselines that fuse input branches early, use efficient temporal-convolution layers, and scale width and depth together. EfficientGCN-B4 achieves SOTA performance on NTU RGB+D 60 and 120 with lower complexity than other models.

  • EfficientGCN constructs efficient but strong baselines using techniques designed to improve model efficiency.
  • Early fusion of three input branches eliminates redundant parameters relative to other multi-stream models.
  • Four temporal-convolution layers based on bottleneck structures and separable convolution significantly save computational cost.
  • Compound scaling uniformly expands model width and depth, further reducing model complexity.
  • EfficientGCN-B4 achieves SOTA performance on NTU RGB+D 60 and 120 with fewer FLOPs and parameters than other models.

APPENDIX A NETWORK ARCHITECTURE

The appendix specifies how EfficientGCN architectures are generated from initial channels, initial temporal-convolution depths, scaling coefficients, and rounding functions, with architecture tables documenting the resulting baselines.

  • Initial channels are set to {48, 24, 64, 128}, while the four blocks begin with temporal-convolution depths {0.5, 0.5, 1, 1}.
  • Scaled channels and temporal-convolution depths are determined using scaling coefficient φ, initial values, α and β, and a step function.
  • Ceiling and floor functions provide the upward and downward rounding operations used in the scaling procedure.
  • EfficientGCN-B0, EfficientGCN-B2, and EfficientGCN-B4 architectures are calculated from initial channels, initial depths, and rounding functions.
  • Architecture tables describe each block’s settings and output shape, including three input branches, stride-2 operations, and Q action classes.
  • A BasicBlock is fixed without attentions for stable training, and rounding can reduce an initial depth of 0.5 to zero.

APPENDIX B GRID SEARCH FOR RECEPTIVE FIELD

The appendix studies spatial and temporal receptive-field settings through accuracy and complexity comparisons, selecting D = 2 and L = 5 as a balance between accuracy and model complexity.

  • D denotes maximum spatial graph distance and L denotes temporal window size; both directly determine the convolutional receptive field.
  • Accuracy declines obviously when D < 2, while D > 3 is not better than D = 2 or D = 3 because oversized fields cause graph over-smoothing.
  • Temporal window size L must balance model accuracy and complexity, similarly to the spatial-distance choice D.
  • D = 2 and L = 5 are selected because they jointly provide high accuracy and low model complexity.D = 2, L = 11 and D = 3, L = 9 are slightly more accurate but considerably more complex.
Loading 2106.15125v2…