PaperlayerView exact revision on arXiv

Source-linked AI summary

Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognition

Yi-Fan Song, Zhang Zhang, Caifeng Shan, Liang Wang

arXiv:2008.03791v2cs.CV

TL;DR

Skeleton action-recognition models are vulnerable to incomplete or noisy joints, so the paper proposes RA-GCN, an ordered multi-stream GCN guided by class activation maps. RA-GCN distributes complementary features across joints and achieves comparable standard-dataset performance while alleviating degradation on synthetic occlusion and jittering data.

  • Problem

    Skeleton-based action-recognition models lack robustness to noisy or incomplete skeleton data, including cases where joints are occluded or disturbed.

  • Method

    RA-GCN uses ordered GCN streams whose accumulated class activation maps direct later streams toward joints left unactivated by earlier streams.

  • Results

    RA-GCN achieves comparable performance to state-of-the-art methods on NTU RGB+D 60 and 120 and significantly alleviates deterioration on synthetic occlusion and jittering datasets.

  • Takeaways & Limitations

    Distributed redundant and complementary features over skeleton joints improve robustness to non-standard skeletons within the evaluated degradation settings.

Abstract

from arXiv · show

Current methods for skeleton-based human action recognition usually work with complete skeletons. However, in real scenarios, it is inevitable to capture incomplete or noisy skeletons, which could significantly deteriorate the performance of current methods when some informative joints are occluded or disturbed. To improve the robustness of action recognition models, a multi-stream graph convolutional network (GCN) is proposed to explore sufficient discriminative features spreading over all skeleton joints, so that the distributed redundant representation reduces the sensitivity of the action models to non-standard skeletons. Concretely, the backbone GCN is extended by a series of ordered streams which is responsible for learning discriminative features from the joints less activated by preceding streams. Here, the activation degrees of skeleton joints of each GCN stream are measured by the class activation maps (CAM), and only the information from the unactivated joints will be passed to the next stream, by which rich features over all active joints are obtained. Thus, the proposed method is termed richly activated GCN (RA-GCN). Compared to the state-of-the-art (SOTA) methods, the RA-GCN achieves comparable performance on the standard NTU RGB+D 60 and 120 datasets. More crucially, on the synthetic occlusion and jittering datasets, the performance deterioration due to the occluded and disturbed joints can be significantly alleviated by utilizing the proposed RA-GCN.

I. INTRODUCTION

Skeleton-based action recognition remains vulnerable to noisy or incomplete skeletons, motivating RA-GCN, which distributes complementary joint features across streams and evaluates robustness under synthetic degradations.

  • Motivation: Skeleton-based models can deteriorate heavily when informative joints are occluded or disturbed in real scenarios.Occlusion may arise from self-occlusion, other people, or contextual objects.
  • Proposed approach: RA-GCN uses ordered GCN streams to learn complementary discriminative features from joints left unactivated by preceding streams.Accumulated class activation maps guide each subsequent stream toward previously unactivated joints.
  • Contributions: Normalization activation expands the activated scope, producing better and more interpretable activation maps than the prior Softmax-based masks.The change is intended to activate more joints within each stream.
  • Contributions: Additional cross-entropy regularizations are applied to individual streams so their features can be learned more effectively.
  • Evaluation: The study extends synthetic evaluation to four occlusion types and two jittering datasets alongside NTU RGB+D 60 and 120.The expanded tests cover spatial and temporal occlusion and Gaussian-noise joint disturbances.

II. RELATED WORK

Related work spans skeleton-based representations, graph models, occlusion handling, and salient-region exploration; RA-GCN builds on these directions to address degraded skeleton data.

  • Graph-based models: Graph-based methods model skeleton joints as connected graphs to capture spatial relationships that CNN- and RNN-based approaches handle less naturally.
  • Occlusion: Partial joint occlusion can substantially reduce existing action-recognition performance, while relatively few studies address noisy or incomplete skeleton data.
  • Salient regions: Prior salient-region studies either constrain attention or erase salient regions to obtain complementary information in other vision tasks.

III. MODEL ARCHITECTURE

RA-GCN preprocesses skeleton sequences into complementary geometric and motion features, then distributes masked inputs across streams that focus on different activated joints.

  • Multi-stream network: RA-GCN uses multiple streams so that discriminative information remains available when joints activated by an earlier stream are occluded.
  • Multi-stream network: The model concatenates outputs from all GCN streams to produce the final class prediction.
  • Stream filtering: Each stream filters the preprocessed sequence with a mask matrix that records currently unactivated joints before GCN processing.The masks are determined by accumulated activation maps from preceding streams.
  • Data preprocessing: Preprocessing forms relative joint coordinates, temporal displacements, and their concatenation with the original coordinates.Relative coordinates subtract the middle-spine center, while temporal displacements use consecutive-frame differences.

B. Richly Activated GCN

The baseline ST-GCN layer performs spatial graph convolution over distance-based joint neighborhoods, followed by temporal convolution and standard deep-learning blocks.

  • Baseline Model: The baseline model consists of ten graph convolutional layers.
  • Baseline Model: ST-GCN aggregates features from neighboring joints using distance-based graph partitions and learnable edge-importance matrices.The normalization term balances neighbor contributions, while each distance-specific adjacency matrix has a learnable matrix.
  • Baseline Model: The spatial graph convolution is followed by a temporal convolution with window size L to model motion over time.
  • Baseline Model: BatchNorm, ReLU, residual connections, and dropout are applied within the ST-GCN layer to stabilize and regularize processing.Dropout uses probability 0.5 between the spatial and temporal convolutional blocks.

2) Activation Module:

The activation module extends CAM from CNNs to skeleton graphs, identifying joints activated by each stream and masking them before subsequent streams process the sequence.

  • Activation Module: CAM scores are adapted from image coordinates to frame and joint indices, locating class-specific activation over skeleton sequences.The ground-truth class is used to form each stream’s joint score map.
  • Activation Module: A threshold δ and a Heaviside step function convert joint score maps into binary activation maps.
  • Activation Module: Each stream receives only joints not activated by preceding streams through accumulated mask matrices and element-wise filtering.The first stream uses an all-one mask, while later masks exclude previously activated joints.
  • Activation Module: The masking design aims to make successive streams explore discriminative features across all skeleton joints.

3) Loss Function:

RA-GCN extends supervision beyond the total network by adding losses for individual streams, encouraging more effective feature learning across the multi-stream model.

  • Loss Function: The loss function supervises each individual stream in addition to the output of the whole model.The formulation includes stream outputs ˆy_s, the overall output ˆy, ground truth y, and the number of streams S.

IV. EXPERIMENTAL RESULTS

Experiments evaluate RA-GCN on standard NTU RGB+D benchmarks and synthetic occlusion and jittering settings designed to test robustness to incomplete or noisy skeletons.

  • Dataset: NTU RGB+D 60 contains 56,880 videos across 60 classes and is evaluated using cross-subject and cross-view top-1 recognition rates.
  • Dataset: NTU RGB+D 120 extends the benchmark to 114,480 videos and 120 classes, with cross-subject and cross-setup evaluations.
  • Dataset: Synthetic occlusion datasets set selected joints to zero across spatial and temporal dimensions to represent incomplete skeletons.The datasets use the NTU 60 cross-subject and NTU 120 cross-setup benchmarks.
  • Dataset: Synthetic jittering datasets add Gaussian noise to randomly selected joints, using N(0, 0.12) and N(0, 0.052).The jittering operation occurs before preprocessing, and the models are trained with standard skeletons.

B. Implementation Details

The implementation uses specified optimization, architecture, and training procedures, including pretraining a one-stream model before fine-tuning the multi-stream RA-GCN.

  • Optimization: The model uses an initial learning rate of 0.1, decays it by 10 every 20 epochs, and trains for 60 iterations with SGD momentum and weight decay.Dropout is set to 0.5; the cited implementation passage gives momentum 0.9 and weight decay 10^-4.
  • Evaluation setup: The implementation evaluates parameter settings on NTU 60, whose comparison is organized around different RA-GCN configurations.
  • Architecture: The first four ST-GCN layers output 64 channels, the middle three output 128, and the last three output 256.
  • Training procedure: A one-stream RA-GCN is pretrained on preprocessed skeleton data to establish the baseline stream before the multi-stream model is fine-tuned.Each stream's mask matrix is initialized to all ones.

C. Parameters Setting

The experiments tune receptive-field and activation parameters, assess component importance, and compare RA-GCN with standard-dataset baselines and SOTA methods.

  • Parameters Setting: D = 2 and L = 5 are optimal on CS, whereas D = 3 and L = 9 are optimal on CV; larger receptive fields can cause over-smoothing.
  • Parameters Setting: The activation threshold δ controls the number of activated joints, with δ = 0.3 giving the best reported CS accuracy for the two-stream model.
  • Ablation Studies: Removing the activation module lowers accuracy by 1.2%, while omitting pretraining causes a 1.5% decline; replacing the thresholding mechanism yields 85.8% accuracy.
  • Ablation Studies: The full preprocessing representation combining original coordinates, relative coordinates, and temporal displacements outperforms temporal displacements alone.
  • Stream analysis: RA-GCN’s accuracy growth becomes moderate beyond three streams, consistent with its goal of discovering redundant representations rather than maximizing standard-dataset accuracy.
  • Standard-dataset results: On NTU RGB+D 120, RA-GCN reaches 81.1% on CSub and 82.7% on CSet, while on NTU RGB+D 60 it remains close to leading methods.

F. Experimental Results on Occlusion Datasets

Across synthetic frame, part, block, and random occlusions, RA-GCN generally preserves recognition performance better than comparison methods, though critical-joint loss remains a failure case.

  • Frame occlusions: Frame occlusions simulate temporal gaps by removing subsequences of 10 to 50 frames, and RA-GCN shows significant superiority over the compared methods.The advantage of three-stream RA-GCN over the baseline increases as more frames are occluded.
  • Part occlusions: Part occlusions target the left arm, right arm, two hands, two legs, and torso; RA-GCN has a large advantage over other SOTA methods.Three-stream RA-GCN is especially advantageous when the two hands or torso are unavailable.
  • Block occlusions: Block occlusions remove joints behind horizontal lines, and RA-GCN obtains the best accuracies in most experiments.
  • Random occlusions: Under random joint loss with occlusion probabilities from 0.2 to 0.6, RA-GCN greatly alleviates performance deterioration while comparison models degrade rapidly.The cited analysis attributes conventional models' degradation to their dependence on an integrated graph structure.
  • Failure cases: When an important joint such as the right arm is occluded, actions such as handshaking may not be inferable from other joints, causing RA-GCN to fail.

G. Experimental Results on Jittering Datasets

Jittering experiments add Gaussian noise at increasing joint probabilities and report that three-stream RA-GCN maintains a large advantage over other models.

  • Dataset construction: The jittering datasets add Gaussian noise with joint disturbance probabilities from 0.02 to 0.10 and evaluate both NTU 60 and NTU 120 benchmarks.
  • Results: Three-stream RA-GCN outperforms other models by a large gap on the NTU 60-based jittering dataset.
  • Results: As jittering probability increases, the performance gap between three-stream RA-GCN and the baseline also increases.

H. Results Analysis

RA-GCN expands joint activation beyond the baseline and discovers additional discriminative joints as streams increase. This distributed activation supports robustness to occluded or disturbed joints, although adding streams does not always improve accuracy.

  • The baseline primarily activates the two arms, whereas RA-GCN also activates auxiliary joints such as the swaying body and stepping legs.These additional joints contribute discriminative information for the throwing action.
  • More activated joints are discovered as the number of RA-GCN streams increases from one to three.Figure 6 compares activated and unactivated joints across the baseline, two-stream, and three-stream models.
  • The distributed activation enables modest recognition performance when some joints are occluded or disturbed.The authors report greater robustness to degraded input than other models.
  • Additional streams do not always improve accuracy because action categories often contain only a limited number of discriminative joints.The authors report that three streams are sufficient in this work.
Loading 2008.03791v2…