Source-linked AI summary
Richly Activated Graph Convolutional Network for Action Recognition with Incomplete Skeletons
Yi-Fan Song, Zhang Zhang, Liang Wang
TL;DR
Incomplete skeletons are common in real scenarios, while existing methods assume complete joints. RA-GCN uses successive GCN streams and class activation maps to learn from unactivated joints; it achieves comparable state-of-the-art performance on NTU RGB+D and significantly alleviates deterioration on synthetic occlusions.
Problem
Existing methods assume complete skeletons, although real scenarios can produce incomplete skeletons that are challenging for action recognition.
Source: §1 · pp. 1–2Method
RA-GCN uses multiple GCN streams and class activation maps from preceding streams to guide each new stream toward unactivated discriminative joints.
Source: §1 · p. 2Results
RA-GCN achieves comparable performance to state-of-the-art methods on NTU RGB+D and significantly alleviates performance deterioration on a synthetic occlusion dataset.
Source: §1 · p. 2Takeaways & Limitations
The authors report that discovering more discriminative joints improves model robustness to incomplete skeletons.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Current methods for skeleton-based human action recognition usually work with completely observed skeletons. However, in real scenarios, it is prone to capture incomplete and noisy skeletons, which will deteriorate the performance of traditional models. To enhance the robustness of action recognition models to incomplete skeletons, we propose a multi-stream graph convolutional network (GCN) for exploring sufficient discriminative features distributed over all skeleton joints. Here, each stream of the network is only responsible for learning features from currently unactivated joints, which are distinguished by the class activation maps (CAM) obtained by preceding streams, so that the activated joints of the proposed method are obviously more than traditional methods. Thus, the proposed method is termed richly activated GCN (RA-GCN), where the richly discovered features will improve the robustness of the model. Compared to the state-of-the-art methods, the RA-GCN achieves comparable performance on the NTU RGB+D dataset. Moreover, on a synthetic occlusion dataset, the performance deterioration can be alleviated by the RA-GCN significantly.
1. INTRODUCTION
Skeleton-based recognition uses spatial and temporal information, but prior approaches assume complete skeletons even though real-world occlusion makes incomplete capture common. RA-GCN uses successive streams to discover discriminative joints and is reported to alleviate performance deterioration on synthetic occlusions.
1. INTRODUCTION
- Skeleton-based recognition models spatial and temporal joint information, while ST-GCN applies graph convolution to skeleton structure and temporal dynamics.Other approaches described include RNNs over joint vectors and CNNs over images formed from temporal joint sequences.Source: §1 · p. 1
- Prior methods assume complete skeletons, although real-world occlusion can make complete capture difficult and incomplete skeletons challenging to recognize.The passage gives pedestrians occluded by parked vehicles or other contextual objects as an example.Source: §1 · pp. 1–2
- RA-GCN uses multiple GCN streams and class activation maps to guide each new stream toward discriminative joints not activated by preceding streams.The preceding streams’ activation maps are accumulated as a mask matrix; the method is intended to improve robustness to incomplete skeletons.Source: §1 · p. 2
- RA-GCN achieves comparable state-of-the-art performance on NTU RGB+D and significantly alleviates performance deterioration on a synthetic occlusion dataset.The synthetic dataset partially occludes NTU joints across spatial and temporal dimensions.Source: §1 · p. 2
2. RICHLY ACTIVATED GCN
RA-GCN uses multiple GCN streams to learn discriminative features from skeleton joints, with each stream directed to joints not activated by earlier streams. Its activation module adapts class activation maps to identify joints and guide later streams.
2. RICHLY ACTIVATED GCN
- Each stream masks out joints activated by earlier streams, passes the remaining skeleton data through ST-GCN, and the output concatenates stream features for classification.Preprocessing concatenates the input, temporal differences, and relative coordinates to the center joint.Source: §2 · p. 2
2.1. Baseline Model
- ST-GCN, the baseline, uses spatial and temporal convolutional blocks, with temporal information extracted by a 1 × L convolution.Its blocks include batch normalization, ReLU, residual connections, and adaptive dropout.Source: §2.1 · p. 3
2.2. Activation Module
- The activation module adapts CAM to locate activated skeleton joints using frame and joint positions, then accumulates preceding streams’ maps to guide each new stream.The class used for the activation map is the true class.Source: §2.2 · p. 3
- RA-GCN streams learn from currently unactivated joints, and the illustrated two- and three-stream examples show more activated joints than the baseline.The first stream’s mask is all ones; later stream inputs are formed from the preprocessed skeleton data and the mask.Source: §2.2 · p. 3, Figure 3 · p. 3
3. EXPERIMENTS
On NTU RGB+D, RA-GCN improves over its baseline and compares favorably with prior methods; on synthetic occlusions, it outperforms other models in most tested settings.
3.1. Dataset and Implemental Details
- NTU RGB+D evaluation uses 56,880 videos, 60 classes, 40 subjects, and CS/CV splits; RA-GCN streams are initialized from a pretrained ST-GCN.Source: §3.1 · p. 3
3.2. Experimental results on complete skeletons
- Compared with the baseline, RA-GCN gains 4.4% and 5.2% on CS and CV, respectively, leading methods on CV but trailing PB-GCN by 1.6% on CS.Source: §3.2 · p. 4, Table 1 · p. 4
3.3. Ablation studies
- Best settings are Dmax=2 and L=5 on CS, and Dmax=3 and L=9 on CV.Source: §3.3 · p. 4, Table 2 · p. 4
3.4. Experimental results on incomplete skeletons
- On CS occlusion tests, 3-stream RA-GCN outperforms other models in most settings except part 2, with increasing superiority under temporal occlusion.Source: §3.4 · p. 4, Table 3 · p. 4
4. CONCLUSION
RA-GCN outperforms its baseline and improves robustness, with experiments verifying its effectiveness in occlusion scenarios. The authors propose adding an attention module in future work.
- RA-GCN achieves much better performance than the baseline and improves model robustness.Source: §4 · p. 4
- Experiments on NTU RGB+D verify RA-GCN's effectiveness in occlusion scenarios.Source: §4 · p. 4
- Future work will add an attention module so each stream can focus more on certain discriminative joints.Source: §4 · p. 4