Source-linked AI summary
Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation
Chao Li, Qiaoyong Zhong, Di Xie, Shiliang Pu
TL;DR
Skeleton-based action recognition and detection require representations of joint co-occurrences and temporal evolution, while prior CNNs are limited by local joint aggregation. The paper proposes hierarchical CNN co-occurrence learning with global cross-joint aggregation, two-stream raw-and-motion inputs, and multi-person fusion, and reports improved performance across recognition and detection benchmarks.
Problem
Prior CNN-based skeleton methods aggregate joint co-occurrences locally, making efficient modeling of long-range joint interactions difficult.
Method
The framework learns point-level joint features independently, aggregates them hierarchically and globally across joints, fuses raw coordinates with temporal differences, and handles multiple persons.
Results
The proposed framework outperforms existing state-of-the-art methods on benchmarks for both action recognition and detection.
Takeaways & Limitations
Global CNN aggregation and hierarchical co-occurrence learning provide a framework for learning skeleton action representations across recognition and detection tasks.
Takeaways & Limitations
The described limitation is that prior CNN models use local aggregation and cannot efficiently learn free parameters for each joint.
Abstract
from arXiv · showhide
Skeleton-based human action recognition has recently drawn increasing attentions with the availability of large-scale skeleton datasets. The most crucial factors for this task lie in two aspects: the intra-frame representation for joint co-occurrences and the inter-frame representation for skeletons' temporal evolutions. In this paper we propose an end-to-end convolutional co-occurrence feature learning framework. The co-occurrence features are learned with a hierarchical methodology, in which different levels of contextual information are aggregated gradually. Firstly point-level information of each joint is encoded independently. Then they are assembled into semantic representation in both spatial and temporal domains. Specifically, we introduce a global spatial aggregation scheme, which is able to learn superior joint co-occurrence features over local aggregation. Besides, raw skeleton coordinates as well as their temporal difference are integrated with a two-stream paradigm. Experiments show that our approach consistently outperforms other state-of-the-arts on action recognition and detection benchmarks like NTU RGB+D, SBU Kinect Interaction and PKU-MMD.
1 Introduction
The paper targets skeleton-based action recognition and detection by learning joint co-occurrences globally through hierarchical CNN aggregation. It combines independently learned point-level features, global cross-joint representations, multi-person fusion, and strong benchmark performance.
- Skeleton joint interactions and combinations are central to characterizing actions, motivating co-occurrence feature learning for recognition and detection.
- Existing CNN-based skeleton methods aggregate co-occurrences locally, limiting efficient modeling of long-range interactions and free parameters for individual joints.
- The framework learns hierarchical features by encoding each joint independently before aggregating joint features globally as convolutional channels.
- Global CNN aggregation is proposed as superior to local co-occurrence modeling for skeleton data.
- The network exploits multi-person feature fusion strategies to scale to activities involving variable numbers of persons.
- The proposed framework outperforms existing state-of-the-art methods on benchmarks for both action recognition and detection.
2 Related Work
Prior CNN approaches learn skeleton features but model joint co-occurrences locally. This work instead separates point-level learning from cross-joint learning and uses CNN aggregation to capture global co-occurrence patterns.
- Earlier skeleton methods used CNNs to encode frame dynamics and joints as image dimensions for action recognition.
- The proposed approach models global co-occurrence patterns with CNN rather than following prior local aggregation strategies.
- A convolution decomposes into local spatial aggregation within each channel and global aggregation across channels.
- The framework separates independent point-level feature learning from cross-joint co-occurrence feature learning.
3 Methods
The framework learns skeleton co-occurrences hierarchically while explicitly modeling temporal motion and fusing multiple persons. It extends the same feature-learning backbone to recognition and temporal detection.
- 3.1 Co-occurrence Feature Learning with CNN: CNN channel aggregation captures global correlations among joints instead of restricting co-occurrence learning to neighboring joints.The method treats joints as channels after tensor reorganization, enabling global response across joints.
- 3.2 Explicit Skeleton Motion: Skeleton motion is represented as the temporal difference of each joint between consecutive frames and processed alongside raw coordinates.The two streams are fused by concatenating feature maps across channels in later network layers.
- 3.3 Hierarchical Co-occurrence Network: Figure 3 specifies convolutional blocks, optional stride-2 max pooling, tensor transposition, and ReLU after selected layers.The two input branches use separate parameters, and their feature maps are concatenated after conv4.
- 3.3 Hierarchical Co-occurrence Network: The network learns point-level joint features first, transposes feature maps to expose joints as channels, and then aggregates higher-level context hierarchically.The point-level stages use 1 × 1 and n × 1 convolutions before the transpose operation.
- 3.4 Multi-person Feature Fusion: Element-wise late fusion generalizes to variable numbers of persons without adding parameters relative to the single-person setting.The evaluated late-fusion operations include concatenation, element-wise maximum, and element-wise mean.
- 3.5 Action Recognition and Detection: The framework supports recognition with softmax classification and temporal detection with appended proposal and regression subnetworks.Detection uses one-dimensional window regression, predicting temporal-window targets rather than two-dimensional bounding-box targets.
4 Experiments
The framework is evaluated through multi-person fusion ablations, recognition benchmarks, temporal detection, and global-versus-local aggregation comparisons. It consistently outperforms prior methods across recognition and detection tasks, while global co-occurrence features particularly benefit long-range joint interactions.
- Experimental Setup: The evaluation covers NTU RGB+D and SBU for action recognition, PKU-MMD for temporal action detection, and an ablation of global co-occurrence aggregation.NTU RGB+D uses cross-subject and cross-view protocols; SBU uses subject-independent five-fold cross-validation.
- Multi-person Feature Fusion: All late multi-person feature-fusion methods outperform early fusion on NTU RGB+D in the cross-subject setting.
- Overall Findings: The framework outperforms state-of-the-art LSTM- and CNN-based methods, scales across dataset sizes, and generalizes from action classification to detection.
- Recognition Results: 7.3% and 3.4% accuracy improvements over the LSTM-based method are achieved on NTU RGB+D cross-subject and cross-view settings, respectively.Against the two-stream CNN method, the improvements are 3.3% and 1.8% in the same settings.
- Recognition Results: 8.2% accuracy improvement over the LSTM-based co-occurrence baseline is achieved on SBU, reaching 98.6% accuracy.The paper reports this as exceeding the previously reported best accuracy of 97.6%.
- Detection Results: 38% mAP improvement over Skeleton boxes is achieved on PKU-MMD cross-subject detection, with additional improvements of 2.2% cross-subject and 0.5% cross-view over another detection method.The detection framework uses temporal proposal segmentation and action classification subnetworks.
- Global versus Local Aggregation: Global co-occurrence features improve most actions, including over 10% absolute gains for wearing a shoe, clapping, wipe face, and take-off a shoe.The analysis reports smaller benefits for actions without obvious joint interaction, such as nausea and typing on a keyboard.
5 Discussion
The discussion argues that whole-body joint context matters for action recognition, including joints unrelated to the salient action. CNNs can encode this background context implicitly through global co-occurrence learning.
- 5 Discussion: Background joints can contribute to recognizing an action by providing contextual information, even when they are not directly involved.For “make a phone call,” the ankle is given as an example of an uninteresting joint whose contribution is encoded implicitly with CNNs.
6 Conclusions
The paper presents an end-to-end hierarchical co-occurrence feature learning framework for skeleton-based action recognition and detection. Experiments show improved performance on both tasks.
- The framework learns hierarchical co-occurrence features for skeleton-based action recognition and detection.It is presented as an end-to-end approach.
- The method uses CNN-based global aggregation to learn joint co-occurrence features.
- A two-stream framework fuses motion features with the learned skeleton representations.
- The HCN model significantly improves performance on both action recognition and detection tasks.
A Exemplary Results
The figures provide exemplary validation-set outputs for action recognition on NTU RGB+D and action detection on PKU-MMD. They show both successful and unsuccessful recognition cases and explain the detection visualization.
- The NTU RGB+D recognition figure shows four correctly recognized validation samples.
- The same figure includes a failure case in its bottom row.
- The PKU-MMD detection figure contains five validation samples with frame indices on the horizontal axis.
- Ground-truth action segments are blue, while detected segments above confidence 0.6 are red and labeled by category ID.