Source-linked AI summary
Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition
Lilang Lin, Jiahang Zhang, Jiaying Liu
TL;DR
Self-supervised skeleton action recognition methods treat motion and static parts equally, despite the need for adaptive modeling of both. ActCLR mines actionlets against a static anchor and adapts transformations and feature pooling by region. Experiments report remarkable action-recognition performance on NTU RGB+D and PKUMMD.
Problem
Self-supervised skeleton action-recognition methods treat motion and static parts equally and lack adaptive design for the different parts.
Source: §1 · p. 1Method
ActCLR mines discriminative actionlet regions by contrasting sequences with an average-motion static anchor, then adapts transformations and feature pooling to motion and non-motion regions.
Source: §1 · p. 2Results
Takeaways & Limitations
The method attends to sequence motion information while reducing static-region interference in feature extraction.
Source: §5 · p. 8
Abstract
from arXiv · showhide
The self-supervised pretraining paradigm has achieved great success in skeleton-based action recognition. However, these methods treat the motion and static parts equally, and lack an adaptive design for different parts, which has a negative impact on the accuracy of action recognition. To realize the adaptive action modeling of both parts, we propose an Actionlet-Dependent Contrastive Learning method (ActCLR). The actionlet, defined as the discriminative subset of the human skeleton, effectively decomposes motion regions for better action modeling. In detail, by contrasting with the static anchor without motion, we extract the motion region of the skeleton data, which serves as the actionlet, in an unsupervised manner. Then, centering on actionlet, a motion-adaptive data transformation method is built. Different data transformations are applied to actionlet and non-actionlet regions to introduce more diversity while maintaining their own characteristics. Meanwhile, we propose a semantic-aware feature pooling method to build feature representations among motion and static regions in a distinguished manner. Extensive experiments on NTU RGB+D and PKUMMD show that the proposed method achieves remarkable action recognition performance. More visualization and quantitative experiments demonstrate the effectiveness of our method. Our project website is available at https://langlandslin.github.io/projects/ActCLR/
1. Introduction
Prior contrastive methods treat skeleton regions uniformly, although motion regions carry richer action information and uniform transformations can degrade it. ActCLR mines actionlets against an average-motion static anchor and adapts transformations and feature pooling to motion and static regions; experiments report strong self-supervised results.
1. Introduction
- Uniform transformations can degrade motion-correlated information, even though motion regions contain richer action information than static regions.For example, masking hand joints in a hand-raising action can impair its motion information and create semantic inconsistency between positive pairs.Source: §1 · pp. 1–2
- Actionlets are mined as the skeletal regions most discriminative from the dataset’s average motion, used as a static anchor.ActCLR treats the region with the largest difference from this anchor as the region where motion occurs.Source: §1 · p. 2
- ActCLR achieves remarkable self-supervised action-recognition results on NTU RGB+D and PKUMMD compared with state-of-the-art methods.The introduction reports thorough experiments on both datasets but gives no numerical comparison here.Source: §1 · p. 2
- ActCLR applies semantics-preserving transformations to actionlets and stronger transformations to non-actionlet regions.The design aims to maintain semantic consistency in motion regions while improving generalization through stronger transformations elsewhere.Source: §1 · p. 2
- Semantic-aware feature pooling extracts actionlet features to emphasize motion joints without interference from motionless joints.The method distinguishes motion and static regions in feature representation.Source: §1 · p. 2
2. Related Work
The related work reviews skeleton-based recognition methods and contrastive learning approaches. It describes progress in graph-based and attention-based methods, alongside contrastive methods that construct positive and negative pairs through data transformations and other mechanisms.
2.1. Skeleton-Based Action Recognition
- Skeleton-based recognition methods include geometric approaches, hierarchical RNNs, attention mechanisms, and spatial-temporal graph convolution networks.Attention methods select important joints or frames; graph attention adapts features based on spatial composition and temporal dynamics.Source: §2.1 · p. 2
2.2. Contrastive Learning
- Contrastive learning trains representations by making positive pairs more similar than negative pairs, with pair construction a central focus.SimCLR uses data augmentation; another approach stores negative samples in a queue updated during training.Source: §2.2 · p. 2
- Skeleton-based contrastive work includes single-stream MoCo, cross-stream or multi-model methods, extreme augmentations, and motion-consistency learning.These approaches use transformations such as shear, crop, rotation, masking, and flipping to form samples or improve consistency.Source: §2.2 · p. 3
3. Actionlet-Based Unsupervised Learning
ActCLR locates motion regions without labels by comparing each action sequence with an average-motion static anchor. It uses the resulting actionlet to guide region-specific transformations and motion-focused feature pooling in contrastive learning.
3.1. Unsupervised Actionlet Selection
- The actionlet is the region whose difference-activation value exceeds a threshold, localized by contrasting each skeleton sequence with an average-motion static anchor.The average is treated as a static sequence because most actions have no motion in most regions; gradients of sequence-to-anchor similarity produce channel importance weights for the actionlet map.Source: §3.1 · p. 3
3.2. Actionlet-Guided Contrastive Learning
- ActCLR applies motion-adaptive transformations by region: actionlet transformations preserve action information, while stronger additional transformations target non-actionlet regions.Actionlet transformations include spatial, temporal, and spatio-temporal operations; non-actionlet operations include higher-variance random noise and Skeleton Mix variants.Source: §3.2 · p. 4
- Semantic-aware feature pooling weights features over the actionlet region to focus motion representation and reduce interference from static regions.The pooled features are used in the offline stream to provide anchor features for contrastive learning.Source: §3.2 · p. 5
- The framework uses MoCo v2 online and offline encoders, with the offline encoder momentum-updated and its actionlet-pooled features serving as the anchor representation.The online stream uses global average pooling; similarity mining optimizes the feature distributions against a memory bank.Source: §3.2 · p. 5
4. Experiment Results
ActCLR is evaluated on NTU RGB+D and PKUMMD across action recognition, transfer learning, and action segmentation. Ablations examine motion-adaptive transformations, semantic-aware pooling, and actionlet/non-actionlet representations.
4.1. Datasets and Settings
- Evaluation uses NTU RGB+D and PKUMMD; the datasets include NTU 60, NTU 120, and PKUMMD parts I and II.NTU 60 has 56,578 videos and 60 action labels, NTU 120 has 114,480 videos and 120 labels, and PKUMMD covers 52 categories; its Part II has greater view variation and more skeleton noise.Source: §4 · p. 5, §4.1 · p. 5
4.2. Evaluation and Comparison
- The paper reports that its model shows superiority over other methods in unsupervised action-recognition comparisons on NTU, NTU 60, and NTU 120 datasets.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
- On NTU, ActCLR achieves better action-recognition performance than state-of-the-art supervised methods.The passage identifies Table 4 as the comparison.Source: §4.2 · p. 6, Table 4 · p. 7
- ActCLR achieves significant performance in transfer learning on PKUMMD and action segmentation on PKUMMD II xview.The transfer evaluation uses linear evaluation pretrained on NTU 60; segmentation uses linear evaluation pretrained on NTU 60 xview.Source: §4.2 · p. 7, Table 5 · p. 7, Table 6 · p. 7
4.3. Ablation Study
- Motion-adaptive transformations outperform full-region transformations across noise settings and degrade less as noise strength increases.Adding more transformation combinations further enhances feature-space consistency and downstream-task performance.Source: §4.3 · p. 7, Table 7 · p. 8, Table 8 · p. 8
- Semantic-aware feature pooling performs better offline; using it online reduces benefits from non-actionlet transformations.The authors attribute the offline result to better positive-sample features for contrastive learning.Source: §4.3 · p. 8, Table 9 · p. 8
- Actionlet-region accuracy is comparable to whole-skeleton accuracy, while non-actionlet-region accuracy is much lower.The authors interpret this as evidence that the actionlet contains the main motion information.Source: §4.3 · p. 8, Figure 3 · p. 8
- The average motion serves as a static background, while the actionlet selects joints where motion mainly occurs and can vary over time.In the “throw” visualization, the selected joints are mainly the hands.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
5. Conclusions
ActCLR uses actionlets to design motion-adaptive transformations and semantic-aware feature pooling, decoupling action and non-action regions. Experiments report strong performance and support the effectiveness of these designs.
5. Conclusions
- ActCLR combines actionlet-based motion-adaptive transformations and semantic-aware feature pooling with a similarity-mining loss to focus on motion while reducing static-region interference.The conclusion reports that experiments verify the effectiveness of the designs.Source: §5 · p. 8