PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition

Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, Runwei Ding

arXiv:2112.03590v1cs.CV

TL;DR

Existing contrastive methods use normal augmentations that limit exploration of novel movement patterns in skeleton action representations. AimCLR mines information from extreme augmentations through diverse positives, gentle distribution matching, and nearest-neighbor expansion, and 3s-AimCLR performs favorably against state-of-the-art methods across evaluation protocols.

  • Problem

    Normal augmentations produce similar positive samples, limiting exploration of novel movement patterns, while labeling large-scale skeleton datasets is costly.

  • Method

    AimCLR combines extreme augmentations and EADM for diverse positive samples, D3M Loss for gentler distribution matching, and NNM to expand the positive set.

  • Results

    3s-AimCLR performs favorably against state-of-the-art methods under a variety of evaluation protocols, with observed higher-quality action representations.

  • Takeaways & Limitations

    The framework supports learning more general, robust, and suitable action representations from abundant information introduced by extreme augmentations.

Abstract

from arXiv · show

In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and the Energy-based Attention-guided Drop Module (EADM) are proposed to obtain diverse positive samples, which bring novel movement patterns to improve the universality of the learned representations. Second, since directly using extreme augmentations may not be able to boost the performance due to the drastic changes in original identity, the Dual Distributional Divergence Minimization Loss (D$^3$M Loss) is proposed to minimize the distribution divergence in a more gentle way. Third, the Nearest Neighbors Mining (NNM) is proposed to further expand positive samples to make the abundant information mining process more reasonable. Exhaustive experiments on NTU RGB+D 60, PKU-MMD, NTU RGB+D 120 datasets have verified that our AimCLR can significantly perform favorably against state-of-the-art methods under a variety of evaluation protocols with observed higher quality action representations. Our code is available at https://github.com/Levigty/AimCLR.

1 Introduction

Skeleton-based action recognition is motivated by broad applications and advances in sensing and pose estimation, while existing supervised and contrastive approaches face data-labeling and augmentation limitations. AimCLR addresses these issues by mining abundant information from extreme augmentations, attention-guided dropping, distribution-divergence minimization, and nearest-neighbor positives.

  • Skeleton-based action recognition has become a significant computer-vision branch as depth sensors and pose-estimation algorithms have advanced.
  • Supervised methods rely on numerous labeled samples, but labeling large-scale datasets is particularly costly, motivating learning from unlabeled skeleton data.
  • Existing contrastive methods use normal augmentations to construct similar positives, limiting exploration of novel movement patterns.
  • Stronger augmentations can introduce abundant semantic information and improve representation generalizability, but extreme changes may alter the original sequence identity.
  • The framework also targets underused drop mechanisms and the unreasonable treatment of all memory-bank samples as negatives in contrastive learning.
  • AimCLR combines extreme augmentations, EADM, D3M Loss, and NNM to introduce novel movement patterns, gently minimize distribution divergence, and expand positive samples.

2 Related Work

Related work spans supervised skeleton recognition, general contrastive self-supervised learning, and self-supervised skeleton-based action recognition. These lines of research provide the modeling and training context for AimCLR, which uses ST-GCN as its encoder.

  • Supervised Skeleton-based Action Recognition: Supervised skeleton recognition progressed from hand-crafted features to RNN, CNN, and GCN-based methods for modeling skeleton sequences.
  • Supervised Skeleton-based Action Recognition: AimCLR adopts the widely used ST-GCN as its encoder to extract skeleton features.
  • Contrastive Self-Supervised Representation Learning: General contrastive self-supervised learning develops pretext tasks and uses mechanisms such as queue-based memory banks and momentum updates.
  • Self-supervised Skeleton-based Action Recognition: Self-supervised skeleton methods include reconstruction, weakened-decoder training, skeleton colorization, and contrastive approaches such as AS-CAL and SkeletonCLR.

3 AimCLR

AimCLR extends skeleton contrastive learning with extreme augmentations, attention-guided dropping, distributional divergence minimization, and nearest-neighbor mining. These components introduce diverse movement patterns while making the resulting positive-sample relationships more reasonable.

  • Augmentations: Extreme augmentations combine spatial, temporal, and spatio-temporal transformations to introduce movement patterns beyond normal Shear and Crop augmentations.The extreme set combines eight augmentations, including Spatial Flip, Rotate, Axis Mask, Temporal Flip, Gaussian Noise, and Gaussian Blur.
  • Energy-based attention-guided dropping: EADM computes parameter-free energy-based attention and drops important spatial-temporal features to encourage more general and robust representations.Lower neuron energy indicates greater distinctiveness from surrounding neurons; attention is obtained from the inverse energy before masking features.
  • Framework overview: AimCLR uses normal and extreme augmentations to generate multiple views, including an attention-guided dropped extreme view, for contrastive representation learning.The query encoder and MLP produce projected representations for augmented views, while the key encoder supplies momentum-updated representations.
  • Dual distributional divergence minimization: D3M Loss minimizes distributional divergence between normally augmented, extremely augmented, and dropped extremely augmented views instead of directly imposing a one-hot target on extreme views.This addresses the risk that drastic extreme augmentations alter the original sequence identity and degrade performance.
  • Nearest neighbors mining: NNM expands the positive set with nearest neighbors from the memory bank because stored samples are not necessarily all negative samples.The memory bank is a first-in-first-out queue of key representations used across training steps.

4 Experiments

AimCLR is evaluated through ablations and multiple protocols across NTU-60, PKU-MMD, and NTU-120. Results show improved representations and performance over SkeletonCLR and other self-supervised or supervised comparisons.

  • Experimental Settings: Experiments cover NTU-60, PKU-MMD, and NTU-120 using KNN, linear, semi-supervised, and finetuned evaluation protocols.PKU-MMD evaluation uses both subsets, while NTU-60 and NTU-120 use their stated cross-subject, cross-view, or cross-setup protocols.
  • Ablation Study: Replacing normal augmentations with extreme augmentations lowers accuracy, but using both improves xsub and xview accuracy by 2.4% and 2.7%.The normal-augmentation baseline reaches 75.0% on xsub and 79.8% on xview.
  • Ablation Study: Adding EADM improves xsub and xview accuracy by 0.8% and 0.3%, respectively, while subsequent NNM produces the highest accuracy.The authors associate EADM and NNM with more robust and suitable downstream features.
  • Ablation Study: Across three datasets and streams, AimCLR outperforms SkeletonCLR, while three-stream AimCLR also exceeds three-stream CrosSCLR under the same training epochs.At 100 epochs, three-stream AimCLR is reported as better than three-stream SkeletonCLR at 300 epochs.
  • Comparison with State-of-the-art: On NTU-60 linear evaluation, three-stream AimCLR leads three-stream SkeletonCLR by 3.9% on xsub and 4.0% on xview.AimCLR also outperforms specified competing methods with or without cross-stream knowledge mining.
  • Comparison with State-of-the-art: AimCLR leads existing self-supervised methods on PKU-MMD and NTU-120, including 68.2% versus 67.9% on NTU-120 xsub and 68.8% versus 67.1% on xset.It also performs better with 1% or 10% labeled data and exceeds reported comparison methods after finetuning.

5 Conclusion

AimCLR explores novel movement patterns from extreme augmentations through diverse positive samples, gentle distribution-divergence minimization, and nearest-neighbor mining. Experiments report favorable performance and higher-quality action representations across evaluation protocols.

  • AimCLR uses extreme augmentations and an energy-based attention-guided drop module to introduce novel movement patterns and improve representation universality.The D3M Loss minimizes distribution divergence more gently, while nearest-neighbor mining makes positive-set construction more reasonable.

A Data Augmentation

The framework combines normal augmentations with a broader set of extreme spatial, temporal, and spatio-temporal transformations. These augmentations are designed to introduce novel movement patterns while preserving useful action information.

  • Normal Augmentations: Normal augmentation uses Shear spatially and Crop temporally, following SkeletonCLR and CrosSCLR.
  • Extreme Augmentations: Extreme augmentation includes Shear, Spatial Flip, Rotate, Axis Mask, Crop, Temporal Flip, Gaussian Noise, and Gaussian Blur.These cover spatial, temporal, and spatio-temporal transformations.
  • Shear: Shear applies a random linear transformation to the spatial coordinates of body joints.The shear factors are sampled from [−β, β], with β = 0.5.
  • Crop: Crop symmetrically pads a temporal skeleton sequence and randomly crops it back to its original length.The padding ratio is γ = 6.
  • Spatial Flip: Spatial Flip exchanges left and right sub-skeletons while keeping the torso position unchanged, with probability p = 0.5.This uses human-body symmetry to preserve action-class equivalence across sides.
  • Temporal Flip: Temporal Flip reverses the skeleton sequence with probability p = 0.5 to expose movement-order information.
  • Rotate and Axis Mask: Rotate randomly selects a main axis and angle, while Axis Mask randomly zero-masks one axis with probability p = 0.5.Rotation angles are sampled within the specified ranges to reflect camera-position variability.
  • Gaussian Noise and Gaussian Blur: Gaussian Noise adds N(0, 0.01) perturbations, while Gaussian Blur smooths joints using a length-15 kernel with σ ∈ [0.1, 2].Blur is applied with 50% chance.

B Visualization Results

On PKU-MMD part I, t-SNE visualizations show AimCLR embeddings with more compact same-class features and more distinguishable different-class features across three streams.

  • AimCLR produces more compact embeddings within the same class and greater separation between different classes than SkeletonCLR across three streams.The visualization uses fixed t-SNE settings on PKU-MMD part I; † denotes motion stream and ‡ denotes bone stream.
Loading 2112.03590v1…