PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Contrastive 3D Action Representation Learning

Fida Mohammad Thoker, Hazel Doughty, Cees G. M. Snoek

arXiv:2108.03656v1cs.CV

TL;DR

Learning useful 3D skeleton features is difficult because labeled data is scarce and prior self-supervised approaches may retain representation-specific variations. The paper combines inter-skeleton contrast across multiple representations with skeleton-specific spatial and temporal augmentations, and reports state-of-the-art self-supervised performance across multiple downstream tasks on NTU and PKU datasets.

  • Problem

    Large labeled skeleton datasets are difficult to obtain, while prior self-supervised tasks may preserve viewpoint and scale variations rather than emphasize higher-level downstream-relevant semantics.

  • Method

    The method learns invariance to multiple skeleton representations and to skeleton-specific spatial and temporal augmentations through contrastive learning.

  • Results

    The final model outperforms prior state-of-the-art self-supervised methods across multiple downstream tasks on NTU RGB+D 60 and 120 and PKU-MMD.

  • Takeaways & Limitations

    Contrasting sequence-based and graph-based representations together with spatiotemporal augmentations results in the paper’s best action features.

Abstract

from arXiv · show

This paper strives for self-supervised learning of a feature space suitable for skeleton-based action recognition. Our proposal is built upon learning invariances to input skeleton representations and various skeleton augmentations via a noise contrastive estimation. In particular, we propose inter-skeleton contrastive learning, which learns from multiple different input skeleton representations in a cross-contrastive manner. In addition, we contribute several skeleton-specific spatial and temporal augmentations which further encourage the model to learn the spatio-temporal dynamics of skeleton data. By learning similarities between different skeleton representations as well as augmented views of the same sequence, the network is encouraged to learn higher-level semantics of the skeleton data than when only using the augmented views. Our approach achieves state-of-the-art performance for self-supervised learning from skeleton data on the challenging PKU and NTU datasets with multiple downstream tasks, including action recognition, action retrieval and semi-supervised learning. Code is available at https://github.com/fmthoker/skeleton-contrast.

1 INTRODUCTION

The paper targets a useful latent feature space for 3D action understanding without requiring large labeled skeleton datasets. It combines inter-skeleton contrast with skeleton-specific augmentations to learn higher-level spatiotemporal semantics.

  • Motivation: Large labeled skeleton datasets are difficult to obtain, motivating a self-supervised contrastive method for 3D skeleton data.Skeletons provide lightweight, faster, privacy-preserving representations, but learning effective action features traditionally requires substantial annotation.
  • Motivation: Prior self-supervised methods preserve viewpoint and scale variations instead of concentrating on higher-level semantics relevant to downstream tasks.RGB augmentations are also unsuitable because skeleton sequences are sparse and image transformations may not alter the estimated skeleton.
  • Method: Inter-skeleton contrast learns from pairs of different skeleton representations of the same sequence in a cross-contrastive fashion.This enriches the sparse input space and reduces dependence on the nuances of one representation.
  • Method: Skeleton-specific spatial and temporal augmentations generate positive pairs that encourage attention to spatiotemporal dynamics while ignoring viewpoint and exact joint positions.The paper introduces these augmentations because common image augmentations are ineffective for sparse skeleton sequences.
  • Evaluation: A comprehensive evaluation reports considerable improvement over prior methods across challenging downstream tasks.The evaluation covers the learned feature space on multiple tasks rather than a single recognition setting.

2 RELATED WORK

Prior work uses self-supervised pretext tasks, instance discrimination, and supervised sequence- or graph-based encoders for skeleton data. This paper instead combines representation diversity with skeleton-specific contrastive augmentations.

  • Self-Supervised Learning: Self-supervised learning commonly exploits unlabeled data through pretext tasks such as spatial puzzles, rotation prediction, context prediction, inpainting, and colorization.Recent methods also learn similarity between sample pairs through instance discrimination and noise contrastive losses.
  • Skeleton Action Recognition: Supervised skeleton methods use sequence-based or graph-based representations to encode joint co-occurrence and spatiotemporal motion, but require time-consuming action annotations.Sequence-based representations are encoded with CNNs, while graph-based representations use spatial and temporal edges with graph convolutions.
  • Skeleton Self-Supervision: Earlier skeleton self-supervised approaches include masked-input reconstruction, motion prediction, multi-task pretext learning, and adversarial neighbor-relation exploration.These approaches provide several alternatives for learning from skeleton data without fully supervised training.
  • Positioning: Unlike prior approaches relying exclusively on sequence representations and reconstruction or motion-prediction tasks, this paper exploits representation diversity and skeleton-specific spatial and temporal augmentations.The proposed regime uses inter-contrastive learning rather than depending on a single sequence-based representation.

3 SKELETON-CONTRASTIVE LEARNING

The method combines skeleton-specific spatio-temporal augmentations with inter-skeleton contrastive learning to learn action features invariant to nuisance variation while preserving higher-level semantics. It contrasts multiple representations of augmented views so embeddings rely on shared information rather than representation-specific shortcuts.

  • 3.1 Skeleton Augmentations: Existing RGB augmentations are unsuitable for skeleton sequences, motivating spatial and temporal transformations that encode skeleton action dynamics.The target invariances include viewpoint, camera distance, skeleton scale, joint perturbations, action speed, and temporal bounds.
  • 3.1.1 Spatial Skeleton Augmentations.: Pose augmentation applies a 3D shear to every joint and time step, creating positive pairs that vary in viewpoint and camera distance while retaining the pose.The transformation encourages reliance on discriminatory pose semantics rather than camera-related information.
  • 3.1.1 Spatial Skeleton Augmentations.: Joint jittering randomly moves a subset of joints while preserving other connections, encouraging invariance to estimated-skeleton noise and attention to common joint-motion semantics.The selected joints are perturbed to irregular positions across the sequence.
  • 3.1.2 Temporal Skeleton Augmentation.: Temporal crop-resize samples different subsequences and re-samples each to a fixed length, changing start and end times as well as effective action speed.The contrastive task is thereby encouraged to focus on common joint-motion dynamics across sampled temporal periods.
  • 3.1.3 Spatio-Temporal Skeleton Augmentations.: Spatial and temporal transformations are composed by applying temporal crop-resize first and a selected spatial augmentation second, producing locally varied positive pairs that retain high-level action semantics.The spatial operation can be pose augmentation, joint jittering, or a randomized choice; the experiments report that randomizing the composition further improves results.
  • 3.3 Inter-Skeleton Contrast: In inter-skeleton contrast, two augmented views are each encoded using different skeleton representations, and cross-representation query-key pairs are attracted while representation-specific negatives are repelled.This extends single-representation contrastive learning by enriching the input space and encouraging reliance on mutual information and higher-level semantics.

4 EXPERIMENTS

Experiments evaluate the learned features across action recognition, retrieval, semi-supervised recognition, transfer learning, and augmentation and representation ablations. The method consistently improves downstream performance, while combining skeleton representations and augmentations provides the strongest features, with three representations trading further gains for computational cost.

  • 3D Action Retrieval: The method also outperforms alternatives for 3D action retrieval, especially under the more challenging cross-subject and cross-setup protocols.Retrieval assigns classes with a 1-nearest-neighbor classifier using cosine similarity between pre-trained features.
  • Semi-Supervised 3D Action Recognition: The method outperforms previous methods on every semi-supervised benchmark and improves over supervised-only training using the available labeled subset.Semi-supervised training first pre-trains with inter-skeleton contrastive learning, then fine-tunes the final classification layer.
  • Transfer Learning for 3D Action Recognition: Inter-skeleton pre-training transfers well across datasets, particularly from PKU-MMD I to PKU-MMD II, which share the same domain.The transfer setup pre-trains on NTU RGB+D 60 or PKU-MMD I and fine-tunes on PKU-MMD II.
  • Benefit of Skeleton Augmentation: Combining all three spatial and temporal augmentations improves accuracy across input representations, with the X_IMG result about 10% above pose augmentation alone and about 28% above no augmentation.The reported augmentations include pose augmentation, joint jitter, and temporal crop-resize.
  • Intra-Skeleton vs. Inter-Skeleton: Pre-training any two skeleton representations improves features over single-representation training, while all three representations give the best result but incur greater computational cost.X_STG accuracy increases by 6% with X_SEQ and by 5% with X_IMG; the caption states that paired training improves every representation.

5 CONCLUSION

The method combines skeleton-specific augmentations with multiple skeleton representations in contrastive learning to capture spatio-temporal dynamics. It achieves gains over prior self-supervised methods across multiple downstream tasks and datasets.

  • The framework uses novel skeleton augmentations and multiple skeleton representations to learn spatio-temporal dynamics.
  • Invariance to spatio-temporal augmentations and cross-contrast between sequence-based and graph-based representations produce the best action features.
  • The final model outperforms prior state-of-the-art self-supervised methods across multiple downstream tasks on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD.

A APPENDIX

The appendix documents downstream training procedures, supervised comparisons, augmentation hyperparameters, multi-representation experiments, and qualitative results.

  • The appendix details training procedures for each downstream task.
  • It compares the method with supervised approaches for skeleton-based action recognition.
  • It examines hyperparameters for the proposed augmentations.
  • It evaluates combining multiple skeleton representations for downstream action recognition and presents qualitative results.

A.1 Downstream Training Details

Downstream tasks use the pretrained query encoder without its projection head, with task-specific handling for recognition and retrieval.

  • The projection head is removed because it tends to focus on information specific to the pretext task.
  • For 3D action recognition, a classifier is appended to the pretrained query encoder.
  • For 3D action retrieval, the feature space is used directly without a classification head.

A.2 Supervised Approaches

The method is competitive with supervised approaches and improves supervised-only training, particularly on challenging evaluation protocols.

  • The comparison evaluates supervised approaches on the NTU 60 and NTU 120 datasets against the sequence-based query encoder f_SEQ.
  • The method is competitive with many supervised approaches despite using an encoder that is not state-of-the-art.
  • Contrastive pre-training boosts performance over supervised-only training.
  • Inter-skeleton contrast improves supervised-only training especially for the more challenging cross-subject and cross-setup protocols.

A.3 Augmentation Hyperparameter Ablations

The ablations show that augmentation strength matters: jittering about half the joints performs best, while a wider temporal crop range improves downstream classification.

  • Joint jittering: Jittering 10 or 15 joints performs best, whereas jittering 2 or 20 joints is sub-optimal.The number of jittered joints controls the augmented view; very few may be insufficient, while jittering too many degrades performance.
  • Joint jittering: Increasing the number of jittered joints improves downstream 3D action classification performance.
  • Temporal crop-resize: A smaller minimum temporal crop length improves downstream performance by exposing the model to a larger temporal range.The authors therefore use l_min=0.1, corresponding to L_ratio ∈ [0.1, 1.0].
  • Temporal crop-resize: Using a larger temporal-length range improves downstream 3D action classification performance.

A.4 Multi-representation Downstream

Combining representations downstream generally improves performance, and inter-skeleton contrast outperforms intra-skeleton contrast at the same combined-training computational cost. These gains involve a trade-off between pretraining resources and inference model size.

  • Contrastive pretraining: Inter-skeleton contrast outperforms intra-skeleton contrast for every tested representation combination.The comparison is made when combining representations for downstream fine-tuning.
  • Contrastive pretraining: Inter-skeleton and intra-skeleton contrast have the same computational cost when training two representations separately or together.
  • Computational trade-offs: Inter-skeleton contrast requires additional pretraining computation, although inference uses the same number of parameters.Each representation is pretrained with another representation under inter-skeleton contrast, whereas intra-skeleton representations are pretrained alone.
  • Representation combinations: Combining representations improves over a single representation in most cases, with X_SEQ and X_STG achieving the best results.The improvement increases inference-time model size.

A.5.1 Visualization of learned features.

The visualizations indicate that inter-skeleton contrastive learning produces more discriminative feature organization and more relevant action-retrieval neighbors than the comparison method.

  • Action retrieval: The proposed method better focuses on query-action motion than pose similarities when retrieving neighbours.This helps distinguish actions such as kicking, staggering, and hopping on one leg, which can share poses with one leg off the floor.
  • Feature visualization: Inter-skeleton contrastive learning forms denser, more separated clusters for the selected NTU RGB+D 60 action classes.The t-SNE comparison uses 10 randomly selected classes and contrasts the proposed features with those learned by Su et al.
  • Action retrieval: Nearest neighbours retrieved with inter-skeleton contrastive learning generally belong to the same action classes as the query.The retrieval visualization compares the top neighbours from the proposed feature space with those from Su et al.
Loading 2108.03656v1…