PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

Haocong Rao, Shihao Xu, Xiping Hu, Jun Cheng, Bin Hu

arXiv:2008.00188v4cs.CV

TL;DR

Existing skeleton-action recognition methods rely on hand-crafted descriptors or substantial labeled data, motivating unsupervised representation learning from unlabeled skeleton sequences. AS-CAL contrasts augmented views with a momentum LSTM and queue, and its evaluation reports improvements over hand-crafted methods and competitive performance against supervised methods.

  • Problem

    Manual labeling for supervised skeleton-action recognition is expensive and non-scalable, while action-label ambiguity further motivates learning from unlabeled data.

  • Method

    AS-CAL learns action representations by contrasting augmented skeleton sequences, using a momentum LSTM key encoder and a queue of encoded keys.

  • Results

    AS-CAL improves existing hand-crafted methods by up to 50% Top-1 accuracy and outperforms many supervised methods across evaluations on four public datasets.

  • Takeaways & Limitations

    Contrastive learning of augmented skeleton sequences provides an unsupervised route to effective action representations evaluated through a frozen-feature linear protocol.

Abstract

from arXiv · show

Action recognition via 3D skeleton data is an emerging important topic in these years. Most existing methods either extract hand-crafted descriptors or learn action representations by supervised learning paradigms that require massive labeled data. In this paper, we for the first time propose a contrastive action learning paradigm named AS-CAL that can leverage different augmentations of unlabeled skeleton data to learn action representations in an unsupervised manner. Specifically, we first propose to contrast similarity between augmented instances (query and key) of the input skeleton sequence, which are transformed by multiple novel augmentation strategies, to learn inherent action patterns ("pattern-invariance") of different skeleton transformations. Second, to encourage learning the pattern-invariance with more consistent action representations, we propose a momentum LSTM, which is implemented as the momentum-based moving average of LSTM based query encoder, to encode long-term action dynamics of the key sequence. Third, we introduce a queue to store the encoded keys, which allows our model to flexibly reuse proceeding keys and build a more consistent dictionary to improve contrastive learning. Last, by temporally averaging the hidden states of action learned by the query encoder, a novel representation named Contrastive Action Encoding (CAE) is proposed to represent human's action effectively. Extensive experiments show that our approach typically improves existing hand-crafted methods by 10-50% top-1 accuracy, and it can achieve comparable or even superior performance to numerous supervised learning methods.

I. INTRODUCTION

AS-CAL addresses the cost and scalability of labeled skeleton-action datasets by learning action representations from unlabeled data through contrastive learning of augmented skeleton sequences. Its momentum LSTM, queue, and CAE representation support consistent action encoding, with strong results across four datasets.

  • Motivation: Supervised skeleton-action methods require extensive annotations, whose manual collection is expensive and difficult to scale.High inter-class similarity can also produce uncertain or incorrect labels.
  • Proposed approach: AS-CAL learns action representations without labels by contrasting augmented instances of the same skeleton sequence.The method uses multiple augmentation strategies to learn invariant action patterns from unlabeled skeleton data.
  • Proposed approach: A momentum LSTM serves as the key encoder through a momentum-based update of the query encoder, producing more consistent action representations.The query and key encoders model long-term dynamics of paired augmented sequences.
  • Proposed approach: The method introduces Contrastive Action Encoding as a representation for action recognition.CAE is formed from action information learned by the proposed encoding process.
  • Results: 50% Top-1 accuracy is the maximum reported improvement over existing hand-crafted methods under linear evaluation across four public datasets.AS-CAL also achieves superior performance to many supervised methods on NTU RGB+D 60 and 120, and outperforms most supervised baselines on SBU and UWA3D.

II. RELATED WORK

Unsupervised skeleton action representation learning addresses the annotation burden of supervised methods. AS-CAL differs from prior approaches by using contrastive learning on augmented skeleton sequences rather than feature engineering or task-specific reconstruction models.

  • Unsupervised skeleton action representation learning remains a newly emerging research topic.
  • Prior unsupervised methods commonly use temporal-order prediction, encoder-decoder reconstruction, generative models, or multiple pretext tasks.
  • AS-CAL learns representations from unlabeled skeleton data through contrastive learning over novel skeleton augmentations, without feature engineering or task-specific reconstruction models.
  • AS-CAL combines a momentum LSTM with a queue-based dictionary to promote more consistent action representations during contrastive learning.

A. Data Augmentation for Skeleton Sequences

The paper treats transformations of one skeleton sequence as instances sharing action patterns. It introduces seven augmentation strategies that perturb geometry, temporal order, noise, detail, or selected joints and channels while pairing query and key sequences with the same strategy or composition.

  • Pattern-invariance means randomly transformed instances of the same skeleton sequence retain similar action patterns for contrastive learning.
  • Rotation applies random rotations around the X, Y, and Z axes, with one main axis sampled from [0, π6] and the other axes from [0, π180].
  • Shear linearly displaces joints in fixed directions, slanting the 3D body-joint shape using randomly sampled shear factors.
  • Reverse augmentation reverses the sequence order at 50% chance, while Gaussian noise adds N(0, 0.05) perturbations to joint coordinates.
  • Gaussian blur smooths joint coordinates with a length-15 kernel and σ sampled from [0.1, 2.0], applied at 50% chance.
  • Joint masking zeros selected joints in random frames, whereas channel masking zeros one coordinate axis to create a 2D projection sequence.
  • Query and key sequences are generated from the same input using the same augmentation strategy or strategy composition before encoding.

B. Augmented Skeleton based Contrastive Action Learning (AS-CAL)

AS-CAL maximizes agreement between augmented instances of the same skeleton sequence through contrastive learning. Its design combines augmentation-based pattern invariance with a momentum LSTM and queue-based dictionary.

  • AS-CAL maximizes agreement between different augmented instances of the same skeleton sequence using an effective contrastive loss.
  • The method uses a momentum LSTM and a queue-based dictionary to support consistent contrastive action learning.
  • The proposed approach is organized around an overview of AS-CAL followed by detailed descriptions of its technical components.

1) Working Flow of AS-CAL:

AS-CAL encodes two augmented versions of a skeleton sequence with an LSTM query encoder and a momentum-updated key encoder, whose smoother evolution promotes consistent key representations.

  • Working Flow: AS-CAL samples query and key sequences through two random augmentations and encodes them with separate LSTM-based query and key encoders.The key encoder processes the key sequence while the query encoder processes the query sequence.
  • Momentum LSTM (mLSTM): The momentum LSTM updates key-encoder parameters using a weighted average of the previous key parameters and current query parameters.The update uses θk ← mθk + (1−m)θq, with m controlling the update speed.
  • Momentum LSTM (mLSTM): Only the query encoder is optimized by back-propagation, while the key encoder changes through the momentum update.This makes the key encoder evolve more smoothly than the query encoder.
  • Momentum LSTM (mLSTM): Lower key-encoder evolution speed with 0.999 ≤ m < 1 benefits contrastive learning compared with full or fast updates.The paper reports that this setting reduces differences among key encoders across iterations and encourages consistent key representations.

3) Temporal Average Pooling:

AS-CAL temporally averages LSTM hidden states to form query and positive-key representations, then uses a queue of encoded keys as a reusable negative dictionary for InfoNCE learning.

  • Temporal Average Pooling: Temporal average pooling aggregates hidden states across time to produce the query representation q and corresponding positive key representation k+.The representations summarize global action encoding information from augmented sequences.
  • Queue-Based Dictionary: At each training step, the current mini-batch of encoded keys is enqueued and the oldest mini-batch is removed.All preceding keys in the queue serve as negative keys for the new mini-batch.
  • Queue-Based Dictionary: The queue is more memory-efficient than a memory bank and maintains immediate mini-batches for a more consistent dictionary.The paper states that AS-CAL with the queue-based dictionary outperforms existing contrastive paradigms in empirical evaluation.
  • Contrastive Learning: InfoNCE maximizes similarity between q and its matched positive key while contrasting them against negative keys stored in the queue.Dot product measures similarity, and τ adjusts the contrastive-learning temperature.
  • Contrastive Action Encoding: The pre-trained query encoder is frozen during linear evaluation, and temporal average pooling of the original sequence yields CAE.CAE is the query representation used as the final action representation for recognition.
  • Contrastive Action Encoding: Among evaluated representations, CAE is reported as comparable to the concatenated q-and-k representation.The comparison includes final hidden states, key representations, CAE, and CAE+.

D. The Entire Approach

The AS-CAL pipeline transforms one skeleton sequence twice, encodes the resulting query-key pair, pools their hidden states, and trains with momentum updates, queued negatives, and InfoNCE.

  • D. The Entire Approach: Two random augmentations based on the same strategy or composition generate the query and key sequences from an input skeleton sequence.The augmented sequences are then processed by the query and momentum-based key encoders.
  • D. The Entire Approach: Temporal average pooling converts the encoder hidden states into the positive pair q and k+ for contrastive learning.The query and key representations are compared using the InfoNCE loss.
  • D. The Entire Approach: During training, the key encoder follows the query encoder through momentum updates, while the InfoNCE loss guides contrastive learning.For recognition, a linear classifier is trained with cross-entropy on CAE q.

IV. EXPERIMENTS

The experiments evaluate unsupervised pre-training and linear evaluation across four public skeleton-action datasets spanning large- and small-scale settings.

  • IV. EXPERIMENTS: Experiments cover NTU RGB+D 60, NTU RGB+D 120, SBU Kinect Interaction, and UWA3D Multiview Activity II.The first two are large-scale datasets, while the latter two are small-scale datasets.
  • IV. EXPERIMENTS: NTU RGB+D 60 contains 60 action classes and 56,578 skeleton sequences, with Cross-Subject and Cross-View evaluation protocols.The Cross-Subject split uses 40,091 training samples and 16,487 testing samples.
  • IV. EXPERIMENTS: NTU RGB+D 120 contains 120 actions from 106 subjects across 113,945 skeleton sequences, with Cross-Subject and Cross-Setup protocols.Its Cross-Subject split uses 63,026 training sequences and 50,919 testing sequences.
  • IV. EXPERIMENTS: SBU contains 8 interaction types in 282 short videos, and evaluation uses 5-fold cross-validation with 3D skeletons containing 15 joints.All skeleton sequences are used to train the model.
  • IV. EXPERIMENTS: UWA3D contains 30 actions performed by 10 subjects across four views and is challenging because of high inter-class similarity and view diversity.The dataset contains 1,075 action sequences.
  • IV. EXPERIMENTS: The experimental protocol separates unsupervised pre-training without skeleton labels from linear evaluation of the learned representation.The learned representation is assessed on the action-recognition task.

1) Unsupervised Pre-training:

AS-CAL is pre-trained on unlabeled skeleton sequences, then evaluated by training a linear classifier on frozen encoders. Experiments span four datasets and compare CAE/CAE+ with hand-crafted, supervised, and unsupervised methods.

  • Unsupervised Pre-training: AS-CAL pre-trains LSTM query and momentum-LSTM key encoders on unlabeled skeleton sequences.Pre-training uses SGD for 60 epochs; data are normalized and augmented only during unsupervised training.
  • Unsupervised Pre-training: Linear evaluation trains a classifier on frozen AS-CAL encoders to measure the learned action representations.The encoders retain their pre-trained parameters during classifier training.
  • Unsupervised Pre-training: Experiments cover NTU RGB+D 60, NTU RGB+D 120, SBU, and UWA3D datasets.The evaluation includes two large-scale and two small-scale skeleton-action datasets.
  • Unsupervised Pre-training: CAE+ improves over three existing unsupervised methods by 5.9%-23.9% accuracy on NTU RGB+D 60.On C-View, CAE+ exceeds Li et al. by 10.0% accuracy.
  • Unsupervised Pre-training: AS-CAL achieves superior performance to depth-image-based methods despite using smaller skeleton inputs.The comparison includes methods identified as depth-image based.

2) Comparison with Hand-Crafted and Supervised Methods:

The comparisons examine AS-CAL against hand-crafted and supervised methods while varying representations, projection heads, encoders, and contrastive-learning hyperparameters. Results favor temporal averaging, no projection head, larger queues, and a momentum coefficient of 0.999.

  • Comparison with Hand-Crafted and Supervised Methods: CAE and CAE+ surpass hand-crafted methods by 8.4%-19.9% on C-Sub and 12.0%-23.4% on C-View of NTU RGB+D 60.The compared hand-crafted baselines include Skeletal Quads and Lie group.
  • Comparison with Hand-Crafted and Supervised Methods: AS-CAL improves over RNN, GRU, and LSTM baselines by at least 6.7% accuracy across all 12 UWA3D view settings.The UWA3D dataset has limited training samples, high inter-action similarity, and frequent self-occlusions.
  • Projection Heads and Projection Output Dimensions: TAP improves accuracy by 20%-25% over No TAP across projection-head and output-dimension settings.The study interprets TAP as a more effective way to aggregate global action encoding information.
  • Projection Heads and Projection Output Dimensions: The No Head model performs best, while the linear projection head substantially outperforms the nonlinear head.The paper attributes added projection transformations to potential action-information loss.
  • Layers and Hidden Units of Encoder: A 2-layer LSTM with 256 hidden units performs comparably to the best 320-unit configuration and is selected for all experiments.The choice targets more compact representations with less training cost.
  • Contrastive-Learning Hyperparameters: Larger queues consistently improve performance, and momentum coefficient m = 0.999 gives the best action-recognition performance on NTU RGB+D datasets.The selected temperature is τ = 0.06 for most cases.

B. Comparison of Different Data Augmentations

Augmentation choices materially affect AS-CAL: most augmentations improve over the original sequence, compositions can improve further, and Reverse plus Shear is consistently strongest.

  • Different Data Augmentations: 3.4%-10.2% accuracy gains result from applying most single augmentations instead of the original sequence.The exception is the “CM” augmentation, whose contrastive-loss curve fluctuates and remains high.
  • Different Data Augmentations: Most two-augmentation compositions further improve AS-CAL by up to 10% accuracy.The augmentations are applied sequentially in different manners.
  • Different Data Augmentations: Double Shear transformations degrade performance, while Reverse plus Shear consistently achieves the best recognition performance.The paper associates double Shear with drastic body-shape changes that make discriminative patterns harder to extract.
  • Different Data Augmentations: Multiple compositions based on Reverse and Shear exceed 50% Top-1 accuracy on NTU RGB+D 60 C-Sub.These results are generally higher than the two-augmentation compositions.
  • Contrastive Dictionary: AS-CAL uses a queue-based contrastive dictionary whose larger size improves accuracy and requires less memory than a memory bank.The queue reuses proceeding encoded keys to form a more consistent dictionary.

D. Evaluation of Different Action Representations

The paper evaluates CAE and CAE+ against alternative action representations and also examines few-label adaptation. Temporal averaging produces stable representations, while insufficient fine-tuning with few labels can reduce performance.

  • Different Action Representations: CAE achieves the best performance on NTU RGB+D 120 and is within 1%-2% accuracy of CAE+ on NTU RGB+D 60.CAE+ concatenates the query representation q and key representation k.
  • Different Action Representations: CAE is constructed by temporally averaging hidden states learned by the query encoder.The representation is evaluated through linear classification after unsupervised pre-training.
  • Different Action Representations: Temporal averaging in CAE yields over 30% accuracy improvement over last-hidden-state representations.The last hidden states achieve only 1%-2% Top-1 accuracy on NTU RGB+D 120, whereas CAE remains stable and competitive.
  • Semi-Supervised Learning: Insufficient fine-tuning with few labels degrades AS-CAL performance, whereas 50% labeled data enables semi-supervised improvement.The study evaluates labeled fractions of 1%, 10%, and 50%.
  • Future Work: The paper identifies adding pretext tasks and theoretically analyzing skeleton augmentations as future research directions.It also proposes efficient encoders and multimodal extensions.
Loading 2008.00188v4…