PaperlayerView exact revision on arXiv

Source-linked AI summary

PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition

Kun Su, Xiulong Liu, Eli Shlizerman

arXiv:1911.12409v1cs.CVcs.LGeess.IV

TL;DR

The paper addresses action recognition methods that rely heavily on labeled data and unsupervised approaches that often require RGB+D inputs. It proposes PREDICT & CLUSTER, an encoder-decoder recurrent model with decoder-weakening strategies for learning separable skeleton representations, and reports performance above prior unsupervised methods, including parity with supervised skeleton recognition in a benchmark setting.

  • Problem

    Current action-recognition methods rely on labeled training data, while existing unsupervised methods often require camera and depth inputs rather than skeleton keypoints alone.

  • Method

    PREDICT & CLUSTER trains an encoder-decoder recurrent network to predict or regenerate skeleton sequences while weakening decoder training so the encoder learns separable hidden-state representations.

  • Results

    The model learns distinctive action features on three benchmark datasets and outperforms prior unsupervised methods, including unsupervised RGB+D methods on cross-view tests.

  • Takeaways & Limitations

    Unsupervised skeleton-based recognition can achieve high performance without action labels and can be competitive with supervised skeleton-based recognition on the reported tests.

Abstract

from arXiv · show

We propose a novel system for unsupervised skeleton-based action recognition. Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions. Our system is based on an encoder-decoder recurrent neural network, where the encoder learns a separable feature representation within its hidden states formed by training the model to perform prediction task. We show that according to such unsupervised training the decoder and the encoder self-organize their hidden states into a feature space which clusters similar movements into the same cluster and distinct movements into distant clusters. Current state-of-the-art methods for action recognition are strongly supervised, i.e., rely on providing labels for training. Unsupervised methods have been proposed, however, they require camera and depth inputs (RGB+D) at each time step. In contrast, our system is fully unsupervised, does not require labels of actions at any stage, and can operate with body keypoints input only. Furthermore, the method can perform on various dimensions of body keypoints (2D or 3D) and include additional cues describing movements. We evaluate our system on three extensive action recognition benchmarks with different number of actions and examples. Our results outperform prior unsupervised skeleton-based methods, unsupervised RGB+D based methods on cross-view tests and while being unsupervised have similar performance to supervised skeleton-based action recognition.

1. Introduction

Action recognition commonly depends on extensive labeled data, motivating an unsupervised approach based only on skeleton sequences. PREDICT & CLUSTER trains an encoder-decoder to learn action representations, strengthened by decoder-weakening strategies, and reports strong benchmark performance.

  • Large-scale action recognition datasets require many labeled examples, while annotation can be impractical and subjective for ambiguous movements.
  • Skeleton keypoints contain less information than RGB+D but can isolate actions and provide robust movement-specific features.
  • PREDICT & CLUSTER replaces supervised classification with sequence prediction or regeneration so encoder hidden states capture action features.
  • Two decoder strategies, Fixed Weights and Fixed States, weaken decoder optimization and guide the encoder toward stronger sequence representations.
  • Across three skeleton-based and RGB+D benchmarks, the combined strategies produce robust representations and outperform prior unsupervised methods.

2. Related Work

Prior action-recognition research is dominated by supervised RGB, depth, and skeleton methods, while unsupervised approaches learn representations through prediction or regeneration. PREDICT & CLUSTER addresses the scarcity of unsupervised skeleton methods by weakening decoder training for larger, multi-view and multi-subject datasets.

  • Action recognition assigns class labels to frame sequences, with skeleton-only methods offering lower data requirements and potential real-time operation.
  • Supervised skeleton recognition includes Lie-group, recurrent, and convolutional approaches that classify body-joint sequences.
  • Unsupervised methods avoid sequence labels and commonly learn action representations by predicting future frames or regenerating input sequences.
  • Prior unsupervised RGB+D methods combine depth, optical flow, or camera-view discrimination to improve encoded representations.
  • Unsupervised skeleton methods use motion prediction and Seq2Seq or GAN architectures, but remain scarce because coordinate-based representations are difficult to learn.
  • PREDICT & CLUSTER replaces masked decoder inputs with strategies that improve encoder learning and weaken the decoder for larger multi-view, multi-subject datasets.

3. Methods

P&C preprocesses skeleton sequences into view-invariant, fixed-length inputs, then learns action representations with a bidirectional encoder-decoder and classifies them using compressed features. Its decoder variants constrain training to improve encoder-side clustering, with training behavior monitored through regeneration loss and accuracy curves.

  • Pre-processing: Skeleton sequences are transformed into a view-invariant coordinate system using rotation and translation based on body-joint geometry.The transformation uses the initial-frame spine, root, and hip positions to define the coordinate frame.
  • Pre-processing: Sequences are down-sampled to a fixed maximum length and zero-padded when shorter than that length.
  • Encoder-decoder architecture: A multilayer bidirectional GRU encoder processes the full keypoint sequence, and its concatenated final state initializes a unidirectional GRU decoder.The decoder regenerates the input sequence, and the regeneration error trains the network.
  • Decoder strategies: Fixed Weights makes the decoder unconditional and randomly weighted, forcing the encoder to represent sequence information in its final state.Only the encoder is trained in this configuration, making it computationally efficient for KNN classification.
  • Decoder strategies: Fixed States replaces each decoder hidden state with the encoder’s final state and adds residual input-output connections to preserve gradients.
  • Feature classification: The final encoder state is compressed by a feature-level autoencoder, then classified with k = 1 KNN using cosine similarity.The classifier operates on the autoencoder’s middle-layer feature vector without learning additional action-placement weights.

4. Experimental Results and Datasets

P&C is evaluated on three datasets containing 3D skeleton sequences, with varied action counts, views, subjects, and recording conditions. The experiments use a common implementation and compare the system with supervised and unsupervised approaches.

  • Implementation: P&C preprocesses keypoint sequences with view-invariant transformation, down-sampling to at most 50 frames, and coordinate normalization to [−1, 1].The reported architecture uses a 3-layer Bi-GRU encoder and a decoder, with additional feature-level auto-encoding.
  • Implementation: The feature-level auto-encoder maps a 2048-dimensional input through fully connected layers to a 256-element final feature vector.The six-layer mapping is 2048 → 1024 → 512 → 256 → 512 → 1024 → 2048.
  • Datasets: The evaluation covers NW-UCLA, UWA3D, and NTU RGB+D, which contain 10, 30, and 60 action classes respectively.The datasets vary in subjects, views, joints, sample counts, and protocols, including cross-view and cross-subject testing on NTU RGB+D.
  • Comparison: Table 1 compares P&C with supervised skeleton, unsupervised RGB+D, and unsupervised skeleton action-recognition methods.The comparison spans the three benchmark datasets and evaluates action-recognition performance across method types.

5. Evaluation and Comparison

P&C is evaluated with 1-nearest-neighbor accuracy across multiple configurations and benchmarks. Its learned features separate action classes, while performance varies with dataset size, class count, and evaluation protocol.

  • Evaluation: P&C uses a K-nearest neighbors classifier with k = 1 to compute action-recognition accuracy across the experiments.Reported configurations include a randomly initialized encoder, the full system with Fixed States, and feature-level auto-encoding.
  • Results: The learned NW-UCLA features form well-separated class clusters in t-SNE, while the method achieves average accuracy above 83%.Three NW-UCLA actions—pick up with two hands, drop trash, and sit down—are recognized with nearly 100% accuracy.
  • Results: Recognition becomes more difficult as the number of action classes increases, while larger training sets can produce smoother and better performance.NTU RGB+D has more classes than UWA3D but approximately 40 times more data, and achieves better performance in the reported comparison.
  • Comparison: P&C outperforms previous unsupervised methods on NW-UCLA and exceeds the first three supervised methods listed in the corresponding comparison.The comparison includes both RGB+D-based and skeleton-based unsupervised methods.
  • Comparison: On UWA3D, P&C Fixed States and Fixed Weights with auto-encoding reach approximately 60%, while only VA-RNN-Aug performs better among the listed supervised skeleton methods.On NTU RGB+D cross-view testing, P&C outperforms prior unsupervised methods and is on par with ST-LSTM.
  • Overall comparison: Using one architecture across all three datasets, P&C outperforms the prior unsupervised skeleton method and most supervised skeleton methods, with strongest comparisons on cross-view tests.On the large-scale cross-subject test, it also outperforms some supervised skeleton and unsupervised RGB+D methods.

6. Conclusion

The paper presents an unsupervised skeleton-based action-recognition model whose decoder-training strategies improve encoder representations. Across three benchmarks, the model learns distinctive action features and outperforms prior unsupervised methods.

  • Conclusion: P&C weakens decoder training so the encoder learns more separable hidden-state representations.The conclusion attributes enhanced performance to novel strategies that weaken the decoder and train the encoder.
  • Conclusion: Across three benchmark datasets, P&C learns distinctive action features and outperforms prior unsupervised methods.The reported conclusion concerns the effectiveness of the unsupervised model for skeleton-based action recognition.
Loading 1911.12409v1…