PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning

Siyuan Yang, Jun Liu, Shijian Lu, Meng Hwa Er, Alex C. Kot

arXiv:2108.01959v3cs.CV

TL;DR

Skeleton action recognition methods often rely on large, costly labeled datasets, and unsupervised skeleton representation learning remains challenging. This paper stacks sequences into 3D skeleton clouds, colorizes points by temporal and spatial order, and learns from those labels with an auto-encoder. The authors report that their method outperforms unsupervised and semi-supervised approaches and is on par with supervised methods.

  • Problem

    Many supervised skeleton-action methods require large numbers of costly labeled samples, while unsupervised skeleton representation learning remains challenging.

    Source: §1 · p. 1

  • Method

    The method stacks all frames’ joints into a 3D skeleton cloud, colorizes points according to temporal and spatial order, and learns features by repainting the cloud.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    The method outperforms state-of-the-art unsupervised and semi-supervised skeleton action recognition methods by large margins and performs on par with supervised methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support skeleton-cloud colorization as an approach to unsupervised skeleton action representation learning from unlabeled data.

    Source: §5 · p. 8

Abstract

from arXiv · show

Skeleton-based human action recognition has attracted increasing attention in recent years. However, most of the existing works focus on supervised learning which requiring a large number of annotated action sequences that are often expensive to collect. We investigate unsupervised representation learning for skeleton action recognition, and design a novel skeleton cloud colorization technique that is capable of learning skeleton representations from unlabeled skeleton sequence data. Specifically, we represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. We evaluate our skeleton cloud colorization approach with action classifiers trained under different configurations, including unsupervised, semi-supervised and fully-supervised settings. Extensive experiments on NTU RGB+D and NW-UCLA datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins, and it achieves competitive performance in supervised 3D action recognition as well.

1. Introduction

Supervised skeleton action methods require many costly labeled sequences, while unsupervised representation learning remains challenging. The paper recasts learning as skeleton-cloud colorization and reports performance above unsupervised and semi-supervised methods, and on par with supervised methods.

1. Introduction

  • Supervised skeleton methods require many labeled samples that are costly to collect, and unsupervised skeleton representation learning remains challenging.The introduction notes only a few prior works on learning from unlabeled skeleton data, often by reconstructing sequences from encoded features.Source: §1 · p. 1
  • The paper treats each sequence as a 3D skeleton cloud and colorizes its points by temporal and spatial order to create self-supervision from unlabeled data.A point-cloud auto-encoder learns spatial-temporal features from the color labels by repainting the cloud.Source: §1 · pp. 1–2
  • The method outperforms state-of-the-art unsupervised and semi-supervised skeleton action recognition methods by large margins, with performance on par with supervised methods.The authors identify this as the outcome of their extensive experiments.Source: §1 · p. 2
  • The authors frame unsupervised action representation learning as a skeleton-cloud repainting task and describe it as the first conversion of this problem to that task.The paper presents colorization-based self-supervision as its central contribution.Source: §1 · p. 2

2. Related work

Prior skeleton-action methods use recurrent, convolutional, or graph networks, but generally depend on labeled data; unsupervised skeleton work is comparatively limited. This paper contrasts frame-by-frame processing with stacked, colorized skeleton clouds for unsupervised spatial-temporal learning.

2. Related work

  • Earlier skeleton-action methods use RNNs, CNNs, and GCNs, while the cited approaches generally require labeled data that is costly to collect.The paper positions unsupervised representation learning as a way to mitigate the labeling constraint.Source: §2 · pp. 2–3
  • Unsupervised action-recognition work largely focuses on RGB or RGB-D video, with examples using sequence prediction, reconstruction, or view-invariant representations.The cited skeleton-specific attempts include masked-sequence regeneration, motion generation, and decoder weakening.Source: §2 · p. 3
  • Unlike prior methods that process skeleton sequences frame by frame, this paper stacks joints into a colored skeleton cloud for unsupervised spatial-temporal representation learning.Its colorization scheme uses information encoded in the cloud rather than extracting temporal features from ordered frame sequences.Source: §2 · p. 3

3. Method

The method stacks skeleton sequences into 3D point clouds and uses colorization to encode temporal, spatial, and person-level information. Encoder-decoder streams learn representations by repainting colorized clouds, after which classifiers are trained under different supervision settings.

3.1. Data Processing

  • Stacking all frames' joint coordinates yields a raw skeleton cloud of N=T×J 3D points.Source: §3.1 · p. 3

3.2. Skeleton Cloud Colorization

  • Temporal and spatial colorization assign RGB features according to frame order and joint index, respectively, to encode temporal dependencies and spatial relations.The temporal color scheme gives adjacent frames similar colors; spatial colors vary with joint order.Source: §3.2 · pp. 3–4
  • Person-level colorization assigns red to the first person's joints and blue to the second person's joints to represent interaction information.The scheme addresses interactions involving two persons.Source: §3.2 · p. 4

3.3. Repainting Pipeline

  • Three encoder-decoder streams repaint temporal-, spatial-, and person-colorized targets; the temporal stream uses a DGCNN encoder, a FoldingNet decoder, and Chamfer-distance reconstruction.Temporal repainting colors odd-frame points, while spatial repainting colors odd-indexed joints; the other sampled points remain uncolored.Source: §3.3 · p. 5

3.4. Training for Skeleton Action Recognition

  • A linear classifier uses the learned encoder features; unsupervised training freezes the encoder, while semi-supervised and supervised settings fine-tune it with the classifier.The classification loss is cross-entropy.Source: §3.4 · pp. 5–6

4. Experiments

Experiments on NTU RGB+D and NW-UCLA assess the colorization approach across unsupervised, semi-supervised, and supervised settings. The reported comparisons favor it over prior unsupervised and semi-supervised methods, with competitive supervised performance.

4. Experiments

  • The experiments evaluate whether skeleton-cloud colorization learns useful action-recognition representations in unsupervised, semi-supervised, and supervised settings.The evaluation uses NTU RGB+D and NW-UCLA.Source: §4 · p. 6

4.1. Datasets

  • NTU RGB+D uses cross-subject and cross-view benchmarks, while NW-UCLA trains on camera views V1 and V2 and tests on V3.NTU RGB+D includes 56,880 sequences; NW-UCLA contains 1,494 samples from 10 subjects across 10 action classes.Source: §4.1 · p. 6

4.3. Comparison with the state-of-the-art methods

  • After unsupervised pretraining and supervised fine-tuning, the method performs well on NW-UCLA and NTU RGB+D, outperforming prior unsupervised-pretrained methods and matching state-of-the-art supervised methods.The paper notes large margins over prior unsupervised-pretrained works on NTU RGB+D.Source: Table 5 · p. 7, Table 6 · p. 7, §4.3 · p. 8
  • Across settings, the method outperforms prior unsupervised and semi-supervised methods and achieves performance comparable to state-of-the-art supervised methods.The comparisons cover NTU RGB+D and NW-UCLA.Source: §4.3 · pp. 7–8
  • On NTU RGB+D, TS outperforms prior unsupervised methods, especially under cross-subject evaluation; TS + SS and TS + SS + PS outperform the state of the art on cross-subject and cross-view protocols.On NW-UCLA, TS and TS + SS also consistently outperform the state of the art.Source: §4.3 · p. 7

4.4. Ablation Study

  • All three colorization strategies improve over the raw-cloud baselines across evaluated settings, although person-level colorization trails the other streams on NTU RGB+D.Person-level colorization improves overall performance when combined with the other two streams.Source: Table 7 · p. 7, Table 8 · p. 8, Table 9 · p. 8

4.3. Comparison with the state-of-the-art methods

  • In semi-supervised evaluation, all three configurations outperform the state of the art on NTU RGB+D; on NW-UCLA, TS and TS + SS do so, with TS + SS winning by large margins.The labeled training subsets range from 1% to 40%, with the available ratios differing by dataset.Source: §4.3 · p. 7

4.4. Ablation Study

  • On NTU RGB+D cross-subject evaluation, temporal-stream performance rises from 65.7% to 71.6% when 50% of points receive color information.The authors frame the ratio as a balance between repainting and spatial-temporal ordered feature learning.Source: §4.4 · p. 8

5. Conclusion

The conclusion presents skeleton cloud colorization as an approach to unsupervised skeleton-action representation learning from unlabeled data. Experiments demonstrate superior unsupervised action-recognition performance.

  • The proposed method achieves superior unsupervised action-recognition performance.Source: §5 · p. 8
  • It forms a 3D skeleton cloud by stacking skeleton sequences and colorizes each point by temporal and spatial order.Source: §5 · p. 8
  • The method learns spatio-temporal features from joint colors using unlabeled data.Source: §5 · p. 8

Related papers

Loading 2108.01959v3…