Source-linked AI summary
MS$^2$L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
Lilang Lin, Sijie Song, Wenhan Yan, Jiaying Liu
TL;DR
Prior single-reconstruction approaches may overfit and fail to learn generalizable skeleton features for action recognition. MS2L integrates motion prediction, jigsaw puzzle recognition, and contrastive learning, and explores training strategies for using those features. Experiments across three datasets and multiple supervision settings show that the resulting representation extractor outperforms the baseline significantly.
Problem
Prior single reconstruction tasks may overfit and produce features that are not generalizable for skeleton-based action recognition.
Method
MS2L jointly uses motion prediction, jigsaw puzzle recognition, and contrastive learning to model skeleton dynamics, temporal patterns, and intrinsic feature representations.
Results
The model is evaluated across three datasets and unsupervised, semi-supervised, fully supervised, and transfer-learning settings, outperforming the baseline significantly.
Takeaways & Limitations
Multi-task self-supervision provides a powerful feature extractor for skeleton-based action recognition within the evaluated datasets and training configurations.
Abstract
from arXiv · showhide
In this paper, we address self-supervised representation learning from human skeletons for action recognition. Previous methods, which usually learn feature presentations from a single reconstruction task, may come across the overfitting problem, and the features are not generalizable for action recognition. Instead, we propose to integrate multiple tasks to learn more general representations in a self-supervised manner. To realize this goal, we integrate motion prediction, jigsaw puzzle recognition, and contrastive learning to learn skeleton features from different aspects. Skeleton dynamics can be modeled through motion prediction by predicting the future sequence. And temporal patterns, which are critical for action recognition, are learned through solving jigsaw puzzles. We further regularize the feature space by contrastive learning. Besides, we explore different training strategies to utilize the knowledge from self-supervised tasks for action recognition. We evaluate our multi-task self-supervised learning approach with action classifiers trained under different configurations, including unsupervised, semi-supervised and fully-supervised settings. Our experiments on the NW-UCLA, NTU RGB+D, and PKUMMD datasets show remarkable performance for action recognition, demonstrating the superiority of our method in learning more discriminative and general features. Our project website is available at https://langlandslin.github.io/projects/MSL/.
1 INTRODUCTION
Skeleton-based action recognition offers lightweight, viewpoint-robust representations, but prior single-task reconstruction methods may miss high-level spatio-temporal information and overfit. MS2L addresses these issues by jointly optimizing motion prediction, jigsaw puzzle recognition, and contrastive learning, with multiple strategies for transferring learned features to action recognition.
- Skeletons represent people through 3D skeletal-joint coordinates, reducing storage and computation demands while remaining robust to viewpoint, appearance, and surrounding distractions.
- Prior skeleton representation methods use encoder-decoder reconstruction, but detailed coordinate recovery can ignore high-level spatio-temporal information critical for action recognition.
- Single-task learning can overfit to its specific objective, producing features that may be insufficiently discriminative and general for recognizing skeleton sequences.
- MS2L jointly optimizes motion prediction, jigsaw puzzle recognition, and contrastive learning to capture skeleton dynamics, temporal evolution, and feature-space regularity.
- The paper explores moving pretraining and jointly training to transfer self-supervised knowledge to action classifiers, evaluating unsupervised, semi-supervised, fully supervised, and transfer-learning settings.
2 RELATED WORK
Prior work spans self-supervised pretext tasks for images and videos, deep skeleton action-recognition models, and encoder-decoder approaches using unlabeled skeletons. These lines of work motivate MS2L’s multi-task learning of skeleton representations without relying solely on action labels.
- Self-supervised learning extracts representations from unlabeled data using automatically generated supervision rather than expensive human annotations.
- Image self-supervision includes jigsaw puzzle reordering, grayscale colorization, and rotation-angle prediction, while sequential methods predict frame order or model spatio-temporal structure.
- Skeleton action recognition evolved from hand-crafted joint-geometry features toward recurrent, attention-based, and convolutional deep networks.
- Recent skeleton models reduce dependence on action labels through encoder-decoder reconstruction of masked input sequences and improvements intended to make encoded representations more informative.
3 MULTIPLE SELF-SUPERVISED LEARNING (MS2L)
MS2L learns skeleton representations through multiple self-supervised tasks that capture complementary information, then transfers those features to action recognition under several training strategies.
- 3.1 Preliminaries: The framework uses a shared encoder to learn skeleton features and an action classifier to assign sequence-level action labels.The classifier produces a probability distribution over action categories from encoded skeleton sequences.
- 3.2 Multiple Self-Supervised Tasks: Motion prediction models skeleton dynamics by encoding masked, noisy prefixes and reconstructing future frames with mean squared error.The input contains frames through T′, while the prediction target covers frames T′+1 through T.
- 3.2 Multiple Self-Supervised Tasks: Contrastive learning regularizes the feature space by pairing each original skeleton with transformed versions as positives and other samples as negatives.A projection head maps encoded sequences into the contrastive feature space, while cosine similarity compares representations.
- 3.2 Multiple Self-Supervised Tasks: Temporal jigsaw recognition divides each skeleton sequence into equal segments, randomly shuffles them, and predicts the correct permutation.With P segments, the task has P! possible permutations and uses a classification head trained with cross-entropy loss.
- 3.3 Training for Action Recognition: Action recognition can use unsupervised, semi-supervised, or fully supervised classifier training, including moving pretraining and joint optimization strategies.Moving pretraining gradually increases the supervised-loss contribution, while joint training optimizes self-supervised and action-recognition objectives together.
4 EXPERIMENT RESULTS
Experiments evaluate MS2L representations across unsupervised, semi-supervised, fully supervised, transfer-learning, and ablation settings on NW-UCLA, NTU, and PKUMMD. Results show improvements over baselines, benefits from joint multi-task training, and stronger performance from temporal rather than spatial transformations and prediction.
- Evaluation settings: Experiments assess MS2L under unsupervised, semi-supervised, fully supervised, and transfer-learning settings across NW-UCLA, NTU, and PKUMMD.Action recognition accuracy measures representation quality.
- Supervised evaluation: MS2L improves supervised accuracy on NW-UCLA from 83.86% to 85.32% with moving pretraining and 86.75% with jointly training.These results compare the baseline with the two training strategies.
- Supervised evaluation: On PKUMMD, jointly training raises accuracy from 83.49% to 85.17% on part I and from 40.97% to 45.70% on part II.Moving pretraining reaches 84.43% on part I and 42.57% on part II.
- Benchmark comparison: The best NTU performance improves from 78.44% to 78.56%, while the model performs better on NW-UCLA and PKUMMD part I than state-of-the-art methods.It remains competitive on PKUMMD part II and NTU.
- Ablation study: Temporal motion prediction and temporal transformations outperform spatial alternatives, whose masked reconstruction or contrastive transformations can hurt recognition.Spatial prediction may lose information and the GRU backbone explicitly models temporal patterns rather than spatial relations.
- Ablation study: Using all three self-supervised tasks achieves the best ablation performance, while contrastive learning alone outperforms the other single-task alternatives.The authors attribute the multi-task gain to retaining more aspects of the original sequences.
- Training strategy: Moving pretraining keeps self-supervised losses smoother than direct joint fine-tuning, which sharply increases those losses and may damage learned representations.The moving strategy is designed to preserve prior self-supervised knowledge while learning action recognition.
5 CONCLUSION
The paper proposes multi-task self-supervised learning for skeleton-based action recognition, combining complementary tasks to learn general features. Experiments across three datasets show the model outperforms the baseline significantly.
- The proposed approach integrates multiple self-supervised tasks to learn more general skeleton representations than a single reconstruction task.The motivation is to address overfitting in single-task representation learning.
- Motion prediction models skeleton dynamics, while jigsaw puzzle recognition models temporal patterns.
- Contrastive learning regularizes the feature space and helps learn intrinsic skeleton features.
- Comprehensive experiments on three datasets show that the model is a powerful feature extractor that significantly outperforms the baseline.