PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-task Self-Supervised Learning for Human Activity Detection

Aaqib Saeed, Tanir Ozcelebi, Johan Lukkien

arXiv:1907.11879v1cs.LGstat.ML

TL;DR

Smartphone HAR needs generalizable representations, but privacy and annotation costs limit large labeled datasets. The paper trains a multi-task temporal convolutional network to recognize signal transformations without semantic labels, and reports performance superior or comparable to fully supervised methods across unsupervised, semi-supervised, and transfer settings.

  • Problem

    Privacy issues and prohibitive annotation costs make large, well-curated labeled HAR datasets difficult to obtain.

  • Method

    A multi-task temporal convolutional network learns sensor representations by recognizing transformations applied to raw input signals as self-supervised tasks.

  • Results

    Performance is consistently superior to or comparable with fully supervised methods across unsupervised, semi-supervised, and transfer-learning settings.

  • Takeaways & Limitations

    Self-supervised features substantially reduce the labeled-data requirement for HAR and support learning from abundant unlabeled sensor data.

  • Takeaways & Limitations

    Real-world deployment on devices remains important for understanding computational, energy, and labeled-data requirements.

Abstract

from arXiv · show

Deep learning methods are successfully used in applications pertaining to ubiquitous computing, health, and well-being. Specifically, the area of human activity recognition (HAR) is primarily transformed by the convolutional and recurrent neural networks, thanks to their ability to learn semantic representations from raw input. However, to extract generalizable features, massive amounts of well-curated data are required, which is a notoriously challenging task; hindered by privacy issues, and annotation costs. Therefore, unsupervised representation learning is of prime importance to leverage the vast amount of unlabeled data produced by smart devices. In this work, we propose a novel self-supervised technique for feature learning from sensory data that does not require access to any form of semantic labels. We learn a multi-task temporal convolutional network to recognize transformations applied on an input signal. By exploiting these transformations, we demonstrate that simple auxiliary tasks of the binary classification result in a strong supervisory signal for extracting useful features for the downstream task. We extensively evaluate the proposed approach on several publicly available datasets for smartphone-based HAR in unsupervised, semi-supervised, and transfer learning settings. Our method achieves performance levels superior to or comparable with fully-supervised networks, and it performs significantly better than autoencoders. Notably, for the semi-supervised case, the self-supervised features substantially boost the detection rate by attaining a kappa score between 0.7-0.8 with only 10 labeled examples per class. We get similar impressive performance even if the features are transferred from a different data source. While this paper focuses on HAR as the application domain, the proposed technique is general and could be applied to a wide variety of problems in other areas.

1 INTRODUCTION

Smartphone HAR benefits from deep models but remains constrained by privacy, annotation costs, and limited labeled data. The paper addresses this through multi-task self-supervised transformation recognition, whose learned features support HAR with limited labels.

  • Privacy sensitivity, annotation costs, and device and usage diversity make large, well-curated HAR datasets difficult to obtain.
  • Self-supervised learning uses auxiliary tasks with labels extracted from data without human intervention to learn semantic representations.
  • The proposed method trains a temporal convolutional network to recognize transformations applied to raw sensor signals as an annotation-free surrogate task.
  • Learned self-supervised features are transferred to activity recognition and improve detection with a small labeled dataset.
  • The study reports significantly better semi-supervised and transfer-learning performance across several public datasets.

2 PRELIMINARIES

The preliminaries frame representation learning, multi-task learning, transfer learning, and semi-supervised learning as foundations for robust HAR with limited labels. They motivate self-supervision as a route beyond manual feature engineering and expensive annotation.

  • 2.1 Representation Learning: Representation learning automatically extracts information useful for downstream machine-learning problems such as classification.
  • 2.2 Multi-task Learning: Multi-task learning jointly optimizes multiple objectives using shared representations and task relationships.
  • 2.2 Multi-task Learning: A multi-task objective generally combines task-specific losses through a weighted linear sum.
  • 2.3 Transfer Learning: Transfer learning reuses previously acquired knowledge to accelerate learning of novel tasks, especially when labeled data are limited.
  • 2.4 Semi-supervised Learning: Semi-supervised learning leverages unlabeled data when labeled-data collection is costly and time-consuming.
  • 2.5 Towards Self-supervision: HAR still faces insufficient labeled data for quality models and difficulties transferring knowledge across related data or tasks.

3 APPROACH

The approach learns general-purpose sensor representations without semantic labels by training a multi-task temporal convolutional network to recognize signal transformations. Shared layers learn generic features while task-specific heads support diverse transformation-recognition objectives, and transferred convolutional weights initialize activity recognition.

  • Overview: The Transformation Prediction Network learns sensor representations by recognizing multiple transformations applied to unlabeled temporal signals.Each task distinguishes original signals from versions altered by a particular transformation.
  • Multi-task design: Each transformation-recognition task uses a separate prediction head, while the network shares a common trunk across tasks.The design encourages shared layers to learn representations useful across transformation types.
  • Signal transformations: Diverse transformations target invariances to noise, amplitude changes, offsets, and temporal variation in sensor signals.Examples include adding random noise and scaling samples within a window.
  • Signal transformations: Multiple auxiliary tasks are intended to produce broader features than signal reconstruction by reducing reliance on trivial low-level input artifacts.The paper contrasts this supervisory signal with autoencoder-based compression.
  • Downstream recognition: After self-supervised pre-training converges, convolutional weights are transferred to an activity-recognition network and kept frozen or partly adapted during supervised training.The transfer connects unlabeled pre-training with downstream activity detection.

4 EVALUATION

The evaluation tests the learned representations across diverse smartphone-based HAR datasets and examines the self-supervised training and downstream activity-recognition pipeline. The study uses multiple public data sources spanning devices, participants, protocols, and activity sets.

  • Evaluation design: The evaluation measures representation quality, feature transferability, and low-data performance across publicly available human activity-recognition datasets.The experiments include self-supervised representations used for downstream activity tasks.
  • Datasets: Six datasets were selected to vary device types, data-collection protocols, participants, and activity-recognition tasks.This dataset diversity is summarized in Table 1.
  • Datasets: The datasets include smartphone and smartwatch inertial signals covering activities such as walking, sitting, standing, stair movement, jogging, and falls.Examples span HHAR, UCI HAR, MobiAct, WISDM, and MotionSense sources.
  • Training procedure: Self-supervised pre-training constructs transformation-labeled examples from unlabeled instances before downstream activity classification.Algorithm 1 creates original and transformed pairs, updates the self-supervised model, then trains the activity classifier with labeled samples.

4.2 Data Preparation and Assessment Strategy

The data-preparation strategy uses consistent windowing, user-based splits, normalization, and dataset-specific sampling choices before evaluating activity-recognition performance. Assessment includes weighted classification metrics and user-split cross-validation where appropriate.

  • Pre-processing: Signals are segmented into 400-sample windows with 50% overlap across datasets.The same window size is used to keep evaluation impartial to this hyperparameter.
  • Data splitting: Users, rather than individual signal windows, define random train-test separation, with 20–30% of users reserved for testing.Training users are further divided into training and validation subsets.
  • Sampling: HHAR transformation data are reduced by sampling 4000 instances per user, while UniMiB receives five times more transformed instances because of its smaller size.These choices reduce computational load for HHAR and expand transformed data for UniMiB.
  • Assessment: Performance is evaluated with Cohen’s kappa and weighted precision, recall, and F-score metrics to address dataset imbalance.User-split five-fold cross-validation is also used where appropriate.

4.3 Results

The self-supervised representations support activity classification across layers, tasks, baselines, and limited-label settings, with multi-task learning and fine-tuning improving downstream performance. They also transfer across related datasets.

  • Quantifying the Quality of Learned Feature Hierarchies: Classifiers trained on self-supervised ConvA, ConvB, and ConvC features assess how representation quality varies across temporal-convolution layers.The three blocks contain 32, 64, and 96 feature maps, respectively, and results average 10 independent runs.
  • Comparison against Fully-Supervised and Unsupervised Approaches: Self-supervised representations outperform existing unsupervised methods and achieve results on par with fully-supervised models.Fine-tuning the last convolutional layer further improves classification, while randomly initialized frozen layers perform poorly.
  • Comparison against Fully-Supervised and Unsupervised Approaches: Multi-task learning improves activity-classification performance over individual self-supervised tasks across evaluated cases.Channel Shuffled, Permuted, and Rotated consistently perform better than other individual tasks, with kappa above 0.60 across activity-recognition problems.
  • Effectiveness under Semi-Supervised Setting: In the low-data regime, self-supervised feature learning significantly outperforms supervised learning on MotionSense, UCI HAR, MobiAct, and HHAR.The comparison uses 2–10 labeled instances per class; fine-tuning the last ConvC layer produces the strongest improvement described for the semi-supervised setting.
  • Evaluating Knowledge Transferability: Self-supervised features improve performance after transfer across datasets and tasks, including a 4-percentage-point kappa gain on UniMiB.On UniMiB, kappa increases from 0.781 to 0.821; WISDM and HHAR improve by 3 percentage points.

4.4 Determining Representational Similarity

The paper examines whether self-supervised representations resemble representations learned directly from activity labels. It uses representational-comparison methods to compare networks and layers.

  • Determining Representational Similarity: The analysis asks whether self-supervised representations are similar to those learned through direct supervision with activity labels.This comparison addresses the relationship between the two feature-learning approaches rather than only their end-task performance.
  • Determining Representational Similarity: SVCCA compares distributed representations by identifying optimal linear relationships between neuron-activation vectors from different networks and layers.The method is applied to activations produced by the trained networks, with dimensionality matched when necessary.

Insights on Representational Similarity with Canonical Correlation.

The representational-similarity analysis uses SVCCA to compare layers across self-supervised and supervised networks. The accompanying visual analyses examine layerwise correspondence and input-time-step importance.

  • Insights on Representational Similarity with Canonical Correlation: Saliency maps identify the time steps that most affect the highest-scoring activity class by using gradients of loss with respect to input time steps.The visualizations use randomly selected MotionSense instances to inspect model predictions.

Visualizing Salient Regions.

Saliency analysis compares which input regions self-supervised and fully-supervised networks use for activity predictions. Their maps generally highlight similar signal regions, including a focused region for Sitting instances.

  • Saliency definition: The saliency score assigns each input element an influence value based on the activity classification loss.The loss is defined as the negative log-likelihood for an input example.
  • Saliency-map comparison: Saliency maps from self-supervised and fully-supervised networks highlight similar input regions important for deciding activity labels.The comparison uses tri-axial accelerometer magnitudes to make influential regions interpretable.
  • Sitting-instance analysis: For Sitting instances, both networks mainly focus on a smaller input region with slightly more variation in signal values.The authors relate this attention pattern to periodic variations and the relatively constant Sitting signal.

Visualization of High-Level Feature Space through t-SNE.

The paper projects global max-pooling representations into two dimensions with t-SNE to inspect their structure. The resulting clusters show correspondence across datasets without using activity labels during embedding.

  • Label usage: Activity labels are used only for final visualization, not to construct the t-SNE embeddings.The embedding technique itself is unsupervised.
  • Projection method: t-SNE projects 96-unit global max-pooling activations from self-supervised and fully-supervised networks into a lower-dimensional space.The method is used to explore patterns and clusters in the learned feature space.
  • Observed structure: The visualized clusters show high correspondence between representations across datasets.In UniMiB, same-class samples cluster near one another while different activities can be separated.

5 RELATED WORK

Related work spans deep temporal representation learning, unsupervised feature extraction, and self-supervised surrogate tasks. The paper distinguishes its approach by learning from completely unlabeled smartphone data and evaluating semi-supervised and transfer settings.

  • Deep representation learning: Convolutional and recurrent networks learn representations directly from temporal signals and have improved activity and stress recognition over hand-crafted features.Deep learning has been applied broadly in ubiquitous computing, pervasive intelligence, health, and well-being.
  • Self-supervised learning: Self-supervised learning uses surrogate tasks with labels readily extracted from data, including transformations, temporal order, and rotation prediction.Such tasks have been studied across images, videos, audio, robotics, and sensor data.
  • Prior time-series methods: Earlier time-series studies used transformations mainly for invariance, augmentation, or synthesis, including wearable-sensor augmentation for Parkinson’s monitoring.These works did not address the paper’s semi-supervised and unsupervised learning settings.
  • Positioning of this work: The paper claims the first self-supervised sensor representation-learning approach for HAR using completely unlabeled data without specialized architecture.It reports features on par with fully-supervised networks, better than autoencoder pre-training, and transferable from related unlabeled data.

6 CONCLUSIONS AND FUTURE WORK

The paper presents self-supervised smartphone HAR using a multi-task temporal convolutional network trained to recognize signal transformations. Across unsupervised, semi-supervised, and transfer settings, the approach is comparable or superior to fully supervised methods and better than autoencoders.

  • Conclusions: The proposed method learns smartphone-HAR representations from unlabeled data by training a multi-task temporal convolutional network to recognize signal transformations.The framework is evaluated in unsupervised, semi-supervised, and transfer-learning settings.
  • Conclusions: The self-supervised features are consistently superior or comparable to fully-supervised methods and significantly better than autoencoder-based unsupervised learning.The strongest improvements occur when only very few labeled instances are available.
  • Future work: Future work includes more sophisticated architectures, domain-specific auxiliary-task selection, architecture search, imbalanced datasets, and real-world device evaluation.The proposed approach is presented as a basis for self-supervised HAR with a few labeled data points.
  • Scope boundary: Evaluation remains bounded by settings where collecting a few labeled examples per activity is feasible.The paper identifies high-class-count datasets and real-world deployment as important unresolved evaluation conditions.

APPENDIX

The appendix reports evaluation protocols for self-supervised representation learning, including convergence tracking, user-split cross-validation, and semi-supervised generalization assessments.

  • APPENDIX: Table 4 evaluates activity recognition using user-split 5-fold cross-validation with no overlap between training and test users.Results are averaged over five folds.
  • APPENDIX: Figure 11 tracks transformation-recognition kappa scores across training epochs using snapshots saved every second epoch.Each snapshot is evaluated on self-supervised test data, which the TPN does not see during training.
  • APPENDIX: Figure 12 reports weighted F-scores for generalization of self-supervised learned features in the semi-supervised setting.Results are averaged over 10 independent runs for each approach.
  • APPENDIX: Figure 13 assesses transferred self-supervised features from the related MobiAct dataset under the semi-supervised setting.Results are averaged over 10 independent runs for each evaluated approach.
Loading 1907.11879v1…