PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data

Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, Soren Brage, Nick Wareham, Cecilia Mascolo

arXiv:2102.06073v1cs.LG

TL;DR

Human activity recognition in real-world settings is constrained by small, limited-diversity labeled datasets. SelfHAR combines teacher-student self-training with self-supervised pre-training to use unlabeled mobile-sensing data. The paper reports stronger performance than supervised and semi-supervised approaches, with similar performance possible using substantially less labeled data.

  • Problem

    HAR datasets can be small and limited in diversity because sensor streams are difficult to label after collection, constraining coverage of real-world behavior.

    Source: §1 · p. 2

  • Method

    SelfHAR combines teacher-student self-training with multi-task self-supervision to leverage unlabeled mobile-sensing data alongside labeled HAR data.

    Source: §1 · p. 3

  • Results

    Up to 12% higher F1 than state-of-the-art supervised and semi-supervised methods was reported without increasing inference model complexity.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Similar performance to the conventional fully supervised approach was achieved with up to 10 times less labeled data.

    Source: §1 · p. 3

    The paper leaves evaluation of learned representations in multi-device, multi-sensor settings and on mobile-sensing tasks beyond HAR as future work.

    Source: §6 · p. 24

Abstract

from arXiv · show

Machine learning and deep learning have shown great promise in mobile sensing applications, including Human Activity Recognition. However, the performance of such models in real-world settings largely depends on the availability of large datasets that captures diverse behaviors. Recently, studies in computer vision and natural language processing have shown that leveraging massive amounts of unlabeled data enables performance on par with state-of-the-art supervised models. In this work, we present SelfHAR, a semi-supervised model that effectively learns to leverage unlabeled mobile sensing datasets to complement small labeled datasets. Our approach combines teacher-student self-training, which distills the knowledge of unlabeled and labeled datasets while allowing for data augmentation, and multi-task self-supervision, which learns robust signal-level representations by predicting distorted versions of the input. We evaluated SelfHAR on various HAR datasets and showed state-of-the-art performance over supervised and previous semi-supervised approaches, with up to 12% increase in F1 score using the same number of model parameters at inference. Furthermore, SelfHAR is data-efficient, reaching similar performance using up to 10 times less labeled data compared to supervised approaches. Our work not only achieves state-of-the-art performance in a diverse set of HAR datasets, but also sheds light on how pre-training tasks may affect downstream performance.

1 INTRODUCTION

HAR models face limited labeled datasets that may not capture the diversity and complexity of real-world behavior. SelfHAR combines teacher-student self-training with self-supervised pre-training, and the paper reports improved performance and data efficiency across diverse datasets.

1 INTRODUCTION

  • Mobile sensing datasets are difficult to label after collection, and HAR studies often involve fewer than 50 participants and fixed, short protocols.These collection constraints limit dataset size, diversity, and quality.Source: §1 · p. 2
  • The paper situates its approach against transfer learning and self-supervised learning, which are commonly used semi-supervised techniques.It notes that earlier HAR self-supervision did not leverage a complementary unlabeled dataset.Source: §1 · pp. 2–3
  • SelfHAR combines teacher-student self-training with self-supervised pre-training to use large-scale unlabeled HAR datasets alongside labeled data.The approach includes signal-transformation discrimination and aims to increase internal and external training-data diversity.Source: §1 · p. 3
  • Up to 12% higher F1 than state-of-the-art supervised and semi-supervised methods was reported without increasing inference model complexity.The evaluation covered seven datasets with different sensor types, populations, and protocols.Source: §1 · p. 3
  • Scores of 0.67–0.88 were attained using only 10 samples per class, with similar performance achieved using up to 10 times less labeled data than the conventional fully supervised approach.Source: §1 · p. 3

2 RELATED WORK AND MOTIVATION

The related work frames SelfHAR against HAR data-collection constraints and prior semi-supervised, self-supervised, self-training, and transfer-learning approaches. It identifies limited success using unlabeled datasets in HAR as a motivation for combining these techniques.

2.1 Human Activity Recognition

  • Real-world HAR performance depends on datasets that capture diverse actions, but collecting labeled sensor data is difficult.The paper notes that laboratory datasets may not represent the richness, noise, and complexity of free-living data.Source: §2.1 · p. 4

2.2 Semi-supervised Learning

  • Semi-supervised HAR methods use unlabeled data, but cited approaches include costly iterative ensembles and sparse coding that requires L1-regularized least-square solutions.En-Co-training adds samples when classifiers agree; sparse-coding features are used to train a classifier.Source: §2.2 · pp. 4–5

2.2.1 Self-supervised Learning.

  • Self-supervised learning creates supervisory signals from data, including signal-transformation tasks, but prior HAR work showed little to no gain from another unlabeled dataset.Other cited work explored varied tasks and modalities, with fully supervised models often outperforming the best self-supervised methods.Source: §2.2.1 · pp. 5–6

2.2.2 Self-training.

  • Self-training uses teacher-generated labels for unlabeled samples, and image-classification work selected confident predictions before student training and fine-tuning.The paper notes that translating those gains to mobile sensing remained unstudied and that the cited method relied on millions of labeled ImageNet images.Source: §2.2.2 · p. 6

2.3 Transfer Learning

  • Network-based transfer learning pre-trains on a large dataset, then fine-tunes on another task or domain, often freezing pretrained layers.The cited approach aims to mitigate catastrophic forgetting while retaining previously learned knowledge.Source: §2.3 · pp. 6–7

2.4 Motivation

  • The paper identifies a gap in leveraging large-scale unlabeled data through self-training and reports limited success using unlabeled datasets for HAR self-supervision.SelfHAR combines these approaches and freezes parts of the network during fine-tuning to retain general feature-extraction knowledge.Source: §2.4 · p. 7

3 APPROACH

SelfHAR is introduced as a semi-supervised HAR framework for leveraging unlabeled datasets to increase training-data diversity.

3 APPROACH

  • SelfHAR is a semi-supervised training framework for HAR that leverages unlabeled datasets.The framework aims to improve the diversity of data used to train models.Source: §3 · p. 7
  • The framework incorporates both self-training and self-supervised learning.These techniques are combined to use unlabeled datasets in HAR.Source: §3 · p. 7
  • SelfHAR aims to improve the diversity of the data on which models are trained.Source: §3 · p. 7

3.1 Problem Statement

The problem is to map sensor time windows to the activity performed and assess whether adding a complementary unlabeled dataset improves recognition over using labeled data alone.

3.1 Problem Statement

  • The model maps each time window in sensor data to an activity label, with the goal of testing whether adding unlabeled data improves recognition over labeled data alone.Evaluation predicts activity labels for an unseen subset of the labeled dataset.Source: §3.1 · p. 7

3.2 The Training Pipeline

SelfHAR uses a teacher-student pipeline to label unlabeled sensor data, augment selected samples, and pre-train a student before fine-tuning it on labeled data.

3.2 The Training Pipeline

  • The pipeline combines self-training and self-supervised pre-training in a multi-step training process.It is designed to increase the internal and external diversity of training data.Source: §3.2.1 · pp. 7, 9

3.2.1 Overview.

  • The teacher is trained on labeled data, then used to label a mixed dataset containing unlabeled and label-stripped training samples.The teacher uses a multi-class activity classification loss.Source: §3.2.1 · pp. 7–8, §3.2.2 · p. 9

3.2.3 Samples Labeling and Selection.

  • Only the top K samples per class that meet confidence threshold C are selected, retaining their probabilistic labels for student training.The confidence threshold excludes uncertain predictions to limit labeling noise.Source: §3.2.3 · p. 9

3.2.4 Data Augmentation and Further Labeling.

  • Eight signal transformations augment selected samples, with activity labels retained and binary labels indicating which transformations were applied.Each sample has an original and eight transformed versions; transformations are not combined on a sample.Source: §3.2.4 · p. 9

3.2.5 Student Model Training.

  • The student jointly predicts activity and eight binary transformation labels from the augmented dataset using a combined loss.The transformations include noise, scaling, rotation, inversion, time reversal, scrambling, time warping, and channel shuffling.Source: §3.2.4 · p. 9, §3.2.5 · p. 10
  • After pre-training, early student layers are frozen and the activity branch is fine-tuned on the original labeled dataset.Early stopping is used during both pre-training and fine-tuning to reduce overfitting.Source: §3.2.5 · p. 10

3.3 Configurations of the SelfHAR Pipeline

The study compares five training configurations built from four pipeline components, while keeping model architecture and prediction complexity constant.

3.3 Configurations of the SelfHAR Pipeline

  • Transformation discrimination can be used to pre-train the teacher before its fine-tuning, forming a distinct component in the pipeline.The study denotes this teacher pre-training step component 0.Source: §3.3 · pp. 10–11
  • Five configurations range from fully supervised training to SelfHAR, combining teacher training, self-labeling, signal transformation, and student training in different ways.SelfHAR uses components 1, 2, 3, and 4; the fully supervised configuration uses component 1 only.Source: §3.3 · p. 10
  • The transformation-discrimination configuration pre-trains on unlabeled data, transfers the convolutional core, and fine-tunes it with a new activity-recognition head.This follows the method proposed by Saeed et al. [39].Source: §3.3 · p. 10
  • All compared configurations share the same neural-network architecture and prediction-model complexity.The configurations are evaluated in an ablation study.Source: §3.3 · p. 11

4 EXPERIMENTAL PROTOCOLS

The experimental protocols evaluate the approach using eight datasets: seven labeled and one unlabeled, differing in users, activities, sizes, and device placements.

4.1.1 HHAR.

  • The HHAR dataset covers six daily activities among nine participants using smartphone accelerometers from heterogeneous devices.The study uses smartphone accelerometer signals for compatibility across datasets.Source: §4.1.1 · p. 11

4.1.2 MotionSense.

  • MotionSense includes six activities from 24 participants, recorded by an iPhone accelerometer in a front trouser pocket.The study uses only the iPhone accelerometer signals.Source: §4.1.2 · p. 12

4.1.3 MobiAct.

  • MobiAct contributes 11 non-fall daily activities recorded with smartphones placed freely in participants’ pockets.The dataset originally includes daily activities and falls; fall-related data are excluded for this activity-classification study.Source: §4.1.3 · p. 12

4.1.4 PAAS.

  • PAAS activity data are grouped into 11 categories and represented by wrist accelerometer recordings.The study uses the triaxial accelerometer on each participant’s non-dominant wrist.Source: §4.1.4 · p. 12

4.1.5 UniMiB SHAR.

  • UniMiB SHAR contributes nine daily-activity classes recorded with a smartphone in participants’ trouser pockets.The dataset also contains fall activities, which are not included in the listed study classes.Source: §4.1.5 · pp. 12–13

4.1.6 UCI HAR.

  • UCI HAR records six activities from 30 volunteers using waist-mounted smartphone accelerometer and gyroscope signals.The phone was worn on the belt in two placements across the protocol.Source: §4.1.6 · p. 13

4.1.7 WISDM.

  • WISDM contains six daily activities recorded by smartphone accelerometers carried in volunteers’ trouser pockets.The dataset includes 29 volunteers and uses a 20 Hz sampling frequency.Source: §4.1.7 · p. 13

4.1 Datasets

  • Fenland supplies free-living wrist accelerometry from a subsample of 2,100 participants, with activity labels derived from metabolic equivalents.The labels distinguish sedentary, light, and moderate-to-vigorous physical activity.Source: §4.1.8 · p. 13

4.2 Data Preparation

The section fixes common sensor preprocessing and user-held-out testing, then specifies a shared TPN-based evaluation framework. It includes standard and linear evaluation alongside two non-deep-learning semi-supervised baselines.

4.2 Data Preparation

  • Windows use 400 timestamps across three accelerometer channels with 50% overlap; 20%–25% of users are held out for testing, without resampling frequencies.The same preprocessing is used across datasets to support fair comparisons focused on training paradigms.Source: §4.2 · pp. 13–14

4.3 Experimental Setup

  • The experimental setup adopts TPN, a temporal convolutional model with three convolutional layers and task-specific heads, to compare training pipelines across architectures.The convolutional layers use 32, 64, and 96 filters, respectively.Source: §4.3.1 · p. 14

4.3.1 Standard Evaluation.

  • Standard evaluation shares the convolutional core across tasks, sums task losses equally, and freezes its first two layers during fine-tuning.Transformation discrimination uses binary cross-entropy; activity recognition uses categorical cross-entropy.Source: §4.3.1 · p. 14

4.3.2 Linear Evaluation.

  • Linear evaluation freezes the pretrained convolutional core and trains a directly connected linear softmax classifier as a proxy for representation quality.No additional nonlinearity is introduced in the new classifier layer.Source: §4.3.2 · p. 14

4.3.3 Other Baseline Algorithms.

  • The non-deep-learning semi-supervised baselines are En-Co-training and Sparse-Encoding.Source: §4.3.3 · p. 14

5 EVALUATION AND DISCUSSION

SelfHAR outperformed fully supervised training across seven target datasets, with further evaluation of class-level performance, training configurations, limited-label settings, representations, and unlabeled-data distributions.

5.1.1 Overview.

  • Up to 12.99% higher weighted F1 on UniMiB SHAR, SelfHAR consistently outperformed the fully supervised baseline across seven datasets, with significant gains on five.The significant improvements were reported for HHAR, MotionSense, MobiAct, PAAS, and UniMiB SHAR.Source: §5.1.1 · p. 15, Table 2 · p. 16

5.1.2 Activity-level comparison.

  • SelfHAR improved or matched supervised performance across nearly all activity classes, rather than gaining by specializing in particular classes.Positive improvements or similar performance across all classes were reported for HHAR, PAAS, UniMiB SHAR, and WISDM.Source: Figure 3 · p. 17, §5.1.2 · p. 18

5.2 Ablation Studies: Effectiveness of the Combined Pre-training Tasks

  • Across ablations, SelfHAR performed best in most cases; combining transformation discrimination with self-training outperformed single approaches, including the prior self-supervised HAR method.In linear evaluation, the performance gap was smaller, consistent with the last convolutional layer being frozen rather than fine-tuned.Source: §5.2 · p. 18, Table 3 · p. 19, Table 4 · p. 20
  • SelfHAR required the most training, while fully supervised training required the least and performed worst in most cases; inference complexity remained unchanged.The compared final models had the same architecture and number of weights.Source: §5.2 · p. 18

5.3 Impact of Training with Limited Labeled Data

  • With 10–100 labeled samples per class, SelfHAR consistently improved over other methods, but below 10 samples transformation discrimination alone sometimes performed better.The authors attribute the very-low-label cases to noise and uncertainty in teacher-student supervisory signals; SelfHAR's largest gain in the 10–100 range was on MotionSense.Source: §5.3 · pp. 21–22, Figure 4 · p. 21

5.4 Visualization of Extracted Features Using t-SNE

  • Before fine-tuning, SelfHAR learned feature manifolds similar to those of the supervised model despite not using ground-truth labels for the projections.The authors note similar clusters in WISDM, including a continuum among jogging, walking, and sitting.Source: Figure 5 · p. 22, §5.4 · p. 23

5.5 Leveraging large unlabeled datasets to improve supervised performance: PAAS and Fenland case study

  • Balanced Fenland pre-training improved F1 macro by 13.44% over the supervised task, exceeding improvements from low- or high-activity-only data.The reported gains were 10.44% for the low-MET dataset and 11.30% for the active dataset; the authors conclude that broader activity coverage is more desirable for pre-training.Source: §5.5 · pp. 23–24, Table 5 · p. 23

6 CONCLUSION AND FUTURE DIRECTIONS

SelfHAR combines self-training and self-supervised learning to use unlabeled data alongside limited labeled data, and the authors report better HAR performance without greater inference-time model complexity. They identify broader representation evaluation as a direction for future work.

6 CONCLUSION AND FUTURE DIRECTIONS

  • SelfHAR outperforms supervised and semi-supervised approaches in overall performance and on individual activities, without increasing inference-time model complexity.The authors present the pipeline as a way to complement limited labeled data with abundant unlabeled data.Source: §6 · p. 24
  • The pipeline combines self-training and self-supervised learning to leverage unlabeled data and support generalization to unseen scenarios.The authors describe the combined pipeline as increasing both internal and external training-data diversity.Source: §6 · p. 24
  • Future work could evaluate learned representations across multiple devices, sensors, and mobile-sensing tasks beyond HAR, including transfer to other tasks.The authors also suggest incorporating other semi-supervised or self-supervised techniques.Source: §6 · p. 24

A ALGORITHM OF SELFHAR

The algorithm uses labeled and unlabeled datasets to train a teacher, select confident pseudo-labeled samples, and build a transformed multitask dataset for student training and fine-tuning.

A ALGORITHM OF SELFHAR

  • The algorithm takes labeled D, unlabeled U, transformations T, activity classes A, and training-epoch counts, and outputs a trained HAR model.The inputs specify separate teacher-training, student-pretraining, and student-fine-tuning epochs.Source: A ALGORITHM OF SELFHAR · p. 28
  • A teacher model is trained on labeled D, then used to predict activity labels for the combined signals from D and U.The teacher is trained with cross-entropy in conventional supervised training.Source: A ALGORITHM OF SELFHAR · p. 28
  • For each activity class, the algorithm selects the top K predictions whose class confidence meets threshold C to form S.Source: A ALGORITHM OF SELFHAR · p. 28
  • Selected samples are augmented with transformations T to form D_prime, with HAR labels and indicators marking which transformations were applied.Source: A ALGORITHM OF SELFHAR · p. 28
  • The student is trained on D_prime, then its core layers are frozen and its activity model is fine-tuned on labeled D.The fine-tuning step uses cross-entropy on D.Source: A ALGORITHM OF SELFHAR · p. 28

B ACTIVITY LABELS

The appendix maps each dataset’s numeric activity-class labels to activity names.

B ACTIVITY LABELS

  • HHAR and MotionSense use classes 0–4 for sitting, standing, walking, walking upstairs, and walking downstairs; class 5 is biking in HHAR and jogging in MotionSense.Source: B ACTIVITY LABELS · p. 29
  • MobiAct maps 0–5 to standing, walking, jogging, jumping, walking upstairs, and walking downstairs; 6–10 cover transitions, chair postures, and car entry or exit.The listed classes are standing-to-sitting, sitting on a chair, sitting-to-standing, stepping into a car, and stepping out of a car.Source: B ACTIVITY LABELS · p. 29
  • PAAS maps 0–10 to sitting, standing, walking, stairs, lying, cycling, home and office activities, personal care, and shopping; UCI HAR maps 0–5 to walking, stairs, sitting, standing, and laying down.Source: B ACTIVITY LABELS · p. 29
  • UniMiB SHAR maps 0–8 to posture transitions, walking, running, stairs, jumping, and lying down; WISDM maps 0–5 to walking, jogging, sitting, standing, and stairs.Source: B ACTIVITY LABELS · p. 29

C COMPONENTS OF THE SELFHAR PIPELINE

Figure 6 presents five evaluated SelfHAR pipeline configurations, each combining a different subset of components.

C COMPONENTS OF THE SELFHAR PIPELINE

  • The fully supervised configuration uses component 1 only, while transformation discrimination uses components 0 and 1.Source: Figure 6 · p. 30
  • Self-training uses components 1, 2, and 4; transformation knowledge distillation adds component 0 to that set.Source: Figure 6 · p. 30
  • The full SelfHAR configuration uses components 1, 2, 3, and 4.Source: Figure 6 · p. 30

Related papers

Loading 2102.06073v1…