PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Harish Haresamudram, Irfan Essa, Thomas Plötz

arXiv:2202.12938v2eess.SPcs.LG

TL;DR

Wearable HAR research faces limited annotated data, while unlabeled sensor data can be collected at scale. This paper introduces a multi-dimensional framework and evaluates seven self-supervised methods, finding robustness across varied conditions and benefits when annotations are scarce.

  • Problem

    Earlier evaluations often used similar source and target conditions, leaving self-supervised HAR performance across varied settings insufficiently assessed.

    Source: §1 · p. 1, §2.1 · p. 6

  • Method

    The paper evaluates seven self-supervised HAR methods with a framework covering source-target robustness, dataset characteristics, and feature-space properties.

    Source: §1 · p. 2, §7 · p. 32

  • Results

    Self-supervised methods showed robustness to class imbalance and significant improvements over supervised learning when annotations were scarce.

    Source: §7 · p. 32

  • Takeaways & Limitations

    The authors present the assessment framework as version 1.0 and call for community contributions toward developing it as a standard for model development and evaluation.

    Source: §7 · p. 32

    SelfHAR was excluded because its target-specific pretraining limits generalizability and does not fit the study’s transfer-learning setup.

    Source: §4.2 · p. 10

Abstract

from arXiv · show

The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems for scenarios where only small amounts of labeled training samples can be collected. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone hand-crafted features for the classic activity recognition chain. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, such that each dimension captures specific aspects of performance, including the robustness to differing source and target conditions, the influence of dataset characteristics, and the feature space characteristics. We utilize this framework to assess seven state-of-the-art self-supervised methods for HAR, leading to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios.

1 INTRODUCTION

The paper argues that self-supervised HAR could help address scarce annotations by learning from plentiful unlabeled sensor data. It introduces a three-dimensional assessment framework and uses it to evaluate seven methods across varied conditions.

1 INTRODUCTION

  • Pretraining on unlabeled movement data may help address limited annotations, including costly healthcare data collection and expert labeling.The paper describes pretraining on background movement data as a way to jumpstart training in such settings.Source: §1 · pp. 1–2
  • Most prior evaluations used similar source and target data, leaving performance under varied conditions insufficiently examined.The paper calls for a multi-faceted evaluation across different scenarios.Source: §1 · p. 2
  • The assessment framework examines robustness to source-target differences, dataset characteristics, and learned feature-space properties.The dimensions include sensor positions, activities and sampling rates; data quantity and imbalance; and representation properties.Source: §1 · p. 2
  • Self-supervised methods were effective across differing sensor positions and unseen activities, and were data efficient and robust to source-dataset imbalances.The evaluation pre-trained on Capture-24 and used an MLP classifier for recognition.Source: §1 · p. 2
  • The paper surveys self-supervised methods for sensor-based HAR and conducts a large-scale evaluation of seven contemporary approaches.The study comprises approximately 50,000 activity-recognition runs.Source: §1 · pp. 2–3

2 BACKGROUND AND MOTIVATION

The background frames self-supervised HAR as learning representations from unlabeled samples for downstream recognition. It reviews approaches and notes that similar source and target conditions limit what existing evaluations establish.

2 BACKGROUND AND MOTIVATION

  • HAR feature representations include statistical, distribution-based, and learned features; this work focuses on learning from unlabeled samples.Learned features include supervised and unsupervised approaches.Source: §2 · p. 3
  • Self-supervised learning uses domain-informed pretext tasks to learn representations from unlabeled data for downstream recognition.It follows a pretrain-then-finetune paradigm.Source: §2.1 · p. 3, §2 · p. 3

2.1 Self-Supervised Learning in Human Activity Recognition

  • Reviewed HAR approaches include transformation-prediction multitask learning, masked reconstruction, and CPC, which predicts future encoded timesteps from a context vector.The reviewed work also includes SelfHAR, which combines self-training with multitask self-supervision, and reports better downstream performance from balanced source subsets.Source: §2.1 · p. 4
  • The paper also evaluates SimCLR, SimSiam, and BYOL; SimCLR contrasts augmented views, while SimSiam avoids negative pairs and BYOL uses online and target networks.BYOL updates its target network using a slow-moving average.Source: §2.1 · p. 5
  • Prior HAR evaluations often used similar source and target conditions, such as overlapping activities and sensor types, limiting their coverage of performance.The paper notes that this leaves robustness under dissimilar conditions untested.Source: §2.1 · p. 6

3 ASSESSMENT METHODOLOGY

The assessment methodology evaluates HAR methods across diverse criteria rather than collapsing performance into one score. Its dimensions cover source-target robustness, dataset characteristics, and learned feature properties.

3 ASSESSMENT METHODOLOGY

  • The framework uses diverse criteria to assess generalizability without reducing performance across tasks to a single value.Its design supports nuanced analysis across multiple aspects of model performance.Source: §3 · p. 6

3.1 Robustness to Differing Source and Target Conditions

  • Source-target robustness criteria examine sensor-position differences, unseen target activities, and sampling-rate variation.The sampling-rate criterion asks whether target data must match the source rate or be downsampled.Source: §3.1 · pp. 6–7

3.2 Influence of Dataset Characteristics

  • Dataset-characteristic criteria examine unlabeled-data quantity, class imbalance, and performance with limited target annotations.The data-quantity criterion considers the number of training users or windows used for pretraining.Source: §3.2 · p. 7

3.3 Feature Space Characteristics

  • Feature-space criteria analyze similarity to supervised representations, linear separability, and implicit dimensionality.These analyses probe representation properties after training.Source: §3.3 · pp. 7–8

4 SETUP

The evaluation uses a single-accelerometer source-to-target setup across nine varied benchmark datasets and seven self-supervised methods, with standardized preprocessing, user-based validation, and macro F1.

4.1 Datasets

  • The study evaluates nine representative target benchmarks with a single accelerometer, selected to vary activities, participants, sensor locations, dataset sizes, and sensors.The study focuses on a single-sensor setup to facilitate transfer of learned weights.Source: §4 · p. 8, §4.1 · p. 8, §4.1.2 · p. 8

4.1.2 Target Datasets.

  • Targets range from wrist-based gym exercises and lab gestures to waist locomotion and transitions and leg/ankle health, exercise, and daily-living tasks; class imbalance is common.Wetlab, Mobiact, MHEALTH, and Myogym have the most severe class compositions.Source: §4.1.2 · pp. 8–10

4.1.1 Capture-24.

  • Capture-24 is the source, while nine targets are grouped three each across waist/trousers, wrist, and leg/ankle.Daphnet FoG is labeled leg/ankle in the table but was recorded on the shank just above the ankle.Source: Table 2 · p. 9

4.2 Self-supervised HAR Methods

  • Methods include multitask transformation classification, masked reconstruction, CPC future prediction, and whole-window autoencoding.Masked reconstruction reconstructs only masked input portions, whereas CPC predicts multiple future timesteps.Source: §4.2 · p. 10
  • Contrastive methods use augmented views (SimCLR), negative-free Siamese learning (SimSiam), or online and target networks (BYOL); SelfHAR is excluded because it produces target-specific pretrained models.SelfHAR's target-specific models do not fit the study's transfer-learning setup.Source: §4.2 · p. 10

4.3 Implementation Details

  • Hyperparameters were tuned by random or grid search, then target classifiers were selected via 50 combinations and five-fold cross-validation; final runs used five seeds, totaling around 50k runs.For most methods, 20 pretraining combinations were sampled; Multi-task self-supervision used a grid search.Source: §4.3 · p. 11

4.4 Data Preparation

  • Raw accelerometer windows were standardized, sampled at 50 Hz, two seconds long, and user-split; Capture-24 training statistics normalized target data.Daphnet remained at 64 Hz because its native rate was comparable.Source: §4.4 · pp. 11–12

4.5 Performance Metric

  • Macro F1 is the main test metric because target classes are imbalanced; it averages per-class precision- and recall-based F1, unlike accuracy or weighted F1.The authors describe unweighted mean F1 as a reasonable strategy for this setting.Source: §4.5 · p. 12

5 RESULTS

This section evaluates self-supervised HAR across mismatched sensor positions, activities, and sampling rates, and examines the effects of unlabeled-data quantity and source-class imbalance. It also analyzes learned representations in terms of similarity to supervised learning, linear separability, and dimensionality.

5.1.1 Performance across Different Sensor Positions.

  • With a 3-layer MLP, wrist-pretrained self-supervised methods often match or exceed supervised baselines at waist and leg targets, though transfer varies by dataset.Waist performance worsens on Mobiact and Motionsense but is similar on USC-HAD; leg performance is similar on Daphnet FoG and PAMAP2 and improves on MHEALTH. Linear evaluation is generally substantially weaker, with SimCLR the main exception (apart from Mobiact).Source: §5.1.1 · p. 13

5.1.2 Performance for Disparate Activities.

  • Self-supervised representations transfer to unseen activities, but results depend on the target dataset and classifier; Wetlab remains a case where supervised baselines outperform.An MLP raises HHAR Multi-task self-supervision from 51% to 57.5%, and CPC and SimCLR from around 56% to 58%; on Myogym, Autoencoder improves from 11.4% to 35.5% and BYOL from 17% to 39%.Source: §5.1.2 · pp. 14–15

5.1.3 Variations in the Sampling Rate.

  • Mismatched target sampling rates reduce performance, especially at higher frequencies; the drop is largest for 200-Hz Mobiact when pretraining used 50 Hz.Masked reconstruction is particularly affected, plausibly because the sampled context differs from pretraining; several other methods also degrade despite time-warping augmentation.Source: §5.1.3 · p. 15, Figure 2 · p. 17

5.2.1 Quantity of the Unlabeled Data Required.

  • Most methods can achieve strong downstream performance with fewer than all 135 pretraining participants, and often with just 10% of training windows.Results vary by method: one participant yields poorer performance overall in some cases, while several methods show little loss or their best performance with few participants; 0.01% (about 410) windows often performs worse, whereas some methods peak at that amount.Source: §5.2.1 · pp. 17, 18, 20, Figure 3 · p. 18, Figure 4 · p. 19

5.2.2 Effect of the Dataset Imbalance on Performance.

  • All self-supervised methods are robust to source-class imbalance, with only minor performance differences between balanced and imbalanced subsets.The imbalance evaluation compares balanced and imbalanced source subsets at ratios ρ ∈ {0.001, 0.01, 0.1}.Source: §5.2.2 · p. 20, Figure 6 · p. 21
  • For six methods, performance on imbalanced and balanced source data is nearly identical; Autoencoder performance worsens as source imbalance increases.For Mobiact, the balanced-versus-imbalanced mean difference grows from ∼0.9% at ρ=0.1 to ∼3.3% at ρ=0.001; the authors attribute the decline to reduced exposure to rare-class windows during reconstruction pre-training.Source: §5.2.2 · p. 21

5.2.3 Performance under Availability of Very Limited Annotated Data.

  • Self-supervised methods generally outperform DeepConvLSTM with very limited labels, with especially substantial gains on fine-grained gym and exercise activities; SimCLR is most effective overall.Multi-task self-supervision also performs strongly. The Conv. classifier is lightweight and often competitive; self-supervision is more effective for wrist- and leg-based targets, while waist-based gains may end at 50 labeled windows per class.Source: §5.2.3 · p. 22, Figure 7 · p. 23

5.3.1 Similarity to Supervised Learning.

  • Earlier self-supervised layers are most similar to supervised representations, while similarity generally decreases with encoder depth and later layers diverge.CPC shows a sharp drop after its second convolutional layer, though its GRU resembles supervised learning; BYOL’s final convolutional layer is less similar, while Autoencoder’s third layer resembles the first supervised layer.Source: §5.3.1 · p. 24

5.3.2 Linear Separability of the Representations.

  • Multi-task self-supervision has a lower ground-truth-versus-random-label train F1 difference, indicating more general representations; contrastive methods, especially SimCLR, SimSiam, and BYOL, have the highest difference.The analysis compares linear-classifier train F1 using ground-truth and random labels; a smaller difference indicates more generic features.Source: §5.3.2 · p. 25, Figure 9 · p. 25

5.3.3 Implicit Dimensionality of the Representations.

  • Around 95% of variance for CPC, SimCLR, BYOL, and Autoencoder is explained by five components, compared with less than 90% for Multi-task and around 85% for Masked reconstruction.SimSiam exceeds 97% with five components; Multi-task and Masked reconstruction therefore have higher implicit dimensionality, while the other methods have relatively lower implicit dimensionality.Source: §5.3.3 · p. 26, Figure 10 · p. 26

6 DISCUSSION

The discussion evaluates experimental choices, summarizes the trade-offs of the self-supervised methods, and distills lessons from the study. It also calls for the community to evolve the assessment framework alongside the field.

6.1 Applying Source Dataset Normalization to Target Scenario for Activity Recognition

  • Source-dataset normalization improves target recognition over target-based normalization, with the largest impact on Masked Reconstruction and Autoencoder.The authors relate this to reconstruction pretexts: source statistics make the target data statistics more similar.Source: Figure 11 · p. 27, §6.1 · p. 27

6.2 Is the Performance of Self-Supervision Getting Hampered by (Too) Simple Encoder Architectures?

  • Adding encoder depth does not improve Multi-task self-supervision or Autoencoder, has little effect on CPC, and reduces SimCLR performance by over 15% on Myogym and over 5% across target datasets.The authors note that the deeper models used hyperparameters selected for the original architectures, which may not be ideal for increased depth.Source: Figure 12 · p. 28, §6.2 · pp. 28–29

6.3 Advantages and Drawbacks of Self-Supervised Approaches

  • CPC performs well across target scenarios and multi-sensor data, while BYOL and SimSiam offer simple, fast setups; Masked reconstruction is computationally intensive and generally does not match other methods on single-accelerometer data.CPC requires considerable tuning; SimSiam can collapse to trivial solutions, while Masked reconstruction has the longest training time in this study.Source: §6.3 · p. 30
  • SimCLR is the most effective evaluated method, while Multi-task self-supervision performs well across downstream scenarios and Autoencoders can be comparably effective with shorter training time.Multi-task self-supervision has a small GPU memory footprint but is less suited to multi-sensor data; Autoencoders require care to avoid learning an identity function.Source: §6.3 · pp. 29–30

6.4 The Necessity for an Evolving Benchmark and a Call to Action

  • The authors present this assessment as version 1.0 and call for community updates that replace solved tasks with more challenging and relevant ones.They advocate future versions as large-scale, decentralized community efforts.Source: §6.4 · p. 31

6.5 Lessons Learned and Insights Gained

  • Self-supervised methods can transfer across sensor positions and activity classes, but optimal performance requires matching target and source sampling rates; they are also data-efficient and robust to extreme source imbalance.Transfer from wrist to leg is reported as more successful than wrist to waist, and a small, participant-diverse dataset can outperform more data from fewer participants.Source: §6.5 · pp. 31–32
  • Early encoder layers resemble supervised features, while feature similarity decreases with depth; Multi-task self-supervision learns generic features and Masked reconstruction uses the representation space more efficiently.Source: §6.5 · p. 32

7 SUMMARY AND CONCLUSION

The paper evaluates self-supervised HAR methods with a framework for examining robustness, dataset effects, and learned representations. Its nine-dataset study reports robustness to class imbalance and improvements over supervised learning when annotations are scarce, while noting that the assessment is limited to single-accelerometer data.

7 SUMMARY AND CONCLUSION

  • The assessment framework examines source–target robustness, dataset characteristics, and representation-space properties through three dimensions, each containing three criteria.The representation-space dimension probes linear separability, implicit dimensionality, and similarity to supervised learning.Source: §7 · p. 32
  • The study found robustness to class imbalance and significant improvements over supervised learning when annotations are scarce.The evaluation used single-accelerometer data from nine diverse benchmark datasets.Source: §7 · p. 32
  • The assessment does not address issues specific to multisensor datasets, including cross-device distribution variation and synchronization.The authors identify its use of single-accelerometer data as the reason for this scope boundary.Source: §7 · p. 32
  • The framework is presented as version 1.0, with the authors inviting the research community to evolve and use it for model development and evaluation.The authors aim to release it in an accessible form for researchers and practitioners.Source: §7 · p. 32

A.1 Architectures, hyperparameters, and implementation details

The implementation uses a three-layer 1D convolutional encoder with global max pooling. Pretraining and classification each tune learning rate and L2 regularization over the same three-value grids.

A.1 Architectures, hyperparameters, and implementation details

  • The encoder has three 1D convolutional layers with 32, 64, and 96 filters and kernel sizes 24, 18, and 8, followed by global max pooling.Each layer uses ReLU and dropout with p=0.1.Source: §A.1.1 · p. 36
  • Pretraining tunes learning rate and L2 regularization over {1e−4, 3e−4, 5e−4}.The encoder architecture is identical to the original paper [62].Source: §A.1.1 · p. 36
  • Classification tunes learning rate and L2 regularization over {1e−4, 3e−4, 5e−4}.Source: §A.1.1 · p. 36

A.1.1 Multi-task Self-supervision.

The described masked-reconstruction implementation uses a Transformer encoder over convolutional embeddings, with fixed sinusoidal embeddings to represent sequence position. Its pretraining search varies layer count, warmup steps, and masking percentage.

A.1.1 Multi-task Self-supervision.

  • Masked reconstruction uses a Transformer encoder, with raw accelerometer data mapped to 128-dimensional embeddings by a 1D convolutional layer.The Transformer setup follows the original paper.Source: §A.1.2 · p. 36
  • Fixed sinusoidal embeddings are added to the convolutional embeddings to inject a sense of time or sequence into the Transformer input.Source: §A.1.2 · p. 36
  • Pretraining tunes Transformer layers from 2 to 6, warmup steps from 20k to 100k, and masked time steps from 10% to 70%.The model uses 8 heads; classifier learning rate and L2 regularization are also tuned.Source: §A.1.2 · p. 36

A.1.2 Masked Reconstruction.

The described setup uses a convolutional encoder with three blocks and a two-layer GRU. The passage specifies the encoder and recurrent-network configurations and begins listing pretraining learning rates.

A.1.2 Masked Reconstruction.

  • The convolutional encoder has three 1D-convolution blocks with 32, 64, and 128 filters, reflect padding, ReLU, and dropout with p=0.2.The setup is identical to [27].Source: §A.1.3 · p. 36
  • The GRU has 256 units, two layers, and dropout with p=0.2.Source: §A.1.3 · p. 36
  • The listed pretraining learning rates are 1e−3, 5e−4, and 1e−4.Source: §A.1.3 · p. 36

A.1.3 CPC.

The CPC encoder uses three convolutional blocks, with a mirrored three-block decoder described in the supplied implementation details.

A.1.3 CPC.

  • Classifier tuning searches learning rates of 1e−4, 5e−4, and 1e−5, with L2 regularization of 0.0, 1e−4, or 1e−5.The supplied setup also lists L2 regularization, kernel size, and batch-size options for training.Source: §A.1.3 · p. 37, §A.1.4 · p. 37
  • CPC uses the same three-block convolutional encoder detailed earlier in the paper.The implementation also searches pre-training learning rate, weight decay, and kernel size.Source: §A.1.4 · p. 37
  • The decoder mirrors the encoder with three convolutional blocks and 128, 64, and 32 filters.The number of filters decreases across the decoder blocks.Source: §A.1.4 · p. 37

A.1.4 Autoencoder.

The described method uses a three-block convolutional encoder and a matching setup with a projection head, cosine-similarity loss, and time-series augmentations.

A.1.4 Autoencoder.

  • The convolutional encoder matches the one used for Multi-task self-supervision [62] and has three 1D blocks with 23, 64, and 96 filters.Kernel sizes are 24, 16, and 8, with ReLU and dropout between blocks.Source: §A.1.5 · p. 37
  • Pre-training uses SGD with momentum 0.9, a cosine learning-rate schedule, and NT-XentLoss.The projection head has three linear layers of 256, 128, and 50 units, with ReLU between layers.Source: §A.1.5 · p. 37
  • The method uses transformations defined in [72], applying one transformation to each batch rather than chaining multiple transformations.Classification learning rates and L2 regularization are also tuned.Source: §A.1.5 · p. 37

A.1.5 SimCLR.

The implementation uses a convolutional backbone, a cosine-similarity-based pre-training loss, and separate projection and prediction heads.

A.1.5 SimCLR.

  • The backbone matches those used for Multi-task self-supervision [62] and [72].Pre-training uses SGD with momentum 0.9 and a cosine learning-rate schedule.Source: §A.1.6 · p. 37
  • Pre-training updates parameters with a symmetric cosine-similarity-based loss.The implementation also uses a three-layer projection head and a two-layer prediction head.Source: §A.1.6 · p. 37
  • Two transformations from the collection defined in [72] are applied to every window in the batch.Classification learning rates and L2 regularization are tuned separately.Source: §A.1.6 · p. 37

A.1.6 SimSiam.

The appendix details self-supervised and classifier implementations, transformation functions, and a source-dataset imbalance setup. It also identifies figures examining performance, representation characteristics, and dataset statistics.

A.1.6 SimSiam.

  • SimSiam uses a lightweight convolutional encoder, SGD with momentum 0.9, a cosine learning-rate schedule, and projection and prediction heads.Both heads contain two linear layers; batch normalization and ReLU are applied between successive layers.Source: §A.1.7 · p. 37

A.1.8 Linear Evaluation.

  • Linear evaluation freezes the pre-trained encoder and updates only a three-layer MLP classifier using cross-entropy loss.The classifier has 256, 128, and num_classes units, with batch normalization, ReLU, and dropout between layers.Source: §A.1.9 · p. 37

A.1.10 DeepConvLSTM.

  • The appendix describes DeepConvLSTM as four convolutional layers followed by a two-layer LSTM and a linear classifier.The convolutional layers each have 64 filters and kernel size 5 × 1.Source: §A.1.10 · p. 38

A.1.11 LSTM and GRU Classifiers.

  • The recurrent classifier uses a single unidirectional LSTM or GRU layer with 128 units, dropout, and a softmax output.A convolutional classifier is also described, using the convolutional encoder followed by a fully connected activity classifier.Source: §A.1.11 · p. 38, §A.1.12 · p. 38

A.2 Transformation Functions used for SimCLR, SimSiam, and BYOL

  • The contrastive methods use eight time-series transformations, including jitter, scaling, rotation, permutation, scrambling, time warping, negation, and reversal.The listed transformations include random channel noise or scaling, permutations of channels or window sections, and time-based alterations.Source: §A.2 · pp. 38–41

A.3 Source Dataset Imbalance Setup

  • Figures show the Capture-24 training class composition, target-dataset compositions, and performance under source-imbalance and dataset-statistics conditions.Other figures address representation similarity to supervised learning and implicit dimensionality across target datasets.Source: Figure 13 · p. 41, Figure 15 · p. 42, Figure 16 · p. 43
  • The source-dataset imbalance factor is the rarest-class window count divided by the most-frequent-class count, and is at most 1.At ρ=0.01, the example has 200 rare-class windows and 20,000 frequent-class windows; balanced subsets divide total windows evenly among six classes.Source: §A.3 · p. 42
  • A further figure compares activity-recognition performance using source-dataset means and variances versus target-dataset statistics.The supplied caption identifies the comparison but does not report its outcome.Source: Figure 20 · p. 47

Related papers

Loading 2202.12938v2…