Source-linked AI summary
Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data
Hang Yuan, Shing Chan, Andrew P. Creagh, Catherine Tong, Aidan Acquah, David A. Clifton, Aiden Doherty
TL;DR
Limited labelled HAR data motivates using self-supervised learning on large unlabelled wearable recordings. The paper pre-trains on UK-Biobank data with temporal transformation tasks and evaluates transfer across seven external benchmarks, where pre-trained models consistently outperform strong baselines, especially in smaller datasets. The authors note that UK-Biobank is mostly Caucasian participants from the UK, limiting demographic coverage.
Problem
Limited labelled HAR datasets constrain deep-learning methods, motivating methods that can exploit feasible large-scale unlabelled wearable data.
Method
The model uses multi-task self-supervised pre-training on roughly 700,000 person-days of free-living UK-Biobank wearable data, followed by fine-tuning on seven external HAR datasets.
Results
Pre-trained models outperformed models trained from scratch and random forests across all seven datasets, with the largest gains occurring in smaller datasets.
Takeaways & Limitations
Self-supervised pre-training can improve downstream HAR and reduce reliance on labelled data while generalising across datasets, tasks, devices, and populations.
Takeaways & Limitations
The UK-Biobank pre-training data consists mostly of Caucasian participants from the UK, limiting demographic representation and motivating evaluation on additional demographic groups.
Abstract
from arXiv · showhide
Advances in deep learning for human activity recognition have been relatively limited due to the lack of large labelled datasets. In this study, we leverage self-supervised learning techniques on the UK-Biobank activity tracker dataset--the largest of its kind to date--containing more than 700,000 person-days of unlabelled wearable sensor data. Our resulting activity recognition model consistently outperformed strong baselines across seven benchmark datasets, with an F1 relative improvement of 2.5%-100% (median 18.4%), the largest improvements occurring in the smaller datasets. In contrast to previous studies, our results generalise across external datasets, devices, and environments. Our open-source model will help researchers and developers to build customisable and generalisable activity classifiers with high performance.
1 Introduction
Limited labelled HAR data restricts deep-learning effectiveness, while large-scale unlabelled wearable data makes self-supervised learning attractive. This paper investigates temporal self-supervised tasks on UK-Biobank data and evaluates their cross-dataset generalisation.
- Motivation: Limited labelled datasets constrain data-hungry deep-learning methods for human activity recognition.HAR labelling is labour intensive because ground truth requires annotating corresponding video streams.
- Motivation: Large-scale unlabelled HAR data is feasible to collect through projects such as UK-Biobank and NHANES.
- Approach: The study trains arrow of time, permutation, and time-warping self-supervised tasks independently and jointly using UK-Biobank wearable data.These tasks prioritise temporal dependencies in human motion.
- Contribution: The resulting multi-task model generalises across seven external datasets differing in activity classes, devices, populations, and recording environments.The evaluation explicitly considers domain shift and task shift.
2 Related work
Earlier HAR approaches used handcrafted features or deep learning, but small labelled datasets limited deep models and sometimes favoured random forests. Prior HAR self-supervised studies also relied on small laboratory-style datasets, motivating evaluation at much larger scale and diversity.
- HAR approaches: HAR traditionally used manual feature engineering and tree-based models, while deep-learning approaches were limited by small datasets.Random forests sometimes generalised better than deep-learning models in small-data regimes.
- Self-supervised learning: Previous HAR self-supervised studies included forecasting, masked reconstruction, contrastive learning, and multi-task learning.
- Self-supervised learning: Prior SSL-HAR studies used laboratory-style datasets with fewer than 10,000 samples, confounding conclusions about utility.
- Study distinction: This study uses more than 6 billion wrist-worn raw accelerometer measurements collected in real-world settings with substantially greater activity diversity.The authors frame this scale and diversity as a more definitive test of SSL for HAR.
3 Methods
The methods combine wrist-worn accelerometer windows, large-scale free-living UK-Biobank pre-training, temporal self-supervised tasks, weighted sampling, and downstream evaluation across diverse external datasets. A unified framework compares fine-tuned, from-scratch, random-forest, and transfer-learning baselines while examining data-volume effects and representations.
- Data: The study uses tri-axial wrist accelerometer signals split into equal-duration windows and linearly resampled to 30 Hz.
- Data: UK-Biobank pre-training uses roughly 700,000 person-days from more than 100,000 participants wearing trackers for seven days.Free-living recordings capture activities beyond scripted laboratory protocols.
- Evaluation datasets: Seven external labelled datasets vary from 600 to 600,000 samples, span 4 to 18 activity classes, use five device brands, and include free-living, scripted, and laboratory settings.
- Self-supervised tasks: The three self-supervised transformations are arrow of time reversal, four-chunk permutation, and random time warping.Each task predicts whether its transformation was applied, with equally weighted cross-entropy losses in multi-task learning.
- Training: Weighted sampling selects windows in proportion to standard deviation to emphasize high-movement periods and improve training stability and convergence.Low-movement periods are less informative because static signals remain nearly unchanged after transformation.
- Model and transfer: The model uses an 18-layer 1D-convolutional ResNet-V2 feature extractor shared across self-supervised tasks, with downstream fine-tuning of all layers or only added fully connected layers.
- Evaluation protocol: Downstream evaluation uses subject-wise cross-validation, 7:1:2 train-validation-test splits, and early stopping with patience five.Datasets with fewer than 10 subjects use held-one-subject-out validation; larger datasets use five-fold validation.
- Evaluation protocol: The shared implementation standardizes network training, preprocessing, and evaluation to support fairer baseline comparisons.The framework addresses inconsistent protocols across benchmark datasets.
4 Results
Across seven downstream datasets, self-supervised pre-training consistently improved human activity recognition, with the largest gains in smaller datasets and under limited labels. The learned representations also clustered activities by type and intensity, while attribution analyses highlighted meaningful motion patterns.
- Downstream performance: Pre-trained models outperformed both from-scratch deep models and random forests across all seven datasets.Random forests outperformed models trained from scratch except on Capture-24, the largest evaluation dataset.
- Downstream performance: 18.4% median relative F1 improvement over the same network trained from scratch, versus 8.5% over random forests.In Capture-24, the improvement over the corresponding model without SSL pre-training was 2.5%.
- Multi-task self-supervised learning: SSL gains were largest on small datasets, whereas differences among task combinations were smaller on Capture-24 and Rowlands than on Opportunity.There was no clear best-performing configuration, so all tasks were used for subsequent experiments.
- Ablation studies: Pre-trained models remained effective across different numbers of labelled subjects, while fully supervised and random-forest models were more sensitive to label quantity.The gain from adding labelled subjects was roughly linear, with larger increases when fewer subjects were available.
- Ablation studies: Downstream performance increased approximately linearly with the number of unlabelled subjects on a logarithmic scale, with the strongest boost in Opportunity.With 10,000 pre-training participants fixed, increasing data per subject did not significantly affect downstream performance.
- Cluster analysis: SSL-derived features clustered similar activities and activity intensities more effectively than raw inputs or untrained features.The clustering analysis used UMAP projections without fine-tuning the SSL-derived features.
5 Conclusion and future work
Self-supervised pre-training improved downstream human activity recognition, particularly with limited labelled data, and generalized across datasets, tasks, devices, and populations. Future work includes broader modalities, demographic diversity, and analysis of subject variability.
- Self-supervised pre-training consistently improved downstream HAR, especially in the small data regime, reducing the need for labelled data.
- Pre-trained representations generalized across external datasets, tasks, devices, and populations.
- Multi-task SSL appears to learn human motion dynamics, intensity levels, and synchronization between sensor axes.
- Future work could incorporate electrocardiogram data and analyze inter-subject and intra-subject variability to inform data curation.
- The UKB pre-training data consists mostly of Caucasians from the UK because raw accelerometer datasets from other regions are limited.The authors identify datasets from different demographic groups as a natural extension to improve model fairness.
A.1 Datasets
Table 5 catalogs the licence, consent, and device information for each dataset used in the study.
- Table 5 lists the license, consent, and device used for each dataset.
A.3 Feature engineering
The random forest baseline uses manually engineered statistical, correlation, norm, distributional, and frequency-domain features from accelerometer data.
- The random forest features include the mean, standard deviation, and range for each accelerometer axis.
- The feature set includes correlations between each pair of accelerometer axes.
- It also includes Euclidean norm summaries such as mean, standard deviation, range, median absolute deviation, kurtosis, and skew.
- The power-spectrum representation contributes the top two dominant frequencies.
A.3.1 Explainable AI Framework for Time-Series Prediction
The explainable-AI framework combines raw signals, time-frequency representations, and LRP attributions to inspect which time-series inputs support or oppose pretext-task predictions.
- The framework interprets AoT predictions using raw data, continuous wavelet transforms, and time-localized LRP attributions.
- The CWT measures similarity between a signal and the Morlet wavelet to produce a time-frequency representation.
- LRP back-propagates the output decision through the network to assign relevance to individual input nodes.
- LRP redistributes relevance layer by layer while conserving relevance within each layer.
- Positive input relevance supports the predicted class, negative relevance contradicts it, and near-zero relevance has little influence.
- The composite LRP rule applies LRP-γ in shallow convolutional layers, LRP-ϵ in middle layers, and LRP-0 in the final classification layer.
- Visualization panels show three-axis traces, CWT scalograms of ∥a∥, and color-coded relevance values, with red indicating positive and blue negative relevance.
- Faithfulness is evaluated by cumulatively masking samples ranked from most to least relevant and observing how quickly model accuracy degrades.
A.4 Results
Results show that self-supervised representations cluster related activities and intensities, while explainability analyses identify signal patterns associated with pretext-task decisions. Performance and robustness varied across tasks, sampling strategies, and data settings.
- Representation quality: SSL-derived features better cluster similar activities and distinguish low- versus high-intensity movements than raw inputs or untrained features.The reported examples separate walking and stair climbing from sitting, writing, and typing, while also grouping lying down, sitting, and standing apart from jogging and sports.
- Explainability: Removing features ranked as relevant usually destroyed class evidence faster than random masking, reducing out-of-sample accuracy.Adding random noise also reduced performance, but more slowly than masking the most relevant samples first.
- Explainability: AoT-specific accuracy degraded more slowly under forward or reverse cumulative noise replacement than under relevance-based or random masking.This suggests the model does not rely on one single forward or reverse sequential pattern, but may use signal morphologies associated with dynamic human motion.
- Explainability: Contextual LRP attributed relevance to natural human motion during unscripted tennis actions, including dropping the ball and serving.The examples came from repetitive, high-intensity tennis activity and used contextual LRP visualizations.