Source-linked AI summary
Self-supervised transfer learning of physiological representations from free-living wearable data
Dimitris Spathis, Ignacio Perez-Pozuelo, Soren Brage, Nicholas J. Wareham, Cecilia Mascolo
TL;DR
Wearable sensing provides low-level signals, while supervised activity recognition relies on costly labels and may not generalize from laboratory to free-living behavior. Step2Heart self-supervises activity-to-heart-rate prediction and transfers aggregated user representations to health-related tasks. Its embeddings outperform strong baselines across upstream and downstream evaluations, including BMI prediction at AUC 0.70.
Problem
Wearable signals are low-level, and supervised activity recognition relies on costly labels and laboratory datasets that may not generalize to diverse free-living behavior.
Source: Introduction · p. 1Method
Step2Heart pretrains activity-to-heart-rate prediction with a quantile-aware loss, then aggregates window embeddings for transfer learning on user-level outcomes.
Source: Introduction · p. 2Results
Step2Heart outperformed strong baselines in upstream and downstream evaluations, with BMI prediction at AUC 0.70.
Source: Introduction · p. 2, Conclusion · p. 7Takeaways & Limitations
The learned representations support personalized prediction of physiologically meaningful health and fitness variables from wearable data.
Source: Introduction · p. 2, Conclusion · p. 7Personalized health representations could be misused for exclusionary insurance policies or unfair credit scoring, motivating data governance that protects users and prevents misuse.
Source: Ethical impact and broader implications · p. 8
Abstract
from arXiv · showhide
Wearable devices such as smartwatches are becoming increasingly popular tools for objectively monitoring physical activity in free-living conditions. To date, research has primarily focused on the purely supervised task of human activity recognition, demonstrating limited success in inferring high-level health outcomes from low-level signals. Here, we present a novel self-supervised representation learning method using activity and heart rate (HR) signals without semantic labels. With a deep neural network, we set HR responses as the supervisory signal for the activity data, leveraging their underlying physiological relationship. In addition, we propose a custom quantile loss function that accounts for the long-tailed HR distribution present in the general population. We evaluate our model in the largest free-living combined-sensing dataset (comprising >280k hours of wrist accelerometer & wearable ECG data). Our contributions are two-fold: i) the pre-training task creates a model that can accurately forecast HR based only on cheap activity sensors, and ii) we leverage the information captured through this task by proposing a simple method to aggregate the learnt latent representations (embeddings) from the window-level to user-level. Notably, we show that the embeddings can generalize in various downstream tasks through transfer learning with linear classifiers, capturing physiologically meaningful, personalized information. For instance, they can be used to predict variables associated with individuals' health, fitness and demographic characteristics, outperforming unsupervised autoencoders and common bio-markers. Overall, we propose the first multimodal self-supervised method for behavioral and physiological data with implications for large-scale health and lifestyle monitoring.
Introduction
Wearable sensors capture low-level signals, while supervised activity models rely on costly labels and may not generalize from laboratory settings to diverse free-living behavior. Step2Heart uses activity-to-heart-rate prediction to learn user-level representations that support health-related transfer tasks.
Introduction
- Deep wearable models can capture temporal dynamics, but supervised human activity recognition depends on costly labels and often uses laboratory data that may not generalize to diverse free-living behavior (Bulling, Blanke, and Schiele 2014; Krishnan, Sharma, and Sundaram 2018).Wrist-worn devices provide low-level signals such as acceleration rather than higher-level events of interest.Source: Introduction · p. 1
- Unsupervised wearable methods often use only movement, whereas movement and its cardiovascular response jointly reflect physiological differences such as fitness.Heart-rate responses to exercise have been reported as predictive of cardiovascular disease, coronary heart disease, and all-cause mortality.Source: Introduction · p. 1
- Step2Heart pretrains a model to forecast ECG-level quality heart rate in real time from activity signals, then transfers its learned representations to personalized health-outcome prediction.The authors motivate the mapping as a way to capture information beyond activity-data autoencoders or other bio-markers.Source: Introduction · p. 2
- The pre-training loss uses heart-rate quantiles to regularize mean squared error and account for the long tails of real-world heart-rate data.The evaluation includes over 1,700 participants with combined heart and activity sensors tracked for a week.Source: Introduction · p. 2
- The learned embeddings outperform autoencoders and common bio-markers, with BMI prediction at AUC 0.70 and Physical Activity Energy Expenditure at AUC 0.80.The embeddings are aggregated from window-level features to user-level representations.Source: Introduction · p. 2
Related work
Prior wearable sensing research spans objective activity monitoring, deep-learning applications, and emerging self-supervised methods. The cited work leaves gaps in free-living forecasting scale, multimodal representation learning, and generalization of learned embeddings.
Related work
- Mobile and wearable monitoring can provide continuous physical-activity profiles, and combining these with cardiorespiratory information offers insights into health and fitness status.Large-scale mobile-accelerometer studies have also reported differences in physical activity across countries and world regions.Source: Related work · p. 2
- FitRec learns personalized heart-rate and activity dynamics during exercise but requires prior activity segmentation; earlier movement-to-heart-rate forecasting used 3 users and PPG rather than ECG ground truth (Ni, Muhlstein, and McAuley 2019; McConville et al. 2018).The segmentation requirement is noted as a constraint for unconstrained free-living applications.Source: Related work · p. 2
- Wearable self-supervision had been applied to activity or ECG in single-modality settings, while this work combines acceleration and ECG guided by their physiological relationship (Saeed, Ozcelebi, and Lukkien 2019; Sarkar and Etemad 2019).The paper characterizes general-purpose physiological and behavioral representations as an under-explored area.Source: Related work · pp. 2–3
- The cardiovascular signature autoencoder required a month of historical data and used more aggregated inputs and fewer outcomes than the present work (Hallgr´ımsson et al. 2018).The authors describe the month-long historical input as making that setup infeasible for real-time applications.Source: Related work · p. 3
Methods
The method forecasts future heart rate from activity sequences and metadata, then transfers the learned representations to user-level health and fitness prediction. Its joint loss combines mean-squared error with quantile losses to account for long-tailed heart-rate data.
Methods
- Activity sequences and metadata predict future heart rate without temporal overlap, and sequence length can be adapted to the task or data granularity.Source: Methods · p. 3
- For downstream transfer, window-level embeddings are aggregated to user level and traditional classifiers predict users’ health- and fitness-related outcomes.The described implementation uses average pooling for user-level aggregation and a linear model for prediction.Source: Methods · p. 3
Model architecture
- Stacked CNNs and GRUs extract temporal features, which are globally averaged into penultimate-layer embeddings and can be concatenated with metadata.A linear-output feed-forward layer performs continuous prediction.Source: Layers · p. 4
- The joint objective adds losses across selected quantiles to MSE, using the quantile terms as a regularizer for long-tailed heart-rate responses.The motivation includes very athletic or sedentary people as potential long-tail outliers.Source: Custom loss function · p. 4
Evaluation
Evaluation uses the Fenland cohort’s wearable measurements and assesses heart-rate forecasting alongside transfer learning to user-level outcomes. It describes the data, training setup, baselines, and evaluation metrics.
Evaluation
- The Fenland study enrolled 12,435 adults aged 35–65, with 2,100 wearing chest heart-rate and movement sensors plus a wrist accelerometer; treadmill tests informed VO2max and resting heart rate measurements.The chest ECG recorded heart rate and uniaxial acceleration in 15-second intervals, while the wrist device recorded 60 Hz triaxial acceleration.Source: Evaluation · p. 4, Study protocol · p. 5
- Signals were cleaned, aligned to one observation per 15 seconds, and paired with cyclical month and hour features to represent temporal context.Participants with less than 72 hours of wear were removed.Source: Study protocol · p. 5
- Training used non-overlapping 512-step windows, an 80/20 user-disjoint train/test split, and forecasts of heart rate 15 seconds after the activity input.The network combined two CNN layers with two bidirectional GRU layers; training stopped after 300 epochs or five validation epochs without improvement.Source: Study protocol · p. 5
- For transfer learning, window embeddings were mean-pooled into user-level representations, then standardized and reduced with PCA before fitting classifiers.Binary labels split each variable at its training-set median; PCA retained 90%–99.9% explained variance, yielding 10–160 components.Source: Study protocol · p. 6
Baselines and metrics
- Baselines included global and user heart-rate means, a movement-only convolutional autoencoder, and XGBoost using summary statistics from sensor windows.The autoencoder’s bottleneck embeddings were used only for transfer learning.Source: Baselines and metrics · p. 6
- Forecasting was evaluated with RMSE, MSE, and MAE, while transfer-learning performance was measured by AUC.Results were averaged across three runs for the ablation and baseline comparisons.Source: Baselines and metrics · p. 6
Results
Step2Heart outperforms forecasting baselines, with resting heart rate and the joint loss improving performance. Its learned embeddings transfer to health, fitness, and demographic prediction, while latent-dimension effects differ across outcomes.
Forecasting
- With acceleration, temporal features, and resting heart rate, Step2Heart reaches 9.54 RMSE and outperforms all forecasting baselines.Adding resting heart rate improves RMSE by approximately 2.5; the ablations compare acceleration alone and with temporal and resting-heart-rate features.Source: Forecasting · p. 6
- The models outperform all forecasting baselines and implicitly learn personalized patterns; the user-mean baseline also vastly outperforms the global-mean baseline.The authors connect the personalized patterns to the strong demographic-prediction results reported in the transfer-learning section.Source: Forecasting · p. 6
- The joint loss L0.5∗MSE+Q achieves the lowest error at 9.5 RMSE, while pure quantile loss achieves the best MAE at 6.75.The authors consider both joint-loss variants their best models because their standard deviations overlap, selecting L0.5∗MSE+Q for its lower average error.Source: Forecasting · p. 6
Transfer learning
- Embeddings predicted sex, height, PAEE, and weight with AUCs of 0.93, 0.82, 0.80, and 0.77, respectively; BMI, VO2max, and age were moderately predictable at 0.70 AUC.Generally, A/R/T outperformed A/T; A/T predicted resting HR at 0.62 AUC, while that comparison does not apply to A/R/T because resting HR is an input.Source: Transfer learning · p. 7
- Using resting heart rate as a single predictor does not exceed 0.55 AUC for BMI or age, and an activity-only autoencoder underperforms Step2Heart A/T.The authors interpret this comparison as evidence that mapping activity to heart rate captures physiological information useful for generalizable embeddings.Source: Transfer learning · p. 7
- Retaining more latent dimensions improves sex and height AUC by about 0.20, while PAEE and VO2max appear robust to dimensionality reduction.The authors suggest demographic predictions use a broader range of latent features than fitness predictions, which can use a feature subsample.Source: Transfer learning · p. 7
- In the t-SNE visualization, extreme PAEE users lie in diametrically opposed regions, while most normal users cluster centrally.Figure 3 colors extreme expenditures; the median participant's PAEE is 48 and is shown in white.Source: Figure 3 · p. 7, Transfer learning · p. 7
Conclusion
The authors present a self-supervised wearable feature extractor evaluated on a dataset of more than 1,700 participants with a week of combined heart and activity sensing. Its embeddings perform well on physiological outcomes, with performance patterns varying by latent dimensionality.
Conclusion
- The model was evaluated on more than 1,700 participants with combined heart and activity sensors recorded for a week.The authors describe the dataset as the largest of its kind.Source: Conclusion · p. 7
- The task-agnostic embeddings perform strongly on physiologically meaningful outcomes such as BMI and fitness, and the model outperforms strong baselines in upstream and downstream tasks.The evaluation includes ablation studies.Source: Conclusion · p. 7
- Increasing latent dimensionality improves some outcomes, while others remain invariant; more fine-grained outcome prediction is left for future work.The authors also suggest applying the method to other domains with parallel time series, including weather and traffic.Source: Conclusion · p. 7
Ethical impact and broader implications
The paper situates self-supervised learning as a way to learn representations from unlabelled data when annotating large datasets is impractical. It also flags possible misuse of personalized health representations and calls for data governance.
Ethical impact and broader implications
- Annotating all healthcare data for deep-learning pattern recognition is described as impractical, motivating representation learning from unlabelled data.The authors identify population health as a setting where ground truth may be inadequate or costly to collect.Source: Ethical impact and broader implications · p. 8
- Personalized health representations could be misused for exclusionary insurance policies or unfair credit scoring.The authors characterize the risks associated with Step2Heart as minimal while noting the possibility of misuse.Source: Ethical impact and broader implications · p. 8
- Future development and use should follow data-governance principles that protect user rights, prevent misuse, and promote trust.Source: Ethical impact and broader implications · p. 8