Source-linked AI summary
Scaling Wearable Foundation Models
Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, Daniel McDuff
TL;DR
Wearable data are continuous and multimodal but difficult to interpret, and conventional supervised models underuse unlabeled data and often generalize poorly. The paper trains LSM on up to 40 million hours from over 165,000 people and studies scaling across compute, data, and model size. Scaling improves generative reconstruction and forecasting as well as label-efficient exercise and activity recognition.
Problem
Wearable sensing produces large, difficult-to-interpret datasets, while supervised models underuse unlabeled data, specialize by task, and often rely on small populations.
Method
LSM is a multimodal foundation model trained with masked signal reconstruction on wearable data spanning 40 million hours from over 165,000 users.
Results
Scaling compute, data, and model size improves interpolation and extrapolation MAE over the best baseline and improves exercise detection and activity recognition over supervised baselines.
Takeaways & Limitations
LSM demonstrates that scaling wearable foundation models can improve generalization and label efficiency across generative and downstream discriminative tasks.
Takeaways & Limitations
The study uses imputation for missing sensor data, although missingness can correlate with real-world events and introduce imputation bias.
Abstract
from arXiv · showhide
Wearable sensors have become ubiquitous thanks to a variety of health tracking features. The resulting continuous and longitudinal measurements from everyday life generate large volumes of data; however, making sense of these observations for scientific and actionable insights is non-trivial. Inspired by the empirical success of generative modeling, where large neural networks learn powerful representations from vast amounts of text, image, video, or audio data, we investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM, a multimodal foundation model built on the largest wearable-signals dataset with the most extensive range of sensor modalities to date. Our results establish the scaling laws of LSM for tasks such as imputation, interpolation and extrapolation, both across time and sensor modalities. Moreover, we highlight how LSM enables sample-efficient downstream learning for tasks like exercise and activity recognition.
1. Introduction
Wearable data are abundant but difficult to interpret, while conventional supervised models underuse unlabeled data, generalize poorly across tasks, and often rely on small populations. This paper investigates whether scaling compute, data, and model size can produce generalizable wearable foundation models.
- Motivation: Supervised wearable models leave much unlabeled data unused, target single tasks, and often train on small, non-diverse populations.These limitations motivate self-supervised learning on large-scale wearable data.
- Research question: The central question is whether scaling laws from language and vision also apply to multimodal wearable sensor data.The answer could inform model design and generalization across tasks and datasets.
- Research scope: 40 million hours from over 165,000 users support the largest and most diverse wearable scaling study described here.The dataset spans accelerometer, PPG, electrodermal activity, skin temperature, and altimeter signals.
- Approach: LSM scales compute, data, and model parameters while targeting generalizable representations across populations, sensor modalities, and downstream tasks.The study evaluates generative imputation, interpolation, and extrapolation alongside downstream classification.
- Downstream evaluation: The study verifies downstream classification on exercise and activity recognition using ecologically valid, user-annotated events.This tests whether learned representations transfer beyond the pretraining objective.
2. Related Work
Prior sensor foundation models use self-supervised learning to improve representations from large behavioral and physiological datasets, while scaling laws characterize how performance changes with compute, data, and model size. LSM extends this literature through masked input modeling and a systematic wearable scaling analysis.
- Sensor Foundation Models: Existing sensor foundation models use self-supervised pretraining on large behavioral and physiological signal corpora to improve accuracy, robustness, and generalizability.Prior work primarily emphasizes contrastive learning with positive and negative data pairs.
- Sensor Foundation Models: LSM instead focuses on masked input modeling for generative capabilities and systematically varies compute, data size, and model size.The study also considers more sensor inputs and a larger data sample than prior work.
- Time-Series Foundation Models: Time-series foundation models have been pretrained for domains including energy, transportation, finance, and climate, with TimeGPT and Lag-Llama among early examples.Wearable sensing is treated as a multivariate time-series setting.
- Scaling Laws in Deep Learning: Prior scaling-law studies report power-law relationships between testing loss and compute, data volume, or model size when the other resources are held constant.Such behavior has been observed in language, vision, and transfer learning.
3. Data for Wearable Foundation Models
The dataset combines one-minute wearable features from four sensors across a large population, producing 26 normalized signals in 300-minute samples. Subject-based splits and scaling slices support evaluation across data quantity while keeping the test set fixed.
- Sensor Data and Processing: Four wearable sensors—PPG, accelerometer, skin conductance, and altimeter/pressure—produce 26 one-minute-resolution signals.Raw sensor data are not stored at this scale because of battery and memory constraints.
- Sensor Data and Processing: The feature set includes heart-rate and PPG-derived measures, skin-temperature and conductance features, altimeter statistics, and multiple accelerometer descriptors.Accelerometer features include jerk, steps, energy measures, covariance, zero crossings, and standard deviation.
- Sensor Data and Processing: Signals are globally z-score normalized, and missing minutes are interpolated or backfilled before masked-autoencoder processing.The preprocessing handles missing values within each 300-minute window.
- Dataset Construction: The pretraining dataset contains 165,090 subjects using Fitbit Sense 2 or Google Pixel Watch 2 devices during January 2023 through July 2024.Participants consented to research and development use, and the data were de-identified.
- Dataset Construction: An 80–20 subject-based train-test split keeps the test set fixed while training slices vary by sample count or number of subjects.Sample-scaling shuffles training data, whereas subject-scaling groups all samples by subject identifier.
4. Sensor Modeling Tasks
LSM is trained and evaluated through generative tasks that reconstruct missing sensor information across time or modalities, then tested on discriminative activity tasks. Random imputation is selected as the pretraining task, while interpolation and forecasting probe temporal completion and prediction.
- Generative Tasks: Random imputation removes patches across time and signal axes, requiring predictions from partial input and serving as the primary pretraining task.The task was empirically chosen for pretraining.
- Generative Tasks: Temporal interpolation fills periods where all sensor data are missing, typically using observations before and after the gap.Such gaps can arise when devices are removed for charging or sensors are disabled to save battery.
- Generative Tasks: Sensor imputation infers missing sensor streams from continuously available modalities by exploiting correlations among physiological signals.The task addresses hardware constraints such as battery and wireless connectivity limitations.
- Generative Tasks: Temporal extrapolation forecasts future sensor measurements from current and historical data, supporting applications such as fatigue detection and anomaly detection.It is described as more challenging than interpolation.
- Discriminative Tasks: Discriminative evaluation covers exercise detection and eight-class activity recognition using user-labeled sensor windows.The activity classes include biking, elliptical, HIIT, strength training, swimming, running, walking, and weightlifting.
- Generative Tasks: The generative task suite includes random imputation, temporal interpolation, sensor imputation, and temporal extrapolation.These tasks test reconstruction of missing patches, missing time periods, missing modalities, and future measurements.
5. Experiments & Results
LSM exhibits scaling behavior across compute, data, and model size, improving generative and discriminative wearable-sensor tasks while requiring sufficient data to exploit larger models. It also shows label-efficient downstream learning, although long-range reconstruction and finer-grained signals remain challenging.
- Training Procedures: Scaling compute follows a power law across multiple orders of magnitude, but performance saturates at the highest compute levels rather than approaching zero error.The fitted relationship is L = aC^b + c, where the additive constant models saturation.
- Training Procedures: Performance improves with more data and larger models, but larger models require substantially more data to prevent overfitting and fully use their capacity.Performance gains diminish around 10^7 hours, while larger models continue benefiting from more than 1 million hours; scaling to 10^8 parameters yields gains in test loss and generative zero-shot performance.
- Results & Discussion: 16% to 23% lower temporal interpolation MAE and 20% to 21% lower extrapolation MAE than the best baseline across five durations.LSM also improves exercise detection and 8-class activity recognition over supervised baselines by 27% / 29% in accuracy and 57% / 54% in mAP.
- Results & Discussion: Total wearable-data hours matter more than subject count at fixed total hours, although scaling both subjects and hours per subject supports intra- and inter-subject variability.The authors note that increasing subject diversity may become more important at larger sample sizes.
- Results & Discussion: Imputation, interpolation, and extrapolation improve with scaling but remain difficult when missing windows are large or predictions extend several hours into the future.These conditions degrade accuracy and increase uncertainty, highlighting persistent challenges with long-range dependencies and large data gaps.
6. Limitations & Future Work
Scaling gains saturate at roughly 10^7 hours and model sizes in the hundreds of millions, while missingness, dataset construction, and limited task coverage remain unresolved challenges.
- Scaling laws saturate around 10^7 data hours and model sizes in the 100 millions, limiting further gains from simply increasing scale.The authors attribute saturation to the masking task, dataset difficulty, and data-cleaning limitations.
- The dataset has uneven temporal coverage, with two years from January–June but only one year from July–December, which may bias activity representation.
- Wearable missingness is often not at random because events such as charging or loose fitting correlate with absent data, potentially biasing imputation.The authors call for models that handle missingness directly rather than relying on potentially biased imputation.
- Future evaluation should include a broader range of discriminative classification and regression tasks to more comprehensively assess pretrained-model benefits.
7. Broader Impact
Wearable foundation models may increase the value of sensor data for health applications, but privacy constraints limit sharing of the underlying streams.
- Foundation models may enable new insights from wearable data for physical activity, sleep, and actionable health applications.
- Some sensor streams cannot be fully anonymized, so participant privacy prevents releasing the training data despite support for open science.
- LSM is positioned as a step toward realistic synthetic sensor datasets that could support cross-institutional research without exposing participant data.
8. Conclusion
The paper introduces LSM, a multimodal wearable foundation model trained on data from more than 165,000 individuals, and reports gains across generative and discriminative tasks as scale increases.
- LSM is trained on 40 million hours of wearable sensor data from over 165,000 individuals and establishes scaling laws for sensor models.
- Scaling data, model size, and compute improves imputation, interpolation, extrapolation, exercise detection, and activity recognition performance.
- LSM demonstrates substantial gains in generalization and efficiency for downstream wearable-health tasks.
A. Model Design Choices and Ablations
The ablations select masked-autoencoder design choices that balance reconstruction quality, downstream performance, and computational cost, while revealing diminishing returns from larger models and datasets.
- Masking Ratio: An 80% masking ratio yields the best temporal interpolation and extrapolation performance among ratios from 30% to 90%.
- Masking Strategy: Random masking is selected for scaling experiments because it consistently performs best across the two primary generative tasks.
- Sensor Order: Clustering signals by sensor type generally improves downstream performance, particularly for heterogeneous accelerometry, EDA, and heart-rate modalities.
- Patch Size: A 10-minute by 5-feature patch offers the best accuracy–compute trade-off, with 15.94 gFlops and MAE values of 0.24 for imputation and 0.37 for forecasting.
- Model Size: Scaling beyond the LSM-B model produces no additional reconstruction-loss or downstream-task improvements, so neural baselines use a ViT-110M backbone.
- Data Scaling: The LSM Base model trained on 6.6 million hours achieves the best overall generative performance, while extending training to 40 million hours yields tapering benefits.
B.2. Results of Scaling Experiments for Discriminative Tasks
Scaling the pretrained LSM improves downstream discriminative performance and supports label-efficient activity recognition, while learned embeddings retain some subject-specific structure.
- Discriminative Performance wrt. Data Scaling: Scaling the dataset significantly benefits downstream discriminative tasks, especially during fine-tuning, and improves label-efficient transfer learning.Few-shot activity recognition also shows more sample-efficient performance than supervised training.
- Activity Recognition Errors: Many activity classes are mistaken for Walk, likely because five-hour inputs contain substantial walking even when another activity is labeled.This confusion pattern provides a concrete interpretation of the activity-recognition errors.
- Representation Analysis: Pretraining-data scaling produces subtle improvements in activity clustering, while fine-tuning remains critical for discriminating activities.The learned representation also embeds subject dependencies linked to individual physiology and activity definitions.
- Signal Structure: The 26 input signals are grouped by sensor in a pairwise correlation matrix, with an alternate ordering minimizing absolute correlation between adjacent features.The matrix covers accelerometer, PPG, temperature, and altimeter-derived signals.
C. Details of Training and Hyperparameters
Training uses hyperparameters selected through sweeps, sensor-valid augmentations, and MAE-related pretraining configurations documented for the compared methods.
- Hyperparameters: Pretraining and fine-tuning hyperparameters were selected through sweeps and are listed alongside settings for linear probes and supervised baselines.Few-shot settings are similar to the fine-tuning and supervised configurations, with slight learning-rate changes.
- Training Augmentations: Sensor-image augmentation uses temporal flips, temporal stretching followed by cropping, and additive Gaussian noise.Random crop and resize are avoided because cropping can remove a subset of sensor signals.
D.1. Pretraining Methods
LSM compares reconstruction-based and contrastive self-supervised pretraining, motivated by generative applications such as imputing missing signals and forecasting future values.
- Contrastive Methods: Contrastive pretraining learns representations by bringing augmented views of the same sample together while separating representations of different samples.SimCLR is an example of this approach, while MSN combines masked denoising with representation matching.
- Pretraining Motivation: Generative pretraining is motivated by the usefulness of imputing missing data and forecasting signals, whereas sensor augmentations can be difficult to design without changing labels.This motivates reconstruction-based methods alongside contrastive baselines.
- Masked Auto Encoder: Masked Auto Encoding randomly masks input data and trains the model to reconstruct the missing portions, encouraging representations that capture missing information.Reconstructions are illustrated for masked sensor patches across imputation, interpolation, extrapolation, and scaling settings.
- Generative Baselines: Generative baselines include mean fill, nearest-neighbor fill, and linear interpolation, with zero replacement when an entire feature is missing.Zero replacement is valid because all features are z-score normalized around zero.