PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals

Yunfei Luo, Yuliang Chen, Asif Salekin, Tauhidur Rahman

arXiv:2412.09758v2cs.LGeess.SP

TL;DR

Wearable-health modeling needs representations that handle changing sensor types and multivariate signals across applications. NORMWEAR combines flexible tokenization, channel-aware fusion, and sensor–text alignment; it reports leading results across task groups and shot settings. The authors note limits in its objectives, regression adaptation, and frequency-band scope.

  • Problem

    Existing approaches are limited by predefined sensor types, univariate inputs, and inconsistent modality-specific preprocessing, restricting generalization to heterogeneous wearable signals.

    Source: §1 · pp. 1–2

  • Method

    NORMWEAR uses multiscale CWT tokenization, channel-aware fusion, and sensor–text representation alignment to support varied sensor configurations and zero-shot inference.

    Source: §1 · p. 2, §3.2 · p. 3, §3.5 · p. 5

  • Results

    NORMWEAR achieves peak performance across activity recognition, EEG analysis, disease-risk evaluation, and vital-sign estimation, with a 3.9% macro-averaged total score improvement over the SoTA time-series self-supervised learning framework.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    The authors present NORMWEAR as a practical starting point for researchers and clinicians using wearable signals, supported by evaluation across health applications.

    Source: §5 · p. 9

    Representation alignment uses a limited set of healthcare objectives, regression adaptation remains open, and the current frequency scope is relatively narrow.

    Source: §5 · p. 9

Abstract

from arXiv · show

Time-series foundation models excel at tasks like forecasting across diverse data types by leveraging informative waveform representations. Wearable sensing data, however, pose unique challenges due to their variability in patterns and frequency bands, especially for healthcare-related outcomes. The main obstacle lies in crafting generalizable representations that adapt efficiently across heterogeneous sensing configurations and applications. To address this, we propose NormWear, the first multi-modal and ubiquitous foundation model designed to extract generalized and informative representations from wearable sensing data. Specifically, we design a channel-aware attention mechanism with a shared special liaison [CLS] token to detect signal patterns in both intra-sensor and inter-sensors. This helps the model to extract more meaningful information considering both time series themselves and the relationships between input sensors. This helps the model to be widely compatible with various sensors settings. NormWear is pretrained on a diverse set of physiological signals, including PPG, ECG, EEG, GSR, and IMU, from various public datasets. Our model shows exceptional generalizability across 11 public wearable sensing datasets, spanning 18 applications in mental health, body state inference, vital sign estimation, and disease risk evaluation. It consistently outperforms competitive baselines under zero-shot, partial-shot, and full-shot settings, indicating broad applicability in real-world health applications.

1 Introduction

Wearable health sensing requires representations that generalize across heterogeneous sensor configurations and multivariate signals. NORMWEAR is introduced to address these gaps with a wearable-focused foundation model and broad downstream evaluation.

1 Introduction

  • Prior wearable-sensing approaches are limited by predefined sensor types or univariate inputs, leaving cross-sensor relationships and heterogeneous multivariate signals unaddressed.These limitations constrain their generalization and usefulness for wearable health monitoring.Source: §1 · pp. 1–2
  • Wearable modalities vary in dimensionality, sampling rate, and resolution, while existing preprocessing approaches lack a consistent pipeline across modalities.The passage identifies a need for a framework accommodating diverse signal requirements.Source: §1 · p. 2
  • NORMWEAR is presented as a wearable-sensing foundation model pretrained on more than 2.5 million multivariate segments totaling 14,943 hours and evaluated on 18 downstream tasks.The evaluation spans zero-shot, few-shot, and full-shot settings.Source: §1 · p. 2
  • The model is designed to process arbitrary configurations of multivariate signals from heart, skin, brain, and physical-body sensors.Its methods include CWT-based modality- and number-agnostic tokenization, channel-aware attention, and a human-sensing-adapted fusion mechanism for zero-shot inference.Source: §1 · p. 2
  • The work integrates wearable datasets with varied input-channel counts and evaluates applications spanning mental and physical state inference, vital-sign estimation, and disease-risk evaluation.The authors report making code and cleaned data publicly available for reproducible research.Source: §1 · p. 2

2 Related Work

NORMWEAR targets gaps in wearable-health evaluation and compatibility by supporting arbitrary sensor modalities and channel counts, and is evaluated across 18 digital healthcare tasks.

2 Related Work

  • Earlier time-series and wearable-health approaches are described as limited by shallow health-task evaluation, specific sensor types, univariate data, and difficulty handling heterogeneous multivariate signals.NORMWEAR is presented as modality- and channel-count-agnostic and evaluated on 18 digital healthcare tasks, with peak performance against the stated time-series baselines.Source: §2 · p. 3

3 Method

NORMWEAR combines flexible dataset construction and multiscale signal tokenization with shared channel processing and channel-aware fusion. It also aligns aggregated sensor representations with text-derived labels to support zero-shot inference.

3.1 Dataset construction for model pretraining and downstream evaluation

  • The pretraining corpus expands nine public datasets from about 230,962 segments and 4,294 hours to 2.5 million segments and 14,943 hours using heuristic augmentation.The pretraining data span PPG, ECG, EEG, GSR, PCG, and IMU, with variable input-channel counts; 11 additional datasets support evaluation across 18 tasks.Source: §3.1 · p. 3

3.2 Tokenization

  • Tokenization computes first and second signal derivatives, applies CWT to raw and derivative series, stacks three scalograms, then divides them into ViT-style patches.The multiscale representation uses Mexican Hat wavelets and scales 1–64, avoiding sensor-specific reconfiguration across signal types.Source: §3.2 · p. 3

3.3 Share-weighted Encoder

  • Each channel is processed as an independent stream by a shared transformer backbone, allowing different sensor configurations during pretraining and fine-tuning.This design is described as reducing computational cost while learning each sensor’s temporal characteristics before cross-channel fusion.Source: §3.3 · p. 4

3.4 Channel-Aware Attention with Liaison Special Token

  • The model uses per-channel [CLS] tokens as liaisons for exchanging information through channel-aware fusion, and [CLS]-attention fusion achieves the best downstream performance among the tested fusion approaches.The fusion is compatible with variable channel counts and channel order; pretraining also reconstructs randomly masked inputs, with temporal-and-scale masking performing best among four strategies.Source: §3.4 · pp. 4–5

3.5 Sensor-Semantic Representation Alignment

  • For zero-shot inference, sensor embeddings are aligned with paired text descriptions in a shared latent space, enabling unseen inputs to be matched to text-derived labels.MSiTF aggregates patches using relevance, recency, and importance scores; the resulting fixed-length representation is trained with contrastive learning and an auxiliary vector-distance loss.Source: §3.5 · pp. 5–6

4 Experiments

The experiments compare NormWear with diverse wearable-signal baselines across zero-, partial-, and full-shot evaluations. NormWear leads broadly, with gains on macro-averaged performance, while vital-sign estimation remains comparable across methods.

4 Experiments

  • Evaluation covers 11 public datasets and 18 tasks, using unified linear probing; partial-shot uses 10% of training data, with AUC ROC for classification and relative accuracy for regression.Full-shot results are reported in Table 2; partial-shot results are reported in Table 11.Source: §4 · p. 6, §4.3 · p. 8

4.1 Selection of baselines covering representative modeling strategies

  • Baselines represent handcrafted statistical, generic time-series, spectrum-based, and sensor-specific strategies; uni-modal signals are encoded separately and concatenated for a matched field of view.The selected sensor-specific baselines include PaPaGei for PPG, ECG-FM for ECG, and CBraMod for EEG.Source: §4.1 · pp. 6–7

4.2 Zero-shot Evaluation

  • Zero-shot inference uses representation alignment to pretrain MSiTF; NormWear with MSiTF outperforms baselines, and removing IMP or text augmentation lowers performance.CLAP's Manhattan-distance and dot-product retrieval show no statistically significant performance difference.Source: §4.2 · p. 7

4.3 Partial-shot and Full-shot Evaluation

  • NormWear achieves peak performance across task groups, with macro-averaged gains of 3.9% over Zhang et al. (2022), 5.3% over Wu et al. (2023), 6.1% over Ansari et al. (2024), and 5.2% over statistical baselines.Figure 4 shows full-shot linear-probing trends; Table 2 details downstream application results.Source: Figure 4 · p. 7, §4.3 · p. 8, Table 2 · p. 8
  • Vital-sign methods are comparable; NormWear prioritizes cross-modal adaptability over modality-specific optimization, which may slightly reduce performance on highly specialized tasks.The paper notes that single-signal models excel in their domains through deeper modality-focused training.Source: §4.3 · p. 9

5 Conclusion and Discussion

The conclusion presents NORMWEAR as a practical starting point for wearable-signal research, with representations intended for several downstream uses. The discussion identifies limits in its alignment objectives, regression support, and frequency-band scope.

5 Conclusion and Discussion

  • NORMWEAR is presented as a practical starting point for wearable-signal research, with representations usable for modeling, clustering, retrieval, and real-time health-state monitoring with minimal tuning.The authors say its generalization was evaluated across ubiquitous health applications.Source: §5 · p. 9
  • The authors identify larger-scale clinical applications and intervention modeling with embedding vectors as future research directions.Source: §5 · p. 9
  • The representation-alignment component is trained on a limited set of healthcare objectives, and broader semantic labels may improve generalization.Source: §5 · p. 9
  • Regression remains an open adaptation challenge, while the current design supports classification tasks well.Source: §5 · p. 9
  • NORMWEAR currently targets physiological signals with relatively narrow frequency bands; audio and lower-resolution clinical summaries are proposed as future extensions.Source: §5 · p. 9

Ethics Statement

The study uses publicly available data for pretraining and evaluation and credits the original authors.

Ethics Statement

A Datasets

The dataset section describes the pretraining collection and a range of downstream datasets spanning wearable modalities and health tasks. The curated pretraining recordings are augmented, and downstream evaluation datasets are presented as unseen during pretraining.

A Datasets

  • The pretraining collection was curated from about 385 hours of recordings and expanded to 4,294 hours through augmentation.Figure 6 summarizes its distributions by sensor modality and physiological information.Source: A Datasets · p. 16, Figure 6 · p. 16
  • Table 4 overviews the downstream evaluation datasets by dataset, modality, and task type.The table caption identifies these datasets as unseen during pretraining.Source: Table 4 · p. 16, A Datasets · p. 16
  • WESAD combines chest- and wrist-worn sensor data for classifying neutral, stress, and amusement states.The collection includes physiological and motion data from 15 subjects; its chest and wrist sensor channels are described separately.Source: A Datasets · pp. 16–17
  • UCI-HAR provides six-channel accelerometer and gyroscope data for classifying six human activities.Source: A Datasets · p. 17
  • The downstream datasets include EEG tasks such as driver-fatigue detection, seizure recognition, and distinguishing eye-relaxation states.Other listed EEG tasks cover tumor area, health area, and eyes-open versus eyes-closed classification.Source: A Datasets · p. 17
  • The collection also includes ECG heartbeat classification, PPG-based health tasks, and datasets for blood-pressure, hemoglobin, and fetal-heart-rate estimation.The cited datasets include ECG Heartbeat Categorization, PPG-China, Non-invasive Blood Pressure Estimation, PPG-HGB, and Fetal-fPCG.Source: A Datasets · pp. 17–18

B.1 Data Preprocess.

The methods describe standardized preprocessing, heuristic time-series augmentation, and a masked-autoencoder pretraining pipeline. Additional sections cover representation alignment, text supervision, statistical baselines, and score visualization.

B.1 Data Preprocess.

  • Signals are resampled to 65 Hz and six-second intervals, detrended, and Gaussian-smoothed; shorter samples are zero-padded for batching.The downstream linear-probing split is 80% training and 20% test, stratified by anonymized subject ID when available.Source: §B.1 · pp. 18–19

B.2 Data Augmentation.

  • Heuristic augmentation swaps a randomly sampled segment from one time series into another, expanding the training set to 2,586,404 segments and 4,294 hours.Each sub-dataset is augmented by a factor of 10.Source: §B.2 · p. 19

B.3 Pretraining Framework.

  • Pretraining uses structured masking across frequency and time, encodes unmasked tokens, and decodes reconstructed input series using MSE loss.The pipeline uses 18 encoder/fusion blocks, a lightweight decoder, and Figure 7 illustrates an original series, masked CWT image, and reconstruction.Source: §B.3 · pp. 19–20, Figure 7 · p. 20

B.4 MSiTF.

B.5 Aligner Module, Objective Function, and Pretraining.

  • The alignment module matches a fused sensor representation to a semantic embedding of a ground-truth sentence, using contrastive learning followed by a similarity-based refinement loss.The passage identifies batch size and a learnable temperature parameter for the contrastive loss, and describes vector distance as a supplemental penalty in refinement.Source: §B.5 · p. 21

B.6 Sentence template example for signal-sext alignment.

  • Categorical labels are converted into varied natural-language descriptions for alignment supervision, with 20 GPT-3.5-generated alternatives per sentence and one sampled during pretraining.At inference, the closest label-sentence embedding to the model embedding is the predicted result.Source: §B.6 · p. 21

B.7 Statistical Feature List.

  • The statistical baseline uses time-domain summary statistics and frequency-domain spectral and quantile features.The lists include mean, standard deviation, extrema, skewness, kurtosis, spectral centroid, spread, and frequency quantiles.Source: §B.7 · pp. 21–22

B.8 Radar Plot or Performance Trend.

  • The radar-plot procedure applies Softmax to raw performance scores to emphasize relative differences, while noting absolute scores may differ from originally reported values.Source: §B.8 · p. 22

C Ablation Study

The ablations compare fusion, masking, input representations, and evaluation settings. [CLS]-token fusion generally gives the best accuracy, while the additional tests examine robustness and the contribution of demographic information.

Fusion Schemes

  • [CLS]-token fusion generally achieves the best accuracy among the compared fusion schemes, with a minor increase in parameters.All-attention fusion was excluded because it was computationally prohibitive.Source: C Ablation Study · p. 22, Table 8 · p. 22

Masking Strategies in Pre-training

  • Masking across both scale and time axes performs best among the tested strategies at a consistent mask ratio of 0.8.The comparison includes temporal-and-scale, scale-only, temporal-only, and unstructured masking.Source: C Ablation Study · p. 23, Table 9 · p. 23

Input Representations

  • The model trained on CWT scalograms performs better than the model trained on raw time series.CWT scalograms represent signals in time-frequency form, whereas raw inputs retain the original sensor data.Source: C Ablation Study · p. 24, Table 10 · p. 24

Semi-Supervised Learning (Partial-shot)

  • With 10% of labels, frozen NormWear features are evaluated using a linear classifier for classification and regression tasks.The study excludes 1% labels because that would leave fewer than ten labeled instances in the downstream medical dataset.Source: C Ablation Study · p. 24, Table 11 · p. 25

Permutation-Invariant Input Channel Analysis

  • Performance remains consistent when the input channel order is circularly shifted, suggesting the model does not depend on a fixed channel configuration.The comparison tests original channel order against channel shuffling across tasks.Source: C Ablation Study · p. 25, Table 12 · p. 25

k-fold Analysis

  • On tasks with 30 or fewer subjects, NormWear outperforms the baselines under 5-fold subject-stratified cross-validation.The protocol applies to all downstream tasks meeting the subject-count threshold.Source: C Ablation Study · p. 25, Table 13 · p. 25

Demographic Analysis

  • Demographic information and wearable-signal representations excel on different tasks, and concatenation improves performance in most cases but sometimes reduces it.The occasional decline suggests a confounding relationship between the two feature sources.Source: C Ablation Study · p. 26, Table 14 · p. 26

D Statistical significance on the model comparison

The paper tests whether NormWear’s performance advantage is statistically significant using repeated evaluations, permutation tests, and a critical difference diagram. The reported tests indicate significant differences from baselines in nearly all cases.

Evaluation procedure

  • Downstream evaluations were repeated 100 times per model and task without fixing the random seed, and the results remained consistent.The paper attributes consistency to stable optimization.Source: D Statistical significance on the… · p. 26

Permutation test

Critical difference analysis

  • The Friedman test finds different model-score distributions across downstream tasks (P < 0.001), and the critical difference diagram separates NormWear from the other models.Conover post hoc tests examine pairwise differences; NormWear-baseline p-values appear in Table 8.Source: D Statistical significance on the… · p. 26

E Supplementary Metrics

The supplementary results report a broad performance advantage over baseline models and provide additional performance metrics and a visual comparison across datasets.

Critical difference diagram

  • The critical difference diagram is a visual comparison of model performance across datasets and is intended to highlight statistically significant differences.The supplied caption does not identify a specific outcome.Source: Figure 9 · p. 27

Supplementary performance results

  • NormWear consistently exceeds the baseline models by a wide margin in the reported performance results.Figure 4 summarizes performance, with further details in Table 15.Source: E Supplementary Metrics · p. 27, Table 15 · p. 27
  • Table 15 provides details of incidental performance metrics.The supplied table caption does not specify the metrics or their values.Source: Table 15 · p. 27

F Scaling up the Pretraining Data Size

Increasing pretraining data from about 37k to nearly 2.5M samples is associated with significantly improved average downstream accuracy. The remaining analyses examine channel-fusion complexity, MSiTF runtime and memory, and visualizations of learned signal representations.

F Scaling up the Pretraining Data Size

  • Average downstream accuracy significantly improves as pretraining data grows from approximately 37k to nearly 2.5M wearable-signal samples.Figure 10 plots average task accuracy against pretraining sample count; the paper interprets the trend as adherence to a scaling law.Source: F Scaling up the Pretraining Data Size · p. 28, Figure 10 · p. 28

G.1 All-Attention

G.2 Cross-Attention

  • Pairwise cross-attention has total complexity O(d · L · C^2).The approach computes cross-attention between channel pairs.Source: §G.2 · p. 29

G.3 [CLS]-Attention

  • The final model uses [CLS]-attention, applying self-attention only to each channel’s [CLS] embedding.The supplied passage identifies this as the adopted approach but does not give its resulting complexity expression.Source: §G.3 · p. 29

G.4 Mean-pool Attention

  • Mean-pool attention first computes a mean representation per channel, giving total complexity O(d · (L · C + C^2)).Self-attention over the mean representations has complexity O(d · C^2).Source: §G.4 · p. 30

H MSiTF Complexity analysis

  • With latent size d treated as constant, MSiTF has runtime complexity O(p + c) and memory complexity O(p + c).Here p is the number of patches and c the number of available answer choices; the memory analysis treats text-encoder, NormWear, and latent sizes as constants.Source: H MSiTF Complexity analysis · p. 30

I.1 The model is agnostic to the input signals

  • Without signal-modality input, last-layer [CLS] representations cluster by body location and separate across locations in the visualization.The analysis samples 500 signals per sensor type and visualizes them with PCA-initialized t-SNE.Source: §I.1 · p. 30

I.2 Waveform visualization

  • For diabetes classification, the waveform visualization marks the top ten positively and negatively associated patterns and uses red dots for time-step relevance.The associations come from linear-probing regression parameters, while the relevance mechanism identifies each time step’s contribution during inference.Source: §I.2 · pp. 30–31

I.3 Quantify the intrinsic behaviors: nonlinear dynamics analysis on the layer-wise waveforms

Layer-wise nonlinear-dynamics analysis indicates that deeper representations become less periodic and more unpredictable for GSR, EEG, and ACC, while PPG and ECG remain more stable. Exploratory embedding and gating visualizations show class structure and task-dependent feature weighting, and reconstruction examples include an untrained modality.

I.3 Quantify the intrinsic behaviors: nonlinear dynamics analysis on the layer-wise waveforms

  • For GSR, EEG, and ACC, deeper layers show lower self-correlation and higher unpredictability, while PPG and ECG remain more stable across layers.The analysis uses nonlinear-dynamics metrics to characterize intermediate-layer representations; decreasing Lyapunov exponent is also reported across layers.Source: §I.3 · p. 31

I.4 T-SNE plot among classes

  • Exploratory t-SNE plots show separation of seizure and normal EEG samples, clustered physical activity types, and cohesive clusters of abnormal ECG beats.The plots reduce embeddings to two dimensions and do not establish superiority or fully capture their high-dimensional properties.Source: §I.4 · p. 32, Figure 14 · p. 32

I.5 Supplementary Qualitative Analysis of MSiTF

  • Relevance and importance scores vary by task: eye-closure detection emphasizes the last patches, while seizure detection emphasizes patches with large fluctuations.The recency score comes from a fixed decay function and remains the same across tasks.Source: §I.5 · p. 33, Figure 15 · p. 33, Figure 16 · p. 33

J Reconstruction Example

  • At 80% masking, reconstruction examples include Phyatt and WESAD scalograms; WESAD IMU data absent from training are generally reconstructed with high accuracy.The samples are uncurated and random.Source: Figure 17 · p. 34, Figure 18 · p. 35

Related papers

Loading 2412.09758v2…