Large-scale Training of Foundation Models for Wearable Biosignals

• Curated medical labels are scarce because health studies are costly, expertise-intensive, and typically involve limited participants, reducing model generalizability.Source: §1 · p. 1
• Self-supervised learning can pre-train biosignal foundation models without explicit labels, making it suitable for unlabeled wearable data.Source: §1 · p. 1
• Prior biosignal self-supervised studies mostly used clinical or controlled data, whereas wearable accelerometer work has begun addressing free-living conditions.Source: §1 · p. 2
• The study trains PPG and ECG foundation models on 141,207 AHMS participants using participant-level pair selection, stochastic augmentation, momentum training, and regularized InfoNCE.Source: §1 · p. 2
• The pre-trained embeddings are evaluated for demographic and health information, with ablations covering pair selection, modality differences, training frameworks, architectures, model sizes, and augmentations.Source: §1 · p. 2

• Joint embedding architectures use contrastive or non-contrastive objectives, often combining mechanisms that prevent representation collapse.Source: §2 · p. 2
• Health self-supervised learning studies usually use datasets containing only hundreds or thousands of participants recorded in clinical or controlled settings.Source: §2 · p. 2
• Self-supervised learning has been effective across health applications, while wearable accelerometer studies have begun using large free-living datasets.Source: §2 · p. 2
• The authors present the first study, to their knowledge, to pre-train foundation models from large-scale wearable-consumer-device PPG and ECG data.Source: §2 · p. 2

3.1 POSITIVE PAIR SELECTION AND AUGMENTATIONS

• Participant-level positive pairs use augmented views of two different segments from the same participant, and this strategy significantly improves embedding quality and utility.Source: §3.1 · p. 3
• The stochastic augmentation module combines cropping, noise, time warping, magnitude warping, and channel operations, with stronger probabilities for ECG because it has less within-participant variability.Source: §3.1 · p. 3

3.2 REGULARIZED INFONCE LOSS

• InfoNCE contrasts positive-pair representations against other batch samples to avoid collapse, while KoLeo regularization encourages a uniform feature span.Source: §3.2 · p. 3

3.3 MOMENTUM TRAINING

• The pre-training data are partitioned into 80% training, 10% validation, and 10% occasional post-training assessment participants.Source: Table 1 · p. 4
• Momentum training updates an online encoder by backpropagation and a momentum encoder by lagging exponential moving average, increasing mismatch between positive representations.Source: §3.3 · pp. 3–4

4.1 DATASETS

• ∼20M PPG segments from ∼141K participants and ∼3.75M ECG segments from ∼106K participants form the pre-training datasets.PPG recordings are 60 seconds, while ECG recordings are 30 seconds; both datasets impose minimum per-participant segment counts and apply modality-specific preprocessing.Source: §4.1 · p. 4

4.2 EVALUATION METRICS

• Downstream evaluation predicts age, biological sex, and BMI using participant-level linear probing with participant-disjoint stratified splits.Classification uses AUC and pAUC at 10% false-positive rate, while regression uses mean absolute error.Source: §4.2 · pp. 4–5
• Survey-question targets are evaluated by comparing AUC from PPG embeddings with a baseline containing demographics and measured heart rate.The survey targets include health conditions, medications, and lifestyle habits, with training and evaluation splits stratified by participant.Source: §4.2 · p. 5
• Smooth effective rank provides an unsupervised proxy for downstream evaluation by measuring entropy in the embeddings’ singular-value distribution.The metric is averaged across batches of 256 in the pre-training validation split.Source: §4.2 · p. 5

4.3 IMPLEMENTATION DETAILS

• The default encoder is an EfficientNet-style 1D convolutional network producing 256-dimensional embeddings, with modality-specific parameter counts for PPG and ECG.The projection head maps embeddings into a 128-dimensional loss space; training uses InfoNCE temperature 0.04, KoLeo weight 0.1, Adam, and 32 A100 GPUs.Source: §4.3 · pp. 5–6
• Figure 1 compares embedding-based and baseline ROC AUC values for AHMS survey targets separately for PPG and ECG.Each marker denotes one questionnaire target, with embedding ROC AUC on the y-axis and baseline-feature ROC AUC on the x-axis.Source: Figure 1 · p. 6

5.1 PPG AND ECG FOUNDATION MODELS ENCODE PARTICIPANTS HEALTH INFORMATION

• PPG embeddings almost always outperformed baseline demographic and heart-rate features for predicting downstream health conditions, whereas ECG embeddings encoded less participant-health information overall.The PPG advantage was not explained by the larger PPG dataset, since a smaller PPG model with ECG-comparable segment counts still performed significantly better.Source: §5.1 · pp. 6–7

5.2.1 POSITIVE PAIR SELECTION

• Participant-level positive pairs consistently produced more predictive PPG and ECG models than segment-level positive pairs.The improvement held for evaluations at both participant and segment granularity.Source: §5.2.1 · p. 7

5.2.2 DISTINCTIONS IN PPG AND ECG PRE-TRAINING AND EMBEDDINGS

• ECG embeddings formed denser participant clusters, had significantly smaller validation InfoNCE loss, and showed statistically smaller dispersion ratios than PPG embeddings.The dispersion ratio measures within-participant variability relative to across-participant variability; Figure 2 summarizes the t-SNE, InfoNCE, and dispersion comparisons.Source: §5.2.2 · p. 7, Figure 2 · p. 8

5.2.3 PRE-TRAINING FRAMEWORK

• Contrastive frameworks were generally more robust and informative than the compared non-contrastive or baseline variations, while removing KoLeo regularization reduced evaluation metrics for both modalities.InfoNCE and smooth effective rank correlated with downstream performance but were not directly comparable across PPG and ECG modalities.Source: §5.2.3 · p. 8, Table 3 · p. 9

5.2.4 ENCODER ARCHITECTURE

• 1D-EfficientNet achieved performance similar to alternative 1D-ResNet and 1D-ViT encoders with significantly fewer parameters.Its smaller memory footprint motivated its selection as the default backbone for potential wearable-device use.Source: §5.2.4 · p. 8, Table 4 · p. 9

5.2.5 AUGMENTATIONS

• PPG pre-training was more sensitive to the choice of single augmentation, while channel permute and cut out were the most effective isolated augmentations for PPG and ECG, respectively.The augmentation module was not comprehensively tuned, and modality-specific augmentation design remains a future direction.Source: §5.2.5 · p. 9

• The models encoded demographics accurately and information predictive of a broad range of self-reported health conditions and medication categories.The downstream labels were opportunistically constructed from self-reported surveys and may be suboptimal compared with more accurate labels.Source: §6 · p. 9
• Because the targets came from self-reported surveys, future work should quantify these predictions using more accurate labels.Source: §6 · p. 9
• Heart rate, heart-rate variability, and physical activity remain valuable health-related biomarkers whose relationships with PPG and ECG embeddings require further examination.Source: §6 · p. 9
• Future work could improve the framework through modality-specific designs, longitudinal embedding analysis, and multimodal pre-training.Source: §6 · p. 9

• Aggregated data may be made available on request, whereas the study’s sensor data cannot be shared because of the Apple Heart and Movement Study protocol and informed-consent restrictions.Code may also be available on request subject to participant-confidentiality and protocol policies.Source: REPRODUCIBILITY STATEMENT · p. 10

• The framework and BYOL used a constant momentum rate of 0.99, while most models used batch size 256 and initial learning rate 0.001.BYOL used an initial learning rate of 0.00025, and step scheduling was used for faster convergence.Source: §A.1 · p. 15
• Augmentation probabilities were not comprehensively tuned; ECG probabilities were instead doubled to create stronger positive-pair mismatch.The starting probabilities were based on prior work, with cut out initially assigned a higher probability.Source: §A.1 · p. 15
• The 1D-ViT used linear learning-rate warmup for its first 10 epochs to support more stable and optimal training.The 1D-ResNet and 1D-ViT otherwise used training details similar to the default 1D-EfficientNet.Source: §A.1 · p. 15

• AHMS demographic distributions cover age, BMI, biological sex, and ethnicity among participants who answered the corresponding surveys.Source: §A.2 · p. 15
• Processed PPG and ECG signals are illustrated as representative 10-second segments in arbitrary units.Source: §A.2 · p. 15
• Survey analyses use each participant’s latest questionnaire entry before 01/28/2022.The survey includes medical conditions, medications, drinking, and smoking questions, with binary-label logic defined for drinking and smoking targets.Source: §A.2 · p. 15

• Table 9 reports linear-probing evaluations of the targets shown in Fig. 1.Source: §A.3 · p. 15
• Table 12 compares downstream age, biological sex, and BMI performance across the proposed framework, SimCLR, and BYOL.Source: §A.3 · p. 16
• Figure 7 compares smooth effective rank after pre-training with each augmentation function isolated for PPG and ECG.The comparison also includes the full augmentation module.Source: §A.3 · p. 16

A.4 SUPPLEMENTARY DISCUSSION

• PPG embeddings are compared with prior demographic-prediction results, while age and BMI comparisons are qualified by differing cutoffs and dataset distributions.Prior biological-sex results are reported as lower than the PPG and ECG embedding results for one cited comparison, whereas some age and BMI results are not directly comparable.Source: §A.4 · p. 16
• The framework forms positive pairs from augmented views of different segments belonging to the same participant.Source: Figure 3 · p. 17, §A.4 · p. 17
• Training combines two cross-view contrastive losses with KoLeo regularization and updates the momentum encoder as an exponential moving average of the online encoder.Source: §A.4 · p. 17
• The encoder architecture uses one-dimensional convolutional blocks, batch normalization, Swish activation, MBConv1D modules, and average pooling for time-series input.Source: Figure 4 · p. 18
• The supplementary figures show AHMS demographic distributions and representative processed PPG and ECG signals.Source: Figure 5 · p. 18, Figure 6 · p. 18
• Figure 7 evaluates isolated augmentation functions against the full augmentation module for both PPG and ECG using smooth effective rank.Source: Figure 7 · p. 19
• Supplementary tables define survey questions and target-label exclusions for medical conditions, medications, drinking, and smoking.Source: Table 5 · p. 20, Table 6 · p. 21, Table 7 · p. 22
• Tables 9–12 cover linear probing, downstream performance, positive-pair strategies, and pre-training-framework comparisons.Source: Table 9 · p. 23, Table 10 · p. 23, Table 11 · p. 24