PaperlayerView exact revision on arXiv

Source-linked AI summary

CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

Hamza Shafiq, Hung Manh Pham, Bin Zhu, Pan Zhou, Jun Hu, Aaqib Saeed

arXiv:2608.12944v1cs.LGeess.IVstat.ML

TL;DR

Existing cardiac foundation models learn within individual sensing modalities, leaving shared physiology across ECG, PPG, and PCG underused. CardioState-JEPA learns a shared, delay-aware latent representation across these signals and transfers across modalities, surpassing strong self-supervised signal baselines on representative tasks.

  • Problem

    Existing cardiac models typically train within one sensing modality, limiting explicit use of shared physiology across electrical, hemodynamic, and acoustic signals.

  • Method

    CardioState-JEPA uses intra-modal masked latent prediction and delay-aware cross-modal alignment to learn shared cardiac representations from ECG, PPG, and PCG.

  • Results

    The unified frozen encoder transfers across ECG, PPG, and PCG tasks, outperforming modality-specific baselines on average for PPG and PCG while remaining competitive on ECG.

  • Takeaways & Limitations

    Heterogeneous cardiac signals can mutually supervise a shared physiological representation that serves as a unified alternative to separate per-sensor learning.

  • Takeaways & Limitations

    Cross-modal alignment relies on comparatively small paired and trimodal corpora, while the delay aligner assumes reliable detection of a reference event.

Abstract

from arXiv · show

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.

1 Introduction

CardioState-JEPA addresses the fragmentation of cardiac representation learning by modeling ECG, PPG, and PCG as time-shifted observations of shared cardiac physiology. It combines shared latent-state prediction, delay-aware alignment, and a two-stage unimodal-to-paired training curriculum, then transfers across 25 downstream tasks.

  • Motivation: ECG, PPG, and PCG capture electrical, hemodynamic, and acoustic aspects of the same cardiac cycle, but cardiac models are typically trained within one modality.This separation leaves structure learned from one physiological view unused by the others.
  • Motivation: Physiological delays and differing waveform appearances make naive timestamp alignment and raw cross-signal reconstruction unreliable.PCG follows electrical activation, PPG arrives later through pulse transit, and one cardiac event can appear as a deflection, burst, or smooth pulse wave.
  • Alignment: The model explicitly estimates and supervises inter-modality delay, aligning electrical, hemodynamic, and acoustic signals in latent cardiac time.This delay-aware cross-modal prediction complements intra-modal masked latent prediction.
  • Model: CardioState-JEPA maps heterogeneous ECG, PPG, and PCG waveforms into a common token space and processes them with one shared Transformer encoder.Its physiology-aware joint-embedding predictive architecture predicts masked latent cardiac states rather than sensor-specific waveform appearance.
  • Training: CardioState-JEPA first learns within-modality structure from abundant unimodal recordings, then aligns modalities with paired recordings through delay-aware cross-modal prediction.The two-stage curriculum addresses the scarcity of synchronized multi-sensor recordings.
  • Evaluation: 25 downstream tasks spanning ECG, PPG, and PCG show that the frozen shared encoder outperforms modality-specific baselines on average for PPG and PCG while remaining competitive on ECG.The results support a unified alternative to separate per-sensor representation learning.

2 Related Work

Related work spans general time-series foundation models and cardiac signal foundation models, with both areas emphasizing self-supervised learning over large unlabeled datasets. Cardiac models build on these methods to learn transferable representations for ECG tasks including arrhythmia, rhythm, and morphology classification.

  • General Time Series Foundation Models: General time-series foundation models learn transferable representations for forecasting, classification, and anomaly detection.These models have become an important research direction across multiple time-series tasks.
  • General Time Series Foundation Models: Most general time-series foundation models use self-supervised learning to exploit large unlabeled time-series corpora.The passage identifies contrastive, masked-prediction, and joint-embedding predictive architectures as major method families.
  • Cardiac Signal Foundation Models: Recent cardiac signal foundation models apply self-supervised learning as large unlabeled cardiac datasets become increasingly accessible.The cardiac-signal literature builds on progress in general self-supervised learning.
  • Cardiac Signal Foundation Models: Early ECG self-supervised models such as ST-MEM transfer to arrhythmia, rhythm, and morphology classification using masked prediction objectives.ST-MEM uses publicly available recordings for pretraining.

3 Method

CardioState-JEPA maps ECG, PPG, and PCG into a shared cardiac representation through modality-specific tokenizers and one Transformer encoder. Its two-stage JEPA training first learns unimodal structure, then aligns paired modalities in physiological cardiac time using learned, supervised delays.

  • Shared representation: ECG, PPG, and PCG are modeled as delayed renderings of one latent cardiac state, differing in sensor nuisance, transduction, and physiological timing.ECG is nearly immediate, PCG follows electromechanical coupling, and PPG follows pulse transit.
  • Shared representation: Modality-specific tokenizers convert ECG, PPG, and PCG into comparable tokens, which a single shared Transformer encoder processes without modality-specific layers.The tokenizer uses modality-specific strides, multiscale depthwise context, projection, positional embeddings, and modality embeddings.
  • Stage I: Unimodal learning: Stage I predicts masked latent regions from visible unimodal context rather than reconstructing waveforms, encouraging codes to capture rhythm and phase structure instead of sensor noise.Contiguous masked blocks can span whole beats, preventing reliance on local interpolation.
  • Stage II: Delay-aware alignment: Stage II performs cross-modal prediction on paired recordings through a learned per-token delay and Gaussian target gathering at shifted cardiac times.The differentiable soft kernel provides gradients to the delay head while avoiding equal-timestamp matching of physiologically offset events.
  • Stage II: Delay-aware alignment: Physiological anchors supervise delay estimates when reliable beats exist, while the curriculum activates paired-data objectives only in Stage II and retains unimodal data to prevent modality drift.Anchors include the ECG–PCG R-peak-to-first-heart-sound interval and ECG–PPG pulse arrival time.

4 Experiments

Across 25 downstream tasks, a single frozen CardioState-JEPA encoder transfers effectively across ECG, PPG, and PCG under patient-disjoint linear probing. Multimodal alignment substantially improves sensor invariance, while trimodal pretraining and masked latent prediction provide complementary benefits.

  • Cross-modal transfer: A single frozen encoder performs consistently across PPG, ECG, and PCG downstream tasks despite serving all three sensing domains.All downstream evaluations use patient-disjoint splits and a task-specific linear head while the pretrained encoder remains frozen.
  • PPG results: 80.4 average classification AUROC and 9.1 average regression MAE improve over PPG baselines from 72.2 and 10.9, respectively.Gains are especially clear for activity recognition, atrial fibrillation detection, respiratory rate estimation, and blood pressure estimation.
  • PCG results: The model achieves the best results on both CirCor murmur detection and CinC2016 abnormal heart-sound detection under linear probing.The same shared encoder is used for ECG, PPG, and PCG.
  • Representation analysis: The modality silhouette falls from 0.121 after Stage I to −0.006 after Stage II, effectively removing sensor-specific separation from the shared cardiac code.The concurrent downstream improvement indicates that the mixing reflects cross-modal alignment rather than representation collapse.
  • Ablations: The full trimodal setting provides the best overall balance, while masked latent prediction gives the strongest overall results among tested objectives.Cross-modal prediction, state alignment, delay modeling, and phase supervision contribute complementary benefits rather than one auxiliary term explaining all improvements.

5 Conclusion

CardioState-JEPA moves cardiac representation learning beyond sensor-specific pretraining by learning a shared latent physiology across ECG, PPG, and PCG. Its intra-modal latent prediction and delay-aware cross-modal alignment let electrical, hemodynamic, and acoustic signals supervise one another.

  • Conclusion: CardioState-JEPA learns a shared foundation-model representation of latent cardiac physiology from ECG, PPG, and PCG.This addresses cardiac representation learning beyond separate sensor-specific pretraining.
  • Conclusion: The framework combines intra-modal latent prediction with delay-aware cross-modal alignment.Modalities are aligned at the corresponding cardiac time while the model predicts latent cardiac states.
  • Conclusion: Electrical, hemodynamic, and acoustic signals supervise one another through the unified representation.These signals correspond to ECG, PPG, and PCG, respectively.

A Appendix · A.1 Data and Training Configuration

The appendix standardizes data reporting and downstream evaluation settings across experiments. It describes corpus and benchmark splits while holding optimization conditions fixed to attribute differences to frozen-encoder quality.

  • A Appendix: Table 5 summarizes the pretraining and downstream linear-probing settings used throughout the experiments.These settings cover both pretraining corpora and downstream benchmarks.
  • A.1 Data and Training Configuration: For each pretraining corpus, the appendix reports the numbers of training and validation recordings.The corpus-level accounting distinguishes training from validation data.
  • A.1 Data and Training Configuration: For each downstream benchmark, it reports the number of classes and the train, validation, and test splits.Benchmark descriptions therefore include both task dimensionality and data partitioning.
  • A.1 Data and Training Configuration: Every downstream task uses the same optimizer.The optimizer is not varied across downstream tasks.
  • A.1 Data and Training Configuration: Every downstream task uses the same epoch budget, batch size, and learning rate.These training-budget and optimization parameters are held constant across tasks.
  • A.1 Data and Training Configuration: The standardized downstream configuration is intended to make reported differences reflect frozen-encoder quality rather than task-specific settings.The appendix frames this as the rationale for using common downstream training conditions.

A.2 Additional Implementation Details

The two pretraining stages use AdamW with weight decay 0.05 and distinct optimization schedules. Stage I runs 300K steps, followed by 200K steps in Stage II, with cosine-decayed learning rates reaching 1 × 10−6.

  • Optimization settings: Both stages use AdamW with weight decay 0.05; Stage I trains 300K steps at batch size 196, while Stage II trains 200K steps at batch size 64.Stage I peaks at 1.2 × 10−4 and Stage II at 6 × 10−5; both decay to 1 × 10−6.

A.3 Training Procedure

CardioState-JEPA uses a two-stage training curriculum: it first learns modality-specific structure from abundant unimodal data, then aligns modalities with scarce paired data while preserving unimodal representations.

  • Stage I learns each modality’s structure from abundant unimodal data through masked latent prediction.
  • Stage II warm-starts from Stage I and uses scarce paired data for delay-aware cross-modal prediction.
  • Stage II continues sampling unimodal data to prevent per-modality codes from drifting.

A.4 Input Length Ablation

The input length ablation evaluates whether CardioState-JEPA remains stable across different ECG window durations, addressing deployment sensitivity to segment length. Table 6 compares four ECG task groups at 2.5-, 5-, and 10-second windows using 10% of training labels.

  • A.4 Input Length Ablation: The ablation tests whether sensitivity to exact input duration could make CardioState-JEPA harder to deploy in practice.The study frames stability across window lengths as a deployment consideration.
  • A.4 Input Length Ablation: Table 6 evaluates four representative ECG task groups at 2.5, 5, and 10 seconds with 10% of the training labels.Each task group is evaluated at all three window lengths.
  • A.4 Input Length Ablation: For each task group, the table reports the score at every window length plus the mean and standard deviation across the three settings.The supplied passage introduces these summary statistics but does not provide their numerical values.

A.5 Sensitivity to Loss Weights … A.10 Limitations

CardioState-JEPA is robust to loss-weight perturbations and learns modality-aware, physiologically aligned representations that improve task-relevant structure across sensing modalities. Its main limitations are scarce synchronized data and comparatively limited PCG pretraining data.

  • A.5 Sensitivity to Loss Weights: The default loss weighting gives the best overall balance, with PPG classification 80.4, PPG regression error 9.1, and ECG average 90.9.No perturbation improves more than one metric at a time, while the ECG average changes by less than one point under substantial weight variation.
  • A.6 Representative Signal Examples: Representative signals show that unimodal data capture shared rhythm and timing despite modality-specific morphology, while scarce synchronized recordings motivate learned delay alignment.Held-out downstream segments differ in morphology, sampling rate, and noise, yet describe the same underlying cardiac process used for cross-modal transfer.
  • A.7 Attention Maps on Cardiac Signals: A single shared encoder attends to modality-specific cardiac events, including ECG depolarization complexes, PPG pulse landmarks, and localized PCG acoustic bursts.The attention patterns indicate recognition of corresponding cardiac events rather than reliance on one fixed waveform shape.
  • A.8 Delay Alignment Visualization: The learned delay aligner consistently maps ECG R-peaks to the PPG systolic upstroke and PCG first heart sound, reflecting physiological electromechanical and pulse-transit timing.For ECG-to-PPG alignment, the reported median absolute error is 4 ms.
  • A.9 Task-Relevant Structure After Cross-Modal Pretraining: Cross-modal training reduces modality identity in the shared cardiac code, lowering the modality silhouette from 0.121 to −0.006.This demonstrates increased modality invariance in the pooled representation.
  • A.9 Task-Relevant Structure After Cross-Modal Pretraining: PPG class structure becomes more organized after Stage II, with atrial-fibrillation silhouette 0.03 to 0.12 and six-class arrhythmia silhouette 0.01 to 0.09.The class analysis indicates sharper task-relevant structure rather than blurred separability after modalities are aligned.
  • A.10 Limitations: Cross-modal alignment is limited by paired and trimodal corpora that are much smaller than the unimodal corpora, leaving comparatively few clean beats for delay-aligner training.The limited synchronized supervision constrains the data available for learning cross-modal timing.
  • A.10 Limitations: PCG pretraining data is the smallest of the three modalities, increasing reliance on cross-modal transfer for acoustic tasks and potentially limiting acoustic representation quality.This data imbalance is identified as a limitation and motivation for future work.
Loading 2608.12944v1…