PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram

Yeongyeon Na, Minje Park, Yunwon Tae, Sunghoon Joo

arXiv:2402.09450v3eess.SPcs.AIcs.LG

TL;DR

Limited disease labels make general ECG representation learning challenging, and a useful approach must account for spatial and temporal relationships across leads. ST-MEM reconstructs masked ECG patches to learn those relationships; the paper reports improved baseline performance and adaptability across lead combinations.

  • Problem

    Limited expert-provided labels make learning ECG representations for diverse disease-detection applications challenging, and ECG signals contain spatial and temporal information across leads.

    Source: §1 · pp. 1–2

  • Method

    ST-MEM uses masked autoencoder reconstruction with spatio-temporal patchifying and lead-indicating modules to learn general ECG representations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    ST-MEM outperforms other baseline methods in linear evaluation, with F1 increases of 0.064 to 0.366 on PTB-XL and 0.292 to 0.561 on CPSC2018.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    Quantitative and qualitative analyses indicate that ST-MEM representations reflect spatial lead relationships and temporal patterns, with evaluations also covering low-resource and reduced-lead settings.

    Source: §5.2 · p. 7, §5.3 · p. 7, §7 · p. 9

Abstract

from arXiv · show

Electrocardiograms (ECG) are widely employed as a diagnostic tool for monitoring electrical signals originating from a heart. Recent machine learning research efforts have focused on the application of screening various diseases using ECG signals. However, adapting to the application of screening disease is challenging in that labeled ECG data are limited. Achieving general representation through self-supervised learning (SSL) is a well-known approach to overcome the scarcity of labeled data; however, a naive application of SSL to ECG data, without considering the spatial-temporal relationships inherent in ECG signals, may yield suboptimal results. In this paper, we introduce ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling), designed to learn spatio-temporal features by reconstructing masked 12-lead ECG data. ST-MEM outperforms other SSL baseline methods in various experimental settings for arrhythmia classification tasks. Moreover, we demonstrate that ST-MEM is adaptable to various lead combinations. Through quantitative and qualitative analysis, we show a spatio-temporal relationship within ECG data. Our code is available at https://github.com/bakqui/ST-MEM.

1 INTRODUCTION

ECG representation learning is difficult because disease labels are limited, while ECG signals carry both spatial and temporal information. ST-MEM uses masked reconstruction to learn those relationships and is reported to perform well across arrhythmia and reduced-lead settings.

1 INTRODUCTION

  • Limited availability of expert-provided disease labels makes learning ECG representations for diverse diagnostic applications challenging.The paper presents self-supervised learning as an approach to address the limited labeled-data resource.Source: §1 · p. 1
  • ECG representation learning needs to account for spatial and temporal information across leads, including cases where only a subset of the standard 12 leads is acquired.The paper describes standard 12-lead ECG as 12 views and notes that reduced-lead ECG is also acquired.Source: §1 · p. 2
  • ST-MEM applies spatio-temporal patchifying and lead-indicating modules within an MAE framework to reconstruct masked ECG data and learn general representations.The paper reports that quantitative and qualitative analyses indicate spatial and temporal features were learned.Source: §1 · p. 2, Figure 2 · p. 2
  • ST-MEM is reported to perform comparably to other contrastive and generative methods and to excel in arrhythmia fine-tuning and linear evaluation, including low-resource and reduced-lead settings.The contribution summary describes these results across scarce-label and reduced-lead scenarios.Source: §1 · p. 2

2 BACKGROUNDS: ECG

A standard 12-lead ECG records electrical signals from different locations, providing spatial and temporal information about the heart. The paper also considers reduced-lead sets, including limb and single-lead ECGs.

2 BACKGROUNDS: ECG

  • A standard 12-lead ECG is described as a multivariate time series whose leads measure electrical signals at different heart locations.The 12 leads include limb and precordial leads.Source: §2 · pp. 2–3
  • ECG waveforms track electrical activity over time, while multiple leads provide distinct spatial views, including frontal-plane limb leads and horizontal-plane precordial leads.The limb leads are I, II, III, aVR, aVL, and aVF; the precordial leads are V1 to V6.Source: §2 · p. 3
  • Reduced lead sets, such as limb leads or a single lead, are often preferred for mobile-device ECG measurements.The paper evaluates its representation in reduced-lead settings as well as the 12-lead setting.Source: §2 · p. 3

3 METHOD

ST-MEM pre-trains an encoder to reconstruct randomly masked spatio-temporal ECG patches, using lead-indicating components and a shared decoder that reconstructs each lead independently. For downstream classification, the encoder is fine-tuned with a classifier head and can be adapted to inputs with fewer leads.

3.1 SELF-SUPERVISED PRE-TRAINING WITH SPATIO-TEMPORAL MASKED AUTO-ENCODER

  • Spatio-temporal patch embeddings let the encoder attend across temporal patches from different leads, while training reconstructs randomly masked raw signal patches.The encoder receives unmasked embeddings; a shared mask embedding supports reconstruction of masked patches.Source: §3.1 · pp. 3–4
  • Because cross-lead unmasked embeddings can make reconstruction too easy, a shared decoder reconstructs each lead independently without using other leads' embeddings.The paper presents this restriction as adding task complexity to motivate the encoder to learn spatio-temporal representations.Source: §3.1 · p. 4
  • Lead-specific embeddings and shared SEP embeddings help distinguish patch sequences by lead, supporting learning of relationships between leads.Each patch receives an embedding for its originating lead, and SEP embeddings are inserted before and after each lead's patch sequence.Source: §3.1 · p. 4

3.2 DOWNSTREAM FINE-TUNING

  • For downstream classification, the pre-trained encoder is paired with a linear classifier head and fine-tuned using ECG inputs and class labels.The decoder is discarded after pre-training.Source: §3.2 · p. 4
  • The model's input patch shape is unaffected by the number of leads, enabling fine-tuning on fewer-lead ECG data after pre-training on 12-lead data.The paper attributes this adaptability to the lead-indicating module's learned lead characteristics.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL SETTINGS

The experiments use three datasets for pre-training and three for downstream evaluation, with comparisons against supervised, contrastive, and generative baselines. Evaluation includes linear evaluation and fine-tuning on unseen 12-lead ECG data.

4.1 DATASETS

  • Three downstream datasets cover 12-lead PTB-XL and CPSC2018 and single-lead PhysioNet2017, while Chapman, Ningbo, and CODE-15 supply pre-training data.All ECGs were resampled to 250 Hz; the merged pre-training set contains 188,480 recordings.Source: §4.1 · p. 5

4.2 BASELINES

  • The comparisons use a ViT-B backbone with supervised initialization, contrastive MoCo v3 and CMSC, and generative MTAE and MLAE baselines.MTAE reconstructs temporal patches, while MLAE reconstructs spatial patches; MTAE+RLM adds random lead masking.Source: §4.2 · pp. 5–6

4 EXPERIMENTAL SETTINGS

  • Linear evaluation and fine-tuning assess arrhythmia and myocardial infarction classification on unseen 12-lead ECG data.The section introduces the datasets and baselines used for these experiments.Source: §4 · p. 5, Table 1 · p. 6

5 EXPERIMENTS AND RESULTS

Experiments assess ST-MEM on unseen-data classification, low-resource training, reduced-lead inputs, and analyses of its learned representations. The results include comparisons with SSL baselines and qualitative evidence of spatial and temporal structure.

5.1 EXPERIMENTAL RESULTS

  • F1 score increases by 0.064 to 0.366 on PTB-XL and by 0.292 to 0.561 on CPSC2018, while ST-MEM consistently outperforms other methods in linear evaluation and fine-tuning.The evaluations use unseen downstream datasets; the passage also reports consistent gains for task adaptation.Source: §5.1 · p. 6

5.2 EFFECTIVENESS OF GENERAL ECG REPRESENTATION IN A LOW-RESOURCE SETTING

  • ST-MEM surpasses other representation-learning methods with 1% and 5% of training data, and 5% of CPSC2018 data yields performance similar to the 100%-data result.The experiments sample 1% and 5% of each dataset for training and validation, with the same test data.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 PERFORMANCE IN REDUCED LEAD SETTINGS

  • ST-MEM is reported as robust across 1-, 6-, and 12-lead combinations, including on PhysioNet2017.The comparison includes RLM and MLAE; Table 3 reports AUROC and marks significant differences against other baselines at p < 0.05.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 ANALYSIS OF GENERAL ECG REPRESENTATION LEARNED FROM ST-MEM

  • ST-MEM representations cluster limb leads I and II separately from precordial leads V2 and V3, and its clustering accuracy is high compared with MTAE and MoCo v3.Table 4 evaluates whether GMM clusters group Chapman representations into limb and precordial leads.Source: §5.4 · pp. 7–8, Table 4 · p. 8

5.5 ABLATION STUDY

  • On CPSC2018, linear-evaluation results increase monotonically when lead-wise shared encoding and lead-indication modules are added after spatio-temporal patchifying alone.The tested indications include SEP and lead embeddings; patchifying without lead type gives the lowest performance.Source: §5.5 · p. 8, Table 5 · p. 8

5.6 INTERPRETATION OF ST-MEM BY ANALYZING SELF-ATTENTIONS

  • Attention for a lead III query is generally high across limb leads, while temporally similar signal shapes also receive high attention.The attention map averages scores across encoder layers and heads.Source: §5.6 · p. 8

6 RELATED WORKS

The related work surveys contrastive and generative SSL for ECG representation learning, alongside work on reduced-lead inputs. It highlights prior spatio-temporal approaches without lead-indicating modules as a challenge for learning spatial relationships between leads.

RELATED WORKS

  • Contrastive ECG SSL uses augmented views to preserve semantic information, while other approaches use temporal or spatial invariance within a single ECG.Source: §6 · p. 9
  • MaeFE applies masked autoencoding to temporal or spatial ECG patches, and prior spatio-temporal patchifying work lacks lead-indicating modules for learning relationships between leads.Source: §6 · p. 9
  • Reduced-lead approaches include random lead masking for robust representations and knowledge distillation from a 12-lead teacher to a single-lead student (Qin et al., 2023).Source: §6 · p. 9

7 CONCLUSION

ST-MEM representations show spatial and temporal relationships in ECGs, and the paper evaluates general ECG representations across reduced-lead and low-resource settings.

7 CONCLUSION

  • ST-MEM representations reflect the distinction between limb leads in the frontal plane and precordial leads in the horizontal plane.The paper also reports observing temporal relationships through an attention map.Source: §7 · p. 9
  • The authors evaluate general ECG representations in reduced-lead and low-resource settings.They also describe evaluating spatial relationships and temporal attention maps.Source: §7 · p. 9
  • Future work will explore applying ST-MEM to general multivariate time-series datasets and alternative transfer-learning approaches for ECG tasks.The authors mention partially fine-tuning the pretrained model as one possible transfer approach.Source: §7 · p. 9

8 REPRODUCIBILITY STATEMENT

The statement provides code, pretrained weights, training details, and dataset access information to support reproducibility.

8 REPRODUCIBILITY STATEMENT

  • The code includes ST-MEM and baseline implementations, pretrained parameters, and training, hyperparameter, and hardware details.Only preprocessed PTB-XL is supplied directly; links provide access to additional datasets.Source: §8 · p. 10

8.1 CODES

  • The code and usage instructions are available in the project repository.The authors direct readers to the README.md file.Source: §8.1 · p. 10

8.2 HYPERPARAMETERS

  • Appendix A.1 contains pretraining, fine-tuning, and linear-evaluation details, including epochs, batch size, and optimizer.Source: §8.2 · p. 10

8.3 DATASETS

  • The paper identifies dataset sizes and recording characteristics in Section 4.1 and reports that ECG data are resampled to 250 Hz.Preprocessed PTB-XL train, validation, and test sets are downloadable through the code repository; raw datasets are linked separately.Source: §8.3 · p. 10

A DETAILS IN EXPERIMENTAL SETTINGS

A.1 IMPLEMENTATION DETAILS

The appendix details model implementation, ECG preprocessing and evaluation, then reports comparisons and additional experiments across benchmark, label, and lead settings.

A.1 IMPLEMENTATION DETAILS

  • ST-MEM uses spatio-temporal patchifying, learned mask, lead, and SEP embeddings, and masked-patch reconstruction with mean squared error.The implementation fixes the backbone as ViT-B and uses a 75% random patch mask for generative pretraining.Source: §A.1 · p. 14

A.2 DATA PREPROCESSING

  • ECGs are standardized through resampling, filtering, normalization, 10-second cropping, label filtering, and 70-10-20 train-validation-test splits.Multi-label ECGs are excluded for the multi-class setting; preprocessing procedures are applied across baselines.Source: §A.2 · pp. 14–15

A.3 EVALUATION ON DOWNSTREAM TASKS

  • Downstream evaluation uses macro F1 and mean one-vs-rest AUROC alongside accuracy, scoring each cropped segment individually.The tasks include arrhythmia and myocardial-infarction classification.Source: §A.3 · p. 15

B.1 COMPARISON OF EXISTING ECG PRE-TRAINED MODELS WITH ST-MEM

  • ST-MEM demonstrates performance comparable to CPC, although CPC was pretrained on PTB-XL and CPSC2018 while ST-MEM and the other baselines treated them as unseen datasets.The comparison includes low-resource settings; the table caption specifies AUROC scores and 12-lead ECG inputs.Source: §B.1 · p. 18, Table 13 · p. 18

B.2 SELECTING AUGMENTATION FOR PRE-TRAINING MOCO V3

  • Using all eight MoCo v3 augmentations performed better than four augmentations in every reported setting except PTB-XL fine-tuning.The authors note that flip, shift, sine, and partial-sine augmentations may affect ECG semantics, making augmentation selection challenging.Source: §B.2 · pp. 18–19, Table 14 · p. 19

B.3 EXPERIMENTS OF DIFFERENT PROBLEM SETTINGS FOR DOWNSTREAM DATASET PTB-XL

  • ST-MEM is reported superior to supervised learning and MTAE across diagnostic, form, and rhythm multi-label settings on PTB-XL.The experiment uses 71 PTB-XL labels grouped into those three categories; the table caption identifies average AUROC results.Source: §B.3 · p. 19, Table 17 · p. 20

B.4 ADDITIONAL SINGLE LEAD EXPERIMENTS ON VARIOUS LEAD TYPES

  • ST-MEM outperforms MTAE+RLM and MLAE in single-lead experiments, with a non-trivial margin on leads II and aVL.The experiments are reported for single-lead PTB-XL ECGs.Source: §B.4 · p. 21, Table 18 · p. 21

B.5 RESULTS OF SINGLE LEAD ECGS IN A LOW-RESOURCE SETTING

In low-resource single-lead evaluation, ST-MEM outperforms the supervised baseline; representation analysis also shows lead-related clustering.

B.5 RESULTS OF SINGLE LEAD ECGS IN A LOW-RESOURCE SETTING

  • 0.786 AUROC was achieved by ST-MEM using only 5% of lead II ECGs, and ST-MEM consistently surpassed the supervised baseline in the reported low-resource settings.The table caption states that results average three samplings with a constant test set.Source: §B.5 · p. 21, Table 19 · p. 21

B.6 ADDITIONAL ANALYSIS OF ECG REPRESENTATION LEARNED FROM ST-MEM

  • With lead-indication modules, ST-MEM representations clustered limb and precordial leads, whereas the model without those modules struggled to cluster them.The analysis used t-SNE on sampled 12-lead ECGs from regular and irregular rhythms in the Chapman dataset.Source: §B.6 · p. 21

B.7 ADDITIONAL ABLATION STUDY

The ablations identify four decoder blocks and a 75% masking ratio as the best tested settings, while using both lead-indicating modules yields the highest AUROC. Attention maps show high weights on similarly shaped temporal patches and neighboring spatial patches across rhythm types.

B.7 ADDITIONAL ABLATION STUDY

  • A decoder depth of 4 blocks achieves the highest PTB-XL fine-tuning performance; both a single block and excessive depth are undesirable.Table 20 presents the decoder-depth ablation.Source: §B.7 · p. 22, Table 20 · p. 22
  • A 75% masking ratio gives ST-MEM its highest PTB-XL fine-tuning performance among the tested ratios.Figure 7 shows performance changes across masking ratios.Source: §B.7 · p. 22, Figure 7 · p. 22
  • Using both lead embeddings and SEP embeddings yields the highest PTB-XL linear-evaluation AUROC; Figure 6 contrasts ST-MEM representations with a model lacking all lead-indicating modules.Table 21 reports the module ablation; Figure 6 plots representations for regular and irregular rhythms.Source: §B.7 · p. 22, Figure 6 · p. 22, Table 21 · p. 23

B.8 ADDITIONAL ANALYSIS OF SELF-ATTENTION MAPS OF ST-MEM

  • Across rhythm types, ST-MEM assigns high attention weights to similarly shaped temporal patches and neighboring spatial patches.Figure 8 shows attention maps for regular and irregular rhythm ECGs.Source: §B.8 · p. 23, Figure 8 · p. 23

C FUTURE WORK: APPLYING ST-MEM TO OTHER MULTIVARIATE TIME-SERIES DATA

The paper tests ST-MEM on multivariate time-series data by adapting its lead embedding for general time-series features. On the Human Activity Recognition dataset, the fine-tuned model shows comparable classification performance.

C FUTURE WORK: APPLYING ST-MEM TO OTHER MULTIVARIATE TIME-SERIES DATA

  • The proposed adaptation transforms the lead embedding into a general time-series feature embedding for the HAR experiment.The model processes 9 x 128 inputs with patch size 32 and randomly masks 60% of patches during pre-training.Source: C FUTURE WORK: APPLYING ST-MEM TO… · p. 24
  • The evaluation uses the HAR dataset, which contains nine sensor features from 30 people performing six activities.Each sample covers 2.56 seconds at 50 Hz, giving 128 readings; the activities include walking, stairs, sitting, standing, and lying down.Source: C FUTURE WORK: APPLYING ST-MEM TO… · p. 24
  • ST-MEM shows comparable performance on fine-tuned human activity classification, supporting its extension to the general multivariate time-series domain.Table 22 reports the classification results.Source: C FUTURE WORK: APPLYING ST-MEM TO… · p. 24, Table 22 · p. 24
Loading 2402.09450v3…