PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised representation learning from 12-lead ECG data

Temesgen Mehari, Nils Strodthoff

arXiv:2103.12676v2eess.SPcs.LGstat.ML

TL;DR

Label scarcity and expensive, difficult clinical ground truth motivate self-supervised learning for 12-lead ECG data. The paper comprehensively assesses adapted self-supervised methods and finds near-supervised linear evaluation, improved finetuned performance, label efficiency, and robustness.

  • Problem

    Expensive labeling and difficult clinical ground truth leave ECG datasets label-limited despite abundant unlabeled data.

  • Method

    The paper presents the first comprehensive assessment of self-supervised representation learning for 12-lead ECG data, adapting instance-discrimination and latent-forecasting methods.

  • Results

    Self-supervised representations via CPC fall only 0.5% behind supervised performance in linear evaluation and improve finetuned performance by 1.0% over supervised training.

  • Takeaways & Limitations

    Finetuned self-supervised models reach comparable performance with roughly 50-60% of the samples and show improved label efficiency and robustness against physiological noise.

  • Takeaways & Limitations

    CPC has a sequential LSTM structure that leads to slow training, requiring approximately 6 days for 200 pretraining epochs on one Tesla V100 GPU.

Abstract

from arXiv · show

Clinical 12-lead electrocardiography (ECG) is one of the most widely encountered kinds of biosignals. Despite the increased availability of public ECG datasets, label scarcity remains a central challenge in the field. Self-supervised learning represents a promising way to alleviate this issue. In this work, we put forward the first comprehensive assessment of self-supervised representation learning from clinical 12-lead ECG data. To this end, we adapt state-of-the-art self-supervised methods based on instance discrimination and latent forecasting to the ECG domain. In a first step, we learn contrastive representations and evaluate their quality based on linear evaluation performance on a recently established, comprehensive, clinical ECG classification task. In a second step, we analyze the impact of self-supervised pretraining on finetuned ECG classifiers as compared to purely supervised performance. For the best-performing method, an adaptation of contrastive predictive coding, we find a linear evaluation performance only 0.5% below supervised performance. For the finetuned models, we find improvements in downstream performance of roughly 1% compared to supervised performance, label efficiency, as well as robustness against physiological noise. This work clearly establishes the feasibility of extracting discriminative representations from ECG data via self-supervised learning and the numerous advantages when finetuning such representations on downstream tasks as compared to purely supervised training. As first comprehensive assessment of its kind in the ECG domain carried out exclusively on publicly available datasets, we hope to establish a first step towards reproducible progress in the rapidly evolving field of representation learning for biosignals.

1 INTRODUCTION

The paper addresses scarce and costly clinical ECG labels by evaluating self-supervised representation learning for 12-lead ECG data. It compares adapted methods and examines representation quality, downstream performance, label efficiency, and robustness.

  • 12-lead ECG is clinically widespread, but interpretation remains mostly manual and can challenge cardiologists.
  • ECG self-supervised learning requires domain-specific adaptations because ECG is multivariate time-series data rather than one-dimensional imagery.
  • The study presents the first comprehensive assessment of self-supervised representation learning for 12-lead ECG data using publicly available datasets.
  • It directly compares adapted SimCLR, BYOL, SwAV, and CPC methods and evaluates finetuned classifiers against purely supervised training.

2 MATERIALS AND METHODS

The methods adapt instance-discrimination and latent-forecasting approaches to clinical ECG data, using established ECG architectures and public datasets. Evaluation covers representation quality, downstream classification, and dataset organization.

  • 2.2 Self-supervised representation learning for ECG data: The study adapts instance-discrimination methods and latent-forecasting methods from computer vision and speech to ECG representation learning.
  • 2.2.1 Instance discrimination (SimCLR/BYOL/SwAV): SimCLR attracts views from the same instance and repels other batch instances, while SwAV contrasts cluster assignments.
  • 2.1 Model architecture: The experiments use xresnet1d-family convolutional networks, including xresnet1d50, for ECG representation learning and classification.
  • 2.2.3 Self-supervised representation learning for physiological time series data: CPC encodes past sequence context and predicts future latent representations using a noise-contrastive objective.
  • 2.3 ECG datasets: Pretraining uses three public datasets totaling 54,566 records, while evaluation uses PTB-XL with 71 multi-label diagnostic, form, and rhythm statements.
  • 2.4 Training and evaluation: The study evaluates macro AUC for the multi-label classification task and uses distinct pretraining and finetuning sequence lengths.

3 EXPERIMENTS

Experiments show that adapted self-supervised methods learn useful ECG representations and can improve downstream performance, label efficiency, and noise robustness, although comparisons are affected by architecture and dataset factors.

  • Self-supervised pretraining: SimCLR achieved the best linear evaluation performance among the instance-based methods, while BYOL produced superior downstream performance after finetuning.The comparison used predetermined ECG transformations after an initial transformation study.
  • Self-supervised pretraining: 0.5% was the CPC model’s linear-evaluation performance gap relative to the same architecture trained supervised, while exceeding the previous supervised result.The evaluation used macro AUC on the PTB-XL task.
  • Downstream performance: 1.0% for CPC, 0.2% for SimCLR, and 0.5% for BYOL were the finetuning improvements over corresponding supervised results.The CPC model reached approximately 0.931 after adjusting batch-normalization statistics and the classification head.
  • Downstream performance: Form and rhythm statements showed the largest average gains from self-supervised pretraining, while lower-performing ECG statements tended to benefit more.The analysis separated diagnostic superclasses, rhythm statements, and form statements.
  • Data efficiency: Pretraining improved label efficiency: the pretrained model using two training folds approximately reached supervised performance obtained with four folds.For BYOL and SimCLR, the effect was present but less pronounced, and pretrained models showed more stable performance across runs.
  • Robustness: CPC pretraining considerably improved noise robustness for the 4FC+2LSTM+2FC model, while domain-specific noise transformations strengthened robustness despite a slight noiseless-performance cost.The highest tested noise levels were described as rarely encountered in real-world scenarios.

4 SUMMARY AND CONCLUSIONS

The paper presents a comprehensive assessment showing that self-supervised learning can produce useful representations for clinical 12-lead ECG data. Adapted methods improve finetuned performance, label efficiency, and robustness, supporting further reproducible work using public datasets.

  • The study establishes a first comprehensive assessment of self-supervised representation learning for 12-lead ECG data using publicly available datasets.
  • Self-supervised representations via CPC fall only 0.5% behind supervised performance during linear evaluation.
  • Finetuning self-supervised representations improves performance by 1.0% compared to supervised training.
  • Pretrained models reach the supervised model’s performance using roughly 50-60% of the samples.
  • Most pretrained models show increased robustness against physiological noise, especially with domain-specific noise transformations.

A.1 Artificial transformations

The study uses artificial signal transformations to generate altered ECG views for self-supervised learning methods. These include noise addition, blurring, channel scaling, cropping, masking, and time warping.

  • Gaussian noise adds zero-mean noise to all channels with σ = 0.01 mV.
  • Gaussian blur applies a one-dimensional kernel with entries (0.1, 0.2, 0.4, 0.2, 0.1) to the ECG signal.
  • Channel resize multiplies each channel by a random factor whose expected value is 1, using b = 3.The transformation is analogous to color transformations in computer vision.
  • Random resized crop selects a contiguous signal segment covering 50%-100% of the original and rescales it.
  • Time out sets a random contiguous segment to zero, covering up to 50% of the signal.
  • Dynamic time warp stretches and squeezes random contiguous segments using 3 warps with radius 10 timesteps.

A.2 ECG-specific physiological noise transformations

The section models four ECG-specific physiological noise types and combines them with the original signal to create noisy ECG examples.

  • Baseline wander: Baseline wander is modeled as a superposition of sinusoidal components representing a low-frequency ECG artifact.The artifact can arise from respiration, electrically charged electrodes, or patient movement.
  • Powerline noise: Powerline noise represents 50 Hz interference and its higher harmonics using three sinusoidal components.Its lead-wise coefficient is sampled uniformly over [−1, 1], while the overall amplitude is bounded by Cmax,pln.
  • Electromyographic noise: Electromyographic noise is modeled as Gaussian noise to represent high-frequency noise typically caused by muscle contractions.The Gaussian variance is controlled by Cmax,emn.
  • Baseline shift: Baseline shift is modeled by multiplying a sampled stepwise function by a random amplitude and lead-specific coefficient.The stepwise function represents baseline changes associated with electrode-skin impedance changes and electrode motion.
  • Superposition: All four noise types are superimposed and added to the original signal s(t), with their strengths adjusted through the corresponding maximum-amplitude parameters.The construction combines baseline wander, powerline, electromyographic, and baseline-shift noise.

A.3 Parameter values used during pretraining and evaluation

During pretraining, physiological transformations use fixed maximum amplitudes for four noise types; robustness evaluation varies noise levels on noisy validation sets.

  • Pretraining parameters: Pretraining uses Cmax,blw = 0.1, Cmax,pln = 0.2, Cmax,emn = 0.5, and Cmax,bls = 1 for physiological transformations.These parameters control baseline-wander, powerline, electromyographic, and baseline-shift noise strengths, respectively.
  • Evaluation parameters: Robustness testing uses noisy validation sets with different noise levels described in Table 3.Table 3 maps noise levels to parameters of the physiological transformations.
  • Transformation types: The physiological transformations comprise baseline wander, powerline noise, electromyographic noise, and baseline shift.These four transformations are shown in Figure A1.

TRANSFORMATIONS

The study compares augmentation choices and self-supervised frameworks using linear evaluation and finetuning, showing that transformation and framework rankings can differ by evaluation mode.

  • Augmentation selection: The transformation grid search uses six augmentations combining computer-vision-inspired and time-series transformations to create semantically equivalent views.The authors identify suitable transformations as crucial to SimCLR effectiveness.
  • Framework comparison: Table 4 compares SimCLR, BYOL, and SwAV with artificial or physiological-noise transformations using linear-evaluation and finetuning performance.Scores are reported after 2000 epochs of pretraining on the All dataset, with mean and standard deviation over 10 finetuning runs.
  • Transformation comparison: Artificial transformations produce considerably stronger models than physiological-noise transformations in both evaluation modes.This comparison applies across the frameworks evaluated in Table 4.
  • Linear evaluation: Linear-evaluation performance is measured as macro AUC on the PTB-XL validation set, averaging three linear-evaluation runs.Figure B1 reports results for a xresnet1d50 model pretrained for 500 epochs with one or two augmentations.
  • Evaluation-mode comparison: SimCLR achieves the best linear-evaluation performance, whereas BYOL yields superior downstream performance after finetuning.Thus, linear-evaluation ranking is not necessarily a perfect proxy for downstream-performance ranking.

SPEECH

The section studies architectural and training modifications for CPC and compares ECG forecasting settings with raw-audio CPC, emphasizing model capacity and forecasting scale.

  • Ablation design: The ablation study varies one CPC architecture or training-procedure aspect at a time while holding the others fixed.It evaluates effects on both linear evaluation and finetuning performance.
  • Architecture modifications: The MLP and classification-head modifications have small but consistent positive effects on linear evaluation and downstream performance.These effects are summarized in Table 5.
  • Model capacity: Reducing LSTM hidden units from 512 to 256 or 128 causes more severe performance degradation, supporting the importance of model capacity.The comparison concerns CPC models with otherwise fixed settings.
  • Finetuning: The largest performance gain comes from a two-step finetuning approach.The supplied passage introduces this result but does not specify the two steps.
  • Comparison with speech: ECG CPC predicts 12 future steps over 0.12 s, comparable in order of magnitude to raw-audio CPC’s 0.192 s look-ahead.The ECG encoder uses no downsampling, whereas the raw-audio encoder downsamples by a factor of 160.
  • Comparison with speech: Using strided convolutions for ECG encoding leads to considerably worse performance, including for supervised models.The ECG setup therefore uses a fully connected encoder without downsampling.

SUPERVISED PRETRAINING

Self-supervised pretraining improves ECG classification, with the largest superclass-level gains appearing in form and rhythm labels and stronger gains where supervised performance is lower. Component ablations assess how architectural and procedural choices affect CPC pretraining and finetuning.

  • Form and rhythm label categories show the largest classification improvements after self-supervised training and finetuning.
  • Pretraining improvements are negatively correlated with supervised performance.Figure D2 presents improvement as a function of supervised performance and includes a linear regression of the data points.
  • Table 5 evaluates architectural and procedural CPC components by comparing performance against a baseline that omits each specified component.

APPENDIX E LABEL DISTRIBUTION OF CINC AND PTB-XL

PTB-XL contains 21,837 samples with 71 finest-level labels, while CinC2020 combines five datasets including PTB-XL under a unified SNOMED-based label scheme. After removing PTB-XL records, the remaining CinC2020 data contain 21,256 signals and differ considerably in label distributions for frequent pathologies.

  • PTB-XL contains 21837 samples annotated with 71 labels at the finest level.
  • CinC2020 was created by compiling five datasets, including PTB-XL, with original labels mapped to SNOMED codes.The mapping introduces ambiguities because the original labels used a different label set.
  • Removing PTB-XL leaves 21256 signals from the four remaining CinC2020 datasets.
  • Label distributions differ considerably between PTB-XL and CinC2020 without PTB-XL, especially for more frequent pathologies.Most pathologies have few samples in both datasets, whereas frequently occurring pathologies show large differences.
Loading 2103.12676v2…