Source-linked AI summary
Generative Pre-Trained Transformer for Cardiac Abnormality Detection
Pierre Louis Gaudilliere, Halla Sigurthorsdottir, Clémentine Aguet, Jérôme Van Zaen, Mathieu Lemay, Ricard Delgado-Gonzalo
TL;DR
The paper explores how transformers can be applied to ECG signals by treating heartbeats as words in a sequence. Its best submitted entry scored 0.12, 0.07, 0.10, 0.10, and 0.07 across the 12-, 6-, 4-, 3-, and 2-lead test sets, respectively.
Problem
The paper explores whether ECGs, viewed as sequences of repeated heartbeats, can be represented in a way that applies transformer attention to periodic time series.
Source: §1 · p. 1Method
The approach represents each heartbeat as a word embedding and an ECG as a sequence of such words, using a transformer encoder for pre-training and classification.
Source: §1 · p. 1, §3.4 · p. 3, §3.5 · p. 3Results
0.12, 0.07, 0.10, 0.10, and 0.07 were the best-entry scores on the 12-lead, 6-lead, 4-lead, 3-lead, and 2-lead test sets, respectively.
Source: §4 · p. 4Takeaways & Limitations
Neither linear probing nor transfer learning of the pre-trained model improved the score over supervised training alone.
Source: §4 · p. 3The approach depends heavily on R-peak detectors, and incorrect R-peak detection could be a major source of error.
Source: §5 · p. 4
Abstract
from arXiv · showhide
ECG heartbeat classification plays a vital role in diagnosis of cardiac arrhythmia. The goal of the Physionet/CinC 2021 challenge was to accurately classify clinical diagnosis based on 12, 6, 4, 3 or 2-lead ECG recordings in order to aid doctors in the diagnoses of different heart conditions. Transformers have had great success in the field of natural language processing in the past years. Our team, CinCSEM, proposes to draw the parallel between text and periodic time series signals by viewing the repeated period as words and the whole signal as a sequence of such words. In this way, the attention mechanisms of the transformers can be applied to periodic time series signals. In our implementation, we follow the Transformer Encoder architecture, which combines several encoder layers followed by a dense layer with linear or sigmoid activation for generative pre-training or classification, respectively. The use case presented here is multi-label classification of heartbeat abnormalities of ECG recordings shared by the challenge. Our best entry, not exceeding the challenge's hardware limitations, achieved a score of 0.12, 0.07, 0.10, 0.10 and 0.07 on 12-lead, 6-lead, 4-lead, 3-lead and 2-lead test set respectively. Unfortunately, our team was unable to be ranked because of a missing pre-print.
1. Introduction
The paper applies the text–time-series analogy to ECGs by treating heartbeats as words and full signals as sentences. It presents this representation as a contribution and outlines the paper’s pipeline, results, and discussion.
1. Introduction
- ECG signals and text are framed as sequences of repeated patterns, with heartbeats corresponding to words.The passage describes text as repeated patterns with semantic meaning and ECGs as repeated heartbeats.Source: §1 · p. 1
- The paper represents each heartbeat as a “word embedding” and an ECG signal as a sequence of such words.This approach draws a conceptual parallel between periodic time-series signals and text.Source: §1 · p. 1
- The paper is organized around background, a classification pipeline and model architecture, results, and a critical review with future research directions.Source: §1 · p. 1
2. Previous Work
Earlier ECG transformer work addressed heartbeat classification or recording-level multi-label classification, but did not treat heartbeats as word embeddings in a sequence of periodic words.
2. Previous Work
- Earlier ECG studies used a CNN-transformer with handcrafted features for CinC2020 and transformers for MIT-BIH heartbeat classification, but did not model heartbeats as words.The cited works include Yan et al. and the CinC2020 approach.Source: §2 · p. 1
3. Methods
The paper represents ECG recordings as sequences of heartbeat embeddings and applies a Transformer Encoder to pre-train for next-heartbeat prediction or classify 28 arrhythmia labels. Its pipeline covers dataset selection, signal preprocessing, heartbeat embedding, transformer architecture, and training.
3. Methods
- Heartbeat sequences serve as “words” in a Transformer network for multi-label classification of 28 cardiac arrhythmias, with next-heartbeat prediction as the pre-training task.The classifier outputs 28 probabilities using sigmoid activation; the pre-training output layer uses linear activation.Source: §3 · pp. 1–2
3.1. Dataset
- The training set draws on eight worldwide datasets totaling 88,000 ECG recordings, with St Petersburg recordings excluded from the main data split because they are 30 minutes long.The remaining recordings were divided into five stratified folds, with four used for training and one for validation; challenge datasets were included for validation.Source: §3.1 · p. 2
3.2. Signal Pre-Processing
- ECG leads were gain-scaled and high-pass filtered at 0.5 Hz; four R-peak detectors were tested before recordings were resampled to 500 Hz.The detectors were Christov, Hamilton, Two-Average, and Pan Tompkins.Source: §3.2 · p. 2
3.3. Word Embeddings
- Each heartbeat spans one-third of the R–R interval before the R-peak and two-thirds after, then is aligned and padded or truncated to 1,000 samples.The RMS of active leads forms a single ECG signal; the amount of zero padding indicates R–R interval variability.Source: §3.3 · p. 2, Figure 2 · p. 3
3.4. Transformer Model
- Five encoder layers combine masked multi-head attention and feed-forward networks, with dropout and normalization; positional encodings are added to heartbeat embeddings.Attention uses eight heads, and the final fully connected layer produces either a linear pre-training output or sigmoid class probabilities.Source: §3.4 · pp. 2–3
3.5. Training
- Pre-training predicts the next heartbeat using mean squared error, while supervised classification uses class-averaged binary cross-entropy and Adam with 4,000 warmup steps.A class is positive above a 0.5 probability threshold; models were trained for 50 epochs.Source: §3.5 · p. 3
4. Results
The Georgia and PTB XL model using the Christov detector was the successful entry, while the best-performing four-source model was not submitted successfully. Pre-training did not improve on supervised-only training, and full-database training exceeded hardware limits.
4. Results
- 0.12, 0.07, 0.10, 0.10, and 0.07 were the scores for 12-, 6-, 4-, 3-, and 2-lead test sets, respectively, for the Georgia/PTB XL Christov entry.The best model used the Two-Average detector and Georgia, PTB XL, Chapman, and Ningbo, but did not yield a successful entry.Source: Table 2 · p. 4, §4 · p. 4
- Neither linear probing nor transfer learning improved on supervised-only training, so pre-training was omitted from the final model.Pre-training on Georgia and PTB XL reached a loss of 1.90E−5 after five epochs; St Petersburg data often produced corrupted inputs and poor predictions.Source: §4 · pp. 3–4
- Adding Chapman and Ningbo to Georgia and PTB XL positively impacted performance, but training on the full database exceeded hardware limits.Source: §4 · p. 4
5. Discussion and Conclusions
The authors describe the approach as promising but report that it did not achieve a satisfying score. Neither linear probing nor transfer learning improved on supervised training alone, and the approach depends strongly on R-peak detection.
5. Discussion and Conclusions
- The model did not achieve a satisfying score despite the promising transformer and heartbeat-as-word framing.The authors characterize the outcome as unsatisfying without providing a score in this passage.Source: §5 · p. 4
- Neither linear probing nor transfer learning of the pre-trained model improved the score over supervised training alone.Source: §5 · p. 4
- The approach depends strongly on R-peak detectors, and incorrect detections could be a major source of error.Source: §5 · p. 4
- Although RMS generalizes well, the authors identify CNN embeddings such as Sigurthorsdottir et al. 2020 as an alternative to explore.Source: §5 · p. 4