PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL

Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, Wojciech Samek

arXiv:2004.13701v1cs.LGstat.ML

TL;DR

Automatic ECG analysis lacks sufficiently appropriate datasets and standardized evaluation procedures. This paper benchmarks multiple tasks and classifier families on PTB-XL, extends evaluation to ICBEB2018 and demographic prediction, and examines uncertainty, hidden stratification, and interpretability. Convolutional architectures perform best across tasks, while PTB-XL pretraining is particularly effective in the small-dataset regime.

  • Problem

    Automatic ECG interpretation lacks appropriate publicly accessible training datasets and well-defined evaluation procedures for comparing algorithms.

  • Method

    The paper benchmarks ECG statement, age, gender, and related analyses on PTB-XL, validates results on ICBEB2018, and studies hidden stratification, uncertainty, and interpretability.

  • Results

    Convolutional neural networks, particularly ResNet- and Inception-based architectures, show the strongest performance across tasks.

  • Takeaways & Limitations

    PTB-XL provides a structured benchmarking and pretraining resource, with transfer learning particularly effective in the small-dataset regime.

Abstract

from arXiv · show

Electrocardiography is a very common, non-invasive diagnostic procedure and its interpretation is increasingly supported by automatic interpretation algorithms. The progress in the field of automatic ECG interpretation has up to now been hampered by a lack of appropriate datasets for training as well as a lack of well-defined evaluation procedures to ensure comparability of different algorithms. To alleviate these issues, we put forward first benchmarking results for the recently published, freely accessible PTB-XL dataset, covering a variety of tasks from different ECG statement prediction tasks over age and gender prediction to signal quality assessment. We find that convolutional neural networks, in particular resnet- and inception-based architectures, show the strongest performance across all tasks outperforming feature-based algorithms by a large margin. These results are complemented by deeper insights into the classification algorithm in terms of hidden stratification, model uncertainty and an exploratory interpretability analysis. We also put forward benchmarking results for the ICBEB2018 challenge ECG dataset and discuss prospects of transfer learning using classifiers pretrained on PTB-XL. With this resource, we aim to establish the PTB-XL dataset as a resource for structured benchmarking of ECG analysis algorithms and encourage other researchers in the field to join these efforts.

I. INTRODUCTION

Automatic ECG interpretation is promising but lacks sufficiently open training datasets and standardized evaluation procedures. The paper presents PTB-XL benchmarking and broader analyses addressing performance, uncertainty, interpretability, and transfer learning.

  • Automatic ECG interpretation could support medical personnel in urgent ECG reading and telemedicine monitoring.
  • Deep-learning approaches have reached or surpassed cardiologist-level performance for selected subtasks and can infer age and gender from ECGs.
  • Open ECG datasets are typically small, while larger datasets are often inaccessible to the general public.
  • PTB-XL is positioned as a pretraining resource for generic ECG classifiers, especially in the small-dataset regime.
  • The paper examines interpretability, robustness, and uncertainty alongside quantitative accuracy for medical decision-support systems.

II. MATERIALS & METHODS

The study benchmarks raw multivariate ECG time-series classifiers using PTB-XL and compares convolutional, recurrent, feature-based, and naive approaches. It also releases the evaluation resources and avoids additional signal preprocessing.

  • PTB-XL contains 21,837 clinical 12-lead ECG records from 18,885 patients, annotated with 71 statements across diagnostic, rhythm, and form categories.
  • The benchmark evaluates algorithms operating directly on raw multivariate time-series data.
  • The compared groups include convolutional neural networks, recurrent neural networks, feature-based classifiers, and naive frequency-based baselines.
  • Convolutional models include standard, ResNet-based, and Inception-based architectures, with selected representatives reported for clarity.
  • The deep-learning models use original time series without preprocessing such as baseline-wander removal or filtering.

C. Multi-label classification metrics

The paper evaluates multi-label classification with complementary label-centric and sample-centric metrics. It favors macro-averaging under class imbalance and summarizes threshold-dependent F1 with Fmax.

  • Multi-label metrics are divided into sample-centric and label-centric categories to provide complementary views of classification performance.
  • The evaluation uses soft classifier outputs before thresholding to assess discriminative power separately from threshold optimization.
  • Macro-averaging is preferred because class imbalance could otherwise allow large classes to dominate the score.
  • The sample-centric evaluation follows procedures used in CAFA classification challenges.
  • Fmax is the maximum threshold-dependent F1 score obtained by varying a single threshold.

III. BENCHMARKING RESULTS ON PTB-XL AND ICBEB2018

The experiments provide PTB-XL benchmarks and use ICBEB2018 to validate findings and examine transfer learning. The section also extends evaluation across ECG-statement selections and granularities.

  • The experiments establish initial benchmark results for future reference and illustrate the range of analyses supported by PTB-XL.
  • ECG-statement experiments cover diagnostic, form, and rhythm categories at subclass and superclass levels.
  • The ICBEB2018 experiments independently validate PTB-XL findings and investigate transfer learning.

A. ECG statement prediction on PTB-XL

The study benchmarks ECG-statement classifiers across six PTB-XL experiments, comparing model families under standardized evaluation. Convolutional architectures, especially ResNet and Inception variants, perform strongest, while ensemble gains are generally modest.

  • Experimental scope: Six ECG-statement experiments evaluate diagnostic, form, and rhythm labels at subclass and superclass granularities.The experiments use the evaluation procedure described in Section II-C and report results for all models in Table II.
  • Predictive performance: 0.89, 0.93, and 0.96 are the macro AUCs reached by the best ResNet or Inception models for form, diagnostic, and rhythm categories, respectively.The reported values range from 0.89 for form, through around 0.93 for diagnostic categories, to 0.96 for rhythm.
  • Caveat: The form task has a considerably smaller training set because approximately 12k ECGs lack form annotations.This limits direct difficulty comparisons across prediction tasks.
  • Model comparison: ResNet and Inception architectures perform best across all experiments, while convolutional models generally remain compatible within error bars.Recurrent models are consistently slightly less performant but remain competitive for diagnostic and rhythm statements.
  • Model comparison: Deep-learning models substantially outperform the wavelet-feature baseline, although feature-based results are sensitive to feature selection and preprocessing details.This comparison should therefore be interpreted cautiously because the baseline depends on design choices outside the model family itself.
  • Ensembling: Ensembling usually produces slight performance increases, with the largest improvement occurring in the rhythm category.The best single ResNet or Inception models remain compatible with the ensemble within error bars, so ensembles serve mainly as rough orientation.

B. ECG statement prediction on ICBEB2018 and transfer learning

The paper benchmarks ECG statement classification on ICBEB2018 and examines PTB-XL pretraining across training-set sizes. Convolutional architectures show a consistent advantage, while transfer learning is most promising in the small-data regime.

  • ICBEB2018 classification: ICBEB2018 benchmarking reports sample-centric Fmax, term-centric macro-AUC, and challenge-oriented Fβ=2 and Gβ=2 metrics.The latter metrics use β=2 and inversely label-frequency-weighted samples.
  • ICBEB2018 classification: ICBEB2018 results show uniform performance among convolutional architectures, slightly weaker recurrent models, and a considerable gap over the feature-based baseline.
  • Transfer learning: PTB-XL classifiers were fine-tuned on ICBEB2018 across training sizes ranging from 85 to 5500 samples and compared with models trained from scratch.The experiment used xresnet1d101 and replaced the classification head for ICBEB2018’s label count.
  • Transfer learning: For large ICBEB2018 training sets, PTB-XL pretraining did not improve performance over training from scratch and potentially slightly deteriorated it within compatible error bars.
  • Transfer learning: PTB-XL pretraining improved performance for a single training fold or fractions of it, supporting its use for relatively small ECG datasets.The paper identifies the small-dataset regime as the natural application domain for generic ECG pretraining.

C. Age regression and gender classification

The study uses PTB-XL metadata to evaluate age regression and gender classification across all, healthy, and non-healthy subjects. Healthy-subject evaluation performs best for age, while gender prediction remains competitive with prior results.

  • Age regression: Age regression achieved MAE=6.86 on healthy subjects, compared with MAE=7.38 on non-healthy subjects and MAE=7.16 on all subjects.
  • Age regression: Healthy-subject age prediction was competitive with prior work reporting MAE=6.9 years and R-squared = 0.7 using approximately thirty times more data.
  • Gender classification: Gender prediction reached 84.9% accuracy and AUC=0.92 on all patients, increasing to 89.8% accuracy and AUC=0.96 on healthy patients.
  • Gender classification: The healthy-subpopulation gender results were competitive with literature values of 90.4% accuracy and AUC=0.97.

D. Signal quality assessment

The paper evaluates signal quality using technical-expert artifact annotations encoded as binary targets. It attributes part of the performance deficiency to label noise and frames the resulting models as prescreening tools.

  • Signal quality assessment: Technical experts annotated ECG samples for signal artifacts, with irregular annotations coded as binary NOISE targets.A target is one when any global static noise or local artifact annotation is present.
  • Signal quality assessment: A significant portion of the performance deficiency is attributed to label noise from missing artifact annotations or misleading annotations on normal signals.
  • Signal quality assessment: The paper suggests incorporating full report strings rather than binary labels in a more thorough analysis.
  • Signal quality assessment: Models trained for signal quality assessment can still be used as a prescreening procedure for ECG quality.

IV. DEEPER INSIGHTS FROM CLASSIFICATION MODELS

The deeper analysis examines hierarchical diagnostic labels, hidden stratification, model uncertainty, and possible links between co-occurring ECG statements and clinical investigation.

  • A. Hierarchical organization of diagnostic labels: Hierarchical labels organize ECG statements into five superclasses and 23 subclasses, enabling predictions to be analyzed at multiple granularities.Predictions are propagated upward by summing child probabilities and limiting outputs to one; maximum and mean aggregation had only minor effects.
  • B. Hidden stratification and co-occurring pathologies: Hidden stratification can conceal inferior performance on diagnostic subpopulations when evaluation reports only superclass performance.The analysis examines this issue top-down by decomposing superclass AUCs into the AUCs of their subclasses.
  • B. Hidden stratification and co-occurring pathologies: Weak performance may reflect sparse classes and unreliable test-set estimates, but some ECG statements remain unusually difficult beyond sample-size effects.NST, LAO/LAE, and IVCD are identified as examples of weakly classified statements within the label hierarchy.
  • B. Hidden stratification and co-occurring pathologies: An unsupervised clustering analysis of IVCD samples was used to investigate groups exhibiting hidden stratification within that class.This analysis was motivated by the comparatively weak classification performance for IVCD.
  • B. Hidden stratification and co-occurring pathologies: The authors suggest studying subgroups defined by combinations of co-occurring ECG statements or pathologies as a direction for future clinical investigation.They caution that PTB-XL provides cardiologist annotations from ECG signals rather than clinical ground truth, which could compromise the analysis.

C. Model uncertainty and diagnosis likelihoods

The analysis compares PTB-XL diagnosis likelihoods with ensemble-based model uncertainty to assess whether the annotation confidence information follows the expected relationship.

  • C. Model uncertainty and diagnosis likelihoods: Diagnosis likelihoods range from 15 to 100, while model uncertainty is estimated as the variance across an ensemble of 10 identically configured models.The likelihoods were extracted from original ECG report strings using keywords, and adversarial training was omitted in this exploratory analysis.
  • C. Model uncertainty and diagnosis likelihoods: The comparison shows the expected monotonic behavior between diagnosis likelihood and model uncertainty, with likelihood 100 producing many outliers.The figure plots model uncertainty against diagnosis likelihood.

D. Prospects of interpretability methods

The paper explores post-hoc interpretability for ECG models by examining attribution maps for clinically recognizable PVC and PACE statements.

  • D. Prospects of interpretability methods: Post-hoc interpretability is considered because clinicians may perceive data-driven algorithms as black boxes.The methods are applied after a model has been trained.
  • D. Prospects of interpretability methods: For PVC, relevance is located at the extrasystole across all ECG leads.The attribution maps use the ϵ-rule with ϵ = 0.1 within layer-wise relevance propagation.
  • D. Prospects of interpretability methods: For PACE, relevance is scattered across the signal and aligns with pacemaker spikes immediately before each QRS complex.PVC and PACE were selected because their characteristic features are relatively easy to verify, including for non-cardiologists.
  • D. Prospects of interpretability methods: The broader benchmark finds convolutional architectures, particularly ResNet- and Inception-based models, strongest across the evaluated tasks.The paper presents interpretability as one of several analyses extending beyond quantitative benchmarking.

APPENDIX I EXPERIMENTAL DETAILS

The appendix details pooling, sliding-window aggregation, feature-based baselines, uncertainty intervals, patient-level splitting, and sampling-frequency experiments used to assess the benchmark setup.

  • Experimental details: The implementations mainly follow published architectures, with concat-pooling combining global average pooling and max pooling along the feature dimension.For ResNets, larger convolutional kernels slightly improved performance.
  • Experimental details: Random fixed-length segments train the classifier, while element-wise maximum aggregation—or mean aggregation for age and gender—produces whole-record predictions.The default window size is 2.5 seconds, and aggregation considerably improves performance over unaggregated random windows.
  • Experimental details: The feature-based baseline trains a classifier on statistical features computed from multilevel Daubechies db4 discrete wavelet transforms for each lead.The transform is applied independently to the leads before deriving statistical features from the coefficients.
  • Experimental details: Performance estimates include 95% confidence intervals obtained by empirical bootstrapping with 1,000 test-set iterations.Summary tables report the point estimate and the maximal absolute deviation from the lower and upper bounds.
  • Train-test splits and sampling frequency: Patient-disregarding random splits can systematically overestimate generalization because records from the same patient may appear in both training and test sets.The experiment compares such random ECG-level splits with patient-based splits while preserving clean validation and test folds.
  • Train-test splits and sampling frequency: Increasing sampling frequency from 100 Hz to 500 Hz showed no significant diagnostic-task gain across the reported metrics.The 500 Hz setup used five-second windows, 2,500 timesteps, and convolutional filters enlarged by a factor of five.
Loading 2004.13701v1…