Source-linked AI summary
Automatic diagnosis of the 12-lead ECG using a deep neural network
Antônio H. Ribeiro, Manoel Horta Ribeiro, Gabriela M. M. Paixão, Derick M. Oliveira, Paulo R. Gomes, Jéssica A. Canazart, Milton P. S. Ferreira, Carl R. Andersson, Peter W. Macfarlane, Wagner Meira, Thomas B. Schön, Antonio Luiz P. Ribeiro
TL;DR
Existing automatic ECG analysis has limited accuracy, while the usefulness of DNNs in the standard 12-lead clinical setting remained an open question. The paper develops an end-to-end DNN using a large 12-lead ECG dataset and reports accurate recognition of six abnormalities, with limitations in statistical comparison and clinical scope.
Problem
Existing automatic ECG models have limited performance, and whether DNN-based analysis extends effectively from single-lead ECGs to standard 12-lead exams remained an open question.
Method
The study adapts a residual-network DNN to one-dimensional signals and trains it end-to-end on a large dataset of labeled 12-lead ECG exams.
Results
The DNN accurately recognized six ECG abnormalities in 12-lead exams, with diagnostic performance at least as good as medical residents and students.
Takeaways & Limitations
The findings support the potential of DNN technology for automatic 12-lead ECG interpretation and improved access to this diagnostic exam.
Takeaways & Limitations
The study cannot establish statistically significant superiority over medical trainees and does not extend its results to untested abnormalities or clinical situations.
Abstract
from arXiv · showhide
The role of automatic electrocardiogram (ECG) analysis in clinical practice is limited by the accuracy of existing models. Deep Neural Networks (DNNs) are models composed of stacked transformations that learn tasks by examples. This technology has recently achieved striking success in a variety of task and there are great expectations on how it might improve clinical practice. Here we present a DNN model trained in a dataset with more than 2 million labeled exams analyzed by the Telehealth Network of Minas Gerais and collected under the scope of the CODE (Clinical Outcomes in Digital Electrocardiology) study. The DNN outperform cardiology resident medical doctors in recognizing 6 types of abnormalities in 12-lead ECG recordings, with F1 scores above 80% and specificity over 99%. These results indicate ECG analysis based on DNNs, previously studied in a single-lead setup, generalizes well to 12-lead exams, taking the technology closer to the standard clinical practice.
1 Results
The study trains and evaluates a residual-network DNN for six ECG abnormalities using a large dataset, then compares its test performance with medical trainees and alternative data splits.
- Dataset: 2,322,513 ECG records from 1,676,384 patients across 811 Minas Gerais counties formed the dataset, with 98% used for training and about 2% for validation.The validation set contained approximately 50,000 exams and supported hyperparameter tuning.
- Targets: The DNN classified first-degree AV block, right bundle branch block, left bundle branch block, sinus bradycardia, atrial fibrillation and sinus tachycardia.These abnormalities include rhythmic and morphologic ECG abnormalities.
- Model specification: A residual-network architecture adapted from image models was used for one-dimensional ECG signals.The architecture had roughly one quarter the layers and parameters of the network used in the referenced single-lead study.
- Testing and performance: The test set contained 827 tracings from distinct patients annotated by three cardiologists, and all abnormalities achieved F1 scores above 80% and specificity over 99%.The model classified an abnormality when its output exceeded a threshold.
- Testing and performance: The DNN matched or outperformed residents and students by F1 score for all abnormalities, although statistical analyses found no significant performance difference for most classes.Performance comparisons used precision, recall, specificity and F1 distributions, with bootstrap resampling and McNemar tests.
- Robustness analysis: Alternative 90%-5%-5% splits produced no statistically significant difference from the original model, except for the chronological split affected by changes in telehealth-center operation over time.The alternative splits were stratified randomly, by patient, or chronologically.
2 Discussion
The study positions end-to-end DNNs as an alternative to feature-based ECG analysis, enabled by a large 12-lead dataset and showing strong recognition of six abnormalities. Performance was at least comparable to medical trainees, but statistical superiority and broader clinical applicability remain unestablished.
- End-to-end classification: End-to-end learning uses raw ECG signals directly, allowing the classifier to learn feature extraction instead of relying on a separate signal-processing stage.This contrasts with classical methods that first extract signal features and then classify them.
- Dataset scale and scope: 2 million entries made the dataset orders of magnitude larger than prior studies and covered short, in-clinic 12-lead ECG exams.The recordings lasted 7 to 10 seconds and included rhythm disorders and conduction disturbances.
- Error analysis: The DNN made most errors on ECG-interval measurements, especially borderline cases requiring sharp diagnostic cutoffs.Examples included sinus bradycardia slightly above 50 bpm and sinus tachycardia slightly below 100 bpm.
- Human comparison: The DNN matched or outperformed medical residents and students across the reported abnormalities, while emergency residents performed worse than medical students for many abnormalities.The authors relate this pattern to differences in routine ECG interpretation and recency of training.
- Limitations: Bootstrap and McNemar analyses did not provide sufficient confidence that the DNN was statistically superior to the medical residents and students for most classes.The authors attribute limited confidence partly to relatively infrequent classes, where a few errors can change scores substantially.
- Limitations: The study did not test acute coronary syndromes or chamber enlargement, and complex or borderline exams may still require experienced specialist review.The authors propose testing multiple complex abnormalities and controlled real-world, real-time use in future work.
3 Methods
The study used ECG data from the Telehealth Network of Minas Gerais and organized CODE-study records into training, validation, and test periods. The dataset primarily came from primary-care facilities using a tele-electrocardiograph.
- Data source: All analyzed ECGs came from the Telehealth Network of Minas Gerais, a public telehealth system serving 811 of 853 municipalities in Minas Gerais.Since September 2017, the network also provided telediagnostic services to other Brazilian states.
- Acquisition setting: The S12L-ECG exams were performed mostly in primary-care facilities using a tele-electrocardiograph manufactured by Tecnologia Eletrônica Brasileira.
- Database organization: The CODE study standardized and consolidated the database for clinical and epidemiological studies.
- Dataset partitioning: Patients above 16 years old from 2010–2016 supplied the training and validation data, while exams from April–September 2018 supplied the test set.
3.3 Training and validation set annotation
Training and validation labels combined automated analyses, measurements, and text diagnoses, followed by rule-based filtering and manual review. The resulting procedure introduced fewer than 3% false positives according to sensitivity analysis.
- Label sources: Training and validation annotations combined Uni-G statements and Minnesota codes, Uni-G measurements, and labels extracted from expert reports with a semi-supervised method.Both automated classifications and textual labels were recognized as potentially erroneous.
- Acceptance and rejection rules: Diagnoses were accepted when expert and automated evidence agreed, while diagnoses supported by only one conflicting automatic classifier were rejected.The procedure separately handled unresolved cases where experts and classifiers disagreed.
- Sensitivity analysis: Sensitivity analysis of 100 manually reviewed exams per abnormality produced additional acceptance rules for RBBB, 1dAVb, SB, ST, and AF.For RBBB, 1dAVb, SB, and ST, all medical diagnoses were accepted; AF required additional agreement criteria.
- Annotation quality: Fewer than 3% of total exams were estimated to receive false-positive diagnoses from the annotation procedure.
- Manual review: 34512 unresolved exams were manually reviewed by medical students under supervision from a certified cardiologist, over several months.Previous reports and automatic measurements were used only to obtain training and validation ground truth, not during later DNN training.
3.4 Test set annotation
The test set was independently annotated by certified cardiologists, with consensus diagnoses used as ground truth. Unlike training and validation, test labels were directly selected from predefined abnormality classes.
- Independent annotation: Two certified cardiologists independently annotated the test ECGs, and disagreements were adjudicated by a third senior specialist.Reported inter-rater kappa values ranged from 0.741 for 1dAVb to 0.964 for LBBB.
- Ground truth: When the two cardiologists agreed, their common diagnosis was treated as the test-set ground truth.
- Label coding: Test annotation used an upgraded TNMG system that presented Uni-G automatic measurements while specialists selected diagnoses from predefined abnormality classes.Therefore, test labels were codified directly rather than extracted from textual reports.
3.5 Neural network architecture and training
The model uses a one-dimensional residual convolutional architecture for 12-lead ECG classification, receiving standardized fixed-length signals and producing multilabel outputs. Training uses residual blocks, normalization, adaptive optimization, and validation-based model selection.
- Network architecture: The architecture adapts residual networks for one-dimensional signals, using skip connections to enable efficient deep-network training.The design is based on a residual network architecture commonly used for images.
- Input representation: The ECG input is resampled to 400 Hz and zero-padded to 4096 samples per lead.Recordings contain 7–10 seconds before padding.
- Network architecture: The network comprises an initial convolutional layer, four residual blocks, and a fully connected sigmoid output layer.Sigmoid activation supports non-mutually-exclusive abnormality classes.
- Network architecture: Convolutional outputs are processed with batch normalization and rectified linear activations throughout the network.Skip connections include pooling and 1x1 convolutions to match signal dimensions.
- Training: Training minimizes average cross-entropy with Adam at learning rate lr = 0.001, reducing the rate after seven validation epochs without improvement.The final model is selected from 50 epochs based on the best validation results.
3.6 Hyperparameter tuning
The final architecture and hyperparameters were selected through iterative training and validation, with manual revisions informed by prior results. The search considered alternative network depths, optimization settings, regularization, and architectural variants.
- Tuning procedure: The final configuration was obtained after approximately 30 iterations of training, validation, and manual architecture or hyperparameter changes.The procedure began from the architecture and hyperparameters used in prior work.
- Search space: The search varied residual-block count, kernel size, batch size, initial learning rate, optimizer, activation, dropout, and learning-rate plateau settings.Residual networks with 2, 4, 8, or 16 blocks and kernel sizes of 8, 16, or 32 were considered.
- Architectural alternatives: Alternative designs included vectorcardiogram dimensionality reduction, an LSTM before convolution, VGG convolutions, and different activation–normalization orders.The study also tested residual networks without the cited preactivation architecture.
3.7 Statistical and empirical analysis of test results
The study evaluates class discrimination with precision–recall analysis and threshold selection based on F1, then compares multiple metrics and error patterns across the DNN and human readers. Robustness is additionally examined across alternative dataset splits.
- Metric analysis: The DNN threshold was fixed at the value maximizing F1, chosen because F1 is robust to class imbalance.Precision–recall curves were preferred to ROC plots for imbalanced classes.
- Metric analysis: Precision, recall, specificity, F1, and confusion matrices were computed for each class and reader group.Bootstrap resampling generated empirical score distributions from 1000 test-set samples with replacement.
- Error analysis: Misclassified exams were reviewed by an experienced cardiologist and categorized into error types specific to the DNN or human readers.DNN categories included measurement, noise, and unexplained errors; human categories included conceptual and attention errors.
- Robustness analysis: F1 performance was evaluated under random, date-ordered, and patient-stratified 90%-5%-5% dataset splits.Models were tested on both the original test set and additional final 5% splits using bootstrap analyses.
Data availability
The test dataset and all developed model weights are openly available, while access to the training set is restricted and considered individually by the Telehealth Network of Minas Gerais.
- Open resources: The test dataset is openly available for download through Zenodo.The passage provides a DOI for the dataset.
- Open resources: Weights for all deep neural network models developed in the paper are available through Zenodo.The passage provides a separate DOI for the model weights.
- Training data: Training-data access is restricted and requests are considered individually by the Telehealth Network of Minas Gerais.Any approved data use is restricted to non-commercial research purposes.
Code availability
The paper provides the code for training and evaluating the DNN and producing its figures and tables, and reports ethics approval for the study.
- The training, evaluation, figure-generation, and table-generation code is publicly available.The paper gives a GitHub repository URL for the implementation.
- The study was approved by the Research Ethics Committee of the Universidade Federal de Minas Gerais under protocol 68496317.7.0000.5149.
Author contribution statement
The author statement assigns responsibilities across study design, DNN development, dataset preparation, analysis, and project leadership, while reporting no competing interests.
- Study design was led by the listed authors, with A.L.R. conceiving the project and acting as project leader.
- A.H.R., M.H.R., and C.A. selected, implemented, and tuned the deep neural network.
- Dataset preprocessing and annotation were assigned to M.H.R., G.M.M.P., and J.A.C., while A.H.R. handled statistical analysis and generated figures and tables.
- The authors declare no competing interests.
Supplementary Information
The supplementary information provides confusion matrices, agreement statistics, bootstrapped performance distributions, alternative data-split analyses, and heart-rate analyses of DNN predictions.
- Supplementary Table 1 reports false positives, false negatives, true positives, and true negatives for each abnormality on the test set.
- Supplementary Table 2 reports kappa coefficients for agreement between the DNN, medical residents, students, and certified cardiologists.Kappa equals 1 for complete agreement and 0 for agreement no greater than chance expectation.
- Supplementary Figure 1 displays boxplots of bootstrapped precision, recall, specificity, and F1-score distributions for the DNN, residents, and students.The distributions use 1,000 samples generated by sampling with replacement from the test set.
- Supplementary Table 3 presents McNemar-test p-values comparing test-set misclassification between the DNN, residents, and students.Statistically significant entries at the 0.05 level are shown in boldface.
- Supplementary Figure 2 shows bootstrapped F1 scores for alternative random, date-ordered, and patient-stratified 90%-5%-5% data splits.Performance is evaluated on the original and secondary test sets.
- Supplementary Figure 3 relates Uni-G heart-rate measurements to correct and incorrect DNN predictions for sinus bradycardia and sinus tachycardia.Horizontal lines mark the consensus thresholds of 50 bpm for sinus bradycardia and 100 bpm for sinus tachycardia.