PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement

Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, Rossella Arcucci

arXiv:2403.06659v3eess.SPcs.AIcs.LG

TL;DR

ECG self-supervised methods can overlook clinical knowledge and lack zero-shot classification, while multimodal ECG-report learning remains underexplored. MERL learns from ECGs and reports and uses clinical-knowledge-enhanced prompts at test time; across six datasets, zero-shot MERL outperforms eSSL methods using linear probing.

  • Problem

    eSSL methods focus on ECG signals rather than clinical concepts, limiting zero-shot classification, while ECG-report multimodal learning and broad benchmarks remain underexplored.

    Source: §1 · p. 2

  • Method

    MERL learns ECG representations through ECG-report multimodal alignment and uses LLMs to structure prompts from external clinical knowledge at test time.

    Source: §3.1 · p. 3, §6 · p. 8

  • Results

    Across six datasets, zero-shot MERL outperforms eSSL methods with linear probing; the benchmark evaluates MERL and 10 eSSL methods across diverse cardiac conditions.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    MERL's reported results support multimodal ECG learning with report supervision for zero-shot and linear-probing classification tasks.

    Source: §6 · p. 8

    LLM prompt generation remains not fully controlled and transparent despite using knowledge from clinical expert-verified databases.

    Source: Broader Impact · p. 9

Abstract

from arXiv · show

Electrocardiograms (ECGs) are non-invasive diagnostic tools crucial for detecting cardiac arrhythmic diseases in clinical practice. While ECG Self-supervised Learning (eSSL) methods show promise in representation learning from unannotated ECG data, they often overlook the clinical knowledge that can be found in reports. This oversight and the requirement for annotated samples for downstream tasks limit eSSL's versatility. In this work, we address these issues with the Multimodal ECG Representation Learning (MERL}) framework. Through multimodal learning on ECG records and associated reports, MERL is capable of performing zero-shot ECG classification with text prompts, eliminating the need for training data in downstream tasks. At test time, we propose the Clinical Knowledge Enhanced Prompt Engineering (CKEPE) approach, which uses Large Language Models (LLMs) to exploit external expert-verified clinical knowledge databases, generating more descriptive prompts and reducing hallucinations in LLM-generated content to boost zero-shot classification. Based on MERL, we perform the first benchmark across six public ECG datasets, showing the superior performance of MERL compared against eSSL methods. Notably, MERL achieves an average AUC score of 75.2% in zero-shot classification (without training data), 3.2% higher than linear probed eSSL methods with 10\% annotated training data, averaged across all six datasets. Code and models are available at https://github.com/cheliu-computation/MERL

1. Introduction

The introduction identifies semantic distortion, limited clinical semantics, and lack of zero-shot capability as challenges for ECG self-supervised learning. It presents MERL, test-time clinical prompt engineering, and a six-dataset benchmark as responses to these gaps.

1. Introduction

  • Contrastive eSSL may distort ECG semantics through input-level augmentation, while generative eSSL can overlook disease-level semantics needed for classification.Both issues motivate learning representations informed by clinical reports.Source: §1 · p. 2
  • Existing eSSL methods focus on signal patterns rather than clinical concepts, limiting zero-shot classification and risking distribution shifts in downstream tasks.The introduction also notes that ECG-report multimodal learning remains underexplored and that benchmarks are needed across cardiac conditions.Source: §1 · p. 2
  • Zero-shot MERL outperforms eSSL linear-probed with 10% annotated data on average across six datasets; without prompt engineering, it surpasses the best eSSL linear-probed with 1% data.With CKEPE, zero-shot MERL also outperforms the best eSSL results obtained with 10% data.Source: Figure 1 · p. 1, §1 · p. 2
  • MERL aligns ECG records with paired clinical reports using CMA and UMA, with latent-level rather than signal-level augmentation to avoid semantic distortion.This replaces naive input-level augmentation with latent augmentation.Source: §1 · p. 2
  • CKEPE uses LLMs to generate customized zero-shot prompts by extracting and restructuring knowledge from clinical-expert-verified databases.The introduction proposes this test-time approach to produce more descriptive prompts.Source: §1 · p. 2
  • The benchmark pretrains MERL and 10 eSSL methods on the largest publicly available ECG dataset and evaluates them across six datasets covering over 100 cardiac conditions.It includes zero-shot, linear-probing, and data-distribution-transfer scenarios.Source: §1 · p. 2

2. Related Work

Related work describes limitations in applying multimodal methods from radiology to ECG, the challenges of signal-only self-supervision, and the need for clinically grounded zero-shot prompts.

2.1. Representation Learning with Multimodal Medical Data

  • Prior ECG-report multimodal approaches are limited by private or small datasets, crude signal-text alignment, or prompts using only condition names.ECG's global temporal and spatial structure also makes direct transfer from image-report methods challenging.Source: §2.1 · p. 2

2.2. Self-supervised Learning for ECG on Signal Domain

  • Contrastive eSSL relies on augmented ECG views that can distort semantics, while generative eSSL reconstructs signals but may miss high-level clinical meaning.Figure 2 contrasts these approaches with MERL's ECG-report alignment and latent augmentation.Source: Figure 2 · p. 3, §2.2 · p. 3

2.3. Customizing Prompt for Zero-shot Classification

  • In specialized medical domains, LLM-generated prompts may be inaccurate or untrustworthy, motivating prompts structured from external clinical knowledge sources.This approach can create clinically relevant prompts without additional annotations.Source: §2.3 · p. 3

3. Method

MERL learns ECG representations from paired reports and combines train-time multimodal alignment with test-time clinically informed prompt generation. Its method includes cross-modal alignment, uni-modal alignment, and CKEPE for zero-shot classification.

3.1. Overview

  • MERL learns transferable ECG representations from ECG signals and associated reports, then applies them to zero-shot classification of unseen diseases.The framework combines train-time ECG-report representation learning with test-time clinical knowledge enhanced prompt engineering.Source: §3.1 · p. 3

3.2. Cross-Modal Alignment

  • Cross-Modal Alignment uses paired ECG records and reports to learn ECG features informed by clinical knowledge under report supervision.Separate ECG and text encoders map each pair into latent embeddings, followed by projectors that place them in the same dimensionality.Source: §3.2 · p. 3

3.3. Uni-Modal Alignment

  • Uni-Modal Alignment is jointly optimized with Cross-Modal Alignment; its dropout masks independently sample embedding entries at a 0.1 ratio to avoid identical paired views.The overall training loss minimizes LUMA and LCMA together.Source: §3.3 · p. 4

3.4. Enhancing Zero-Shot Prompts with External Clinical Knowledge Databases

  • CKEPE queries expert-evaluated clinical databases, discards terms not found or relevant, and reformulates verified attributes and subtypes into condition-specific prompts.The resulting structured prompts describe possible disease subtypes and ECG signal patterns for zero-shot classification.Source: §3.4 · pp. 4–5, Figure 3 · p. 4

4. Experiments

Experiments evaluate MERL through zero-shot classification, linear probing, and cross-dataset domain shift. Across six datasets, zero-shot MERL outperforms linear-probed eSSL overall, while MERL also performs strongly in probing and transfer settings.

4.1. Pre-training Configuration

  • After curation, MERL pre-training uses 771,693 paired ECG-report samples from a dataset initially containing 800,035 pairs.Pre-training runs for 50 epochs with a 1D-ResNet18 ECG encoder and Med-CPT text encoder.Source: §4.1 · p. 5

4.2. Downstream Tasks Configuration

  • The downstream evaluation includes zero-shot classification and linear probing across public ECG datasets; probing freezes the ECG encoder, while zero-shot classification uses CKEPE prompts and macro AUC.The datasets cover over 100 cardiac conditions, and Figure 4 compares zero-shot MERL with linear-probed eSSL using 1% and 10% data.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3. Evaluation on zero-shot learning

  • With CKEPE, zero-shot MERL exceeds the best eSSL method probed with 10% data on average across six datasets, and outperforms eSSL across all datasets even without additional samples.Its advantage is smaller on PTBXL-Super, while it outperforms eSSL methods probed with 1%–10% data on the other five, more challenging datasets.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4. Evaluation as ECG Representations

  • MERL consistently outperforms eSSL across training-data ratios and six datasets in linear probing; 1% MERL probing beats all eSSL methods with 100% data on PTBXL-Super, and 10% does so on the other five datasets.The results are reported in Table 1.Source: §4.4 · p. 6, Table 1 · p. 7

4.5. Robustness to Distribution Shift

  • Under domain shift, zero-shot MERL outperforms eSSL methods probed with 100% data in all but the PTBXL-Super→CSN setting.The evaluation trains on a source dataset and tests on a target dataset with shared categories; Table 2 reports these comparisons.Source: §4.5 · pp. 6–7, Table 2 · p. 7

5. Ablation Studies

The ablations assess average zero-shot classification and 1%-data linear probing across six ECG datasets. They examine loss components, encoders, knowledge databases, augmentation, dropout, and backbone scaling.

Ablation Studies

  • The ablations report average zero-shot classification and 1%-data linear-probing performance across six ECG classification datasets.This is the evaluation scope for the section’s component studies.Source: §5 · p. 7

Loss Function

  • Combining LCMA and LUMA improves performance over using CMA alone.The authors attribute UMA’s benefit to improved ECG representation learning in latent space for downstream tasks.Source: §5 · p. 7

Text Encoder

  • Med-CPT achieves the highest performance of the tested text encoders in both tasks.The authors attribute its advantage to discriminative text embeddings from contrastive pretraining.Source: §5 · p. 7, Table 4 · p. 7

Clinical Knowledge Database

  • Removing either clinical knowledge database decreases zero-shot classification performance.Removing the larger web database, SNOMEDCT, causes a notable reduction, and the authors report that the larger database provides more improvement.Source: §5 · p. 8, Table 5 · p. 8

Data Augmentation Strategies

  • Latent-space augmentation outperforms the tested raw-ECG augmentation strategies, which can distort ECG semantics and reduce representation quality.The authors report four augmentation strategies in Table 6.Source: §5 · p. 8, Table 6 · p. 8

Dropout Ratio

  • A dropout ratio of 0.1 gives the best latent-augmentation results; both higher and lower ratios reduce performance.The authors select 0.1 for their method.Source: §5 · p. 8, Table 7 · p. 8

Feature Extractors for ECG

  • MERL with a CNN backbone outperforms its ViT backbone variant.The authors suggest ViT tokenization of continuous signals may lose information.Source: §5 · p. 8, Table 8 · p. 8

Scalability

  • MERL consistently outperforms eSSL across scaled backbones, and its smallest backbone exceeds the largest eSSL backbones.MERL performance improves with larger backbones, while eSSL performance fluctuates; Figure 5 reports average linear-probing performance across six datasets.Source: §5 · p. 8, Figure 5 · p. 8

6. Conclusion

The paper introduces MERL, a multimodal ECG framework with alignment strategies for training and customized prompts at test time. It reports that MERL outperforms eSSL in zero-shot and linear-probing tasks and establishes a benchmark across ECG classification tasks.

Conclusion

  • MERL combines CMA and UMA alignment during training with CKEPE prompt customization at test time.CKEPE uses LLMs to extract and restructure clinical knowledge from a provided database.Source: §6 · p. 8
  • The paper establishes a benchmark of 10 eSSL methods and MERL, evaluated across broad classification tasks after pretraining on the largest public ECG datasets.The conclusion describes this as the first such benchmark.Source: §6 · p. 8
  • MERL outperforms signal-domain eSSL methods in both zero-shot and linear-probing classification tasks.The authors present this as evidence of the advantages of multimodal ECG learning with report supervision over signal-only representation learning.Source: §6 · p. 8

Broader Impact

The paper presents MERL as a zero-shot approach that needs no annotated data for downstream ECG tasks and reports a benchmark spanning six datasets and domain-transfer scenarios. It also identifies limits in controlling and transparently inspecting LLM-generated prompts, and proposes aligning ECGs with additional medical modalities as future work.

Broader Impact

  • MERL’s zero-shot approach eliminates the need for annotated data in downstream ECG tasks.The paper also reports that it outperforms linearly probed self-supervised learning methods through prompt engineering.Source: Broader Impact · p. 9
  • The benchmark covers six datasets and includes domain-transfer scenarios.The authors describe it as a comprehensive ECG representation-learning benchmark.Source: Broader Impact · p. 9
  • LLM prompt generation remains not fully controlled or transparent, leaving safe, controllable, and trusted clinical generation as an area for further development.The LLM is prompted to extract and reformat knowledge from clinical expert-verified databases.Source: Broader Impact · p. 9
  • Future work aims to align ECG records with electronic health records, cardiac imaging, and cardiologist reports.The stated aim is to enhance multimodal medical data understanding.Source: Broader Impact · p. 9

A.1. Downstream Task Data Split

The data-split procedure follows the official split for four PTBXL subsets and uses random splits for CPSC2018 and CSN. The authors state that split information will be released after acceptance.

A.1. Downstream Task Data Split

  • The four PTBXL subsets use the official split from the original work.The passage identifies the source as Wagner et al. (2020).Source: §A.1 · p. 12
  • CPSC2018 and CSN use random data splits.The passage cites Liu et al. (2018) for CPSC2018 and Zheng et al. (2022; 2020) for CSN.Source: §A.1 · p. 12
  • The authors state that all data-split information will be released post-acceptance.Source: §A.1 · p. 12

A.2. Downstream Task Configuration

Downstream configurations include category matching for domain transfer and hyperparameter settings for downstream tasks.

A.2. Downstream Task Configuration

  • Target-domain categories are merged into the most similar source category when possible, while categories absent from the source domain and their samples are removed.For example, “ST wave tilt up” and “ST wave drop down” map to “ST-T wave change.”Source: §A.2 · p. 12
  • Downstream-task hyperparameter settings are listed in Table 10.Source: Table 10 · p. 12, §A.2 · p. 12
  • Category-matching tables document PTBXL-Super-to-CPSC2018 transfer, with “None” marking target categories that belong to no source category.Source: Table 11 · p. 12, §A.2 · p. 12

B. Visualization on Learned ECG Representation

On the CSN test set, MERL’s learned ECG representations separate categories more clearly than SimCLR and ST-MEM in the visualization. The comparison also illustrates limitations attributed to the baselines’ pre-training targets.

B. Visualization on Learned ECG Representation

  • MERL distinguishes ECG samples from different categories without supervised learning, while SimCLR and ST-MEM struggle to separate mixed features.The visualization uses the CSN test set and retains unique categories with at least 50 samples.Source: B. Visualization on Learned ECG… · p. 12, Figure 6 · p. 13
  • The accompanying category-matching tables cover PTBXL-Super-to-CSN and CPSC2018-to-CSN transfer.Source: Table 12 · p. 13, Table 13 · p. 13
  • ST-MEM’s reconstruction target focuses on low-level signal patterns, while SimCLR learns from semantically distorted samples, which the authors identify as limitations for discriminative semantics.The authors attribute the learned-feature differences to clinical report supervision benefiting the ECG encoder.Source: B. Visualization on Learned ECG… · pp. 12–13

Related papers

Loading 2403.06659v3…