PaperlayerView exact revision on arXiv

Source-linked AI summary

From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining

Fuying Wang, Jiacheng Xu, Lequan Yu

arXiv:2506.21803v1eess.SPcs.AIcs.LG

TL;DR

Existing ECG-language methods often overlook ECGs’ multi-scale structure, limiting representation of their hierarchy. MELP combines cardiology-specific language pretraining with token-, beat-, and rhythm-level ECG-text supervision. Experiments on three benchmark datasets report superior performance over existing baselines.

  • Problem

    Existing ECG-language methods often miss ECGs’ multi-scale structure, leaving hierarchical ECG-text alignment insufficiently addressed.

    Source: §1 · p. 2

  • Method

    MELP pretrains a cardiology-specific language model and aligns ECGs with reports using token-, beat-, and rhythm-level supervision.

    Source: §1 · p. 2

  • Results

    Extensive experiments on three benchmark datasets demonstrate superior performance over existing baselines.

    Source: §5 · p. 9

  • Takeaways & Limitations

    MELP demonstrates effectiveness in aligning ECG signals with clinical text at multiple levels of abstraction.

    Source: §5 · p. 9

    Token-level supervision lacks explicit clinical interpretability, limiting applicability in explainable diagnostic settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Electrocardiograms (ECGs) play a vital role in monitoring cardiac health and diagnosing heart diseases. However, traditional deep learning approaches for ECG analysis rely heavily on large-scale manual annotations, which are both time-consuming and resource-intensive to obtain. To overcome this limitation, self-supervised learning (SSL) has emerged as a promising alternative, enabling the extraction of robust ECG representations that can be efficiently transferred to various downstream tasks. While previous studies have explored SSL for ECG pretraining and multi-modal ECG-language alignment, they often fail to capture the multi-scale nature of ECG signals. As a result, these methods struggle to learn generalized representations due to their inability to model the hierarchical structure of ECG data. To address this gap, we introduce MELP, a novel Multi-scale ECG-Language Pretraining (MELP) model that fully leverages hierarchical supervision from ECG-text pairs. MELP first pretrains a cardiology-specific language model to enhance its understanding of clinical text. It then applies three levels of cross-modal supervision-at the token, beat, and rhythm levels-to align ECG signals with textual reports, capturing structured information across different time scales. We evaluate MELP on three public ECG datasets across multiple tasks, including zero-shot ECG classification, linear probing, and transfer learning. Experimental results demonstrate that MELP outperforms existing SSL methods, underscoring its effectiveness and adaptability across diverse clinical applications. Our code is available at https://github.com/HKU-MedAI/MELP.

1. Introduction

Existing ECG-language models largely align whole recordings and miss ECGs’ token-, beat-, and rhythm-level structure. MELP combines clinical-text pretraining with supervision at all three scales and is reported to achieve state-of-the-art performance across three downstream task types.

1. Introduction

  • MELP aligns ECGs and clinical text at token, beat, and rhythm levels, addressing the limits of global-only alignment.The three views represent waveform segments, cardiac cycles, and the full recording’s rhythm.Source: §1 · p. 2
  • The model first pretrains a cardiology-specific language model, then applies cross-modal supervision to paired ECG-text data.Its token-level task generates waveform-segment descriptions, beat-level learning contrasts beats with clinical descriptions, and rhythm-level learning uses a global contrastive loss.Source: §1 · p. 2
  • MELP is introduced as a hierarchical ECG representation-learning model that integrates clinical text knowledge through structured supervision.The introduction identifies this model and framework as the paper’s contributions.Source: §1 · p. 2
  • MELP achieves state-of-the-art performance in zero-shot classification, linear probing, and cross-domain transfer learning across three public ECG datasets.The evaluation spans self-supervised and multimodal baselines.Source: §1 · p. 2

2. Related Works

Prior ECG representation learning includes contrastive, generative, and combined approaches, while ECG-language work has explored several forms of multimodal alignment. The related-work section identifies limited modeling of localized ECG patterns as a remaining gap.

2.1. ECG Representation Learning

  • ECG representation learning includes contrastive and generative methods, as well as foundation models combining both objectives, but these approaches use ECG alone.Contrastive methods use augmented views; generative methods reconstruct masked signal portions or beats.Source: §2.1 · p. 2

2.2. ECG-Language Pretraining

  • ECG-language studies have used EHR-guided representation learning, ECG-report alignment, and joint masked language and ECG modeling.Examples include ESI, MERL, C-MELT, and ECG-Chat.Source: §2.2 · pp. 2–3
  • Existing multimodal models predominantly use global ECG representations, overlooking fine-grained local patterns that may limit adaptability to downstream tasks.Source: §2.2 · p. 3

3. Method

MELP first pretrains a cardiology-focused text encoder, then aligns ECGs and reports at token, beat, and rhythm scales. Its pretrained representations support retrieval-based zero-shot evaluation and linear or fine-tuned classification.

3.1. Overview

  • MELP’s overview combines domain-specific language pretraining, multimodal alignment, and downstream transfer into one ECG-language framework.The method structures cross-modal supervision at token, beat, and rhythm scales.Source: §3.1 · p. 3, §3.3 · p. 3, Figure 2 · p. 4

3.2. Cardiology Language Pretraining

  • The cardiology text encoder is pretrained with masked language modeling on cardiology-related PubMed, Wikipedia, and MIMIC-IV-ECG text.MELP then initializes its text encoder from this pretrained model.Source: §3.2 · p. 3, §3.3 · p. 3

3.3. Multimodal Pretraining

  • At the token level, ECG features condition an autoregressive decoder that generates the paired report, exposing the model to waveform details at multiple report granularities.The decoder uses 128 learnable query tokens to summarize ECG embeddings; report generation may help it learn relationships between local features and high-level findings.Source: §3.3 · pp. 3–4
  • The method targets fine-grained beat–sentence matching, using attention-pooled ECG beat embeddings and sentence embeddings formed by averaging word tokens, with an attention-weighted beat representation for each sentence.It uses 10 learnable tokens to summarize beats within a 10-second ECG segment; this default is not necessarily optimal.Source: §3.3 · p. 4
  • MELP’s global rhythm view averages beat embeddings and uses report-level contrastive supervision to align ECG and text representations.The framework jointly optimizes global, language-modeling, and local alignment losses.Source: §3.3 · p. 5

3.4. Transferring into Downstream Tasks

  • For zero-shot retrieval, MELP uses global ECG and text embeddings; for fine-tuning, it average-pools beat embeddings before a linear classification layer.Linear probing freezes the network and trains only the linear layer.Source: §3.4 · p. 5

4. Experiments

MELP is evaluated through linear probing, zero-shot classification, transfer learning, and ablations across ECG datasets. It performs strongly across these settings, while ablations assess multi-scale supervision, encoder choice, text pretraining, and hyperparameters.

4.1.1. PRETRAINING TASK

  • Pretraining used 760,618 curated ECG-text pairs from MIMIC-IV-ECG; downstream evaluation covered PTB-XL, CSN, and CPSC2018 using AUROC.The pretraining database initially contained 800,035 recordings; paired-text and signal-quality exclusions yielded the final set.Source: §4.1.1 · p. 5, §4.1.2 · p. 6

4.2.1. EVALUATION ON LINEAR PROBING FOR ECG CLASSIFICATION

  • MELP achieved the highest AUROC in 16 of 18 linear-probing settings and ranked first or second in all full-data dataset evaluations; it also led zero-shot results across all tasks.The reported average zero-shot gain over MERL was 3.7%.Source: Table 2 · p. 6, §4.2.1 · pp. 6–7, Table 3 · p. 6

4.2.3. EVALUATION ON TRANSFER LEARNING FOR ECG CLASSIFICATION

  • MELP had the best average performance across six transfer tasks, ranking first in three and second in two, with reported gains over MERL in three domain shifts.The reported gains were +1.12% for CPSC2018 → PTBXL-Super, +3.76% for CPSC2018 → CSN, and +0.52% for CSN → PTBXL-Super.Source: Table 4 · p. 7, §4.2.3 · p. 7

4.3.1. ABLATION ON MULTI-SCALE SUPERVISION

  • The full multi-scale model exceeded the best partial-supervision variant by 2.86% average AUROC, while variants without rhythm-level supervision performed worse.All configurations including rhythm-level supervision achieved AUROCs above 82%; the full model also improved zero-shot AUROC by 2.1% on average over ablated versions.Source: §4.3.1 · pp. 7–8, Table 6 · p. 8

4.3.2. ANALYSIS ON ECG REPORT GENERATION

  • On 500 curated PTB-XL report samples, MELP significantly outperformed PULSE on report-generation metrics, while broader fine-grained prediction analysis remains future work.The evaluation used NLG metrics and BERTScore.Source: §4.3.2 · p. 8, Table 9 · p. 8

4.3.3. ABLATION ON ECG ENCODER

  • Wav2Vec 2.0 outperformed the CMSC-based variant across evaluation settings by an average 3.53% and beat the ResNet-18 variant in 17 of 18 settings.Its average improvement over the ResNet-18 variant was 3.63%.Source: §4.3.3 · p. 8, Table 7 · p. 8

4.3.4. ABLATION ON UNI-MODAL TEXT PRETRAINING

  • With cardiology-specific text pretraining, the full model outperformed its no-text-pretraining baseline in 13 of 18 settings, improving by 0.35% on average.The authors associate text pretraining with better feature separation in the joint embedding space.Source: §4.3.4 · pp. 8–9

4.3.5. ABLATION ON HYPERPARAMETERS

  • Zero-shot performance remained consistent across four hyperparameter configurations, with λ1 = 2 and λ2 = 0.2 selected as defaults; MELP’s CSN embeddings appeared more separated than MERL’s.The embedding comparison visualizes seven common ECG abnormalities from the CSN test set.Source: §4.3.5 · p. 9, Table 10 · p. 9, §4.4 · p. 9

5. Conclusion

MELP learns fine-grained ECG representations from clinical reports and outperforms existing baselines across three benchmark datasets. Its limitations include token-level interpretability and reliance on the MedCPT text encoder.

5. Conclusion

  • MELP is a multimodal ECG foundation model that uses multi-scale supervision from clinical reports to learn fine-grained representations for diverse downstream tasks.Source: §5 · p. 9
  • MELP demonstrates superior performance over existing baselines in experiments on three benchmark datasets.The authors attribute this to aligning ECG signals with clinical text at multiple abstraction levels.Source: §5 · p. 9
  • Token-level supervision lacks explicit clinical interpretability, and the MedCPT text encoder does not fully capitalize on recent large language model advances.The authors identify explainable diagnostic settings as a constraint and propose exploring ECG instruction tuning and external medical knowledge bases.Source: §5 · p. 9

Impact Statement

The impact statement flags privacy, potential demographic bias, and regulatory considerations for clinical deployment. The cardiology corpus combines curated PubMed and Wikipedia content with ECG reports from MIMIC-IV-ECG.

Impact Statement

  • De-identification of MIMIC-IV-ECG does not remove privacy considerations; predictions may reflect subtle gender or race biases, and clinical adaptation requires regulatory consideration.The authors call for further study of privacy implications and bias-mitigation strategies.Source: Impact Statement · p. 9

A.1. Cardiology Corpus Details

  • The cardiology corpus uses PubMed abstracts selected with cardiology-related queries and Wikipedia articles collected under the Cardiology category and its subcategories.The curated corpus is approximately 5.6 GB, and MIMIC-IV-ECG training reports are added for ECG-specific clinical language.Source: §A.1 · p. 13

A.2. Training Details

The text pretraining adapts MedCPT’s query encoder to cardiology using masked language modeling on ECG diagnostic reports. Its workflow prepares masked, sentence-segmented inputs for the encoder.

A.2. Training Details

  • The text encoder is initialized from MedCPT’s query encoder, originally trained on PubMed search logs, and fine-tuned for cardiology.Source: §A.2 · p. 13
  • Masked language modeling trains the encoder to reconstruct randomly masked tokens in ECG diagnostic reports from their surrounding context.The objective is intended to capture domain-specific semantics in cardiology narratives.Source: §A.2 · p. 13
  • Three medical datasets are combined and segmented into sentences; portions are masked, special tokens are inserted between sentences, and the sequences are processed by the MedCPT query encoder.Source: Figure 4 · p. 13

B. Discussion on Multi-Scale ECG Interpretation

The discussion argues that ECG reports contain waveform-level details and that broad diagnoses can reflect fine-grained waveform criteria. It motivates token-level ECG embeddings while noting that richer textual descriptions remain future work.

B. Discussion on Multi-Scale ECG Interpretation

  • MIMIC-IV-ECG reports frequently describe detailed waveform-level features, as illustrated by complete report examples in Table 11.The table caption says bold text marks fine-grained ECG details.Source: B. Discussion on Multi-Scale ECG… · p. 14, Table 11 · p. 14
  • Even “sinus rhythm” assessments are based on low-level criteria such as P-wave placement, regular R-R intervals, consistent PR intervals, and normal heart rate.The authors say these criteria are not explicitly mentioned in MIMIC-IV-ECG reports but can be captured by token-level representations.Source: B. Discussion on Multi-Scale ECG… · p. 14, Table 12 · p. 14
  • The authors argue token-level ECG embeddings for report generation are appropriate and essential for capturing clinically meaningful information.Source: B. Discussion on Multi-Scale ECG… · p. 14
  • Adding explicit waveform-component descriptions alongside general diagnostic terms may strengthen the approach, but this direction is beyond the current work’s scope.The authors suggest details such as P waves before each QRS complex and consistent PR intervals as richer supervision.Source: B. Discussion on Multi-Scale ECG… · p. 14

C.1. Results for Patient Identification task

MELP outperforms three baselines on all reported patient-identification recall metrics. An ablation finds 16 learnable beats perform best among tested variants, with a +0.66% gain over 10 beats.

C.1. Results for Patient Identification task

  • Patient identification evaluates whether ECG representations distinguish patients while keeping recordings from the same patient similar.The experiments follow the settings of Oh et al. (2022).Source: §C.1 · pp. 14–15
  • MELP outperforms all three baselines in top-1, top-5, and top-10 patient-identification accuracy.The task trains models to produce similar ECG representations for the same patient and distinct representations for different patients.Source: §C.1 · pp. 14–15, Table 13 · p. 15

C.2. Ablation on Number of Beats.

  • 16 learnable beats perform best among tested variants, gaining +0.66% over the 10-learnable-beat setting.The authors report a median of 12–13 beats in the training set, compared with their 10-beat default.Source: §C.2 · p. 15, Table 15 · p. 15

D. Dataset Details.

The study uses MIMIC-IV-ECG for unlabeled pretraining and PTB-XL, CPSC2018, and CSN for labeled downstream evaluation across diagnostic categories.

D. Dataset Details.

  • MIMIC-IV-ECG provides 760,618 unlabeled pretraining records, while PTB-XL, CPSC2018, and CSN provide labeled downstream evaluation across five, nine, and four categories.PTB-XL has 12,978 training, 1,642 validation, and 1,652 test samples; CPSC2018 has 8,958, 1,303, and 2,598; CSN has 7,651, 851, and 2,126, respectively.Source: D. Dataset Details · p. 15

E.1. Evaluation Metric

The section defines AUC as the classification metric, specifies label-aligned cross-dataset transfer evaluation, and details MELP’s ECG and text network architecture.

E.1. Evaluation Metric

  • AUC measures classification performance from the ROC curve, with higher values indicating better discrimination; optimization prioritizes AUC.It estimates the probability that a randomly selected positive instance ranks above a negative one and reflects TPR-FPR trade-offs.Source: §E.1 · p. 15

E.2. Domain Transfer Experimental Details

  • Transfer learning trains on one dataset and evaluates on another using aligned overlapping labels, excluding categories without direct correspondence.MELP is evaluated directly on the overlapping label set because it supports zero-shot classification.Source: §E.2 · p. 16

E.3. Network Architecture

  • MELP combines an ECG feature extractor and 8-layer Transformer with a 12-layer causal-attention text Transformer, text projector, and 6-layer text decoder.Both branches include attentional pooling; the ECG branch also uses Conv1d-based positional encoding.Source: §E.3 · p. 16
Loading 2506.21803v1…