PaperlayerView exact revision on arXiv

Source-linked AI summary

Teach Multimodal LLMs to Comprehend Electrocardiographic Images

Ruoqi Liu, Yuelin Bai, Xiang Yue, Ping Zhang

arXiv:2410.19008v1eess.IVcs.AIcs.CV

TL;DR

Automatic ECG interpretation often focuses on limited conditions and depends on raw signals that may not be available when only ECG images can be accessed; general MLLMs can also interpret those images inaccurately. The paper introduces ECGInstruct, PULSE, and ECGBench, and reports state-of-the-art results for PULSE, while noting difficulties on complex open-ended tasks.

  • Problem

    Existing ECG automation has limited generalizability and often relies on raw signals, while general MLLMs can produce inaccurate ECG image interpretations.

    Source: §1 · p. 2

  • Method

    The authors build ECGInstruct, an instruction-tuning dataset with over one million samples, train the ECG image MLLM PULSE, and curate ECGBench.

    Source: §1 · p. 2

  • Results

    PULSE sets a new state-of-the-art, with a 15% to 30% average accuracy gain over GPT-4o on out-of-domain datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that PULSE demonstrates potential for enhancing ECG image analysis and interpretation in clinical applications.

    Source: §5 · p. 11

    PULSE faces challenges on complex open-ended tasks, including report generation and multi-turn conversations, and struggles more with waveform and diagnosis identification than rhythm interpretation.

    Source: §4.6 · p. 10

Abstract

from arXiv · show

The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for assessing cardiac conditions. Existing automatic interpretation methods suffer from limited generalizability, focusing on a narrow range of cardiac conditions, and typically depend on raw physiological signals, which may not be readily available in resource-limited settings where only printed or digital ECG images are accessible. Recent advancements in multimodal large language models (MLLMs) present promising opportunities for addressing these challenges. However, the application of MLLMs to ECG image interpretation remains challenging due to the lack of instruction tuning datasets and well-established ECG image benchmarks for quantitative evaluation. To address these challenges, we introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset of over one million samples, covering a wide range of ECG-related tasks from diverse data sources. Using ECGInstruct, we develop PULSE, an MLLM tailored for ECG image comprehension. In addition, we curate ECGBench, a new evaluation benchmark covering four key ECG image interpretation tasks across nine different datasets. Our experiments show that PULSE sets a new state-of-the-art, outperforming general MLLMs with an average accuracy improvement of 15% to 30%. This work highlights the potential of PULSE to enhance ECG interpretation in clinical practice.

1 INTRODUCTION

PULSE addresses the challenge of interpreting ECG images with specialized training data, a tailored MLLM, and a benchmark for evaluation. The authors report state-of-the-art performance and identify remaining challenges on complex, open-ended tasks.

1 INTRODUCTION

  • Existing automatic ECG interpretation methods often focus on limited cardiac conditions and use raw signals, which may be unavailable where only ECG images can be accessed.ECGs are non-invasive, cost-effective, and widely accessible for assessing cardiac function.Source: §1 · p. 2
  • General MLLMs can give contextually relevant but inaccurate ECG image interpretations, motivating specialized models for this task.The paper identifies missing large-scale ECG image datasets, instruction-tuning datasets, and established evaluation benchmarks as development challenges.Source: §1 · p. 2
  • The paper introduces ECGInstruct, an ECG image-text instruction dataset with over one million samples, and uses it to develop PULSE for ECG image comprehension.ECGBench covers four interpretation tasks across nine datasets, including repurposed tasks and challenging tasks using real-world ECG images.Source: §1 · p. 2
  • PULSE sets a new state-of-the-art, outperforming proprietary MLLMs across benchmarks with a 15% to 30% average accuracy gain over GPT-4o on out-of-domain datasets.The authors also report that ablations support the importance of diverse data sources and ECG instruction tasks in training.Source: §1 · p. 2
  • The paper frames its work as the first study focused on assessing MLLMs for image-based ECG interpretation.Its evaluation considers performance across downstream tasks.Source: §1 · p. 2

2 ECGINSTRUCT: TEACH MLLMS TO COMPREHEND ECG IMAGES

ECGInstruct combines synthesized ECG images, clinically informed tasks, geographically diverse data, and generated instruction-response pairs. The section also describes quality checks and the model’s training setup.

2 ECGINSTRUCT: TEACH MLLMS TO COMPREHEND ECG IMAGES

  • ECGInstruct is designed as a multifaceted instruction-tuning dataset, combining realistic ECG image synthesis, clinically informed task types, and data from distinct geographic regions.Its curation uses clinician-defined tasks, diagnoses and clinical reports, diverse task types, and quality checking of lower-scored instructions.Source: §2 · pp. 2–3, Figure 2 · p. 3

2.1 ECG IMAGE SYNTHESIS WITH VARIOUS DISTORTIONS

  • ECG images are synthesized with artifacts such as grid interference, creases, wrinkles, and rotations to mimic imperfect paper ECGs.The authors aim to make the images more representative of less-than-ideal clinical conditions.Source: §2.1 · pp. 3–4

2.2 ECG-RELATED TASKS WITH CLINICAL EXPERTS’ INSIGHTS

  • Clinician-informed tasks cover basic feature recognition, rhythm analysis, morphology and pathology identification, and clinical report generation.The paper describes these categories as spanning foundational ECG parameters through complex conditions and comprehensive interpretation.Source: §2.2 · p. 4

2.3 DIVERSE TYPES OF TASKS AND DATA SOURCES

  • ECGInstruct draws on four datasets from geographically distinct regions and uses multiple-choice, fill-in-the-blank, close-ended, and open-ended question formats.The paper presents geographic diversity as supporting applicability across populations and healthcare systems.Source: §2.3 · p. 4

2.4 DATA SYNTHESIZING AT SCALE

  • To scale instruction creation, the authors use reports from PTB-XL and MIMIC-IV-ECG as seed data with Llama-3-70B-Instruct, and create templates for CODE-15%.The pipeline generates ECG-related instructions and responses from expert examples and real-world scenarios.Source: §2.4 · p. 4

2.5 QUALITY CONTROL

  • An independent Llama 3 judge scores instruction-response pairs, with low-scoring items flagged for expert review, revision, or removal.The process iteratively refines generation using score patterns and human expert input.Source: §2.5 · pp. 4–5

2.6 TRAINING

  • PULSE uses a vision encoder and language-model decoder connected by a projection layer; training freezes the encoder and updates the projection layer and language model.The image begins each conversation, and training uses an autoregressive objective masking image and instruction tokens.Source: §2.6 · p. 5

3 ECGBENCH

ECGBench evaluates ECG image interpretation using repurposed datasets and newly curated tasks from external resources. Its task set spans detection, report generation, multiple-choice interpretation, and open-ended multi-turn interactions.

3 ECGBENCH

  • ECGBench combines tasks repurposed from six existing datasets with newly created tasks from external resources, spanning in-domain and out-of-domain problems.The benchmark covers four key tasks with diverse answer types.Source: §3 · p. 5, Table 2 · p. 6

3.1 EVALUATION TASK CURATION

  • Abnormality detection repurposes six ECG datasets, synthesizing images from raw signals and deriving queries from diagnoses and reports.Datasets with fewer than 10 diagnostic labels use close-ended questions; those with more labels use eight-option multiple-choice questions.Source: §3.1 · p. 6
  • Report generation evaluates 500 randomly selected PTB-XL test reports written and validated by cardiologists, using synthesized ECG images as inputs.Non-English ground-truth reports are translated into English for evaluation.Source: §3.1 · p. 6
  • MMMU ECG contains 200 multiple-choice questions curated from ECG textbooks, clinical cases, and learning materials, with questions and answers checked by multiple annotators.An independent reviewer cross-checks final materials against their sources.Source: §3.1 · p. 6
  • ECG Arena contains 50 open-ended, multi-turn ECG conversations, with follow-up questions contingent on the initial question and response.Annotators created conversations using clinical interpretations and ECG images because multi-turn examples are rare in existing sources.Source: §3.1 · pp. 6–7

3.2 EVALUATION METRICS

  • Evaluation uses task-specific measures: classification metrics or accuracy for abnormality detection, GPT-4o scores for reports and Arena, and accuracy for MMMU ECG.GPT-4o judges reports on rhythm, waveform, and diagnosis, and Arena responses on accuracy, completeness, and instruction adherence.Source: §3.2 · p. 7

4 EXPERIMENTS

Experiments compare PULSE with domain-specific methods and general MLLMs, evaluate it on in- and out-of-domain ECG tasks, and examine training data, instruction tasks, and case studies. PULSE achieves state-of-the-art results, while discussion notes challenges on complex, open-ended tasks.

4.1 METHODS FOR COMPARISON

  • Comparisons include signal- and image-based ECG methods, proprietary and open-source MLLMs; PULSE follows a LLaVA-v1.6-Vicuna-7B architecture with a frozen vision encoder and fine-tuned projector and LLM.The implementation uses anyres to support ECG images of varying sizes.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 MAIN RESULTS

  • PULSE achieves state-of-the-art performance across in-domain datasets, improving over GPT-4o by 27% in AUC on PTB-XL Super, 11 points in report score on PTB-XL Report, and 39% in accuracy on ECG-QA.Table 3 lists in-domain evaluation results; the paper notes that some domain-specific methods use different setups and are included for reference.Source: §4.3 · p. 7, Table 3 · p. 8
  • PULSE improves accuracy on MMMU ECG by 15% over GPT-4o and leads the best proprietary and leading open-source models by 2 and 11 ECG Arena score points, respectively.ECG Arena is described as a more challenging multi-turn, open-ended task; its model-performance gap leaves room for improvement.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 ABLATION STUDY

  • Adding training sources and ECG instruction tasks progressively improves performance across datasets and tasks, with all four instruction tasks achieving the highest performance.A model trained only on PTB-XL performs lowest across datasets, while one trained only on basic feature recognition performs poorly across metrics.Source: §4.4 · p. 9, Table 5 · p. 10, Table 6 · p. 10

4.5 CASE STUDY

  • In case studies, PULSE responses align more closely with ground truths, while GPT-4o can rely on printed text such as axis labels instead of waveform patterns.The example describes GPT-4o identifying left axis deviation from a printed QRS-axis value.Source: §4.5 · p. 10

4.6 DISCUSSION

  • PULSE excels at rhythm interpretation but struggles with waveform and diagnosis identification, and faces challenges on report generation and multi-turn conversations.The discussion suggests broader waveform and diagnosis coverage and step-wise instruction-tuning data as future directions.Source: §4.6 · p. 10, Figure 4 · p. 10

5 CONCLUSION

The paper presents PULSE, fine-tuned on ECGInstruct, and reports state-of-the-art performance across in- and out-of-domain datasets. It situates this work amid limitations of signal-based or single-task ECG methods and focuses on ECG images synthesized from raw signals.

5 CONCLUSION

  • PULSE, fine-tuned on over 1 million ECGInstruct samples, achieves state-of-the-art performance across multiple in- and out-of-domain datasets, surpassing proprietary and open-source MLLMs.The paper describes this as demonstrating the potential of MLLMs for ECG image analysis and interpretation in clinical applications.Source: §5 · p. 11

A RELATED WORK

  • Many prior ECG methods use temporal signals, while image-based report-generation work is limited to that task; this study uses diverse instruction-tuning data to target broader generalizability.The cited image-based report-generation framework uses a BEiT vision encoder and GPT-2 decoder.Source: A RELATED WORK · pp. 15–16

B PRELIMINARY ON 12-LEAD ECG

  • ECGs are typically 12-lead time series, and this paper focuses on images synthesized from raw signals.The six limb leads provide frontal-plane views, while six precordial leads provide horizontal-plane views.Source: B PRELIMINARY ON 12-LEAD ECG · p. 16

C DETAILS OF ECG IMAGE SYNTHESIS

ECG-image-kit is used to synthesize ECG images from raw signals, with distortions and layout variations intended to reflect real-world clinical data. The process also adds printed metadata and balances augmented and standard images.

C DETAILS OF ECG IMAGE SYNTHESIS

  • ECG-image-kit generates ECG images from raw signals under varied conditions, including distortions and noise to simulate clinical data.Augmentations include wrinkles, creases, rotations, resolution and background variation, noise, aspect ratios, image sizes, plot positions, and occasional grid-line removal.Source: C DETAILS OF ECG IMAGE SYNTHESIS · p. 16
  • Synthesized images may include randomly printed patient and ECG metadata, and use lead configurations beyond the standard 4x3 layout.Examples include 12x1 and 6x2 layouts; PTB-XL demographics and features are drawn from PTB-XL+ annotations.Source: C DETAILS OF ECG IMAGE SYNTHESIS · p. 16
  • The augmentation process targets an approximate 1:1 ratio of augmented to standard ECG images.The stated aim is exposure to both clean and distorted images.Source: C DETAILS OF ECG IMAGE SYNTHESIS · p. 16

D DETAILS OF INSTRUCTION TUNING DATASETS

The appendix identifies a table of ECGInstruct data statistics and a figure with examples of basic feature recognition instructions.

D DETAILS OF INSTRUCTION TUNING DATASETS

  • Table A1 presents detailed data statistics for ECGInstruct.Source: Table A1 · p. 17
  • Figure A1 shows examples of basic feature recognition instructions used for PULSE finetuning.Source: Figure A1 · p. 18

Training Data Examples: Heart Rhythm Analysis

Appendix figures provide examples of the ECG instruction types used to finetune PULSE, spanning heart rhythm analysis, morphology and pathology identification, and clinical reporting.

Training Data Examples: Heart Rhythm Analysis

  • Figure A2 presents examples of heart rhythm analysis instructions for finetuning PULSE.Source: Figure A2 · p. 19
  • Figure A3 presents examples of morphology and pathological condition identification instructions for finetuning PULSE.Source: Figure A3 · p. 20
  • Figure A4 presents examples of clinical reporting instructions for finetuning PULSE.Source: Figure A4 · p. 21

Prompt: Multi-task Data Synthesizing

The appendix identifies prompts used to synthesize and revise ECG-related training data and to evaluate generated instructions, reports, and ECG Arena responses.

Prompt: Multi-task Data Synthesizing

  • Figure A5 shows the prompt used to synthesize ECG instruction tasks based on clinical reports.Source: Figure A5 · p. 22

Prompt: Multi-turn Dialogue Synthesizing

  • Figure A6 shows the prompt used to synthesize ECG multi-turn dialogue as instruction-tuning data.Source: Figure A6 · p. 23
  • Figure A7 shows the prompt used to revise and translate original reports.Source: Figure A7 · p. 24

Prompt: Instruction Data Scoring

Prompt: Evaluation of Report Generation

Prompt: Evaluation of ECG Arena

F DETAILS OF EVALUATION METRICS

The evaluation uses task-specific metrics and LLM judges, with separate scoring procedures for abnormality detection, report generation, MMMU ECG, and ECG Arena.

Abnormality Detection

Report Generation

  • GPT-4o evaluates reports against cardiologist-written reports on rhythms, waveform morphology, and diagnosis, averaging three 0–10 component scores and scaling the result to 100.Source: F DETAILS OF EVALUATION METRICS · p. 27

MMMU ECG

ECG Arena

G CASE STUDY

The case study contrasts PULSE’s report for a PTB-XL ECG with GPT-4o’s report, using the clinician’s report as ground truth and judge scores to assess alignment.

Case Study: PTB-XL Report

  • The ground-truth report lists sinus rhythm, prolonged PR interval, left-shifted R-S transition, high limb-lead voltages suggesting LVH, and ST depression with flat T waves in specified leads.Source: G CASE STUDY · p. 28
  • PULSE’s generated report includes a diagnosis of myocardial ischemia alongside LVH, T-wave abnormality, prolonged PR interval, and sinus rhythm.Source: G CASE STUDY · p. 28
  • The judge assigns PULSE a final score of 100, giving 10 each for diagnosis, waveform form, and rhythm.Source: G CASE STUDY · p. 28
  • GPT-4o’s report instead describes a normal ECG, including a normal PR interval and no significant ST-segment or T-wave abnormalities.Source: G CASE STUDY · p. 28
  • PULSE’s report fully aligns with the ground truth and receives a report score of 10, whereas GPT-4’s structurally sound report contains notable inaccuracies.Source: Figure A11 · p. 28

Case Study: PTB-XL Report

The cases show PULSE matching many report findings while omitting some details; general MLLM outputs also contain substantial interpretation errors.

Case Study: PTB-XL Report

  • 83.3 report score: PULSE identified sinus rhythm and ischemic ST/T-wave abnormalities but omitted specific T-wave changes; GPT-4 identified only the rhythm.The ground truth specifies low or flat T waves in V3–V5 and slight inversion in V2.Source: Case Study: PTB-XL Report · p. 29, Figure A12 · p. 29
  • 73 report score: PULSE identified sinus arrhythmia and major conduction blocks but omitted abnormal QRS(T) and possible lateral damage; GPT-4o’s report only partially matched ground truth.The ground truth lists left axis deviation, left anterior hemiblock, right bundle branch block, and bifascicular block.Source: Case Study: PTB-XL Report · p. 30, Figure A13 · p. 30
  • 56.7 judge score: PULSE associated qR/rS complexes with LAFB but generalized the explanation, added unsupported LVH, and omitted the specific leads and left axis deviation.The ground truth identifies qR complexes in I and aVL, rS complexes in II, III, and aVF, and left axis deviation.Source: Case Study: PTB-XL Report · p. 31
  • On a challenging ECG Arena example, PULSE’s output partially aligned with the reference, while GPT-4o’s largely deviated from it.GPT-4o interpreted qR complexes in V1/V2 and rS complexes in V5/V6 as right bundle branch block.Source: Figure A14 · p. 31, Case Study: PTB-XL Report · p. 31

Related papers

Loading 2410.19008v1…