PaperlayerView exact revision on arXiv

Source-linked AI summary

ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis

Yubao Zhao, Jiaju Kang, Tian Zhang, Puyu Han, Tong Chen

arXiv:2408.08849v2eess.SP

TL;DR

ECG-language models face a gap between ECG signals and text reports, alongside unreliable cardiology outputs from hallucinating general LLMs. ECG-Chat combines contrastive waveform-report alignment with instruction-tuning data and diagnosis-driven prompting; the authors report state-of-the-art ECG medical text generation and validation of its encoder in retrieval and classification tasks.

  • Problem

    ECG-language models have not bridged ECG signals and text reports, while general LLMs can hallucinate in cardiology, limiting reliable ECG report generation and question answering.

    Source: §I · p. 1

  • Method

    ECG-Chat combines contrastive ECG-report alignment with instruction-tuning data and diagnosis-driven prompting for report generation and ECG question answering.

    Source: §I · p. 1

  • Results

    ECG-Chat achieves state-of-the-art ECG medical text generation, with WDE validated through ECG-text retrieval and classification tasks.

    Source: §VI · p. 5

  • Takeaways & Limitations

    The authors highlight multi-stage information fusion and the potential of large models in medical scenarios.

    Source: §VI · p. 5

    The instruction-tuning dataset is small and not drawn from the real world, which the authors say leads to bias and hallucinations.

    Source: §VI · p. 5

Abstract

from arXiv · show

The success of Multimodal Large Language Models (MLLMs) in the medical auxiliary field shows great potential, allowing patients to engage in conversations using physiological signal data. However, general MLLMs perform poorly in cardiac disease diagnosis, particularly in the integration of ECG data analysis and medical report generation, mainly due to the complexity of ECG data analysis and the gap between text and ECG signal modalities. To address these issues, we propose ECG-Chat, a multitask MLLMs focused on ECG medical report generation, providing multimodal conversational capabilities based on cardiology knowledge. We propose a contrastive learning approach that integrates ECG waveform data with text reports, aligning ECG features with reports in a fine-grained manner. This method also results in an ECG encoder that excels in zero-shot report retrieval tasks. Additionally, expanding existing datasets, we constructed a 19k ECG diagnosis dataset and a 25k multi-turn dialogue dataset for training and fine-tuning ECG-Chat, which provides professional diagnostic and conversational capabilities. Furthermore, ECG-Chat can generate comprehensive ECG analysis reports through an automated LaTeX generation pipeline. We established a benchmark for the ECG report generation task and tested our model on multiple baselines. ECG-Chat achieved the best performance in classification, retrieval, and medical report generation tasks. Our code is available at https://github.com/YubaoZhao/ECG-Chat.

I. INTRODUCTION

The introduction frames ECG-language diagnosis as limited by the gap between ECG signals and text reports, as well as hallucinations in cardiology. It presents ECG-Chat and ECG-Instruct as a response, combining waveform-report alignment, instruction-tuning data, and report-generation components.

I. INTRODUCTION

  • Existing ECG algorithms focus mainly on classification, while ECG-language models have not bridged signal-report modalities and general LLMs hallucinate in cardiology.The introduction says these gaps leave report generation and question answering inadequately addressed.Source: §I · p. 1
  • Contrastive waveform-report alignment is introduced to improve ECG encoder performance in retrieval tasks.The contribution is described as fine-grained alignment of ECG waveform data with text reports.Source: §I · p. 1
  • ECG-Instruct provides 19K diagnosis and 25K dialogue examples for instruction tuning.The dataset pipeline uses GPT-4 to generate the instruction-tuning data.Source: §I · p. 1
  • ECG-Chat fine-tunes Vicuna-13B with LoRA and uses diagnosis-driven prompting and automated LaTeX vocabulary explanations for ECG report generation.The system also supports ECG question answering and conversational use.Source: §I · p. 1

A. Cross-modal Medical Text Generation

  • Research on signal-text multimodality, particularly for ECGs, remains underexplored despite progress in image-text medical applications.Cross-modal medical text generation uses modalities such as images, text, and signals to produce medically informed diagnostic or explanatory text.Source: A. Cross-modal Medical Text Generation · pp. 1–2

B. Intelligent Interpretation of ECG

ECG interpretation remains less developed for diagnostic text generation than for retrieval and classification, with clinical relevance and interpretability still challenging. ECG-Chat combines ECG-text alignment, report generation, and multi-round conversation in an end-to-end pipeline.

B. Intelligent Interpretation of ECG

  • ECG interpretation includes retrieval and classification, but descriptive diagnostic text generation remains nascent, with clinical relevance and interpretability unresolved challenges.Complex cardiac conditions are another area requiring further exploration.Source: B. Intelligent Interpretation of ECG · p. 2

A. Architecture and General Pipeline

  • ECG-Chat encodes ECG signals, aligns their features with reports, adapts them to an LLM, and adds diagnosis-driven prompting and GraphRAG for report generation and multi-round conversations.The system also integrates automated prompt tuning and GraphRAG into long-form report generation, with the approach extending to other chat functions.Source: A. Architecture and General Pipeline · p. 2, Figure 2 · p. 3

B. Aligning ECG Features with Text Reports

  • ECG-text alignment uses contrastive learning in both ECG-to-text and text-to-ECG directions, with normalized paired embeddings and temperature-scaled logits.The text encoder is pretrained and only its last few layers are fine-tuned.Source: B. Aligning ECG Features with Text… · p. 2
  • The training objective combines contrastive alignment with autoregressive captioning, while waveform data appended to reports distinguishes otherwise identical reports and adds waveform information to ECG representations.The captioning loss trains ECG features to predict tokenized text; the two losses are weighted by hyperparameters.Source: B. Aligning ECG Features with Text… · pp. 2–3

C. Multimodal Instruction Turning and Inference

ECG-Chat maps ECG features into an LLM through a two-layer adapter and trains in feature-alignment and instruction-fine-tuning stages. Diagnosis-Driven Prompting supplies thresholded predicted labels to guide report generation.

C. Multimodal Instruction Turning and Inference

  • A two-layer MLP adapter maps ECG features into the LLM; training freezes the ECG encoder, first trains only the projection layer, then fine-tunes the LLM with LoRA.The stages are feature alignment and instruction fine-tuning.Source: C. Multimodal Instruction Turning and… · p. 3
  • Diagnosis-Driven Prompting inserts predicted labels into the prompt, grouping multi-label predictions as disease, rhythm, and waveform categories and including labels above a threshold.Rhythm is treated as single-label; example prompt labels include sinus rhythm and left bundle branch block.Source: C. Multimodal Instruction Turning and… · p. 3

D. Hallucination Elimination and Visualization Output

The paper combines a textbook-derived cardiology knowledge graph, automated prompt tuning, and LaTeX report generation. Its datasets cover ECG-text pretraining, diagnosis and dialogue instruction tuning, and evaluation across retrieval, classification, and report generation.

D. Hallucination Elimination and Visualization Output

  • GraphRAG builds a knowledge graph from seven cardiology textbooks, DSPy automates prompt tuning with patient data and retrieved knowledge, and a LaTeX pipeline produces structured clinical reports.The authors describe GraphRAG as intended to mitigate hallucinations and ground generated text in established medical knowledge.Source: D. Hallucination Elimination and… · p. 3

A. Datasets for Contrastive Learning

  • ECG encoder pretraining combines MIMIC-IV-ECG, CSN, and SPH into an 805K-record dataset, with lead-II waveform measurements appended to reports.Extracted measurements include intervals, QRS duration, and P, R, and T wave peaks.Source: A. Datasets for Contrastive Learning · pp. 3–4

B. Datasets for Instrution Tuning

  • Instruction tuning uses MIMIC-IV-ECG records for 19K diagnosis samples and 25K conversation samples, with conversations spanning 4–15 rounds.The pretraining set contains 619K records with GPT-4-generated user instructions and reports as answers.Source: B. Datasets for Instrution Tuning · p. 4

C. Datasets for Evaluations

  • Evaluation covers ECG-report retrieval, zero-shot and linear-probe classification, and report generation on PTB-XL and CPSC2018; GraphRAG and DSPy ablations use ECG-ExpertQA.The passage does not report comparative evaluation outcomes.Source: C. Datasets for Evaluations · p. 4

V. EXPERIMENT

The experiments assess ECG-Chat’s ECG encoder on zero-shot ECG-report retrieval and describe the model’s training setup. The retrieval comparison reports the strongest results for CoCa with Waveform Data Enhancement (WDE).

A. Implementation Details

  • The ECG encoder uses a 12-layer 1D-ViT and a fine-tuned Med-CPT text encoder; ECG-Chat trains a projection layer and fine-tunes Vicuna-13B with LoRA.Encoder training lasts 20 epochs; ECG-Chat uses one projection-only pretraining epoch and three fine-tuning epochs, with experiments run on 8×V100 32GB GPUs.Source: A. Implementation Details · p. 4

B. Zero-shot ECG-Report Retrieval

C. ECG Reports Generation

ECG-Chat is evaluated on ECG report generation using clinical-efficacy and language-generation metrics. The authors report strong DDP results while noting data-diversity, rare-diagnosis, and modality limitations.

C. ECG Reports Generation

  • Evaluation uses PTB-XL test records, with clinical efficacy measured by GPT-4-annotated precision, recall, and F1, and NLG by BLEU, METEOR, and ROUGE-L.The NLG references are GPT-4o answers based on translated PTB-XL reports.Source: C. ECG Reports Generation · p. 5
  • ECG-Chat with DDP achieves the best results across all reported NLG indicators.The comparison is against translated original PTB-XL reports.Source: C. ECG Reports Generation · p. 5
  • Normal and sinus-rhythm labels have high F1 scores, but uncommon labels can score zero; without DDP, outputs favor a few responses.The authors nevertheless characterize the generated ECG interpretation reports as relatively accurate overall.Source: C. ECG Reports Generation · p. 5

VI. CONCLUSION AND LIMITATIONS

  • The authors attribute improved generation to WDE and DDP, with GraphRAG and DSPy used to address medical-text hallucinations.They describe WDE as enhancing the encoder and DDP as correcting report-generation errors using classifier information.Source: §VI · p. 5
  • The authors cite limited training-data diversity, small non-real-world instruction data, weak performance on rare symptoms, and ECG-only scope as limitations.They propose combining ECG with modalities such as X-Ray and EHR for more comprehensive medical-record interpretation.Source: §VI · pp. 5–6

APPENDIX A RAG OF CARDIOLOGY KNOWLEDGE

The appendix describes three report-support components: cardiology knowledge retrieval, automatic prompt tuning, and structured report generation.

APPENDIX A RAG OF CARDIOLOGY KNOWLEDGE

  • GraphRAG indexes seven cardiology textbooks as a knowledge graph, retrieves relevant elements for global answers, and is reported effective in ablation experiments.The authors describe it as a way to mitigate hallucinations in ECG diagnostic reports.Source: Appendix A · p. 8

APPENDIX B AUTOMATED PROMPT TUNING

  • DSPy is trained on ECG-Instruct and combines GraphRAG knowledge with patient data to generate diagnostic text while optimizing prompts and model weights.Its automated, modular workflow is intended to reduce manual tuning and improve scalability and maintainability.Source: Appendix B · p. 8

APPENDIX C LATEX-BASED AUTOMATED MEDICAL REPORT GENERATION PIPELINE

  • The plasTeX pipeline structures reports into patient information, medical history, ECG analysis, pathological analysis, diagnosis, and recommendations.The appendix points to Figures 5 and 6 for template examples.Source: Appendix C · p. 8

APPENDIX D DESCRIPTION OF DATASETS

The appendix describes ECG datasets used for training and evaluation, including clinical records paired with reports or diagnostic labels. It also outlines construction of the pretraining and instruction-tuning data.

APPENDIX D DESCRIPTION OF DATASETS

  • After preprocessing MIMIC-IV-ECG, the authors use 788,822 samples for training; the feature-alignment pretraining description specifies a 619K subset.The MIMIC-IV-ECG records are 10-second, 500 Hz recordings paired with machine-generated reports.Source: Appendix D · pp. 8–9
  • The CSN dataset contributes 40,637 training samples from 45,152 records, with SNOMED CT codes converted to textual reports.The SPH database contributes 20,616 training samples; the first 10 seconds of each 10–60-second record are used.Source: Appendix D · pp. 8–9
  • PTB-XL contains 21,837 ten-second ECG signals with free-text reports and SCP-code labels, while CPSC2018 has 6,877 standard 12-lead ECG records annotated with nine labels.CPSC2018 recordings longer than 60 seconds are truncated to 10 seconds; those shorter than 10 seconds are filled with zeros.Source: Appendix D · p. 9
  • ECG-Instruct contains 19K diagnosis question-answer samples and 25K multi-turn dialogue samples, with each dialogue spanning 4–15 rounds.Dialogue prompts include waveform features; 19K of the 25K samples come from the diagnosis dataset.Source: Appendix D · pp. 9–10

APPENDIX E DATASET STATISTICS AND ANALYSIS ON ECG-INSTRUCT

ECG-Instruct includes diagnosis and multi-turn conversation components, with statistics describing the scale and textual detail of each. The figures visualize probability-density distributions for the full dataset and its components.

APPENDIX E DATASET STATISTICS AND ANALYSIS ON ECG-INSTRUCT

  • 45,044 ECG samples contain 325,730 sentences, averaging 7.23 sentences per description; descriptions average 73.15 words and use 6,319 distinct words.The dataset contains 3,294,880 words overall.Source: Appendix E · p. 10, Table III · p. 10
  • 20,258 diagnosis question-answer samples are single-turn exchanges; captions average 66.77 words and 4.10 sentences, with lengths ranging from 11 to 303 words.Figure 8 visualizes caption-length, sentence-count, and vocabulary-frequency distributions.Source: Appendix E · p. 10, Figure 8 · p. 10
  • 24,786 conversation samples contain 4–30 dialogue rounds on ECG topics including heart rate, waveform, rhythm, cardiac axis, and diagnostic results.Conversations average 78.36 words and 9.79 sentences; Figure 9 presents their length, sentence-count, and vocabulary-frequency distributions.Source: Appendix E · pp. 10–11, Figure 9 · p. 10

APPENDIX F ABLATION STUDY

The ablations report gains from mixed-dataset training and increased training data, different best text encoders by task, and improved ECG-ExpertQA metrics with GraphRAG and DSPy together.

Datasets

  • Mixed-dataset encoder training effectively enhances generalization compared with training on MIMIC-IV-ECG alone.The comparison is reported in Table IV.Source: Appendix F · p. 11, Table IV · p. 11

Text Encoder

  • BioLinkBert achieves the best retrieval results, while Med-CPT achieves the best classification results among the three tested text encoders.Table V compares Med-CPT with BioLinkBert and BioClinicalBert.Source: Appendix F · p. 11, Table V · p. 11

Scalability

  • Increasing training-data volume significantly improves model performance, especially retrieval, while changing encoder size from 43M to 128M parameters has little impact.The scalability analysis compares 10%, 50%, and 100% data settings and parameter counts.Source: Appendix F · p. 11, Table VI · p. 11

Effectiveness of GraphRAG and DSPy

  • GraphRAG and DSPy together significantly outperform the other tested combinations across all metrics; GraphRAG improves Faithfulness to 76.60 and Answer Relevancy to 68.29.DSPy improves Context Recall to 27.57 and Summarization Score to 47.72; without either module, Faithfulness is 39.87 and Answer Relevancy is 32.56.Source: Appendix F · pp. 11–12, Table VII · p. 12

APPENDIX G CASE STUDY

The case study compares ECG-Chat reports with translated PTB-XL reports and describes the generated text as fluent, colloquial, and relatively accurate.

APPENDIX G CASE STUDY

  • ECG-Chat’s reports are described as more fluent and colloquial than translated PTB-XL reports, with relatively good diagnostic accuracy.Figure 10 shows examples of the generated and translated reports.Source: Appendix G · p. 12
Loading 2408.08849v2…