Source-linked AI summary
ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography
Mohammadsina Hassannia, Matthew A. Reyna, Reza Sameni
TL;DR
ECG interpretation demands contextual knowledge that existing medical benchmarks do not fully assess. ECGQuest provides a source-grounded ECG benchmark and evaluates zero-shot and LoRA fine-tuning strategies. Fine-tuning made compact open models competitive with larger proprietary systems, although external generalization was uneven and labels remain weakly validated.
Problem
Existing language-model benchmarks mainly assess broad medical knowledge or individual ECG signals and images, not the broader contextual knowledge required for ECG interpretation.
Method
The study constructs a literature-grounded, source-linked True/False ECG benchmark and evaluates language models zero-shot, externally, and after LoRA fine-tuning.
Results
Fine-tuning improved compact open models by 6.5–14.1 percentage points on ECGQuest; DeepSeek-R1-14 B reached 76.3% accuracy and the ensemble 78.5%, while external gains were uneven.
Takeaways & Limitations
Parameter-efficient fine-tuning can make 7–14 B open models competitive with substantially larger proprietary models on ECGQuest.
Takeaways & Limitations
ECGQuest labels were generated by GPT-4o and lack expert over-read, so they should be treated as weak labels rather than clinical ground truth.
Abstract
from arXiv · showhide
Electrocardiogram (ECG) interpretation requires knowledge of cardiology, electrophysiology, clinical diagnosis, ECG waveforms, signal acquisition, and instrumentation. Existing language-model benchmarks, however, primarily assess broad medical knowledge or interpretation of individual ECG signals and images rather than the broader contextual knowledge required for ECG interpretation. We developed ECGQuest, a literature-grounded resource for evaluating and fine-tuning ECG-specific language models. A GPT-4o-based pipeline generated questions from 23 ECG references and Computing in Cardiology proceedings from 2003-2025. The final dataset contains 10,904 unique True/False questions paired with their negated forms (21,808 Q&A pairs). We evaluated three commercial and 20 open-source language models on a held-out test set in a zero-shot setting. Five open-source models with 7-14B parameters were fine-tuned using Low-Rank Adaptation, with BERT and BiomedBERT included as supervised encoder baselines. Generalization was assessed on ECG-related subsets of MedMCQA and MedQA converted to binary True/False questions using official answer keys. Zero-shot accuracy on ECGQuest ranged from 49.5% to 74.4%, with GPT-5 performing best. General-purpose models outperformed medically specialized models, several models showed strong True/False bias, and encoder baselines performed near chance. Fine-tuning improved all open-source models by 6.5-14.1%. Fine-tuned DeepSeek-R1-Distill-Qwen-14B reached 76.3% accuracy, while a five-model voting ensemble reached 78.5%. On MedMCQA and MedQA, fine-tuning mainly benefited weaker or class-biased models and did not consistently improve strong base models. ECGQuest provides a reproducible benchmark for contextual ECG knowledge and shows that parameter-efficient fine-tuning can make smaller language models competitive with substantially larger commercial models.
1 INTRODUCTION
ECG interpretation is widely used but difficult, requiring broad technical and clinical knowledge beyond simple classification. Variability among annotators further motivates reliable computational support.
- ECGs are routinely used to detect arrhythmias, ischemia, conduction abnormalities, and other cardiac conditions across clinical settings.
- Accurate interpretation requires substantial training and experience in cardiac electrophysiology.
- ECG interpretation combines electrophysiology, acquisition and instrumentation, waveform measurements, and clinical context.
- Experienced annotators show inter-rater and intra-rater variability, contributing to missed diagnoses, inappropriate interventions, and delayed treatments.
2 EXISTING MEDICAL Q&A AND DOMAIN-SPECIFIC LANGUAGE MODELS
Existing medical benchmarks evaluate broad clinical knowledge, while ECG interpretation requires more specialized contextual knowledge. ECGQuest addresses this gap through a dedicated ECG-focused evaluation framework.
- Medical Q&A benchmarks such as MedQA, MedMCQA, and PubMedQA primarily assess broad clinical knowledge rather than ECG knowledge.
- ECGQuest evaluates closed-source, open-source, and medically specialized models on ECG-focused questions and external exam-derived questions.
- Prior ECG language-model work has integrated signal representations into latent spaces and evaluated more than 800,000 reports.
3 METHODS
The study constructs a source-linked, balanced ECG question benchmark, evaluates models in zero-shot and external settings, and fine-tunes compact open models with LoRA.
- The study constructs ECGQuest from ECG literature, evaluates diverse models on held-out and external ECG questions, and fine-tunes compact open models with LoRA.
- 3.1 ECGQUEST DATASET CONSTRUCTION: The automated dataset pipeline assembles and extracts a corpus, generates questions, balances labels through paired negation, and validates and deduplicates items.
- 3.1 ECGQUEST DATASET CONSTRUCTION: Each generated item is linked to its source, making the reference text the basis for question correctness.
- 3.1 ECGQUEST DATASET CONSTRUCTION: Source PDFs were anonymized with MD5-based identifiers and split programmatically into single-page files for text extraction.
- 3.1 ECGQUEST DATASET CONSTRUCTION: Paired counterparts retain the source text, flip the label, and are discarded when verification criteria fail.
- 3.2 MODEL FINE-TUNING: Training, validation, and test splits preserve both members of each True/False pair within the same split.
4 RESULTS
Zero-shot performance varied widely, with larger same-family models generally improving and medical specialization not guaranteeing stronger ECG performance. Fine-tuning substantially improved compact models, while agreement analyses tracked benchmark alignment rather than clinical correctness.
- 4.1 ZERO-SHOT BENCHMARKING: 49.5%–74.4%: zero-shot ECGQuest accuracy, with GPT-5 highest at 74.4%.
- 4.1 ZERO-SHOT BENCHMARKING: Within model families, accuracy generally increased with parameter count, but medical specialization did not necessarily improve ECG performance.
- 4.2 FINE-TUNED MODEL PERFORMANCES: 6.5–14.1 percentage points: fine-tuning gains across compact open models, enabling several 7–14 B models to match or exceed leading commercial models.
- 4.2 FINE-TUNED MODEL PERFORMANCES: 76.3%: fine-tuned DeepSeek-R1-14 B accuracy, exceeding GPT-5’s 74.4%; the five-model voting ensemble reached 78.5%.
- 4.2 FINE-TUNED MODEL PERFORMANCES: 52.1% and 54.0%: fine-tuned BERT and BiomedBERT accuracy, respectively, both near the 50% chance level.
- 4.3 AGREEMENT WITH BENCHMARK LABELS: Cohen’s κ agreement generally followed accuracy, with fine-tuning producing the largest gains for initially weak models.
- 4.3 AGREEMENT WITH BENCHMARK LABELS: Agreement values quantify alignment with benchmark labels rather than clinical correctness.
5 DISCUSSION AND FUTURE WORK
ECG-specific knowledge varies across models, and generic medical specialization does not guarantee ECG competence. External evaluation suggests fine-tuning benefits weaker or biased models but can reduce accuracy for balanced models, while weak labels and possible leakage constrain interpretation.
- ECG knowledge varies across models, and medically specialized models can underperform larger general-purpose models.Within families, larger models generally perform better, but pretraining data and base-model quality also matter.
- MedMCQA and MedQA scores may partly reflect training-data leakage rather than ECG understanding.These long-standing public resources may have entered existing models’ training corpora.
- Future work: ECGQuest labels are weak labels because GPT-4o generated them from source materials without expert over-read.The authors propose expert review to identify ambiguity, correct inaccurate labels, and establish an expert-validated benchmark.
6 CONCLUSION
ECGQuest is a source-linked, balanced benchmark for conceptual ECG knowledge and model fine-tuning. Its results show substantial model variation, generally improved within-family performance with size, and that LoRA can make smaller models competitive, especially on the benchmark itself.
- ECGQuest is a literature-grounded balanced True/False dataset for evaluating and fine-tuning models on conceptual ECG knowledge.Each item is source-linked and paired with a minimally modified opposite-label counterpart.
- ECG knowledge varies substantially across models and is not guaranteed by generic medical specialization.The conclusion identifies this as one of the study’s three main findings.
- Within model families, performance generally improves with increasing model size.This is reported as a general pattern across the evaluated models.
- Parameter-efficient fine-tuning enables relatively small 7–14 B models to match or outperform substantially larger proprietary models on ECGQuest.External exam-labeled ECG questions show that this benefit is largest for weak or class-biased base models.
A ECGQUEST QUESTION COMPOSITION AND LEVEL OF DIFFICULTY
ECGQuest organizes question pairs across ECG knowledge categories and examines both question-level difficulty and category-level model performance. Difficulty varies widely, and different models lead in different categories rather than one model dominating all areas.
- Dataset Composition: ECGQuest question pairs are distributed across dataset splits and four ECG knowledge categories.The categories are ECG Diagnosis, ECG Acquisition Technologies, ECG Characteristics, and Basic Cardiac Physiology.
- Question-Level Difficulty Profile: Approximately 73% of the 1,050 test questions were answered correctly by at least half of the evaluated models.Questions were ordered from easiest or clearest to hardest or most ambiguous based on model agreement.
- Question-Level Difficulty Profile: Question difficulty and clarity vary substantially, from items answered correctly by nearly all models to a small subset answered correctly by almost none.The ranked profile shows a gradual decline in the fraction of models answering correctly.
- Category-Level Performance: No single model achieved the best performance across all four ECG knowledge categories.Category leaders included fine-tuned DeepSeek-R1-Distill-Qwen 14 B in ECG Diagnosis and ECG Characteristics, GPT-4o in ECG Acquisition Technologies, and fine-tuned Qwen2.5-Instruct 14 B in Basic Cardiac Physiology.
- Category-Level Performance: 90.7% accuracy was achieved by fine-tuned Qwen2.5-Instruct 14 B in Basic Cardiac Physiology, the strongest reported category-level result.Fine-tuned DeepSeek-R1-Distill-Qwen 14 B followed with 88.9% in that category.
B EXTERNAL EXAM-LABELED EVALUATION
External ECG questions from MedMCQA and MedQA were converted into exam-key-labeled True/False sets to test generalization beyond ECGQuest. Fine-tuning helped weaker or biased models but reduced performance for stronger base models.
- Dataset preparation: 2,084 MedMCQA and 294 MedQA True/False questions were produced using official examination keys.False answers were randomly selected from the three incorrect multiple-choice options before conversion.
- Zero-shot results: GPT-5 led external-set accuracy at 86.5% on MedMCQA and 93.2% on MedQA, followed by GPT-4o at 83.7% and 89.8%.
- Fine-tuned results: Fine-tuning improved BioMistral-7 B accuracy from 51.3% to 60.5% on MedMCQA and from 50.7% to 58.5% on MedQA.Macro-F1 roughly doubled from 36.7 to 60.3 and from 35.4 to 58.4 by correcting True-class bias.
- Fine-tuned results: Fine-tuning improved Qwen2.5-Instruct-14 B on MedMCQA but left it close to its starting point on MedQA.MedMCQA accuracy rose from 67.3% to 73.0%, whereas MedQA accuracy changed from 66.3% to 64.9%.
- Fine-tuned results: For already strong Llama-3.1-8 B and DeepSeek-R1-14 B models, fine-tuning reduced accuracy on both external datasets.The authors relate strong zero-shot scores on these public resources partly to possible pretraining-data leakage.
C QUESTION-GENERATION, VALIDATION, AND LABEL-BALANCING PROMPTS
The prompts constrain ECG True/False question generation to source-grounded content, permitted ECG topics, and self-contained statements. Separate validation and label-balancing procedures reject unsupported items and create minimally edited opposite-label counterparts.
- Question generation: GPT-4o generates ECG questions from extracted text or rendered page images, using only explicitly written source content.The vision workflow forbids visual inference from waveform shapes and requires exact text from captions, tables, labels, or annotations.
- Question generation: True questions restate source facts, while false questions alter only one supported detail without introducing outside knowledge.The prompts require verbatim source text and direct transformations, with examples changing the normal PR upper bound from 200 ms to 400 ms.
- Validation: Candidate items are discarded when source support, label determinability, self-containment, or other validation checks fail.Representative rejections include an unsupported cardioversion recommendation and a question requiring interpretation of Figure 4.7.
- Question generation: The allowed topics cover ECG waveforms, clinical interpretation, physiological mechanisms, and device technical specifications.Examples include P, QRS, ST, T, and U waves; diagnostic patterns; conduction and repolarization; and leads, filters, calibration, and artifacts.
- Label balancing: A label-balancing prompt generates the opposite-label counterpart from the same unchanged source text using minimal edits or faithful restatement.Counterparts must differ from the original, flip the label correctly, remain self-contained, and avoid references to images, figures, strips, or diagrams.