ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography

• ECGs are widely used across care settings, yet interpretation requires substantial training, and annotator variability can cause missed diagnoses, inappropriate interventions, and delayed treatment.Source: §1 · pp. 1–2
• ECG interpretation combines electrophysiology, acquisition technologies, waveform measurements, descriptive diagnosis, and patient context rather than only classifying waveform patterns.Source: §1 · p. 2
• The authors constructed a text-only database of 10,904 True/False question pairs (21,808 questions in total) from 23 ECG resources and specialized conference proceedings.The questions span clinical, electrophysiological, biomedical signal-processing, diagnostic, and ECG hardware and electronics topics.Source: §1 · p. 2
• They evaluated proprietary, general-purpose, biomedical, and open-access language models on ECGQuest in a zero-shot setting.Source: §1 · p. 2
• They explored low-rank adaptation to specialize smaller language models with 7 to 14 billion parameters for ECG knowledge.Source: §1 · p. 2
• They tested generalization on ECG-related MedQA (USMLE style) (Jin et al., 2021) and MedMCQA (Pal et al., 2022) sets labeled using official examination keys independent of their generation pipeline.Source: §1 · p. 2

Existing medical Q&A and domain-specific language models

• MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), and PubMedQA (Jin et al., 2019) assess broad clinical knowledge rather than ECG concepts.General-purpose and domain-adapted models have achieved strong, sometimes expert-level accuracy on these benchmarks through prompting, instruction tuning, and parameter-efficient fine-tuning; Health-LLM focuses on wearable-sensor time series rather than conceptual ECG knowledge.Source: §2 · p. 2
• ECG-QA (Oh et al., 2023) evaluates text-translated signal-model outputs using expert-validated question templates, and Q-HEART (Pham et al., 2025) achieved state-of-the-art performance on ECG-QA.ECG-QA pairs PTB-XL and MIMIC-IV data with 70 verify, choose, and query templates; Q-HEART combines ECG representations, retrieved clinical context, and instruction tuning.Source: §2 · p. 2
• ECGInstruct (Liu et al., 2024) compiled over one million ECG-image instruction-tuning samples to train PULSE, evaluated on ECGBench across four interpretation tasks.The tasks cover feature recognition, rhythm analysis, morphology assessment, and report generation.Source: §2 · p. 2
• MEIT (Wan et al., 2025) instruction-tuned nine language models for ECG report generation by integrating signal representations into the model’s latent space.MEIT benchmarked the models on more than 800,000 reports.Source: §2 · pp. 2–3

3 METHODS

• The study evaluated closed-source, open-source, and medically specialized models on a held-out test set, checked generalization on two external exam-labeled ECG sets, and fine-tuned compact open-source models with LoRA (Hu et al., 2022).Fine-tuning used an approximately 80:10:10 split with paired questions assigned together; five generative models and BERT and BiomedBERT encoder baselines were evaluated.Source: §3 · p. 3, §3.2 · p. 5

3.1 ECGQUEST DATASET CONSTRUCTION

• GPT-4o generated factual questions from extracted text, rendered page images, or both, selecting among these modes by page character count and covering four ECG knowledge topics.Pages with at least 400 characters used text; pages with 100–399 used text and image; pages below 100 used vision alone.Source: §3.1 · p. 4
• Automated verification rejected 638 responses for unsupported, image-referential, malformed, or ambiguously labeled questions, while preserving each retained item’s source-page provenance.Text- and image-conditioned outputs were deduplicated for pages processed in both modes.Source: §3.1 · p. 4
• ECGQuest contains 10,904 unique question pairs (21,808 True/False questions) with balanced labels, assembled from 23 ECG reference documents spanning clinical, physiological, waveform, acquisition, and device knowledge.Pairs were deduplicated across documents; the largest category was ECG diagnosis, with 6,735 pairs.Source: §3.1 · pp. 3, 5
• Each validated question was paired with a minimally edited opposite-label counterpart derived from the same source text, preventing label shortcuts; final deduplication used a source-text Jaccard threshold of 0.90.Counterparts were programmatically checked for unchanged source text and correctly flipped labels.Source: §3.1 · pp. 4–5

3.2 MODEL FINE-TUNING

• Five instruction-based models were adapted with LoRA on attention and MLP projections, while BERT and BiomedBERT were fine-tuned as two-class classifiers on the same training split.Class-weighted loss was used to reduce each base model’s tendency to overpredict True or False.Source: §3.2 · p. 5

4.1 ZERO-SHOT BENCHMARKING OF LLMS AND SLMS ON ECGQUEST

• 74.4% accuracy was the zero-shot high, achieved by GPT-5; Gemma-4-31B led open-source models at 71.0%.Zero-shot accuracy ranged from 49.5% to 74.4%; response bias affected several models, while accuracy rose with parameter count within model families. Medical specialization did not necessarily improve results: eight general-purpose open-source models outperformed the best medical model, MedGemma-4B, at 59.5%.Source: §4.1 · pp. 5–6

4.2 FINE-TUNED MODEL PERFORMANCES

• Fine-tuning improved some weaker or class-biased models on exam-derived ECG questions, but reduced accuracy for already balanced Llama-3.1-Instruct-8B and DeepSeek-R1-14B.On MedMCQA, BioMistral-7B improved from 51.3% to 60.5% and Qwen2.5-Instruct-14B from 67.3% to 73.0%; fine-tuned BERT and BiomedBERT remained near chance at 52.1% and 54.0%.Source: §4.2 · p. 6
• 78.5% accuracy (macro-F1 78.4) was the best ECGQuest result, achieved by majority voting across five fine-tuned models.DeepSeek-R1-14B reached 76.3% accuracy (macro-F1 76.1), exceeding GPT-5 at 74.4%; Qwen2.5-14B and Gemma-4-12B also surpassed GPT-4o, while Llama-3.1-8B matched it. Fine-tuning gains across the five open-source models ranged from 6.5 to 14.1 percentage points.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 AGREEMENT WITH BENCHMARK LABELS

• Approximately 73% of test items were answered correctly by at least half of evaluated models, while fine-tuned models led in most ECG knowledge categories.No single model led across all four categories.Source: §4.3 · p. 7
• κ = 0.49 was the highest zero-shot agreement with benchmark labels, achieved by GPT-5; agreement rose after fine-tuning across all five adapted models.Fine-tuned DeepSeek-R1-14B matched GPT-5 at κ = 0.49, and the fine-tuned models agreed more with one another (κ = 0.42 to 0.59) than with their base models. These κ values measure agreement with benchmark labels, not clinical correctness.Source: §4.3 · pp. 6–7, Figure 2 · p. 8

4.4 PARETO FRONTIERS ON RECEIVER OPERATING CHARACTERISTICS AND PRECISION RECALL CURVES

• J increased after fine-tuning in every evaluated model, and the ensemble reached J=0.57 versus GPT-5’s 0.49.For Qwen2.5-Instruct-14B, J rose from 0.211 to 0.480 as recall increased from 27.5 to 75.0 with essentially unchanged specificity; for BioMistral-7B, J rose from 0.09 to 0.38 as specificity increased from 18.7 to 67.1.Source: §4.4 · p. 8, Figure 3 · p. 9

5 DISCUSSION AND FUTURE WORK

• Performance generally improved with model size within Gemma families, but across families pretraining data and base-model quality mattered, and medically tuned models underperformed larger general-purpose models.MedGemma-4 B was outperformed by Gemma-4-4 B at the same parameter size, suggesting medical-specific tuning does not necessarily provide ECG-specific knowledge.Source: §5 · p. 9
• All five adapted models outperformed encoder baselines and matched or exceeded GPT-4o; DeepSeek-R1-14 B and the five-model majority vote also exceeded GPT-5.The discussion notes that this shows ECG knowledge can be transferred into a compact model and that adaptation mitigated initial class biases.Source: §5 · p. 9
• On balanced True/False data, accuracy can conceal class imbalance, so macro-F1 and class-specific results are needed to identify asymmetric performance.A model always predicting one class can achieve 50% accuracy but only 33.3% macro-F1; a large accuracy–macro-F1 gap signals asymmetry, though class-specific results establish its direction.Source: §5 · p. 9
• LoRA improved performance and corrected directional bias for BioMistral-7 B and Qwen2.5-14 B on external exam sets, but reduced accuracy for balanced base models by specializing them toward ECGQuest.The authors interpret this as a generalization gap; strong zero-shot MedMCQA and MedQA scores may partly reflect training-data leakage because both are longstanding public resources.Source: §5 · pp. 9–10
• ECGQuest labels generated by GPT-4o have not received expert over-read and should be treated as weak labels rather than clinical ground truth.The authors call for human review to correct inaccurate or ambiguous labels and establish an expert-validated benchmark; source-linked quotes enable systematic, traceable review.Source: §5 · p. 10
• Future work should expand ECGQuest with additional public resources and pair ECG-specific language models with waveform- or image-based models, then extend the model-agnostic pipeline to other cardiovascular modalities.Suggested additions include guidelines, educational materials, open-access articles, and cardiology conference abstracts; other modalities include echocardiography, cardiac MRI, and cardiac CT.Source: §5 · p. 10

6 CONCLUSION

• Fine-tuning enabled 7–14 B models to match or outperform substantially larger proprietary models on ECGQuest, but external exam-question benefits were largest for weak or class-biased base models.The external evaluation used labeled ECG questions from MedMCQA and MedQA.Source: §6 · p. 10
• ECG knowledge varied substantially across models, generic medical specialization did not guarantee performance, and larger models generally performed better within the same family.Source: §6 · p. 10
• ECGQuest is a balanced, source-linked True/False dataset whose items have minimally modified opposite-label counterparts; the Q&A set and fine-tuned models are publicly available.Source: §6 · p. 10

A ECGQUEST QUESTION COMPOSITION AND LEVEL OF DIFFICULTY

• GPT-4o categorized ECGQuest question pairs across four ECG knowledge categories, with counts reported across training, validation, and test splits.The supplied passage identifies the categories as ECG Diagnosis, ECG Acquisition Technologies, ECG Characteristics, and Basic Cardiac Physiology but gives no counts.Source: A ECGQUEST QUESTION COMPOSITION AND… · p. 15, Table 2 · p. 15
• Question difficulty or clarity varied substantially, from items nearly all models answered correctly to a small subset almost none answered correctly.Without expert review, model disagreement may also reflect question ambiguity.Source: A ECGQUEST QUESTION COMPOSITION AND… · p. 15, Figure 4 · p. 15
• Approximately 73% of the 1,050 test questions were answered correctly by at least half of evaluated models, while fewer than half answered the remaining 27% correctly.The profile orders questions from easiest or clearest to hardest or most ambiguous according to the fraction of models answering correctly.Source: Figure 4 · p. 15, A ECGQUEST QUESTION COMPOSITION AND… · p. 15
• No single model led every ECG knowledge category; category-level results showed complementary strengths, with fine-tuning particularly beneficial.The fine-tuned DeepSeek-R1-Distill-Qwen 14 B model also ranked second in Basic Cardiac Physiology, at 88.9%.Source: A ECGQUEST QUESTION COMPOSITION AND… · pp. 15–16
• 76.8% accuracy: fine-tuned DeepSeek-R1-Distill-Qwen 14 B led ECG Diagnosis, while GPT-4o led ECG Acquisition Technologies at 71.6%.The fine-tuned Gemma-4 12 B model followed in ECG Acquisition Technologies at 70.4%.Source: A ECGQUEST QUESTION COMPOSITION AND… · p. 15, Figure 5 · p. 16
• 76.5% accuracy: fine-tuned DeepSeek-R1-Distill-Qwen 14 B led ECG Characteristics; fine-tuned Qwen2.5-Instruct 14 B led Basic Cardiac Physiology at 90.7%.The next-highest accuracies were 71.1% for Qwen2.5-Instruct in ECG Characteristics and 88.9% for DeepSeek-R1-Distill-Qwen in Basic Cardiac Physiology.Source: A ECGQUEST QUESTION COMPOSITION AND… · pp. 15–16, Figure 5 · p. 16

B EXTERNAL EXAM-LABELED EVALUATION

• The external sets contained 2,084 MedMCQA and 294 MedQA True/False questions converted using official answer keys, providing ECG-knowledge evaluation independent of ECGQuest question generation.GPT-4o filtered questions to four ECG topics; negated or except-style items were excluded, and false versions appended randomly selected incorrect options.Source: B EXTERNAL EXAM-LABELED EVALUATION · p. 16
• Fine-tuning raised BioMistral-7B accuracy from 51.3% to 60.5% on MedMCQA and from 50.7% to 58.5% on MedQA.Its macro-F1 roughly doubled on both sets by correcting True-class bias; Qwen2.5-Instruct-14B improved on MedMCQA but remained close to its starting point on MedQA.Source: B EXTERNAL EXAM-LABELED EVALUATION · pp. 16–17
• GPT-5 led on both exams, with 86.5% on MedMCQA and 93.2% on MedQA, followed by GPT-4o at 83.7% and 89.8%.Gemma-4-31B and Gemma-4-26B also outperformed GPT-3.5 on both sets, and Gemma accuracy increased monotonically with model size; MedGemma-4B was the only competitive medical model, while three other medical models were near chance.Source: B EXTERNAL EXAM-LABELED EVALUATION · p. 16, Table 3 · p. 17
• Gemma-4-12B stayed near its base accuracy on MedMCQA, 74.4% to 72.6%, and improved on MedQA, 68.7% to 72.8%.The fine-tuning effect therefore differed across external datasets even for this model.Source: B EXTERNAL EXAM-LABELED EVALUATION · p. 17
• Fine-tuning reduced accuracy for Llama-3.1-8B and DeepSeek-R1-14B on both exams, and the five-model ensemble scored 67.2% on MedMCQA and 65.6% on MedQA.The authors note that these long-standing public exam resources may have entered pretraining corpora, so strong zero-shot scores may partly reflect data leakage rather than ECG understanding.Source: B EXTERNAL EXAM-LABELED EVALUATION · p. 17, Table 3 · p. 17

• Text-mode GPT-4o generation limits questions to ECG topics and page text, requiring verbatim source_text and direct true or single-detail-false transformations.Questions must be self-contained, and their labels must be determinable from source_text alone.Source: C QUESTION-GENERATION, VALIDATION, AND… · pp. 18–19
• Vision-mode generation uses only explicitly written page text, forbids waveform interpretation and image references, and requires each question to derive directly from source_text.Source text may come from captions, labels, tables, or annotations.Source: C QUESTION-GENERATION, VALIDATION, AND… · pp. 19–20
• Verification discards candidates that violate grounding or image-reference constraints, including unsupported cardioversion claims and questions answerable only by viewing a figure.The examples show rejection when the cited text instead recommends vagal manoeuvres and adenosine, or when the question refers to Figure 4.7.Source: C QUESTION-GENERATION, VALIDATION, AND… · p. 20
• The label-balancing prompt creates an opposite-label counterpart from the same source_text, which must remain unchanged and support the inverse label without outside concepts.A self-check requires the counterpart to differ from the original, remain self-contained, and be faithful to source_text.Source: C QUESTION-GENERATION, VALIDATION, AND… · p. 21