Source-linked AI summary
ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
Jungwoo Oh, Hyunseung Chung, Junhee Lee, Min-Gyu Kim, Hangyul Yoon, Ki Seong Lee, Youngchae Lee, Muhan Yeo, Edward Choi
TL;DR
MLLMs can produce plausible ECG interpretations, but it remains unclear whether their reasoning is grounded in visual signal evidence. ECG-Reasoning-Benchmark evaluates this question with a multi-turn benchmark spanning 17 diagnoses and over 6,400 samples, finding that models often retrieve criteria yet fail to complete the grounded reasoning chain.
Problem
Existing ECG-MLLM evaluations and synthetic explanations can assess fluent medical plausibility without establishing whether interpretations are supported by the underlying ECG signal.
Method
The paper introduces ECG-Reasoning-Benchmark, using structured ECG features and step-wise verification to evaluate clinical reasoning and visual grounding across 17 diagnoses.
Results
Models generally identify required clinical criteria but fail to sustain complete reasoning chains grounded in granular ECG evidence, with Completion success rates reaching only around 6%.
Takeaways & Limitations
The findings indicate that current MLLMs bypass actual visual interpretation, exposing a limitation in their visual grounding capabilities.
Takeaways & Limitations
Existing reasoning approaches remain constrained by reliance on synthetic ground truth, which can encourage linguistic imitation rather than evidence derivation from raw signals.
Abstract
from arXiv · showhide
While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce \textbf{ECG-Reasoning-Benchmark}, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.
1 Introduction
Existing ECG models can achieve strong diagnostic classification, but clinical adoption requires explanations grounded in the ECG signal rather than fluent black-box predictions. ECG-Reasoning-Benchmark addresses this gap by evaluating multi-stage clinical deduction and visual grounding.
- Diagnostic accuracy alone is insufficient in high-stakes healthcare, where clinicians require evidence and reasoning supporting the label.
- Synthetic explanations can recite textbook criteria regardless of what the underlying ECG signal shows, creating hallucination risk.
- LLM-as-a-Judge evaluation compares outputs with synthetic references and cannot verify grounding in the actual ECG image.
- ECG-Reasoning-Benchmark evaluates ECG interpretation as a multi-stage process involving medical knowledge, perceptual detection, and visual grounding.
- The benchmark shifts evaluation from subjective fluency scoring to step-by-step verification of decisions based on the actual ECG signal.
- Current models often identify required ECG findings but fail to ground those findings within the signal, revealing a critical multi-step deduction failure.
2 Related Works
Earlier ECG-MLLMs supported report generation and question answering, while later systems added explicit reasoning but continued to rely heavily on synthetic supervision. ECG-Reasoning-Benchmark targets the resulting gap with objective signal-grounded evaluation.
- Early ECG-MLLMs adapted vision-language architectures to map global signal embeddings into reports or text-based answers.
- Later approaches incorporated explicit reasoning or physical measurements, but their datasets and reasoning chains remained dependent on synthetic generation.
- Synthetic ground truth encourages models to emulate a teacher’s linguistic style rather than derive evidence from the raw ECG signal.
- The benchmark provides an objective, quantitative examination of whether clinical interpretations are supported by input-signal evidence.
3 Automated ECG Analysis Pipeline
The Automated ECG Analysis Pipeline converts raw 12-lead signals into transparent diagnostic ground truth through waveform delineation, feature quantification, clinical finding extraction, and diagnosis derivation.
- Wave Detection and Segmentation: The pipeline detects P waves, QRS complexes, T waves, and isoelectric background separately for each ECG lead using U-Net3+.
- Wave Detection and Segmentation: P-wave recovery searches RR intervals lacking detections to identify non-conducted P waves missed by the initial model.
- Wave Detection and Segmentation: 1.000 recall and precision were achieved for QRS complexes, while P-wave recall and precision were 0.978 and 0.937, and T-wave values were 0.996 and 0.992.
- Feature Extraction: Feature extraction quantifies wave durations, PR/RR/QT intervals, amplitudes, ST deviations, QRS morphologies, pathological Q waves, and electrical axis.
- Finding Extraction: Continuous measurements are mapped to discrete ECG findings using established clinical criteria, such as identifying PR intervals over 200 ms as prolonged.
- Diagnosis Derivation: Hierarchical logic diagrams combine extracted findings into 17 core ECG diagnoses based on guidelines and validation by three specialists.
4 Construction of ECG-Reasoning-Benchmark
ECG-Reasoning-Benchmark tests whether models can complete a recursively verified clinical reasoning chain grounded in physical ECG evidence. It combines structured signal-derived annotations, multi-step questioning, balanced sampling, and expert review.
- Benchmark Construction: The benchmark verifies every step of clinical deduction against physical signal evidence rather than relying on traditional QA or LLM-as-a-Judge scoring.
- Evaluation Workflow: Evaluation begins with an initial diagnostic question, then proceeds to step-wise verification regardless of the initial answer.
- Evaluation Workflow: Each clinical finding is evaluated through criterion selection, finding identification, and granular ECG grounding tasks.
- Evaluation Workflow: ECG grounding requires identifying abnormal leads, locating relevant waveforms in the 10-second strip, and quantifying measurements.
- Evaluation Workflow: The four-step validation loop is repeated for every criterion, and diagnosis is concluded only after all supporting findings are validated.
- Dataset Construction: The dataset balances 100 positive and 100 negative samples for each of 17 diagnoses across their logical reasoning paths.
- Dataset Construction: Human labels were filtered against pipeline diagnoses, producing over 6,400 samples from PTB-XL and MIMIC-IV-ECG, with expert validation of 143 representative samples.
5 Experiments
The benchmark evaluates diverse ECG-capable MLLMs through multi-turn, stepwise verification across two ECG datasets. Models generally identify diagnostic criteria but struggle to sustain complete reasoning and ground criteria in ECG evidence.
- Evaluated Models: The evaluation covers ECG-specific, medical-domain, general-domain, and proprietary MLLMs using visual or time-series ECG inputs.OpenTSLM receives 100Hz 12-channel time-series data, while other vision-capable models receive converted 12-lead ECG images.
- Evaluation Metrics: The multi-turn protocol measures initial diagnosis accuracy, complete reasoning-chain success, reasoning depth, and diagnosis accuracy after ground-truth reasoning guidance.Semantic verification is performed at each step rather than through rigid string matching.
- Reasoning Completion: 6.26% Completion is the highest reported success rate on PTB-XL, while ECG-R1-RL reaches 5.81% on MIMIC-IV-ECG.OpenTSLM, MedGemma, and MedGemma-1.5 achieve around or below 1% Completion.
- Depth Analysis: Average Depth generally exceeds 1.0 but rarely exceeds 2.0, indicating success in Criterion Selection followed by failures in Finding Identification and ECG Grounding.The results suggest models retrieve relevant medical criteria but do not reliably connect them to granular ECG evidence.
- IDA vs. GT-RDA: Hulu-Med rises from 57.49% to 99.42% on PTB-XL and from 51.62% to 97.20% on MIMIC-IV-ECG under GT-RDA, whereas ECG-R1-RL declines from 85.41% to 22.70% and from 80.17% to 22.95%.The comparison contrasts non-ECG-specific models’ use of supplied reasoning with ECG-specific models’ degradation under the same guidance.
6 Discussion
The benchmark exposes a reasoning gap in ECG-MLLMs: plausible diagnoses do not reliably reflect granular ECG evidence. Its strict design also excludes uncertainty and may not match clinicians’ heuristic workflows.
- Discussion: The benchmark finds that plausible diagnoses from current MLLMs often lack grounding in granular visual evidence from the ECG signal.ECG-specific models’ collapse under ground-truth reasoning further suggests reliance on superficial pattern matching rather than valid medical deduction.
- Limitations: The benchmark excludes borderline or ambiguous samples to maintain indisputable ground truth, limiting evaluation of diagnostic uncertainty.Future iterations aim to reintroduce borderline cases and assess recognition of ambiguity and expression of diagnostic doubt.
- Limitations: The formal sequential reasoning process may not mirror clinicians’ heuristic workflows and penalizes shortcuts that bypass exhaustive finding verification.The authors plan dynamic reasoning diagrams with severity and certainty weightings to support early termination.
A.1 Wave Detection and Segmentation Performance
The ECG segmentation evaluation uses LUDB with AAMI-style 150 ms onset/offset tolerance across three train-validation-test splits. The complete pipeline improves recall and achieves near-perfect QRS segmentation, while annotation omissions artificially reduce P-wave precision.
- Table 2 evaluates ECG segmentation on LUDB using mean performance and 95% confidence intervals across three train-validation-test splits.A detection is correct when its onset or offset lies within 150 ms of the expert annotation.
- The comparison includes traditional tools, raw UNet3+, and the complete pipeline combining UNet3+ with context-aware post-processing.
- QRS segmentation reached Recall and Precision of 1.000 with the complete pipeline.
- Post-processing consistently improved recall across waveforms but reduced P-wave Precision for P Onset from 0.942 to 0.934.
- Unannotated but visually present P waves were counted as False Positives, artificially lowering precision under LUDB’s annotation policy.Figure 3 illustrates these detections before and after post-processing.
A.2 Diagnostic Logic Diagrams for 17 Core Diagnoses
The benchmark organizes 17 core ECG diagnoses into six clinical groups and represents each diagnosis with a hierarchical logic diagram. These diagrams encode both clinical finding sequences and the wave, lead, and measurement grounding required at each step.
- The 17 diagnoses span AV Block, Conduction Disturbance, Hypertrophy, Ectopic Beat, Myocardial Infarction, and additional core diagnostic categories.
- The benchmark’s core diagnoses directly cover 56.05% of PTB-XL samples and 82.03% of MIMIC-IV-ECG samples.
- Including indirectly derivable conditions expands coverage to 76.85% for PTB-XL and 96.18% for MIMIC-IV-ECG.
- The hierarchical diagrams specify clinical finding sequences together with wave, lead, and measurement grounding elements.
B Dataset Statistics
ECG-Reasoning-Benchmark combines PTB-XL and MIMIC-IV-ECG recordings into a large multi-turn evaluation with roughly eight reasoning turns per case. Its case allocation targets balanced positive and negative coverage across valid reasoning paths.
- The benchmark contains 2,868 PTB-XL recordings and 3,316 MIMIC-IV-ECG recordings, with 24,097 and 27,025 QA pairs, respectively.
- Evaluation loops average 7.81 reasoning turns per PTB-XL case and 8.05 per MIMIC-IV-ECG case.
- Each diagnosis and dataset targets 100 positive and 100 negative cases, distributed uniformly across valid reasoning paths.
C.1 Detailed Computation of the Depth Metric
The Depth metric records how far models progress through sequential verification loops, from criterion selection to diagnostic decision, with fractional scoring for ECG grounding sub-tasks. A CLBBB example shows how finding-level scores are pooled into a micro-average.
- Each reasoning loop evaluates Criterion Selection, Finding Identification, ECG Grounding, and Diagnostic Decision in sequence.ECG Grounding receives fractional credit according to the number of required grounding sub-tasks.
- The example’s microaveraged Depth is 2.375, computed by pooling all four finding-level scores rather than averaging within the sample first.
- Depth pooling across all samples represents the exact verification stage reached for each clinical finding on average.
- The benchmark evaluates models with deterministic decoding at temperature 0 across the generation pipelines.