PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang, Jiarong Guo, Ningshu Li, Lehan Wang, Chenxu Zhao, Jiexin He, Machteld J. Boonstra, Stephan van der Zwaard, Marielle C. van de Veerdonk, Pere Lopez-Gutierrez, Lydia Dux-Santoy, Andrea Guala, Enrique Almar-Munoz, Sebastian Johannes Reinstadler, Radek Debiec, Yi-Hang Zhu, Huiyu Zhou, Adrian P Cheong, Yike Guo, Agnes Mayr, Karim Lekadir, Yong Jiang, Nilesh J. Samani, Dacheng Tao, Folkert W. Asselbergs, Hongwen Fei, Alex PW Lee, Xiaomeng Li

arXiv:2610.32880v1cs.CV

TL;DR

Echocardiography AI remains fragmented and often opaque, while experienced interpreters are in short supply. EchoChat uses a multimodal model with knowledge grounding and guideline-oriented reasoning for comprehensive interpretation. It broadens diagnostic scope and improves shared-task performance over foundation-model comparators, with collaborative reports also outperforming unassisted routine-reader reports.

  • Problem

    Existing echocardiography AI is fragmented, narrow or opaque, leaving no system for comprehensive, explainable analysis amid a shortage of experienced interpreters.

    Source: §1 · p. 2

  • Method

    EchoChat combines a multimodal vision-language model with a cardiology knowledge library and guideline-oriented reasoning that links conclusions to visual evidence.

    Source: §1 · p. 2

  • Results

    EchoChat expanded coverage to 33 conditions and improved average recall over PanEcho by 13.54% on shared recognition tasks, while collaborative reports raised primary-diagnosis F1 from 72.78% to 82.75%.

    Source: §1 · p. 2, §3 · p. 12

  • Takeaways & Limitations

    EchoChat is designed to support echocardiographers’ review, verification, revision, and finalization of reports rather than replace clinical judgment.

    Source: §3 · p. 13

    EchoChat requires standardized, high-resolution records, and its user study was a controlled simulation rather than live-laboratory deployment.

    Source: §3 · p. 13

Abstract

from arXiv · show

Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating as opaque "black boxes" or focusing on narrow tasks that fail to capture the holistic, guideline-driven reasoning of a clinician. Here we introduce EchoChat, a generative multimodal vision-language model that performs comprehensive echocardiographic analysis to support clinician interpretation. By emulating the cognitive workflow of a cardiologist, EchoChat integrates vast medical knowledge with a chain-of-thought reasoning mechanism to explicitly ground its conclusions in visual evidence. Trained on echocardiography-instruction pairs from 46,358 examinations, our model establishes a new state-of-the-art across diverse cohorts from six international medical centers. It supports the broadest diagnostic scope of 33 cardiac conditions and findings, including 15 newly added conditions for which it achieves an average F1 of 73.7% across available external evaluations. It surpasses existing best-performing foundation models in cardiac condition recognition (improving average recall by 13.54% on shared tasks) and quantification (average R2 score increase of 0.275 on shared tasks). Furthermore, in a source-blinded echocardiographer reader study, echocardiography reports co-generated with EchoChat were rated higher in diagnostic accuracy and completeness by two senior echocardiographers, and improved diagnostic performance from 72.78% to 82.75% in terms of F1 score over routine-reader only reports. These results demonstrate that integrating clinical reasoning into multimodal AI can democratize specialist expertise and transform the standard of cardiovascular care.

1 Introduction

EchoChat is designed to address fragmented, opaque echocardiography AI with comprehensive analysis that grounds conclusions in medical knowledge, guidelines, and visual evidence. Across external evaluations, it broadens diagnostic coverage and improves performance over existing foundation models, while collaborative reports outperform routine-reader-only reports.

1 Introduction

  • Existing AI tools are limited to narrow tasks or isolated measurements and diagnoses, often as black boxes that do not reproduce holistic, guideline-driven reasoning.The shortage of experienced interpreters further constrains access to echocardiography interpretation.Source: §1 · p. 2
  • EchoChat is a multimodal vision-language assistant for interpreting echocardiograms, estimating measurements, drafting reports, and answering evidence-based questions.Its framework combines a knowledge library with guideline-oriented reasoning that links conclusions to visual evidence.Source: §1 · p. 2
  • 82.75% versus 72.78% primary-diagnosis F1 was achieved by EchoChat-routine echocardiographer collaborative reports compared with unassisted reports.Two senior echocardiographers also rated collaborative reports higher in diagnostic accuracy and completeness.Source: §1 · p. 2
  • EchoChat extended diagnostic coverage to 33 cardiac conditions, including 15 newly available challenging conditions averaging 83.0% accuracy across four external test sets.The evaluation included over 5,000 studies across five external hospitals in the Netherlands, Austria, Spain, and China.Source: §1 · p. 2
  • EchoChat improved average recall over EchoPrime by 28.00% and PanEcho by 13.54%, and average R2 over them by 0.294 and 0.275, respectively.These comparisons cover shared cardiac-condition recognition and quantification tasks.Source: §1 · p. 2

2 Results

EchoChat was trained on 46,358 echocardiographic studies and evaluated across external hospitals on recognition, measurement, report generation, and question answering. Results include stronger shared-task performance than foundation-model baselines and better diagnostic performance and report ratings when clinicians revised its drafts.

2.3 EchoChat generalizes across external hospitals in cardiac condition recognition and measurement assessment

  • Across 41 external evaluations of 15 further conditions beyond shared baseline tasks, mean F1 was 73.7%; external cohorts also covered all 33 findings where available.External cohorts showed condition-specific variation, including ventricular and atrial septal defects and left ventricular apical aneurysm.Source: §2.1 · p. 4, §2.3 · p. 8

2.1 EchoChat integrates condition recognition and quantitative measurement across major domains of echocardiography

  • EchoChat estimated 22 parameters and graded 11 abnormalities; internal left ventricular ejection fraction estimation achieved MAE 2.11 and R2 0.90.The parameters span cardiac structure, function, and blood flow; severity grading used condition-specific ordinal scales.Source: §2.1 · p. 4

2.2 EchoChat supports structured report drafting and case-specific interaction with echocardiographers

  • On Hong Kong SAR internal held-out testing, EchoChat’s abnormality-finding F1 was 0.7341 (95% CI, 0.7235-0.7448) versus 0.1189 (95% CI, 0.1080-0.1305) for Qwen2.5-VL [5]; on MIMIC-Echo-VQA, direct-inference accuracy was 66.4% versus Echo-CoPilot’s 54.0%.The MIMIC-Echo-VQA evaluation used direct inference on 622 held-out multiple-choice echocardiography video questions.Source: §2.2 · pp. 5, 8

2.1 EchoChat integrates condition recognition and quantitative measurement across major domains of echocardiography

  • Across shared external tasks, condition F1 was 79.6% vs 62.9% (PanEcho) and 79.7% vs 56.5% (EchoPrime), while measurement R2 was 0.518 vs 0.244 (PanEcho).Condition comparisons covered 50 entries with PanEcho and 47 with EchoPrime; the measurement comparison covered 26 entries with PanEcho.Source: §2.1 · p. 4, Figure 2 · p. 6, Figure 3 · p. 7

2.4 EchoChat combines domain knowledge and structured reasoning to improve heart condition recognition and support diagnostic review

  • The combined Echocardiography Knowledge Library and Cardiologist-Like Reasoning configuration achieved the highest accuracy (87.56%), recall (82.86%), and F1 (87.13%) among evaluated configurations.Both components contributed complementary gains; explanations also exposed unsupported exclusions and mismatches in stated imaging findings.Source: §2.4 · pp. 8–9

2.5 EchoChat-echocardiographer collaboration improves cardiac condition recognition

  • Collaborative reports achieved micro-F1 scores of 82.4% and 83.1%, versus 69.5% and 76.1% for routine readers A and B.The evaluation used 100 examinations and the same 21 clinician-defined primary diagnostic targets for all outputs.Source: §2.5 · p. 9, Figure 4 · p. 10

2.6 EchoChat-echocardiographer collaboration enhances echocardiography report quality

  • Source-blinded senior ratings favored collaboration: mean diagnostic-accuracy scores were 4.15 and 3.86 versus 3.35 and 3.57 for routine reports, and completeness scores were 4.36 and 4.15 versus 3.42 and 3.62.The diagnostic-accuracy differences were significant in all four paired comparisons; completeness gains were also significant in all four comparisons after Holm correction.Source: §2.6 · p. 9, Figure 4 · p. 10
  • In a representative case, clinician revision corrected the AI draft’s mitral-leaflet localization from A2 to A3 while retaining the reference-matching regurgitation severity.Mean finding-accuracy and measurement/severity-accuracy scores were each 4.5/5 for the collaboration, compared with 3.5/5 for the AI draft and 2.5/5 for the routine-only report.Source: Figure 5 · p. 11, §2.6 · p. 12

3 Discussion

EchoChat broadens echocardiographic AI to cover more conditions, measurements, and report-generation tasks, with higher average performance on shared tasks and improved clinician-collaborative reporting. Its intended use keeps clinicians responsible for verification and final reports, while current evidence is bounded by standardized high-resolution records and a controlled user study.

3 Discussion

  • Collaborative reports achieved 82.75% micro-F1 versus 72.78% for routine-reader-only reports, a 9.97 percentage-point gain, and EchoChat outperformed evaluated foundation models on shared tasks.External evaluations across five international test centers support generalizability; condition recognition was assessed at four centers and measurements at all five.Source: §3 · p. 12
  • EchoChat covers 33 conditions, 22 measurements, and severity grading for 11 abnormalities, extending prior models’ scope of up to 18 conditions and 21 measurements.Across 15 additional condition-recognition tasks, mean external F1 was 73.7%.Source: §3 · p. 12
  • Combining the Echocardiography Knowledge Library and echocardiography-based reasoning increased F1 from 81.63% to 87.13% and exposes reasoning trajectories for inspection.The paper describes these components as supporting domain expertise and making stated findings and inferences inspectable.Source: §3 · p. 13
  • EchoChat supports clinician review of abnormalities, severity, measurements, and draft reports, with routine-report-relative gains of 15.84% in diagnostic accuracy and 20.74% in completeness.These correspond to gains of 0.55 and 0.73 points, respectively, on five-point scales rated by senior echocardiographers; clinicians retain final judgment.Source: §3 · p. 13
  • EchoChat currently requires standardized, high-resolution records manually collected by clinicians, and its user study was a controlled simulation rather than live-laboratory deployment.Further validation is needed for automation bias, model drift, system integration, patient outcomes, and multi-turn conversations.Source: §3 · p. 13
  • Future work will seek greater accuracy and robustness and explore collaborative frameworks for integrating multimodal models into cardiology departments’ diagnostic workflows.The paper presents this as a direction for further exploration, not an established deployment outcome.Source: §3 · p. 13

4 Methods

The methods describe EchoChat’s development using multicenter echocardiography data, structured clinical-task supervision, an echocardiography knowledge library, and cardiologist-like reasoning. They also define its evaluation across external cohorts and human-reader studies, including task-specific metrics, statistical procedures, report and VQA assessment, and evaluation of an LLM judge.

4.1 Echocardiography Data Curation and Training Data Construction

  • Data from six centers across four countries included 58,768 examinations, with 46,358 Hong Kong SAR examinations used to train the base model.Reports supplied binary and severity labels, while the training set supported measurement estimation, condition recognition, severity grading, report generation, visual question answering, and diagnostic reasoning.Source: §4.1 · pp. 14–15

4.2 Multi-Agent Collaborative Echocardiography Knowledge Injection

  • The Echocardiography Knowledge Library combines authoritative resources covering over 100 cardiac conditions with a multimodal, UMLS-indexed curation and retrieval pipeline.Its sources include 14 textbooks, 91 guidelines, 573 educational entries, and 176 case studies; top-2 relevant text entries enrich image-based VQA training examples.Source: §4.2 · pp. 15–16

4.3 Cardiologist-Like Reasoning

  • CLR trains EchoChat to explain diagnoses through visual evidence, disease hypotheses, and both supporting and exclusionary reasoning.LLM-generated reasoning paths use report-derived positive and negative disease candidates, treating clinician-approved reports as proxies for imaging findings and assuming unmentioned conditions are absent.Source: §4.3 · pp. 16–17

4.4 Model Design and Training

  • EchoChat uses Qwen2.5-VL-7B with a 32-layer vision transformer and was instruction-tuned for one epoch on 32 NVIDIA H800 GPUs.Inference averaged 1.939 ± 1.869 s per condition-recognition query and 7.884 ± 0.959 s per ten-section report; evaluation also used EchoNet-Dynamic and held-out MIMIC-Echo-VQA questions without benchmark-specific fine-tuning.Source: §4.4 · pp. 17–18

4.5 Cardiac disease recognition in comparison with human echocardiographers

  • A reader study compared EchoChat with routine and senior echocardiographers on 100 held-out examinations, using 21 primary diagnostic labels to assess precision, recall, and micro-F1.The cases comprised 72 with cardiac conditions and 28 without detected abnormalities; the report-derived reference conclusions were independently validated by a senior echocardiographer.Source: §4.5 · p. 18

4.6 Expert evaluation of EchoChat-echocardiographer collaboration

  • Two routine echocardiographers revised EchoChat reports, and two source-blinded senior echocardiographers scored five randomized report types per examination across six dimensions.The rubric separately assessed imaging findings, diagnostic conclusions, measurements and severity, completeness, consistency, and clarity; paired comparisons used examinations as the units of analysis.Source: §4.6 · pp. 18–19

4.7 Evaluation metrics and statistics

  • Condition-recognition metrics included accuracy, precision, and recall, while reader-study micro-F1 pooled diagnostic counts and macro-F1 averaged label-specific F1 scores.Foundation-model comparisons gave equal weight to shared supported condition- or measurement-center entries; discrete generated labels were not treated as continuous scores for conventional ROC analysis.Source: §4.7 · p. 19
  • Severity recognition used precision, recall, and F1 at at-least-mild and at-least-moderate thresholds, with ordinal agreement assessed within condition-specific grade schemes.Four-grade schemes distinguished none/normal, mild, moderate, and severe; three-grade schemes combined moderate and severe. External-center PanEcho aortic-stenosis comparisons used its merged mild/moderate grade scheme, so its at-least-moderate endpoint was unsupported.Source: §4.7 · p. 19
  • Heart measurement estimation used MAE, with LVEF scatter plots and Bland–Altman plots comparing predictions with reference values and showing bias and 95% limits of agreement.The limits of agreement were defined as mean bias ±1.96 times the standard deviation of prediction-minus-reference differences.Source: §4.7 · p. 19
  • Report generation was evaluated with BLEU, ROUGE, and METEOR, while VQA also used sentence recall and F1 computed at the token level.BLEU, ROUGE, and METEOR are identified in the passage as NLG metrics.Source: §4.7 · p. 20
  • Confidence intervals used 1,000 bootstrap iterations for recognition, measurement, and language metrics and 5,000 patient-bootstrap iterations for reader-study comparisons.Ablation configurations were compared using paired patient-level permutation tests with Holm correction across 12- and 16-test families; human–LLM VQA agreement used 20,000 patient-cluster bootstrap replicates.Source: §4.7 · p. 20
  • Abnormality-finding precision, recall, and F1 assessed report concept support and coverage using Qwen3.6-27B to extract and semantically match findings.Extraction excluded normal findings, recommendations, comparisons, dates, and clinical history while retaining trace and mild abnormalities; matching required compatible abnormality identity and anatomy. The matching pipeline was similar to NeuroVFM.Source: §4.7 · p. 20
  • The LLM judge agreed with raters 1 and 2 on 226/240 items (94.2%; κ=0.786, 95% CI, 0.671-0.883) and 206/240 (85.8%; κ=0.536, 95% CI, 0.394-0.662), respectively.Human-human agreement was 202/240 (84.2%; κ=0.464); among 202 items where both echocardiographers agreed, the LLM matched their shared judgment on 197 (97.5%). The judge assessed response consistency against report-derived reference answers without external medical knowledge or unstated clinical thresholds.Source: §4.7 · p. 21

6 Code availability

The study’s code and model will be publicly released upon publication, with implementation details provided in the Methods section.

6 Code availability

  • The code and model are slated for public release upon publication.Implementation details, including network architecture and training strategies, are provided in the Methods section.Source: §6 · p. 21

8 Author contributions statement

The authors divide contributions across study design, method development, benchmarking, clinical input, data preparation, and manuscript preparation and revision.

8 Author contributions statement

  • Y.Q., M.E., and S.S. developed the method, benchmarked AI models, and analyzed results, with Y.Q. and S.S. focused on new methods and M.E. on training and external-hospital benchmarking.X.M.-L. designed the study.Source: §8 · pp. 21–22
  • Clinical collaborators contributed study-design input and user-study organization, while contributors across centers supplied external testing data and supported its processing and model inference.The passage also credits contributors for raw-data preparation, preprocessing, deployment, and visual-question-answering experiments.Source: §8 · p. 22
  • Y.Q. and S.S. drafted the manuscript, other authors contributed to drafting and revisions, and all authors discussed the results and contributed to the final manuscript.Source: §8 · p. 22

Related papers

Loading 2610.32880v1…