PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Heterogeneous Echocardiography AI Outputs in Language Space: Learnable Late Fusion with a Supervised LLM

Ryo Takizawa, Tempei Kabayama, Chiemi Yamazaki, Shun Kitamura, Ryo Matsuoka, Yuta Ando, Satoshi Kodera, Norihiko Takeda

arXiv:2610.39464v1cs.CLcs.AI

TL;DR

Heterogeneous echocardiography AI outputs may be complementary, but learned ensembles often require converting them into fixed numerical representations. This study trains a supervised LLM to fuse report statements and measurements in language space; it performed comparably to evaluated reference strategies and outperformed naive fusion and a general-purpose LLM.

  • Problem

    Individual echocardiography models may not achieve adequate examination-level performance, while conventional learned ensembles require heterogeneous outputs to be converted into numerical decisions or features.

    Source: §1 · p. 1

  • Method

    A supervised MedGemma-based LLM jointly fused view-specific predicted finding statements and continuous measurements in a shared text context to classify four examination-level findings.

    Source: §1 · p. 2

  • Results

    The supervised LLM performed comparably to reports OR and LightGBM while outperforming naive fusion and a general-purpose LLM without task-specific training.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings support the potential of language-based fusion to combine heterogeneous upstream outputs without target-specific structured feature engineering.

    Source: §5 · p. 9

    The retrospective single-center study did not evaluate generalization to external institutions or future time periods.

    Source: §5 · p. 9

Abstract

from arXiv · show

View- and target-specific echocardiography AI models capture useful but complementary information and may not achieve sufficient predictive performance in isolation. A learned ensemble is a natural way to combine these outputs, but heterogeneous models typically require each output to be converted into a binary decision, probability, confidence score, or fixed-length feature vector. We developed a supervised large language model (LLM)-based late-fusion module that directly aggregates view-specific predicted finding statements and predicted measurements in language space and learns a case-level fusion function. In patient-level three-fold cross-validation of 859 examinations from 646 patients, the proposed method integrated heterogeneous outputs within a single framework, performed comparably to the best-performing reference strategy evaluated, and outperformed a general-purpose LLM without task-specific training. These results suggest that a supervised LLM can combine multiple upstream models without model-specific output conversion or feature engineering that requires medical knowledge.

1 Introduction

Multiple echocardiographic views and AI models can provide complementary evidence, yet individual models may not achieve adequate examination-level performance. The paper motivates supervised language-space fusion as a way to combine heterogeneous outputs without model-specific conversions or feature engineering.

1 Introduction

  • Individual echocardiography AI models may not provide adequate examination-level performance, motivating integration of complementary evidence across views and models.Prior work spans view classification, segmentation, ejection-fraction estimation, disease classification, and newer vision–language and multi-task models.Source: §1 · p. 1
  • Conventional learned ensembles require heterogeneous model outputs to be converted into binary decisions, probabilities, confidence scores, or fixed-length numerical features.Changing upstream models or targets can require redesigning output layers, thresholds, feature schemas, and missing-value handling.Source: §1 · p. 1
  • Echocardiography outputs can include severity, uncertainty, continuous measurements, units, and missing items, which advance binarization or fixed features can discard.A common natural-language and numeric context may reduce feature engineering as upstream models or inputs change.Source: §1 · pp. 1–2
  • A supervised LLM late-fusion module combines view-specific finding statements and measurements in language space without model-specific engineering.The study evaluates simultaneous prediction of four targets from imperfect upstream models using patient-level three-fold cross-validation of 859 examinations from 646 patients.Source: §1 · p. 2
  • The study characterizes supervised LLM fusion through its common representation, behavior with weak inputs, and potential to extend the input configuration.Its three contributions also include evaluating four-target prediction and patient-level cross-validation across 859 examinations from 646 patients.Source: §1 · p. 2

2 Related work

Prior echocardiography work has advanced upstream representation learning and examination interpretation, while this study focuses on learned downstream fusion of imperfect models. The paper positions language space as a common interface for heterogeneous predictions that can retain information conventional numerical schemas may lose.

2.1 Automated echocardiography analysis and model integration

  • Earlier echocardiography systems emphasize upstream representation learning or end-to-end examination interpretation, whereas this work studies downstream fusion of individually imperfect models with heterogeneous outputs.The cited examples include a multi-stage pipeline, EchoNet-Dynamic, EchoCLIP, and PanEcho.Source: §2.1 · p. 2

2.2 Learned ensembles and fusion in language space

  • Conventional stacking requires consistent numerical features and a fixed-length schema, adding design work and potentially discarding severity, negation, or ambiguity in text.Language-space fusion can directly process statements, numbers, units, missing items, and variable numbers of outputs without standardized probabilities or confidence scores.Source: §2.2 · p. 2
  • Unlike systems that provide images to an LLM or let it select tools at inference, this work applies supervised late fusion to fixed outputs from view-specific report and measurement models.Echo-CoPilot combines multiple LLM agents and tools, while EchoSonar-R unifies classification and report generation across views.Source: §2.2 · p. 3

3 Method

The study applies patient-level evaluation to retrospective single-center echocardiography examinations, using report and measurement predictions as inputs to a supervised LLM fusion module. It preserves heterogeneous predictions in text and jointly returns four binary examination-level findings.

3.1 Problem setting and dataset

  • The evaluation included 859 examinations from 646 patients acquired at one center between January 2015 and December 2023.It was a retrospective observational study, and 135 patients had more than one examination.Source: §3.1 · p. 3

3.2 Reference labels

  • Four binary targets were defined from predefined findings in routine-care physician reports: LV dilatation, LA dilatation, aortic root dilatation, and global LV systolic dysfunction.Mild-or-greater enlargement was positive for LV and LA; aortic root enlargement required explicit documentation, while prespecified descriptions determined systolic dysfunction labels.Source: §3.2 · p. 3

3.3.1 View-specific predicted finding statements

  • View-specific multitask report models predicted finding severity or type from up to five echocardiographic views, with available views differing by target.The highest-probability class was deterministically mapped to an English statement retaining finding type and severity; class probabilities were not used for fusion.Source: §3.3.1 · p. 3

3.3.2 Predicted measurements

  • Measurement models combined segmentation of 4CH, 2CH, and LAX videos with geometric postprocessing to generate continuous measurements for the four targets.The measurements included Dd, LAD, sinus of Valsalva diameter, EFMOD, and EFTeichholz; measurement parameters were fixed before late-fusion training.Source: §3.3.2 · pp. 4–5

3.4 Fusion task in language space

  • For each examination, available view-specific findings and measurements were templated as text, then jointly fused into four binary outputs in a fixed JSON schema.Report predictions remained statements and measurements remained continuous; upstream outputs were not binarized in advance or required to provide probabilities or confidence scores.Source: Figure 1 · p. 4, §3.4 · p. 5

3.5 Supervised fine-tuning

  • The fusion module used MedGemma-4b-it with LoRA fine-tuning, and training examples were resampled to bring each label's positive rate close to 0.30.Each fold retained original training examinations and added a replacement sample twice the original training-set size; test sets were not modified.Source: §3.5 · p. 5

4 Results

Across patient-level three-fold cross-validation, supervised LLM fusion performed comparably to the strongest reference strategies, while a general-purpose LLM performed worse. Upstream report predictions generally outperformed measurement predictions, and target-level results varied across methods.

4.1 Evaluation setting

  • All examinations from each patient stayed in one fold, and each examination appeared in a test fold exactly once across the three-fold evaluation.The concatenated test-fold predictions formed the cohort’s out-of-fold predictions.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Standalone performance of upstream models

  • Report-prediction F1 exceeded measurement-prediction F1 for LV dilatation, LA dilatation, and global LV systolic dysfunction; aortic-root performance differed substantially between views.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Main results

  • 0.690 Macro-F1 for SFT all was close to LightGBM (0.688) and reports OR (0.703); its +0.002 difference from LightGBM had a 95% CI including zero.The exploratory paired patient-cluster bootstrap did not establish equivalence or non-inferiority.Source: Table 3 · p. 7, §4.3 · pp. 7–8, Figure 2 · p. 8
  • 0.615 Macro-F1 for all-signals OR was below SFT all (0.690) and LightGBM (0.688), while both learned methods avoided degradation from indiscriminately adding measurements.Report predictions were the dominant upstream signals in this study.Source: Table 3 · p. 7, §4.3 · p. 9
  • Adding measurements increased recall for LA dilatation and global LV systolic dysfunction, but precision–recall trade-offs kept F1 differences small; aortic-root predictions were identical.SFT all and the report-only ablation also had nearly identical overall performance, with a paired-difference CI including zero.Source: Table 4 · p. 8, §4.3 · p. 9
  • No single method consistently led across targets: SFT all’s relative F1 performance differed by target against both LightGBM and reports OR.SFT all was lower than LightGBM for LV dilatation, and lower than reports OR for LV and aortic-root dilatation.Source: Table 4 · p. 8, §4.3 · p. 8
  • Fixed-rule rankings depended on the metric: reports AND had lower Macro-F1 but the highest exact match and Hamming accuracy, while reports OR better detected positive classes.The authors attribute the conservative AND rule’s advantage on the latter metrics to negative examples dominating the dataset.Source: Table 3 · p. 7, §4.3 · p. 8
  • GPT-5.5 without additional training had lower Macro-F1 and exact match than SFT all, with a paired-difference CI excluding zero.For global LV systolic dysfunction, GPT-5.5 produced many false positives, whereas SFT all achieved a better precision–recall balance.Source: Table 3 · p. 7, Figure 2 · p. 8, §4.3 · p. 9

5 Conclusion and limitations

Supervised LLM late fusion combined report and measurement predictions, performing comparably to the evaluated fixed-rule and learned fusion references while outperforming naive fusion and an untrained general-purpose LLM. The study was retrospective and single-center, and external generalization and robustness to input changes were not evaluated.

5 Conclusion and limitations

  • The supervised LLM performed comparably to reports OR and LightGBM, while outperforming naive fusion and a general-purpose LLM without task-specific training.Its performance also remained comparable to a report-only model when report predictions dominated.Source: §5 · p. 9
  • Language-based fusion may achieve performance similar to conventional learned fusion without target-specific structured feature engineering.Source: §5 · p. 9
  • The retrospective, single-center study did not evaluate external generalization, input-configuration changes, missing views, or robustness to perturbations.Only 77 examinations were positive for LV dilatation and 70 for global LV systolic dysfunction, resulting in wide CIs for some estimates.Source: §5 · p. 9

A Cohort characteristics

Table 5 summarizes sex and age distributions for 859 examinations, with characteristics reported at the examination level because some patients contributed multiple examinations.

A Cohort characteristics

B Target-specific input signals

Table 6 summarizes report and measurement inputs associated with each evaluation target.

B Target-specific input signals

  • Table 6 summarizes report and measurement inputs associated with each evaluation target.Source: Table 6 · p. 10

C Prompts and output schema

The prompts frame the model as an echocardiography interpreter that classifies four conditions as positive or negative. The all-input example supplies view-specific reports and measurements, while the report-only setup ends after the report blocks.

C Prompts and output schema

  • The system prompt frames the model as an expert echocardiography interpreter tasked with classifying conditions as positive or negative.Source: C Prompts and output schema · p. 11
  • Conditions are classified as negative when normal, absent, trace, borderline, or low-normal, and otherwise as positive.Source: C Prompts and output schema · p. 11
  • The requested output is a four-target JSON object covering LV dilatation, LA dilatation, aortic root dilatation, and global LV systolic dysfunction.Source: C Prompts and output schema · p. 11
  • The all-input example provides view-specific finding statements and measurements, including Teichholz EF, Valsalva, LAD, and Dd values.Source: C Prompts and output schema · p. 11
  • For report-only inference, the input ends after the five view-specific finding-statement blocks, with the same answer tag and four-target JSON format.Source: C Prompts and output schema · p. 11

D Supervised fine-tuning configuration

Table 7 is identified as the supervised fine-tuning configuration.

D Supervised fine-tuning configuration

  • Table 7 presents the supervised fine-tuning configuration.Source: Table 7 · p. 12
Loading 2610.39464v1…