PaperlayerView exact revision on arXiv

Source-linked AI summary

EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence

Chaoyin She, Ruifang Lu, Lida Chen, Wei Wang, Qinghua Huang

arXiv:2509.14977v3cs.CV

TL;DR

Ultrasound diagnosis remains expertise-dependent, while general VLMs have limited ultrasound-specific knowledge and multi-organ generalization. EchoVLM combines broad ultrasound data with an expert-validated instruction pipeline and dynamically routed Dual-path MoE. It outperforms state-of-the-art models across multiple anatomical systems, with the highest average BLEU-1 score of 53.87 in report generation.

  • Problem

    General-purpose VLMs have limited ultrasound-specific knowledge and multi-organ generalization, while conventional ultrasound diagnosis depends heavily on physician expertise.

    Source: §1 · p. 1

  • Method

    EchoVLM combines a 10B ultrasound-specialized VLM, seven-system clinical data, expert-validated instruction tuning, and a dynamically routed Dual-path MoE.

    Source: §1 · pp. 1–2, §5 · p. 9

  • Results

    EchoVLM outperforms state-of-the-art models across multiple anatomical systems and achieves the highest average BLEU-1 score of 53.87 in report generation.

    Source: §4.2.1 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The reported results support EchoVLM as a universal ultrasound-specialized VLM spanning multiple anatomical systems.

    Source: §5 · p. 9

    EchoVLM performs suboptimally in vascular analysis, which the authors hypothesize may reflect the dataset’s long-tailed distribution and the small proportion of vascular cases.

    Source: §6 · p. 9

Abstract

from arXiv · show

Ultrasound is the preferred early cancer screening modality due to non-ionizing radiation, cost-effectiveness, and real-time imaging, yet conventional diagnosis relies heavily on physician expertise, causing significant subjectivity and limited efficiency. Vision-Language Models (VLMs) show promise but lack ultrasound-specific knowledge and multi-organ generalization. We propose EchoVLM, the first open-source 10-billion-parameter ultrasound-tailored VLM with a Mixture-of-Experts (MoE) architecture. It is infused with knowledge across seven anatomical systems, trained on 208,941 clinical cases, 1.47 million ultrasound key-frame images, and over 100 diseases or imaging findings. Supporting clinical report generation, diagnosis prediction, and Visual Question Answering (VQA), it outperforms Qwen2-VL by 7.58 BLEU-1 and 3.45 ROUGE-1 points in report generation. This work shows substantial potential for establishing a general-purpose ultrasound VLM and lays a technical foundation for clinical translation. Source code and model weights are available at https://github.com/Asunatan/EchoVLM.

1 Introduction

Ultrasound offers real-time, cost-effective imaging without ionizing radiation, but diagnosis remains expertise-dependent and general VLMs have limited ultrasound generalization. EchoVLM addresses this gap with multi-organ data, expert-validated instruction tuning, and a Dual-path MoE.

1 Introduction

  • General-purpose VLMs have limited ultrasound generalization, restricting their clinical utility in ultrasound workflows.The introduction reports a substantial performance gap between generic models and their supervised fine-tuned variants.Source: §1 · p. 1
  • EchoVLM’s multi-organ dataset covers seven anatomical systems, 208,941 cases from 15 hospitals, 1.47 million key-frame images, and over 100 diseases or imaging findings.The introduction describes the dataset as the largest multi-organ ultrasound dataset to date.Source: §1 · pp. 1–2
  • An expert-validated few-shot prompting mechanism generated 1.8 million instruction-tuning pairs while using expert oversight to ensure clinical accuracy.Source: §1 · pp. 1–2
  • A Dual-path MoE injects ultrasound knowledge while preserving pre-acquired knowledge, with dynamic routing adapting task-specific subnetworks to heterogeneous diagnostic domains.Source: §1 · p. 2

2 Related Work

VLM research has advanced from visual-text alignment toward richer feature integration and instruction tuning, and has increasingly entered medical applications. Existing systems nevertheless face ultrasound-specific knowledge and context-length limitations in specialized clinical settings.

2 Related Work

  • VLM development progressed from image-text alignment toward semantic embeddings, cross-attention, visual instruction tuning, and finer-grained visual understanding.The related-work section also describes advances in dataset scale, resolution, multi-image or video understanding, projection layers, and visual encoders.Source: §2 · p. 2
  • Medical VLMs include LLaVA-Med (Li et al., 2023a), Medgemma (Sellergren et al., 2025), HuatuoGPT-Vision (Chen et al., 2024), and Lingshu (Team et al., 2025b).Source: §2 · p. 2
  • In specialized clinical settings, existing systems face insufficient ultrasound-specific knowledge and context lengths that fail to handle common multi-image ultrasound scenarios.The paper proposes an ultrasound-specific VLM architecture optimized for diagnostic workflows and standardized sonographic terminology.Source: §2 · p. 2

3 Method

EchoVLM combines a multicenter, multi-organ ultrasound dataset and instruction-tuning pipeline with a Qwen2-VL-based architecture that adds ultrasound-specialized experts. Its training separates MoE initialization from cooperative instruction fine-tuning and uses a compound autoregressive and expert-load-balancing objective.

3.1 Data Collection and Instruction-Tuning Data Generation Pipeline

  • The pipeline assembled 208,941 cases from 15 hospitals, covering seven anatomical systems and over 100 diseases or imaging findings, then generated 1.8 million instruction-tuning pairs.It used 1.47 million key-frame images, expert-designed few-shot templates for reports, diagnoses, and VQA, and filtering and validation steps.Source: §3.1 · pp. 2–3

3.2 Architecture of EchoVLM

  • EchoVLM adapts Qwen2-VL for clinical ultrasound analysis by adding a Dual-path MoE to inject domain knowledge while preserving pre-existing representations.Visual tokens and text tokens are concatenated and processed by Transformer blocks containing attention, normalization, residual connections, and the Dual-path MoE.Source: §3.2 · pp. 3–4

3.3 Dual-path MoE

  • The Dual-path MoE combines a frozen copy of the original Qwen2 FFN with trained active experts, including a shared expert for every token and sparsely activated top-2 routing experts.A learnable scalar modulates the balance between generic knowledge and ultrasound-specific information.Source: §3.3 · pp. 4–5

3.4.1 Stage I: Parameter-isolated pre-training.

  • Stage I freezes the original Qwen2-VL parameters and trains only newly introduced MoE blocks to acquire ultrasound-specific representations.This parameter isolation is described as preventing catastrophic forgetting of generic multimodal knowledge.Source: §3.4.1 · p. 5

3.4.2 Stage II: Cooperative instruction fine-tuning.

  • Stage II uses LoRA for lightweight tuning of the base model while fully updating MoE components, with dynamic modulation balancing general and ultrasound-domain knowledge.The stage is identified as Cooperative Instruction Fine-tuning.Source: §3.4.2 · p. 5

3.4.3 Training Objectives.

  • Training minimizes a compound objective that combines autoregressive response loss with an auxiliary penalty for balancing expert utilization.The autoregressive loss uses concatenated visual and textual input; the balancing term is controlled by hyperparameter γ.Source: §3.4.3 · pp. 5–6

4 Experiments

Experiments compare EchoVLM with general and medical-specific models across report generation, ultrasound diagnosis, and VQA, then assess MoE design choices. Reported results include strong task performance and gains from shared experts, Top-2 routing, and four routing experts.

4.1 Implementation Details

  • The held-out evaluation uses 27,577 ultrasound images and 3,000 reports, with greedy decoding for deterministic outputs.Stage I trains MoE modules on clinical reports and ultrasound key frames while leaving base-model parameters unchanged; Stage II instruction-finetunes the base model and MoE modules on 1.8 million samples.Source: §4.1 · p. 6

4.2.1 Report Generation

  • 53.87 BLEU-1, 61.69 ROUGE-1, and 55.78 ROUGE-L are EchoVLM (11B)'s highest average scores; it ranks second-highest in METEOR and BERTScore.It leads multiple anatomical systems and ranks first in BLEU-1 and METEOR for most domains.Source: §4.2.1 · p. 6, Table 2 · p. 7

4.2.2 Ultrasound Diagnosis

  • BERTScore:75.44 and ROUGE-1:72.51 are among EchoVLM (11B)'s leading average ultrasound-diagnosis scores; it outperforms other models in most categories.The passage also reports SOTA results in kidney and liver systems.Source: §4.2.2 · p. 6, Table 3 · p. 7

4.2.3 Vision Question Answering

  • 21 of 35 metric-anatomy combinations surpass most comparison models, with EchoVLM among the top three in another eight and overall superior to Lingshu.It nevertheless lags behind Lingshu on certain metrics.Source: §4.2.3 · p. 7, Table 4 · p. 8

4.3.1 Impact of the share expert

  • BLEU-1 gains of +4.58 for report generation and +3.48 for ultrasound diagnosis accompany ROUGE-L gains of +3.45 and +5.49, respectively.BERTScore also improves by +1.27 and +4.06, respectively; the authors associate shared experts with cross-task knowledge transfer and multimodal coherence.Source: §4.3.1 · pp. 7–8, Table 5 · p. 8

4.3.2 Impact of the Top-K Routing.

  • Top-2 routing generally outperforms Top-1 on complex multimodal tasks, with task-dependent gains: report generation improves by +3.94 BLEU-1, +2.81 ROUGE-L, and +3.29 METEOR; ultrasound diagnosis by +4.10 BLEU-1 and +4.76 ROUGE-1.Source: §4.3.2 · p. 8

Impact of the Number of Routing Experts.

  • 10.15 BLEU-1 for report generation and 4.11 BLEU-1 for ultrasound diagnosis are the gains from increasing routing experts from zero to four.Six experts yield only marginal further gains (53.87 to 54.03 and 62.81 to 63.23 BLEU-1) while increasing training time from 55.36 to 59.10 hours per epoch; four experts are reported as the best trade-off.Source: Impact of the Number of Routing Experts · p. 9, Table 7 · p. 9

5 Conclusion

EchoVLM is presented as an open-source 10B-parameter ultrasound VLM combining a multi-organ dataset, expert-validated instruction tuning, and dynamic Dual-path MoE routing. The authors report that it outperforms state-of-the-art models across multiple anatomical systems and identify limitations and future directions.

5 Conclusion

  • EchoVLM outperforms state-of-the-art models across multiple anatomical systems.The conclusion also identifies long-tail data distribution and complex VQA requiring multi-step reasoning as limitations.Source: §5 · p. 9
  • The study introduces an open-source 10B-parameter universal ultrasound VLM with an expert-validated instruction-tuning pipeline and dynamically routed Dual-path MoE.Its dataset contains 208,941 clinical cases and 1.47 million images across seven anatomical systems.Source: §5 · p. 9
  • Planned work includes data rebalancing, broader anatomical coverage, longitudinal patient data, and routing refinement.The stated aims are to advance precise and efficient AI-assisted ultrasound diagnostics.Source: §5 · p. 9

6 Limitations

EchoVLM has suboptimal vascular-analysis performance, which the authors hypothesize is related to the dataset’s long-tailed distribution. They also report that it underperforms Lingshu on certain open-ended VQA metrics.

6 Limitations

  • Vascular analysis is suboptimal, and the authors hypothesize that the dataset’s long tail contributes to this limitation.Vascular cases are the smallest proportion, while domain experts may prioritize patterns from majority classes such as breast and liver.Source: §6 · p. 9
  • In open-ended VQA, EchoVLM underperforms Lingshu (7B) on certain metrics.The authors identify the unstructured, open-ended nature of VQA as an area for improvement.Source: §6 · p. 9
  • The authors propose data rebalancing as a way to improve vascular-analysis performance.They state that vascular-case scarcity likely contributed to the suboptimal results.Source: §6 · p. 9

A Implementation of Data Generation Pipeline: Desensitization and Quality Control

The implementation combines privacy-conscious data handling with a desensitization approach that preserves clinically useful measurements and anatomical landmarks. Its few-shot generation pipeline applies deduplication and expert validation to produce 1.8 million high-quality pairs from 3.42 million raw instructions.

Implementation of Data Generation Pipeline: Desensitization and Quality Control

  • Data came from 15 hospitals in China; the retrospective records were de-identified before training, and the protocol and use received institutional ethics approval.Source: A Implementation of Data Generation… · p. 13
  • The redefined desensitization approach removes sensitive ROI information while retaining measurements and anatomical landmarks.Measurements provide quantitative features, while landmarks support anatomical localization and clinical relevance; Figure 4 compares this approach with traditional ROI-based desensitization.Source: A Implementation of Data Generation… · p. 13, Figure 4 · p. 13
  • Few-shot prompting uses diverse language and vision-language models, deduplicates overlapping instances, and applies AI prescreening followed by expert verification for open-ended data.The pipeline discards instances above the stated ROUGE-L similarity threshold of 0.7 or with Simhash Hamming distance ≤3; Figures 5 and 6 show generation and verification prompts.Source: A Implementation of Data Generation… · p. 13, Figure 5 · p. 14
  • Quality control reduces 3.42 million raw instructions to 1.8 million high-quality data pairs.Source: A Implementation of Data Generation… · p. 14

B Model Architecture and Training Configuration

The architecture uses a Dual-path MoE and PatchMerger, trained in two stages that first adapt the new MoE components and then fine-tune selected model components. Baselines use full-parameter fine-tuning for comparable parameter updates.

Model Architecture and Training Configuration

  • The Dual-path MoE uses Top-2 selection among four routing experts, each with dimensionality 1408, and a shared expert dimension four times larger.Source: B Model Architecture and Training… · p. 14
  • PatchMerger fuses and compresses visual representations by consolidating adjacent 2×2 tokens into one token; in Stage I, base-model parameters remain frozen and optimization is confined to newly integrated Dual-path MoE components.In Stage II, LoRA selectively updates the visual encoder, vision-to-text projection layer, and large language model attention layers. Figure 7 depicts the model structure, training strategy, and overall performance.Source: B Model Architecture and Training… · p. 14, Figure 7 · p. 15
  • Comparative and preliminary baselines use full-parameter fine-tuning, matching EchoVLM’s parameter-update paradigm across both training stages.The authors describe this setup as supporting a fair experimental comparison.Source: B Model Architecture and Training… · p. 15

C Routing Distributions

The routing analysis finds balanced expert use overall despite image-heavy token dispatch linked to the dataset’s image-to-report composition. Visualizations show attention to clinically relevant regions alongside irrelevant artifacts, while selected case studies report advantages and a remaining nodule-identification limitation.

C Routing Distributions

  • Routing frequencies are balanced across experts, but image tokens dominate textual tokens because each report is associated with multiple key-frame images.The analysis attributes the imbalance to 208,941 radiological reports and 1.47 million ultrasound key-frame images.Source: C Routing Distributions · p. 15

D Visualization

  • Grad-CAM visualizations show attention to anatomical landmarks, measurements, and hemodynamic charts, alongside irrelevant image edges and black areas.Red boxes mark meaningful regions and blue boxes mark irrelevant areas.Source: D Visualization · pp. 15–16

E Case Study

  • Across randomly selected anatomical case studies, EchoVLM shows advantages over general-purpose and specialized ultrasound models, but some nodule false negatives remain.The examples cover report generation across different anatomical regions.Source: E Case Study · p. 16

F Fine-grained Medical Entity-level Evaluation

The evaluation measures report quality through extracted medical entities rather than surface-level text similarity. EchoVLM outperforms existing multimodal models across most reported metrics for report generation and diagnosis.

F Fine-grained Medical Entity-level Evaluation

  • Entity-level evaluation emphasizes extracted medical-entity accuracy over surface-level text similarity as a clinically aligned measure of report quality.The methodology is adopted from Li et al. (2024b).Source: F Fine-grained Medical Entity-level… · p. 16
  • Table 9 presents the model architecture and training configuration.Source: Table 9 · p. 16
  • EchoVLM consistently outperforms existing multimodal models across most metrics for ultrasound report generation and diagnosis entity evaluation.Table 8 reports report-generation results across seven anatomical systems, and Table 10 reports ultrasound diagnosis results.Source: F Fine-grained Medical Entity-level… · p. 16

G Out-of-Distribution (OOD) Evaluation

The OOD evaluation uses a public ultrasound dataset with no samples included in training. The reported results indicate adaptability to unseen ultrasound data, assessed with both text-based and medical-entity metrics.

G Out-of-Distribution (OOD) Evaluation

Loading 2509.14977v3…