PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang

arXiv:2607.24743v2cs.CVcs.AIcs.CL

TL;DR

Medical MLLMs need clinically meaningful region-level visual understanding and diverse, high-quality training signals. ClinFusion combines compositional 2D and 3D vision processing with vision-grounded evaluation, achieving state-of-the-art results across multimodal and textual benchmarks and the highest-ranked reports in blinded radiologist evaluation.

  • Problem

    Medical MLLMs still need clinically meaningful region-level visual understanding and diverse, high-quality visual training signals.

    Source: §1 · p. 2

  • Method

    ClinFusion combines a compositional, cascaded vision encoder for diverse 2D and native 3D data with vision-grounded instruction-following and report-generation evaluation.

    Source: §3 · pp. 10–11

  • Results

    Across 2D, 3D, and textual benchmarks, ClinFusion achieves state-of-the-art performance; blinded radiologist evaluation ranked its reports highest and found its RoI-grounded metric best reflected expert judgment.

    Source: §1 · p. 3, §3 · p. 11

  • Takeaways & Limitations

    ClinFusion’s agentic tool-use extension supports retrieval-augmented generation and external specialist-model use, with reported improvements in text-only and multimodal clinical scenarios.

    Source: §3 · p. 11

    Clinical translation requires rigorous human-in-the-loop evaluation of effects on diagnostic accuracy, efficiency, and patient outcomes, while navigating ethical and regulatory considerations.

    Source: §3 · p. 11

Abstract

from arXiv · show

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.

1 Introduction

Medical MLLMs could support diagnostics, report generation, and interactive decision-making, but clinical deployment is a vision-centric challenge requiring heterogeneous 2D/3D knowledge and clinically grounded evaluation.

Motivation

  • Clinical deployment requires models to absorb heterogeneous visual knowledge and evaluate clinically meaningful, region-level understanding, supported by diverse, high-quality visual training data.Medical evidence spans 2D images and volumetric CT and MRI scans.Source: §1 · p. 2

Challenges

  • Existing approaches either sample 3D volumes as 2D slices, sacrificing structure, or use 3D encoders requiring complex alignment; monolithic encoders struggle with heterogeneous medical data.Current report evaluation also overlooks instruction-following and patient context, while lacking fine-grained accounting of matched, missed, and hallucinated findings.Source: §1 · p. 2

Architecture

  • ClinFusion combines a foundational vision transformer, specialist 2D encoders, and a native 3D encoder, using CaSL Fusion for cascaded 2D enrichment and 2D-anchored 3D alignment.This compositional architecture progressively enriches a well-aligned visual representation.Source: §1 · p. 3

Evaluation

  • The evaluation framework introduces MedIF-Bench for medical instruction-following and RoI-grounded report assessment that uses clinical context to classify findings as matched, missed, or hallucinated.The RoI method uses an LLM judge for factualness-driven assessment.Source: §1 · p. 3

Agentic tool use

  • ClinFusion’s agentic system adds retrieval-augmented access to current medical literature and specialist perception tools for tasks such as segmentation and disease classification.The paper reports consistent improvements in text-only and multimodal clinical scenarios.Source: §1 · pp. 3–4

Results

  • ClinFusion achieves state-of-the-art results across 2D/3D multimodal medical benchmarks and textual medical tasks; six board-certified radiologists in a blinded evaluation of 300 clinical cases ranked its reports highest, and its RoI-grounded metric best reflects expert clinical judgment.Source: §1 · p. 3

2 Results

The Results section evaluates ClinFusion’s compositional vision encoder and evaluation methods across multimodal 2D and 3D, textual, and instruction-following benchmarks, including its native 3D perception and agentic extension.

2.2.1 Evaluation on 2D Multimodal Medical Benchmarks

  • ClinFusion-8B led medical MLLMs on 7/8 2D VQA benchmarks and beat Gemini-3-Flash on six; report F1 was 37.8 (CheXpert-Plus) and 57.3 (IU-XRAY), versus Hulu-Med-7B’s 31.9 and 46.5.A qualitative chest X-ray example also shows ClinFusion identifying cardiomegaly and pulmonary vascular prominence missed by Hulu-Med, Lingshu, and Gemini-3-Flash.Source: §2.2.1 · p. 5

2.2.2 Evaluation on 3D Multimodal Medical Benchmarks

  • On AMOS-MCQ, ClinFusion-8B scored 80.2 versus Hulu-Med-7B’s 65.7 and Hulu-Med-32B’s 73.9; ClinFusion-32B scored 81.7 on AMOS-MCQ and 89.0 on CT-Rate MCQ.For 3D report generation, ClinFusion-32B led CT-Rate Report with F1 23.9 versus Hulu-Med-32B’s 23.4, while Gemini-3-Flash led AMOS Report with 23.4; ClinFusion-32B scored 16.1 there. ClinFusion reports also identified subtle liver density changes and followed requested organ-by-organ areas of focus in a qualitative CT case.Source: §2.2.2 · p. 6

2.2.3 Evaluation on MedIF-Bench

  • Overall-IF scores were 98.1 (ClinFusion-8B) and 98.9 (ClinFusion-32B), exceeding GPT-5.2 (96.0), Gemini-3-Flash (96.6), and Claude-Sonnet-4.5 (86.5).MedIF-Bench also exposes evaluability problems: MedGemma-1.5-4B-IT scored 27.4 Overall-IF and its format failures coincided with low PMC-VQA (18.7) and SuperGPQA (2.25) scores.Source: §2.2.3 · p. 6

2.2.4 Evaluation on Textual Medical Benchmarks

  • ClinFusion-32B led medical MLLMs on 7/8 textual benchmarks, including MedXpertQA (26.7 vs. Hulu-Med-32B’s 19.8), Medbullets (74.8 vs. 67.5), and MedQA-MCMLE (93.8 vs. 87.0).ClinFusion-8B led on PubMedQA (77.6), MedQA-MCMLE (89.1), and MMLU-Med (81.3) but trailed on MedMCQA and MedQA-USMLE; textual reasoning gaps with proprietary models narrowed from 8B to 32B, including MedXpertQA (20.0 → 26.7) and SuperGPQA (32.6 → 41.8).Source: §2.2.4 · p. 7

2.3 Clinical Validation with Expert Radiologists

  • ClinFusion with agentic tools ranked first in Overall, Accuracy, Completeness, and Operability, significantly ahead of Hulu-Med and Gemini-3-Flash (p < 0.001).Standalone ClinFusion also outperformed both systems, with further consistent gains from tools. The RoI-grounded metric led eleven metrics on Kendall’s τ (0.511), Spearman’s ρ (0.572), and Top-1 Accuracy (55.5%); radiologist inter-annotator agreement was W = 0.665 ± 0.275 across 300 cases.Source: §2.3 · p. 7

2.4.1 Analysis of RoI-Grounded Report Generation

  • Providing clinical indication and areas of focus prevented context-free report evaluation from compressing ClinFusion-8B’s advantage over Lingshu-7B and penalizing unverifiable extra findings as false positives.Without clinical context, models may describe anatomy outside the ground-truth report’s scope, making legitimate findings indistinguishable from hallucinations; the RoI-grounded protocol constrains evaluation to relevant regions so claims can be classified as true positives, false negatives, or false positives.Source: §2.4.1 · p. 8
  • The RoI-grounded LLM-as-a-Judge distinguished report quality better than RadGraph-F1 and penalized verbosity rather than rewarding it.For Long Output, RadGraph-F1 rose from 11.8 to 18.8 on CheXpert-Plus and 16.3 to 33.0 on IU-XRAY, while LLM-as-a-Judge fell from 16.8 to 15.1 and 36.7 to 33.6, respectively.Source: §2.4.1 · p. 8

2.4.2 Ablation Studies on Architectural Design

  • Architectural ablations favored local cross-attention, DINOv2 as the balanced single specialist encoder, and cascade fusion for report generation; a second specialist brought diminishing returns.The final architecture used Qwen ViT, DINOv2, and ConvNeXt: the second specialist added at most 0.4 on VQA and 0.9 on report generation, while cascade fusion scored 28.8 versus parallel fusion’s 27.3 on report generation.Source: §2.4.2 · p. 9
  • 3.0 points: removing the native 3D encoder lowered average 3D VQA from 65.8 to 62.8, showing that sparse 2D slices cannot replace volumetric features.Average 3D report-generation F1 fell by 4.0 points (15.7 →11.7); the largest single-task drops were CT-Rate Open VQA (63.6 →56.4) and CT-Rate Report F1 (20.3 →15.5).Source: §2.4.2 · pp. 9–10

2.5 Practical Extension with Agentic Tool Use

  • The agentic framework combines RAG and perception tools in a plan–act workflow, making decisions verifiable through exposed, auditable intermediate evidence.Examples include provenance-aware retrieval from sources such as PubMed and textbooks for a text-only case, and structured lesion findings from a Multi-Organ Lesion Segmenter for a 3D chest CT report.Source: §2.5 · p. 10
  • Agentic tools improved ClinFusion’s 2D/3D multimodal evaluations at both scales, with the largest gains on descriptive tasks rather than closed-form 3D MCQs.For 8B, gains were CheXpert-Plus 2D report +12.4, 3D-RAD Open VQA +8.4, AMOS Report +4.6, and CT-Rate Report +3.7; MCQ gains were AMOS MCQ +5.5, 3D-RAD MCQ +2.2, and CT-Rate MCQ +2.8. For 32B, descriptive-task gains were 3D-RAD Open VQA +9.4, CheXpert-Plus +6.8, CT-Rate Report +3.7, and AMOS Report +3.0.Source: §2.5 · p. 10
  • +12.8 on SuperGPQA was the largest reported 8B text-only gain; agentic tools generally improved knowledge-intensive benchmarks, with MedQA-M an exception.Other 8B gains were MedQA-U +11.1, MedXpertQA +7.7, and MedMCQA +7.3, while MedQA-M fell by 4.8 points, attributed to limited Chinese-language retrieval coverage. At 32B, gains included SuperGPQA +9.1, MedXpertQA +4.5, MedQA-U +4.2, and MedQA-M +0.7; the paper links the latter to the larger backbone’s stronger multilingual capability.Source: §2.5 · p. 10

3 Discussion

ClinFusion presents a vision-centric framework for holistic medical understanding that addresses heterogeneous medical data, clinically misaligned evaluation, and practical deployment. Its contributions combine a compositional vision architecture, vision-grounded evaluation, and agentic tool use, while highlighting limitations that require further research.

Discussion

  • ClinFusion’s CaSL Fusion architecture combines cascaded, 2D-anchored processing of diverse 2D and native 3D data, with ablations confirming benefits for nuanced diagnostic tasks.The compositional design moves beyond monolithic vision encoders to create a synergistic visual representation.Source: §3 · p. 10
  • The vision-grounded evaluation framework introduces MedIF-Bench for reliable instruction-following and RoI-grounded report evaluation, while agentic tools enable retrieval-augmented generation and specialist-model use.The tool-use extension demonstrates consistent improvements in text-only and multimodal clinical scenarios.Source: §3 · p. 11
  • ClinFusion achieves state-of-the-art results across 2D, 3D, and textual benchmarks, surpasses proprietary APIs on multimodal tasks, and earns strong radiologist-aligned evaluation and instruction-following results.The RoI-grounded metric correlates strongly with expert radiologist judgment, and agentic tool use and MedIF-Bench results show consistent improvements and preserved instruction-following, respectively.Source: §3 · p. 11
  • The paper argues that medical AI should advance perception architecture and evaluation together, with modular, verifiable systems assessed against clinical needs.ClinFusion’s compositional design can integrate new specialist encoders as imaging technology evolves, while auditable tools support continuous knowledge updates and delegation.Source: §3 · p. 11
  • ClinFusion’s toolset remains limited, proprietary models retain an advantage on knowledge-intensive text benchmarks, and clinical translation requires human-in-the-loop evaluation and attention to ethics and regulation.Future work should expand tool modalities and functions; the authors attribute the text-benchmark gap primarily to model capacity and training-data scale, which retrieval partially mitigates.Source: §3 · p. 11

4 Methods · 4.1 Architecture

ClinFusion extends Qwen-VL with compositional 2D and native 3D vision encoders, fused through CaSL to preserve the foundation encoder’s language alignment while adding complementary medical-image features. Its 3D pathway anchors volumetric features to aligned 2D representations.

4.1 Architecture

  • ClinFusion extends Qwen-VL [2] [1] with a compositional ensemble of specialized vision encoders to capture medical images’ local detail and global structure across native 2D and 3D representations.The design addresses the challenge of a single generalist encoder capturing both high-frequency pathology detail and global anatomical semantics.Source: §4.1 · p. 12

4.1.1 Perception with Compositional Vision Encoders

  • ConvNeXt supplies local texture features, while DINOv2 [54] provides robust semantics; ClinFusion selects DINOv2 over MedSigLIP [8] for its overall VQA and report-generation balance.The paper says MedSigLIP did not offer a consistent advantage in ablations and that clinical knowledge is instead injected through staged training.Source: §4.1.1 · p. 12
  • CaSL Fusion enriches Qwen ViT features sequentially using spatial local cross-attention, prioritizing information density over token length and retaining the foundation encoder’s established vision-language alignment.Its local neighborhood operation reduces attention cost from quadratic in the number of tokens to linear.Source: §4.1.1 · pp. 12–13
  • CaSL is asymmetric and left-associative, keeping Qwen ViT as the query backbone as specialist features are fused into a representation for the LLM.Stochastic residual regularization drops the query representation during training to encourage specialist-feature use; inference disables this stochasticity, with pdrop set to 0.1 by default during training.Source: §4.1.1 · p. 13
  • CaSL also supports DeepStack by fusing specialist features with Qwen ViT features at each DeepStack layer, then injecting the enriched features into the LLM.This keeps the LLM’s reasoning continuously grounded in visual evidence.Source: §4.1.1 · pp. 13–14

4.1.2 2D-Anchored Native 3D Volumetric Understanding

  • ClinFusion addresses lost inter-slice context by combining a dedicated native 3D encoder [61] with 2D anchor features, using the aligned 2D features to accelerate alignment and transfer pretrained knowledge.It randomly samples four 2D slices from each volume at both training and inference.Source: §4.1.2 · p. 14
  • Depth-aware CaSL lets each 2D anchor attend across its spatial neighborhood and the volume’s depth, fusing anchor features with aligned 3D features before feeding the resulting sequence to the LLM.The described 3D feature volume has depth Dvol = 4 from a 32-frame input downsampled by a factor of 8 along depth.Source: §4.1.2 · p. 14

4.2 Vision-Grounded Evaluation System

ClinFusion’s evaluation system assesses visual understanding, medical knowledge, report generation, and instruction following, adding MedIF-Bench and RoI-grounded report evaluation to align assessment with clinical practice.

4.2 Vision-Grounded Evaluation System

  • The evaluation system covers 2D and 3D VQA, text-based medical QA, report generation, and instruction following, a capability overlooked by existing benchmarks.It builds on prior frameworks including MedGemma [8], Lingshu [6], HuLu-Med [7], and HuatuoGPT [63].Source: §4.2 · p. 15

4.2.1 Evaluation Prompt Templates for VQA and QA

  • For VQA and text QA, MCQ prompts require one tagged uppercase option, while open-ended answers are judged for semantic equivalence to the ground truth.For text-only tasks, prompts omit references to images; open-ended judging uses the question, ground-truth answer, and model answer.Source: §4.2.1 · p. 15

4.2.2 MedIF-Bench

  • MedIF-Bench evaluates format compliance, not medical accuracy, across 900 samples and seven task categories using exact-match regular-expression scoring.A response scores 1.0 only if it fully matches the required format, and 0.0 otherwise; categories include MCQ, open-ended tasks, prognosis, report generation, SEER, and treatment.Source: §4.2.2 · p. 16

4.2.3 RoI-Grounded Report Generation

  • RoI-Grounded Report Generation uses GT-derived clinical context to guide focused reporting, then compares generated claims with the reference to calculate clinical precision, recall, and F1.The extracted context includes clinical indication and areas of focus but excludes specific findings; the judge classifies abnormalities as matched, missed, or hallucinated. The paper reports that context-free generation makes metrics unreliable and RadGraph-F1 has synonymy insensitivity and length bias.Source: §4.2.3 · pp. 16–17

4.3 Data Curation

ClinFusion's training corpus combines medical and general data to support multimodal perception, evidence-based reasoning, and clinical workflows, with four synthesis pipelines addressing targeted training bottlenecks.

Data Curation

  • The corpus comprises 22.2 million samples and is paired with four targeted synthesis pipelines for the multi-stage training curriculum.It includes 12.6M medical multimodal, 4.7M medical text, 3.4M general multimodal, and 1.5M general text samples, drawing on open-source medical datasets, medical evidence corpora, and CT–report aligned data.Source: §4.3 · p. 17

Instruction Warping

  • Instruction Warping expands 120 manually authored clinical instructions into 21k templates applied to existing samples through schema-based recomposition.The templates cover diverse clinical roles, scenarios, and formatting requirements to preserve instruction-following ability during medical fine-tuning.Source: §4.3 · p. 17

Dense Caption Generation

  • Dense-caption synthesis yields 610k visually grounded samples from 100k journal articles and 785k case reports.Sub-figure segmentation and multi-stage filtering isolate high-quality images, while dual-VLM consensus generates the descriptions.Source: §4.3 · p. 17

Interactive Multimodal CoT Annotation

  • Interactive multimodal CoT annotation retains 67k reference-matching trajectories from 100k reasoning-hard instances using an iterative multi-agent pipeline [66].A text-only orchestrator and multimodal visual grounder interact to seek evidence and solve the instances.Source: §4.3 · p. 17

Large-Scale CT Caption with Expert Toolset

  • The 3D CT pipeline curates 0.2M open-source studies with reports [67] and uses lesion [70] and organ segmentation to supervise 1.1M hospital CT scans.The resulting CT–report aligned corpus supports rapid 3D encoder alignment and holistic 2D/3D instruction tuning.Source: §4.3 · pp. 17–18

4.4 Agentic Tool Use

ClinFusion’s agentic extension combines planned retrieval and specialist image analysis with central multimodal reasoning, producing clinical responses grounded in auditable evidence. Its knowledge and perception tools address the need for current, traceable medical evidence and precise visual analysis.

4.4 Agentic Tool Use

  • ClinFusion plans and sequentially executes knowledge-retrieval and perception tools, then synthesizes their structured outputs with the query and image into a traceable clinical response.It identifies needed evidence and orders tool calls; the resulting intermediate evidence can be audited, while ClinFusion remains the reasoning and orchestration engine.Source: §4.4 · p. 18

4.4.1 Knowledge Tools for Evidence-Grounded Retrieval

  • The RAG tool normalizes medical concepts with UMLS, combines vector retrieval with UMLS and PrimeKG graph expansion, and clusters matching claims into cited evidence.Resources include broad text corpora, open medical knowledge graphs, 18K guidelines and expert consensuses, and 100K journal articles.Source: §4.4.1 · p. 18

4.4.2 Perception Tools for Specialized Visual Analysis

  • For voxel-precise imaging tasks, ClinFusion delegates to three CT experts for lesion inventories [70], fatty-liver grading, and lymph-node enumeration.Their structured outputs include lesion counts, locations, and sizes; fatty-liver severity with the supporting liver region; and lymph-node size measurements.Source: §4.4.2 · pp. 18–19
  • For chest X-rays, ClinFusion integrates finding classification [81] [82] and finding-description generation [83], returning normalized evidence objects as natural-language statements.The classification experts provide probabilities for 18 and 40 findings, while generated descriptions supply sentence-level evidence when scores alone are insufficient.Source: §4.4.2 · p. 19

4.5 Progressive Staged Training · 4.6 Implementation Details

ClinFusion uses five progressive stages to build from 2D alignment to holistic 2D/3D instruction following. The implementation specifies its encoder configuration, fusion settings, and shared optimization setup.

4.5 Progressive Staged Training

  • ClinFusion’s progressive curriculum gradually increases task complexity and data modality, moving from 2D alignment to holistic 3D instruction following, with each capability built on a stable foundation.Source: §4.5 · pp. 19–20
  • Stages 1–3 align specialist 2D encoders, deepen vision-language alignment, then instruction-tune the full model for diverse medical tasks.Stage 1 trains projection layers, CaSL Fusion modules, and specialist encoders while freezing the LLM and base vision encoders; Stage 2 unfreezes the LLM on the same image-text data; Stage 3 trains all components on multi-task 2D visual instructions.Source: §4.5 · p. 19
  • Stages 4–5 add 3D capability while preserving 2D skills, then fine-tune all components on combined 2D/3D visual instructions.Stage 4 trains only the 3D encoder and 2D-anchored depth-aware fusion modules; Stage 5 unlocks all components for holistic instruction tuning.Source: §4.5 · p. 19

4.6 Implementation Details

  • The final system builds on Qwen3-VL with DINOv2-Large and ConvNeXt-Large-d-320 specialist 2D encoders and a native PE-3D encoder.Qwen3-VL uses dynamic-resolution image processing with patch size 16 and 2 × 2 token merging; each encoder uses its pre-trained checkpoint’s input resolution. DINOv2 uses 224 × 224 center crops, ConvNeXt uses 320 × 320, and PE-3D processes 32 × 256 × 256 volumes with patch size 83. MedSigLIP at 448 × 448 is only an ablation candidate, not part of the final configuration; 3D inputs use S = 4 anchor slices and Dvol = 4.Source: §4.6 · p. 20
  • CaSL Fusion uses encoder-specific local attention windows, with ConvNeXt upsampled to preserve finer spatial detail.The window size is k = 3 for DINOv2 and PE-3D and k = 5 for ConvNeXt, whose feature map is upsampled to 5H × 5W; larger upsampling was avoided due to training memory constraints. Stochastic residual regularization uses pdrop = 0.1 during training and is disabled at inference by setting λ = 1.Source: §4.6 · p. 20
  • All stages use AdamW, a 1 × 10−5 learning rate, cosine scheduling, BF16 precision, and DeepSpeed ZeRO-3 with FlashAttention-2 and gradient checkpointing.The setup uses β1 = 0.9, β2 = 0.999, ϵ = 10−8, zero weight decay, gradient clipping at 1.0, 100 warmup steps, sequence cutoff 4096, and NVIDIA H20 GPUs.Source: §4.6 · p. 20
  • Stage configurations vary in compute and duration: Stages 1–3 use 16 GPUs, Stage 4 uses 16, and Stage 5 uses 32.Effective batch sizes and epochs are 128 and 1, 1, and 3 for Stages 1–3; 256 and 1 for Stage 4; and 256 and 3 for Stage 5.Source: §4.6 · p. 20

Data Availability

Public training and evaluation datasets are available from their original sources, while hospital-collected CT data and the in-house clinical knowledge base are restricted. De-identified research data may be available to academic, non-commercial researchers upon request and institutional approval.

Data Availability

  • Public datasets are available from their original sources, with download links and access terms listed in Extended Data Tables 1, 2, and 4; SEER data are available online.SEER data were used to construct part of MedIF-Bench.Source: Data Availability · p. 29
  • The knowledge tool draws on public sources including UMLS, PrimeKG, PubMed, StatPearls, Wikipedia, and standard medical textbooks.Source: Data Availability · p. 29
  • Hospital-collected CT data and the in-house clinical-guidelines and journal-article knowledge base are not publicly available because of privacy, ethical, and institutional restrictions.De-identified research data may be provided for academic, non-commercial purposes upon reasonable request, subject to institutional approval and a data-use agreement.Source: Data Availability · p. 29

Extended Data

The Extended Data provides supplementary examples and methodological details spanning ClinFusion’s report generation, agentic workflows, evaluation, training, and benchmark framework.

Extended Data Fig. 1

  • In the 2D case, ClinFusion reports mild cardiomegaly and pulmonary vascular prominence, while Hulu-Med, Lingshu, and Gemini-3-Flash report normal findings.The ground-truth report describes an enlarged heart and engorged pulmonary vasculature.Source: Extended Data · p. 39
  • In the 3D case, ClinFusion reports mild fatty liver, consistent with the ground-truth hepatosteatosis, while Gemini-3-Flash describes a normal liver.The figure compares 2D and 3D report-generation cases with prompts directing models to focus on specific regions of interest.Source: Extended Data · p. 39

Extended Data Fig. 2

  • The text-only benchmark figure covers eight tasks spanning complex reasoning, text understanding, and medical examinations, re-benchmarked under a fair comparison protocol.The caption describes the benchmark scope but provides no individual scores.Source: Extended Data · p. 40

Extended Data Fig. 3

  • A CT case compares report generation with and without extracted clinical context, and underlining marks claims that cannot be verified against the ground-truth report.The extracted context specifies the clinical indication and areas of focus.Source: Extended Data · p. 40

Extended Data Fig. 4

  • Agentic case studies show query rewriting and evidence retrieval for a text-only diagnosis, plus modality-aware CT tools that supply quantitative findings for report generation.The illustrated liver workflow combines lesion segmentation, fatty-liver assessment, and retrieval-augmented evidence for case analysis and management recommendations.Source: Extended Data · p. 41

ROI-grounded report generation evaluation

  • The worked categorization example marks pleural effusion, pericardial effusion, lung opacities and consolidations, and perihepatic ascites as matched, while listing two liver findings as missed.It also labels hepatomegaly, left-lung compressive atelectasis, and perisplenic ascites as hallucinated abnormalities.Source: Extended Data · p. 43
  • 5 TP, 2 FN, and 3 FP yield Precision 0.6250, Recall 0.7143, and F1-score 0.6667 in the illustrated ROI-grounded evaluation example.The pipeline extracts regions of interest from the ground-truth report while minimizing diagnostic-answer leakage, conditions generation on those regions, then uses an LLM judge to categorize matched, missed, and hallucinated abnormalities.Source: Extended Data · p. 43

Training data curation

  • 22.2M training samples spanning 12 medical imaging modalities are produced through dense caption generation, medical multimodal CoT, and large-scale CT caption pipelines for diverse clinical tasks.The caption names medical dialogue, report generation, anomaly detection, medical computation, treatment planning, and prognosis prediction among the supported tasks.Source: Extended Data · p. 44

Evaluation benchmarks

  • The evaluation benchmark framework covers five dimensions: 2D multimodal medical VQA, text-only medical knowledge, 3D volumetric medical VQA, clinical report generation, and instruction following.It distinguishes single- from multi-image inputs and classifies tasks as multiple choice, open-ended, report generation, or instruction following.Source: Table 2 · p. 46

Progressive training strategy

  • ClinFusion’s five-stage training progresses from 2D encoder alignment and instruction tuning to rapid 3D alignment and holistic 2D/3D instruction tuning.The stages successively train specialist encoders and fusion modules, align fused 2D features with the LLM, teach 2D task execution, align the 3D encoder, and tune the full model.Source: Extended Data · p. 46

Supplementary Information · S1 Supplementary Methods: Data Synthesis Pipelines

The supplementary methods detail four data-synthesis pipelines that address instruction-following, dense vision–language supervision, long-horizon multimodal reasoning, and CT report-supervision bottlenecks. Together, they produce instruction templates and grounded training data for medical images and volumes.

S1 Supplementary Methods: Data Synthesis Pipelines

  • The four pipelines diversify medical instructions, turn paper figures into grounded captions, synthesize visually grounded reasoning trajectories, and generate CT pseudo-reports.The CT pipeline targets hospital-collected volumes without reports.Source: S1 Supplementary Methods: Data… · p. 48

S1.1 Instruction Warping

  • Instruction warping uses 120 manually authored seed instructions and GPT-5.1 expansion to create 21k templates, recomposing structured sample fields while aiming to preserve medical semantics.Seeds span dialogue roles, clinical scenarios, output formats, and constraints to counter instruction-following erosion during medical adaptation.Source: S1.1 Instruction Warping · p. 48

S1.2 Dense Caption Generation

  • Dense captioning segments and filters composite paper figures, then uses GPT-5.1 and Gemini 3 Pro consensus inventories to generate clinically coherent descriptions grounded in visible findings.The pipeline removes non-image panels, heavily annotated or low-resolution regions, and near-duplicates, while validating modality and medicalness.Source: S1.2 Dense Caption Generation · pp. 48–49
  • 610k dense-caption samples were produced from 749k high-quality medical images curated from 100k journal articles and 785k case reports.The captions support early vision–language alignment and later instruction tuning.Source: S1.2 Dense Caption Generation · p. 49

S1.3 Interactive Multimodal CoT Annotation

  • The reasoning pipeline mines 100k diverse reasoning-hard instances by prioritizing low-correctness, long-generation QA and retaining instances with Shard > 4.Difficulty scoring uses pass@1 and average output length from Qwen3-VL-8B [2], followed by diversity-aware sampling.Source: S1.3 Interactive Multimodal CoT… · p. 49
  • A text-only GPT-5.1 orchestrator iteratively queries a Gemini 3 Pro visual grounder, yielding 67k reference-matching hard multimodal reasoning instances for instruction tuning.Each reasoning step is tied to newly retrieved visual evidence to mitigate grounding drift over multiple rounds.Source: S1.3 Interactive Multimodal CoT… · p. 49

S1.4 Large-Scale CT Caption with Expert Toolset

  • For hospital CT scans lacking reports, lesion and organ segmentation tools produce structured annotations that are converted into lesion-focused captions and standardized Findings/Impression text.Quality control includes mask sanity checks, cross-slice consistency checks, and text validation; the complementary open-source CT data provides report-paired supervision.Source: S1.4 Large-Scale CT Caption with… · pp. 49–50
  • 1.1M hospital CT cases received pseudo-report supervision, expanding the CT–report corpus to support 3D encoder alignment and holistic 2D/3D instruction tuning.The full corpus combines 0.2M open-source studies with high-quality reports and 1.1M pseudo-labeled cases; the open-source portion provides high-fidelity alignment anchors.Source: S1.4 Large-Scale CT Caption with… · p. 50

S2 Supplementary Methods: Agentic Tool Use

The supplement details ClinFusion’s knowledge retrieval and evidence aggregation, along with its perception tools and their routing interface.

S2.1 Knowledge Tool: Hybrid Retrieval and Cross-Source Aggregation

  • UMLS normalization precedes complementary vector retrieval and UMLS/PrimeKG knowledge-graph recall, with normalized concepts and semantic similarity integrating candidate evidence.Vector search targets semantically matched passages; graph recall expands through clinical relations to upstream and downstream concepts, helping address surface-form mismatches.Source: S2.1 Knowledge Tool: Hybrid Retrieval… · p. 50
  • Retrieved chunks are clustered across vector and graph sources by semantic similarity and shared normalized entities, then summarized with qualifiers and supporting citations.The resulting context combines embedding retrieval’s precision with knowledge-graph breadth for compact, traceable clinical answers.Source: S2.1 Knowledge Tool: Hybrid Retrieval… · pp. 50–51

S2.2 Perception Expert Tools: Tool Descriptions and Routing

  • Experts [70],, and provide lesion inventories across 29 types and eight organs, CT steatosis grading with liver localization, and size-sensitive abdominal node segmentation.These outputs support lesion counts, locations and sizes; standardized fatty-liver assessment; and structured nodal evaluation for lymphadenopathy.Source: S2.2 Perception Expert Tools: Tool… · p. 51
  • Chest X-ray experts [81] [82] classify 18 or 40 findings, while expert [83] drafts findings-style descriptions using CheXagent and CheXinstruct.Classification supplements free-form perception with pathology-level evidence; generation supplies sentence-level imaging context when scores alone are insufficient.Source: S2.2 Perception Expert Tools: Tool… · p. 51
  • ClinFusion routes tools to evidence gaps rather than following a rigid workflow, using capability-card schemas and normalized evidence objects verbalized for reasoning.Capability cards specify supported anatomy, labels, measurable endpoints and resolvable questions; voxel-level coordinates can be mapped to coarse standardized locations such as liver_top.Source: S2.2 Perception Expert Tools: Tool… · p. 51

S3 Supplementary Ablation Studies

Supplementary ablations show that stochastic residual regularization improves 3D VQA and report-generation averages with adequate training, while Grad-CAM suggests the specialist encoders contribute complementary, context-dependent visual evidence.

Stochastic residual regularization

  • 65.8 vs. 64.8 average 3D VQA and 23.7 / 14.0 / 15.7 vs. 23.4 / 13.3 / 15.1 average report-generation metrics favor stochastic residual regularization.The model adds a fourth vision encoder, a 3D ViT, for native volumetric understanding; in shorter runs, the stochastic-versus-deterministic gap was negligible or even inverted.Source: S3 Supplementary Ablation Studies · p. 52

Interpreting the functionality of additional encoders

  • Grad-CAM shows DINOv2 focusing on the cardiac silhouette and effusions/lung bases, while ConvNext highlights sternotomy wires but shows little activation in the second example.The examples associate these patterns with cardiomegaly and bibasilar atelectasis; CaSL Fusion combines the specialists’ evidence, with ConvNext more relevant to fine-grained texture and DINOv2 to global semantic patterns.Source: S3 Supplementary Ablation Studies · p. 52

S4 Supplementary Figures

The supplementary figures document ClinFusion’s data-synthesis pipeline and illustrate complementary specialist-encoder attention through Grad-CAM visualizations.

S4 Supplementary Figures

  • Supplementary Fig. 1 combines dense caption synthesis, expert-guided CT lesion pseudo-reporting, and interactive multimodal reasoning annotation for hard instruction-tuning samples.The pipeline extracts and grounds panel-level descriptions, generates lesion-focused CT captions and standardized pseudo-reports, and rewrites interactive visual fact-checking trajectories into coherent reasoning traces.Source: S4 Supplementary Figures · p. 53

DINOv2 Grad-CAM

  • Grad-CAM visualizations show complementary attentional focus for DINOv2 and ConvNeXt specialist encoders alongside related report findings.Examples include cardiomegaly without acute cardiopulmonary findings and persistent bilateral pleural effusions with bibasilar atelectasis.Source: DINOv2 Grad-CAM · p. 54

S5 Supplementary Tables

The supplementary tables document benchmark results across 2D, 3D, and textual medical tasks, MedIF-Bench, and ablations of report generation and vision-encoder design. Their captions specify evaluated comparisons and metrics, but the supplied passages do not include table values.

Supplementary Tables 1–5

  • Tables 1–5 cover 2D multimodal benchmarks, 2D report generation, textual medical benchmarks, 3D multimodal benchmarks, and MedIF-Bench instruction-following tasks.The captions state that results include task-level and composite instruction-following scores for MedIF-Bench; several tables distinguish medical-MLLM best and second-best scores, and some note re-benchmarking.Source: S5 Supplementary Tables · pp. 55–58

Supplementary Table 6

  • Table 6 tests whether extracted clinical context affects report generation by comparing ClinFusion-8B and Lingshu-7B [6] with and without context.The no-context setting removes Clinical Indication and Area of Focus from the generation prompt.Source: S5 Supplementary Tables · p. 59

Supplementary Table 7

  • Table 7 compares RadGraph-F1 with the LLM-as-a-Judge metric across four model variants, including verbose Qwen3-VL-8B to expose length bias.RadGraph-F1 reports entity and entity-relation F1; the judge reports precision, recall, and F1.Source: S5 Supplementary Tables · p. 59

Supplementary Tables 8–10

  • Tables 8–10 ablate compositional vision-encoder design, comparing single and dual encoders, specialist 2D encoder choices, and multiple-encoder fusion strategies and order.They evaluate VQA accuracy and report-generation metrics, including RadGraph-F1 for entities and entity-relations.Source: S5 Supplementary Tables · p. 59

Supplementary Table 11

  • Table 11 tests the native 3D encoder’s inference contribution by comparing the full model with a version relying only on sampled 2D slices for 3D understanding.It reports VQA accuracy and report-generation precision, recall, and F1.Source: Table 11 · p. 60

Supplementary Table 12

  • Table 12 compares deterministic residual connections with stochastic residual regularization on 3D VQA and report-generation tasks.The caption specifies accuracy for AMOS, 3D RAD, and CT Rate VQA, and average precision, recall, and F1 for report generation.Source: S5 Supplementary Tables · p. 60

Related papers

Loading 2607.24743v2…