PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang

arXiv:2607.24743v2cs.CVcs.AIcs.CL

TL;DR

Medical MLLMs need to integrate heterogeneous 2D and 3D images while evaluating outputs in ways aligned with radiologists’ clinical practice. ClinFusion addresses this with a compositional vision system and vision-grounded evaluation framework, achieving state-of-the-art results across multimodal and textual medical benchmarks and the highest-ranked reports in blinded radiologist evaluation.

  • Problem

    Medical MLLMs lack clinically meaningful region-level visual understanding and diverse, high-quality visual training signals for heterogeneous medical data.

  • Method

    ClinFusion combines a compositional cascaded vision encoder for 2D and native 3D data with MedIF-Bench and region-of-interest-grounded report evaluation.

  • Results

    ClinFusion achieves state-of-the-art performance across 2D, 3D, and textual medical benchmarks, with reports ranked highest by six radiologists across 300 clinical cases.

  • Takeaways & Limitations

    The results support vision-grounded, factualness-driven metrics as better reflections of expert radiologist judgment than automatic alternatives.

  • Takeaways & Limitations

    Clinical deployment remains subject to rigorous human-in-the-loop evaluation of diagnostic accuracy, efficiency, and patient outcomes.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.

1 Introduction

ClinFusion addresses medical MLLM deployment as a vision-centric challenge involving heterogeneous 2D/3D visual knowledge absorption and clinically grounded evaluation. It combines a compositional cascaded vision architecture, MedIF-Bench, RoI-grounded report evaluation, expert validation, and agentic tool use for holistic medical understanding.

  • Motivation and challenges: Medical MLLM deployment is fundamentally vision-centric because clinical decisions rely on heterogeneous visual evidence from radiographs, pathology slides, CT, and MRI.The paper identifies visual knowledge absorption and vision-grounded evaluation as tightly coupled dimensions, supported by principled data curation.
  • Evaluation: ClinFusion’s vision-grounded evaluation framework adds MedIF-Bench for medical instruction-following and RoI-grounded methodology for clinically aligned report-generation evaluation.The framework targets overlooked instruction-following capability and region-level, factualness-oriented assessment beyond surface-level text matching.
  • Architecture: ClinFusion introduces a compositional and cascaded vision architecture that unifies heterogeneous 2D and 3D medical imaging through CaSL Fusion.The architecture combines a foundational, well-aligned vision transformer with specialist 2D encoders and progressively enriches visual representations.
  • Expert validation: Six board-certified radiologists ranked ClinFusion’s reports highest across factual accuracy, completeness, and clinical utility in 300 CT and X-ray cases.The blinded evaluation compared ClinFusion, Gemini-3-Flash, and Hulu-Med.
  • System integration: Agentic tool use extends ClinFusion with retrieval-augmented generation and specialist perception models, yielding consistent improvements in text-only and multimodal clinical scenarios.The tools support current, citable medical literature retrieval and high-precision analyses such as organ segmentation and disease classification.

2 Results

ClinFusion achieves state-of-the-art or near-best performance across 2D and 3D medical benchmarks, while preserving strong instruction-following and textual competence. Radiologist evaluations further validate its clinical advantage and the benefit of agentic augmentation.

  • 2D medical understanding: ClinFusion ranks first or second across all eight 2D VQA benchmarks, demonstrating robust and versatile visual reasoning.ClinFusion-8B achieves top scores among medical MLLMs on seven benchmarks.
  • 2D medical understanding: 37.8 F1: ClinFusion-8B leads 2D clinical report generation on CheXpert-Plus, substantially improving over Hulu-Med-7B’s 31.9.ClinFusion models are reported as matching or outperforming proprietary systems, especially on CheXpert-Plus.
  • 3D medical understanding: 80.2 on AMOS-MCQ: ClinFusion-8B surpasses Hulu-Med-7B by 14.5 points and exceeds Gemini-3-Flash by 16 points.ClinFusion-32B reaches 81.7 on AMOS-MCQ and 89.0 on CT-Rate MCQ.
  • 3D medical understanding: 23.9 F1: ClinFusion-32B leads CT-Rate Report, surpassing Hulu-Med-32B’s 23.4 and Gemini-3-Flash’s 20.2.The model achieves the best F1 across evaluated models under the RoI-Grounded protocol.
  • Textual competence: Seven of eight textual benchmarks: ClinFusion-32B achieves the best scores among medical MLLMs, including MedXpertQA 26.7 versus Hulu-Med-32B’s 19.8.Scaling from 8B to 32B improves MedXpertQA from 20.0 to 26.7 and SuperGPQA from 32.6 to 41.8.
  • Expert evaluation: ClinFusion with agentic tools ranks first in every radiologist-rated dimension, with statistically significant margins over Hulu-Med and Gemini-3-Flash (p < 0.001).The standalone model already outperforms both baselines, while agentic tools provide a further consistent uplift.

3 Discussion

ClinFusion addresses heterogeneous medical data, clinically misaligned evaluation, and static model knowledge through a vision-centric architecture, vision-grounded evaluation, and agentic tool use. Experiments show state-of-the-art performance, while the discussion identifies limited tooling, proprietary-scale gaps, and the need for human-in-the-loop clinical validation.

  • Contributions: ClinFusion’s CaSL Fusion operator uses a compositional, cascaded, and 2D-anchored architecture to unify diverse 2D and native 3D medical data processing.The design creates synergistic visual representations intended to benefit nuanced diagnostic tasks.
  • Contributions: The vision-grounded evaluation framework combines MedIF-Bench for reliable instruction following with RoI-grounded evaluation for clinically aligned, factual report generation.An agentic extension adds retrieval-augmented generation and external specialist models as tools.
  • Results: ClinFusion achieves state-of-the-art performance, outperforming leading open-source models across 2D, 3D, and textual benchmarks and surpassing powerful proprietary APIs on multimodal tasks.These results support the hypothesis that compositional vision architecture benefits holistic medical perception.
  • Implications: The discussion advocates advancing medical AI through equal attention to perception architecture and evaluation methodology in modular, verifiable systems aligned with clinical needs.ClinFusion’s compositional design supports flexible integration of new specialist encoders as imaging technologies evolve.
  • Limitations: ClinFusion’s current agentic toolset is limited, motivating broader modality coverage, including ultrasound and MRI, and additional functionality such as treatment-planning simulators.The discussion also attributes remaining gaps against proprietary systems primarily to capacity and training-data scale differences.
  • Limitations: Clinical translation requires rigorous human-in-the-loop evaluation of diagnostic accuracy, efficiency, and patient outcomes alongside ethical and regulatory consideration.The agentic extension partially mitigates the proprietary performance gap through knowledge retrieval.

4 Methods · 4.1 Architecture

ClinFusion builds on Qwen-VL with compositional vision encoders and Cascade Spatial-Aware Locality Fusion (CaSL Fusion), extending vision processing from heterogeneous 2D inputs to native 3D volumes. Its architecture preserves spatially grounded representations through local cross-attention, stochastic residual regularization, and DeepStack-compatible fusion.

  • 4.1 Architecture: ClinFusion uses Qwen-VL’s LLM, redesigned Qwen ViT, and vision-language merger as the foundational architecture.Qwen ViT provides general visual understanding and serves as the perception backbone.
  • 4.1.1 Perception with Compositional Vision Encoders: The 2D perception system composes Qwen ViT with specialist encoders selected to complement its limitations, including ConvNeXt for local features.ConvNeXt contributes a strong spatial-locality inductive bias.
  • 4.1.1 Perception with Compositional Vision Encoders: CaSL Fusion prioritizes information density over token length by spatially and efficiently enriching Qwen ViT representations through cascaded local cross-attention.Features are projected into Qwen ViT’s hidden dimension and resized to its spatial shape; ConvNeXt uses a larger resolution to preserve high-frequency detail.
  • 4.1.1 Perception with Compositional Vision Encoders: Local neighborhood extraction reduces fusion complexity from quadratic over N to linear while enforcing a spatial inductive bias.The asymmetric, left-associative operator keeps Qwen ViT as the query backbone, progressively enhanced by specialist features before entering the LLM.
  • 4.1.1 Perception with Compositional Vision Encoders: During training, stochastic residual regularization prevents reliance on Qwen ViT by randomly scaling its residual path; inference sets λ = 1 and pdrop = 0, with pdrop = 0.1 by default.The method forces learning from specialist features while retaining stable full-feature utilization at inference.
  • 4.1.1 Perception with Compositional Vision Encoders: CaSL Fusion is compatible with DeepStack, replacing layer-specific Qwen ViT features with enriched features injected into multiple LLM layers to maintain visual grounding.At each DeepStack layer, the foundational feature serves as the query for cascaded specialist fusion.
  • 4.1.2 2D-Anchored Native 3D Volumetric Understanding: ClinFusion addresses lost inter-slice context by combining native 3D encoding with 2D anchor slices and a 2D-anchored 3D CaSL Fusion.The 3D encoder is pre-aligned with language using 3D volumes and radiology reports, while four representative slices are sampled for 2D anchors.
  • 4.1.2 2D-Anchored Native 3D Volumetric Understanding: The depth-aware 3D fusion lets each 2D anchor attend across spatial neighborhoods and the full depth dimension, producing features concatenated into an S × N sequence for the LLM.For a 32-frame input, Dvol = 4 after depth downsampling by 8; the 3D fusion also supports DeepStack and stochastic residual regularization.

4.2 Vision-Grounded Evaluation System

ClinFusion’s vision-grounded evaluation system mirrors radiologists’ integrated skills across five critical dimensions, while MedIF-Bench isolates instruction-format compliance. Its RoI-Grounded Report Generation paradigm adds clinical context and factualness-driven claim evaluation to address limitations of conventional report-generation pipelines.

  • Comprehensive evaluation system: The evaluation system assesses five critical dimensions, including foundational 2D and 3D visual understanding through VQA and text-based medical knowledge through QA.It builds on prior frameworks including MedGemma, Lingshu, HuLu-Med, and HuatuoGPT.
  • MedIF-Bench: MedIF-Bench contains 900 samples that pair medical tasks with manually curated formatting instructions and evaluate format compliance independently of medical accuracy.Its instruction pool spans seven task categories with corresponding regex patterns.
  • RoI-Grounded Report Generation: RoI-Grounded Report Generation combines context-aware report generation with factualness-driven LLM-as-a-Judge evaluation in a three-stage workflow.The workflow extracts clinical context and regions of interest from the GT report, uses them to guide focused generation, and evaluates diagnostic claims.
  • Context-aware report generation: The method extracts clinical indication and anatomical focus from the GT report while enforcing a security constraint that excludes specific findings from the extracted context.This context is injected into the model-under-test’s report-generation prompt.
  • Factualness-driven evaluation: The factualness judge categorizes diagnostic claims as matched abnormalities, missed abnormalities, or false-positive abnormalities, then computes Precision, Recall, and F1-score from TP, FN, and FP counts.These metrics provide transparent, granular indicators of diagnostic correctness.
  • Validation against conventional pipelines: Compared with context-free generation plus RadGraph-F1, the proposed design addresses unreliable penalties for unverifiable claims and avoids RadGraph-F1’s synonymy insensitivity and length bias.The comparison is reported in Section 2.4.1.

4.3 Data Curation

ClinFusion’s 22.2-million-sample corpus is curated to support multimodal perception, evidence-based medical reasoning, and realistic clinical workflows, especially 3D CT interpretation. It combines complementary medical data sources with targeted synthesis pipelines for instruction following, visual grounding, long-horizon reasoning, and native 3D understanding.

  • Corpus objectives: 22.2 million samples comprise a training corpus designed for ClinFusion’s multi-stage curriculum, targeting heterogeneous medical imagery, medical reasoning, and realistic clinical workflows.The corpus particularly emphasizes 3D volumetric CT interpretation.
  • Data sources: Three complementary sources provide the data: open-source medical multimodal datasets, medical evidence corpora, and CT–report aligned data.The evidence corpora include clinical guidelines, journal articles, and case reports; CT–report data combines open-source and hospital-collected studies with reports.
  • Data synthesis pipelines: Four synthesis pipelines address training bottlenecks involving instruction following, dense visual captioning, interactive multimodal reasoning, and large-scale CT captioning.The pipelines are Instruction Warping, Dense Caption Generation, Interactive Multimodal CoT Annotation, and Large-Scale CT Caption with Expert Toolset.
  • Data synthesis pipelines: 610k dense-caption samples are generated from 100k journal articles and 785k case reports using sub-figure filtering and dual-VLM consensus.The pipeline isolates high-quality medical images before producing dense, visually grounded descriptions.
  • Data synthesis pipelines: 67k reasoning trajectories are retained from 100k reasoning-hard instances after multi-agent, iterative evidence-seeking solutions match reference answers.The pipeline couples a text-only orchestrator with a multimodal visual grounder to train long-horizon multimodal reasoning.
  • Data synthesis pipelines: 0.2M open-source CT studies and pseudo-report supervision for 1.1M hospital-collected CT scans expand CT–report aligned data for rapid 3D alignment and holistic 2D/3D instruction tuning.The supervision is generated through segmentation-assisted annotation using lesion and organ segmentation models.

4.4 Agentic Tool Use

ClinFusion extends its clinical deployment with an explicit plan–act workflow that orchestrates retrieval and specialized perception tools. Their outputs are normalized into traceable evidence that the model integrates with the original query and images into a clinical response.

  • Plan–act orchestration: ClinFusion analyzes multimodal inputs, identifies required evidence, creates an ordered tool-invocation plan, and synthesizes tool outputs with the original visual input into a response.The workflow supports optional multi-query rewriting for text inputs and emphasizes traceable evidence over implicit parametric recall.
  • Retrieval-augmented generation: The RAG knowledge tool combines vector passage retrieval with knowledge-graph concept recall to provide up-to-date, authoritative, and traceable medical evidence.Its resources include broad text corpora, UMLS and PrimeKG, 18K clinical guidelines, and 100K research articles.
  • Perception tool ecosystem: Five perception experts serve as external senses, allowing ClinFusion to delegate fine-grained imaging analysis to specialized models before reasoning over structured results.The integrated set comprises three CT-focused experts and two chest X-ray-focused experts.
  • Perception tool ecosystem: 29 common lesion types across eight organs are segmented into inventories containing lesion counts, locations, and sizes by the multi-organ lesion expert.The covered organs are breast, colon, esophagus, stomach, kidney, lung, liver, and pancreas.
  • Perception tool ecosystem: Chest X-ray tools provide structured probabilities for 18 or 40 thoracic findings and generate clinically coherent descriptions when classification scores alone are insufficient.The classification expert integrates TorchXRayVision and CheXFound, while the generation expert supplies sentence-level imaging evidence.

4.5 Progressive Staged Training · 4.6 Implementation Details

ClinFusion is trained through five progressive stages that move from 2D encoder alignment to unified 2D/3D instruction tuning, building each capability on a stable foundation. Its implementation combines Qwen3-VL with specialist 2D and native 3D encoders, encoder-specific fusion settings, and a shared optimization setup.

  • 4.5 Progressive Staged Training: The curriculum progressively increases task complexity and modality, transitioning from 2D alignment to holistic 3D instruction following.This strategy is designed to stabilize training and transfer knowledge from pretrained components.
  • 4.5 Progressive Staged Training: Stage 1 aligns specialist 2D encoders with Qwen ViT using 2D image-text pairs while training projection layers, CaSL Fusion modules, and encoders.The LLM and base vision encoders remain frozen during this targeted alignment stage.
  • 4.5 Progressive Staged Training: Stage 2 unfreezes the LLM for deep semantic alignment between fused 2D visual representations and the language model.Training continues on the same 2D image-text alignment data.
  • 4.5 Progressive Staged Training: Stage 3 fine-tunes the entire model on multi-task 2D visual instructions to develop grounded perception, reasoning, and task execution.This stage targets diverse medical tasks involving 2D imagery.
  • 4.5 Progressive Staged Training: Stage 4 introduces 3D volumes by training only the 3D encoder and depth-aware fusion modules while freezing the LLM and 2D perception components.The 2D-anchored strategy aims to retain established 2D task-execution capabilities during 3D alignment.
  • 4.5 Progressive Staged Training: Stage 5 fine-tunes all components on combined 2D and 3D visual instructions, unifying multimodal capabilities for complex medical tasks.The staged process culminates in a robust multimodal medical model built on a stable foundation.
  • 4.6 Implementation Details: The final architecture builds on Qwen3-VL and adds DINOv2-Large, ConvNeXt-Large-d-320, and PE-3D specialist encoders.Qwen3-VL uses dynamic-resolution processing with patch size 16 and 2 × 2 token merging; encoder inputs retain their pretrained checkpoint resolutions.
  • 4.6 Implementation Details: CaSL Fusion uses k = 3 for DINOv2 and PE-3D and k = 5 for ConvNeXt, whose feature map is upsampled to 5H × 5W.Residual regularization uses pdrop = 0.1 during training and is disabled at inference with λ = 1.

Data Availability

Public training and evaluation datasets are available from their original sources, with download links and access terms documented in the supplementary tables. The in-house clinical knowledge base and hospital-collected CT data are not publicly available, although de-identified research data may be shared under specified conditions.

  • Public datasets: Public training and evaluation datasets are listed in Extended Data Tables 1 and 2, with download links and access terms in Extended Data Table 4.These datasets are available from their respective original sources.
  • Public datasets: SEER cancer registry data used for part of MedIF-Bench are available at the SEER website.The passage provides the access URL: https://seer.cancer.gov/data/.
  • Knowledge resources: The knowledge tool draws on publicly available medical knowledge sources, including UMLS, Wikipedia, and standard medical textbooks.The supplied passage lists these sources across consecutive text passages.
  • Restricted data: The in-house clinical-guideline and journal-article knowledge base and hospital-collected CT data are not publicly available because of privacy, ethical, and institutional restrictions.The restrictions arise from patient privacy, ethical requirements, and institutional data-use agreements.
  • Restricted data: De-identified research data may be provided for academic and non-commercial purposes upon reasonable request, subject to institutional approval and a data-use agreement.Availability is conditional rather than open access.

Extended Data

The extended data presents qualitative 2D and 3D report-generation cases, textual medical benchmark coverage, and examples of tool-assisted clinical workflows. These examples compare ClinFusion with other methods, demonstrate ROI-focused prompting, and show generated findings alongside retrieved evidence and lesion-segmentation outputs.

  • 2D report generation: ClinFusion’s 2D report identifies mild cardiomegaly and pulmonary vascular prominence, matching the ground-truth abnormalities that competing reports describe as normal.The ground truth reports cardiomegaly and pulmonary venous hypertension, whereas Hulu-Med, Lingshu, and Gemini-3-Flash report normal cardiac size and pulmonary vasculature.
  • 3D report generation: The 3D case compares ground-truth thoracic and upper-abdominal findings with Gemini-3-Flash and ClinFusion reports under ROI-focused prompting.Extended Data Fig. 1 explicitly identifies separate 2D and 3D report-generation cases and states that the input prompt directs MLLMs to focus on specific ROIs.
  • Textual medical benchmarks: Extended Data Fig. 2 evaluates ClinFusion on eight text-only benchmarks spanning reasoning, text understanding, and medical examinations.The figure compares open-source general-purpose and medical MLLMs, proprietary models, and ClinFusion under a fair protocol re-benchmarked by the authors.
  • Report generation with context: 28 mm pleural effusion and 24 mm pericardial effusion are among the findings shown in a context-conditioned generation example.The same example describes bilateral ground-glass appearances and consolidations, while also containing a contradictory statement that no pleural or pericardial effusion was detected.

Supplementary Information · S1 Supplementary Methods: Data Synthesis Pipelines

ClinFusion’s supplementary data synthesis pipelines address instruction-following erosion, weak figure-caption supervision, insufficient multimodal reasoning difficulty, and limited CT–report pairs. They generate diverse instruction templates, grounded dense captions, reasoning-hard trajectories, and 1.1M pseudo-reports for hospital CT volumes.

  • S1 Supplementary Methods: Data Synthesis Pipelines: Four synthesis pipelines target instruction following, dense vision–language supervision, long-horizon multimodal reasoning, and report generation for unpaired CT volumes.The pipelines are instruction warping, dense caption generation, interactive multimodal CoT annotation, and large-scale CT captioning with an expert toolset.
  • S1.1 Instruction Warping: Medical adaptation erodes general instruction following, so instruction warping expands 120 manually authored seed instructions into 21k templates using GPT-5.1.The seeds span clinical roles, scenarios, formats, and explicit constraints such as JSON schemas, strict headers, and region-of-interest restrictions.
  • S1.2 Dense Caption Generation: 610k dense-caption samples are produced from 749k high-quality medical images, replacing sparse paper-figure captions with grounded descriptions of visible findings, anatomy, and modality-specific patterns.The pipeline segments and filters sub-figures, then uses two-pass generation and cross-model consistency filtering before producing clinically coherent descriptions.
  • S1.3 Interactive Multimodal CoT Annotation: Interactive multimodal CoT annotation mines reasoning-hard instances and uses iterative text-only orchestration with multimodal visual grounding to produce evidence-supported trajectories.Instances with Shard > 4 yield 100k diverse reasoning-hard samples, while multi-round visual checks reduce grounding drift and retain trajectories matching reference answers.
  • S1.4 Large-Scale CT Caption with Expert Toolset: 1.1M CT cases receive pseudo-report supervision through a visual-expert pipeline that structures lesions, retrieves representative slices, and generates reports with quality control.The hospital collection contains 1.5M de-identified CT scans without paired reports; lesion masks, anatomical labels, marked slices, and metadata support report-style generation.
  • S1.4 Large-Scale CT Caption with Expert Toolset: The CT–report corpus combines 0.2M open-source studies with 1.1M pseudo-labeled cases to support rapid 3D encoder alignment and holistic 2D/3D instruction tuning.Open-source studies provide high-fidelity alignment anchors, while pseudo-labeled cases expand volumetric diversity and robustness.

S2 Supplementary Methods: Agentic Tool Use

ClinFusion’s agentic tool-use system combines normalized hybrid retrieval with cross-source evidence aggregation and routes questions to specialized perception experts when precise visual evidence is needed. Five perception tools provide structured findings for lesion analysis, fatty liver assessment, lymph nodes, and chest X-ray interpretation.

  • Knowledge tool: UMLS concept normalization precedes complementary vector and knowledge-graph retrieval to reduce ambiguity from synonyms, abbreviations, and variant medical terminology.The knowledge graph supports link-based expansion through clinically relevant upstream and downstream concepts.
  • Knowledge tool: Retrieved chunks are clustered across channels and summarized into compact, evidence-grounded context for faithful, traceable clinical answers.This combines embedding retrieval precision with the relational breadth of knowledge-graph recall.
  • Perception experts: Five perception experts produce structured findings for lesion segmentation, fatty liver severity, abdominal lymph nodes, and chest X-ray classification or generation.The lesion expert covers 29 common lesion types across eight organs; the chest X-ray classification expert supports 18 or 40 findings through its component models.
  • Tool routing: ClinFusion uses a schema-first, non-rigid routing interface that calls tools when a question requires precise localization, measurement, grading, or lesion counting.Examples include largest-lesion size, lymph-node enlargement, fatty-liver grade, and lesion counts in a target organ.

S3 Supplementary Ablation Studies

The ablation studies show that stochastic residual regularization benefits 3D volumetric understanding, while specialist encoders contribute complementary global and fine-grained visual evidence through cascaded fusion. Grad-CAM visualizations illustrate this dynamic interplay in chest X-ray report generation.

  • 3D volumetric understanding: Stochastic residual regularization benefits 3D volumetric understanding, motivating addition of a fourth 3D ViT vision encoder for native volumetric understanding.The model is trained across all five stages with partial 2D and 3D data to ensure fast feedback.
  • Encoder ablation: Adding a third vision encoder showed diminishing returns, suggesting later-stage components may struggle to integrate into an already powerful model.
  • Encoder functionality: Grad-CAM chest X-ray examples show DINOv2 focusing on the cardiac silhouette for semantic assessment and ConvNext emphasizing diagnostically relevant fine-grained details.DINOv2’s global focus supports identifying cardiomegaly, while ConvNext becomes more important when fine-grained textural details matter.
  • Cascaded fusion: Cascaded fusion dynamically combines global semantic and fine-grained textural evidence, enabling more robust and holistic perception.The relative contribution depends on the case: DINOv2 dominates large-scale fluid and density changes, whereas ConvNext is more critical for diagnostic texture.

S4 Supplementary Figures

The supplementary figures illustrate ClinFusion’s data-synthesis pipeline and specialist-encoder attention patterns. The examples pair complementary Grad-CAM visualizations with clinically relevant report findings.

  • Data synthesis pipeline: ClinFusion’s data-synthesis pipeline extracts figure images, captions, and surrounding context from case reports and journal articles, then segments composite figures into sub-panels.Panel-wise subcaptions are aligned to each sub-figure before multi-stage filtering removes non-clinical, heavily annotated, or low-quality regions.
  • Related clinical findings: The report examples describe postoperative sternotomy with cardiomegaly but no acute cardiopulmonary findings, alongside persistent bilateral pleural effusions and bibasilar atelectasis.They also mention right subclavian central venous catheter and left internal jugular sheath placement.
  • Specialist-encoder attention: Supplementary Fig. 2 visualizes complementary attentional focus from DINOv2 and ConvNext specialist encoders using paired Grad-CAM heatmaps and related report findings.Each example presents the original image, both encoder heatmaps, and corresponding report content.

S5 Supplementary Tables

The supplementary tables document ClinFusion’s evaluations across 2D, 3D, textual, and instruction-following medical benchmarks, alongside ablations of clinical context, metrics, and vision-encoder design. They also examine native 3D encoding and stochastic residual regularization for 3D tasks.

  • Benchmark evaluations: Supplementary Tables 1–5 cover 2D multimodal, 2D report-generation, textual medical, 3D multimodal, and MedIF-Bench instruction-following evaluations.The tables distinguish proprietary, generalist, and medical MLLMs where applicable, and report individual and composite instruction-following scores for MedIF-Bench.
  • Clinical-context ablation: Supplementary Table 6 tests whether extracted clinical context improves report generation for ClinFusion-8B and Lingshu-7B across four datasets.The comparison uses “w/ Context” and “w/o Context” settings, with Clinical Indication and Area of Focus removed from the latter prompts.
  • Metric comparison: Supplementary Table 7 compares RadGraph-F1 with the proposed LLM-as-a-Judge metric across CheXpert-Plus and IU-XRAY using four model variants.The evaluation includes a verbose Qwen3-VL-8B variant to expose length bias; RadGraph-F1 measures entity and entity-relation F1, while LLM-as-a-Judge reports Precision, Recall, and F1.
  • 3D ablations: Supplementary Tables 11–12 assess native 3D encoding and stochastic residual regularization on 3D VQA and report-generation tasks.Table 11 compares the full model with inference using only sampled 2D slices, while Table 12 contrasts deterministic and stochastic residual connections.
Loading 2607.24743v2…