PaperlayerView exact revision on arXiv

Source-linked AI summary

Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, Jin Hao, Zijian Chen, Ruijia Wu, Tao Tang, Junhui Lv, Hongxia Xu, Hongwei Wang, Jun Xiao, Bin Feng, Fudong Zhu, Kenli Li, Weidi Xie, Jimeng Sun, Jian Wu, Zuozhu Liu

arXiv:2510.08668v2cs.CV

TL;DR

Clinical AI remains fragmented across medical text, images, volumes, and videos, motivating Hulu-Med, a transparent generalist VLM that unifies these modalities in one architecture. Across 30 diverse medical benchmarks, it delivers leading open-model performance and remains highly competitive with proprietary systems.

  • Problem

    Clinical AI systems remain fragmented and task-specific, while existing medical VLMs lack comprehensive multimodal coverage and transparent development.

  • Method

    Hulu-Med unifies text, 2D images, 3D volumes, and videos through a generalist architecture trained on 16.7 million public or synthetic samples.

  • Results

    Across 30 diverse medical benchmarks, Hulu-Med delivers leading performance among open medical and general VLMs and remains highly competitive with proprietary systems.

  • Takeaways & Limitations

    The results support the feasibility of a unified, accessible, and transparent pipeline for holistic medical vision-language understanding.

  • Takeaways & Limitations

    Hulu-Med currently accepts only medical text and visual data, excluding genomic and molecular inputs.

Abstract

from arXiv · show

Real-world clinical decision-making requires integrating heterogeneous data, including medical text, 2D images, 3D volumes, and videos, while existing AI systems fail to unify all these signals, limiting their utility. In this paper, we introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM) designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture. Hulu-Med is trained on a curated corpus of 16.7 million samples, comprising exclusively public or synthetic data, spanning 12 major anatomical systems and 14 medical imaging modalities. Hulu-Med employs a medical-aware token-reduction strategy that prunes redundant visual tokens, achieving up to a 55% reduction for 3D and video inputs, improving cross-modal efficiency, and enabling training at 7B-32B parameter scales in approximately 4,000-40,000 GPU hours. Across 30 public in-domain and out-of-domain medical benchmarks-covering text reasoning, visual question answering, report generation, multilingual dialogue, video understanding, and rare disease diagnosis-Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks. Despite being a VLM, Hulu-Med outperforms GPT-4o and matches GPT-o1 on the text-only HealthBench. For the first time in the community, we provide a fully transparent, reproducible and cost-effective pipeline for holistic medical vision-language understanding by releasing our end-to-end data curation, training procedures, and model parameters. Code and models are available at https://github.com/ZJUI-AI4H/Hulu-Med.

Introduction

Hulu-Med is presented as a unified, transparent medical VLM that processes text, 2D images, 3D volumes, and videos within one architecture. It combines universal patch-based encoding with broad benchmark validation, achieving leading results across diverse medical tasks and modalities.

  • Data and Transparency: Hulu-Med is trained on 16.7 million public or synthetic samples spanning 12 anatomical systems and 14 medical imaging modalities.The training corpus supports holistic coverage while maintaining an end-to-end transparent and reproducible pipeline.
  • Architectural Unification: The model unifies visual processing through a SigLIP encoder, 2D RoPE extended to 3D and video, and universal patch-based encoding.The strategy eliminates separate encoders, supports arbitrary medical image resolutions, and models spatio-temporal inputs.
  • Evaluation: 30 public medical benchmarks cover language reasoning, 2D/3D/video VQA, report generation, and related tasks, with strong cross-modal and cross-task generalization.The evaluation is described as the first systematic benchmarking of a medical VLM at this scale and diversity.
  • Architectural Unification: Hulu-Med natively processes text, 2D images, 3D volumes, and video within a single unified architecture.This addresses the limitation of fragmented, task-specific healthcare AI systems and avoids modality-specific encoders.
  • Data and Transparency: 7B–32B parameter models are trained within realistic compute budgets through a fully transparent and efficient pipeline.The paper emphasizes reproducibility, accessibility, and mitigation of privacy and copyright concerns associated with proprietary systems.
  • Evaluation: 10 of 11 benchmarks were won against open-source models, while leading proprietary models were outperformed on 8 benchmarks.These results span seven medical VQA datasets, three report-generation benchmarks, and MedMNIST classification.

Methods · Model Architecture

Hulu-Med unifies text, 2D images, 3D volumes, and videos in one end-to-end VLM with four core components. Its architecture combines rotary position-adaptive visual encoding, dynamic token reduction, multimodal projection, native text tokenization, and autoregressive language decoding.

  • Model Architecture: Hulu-Med processes text, 2D images, 3D volumes, and videos through a single end-to-end framework comprising four core components.The components are a rotary position-adaptive visual encoder, text tokenizer, multimodal projector, and LLM decoder.
  • Model Architecture: 27-layer Vision Transformer encoding treats heterogeneous visual inputs as unified sequences of 2D image planes and uses image patches as the universal processing unit.The encoder has hidden size 1152, intermediate MLP size 4304, and 16 attention heads.
  • Model Architecture: 2D RoPE replaces fixed-size absolute positional embeddings by independently encoding patch positions along image height and width.This embeds relative spatial information directly into self-attention and supports arbitrary image resolutions and aspect ratios without learned positional embeddings.
  • Model Architecture: A two-stage token reduction strategy first applies local spatial pooling with merge factor 2 to 3D and video inputs.Each 2×2 block of adjacent patch tokens is combined into one token via bilinear interpolation.
  • Model Architecture: τ = 0.1 governs dynamic temporal pruning: patches with diff(i)_t < τ are removed from frame t during the forward pass.The strategy reduces final visual token count by up to 60% for 3D and video inputs while maintaining comparable performance.
  • Model Architecture: 152,064-token BPE tokenization uses the tokenizer native to the LLM backbone for textual input.The tokenizer is Byte-Pair Encoding (BPE).
  • Model Architecture: A two-layer MLP projector maps final visual patch embeddings from dimension 1152 into language-compatible embeddings for the LLM.This alignment enables the decoder to interpret visual information within its native language embedding space.
  • Model Architecture: Hulu-Med-7B uses Qwen2.5-7B-Instruct, concatenates projected visual and text embeddings, and autoregressively predicts tokens without task-specific modifications.The unified sequence lets the model support diverse generative tasks.

Training Strategy

Hulu-Med uses a progressive three-stage curriculum that moves from vision–language alignment and medical multimodal pretraining to mixed-modality instruction tuning. The strategy combines public datasets with targeted synthetic pipelines to expand medical knowledge, visual understanding, multilingual reasoning, and multimodal task coverage.

  • Progressive Curriculum: Three stages progressively establish vision–language alignment, inject medical multimodal knowledge, and enable interleaved multi-image plus 3D/video spatiotemporal reasoning.Stages 1 and 2 consolidate 2D single-image competence, while stage 3 introduces broader multimodal contexts.
  • Stage 1: Vision-Language Alignment: 1.4 million image-text pairs train foundational alignment through short caption generation against ground-truth captions.The corpus consists entirely of public medical datasets, including Quilt and MedICaT.
  • Stage 2: Medical Multimodal Pre-training: 4.9 million samples support medical multimodal pretraining through long-form caption generation and open-ended question answering.The stage begins with 2.6 million samples compiled from public datasets.
  • Synthetic Data Expansion: 2.3 million synthetic long captions address underrepresented modalities such as ultrasound and dermatology.A multi-agent pipeline used Gemini-2.5-Pro to rewrite short captions into richer descriptions, yielding 1.4 million enhanced captions.
  • Stage 3: Mix-Modality Instruction Tuning: 10.5 million instruction-tuning instances cover VQA, classification, MRG, and CoT reasoning across 5.9 million text-based and 4.5 million multimodal instructions.The dataset combines public instruction-tuning data with diverse downstream tasks.
  • Synthetic Data Expansion: Three synthesis pipelines add 45K multilingual CoT samples and 600K VQA pairs using role-play prompting, rejection sampling, and Gemini-2.5-Pro.CoT reasoning paths were retained only when their final answers were correct.

Evaluation Framework and Metrics

Hulu-Med is evaluated across text, 2D images, 3D volumes, and video using benchmarks and metrics tailored to each modality and clinical task. The framework combines closed-ended accuracy, open-ended recall, report-generation metrics, and LLM-based judging for complex responses.

  • Evaluation Framework: Four modalities—text, 2D images, 3D volumes, and video—organize the evaluation framework, with task-specific metrics assessing model strengths and limitations.The framework is designed for holistic assessment across heterogeneous medical data and clinical tasks.
  • Text-Based Medical Reasoning and Generalization: Eight text-only benchmarks assess medical knowledge and reasoning on challenging professional medical board-style questions.The evaluation also includes multilingual understanding across six languages and conversational safety and clinical performance in realistic multi-turn dialogues.
  • 2D Medical Image Understanding: Seven 2D VQA benchmarks test multimodal understanding, radiology and pathology knowledge, and higher-order visual reasoning, while MRG evaluates chest-radiograph report generation.MRG uses BLEU (1-4), ROUGE-L, METEOR, recall, and RaTEScore to measure fluency, clinical concepts, and clinical utility.
  • 3D Volumetric and Spatiotemporal Analysis: 3D evaluation covers spatial reasoning, broader clinical reasoning, and report generation; closed-ended questions use accuracy, descriptive subtasks use recall, and AMOS-MM uses NLG metrics.The 3D strategy mirrors the 2D domain and assesses anatomical understanding, temporal reasoning within image series, and clinical fidelity.
  • Surgical and Medical Video Comprehension: Video benchmarks assess surgical phases, instruments, actions, and multi-frame temporal reasoning, using accuracy for closed-ended questions and recall for short descriptive answers.HealthBench responses are judged by Gemini-2.5-Pro, while ChatGPT-4o-latest evaluates differential-diagnosis correctness and clinical appropriateness in RareBench.

Public Data Subtotal 2,582,461

The public-data subtotal comprises large language-focused datasets alongside multimodal medical supervision. Its visual portion spans 2D VQA, classification, report generation, 3D captioning, video tasks, interleaved instruction, and 3D imaging.

  • Language-focused data: 1,859,880 examples make Factoid QA Apollo-Pre the largest listed language dataset, followed by 1,255,356 Clinical Dialogue Miriad samples.LongCoT Data II-Medical SFT contributes 700,000 examples, while medical-o1 contributes 65,531 reasoning examples.
  • Instruction and multilingual data: 496,743 Openhermes instruction samples and 182,240 Glaive-code-assist samples supplement the general instruction mixture.Smaller medical-language additions include AlpaCare with 52,002 samples and MMedC with 45,048 multilingual QA samples.
  • Video data: 24,829 Cholec80-VQA and 17,010 Cholec80-Cap samples supply video question-answering and captioning supervision.Together, these entries explicitly cover both video QA and video caption tasks.
  • 2D and report data: 779,287 Instruction LLaVA_NeXT and 646,750 2D VQA PubMedVision samples provide the largest listed multimodal components.Additional supervision includes 242,310 MIMIC-CXR-MRG report-generation samples and 74,689 MedMNIST classification samples.
  • 3D and interleaved data: 4989 Embodied 3D samples and 31,928 M3D-Cap 3D-caption samples extend the corpus to volumetric medical imaging.The listed interleaved resources include Quilt-Instruct with 105,745 samples, Llava-Interleaved with 36,541, and Llava-Med-Instruct with 56,408.

GRAND TOTAL 10,457,117

Extended Table 7 presents prompt templates used for different task types during inference.

  • Extended Table 7 provides inference prompt templates for different task types.

Task Type Prompt for Direct Answering Prompt for CoT Reasoning

The section specifies task-dependent prompts for direct answering and chain-of-thought reasoning across multiple-choice, judgment, closed-ended, open-ended, and report-generation tasks. Prompts enforce answer formats ranging from option letters and yes/no responses to concise phrases, boxed reasoning answers, and structured reports.

  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Multiple-choice prompts provide the question and options, then request the selected option’s letter directly.
  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Chain-of-thought prompts ask the model to reason step by step and place the final answer within \boxed{}.
  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Judgment prompts require outputting only ’yes’ or ’no’ with no extra output.
  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Close-ended prompts require answering the question using a single word or phrase.
  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Open-ended prompts request a concise answer to the question.
  • Task Type Prompt for Direct Answering Prompt for CoT Reasoning: Report-generation prompts instruct the assistant to generate findings and impressions from given images using the format “Findings: {} Impression: {}.”

MMLU-Pro-Med … MedMCQA

Hulu-Med’s text-medical evaluation compares 7B-scale VLMs across eight benchmarks and reports an overall cross-benchmark comparison. Results include named MedQA-related benchmarks and are averaged over three runs at decoding temperature 0.6.

  • MedQA: The evaluation compares 7B-scale VLM performance across eight medical text benchmarks.Results were averaged over three random runs.
  • MedQA: Each benchmark result was averaged over three random runs.The decoding temperature was 0.6.
  • MedQA: The evaluation used a decoding temperature of 0.6.This setting applied to the reported benchmark results.
  • MedQA: MedQA denotes the MedQA-USMLE benchmark.The caption defines the abbreviated benchmark name explicitly.
  • MedQA: MedXQA denotes the MedXpertQA-Text benchmark.The caption defines MedXQA as an abbreviated benchmark name.
  • MedQA: SGPQA denotes the SuperGPQA-Medical benchmark.The caption defines SGPQA as an abbreviated benchmark name.
  • MedQA: The overall comparison summarizes model performance across the eight medical text benchmarks.The overall comparison is presented separately from the per-benchmark comparisons.

MedMNIST … VQA-RAD

Across the merged sections, Hulu-Med benefits from more training data, modality re-balancing, token pruning, progressive curriculum learning, and larger language-model backbones. Qualitative examples further demonstrate medical understanding across text, video, 2D, and 3D inputs, including diagnosis and report generation.

  • VQA-RAD: Increasing the training-data proportion consistently improves model performance, demonstrating the effectiveness of data scaling.
  • VQA-RAD: Re-balancing improves performance on rare modalities while preserving strong capabilities on common modalities.
  • VQA-RAD: 55% token reduction improves token efficiency for medical vision-language inputs.
  • VQA-RAD: Progressive curriculum training consistently outperforms mixed-stage training, while 2D-only training achieves competitive results on 3D and video benchmarks.The mixed-stage approach incurs significant performance drops, whereas the 2D-only model rivals much larger specialized models on 3D volumetric and dynamic video benchmarks.
  • MedQA: Performance scales with increasing model parameters on the Qwen3 series LLM backbone.
  • VQA-RAD: Qualitative outputs cover text diagnosis, video procedure understanding, and 2D and 3D medical interpretation.Examples include rare-disease differential diagnosis, surgical dissection descriptions, radiographic summaries, histopathological findings, pneumonia assessment, pleural effusions, and abdominal CT findings.
  • VQA-RAD: Complex medical diagnosis reasoning identifies lateral collateral ligament complex repair or reconstruction as the key contributor to elbow rotatory stability.The explanation attributes this role to the lateral collateral ligament complex’s importance in resisting external rotation forces.

B Ground Truth

The ground-truth examples cover multimodal clinical reasoning across imaging, surgical video, emergency advice, documentation, and rare-disease diagnosis. They include both specific clinical answers and criteria-based evaluations of response completeness.

  • Clinical reasoning: Gottron’s papules support dermatomyositis with elevated creatine kinase as the likely laboratory abnormality.The passage identifies the skin finding as characteristic of dermatomyositis and links muscle inflammation to elevated CK.
  • Clinical reasoning: The imaging answers identify a small-bowel transition point in the left lower quadrant and classify arterial-wall calcification as a refractory lesion.Both answers are grounded in CT findings: clustered dilated loops and persistent or recurrent calcification-associated disease.
  • Video understanding: Surgical-video ground truth distinguishes robotic dissection, clip-assisted closure, and laparoscopic gallbladder removal.The passages describe a da Vinci Si robotic arm, subsequent suturing with U-shaped clips, and a laparoscopic cholecystectomy.
  • Video understanding: The visual-description ground truth rewards accurate procedure identification and detailed reporting of colors, textures, and instruments.It specifically cites pinkish-red and darker-red colors, smooth or irregular textures, and metallic surgical tools among the expected details.
  • Clinical assistance: Additional ground truth includes the coronary sinus as the site of lowest oxygen content, emergency activation for an unresponsive person, and structured cardiac-rehabilitation documentation.The emergency criteria emphasize contacting EMS and monitoring responsiveness, while the rehab template records vital signs, exercise duration, intensity, and comments.
  • Clinical assistance: The examples also include asthma-management reasoning, repeated identification of glutaric acidemia type I, and a broad differential-diagnosis case.One documentation criterion flags omission of medication changes as a limitation in cardiac-rehabilitation notes.
Loading 2510.08668v2…