Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

Introduction

• Hulu-Med addresses fragmented, task-specific medical AI with a unified architecture for text, 2D images, 3D volumes, and video, alongside an end-to-end transparent, reproducible pipeline.Its public and synthetic corpus contains 16.7 million samples spanning 12 anatomical systems and 14 imaging modalities.Source: Introduction · pp. 2–3

Overview of Hulu-Med

• A shared patch-based encoder and progressive three-stage curriculum support text-only and multimodal tasks, while medical-aware token reduction cuts visual tokens by approximately 55%.The corpus has 9 million medical multimodal samples, 4.9 million medical text QA pairs, 1.3 million general multimodal samples, and 1.5 million general text QA pairs; five synthesis pipelines generate instruction-aligned visual-text data.Source: Introduction · p. 2, Overview of Hulu-Med · pp. 3–4

Evaluation on 2D Medical Vision–Language Understanding

• Hulu-Med surpassed open-source models on 10 of 11 2D benchmarks and proprietary models on 8, with all variants leading on three report-generation benchmarks.On MIMIC-CXR, Hulu-Med-7B achieved a RaTEScore of 57.0 versus MedGemma-4B/27B at 51.3; on MedMNIST-2D, its average accuracy exceeded 85%, while GPT-4o attained less than 40%.Source: Evaluation on 2D Medical… · p. 5

Evaluation on 3D Medical Vision–Language Understanding

• Hulu-Med achieved state-of-the-art performance on open- and closed-ended 3D VQA, outperforming specialized 3D and general-purpose models on M3D and excelling on 3D-RAD.Hulu-Med-7B exceeded the best baseline by 22.8% on longitudinal temporal diagnosis; all variants led on AMOS-MM conventional generation metrics and RaTEScore.Source: Evaluation on 3D Medical… · p. 6

Evaluation on Medical Video Benchmarks

• On zero-shot MedFrameQA, Hulu-Med outperformed leading proprietary models in accuracy and variance as frame counts increased, and surpassed video foundation models in accuracy and recall on two surgical benchmarks.It led open-source models on SurgeryVideoQA with 30.1%, versus Lingshu-32B at 29.9%, while GPT-4o scored 44.8%.Source: Evaluation on Medical Video Benchmarks · p. 6

Evaluation on Medical Text-Only Benchmarks

• Across eight text benchmarks, Hulu-Med-32B averaged 65.9% accuracy, exceeding DeepSeek-V3 (59.8%) by 6.1 points and approaching o3-mini (67.9%) within 2.0 points.Performance improved from 7B to 32B across all benchmarks, including gains of 4.6 percentage points on MedXQA, 7.3 on Medbullets, 10.7 on SGPQA, and 6.9 on MedQA. Hulu-Med-32B nevertheless trailed Claude Sonnet 4 by 9.4 points on MedXpertQA, 11.4 on Medbullets, and 11.7 on MedQA.Source: Evaluation on Medical Text-Only… · p. 7
• Hulu-Med-32B scored 75.13% on MMedBench, surpassing GPT-4 (74.27%), matched GPT-o1 on HealthBench, and with CoT prompting surpassed proprietary models in seven of eight RareBench scenarios.On HealthBench, Hulu-Med-32B scored 41.6 versus GPT-4o at 32.0; Hulu-Med-32B-Thinking scored 78.41% on MMedBench, above Gemini 2.5 Flash (77.63%) and Claude Sonnet 4 (76.04%).Source: Evaluation on Medical Text-Only… · pp. 7–8

Analysis of Model Design and Data Curation

• Hulu-Med outperformed five modality-specific models, performance improved with more training data, and the best tested mixture was 3:1 medical-to-general and 1:1 text-to-multimodal.Removing any domain or modality component reduced performance, while synthetic long captions improved OmniMedVQA accuracy and generated CoT data boosted textual and multimodal reasoning.Source: Analysis of Model Design and Data… · p. 8
• Approximately 55% fewer visual tokens leave surgical-video performance effectively unchanged and cause minimal degradation on 3D benchmarks; 7B/32B training takes roughly 4,100/38,000 GPU-hours.The token reduction applies to 3D and video inputs; the training-hour estimates are on 80G-memory GPUs.Source: Analysis of Model Design and Data… · p. 8

Discussion

• Hulu-Med was trained in a three-stage pipeline with an open corpus of 16.7M samples, and publicly releases all data pipelines, training code, and model parameters to ensure reproducibility.Source: Discussion · pp. 8–9
• Hulu-Med achieves leading performance across 30 diverse medical benchmarks while unifying text, 2D/3D images, and video understanding.The discussion describes this as state-of-the-art performance across medical text, imaging, and video tasks.Source: Discussion · pp. 8–9
• A unified visual encoder, adaptive token reduction, and progressive 2D-to-3D/video training enable holistic understanding with computational efficiency.The design uses universal visual patches and 2D RoPE for variable-length multimodal inputs; shorter sequences use bilinear interpolation, while longer ones use medical-aware pruning. Progressive training outperforms mixed-modality training.Source: Discussion · p. 9
• Hulu-Med shows clinical-use potential across benchmarks and case studies, including more accurate, less hallucinatory reports than Med-Gemma and hour-long video processing with 43% lower GPU memory.The discussion also reports multilingual and multi-turn dialogue, rare-disease diagnosis, detailed surgical-video captions, and prompted reflective reasoning with self-correction, without reinforcement learning.Source: Discussion · p. 9
• Future work should add genomic and molecular inputs, broaden public-data aggregation, strengthen reasoning and continual pretraining, and validate integration into specialist models and clinical workflows.The authors specifically propose large-scale reinforcement learning on diverse long-CoT and long-horizon data, alongside further clinical validation.Source: Discussion · pp. 9–10

Model Architecture

• The unified architecture processes text, 2D images, 3D volumes, and videos through a visual encoder, text tokenizer, multimodal projector, and LLM decoder.It generates textual responses within a single end-to-end framework.Source: Model Architecture · p. 27
• A 27-layer ViT with 1152 hidden size, 4304 intermediate MLP size, and 16 attention heads encodes images, volume slices, and video frames as 16×16-pixel patches.3D volumes are decomposed into slices and videos sampled into frames before patching.Source: Model Architecture · p. 27
• 2D RoPE encodes height and width positions independently in self-attention, enabling arbitrary image resolutions and aspect ratios without learned positional embeddings.The first and second halves of the features encode height and width, respectively.Source: Model Architecture · p. 27
• For 3D and video inputs, 2×2 pooling reduces tokens per plane fourfold, then adjacent-slice or frame pruning removes redundant patches, reducing final tokens by up to 60%.Video patches are pruned when their embedding L1 distance from the corresponding patch in the previous frame is below τ = 0.1; pruning is dynamic during the forward pass and maintains comparable performance.Source: Model Architecture · p. 27
• Text uses the LLM backbone’s 152,064-token BPE tokenizer, while a two-layer MLP projector maps visual patch embeddings into the LLM embedding space.The projector bridges visual and text representations.Source: Model Architecture · p. 27
• Hulu-Med-7B uses Qwen2.5-7B-Instruct; concatenated text and projected visual embeddings are processed autoregressively for diverse generative tasks without task-specific modifications.The framework also includes Hulu-Med-14B with Qwen3-14B and Hulu-Med-32B with Qwen2.5-32B backbones.Source: Model Architecture · p. 28

Training Strategy

• The three-stage curriculum combines distinct public-data mixtures and targeted synthetic pipelines to address limited visual-instruction data and weak integration of general and medical knowledge.Stages 1 and 2 consolidate 2D single-image competence; stage 3 introduces interleaved multi-image contexts and spatiotemporal reasoning over 3D volumes and videos.Source: Training Strategy · p. 28

Stage 1: Vision-Language Alignment

• Stage 1 aligns the vision encoder and LLM using 1.4 million public image-text pairs for short caption generation, while keeping the LLM backbone frozen.Only the multimodal projector and vision encoder are fine-tuned, at learning rates of 1 × 10−3 and 1 × 10−5, respectively.Source: Training Strategy · p. 28

Stage 2: Medical Multimodal Pre-training

• Stage 2 uses 4.9 million samples for long-form caption generation and open-ended question answering, including a 2.6 million-sample public corpus with approximately 10% general-domain text.To address underrepresented modalities and sparse annotations, the team synthesizes 2.3 million long captions, including 1.4 million rewrites of short captions and captions generated and ranked for images lacking annotations; all components are trainable.Source: Training Strategy · p. 28

Stage 3: Mix-Modality Instruction Tuning

• Stage 3 trains on 10.5 million instances, including 5.9 million text-based and 4.5 million multimodal instructions across multi-image, interleaved, 3D, and video formats.Tasks include VQA and classification as well as MRG and CoT reasoning. Synthetic additions include 45K multilingual CoT samples, 600K VQA pairs, and 20K video captions; all parameters remain trainable, with the LLM learning rate increased to 5×10−5.Source: Training Strategy · p. 29

Text-Based Medical Reasoning and Generalization

• Text-only evaluation tests medical knowledge and expert reasoning across eight benchmarks, using accuracy for multiple-choice questions.The benchmarks span professional medical exams, biomedical-literature factual recall, and expert-level reasoning tasks.Source: Evaluation Framework and Metrics · p. 29
• Deployment evaluation covers multilingual understanding, rubric-based clinical dialogue, and rare-disease diagnosis.MMedBench covers six languages; Gemini-2.5-Pro judges HealthBench dialogues, while ChatGPT-4o-latest assesses RareBench differential diagnoses.Source: Evaluation Framework and Metrics · p. 29

2D Medical Image Understanding

• 2D evaluation spans seven VQA benchmarks, MedMNIST classification, and chest-radiograph report generation across three datasets.Closed-ended VQA and MedMNIST use accuracy; report generation uses BLEU (1-4), ROUGE-L, METEOR, recall, and RaTEScore.Source: Evaluation Framework and Metrics · pp. 29–30

3D Volumetric and Spatiotemporal Analysis

• 3D evaluation tests spatial reasoning and clinical tasks from descriptive generation to static and longitudinal diagnosis, plus report generation.Closed-ended reasoning uses accuracy, open-ended tasks use recall, and AMOS-MM reports are assessed with BLEU, ROUGE-L, METEOR, and RaTEScore.Source: Evaluation Framework and Metrics · p. 30

Surgical and Medical Video Comprehension

• Video benchmarks assess surgical phases, instruments, actions, and multi-frame temporal reasoning across surgical and medical sequences.Closed-ended Cholec80-VQA and MedFrameQA use accuracy; descriptive answers use recall, and ChatGPT-4o-latest provides semantic evaluation for open-ended or supplementary assessment.Source: Evaluation Framework and Metrics · p. 30

Text Data (5.9M)

• Text data included Apollo-Pre factoid QA (1,859,880), II-Medical SFT long-CoT (700,000), and sampled Miriad clinical dialogue (1,255,356).Source: Text Data (5.9M) · p. 36
• Other listed text datasets were medical-o1 reasoning (65,531), AlpaCare medical instruction (52,002), and MMedC multilingual QA (45,048).Source: Text Data (5.9M) · p. 36

Multimodal Data (4.5M)

• Multimodal data included PubMedVision 2D VQA (646,750), MedMNIST classification (74,689), MIMIC-CXR-MRG report generation (242,310), M3D-Cap 3D captioning (31,928), Cholec80-Cap video captioning (17,010), and Cholec80-VQA video QA (24,829).Source: Multimodal Data (4.5M) · p. 36

• The table documents prompt templates for different task types during inference.Source: GRAND TOTAL 10,457,117 · p. 37

Task Type Prompt for Direct Answering Prompt for CoT Reasoning

• Multiple-choice prompts provide a question and options, with direct-answer instructions requiring only the selected option’s letter.The templates include both “Question” and “Multiple-Choice Question” forms.Source: Task Type Prompt for Direct Answering… · p. 37
• Chain-of-thought prompts ask the model to reason step by step and place its final answer within \boxed{}.The template appears both with and without an explicit “Please” before the reasoning instruction.Source: Task Type Prompt for Direct Answering… · p. 37
• Task-specific answer formats request yes/no for judgment, a single word or phrase for close-ended questions, and concise responses for open-ended questions.Source: Task Type Prompt for Direct Answering… · p. 37
• The report-generation prompt asks for an image-based report divided into Findings and Impression sections.Source: Task Type Prompt for Direct Answering… · p. 37
• Extended Table 8 is described as comparing MedFrameQA accuracy by frame count and modality, with SD measuring accuracy variation across 2–5 frame counts.The supplied passages describe the table’s scope but provide no result values.Source: Task Type Prompt for Direct Answering… · p. 38
• The appendix catalogs datasets across three stages and notes that synthetic datasets and those requiring specific permissions are marked Credentialed Access.The catalog includes medical and general text or multimodal datasets with listed access or license information.Source: Task Type Prompt for Direct Answering… · p. 39
• The appendix labels a Multilingual Chain-of-Thought component and lists dataset groups spanning medical QA, report generation, and video tasks.Source: Task Type Prompt for Direct Answering… · p. 40

VQA-RAD

• Extended Figure 3 compares 7B-scale VLMs on medical multimodal benchmarks using three seeds at temperature 0.6, excluding MMMU because of EvalAI submission limits.The supplied caption does not provide benchmark results.Source: VQA-RAD · p. 42

MedQA

• The evaluation compares 7B-scale VLMs across eight medical text benchmarks, averaging each result over three runs at decoding temperature 0.6.The figure also presents an overall comparison across the eight benchmarks; the supplied caption does not report scores or identify a winner.Source: MedQA · p. 44

VQA-RAD

• Performance improves as training data increases, while re-balancing improves rare-modality performance and maintains strong capabilities on common modalities.Source: VQA-RAD · p. 45
• 55% token reduction is reported, and Extended Figure 7 covers performance after token pruning, 3D-RAD sub-task performance across pruning coefficients, and training cost at three model scales.Source: VQA-RAD · p. 46
• Progressive curriculum training consistently outperforms mixed-stage training, which incurs significant performance drops.The comparison is presented as evidence for hierarchical learning.Source: VQA-RAD · p. 47
• A model trained exclusively on 2D data achieves competitive results on 3D and video benchmarks, rivaling much larger specialized models.The figure describes this as an emergent cross-modal capability.Source: VQA-RAD · p. 47
• Adding 3D and video data does not compromise 2D performance and further enhances 2D learning.Source: VQA-RAD · p. 47

MedQA

• Performance scaling with increasing model parameters is examined for the Qwen3 series LLM backbone in MedQA.Source: MedQA · p. 48

VQA-RAD

• Qualitative examples show video descriptions of surgical actions and 2D image interpretations, including a response describing controlled tissue dissection and a chest-image summary of no acute disease.Source: VQA-RAD · pp. 50–51
• A 3D report example identifies mild intestinal wall thickening and likely ingested foreign bodies, while a complex elbow case is answered with lateral collateral ligament complex repair or reconstruction.Source: VQA-RAD · pp. 53–54

Medical Diagnosis Task

• Gottron’s papules suggest dermatomyositis, for which elevated creatine kinase is the most likely abnormal test.The example rules out rheumatoid factor, zinc, thyrotropin, and CD4 count as less relevant to the presentation.Source: B Ground Truth · p. 54

3D Medical Reasoning

• The 3D reasoning examples locate a small-bowel transition point in the left lower quadrant and classify visible arterial calcification as a refractory lesion.The explanations associate the transition point with a cluster of dilated loops and refractory status with persistent or recurrent calcification.Source: B Ground Truth · p. 55

Video Caption Generation

• A video-caption example describes a laparoscopic gallbladder removal, while another narrates robotic-assisted dissection followed by clipping and suturing.The robotic-surgery description follows the procedure from instrument engagement with tissue through closure.Source: B Ground Truth · p. 56

Long Video Understanding

• The long-video example gives a chronological description of an internal surgical scene, but evaluation notes it omits suturing and uses vague anatomical localization.The evaluation says the description could have specified the nasal septum or turbinates.Source: B Ground Truth · p. 57

Multi-linguistic Benchmark

• Multilingual examples answer questions about cardiolipin’s membrane location, nimesulide’s COX-1 activity, and which organization is not directly involved in workplace suicide prevention.The supplied answers identify the inner mitochondrial membrane, deny active COX-1 inhibition, and select the regional mental health and welfare center, respectively.Source: B Ground Truth · p. 58
• Further multilingual examples identify 24 hours for reporting a hospital infection outbreak, the coronary sinus as lowest in oxygen content, and pancreatic juice as the highest-pH secretion.The French-language answer attributes pancreatic juice’s high pH to its sodium bicarbonate content.Source: B Ground Truth · p. 59

HealthBench

• For an unresponsive older adult with a pulse and slow breathing, the HealthBench example advises immediate emergency services and monitoring while awaiting help.Its response also advises maintaining airway patency, avoiding food or drink, and staying with the person; evaluation highlights emergency activation and continued monitoring.Source: B Ground Truth · p. 60

B Ground Truth

• Emergency guidance should begin with a clear directive to seek emergency care, contact EMS, and locate an AED if the patient is pulseless or not breathing.The criteria also call for concise checks of breathing, pulse, and responsiveness, and discourage distracting advice to look for clues.Source: B Ground Truth · p. 60
• Cardiac rehabilitation notes are expected to document exercise type, duration, intensity, MET level, tolerance, key vitals, symptoms, and follow-up plans.The example note uses headings for exercise tolerance, pre- and post-exercise vitals, medication changes, symptoms, overall progress, and next steps.Source: B Ground Truth · p. 61
• The cardiac rehabilitation response is flagged for failing to document medication changes, including new, discontinued, or dose-changed medications.The criterion notes that medications such as beta blockers can affect exercise tolerance or heart-rate response.Source: B Ground Truth · p. 61
• Glutaric acidemia type I is the reference diagnosis for the phenotype combining macrocephaly, dystonia, cerebral atrophy, glutaric aciduria, and subdural hemorrhage.The model’s differential also lists it first among ten diagnoses.Source: B Ground Truth · p. 63
• Giant cell arteritis (temporal arteritis) is the reference diagnosis for the phenotype that includes jaw claudication, elevated ESR, headache, and weight loss.The supplied answer key repeats this diagnosis.Source: B Ground Truth · p. 64

Rare disease diagnosis example 1

• The response ranks giant cell arteritis first for jaw claudication, headache, elevated ESR, leukocytosis, and systemic symptoms, while presenting nine alternative diagnoses.It notes that jaw claudication is more specific to GCA than PMR and that several alternatives fit less well.Source: B Ground Truth · p. 64

Rare disease diagnosis example 2

• For congenital craniofacial and ear anomalies with choanal atresia and developmental delay, the response ranks Treacher Collins first and CHARGE second.It also lists eight other syndromic possibilities, noting that CHARGE is associated with choanal atresia, hearing loss, microcephaly, and feeding difficulties.Source: B Ground Truth · p. 65

Rare disease diagnosis example 3

• The response identifies methylmalonic acidemia with homocystinuria as the leading diagnosis for a phenotype combining neurologic, congenital, and metabolic findings.It lists homocystinuria, propionic acidemia, biotinidase deficiency, and other metabolic or multisystem disorders as alternatives; methylmalonic aciduria and hyperhomocystinemia are described as most consistent with the leading diagnosis.Source: B Ground Truth · p. 66