Source-linked AI summary
MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs
Baorong Shi, Bo Cui, Boyuan Jiang, Deli Yu, Fang Qian, Haihua Yang, Huichao Wang, Jiale Chen, Jianfei Pan, Jieqiong Cao, Jinghao Lin, Kai Wu, Lin Yang, Shengsheng Yao, Tao Chen, Xiaojun Xiao, Xiaozhong Ji, Xu Wang, Yijun He, Zhixiong Yang
TL;DR
Medical VLMs still face gaps in long-tail coverage, evidence-grounded reliability, reasoning data, and reproducible evaluation for clinical use. MedXIAOHE addresses these gaps with entity-aware continual pretraining, synthesized reasoning and caption data, evidence-grounded generation, tool-augmented reasoning, and unified evaluation, reporting broad benchmark strength and practical workflow generalization. Persistent challenges include hallucinations, distribution shift, and multi-step reasoning reliability.
Problem
Real-world medical VLMs require broader long-tail coverage, reliable evidence use, faithful long-form generation, diverse reasoning data, and more reproducible evaluation.
Method
MedXIAOHE combines entity-aware continual pretraining, dense caption and reasoning-data synthesis, evidence-grounded generation, reinforcement learning, tool-augmented interaction, and unified evaluation.
Results
MedXIAOHE reports strong integrated capabilities across medical text, medical images, and clinical understanding, with generalization to evidence-grounded, tool-assisted multi-step workflows.
Takeaways & Limitations
The model connects broad medical benchmark performance with workflows requiring evidence grounding and multi-step reasoning with external tools.
Takeaways & Limitations
Current medical VLMs still require better hallucination control in long-form generation, reliability under distribution shift, and multi-step medical reasoning.
Abstract
from arXiv · showhide
We present MedXIAOHE, a medical vision-language foundation model designed to advance general-purpose medical understanding and reasoning in real-world clinical applications. MedXIAOHE achieves state-of-the-art performance across diverse medical benchmarks and surpasses leading closed-source multimodal systems on multiple capabilities. To achieve this, we propose an entity-aware continual pretraining framework that organizes heterogeneous medical corpora to broaden knowledge coverage and reduce long-tail gaps (e.g., rare diseases). For medical expert-level reasoning and interaction, MedXIAOHE incorporates diverse medical reasoning patterns via reinforcement learning and tool-augmented agentic training, enabling multi-step diagnostic reasoning with verifiable decision traces. To improve reliability in real-world use, MedXIAOHE integrates user-preference rubrics, evidence-grounded reasoning, and low-hallucination long-form report generation, with improved adherence to medical instructions. We release this report to document our practical design choices, scaling insights, and evaluation framework, hoping to inspire further research.
1 Introduction
Medical VLMs could support multimodal clinical work, but real-world deployment requires broader coverage, reliable evidence use, stable long-form generation, and reproducible evaluation. MedXIAOHE addresses these needs through entity-aware training, reasoning-oriented post-training, evidence-grounded generation, and a unified benchmark framework.
- Clinical decision-making combines symptoms, histories, examinations, images, and reports, motivating medical foundation models that handle heterogeneous multimodal inputs.
- Real-world medical models must address long-tail cases, heterogeneous inputs, evidence discipline, stable behavior, and reliable long-form generation.
- Medical multimodal evaluation remains difficult because benchmarks use inconsistent prompting, scoring, contamination control, and coverage of clinically important capabilities.
- MedXIAOHE organizes heterogeneous medical corpora around an entity-centric taxonomy to expand knowledge coverage and reduce long-tail gaps across specialties and modalities.
- Its evidence-grounded report pipeline combines drafting, entity extraction and retrieval, critique-and-rewrite, and rubric-trained rewards to suppress subtle hallucinations and over-interpretation.
- The Unified Med-VLM Benchmark consolidates 30+ public benchmarks with standardized prompting, scoring, and decontamination, complemented by targeted in-house evaluations.
- The report highlights strong end-to-end results across more than 30 public and in-house benchmarks, alongside reliable mixed-evidence reasoning and tool-augmented diagnostic traces.
2 Overview and Architecture
MedXIAOHE extends a Seed-based multimodal decoder to unify medical text, images, OCR-style understanding, and long-form generation under one interactive interface. It reports strong performance across visual, imaging, diagnostic, medical-text, report-generation, and instruction-following evaluations.
- Architecture: MedXIAOHE uses Seed-ViT, a multimodal connector, and an autoregressive LLM that generates from interleaved visual and textual context.
- Architecture: The same interface handles histories, guidelines, reports, clinical photos, medical images, medical-document understanding, and structured or long-form outputs.
- Design choice: The model extends Seed through continual training rather than architectural redesign, emphasizing domain data quality, long-tail coverage, and post-training alignment.
- Evaluation: MedXIAOHE is evaluated in thinking mode with greedy decoding against GPT-5.2 Thinking, Gemini 3.0 Pro, and Gemini 2.5 Pro.
- Results: It outperforms SOTA models on in-house VQA, caption, OCR, and MMMU medical-series evaluations, indicating strong medical visual understanding.
- Results: The model shows strengths on heterogeneous radiology and pathology VQA tasks, including SLAKE, PATH_VQA, and OmniMedVQA.
- Results: It delivers competitive diagnostic reasoning on RareBench and MedBrowseComp while remaining strong on DiagnosisArena and HLE-Med.
- Results: MedXIAOHE demonstrates strong medical knowledge and reasoning across PubMedQA, MedQA, MedBullets, CMExam, MedMCQA, and HealthBench.
3 Continual Pre-training
MedXIAOHE’s continual pre-training pipeline combines corpus cleaning, entity-aware organization, and dense multimodal instruction construction. These components target balanced medical knowledge coverage, long-tail entities, and clinically grounded image understanding.
- Corpus cleaning: The corpus pipeline combines hash-based deduplication, rule-based filtering, normalization, and model-based quality control for high-quality pre-training data.The broader collection process applies source-specific processing before the three-stage cleaning workflow.
- Medical Entity Tree: The Medical Entity Tree balances entity training, quantifies knowledge coverage, and guides targeted acquisition of sparse medical domains.Its construction progresses from efficient extraction to joint typing, hierarchical clustering, and controlled tree attachment.
- Medical Entity Tree: The resulting five-tier taxonomy contains 1.4 million entities and achieves Forward Coverage AMCS scores above 0.95 across all evaluated datasets.Reported scores are 0.96, 0.95, and 0.97.
- Medical Entity Tree: Lower Backward Coverage AMCS scores, including 0.68 for Clinical Knowledge Data and 0.79 for CMeKG, indicate additional long-tail concepts in the tree.The analysis attributes these gaps to fine-grained entities absent from the baselines.
- Image knowledge: The caption pipeline transforms structured annotations into dense narratives and diverse descriptive, diagnostic, localization, and reasoning instruction-response pairs.This design goes beyond sparse labels to support fine-grained cross-modal alignment and complex clinical instructions.
- Image knowledge: A rubric-based Generative Reward Model filters generated captions using criteria including anatomical localization, morphological accuracy, and evidence-based assertions.Human-annotated errors provide rubrics for training the GenRM through supervised fine-tuning followed by reinforcement learning.
4 Mid-Training
Mid-training strengthens advanced medical reasoning by synthesizing structured, diverse, and verifiable reasoning data, then trains tool-augmented interaction for multi-step clinical tasks.
- Mid-training targets atomic combinational skills that integrate capabilities such as visual grounding and tool use into coherent workflows.
- Mid-Training Data Construction: The data synthesis engine combines diverse corpora, knowledge graphs, and multiple synthesis strategies to address the scarcity of explicit medical reasoning data.It supports multi-turn QA, detailed image captioning, and visual grounding, while generating multi-hop and verifiable reasoning examples.
- Mid-Training Data Construction: Multi-expert reject sampling distills diverse reasoning trajectories across visual examination, medical imaging, diagnosis, and text.The pipeline uses expert models with varying cognitive configurations and best-of-N rejection sampling.
- Mid-Training Data Construction: Structured and personalized Chain-of-Thought pipelines bind logical deduction to low-hallucination visual captions and preserve visual fidelity in perception-intensive tasks.The personalized pipeline separates logical content from cognitive style and uses a visual-grounded reasoning core.
- Agentic Reasoning: Agentic training uses general search, medical search, and image-editing tools to create challenging multi-step QA pairs and structured interaction traces.Agentic RL trains the model to interact, observe, and summarize tool definitions, thought processes, and execution results.
- Agentic Reasoning: The filtering pipeline produces a high-quality RL dataset focused on diagnostically challenging, tool-enhanced cases.
5 Post-training
Post-training combines expert-informed supervised fine-tuning with reinforcement learning to improve instruction following, clinical reasoning, safety alignment, and iterative capability integration.
- Post-Training Pipeline: Post-training applies the mid-training synthesis pipeline with expert annotations, followed by SFT and RL, while rejection sampling supplies hard negatives for iterative refinement.
- Supervised Fine-Tuning: SFT uses the highest-quality data to provide broad task coverage, strong instruction following, and compliance with complex medical constraints.Instruction data spans departments, task types, single-turn and multi-turn interactions, and explicit constraint-checking rationales.
- Reinforcement Learning: RL targets difficult clinical cases after filtering candidate samples and constructing discriminative evaluation points for clearer reward signals.
- Hybrid Reward System: The hybrid reward system routes samples to task-appropriate rule-based or rubric-based graders and combines their signals into a unified objective balancing accuracy, coherence, and safety.
- RFT-Enhanced Curriculum Reinforcement Learning: The RFT-enhanced iterative curriculum alternates supervised capability distillation with curriculum-based policy optimization under dynamic multi-objective constraints.It uses iterative RFT and RL cycles rather than static mixing or rigid sequencing.
- RFT-Enhanced Curriculum Reinforcement Learning: Iterative cycling improves the hybrid reward signal while avoiding the gradient interference of simultaneous training and the inflexibility of sequential paradigms.
6 Unified Med-VLM Benchmark
The Unified Med-VLM Benchmark standardizes evaluation across 30+ public benchmarks and supplements them with deployment-focused in-house tests for clinically important capabilities.
- Benchmark Design: 30+ public benchmarks are consolidated into a Unified Med-VLM Benchmark with a standardized evaluation harness.The framework addresses fragmented benchmark subsets, prompting styles, scoring scripts, and data-hygiene assumptions.
- Benchmark Design: The suite emphasizes broad capability coverage, reproducible protocols, and deployment relevance beyond answer correctness.
- Evaluation Harness: Evaluation is normalized through task families, capability-aware prompts, deterministic closed-form parsing, task-appropriate open-form scoring, and macro-averaged category summaries.
- Capability Categories: Six capability categories cover visual diagnosis, medical imaging, diagnosis, medical text, medical reports, and instruction following.Together they assess visual grounding, multimodal fusion, differential diagnosis, multilingual reasoning, faithful generation, and long-context controllability.
- In-house Benchmarks: In-house evaluations target deployment-critical axes that public benchmarks under-measure, especially real-world patient images and medical documents.
- In-house Benchmarks: The in-house suite includes more than 100k clinical-image VQA questions, distorted real-world report OCR images, and clinically structured lesion-caption evaluations.
7 Conclusion
MedXIAOHE provides broad multimodal medical capabilities and generalizes to evidence-grounded, tool-assisted workflows. The authors identify hallucination, distribution-shift, and multi-step reasoning reliability as continuing challenges.
- MedXIAOHE is competitive across a broad range of medical benchmarks and integrates medical text, images, and clinical understanding.
- The model generalizes to practical workflows requiring evidence-grounded outputs and multi-step reasoning with external tools.These behaviors support movement from isolated tasks toward clinician-facing systems in realistic settings.
- Current medical VLMs still face limitations in long-form hallucination control, reliability under distribution shift, and multi-step medical reasoning.The paper identifies deeper evidence grounding, more reliable tool use, and broader concept and modality coverage as ongoing work.
- The accompanying evaluation framework is intended to encourage standardized and clinically meaningful assessment of trustworthy medical multimodal foundation models.
8 Contributions
The listed authors are ordered alphabetically by first name.
- Author names are arranged alphabetically by first name.
A.1 Medical DeepResearch
The report highlights first-place performance in both Cure-Bench reasoning tracks and illustrates tool-assisted medical reasoning through evidence retrieval and image magnification.
- #1 place was achieved in both the Internal Reasoning and Agentic Reasoning tracks of Cure-Bench@NeurIPS 2025.The benchmark evaluates AI models in drug decision-making and treatment planning.
- Tool-assisted reasoning retrieves and verifies evidence before producing an answer.
- Magnification-assisted reasoning analyzes the original image, calls a zoom tool, inspects the crop, and concludes the diagnosis.
A.3 Think with Grounding
The grounding workflow analyzes chest X-ray regions systematically and outputs abnormality locations, identifying cardiomegaly and mild vascular congestion while describing other structures as unremarkable.
- The lung fields are described as clear across the right and left upper, middle, and lower zones.
- The cardiovascular assessment identifies a widened mediastinum, enlarged cardiac silhouette, and prominent aortic arch.
- The model reports normal diaphragms and sharp costophrenic angles without pleural effusion.
- The model concludes that the primary abnormalities are cardiomegaly and mild vascular congestion.It reports clear lungs without consolidation, effusion, or pneumothorax.
- Grounding reasoning takes a raw X-ray, applies logical analysis, and outputs bounding boxes for abnormalities.
A.4 Expert-Level Complex Diagnostic
The complex diagnostic example links pancytopenia and splenomegaly with hairy lymphoid cells on a peripheral smear to Hairy Cell Leukemia, identified as TRAP-positive.
- The model identifies fine, irregular hair-like projections around abnormal lymphoid cells as the key smear feature.
- Pancytopenia, splenomegaly, and hairy cells on the peripheral smear support a diagnosis of Hairy Cell Leukemia.
- The example specifies TRAP positivity as a characteristic associated with Hairy Cell Leukemia.
A.5 Real-World Clinical Report Generation
MedXIAOHE generates structured radiology reports by moving from task understanding and image observation through clinical correlation to separate Findings and Impression sections. The example produces a concise report describing no acute cardiopulmonary or bony abnormality.
- Task Understanding: The workflow first identifies the required IU-XRAY format and separates objective Findings from clinical Impression.The prompt requires both sections, and the reasoning explicitly assigns observations to Findings and interpretation to Impression.
- Image Observation: Image observation examines the trachea, cardiomediastinal silhouette, lungs, pleural spaces, and bony structures across views.The chest X-ray analysis checks midline tracheal position, cardiomediastinal shape, infiltrates, effusion, pneumothorax, and bone integrity.
- Correlation & Reasoning: Clinical correlation links midline tracheal position and clear lungs to the absence of specific acute abnormalities such as tracheal deviation, pneumonia, edema, effusion, and pneumothorax.These links connect visual observations to medical interpretations before report generation.
- Structuring the Report: The report structure systematically summarizes each anatomical component in Findings and synthesizes them into an Impression.The example Impression states that no obvious cardiopulmonary abnormalities were observed.
- Structuring the Report: Figure 14 presents the broader structured-CoT pattern with three chest X-ray views, a prompt, stepwise reasoning, and final Findings/Impression output.This figure depicts the same input-to-report organization in a visual example.
A.6 Deformed Clinical Report Understanding
The deformed-report understanding workflow constrains OCR answers to image-supported information and uses targeted zooming to resolve a laboratory-table query. In the example, the model determines that an MCH value of 26.2 pg is below the 27–34 pg reference range.
- OCR QA: The OCR QA system requires image-grounded, direct answers without information beyond the image or additional explanation.It returns “NOT_FOUND” when no relevant information is present and otherwise preserves recognized content.
- Targeted Extraction: For the MCH query, the model locates the relevant table row and invokes ZOOM on the first image to improve inspection.The tool call targets “Mean Corpuscular Hemoglobin” with a bounding box and scale factor of 2.
- Result: 26.2 pg is below the 27–34 pg reference range, so the measured MCH is not within range.The result includes both the measured value and the reference interval.
- Result: Figure 14 illustrates precise extraction from a deformed clinical report by identifying the measurement-table row and using ZOOM to correct resolution and deformation issues.The example is presented as originally Chinese content.
B Evaluation Details
The evaluation details organize public medical benchmarks by capability and describe prompt-based procedures for constructing and attaching medical entities to a taxonomy. The framework includes visual, imaging, diagnostic, textual, reporting, and instruction-following evaluations.
- Benchmark Coverage: Public benchmarks are grouped into Visual Diagnosis & Image Recognition, Medical Imaging, Diagnosis, Medical Text, Medical Report, and Instruction Following.Tables 3–8 provide one benchmark group for each capability area.
- Entity Processing: Entity extraction identifies informative medical proper nouns, excludes non-medical entities, and splits overly long entity nouns when appropriate.The instructions also require ignoring meaningless spaces and special symbols.
- Entity Processing: Entity outputs use specific entity categories and names in a strict JSON format.The required output replaces EntityType and EntityName placeholders for each sentence.
- Entity Tree Attachment: The taxonomy-integration stage inserts valid medical entities into appropriate parent subcategories while preserving the existing tree structure.The entity tree and entity nouns are supplied as inputs to this attachment procedure.
- Conflict Resolution: Conflict resolution compares multiple parent paths using etiological dominance and specificity, then retains the more appropriate classification.The procedure calls for definition search, path comparison, and deletion of competing paths.