PaperlayerView exact revision on arXiv

Source-linked AI summary

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, Yu Rong

arXiv:2506.07044v4cs.CLcs.AIcs.CV

TL;DR

Medical MLLMs have limited medical knowledge coverage and reasoning, while existing curation approaches can produce insufficiently reliable data. Lingshu combines broad medical data curation with multi-stage training and achieves state-of-the-art performance across most evaluated medical tasks, with Lingshu-32B averaging 7.2 accuracy points above the second-best model across seven medical VQA tasks.

  • Problem

    Existing medical MLLMs have limited coverage beyond imaging and rely on curation processes that may lack sufficient supervision and diverse medical knowledge.

  • Method

    The paper curates multimodal, textual, and general-domain data, synthesizes captions, QA, and reasoning samples, and trains the medical-specialized Lingshu model.

  • Results

    Lingshu achieves state-of-the-art performance across most multimodal and textual medical VQA tasks and report generation, with Lingshu-32B averaging 7.2 accuracy points above the second-best model across seven medical VQA tasks.

  • Takeaways & Limitations

    The results support broad medical knowledge coverage and data quality as important components of Lingshu’s evaluated performance across medical tasks.

  • Takeaways & Limitations

    Lingshu still underperforms state-of-the-art proprietary models, and its generalization to broader medical tasks remains insufficiently explored.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

1 Introduction

Lingshu addresses medical MLLMs’ limited knowledge coverage, hallucination risks, and weak complex-case reasoning through curated multimodal and textual data, multi-stage training, RLVR exploration, and unified evaluation. It achieves strong performance across medical VQA, report generation, and related applications.

  • Motivation: Existing medical MLLMs lack broad domain knowledge, risk hallucinations, and struggle with complex cases requiring sequential decision-making beyond basic image-text correlations.Distillation from medical imaging prompts inadequately covers pharmacology, public health, and clinical contexts, while processes often lack additional supervision.
  • Data: Lingshu’s data curation pipeline combines multimodal medical, medical text, and general-domain data with synthesized captions, QA pairs, and chain-of-thought reasoning samples.The pipeline is designed to improve medical knowledge coverage and produce accurate training data.
  • Training: Lingshu uses multi-stage training in 7B and 32B variants to progressively infuse medical knowledge and improve problem-solving, while Lingshu-RL explores RLVR for multimodal medical reasoning.The training contribution includes medical-specific models and an investigation of reinforcement learning with verifiable rewards.
  • Evaluation: MedEvalKit unifies major multimodal and textual medical benchmarks to enable efficient, standardized, and fairer model assessment.The framework addresses the lack of accessible, unified evaluation environments in medical model development.
  • Results: Lingshu-32B outperforms the second-best model by an average of 7.2 accuracy points across seven medical VQA tasks, surpassing GPT-4.1 and Claude Sonnet 4.Lingshu achieves state-of-the-art performance across most multimodal and textual medical VQA tasks and report generation for both 7B and 32B configurations.

2 Data Curation

Lingshu’s data curation combines diverse medical and general-domain sources with quality-controlled synthesis of captions, OCR, VQA, and reasoning data. The resulting corpus contains 3.75M open-source and 1.30M synthetic medical samples spanning over 12 imaging modalities.

  • 2.1.1 Medical Multimodal Data Collection: The collection integrates medical multimodal data, unimodal medical texts and images, and general-domain datasets, all individually preprocessed and quality-filtered.Medical multimodal data includes captions, VQA, and radiology reports; unimodal data supports domain knowledge and synthesis, while general-domain data improves visual, language, and multimodal instruction following.
  • 2.2 Medical Data Synthesis: The synthesis pipeline targets long-form captions, OCR instructions, VQA, and distilled reasoning examples, with rigorous quality control for accuracy and reliability.These samples are designed to improve diagnostic detail extraction, medical-image text recognition, core medical VQA skills, and advanced reasoning.
  • 2.1.3 General Domain Data Collection: General-domain captioning and instruction datasets augment Lingshu’s general visual understanding, text generation, multimodal instruction following, and vision-language generalization.Sources include LLaVA-1.5, PixMo, OpenHermes-2.5, and ALLaVA.
  • 2.1.4 Data Cleaning Process: Medical data cleaning removes low-resolution, duplicate, overly short, and overly long image-caption samples while preserving concise instruction responses and allowing diverse cross-dataset instructions.Text data is also sanitized for identity information, direct diagnoses, and prescriptions, then deduplicated with min-hash LSH.
  • 2.2.1 Medical Long-form Caption Synthesis: Approximately 100K long-form captions are generated from multi-modal medical datasets through metadata preparation, region-of-interest grounding, factual annotation, doctor-preference annotation, and summarization.Stage-3 factual descriptions are prioritized during summarization to reduce conflicts, while expert instructions capture clinically important details beyond annotated regions.
  • 2.2.2 Medical OCR-based Instruction Data Synthesis: 50K high-quality OCR-based instruction samples are created by rendering validated biology and chemistry questions as images, retaining only answers that exactly match ground truth.Gemini-2.0-Flash-Thinking annotates detailed reasoning before validation.
  • 2.2.3 Medical VQA Data Synthesis: Medical VQA combines template-based questions grounded in annotated anatomy, abnormalities, and modalities with GPT-4o self-instruction from image-caption pairs, while GPT-4o-generated reasoning traces are validated against ground-truth answers.The combined approach improves question diversity and linguistic variation, and inconsistent reasoning samples are excluded.
  • 2.3 Dataset Summary: 3.75M open-source and 1.30M synthetic medical samples were curated, with strict benchmark-overlap deduplication and coverage of over 12 medical imaging modalities.A BiomedCLIP-based modality classifier inferred modalities for unlabeled images to characterize the training distribution.

3 Model Training

Lingshu is trained from Qwen2.5-VL-Instruct backbones through a staged medical adaptation pipeline spanning alignment, instruction tuning, and optional reinforcement learning. The design progressively expands medical vision-language alignment, clinical task coverage, and reasoning capabilities.

  • 3 Model Training: Lingshu uses Qwen2.5-VL-Instruct 7B and 32B backbones, whose instruction capabilities enable instruction-formatted data from the initial stage and improve training consistency and stability.The architecture combines an LLM, vision encoder, and MLP-based projector; instruct variants reduce alignment-data requirements and additional instruction-alignment overhead.
  • 3.1 Training Recipe: The training recipe follows four sequential stages—medical shallow alignment, medical deep alignment, medical instruction tuning, and medical-oriented reinforcement learning—producing Lingshu after the first three and Lingshu-RL after RL.The progression is designed to adapt the backbone from foundational medical vision-language learning toward practical task solving and reasoning.
  • 3.1.1 Medical Shallow Alignment: Medical shallow alignment freezes the LLM and fine-tunes the vision encoder and projector on concise medical image-caption data to establish efficient modality-to-text alignment.Coarse captions from PMC-OA and ROCO support rapid learning of general imaging characteristics while avoiding potential language-generation degradation from sparse text.
  • 3.1.2 Medical Deep Alignment: Medical deep alignment unfreezes all parameters and uses richer medical image-text data plus general-domain multimodal data to integrate medical knowledge while preserving structured visual and reasoning capabilities.The data expands modality diversity, caption complexity, and structural completeness, including synthetic pairs and formats such as charts, tables, graphs, mathematics, and scientific illustrations.
  • 3.1.3 Medical Instruction Tuning: Medical instruction tuning unlocks all parameters and combines clinical scenario queries, medical and general text, multimodal data, and complex multi-image or multi-turn reasoning tasks.The corpus covers diagnosis, examination, knowledge retrieval, report generation, localization, OCR, QA, and chain-of-thought data to counterbalance image-centric training and broaden medical competence.
  • 3.1.4 Medical-oriented Reinforcement Learning: Medical-oriented RL applies GRPO with verifiable rewards to encourage deliberate, autonomously discovered medical reasoning rather than answer memorization or teacher-guided chain-of-thought imitation.The curated 100K-example dataset balances multiple-choice and open-ended QA, limits binary-answer questions to approximately 5%, and uses format and accuracy rewards weighted 0.5 and 1.
  • 3.2 Implementation Details: Implementation uses AdamW, cosine scheduling, 100-step warm-up, 8,192-token sequences, batch size 1, and gradient accumulation 8; data packing is restricted to instruction tuning because it harms alignment-stage performance.The RL stage initializes from Lingshu, runs GRPO for one epoch, and produces Lingshu-RL with a 4,096-token limit, rollout batch 512, global batch 128, and learning rate 1e-6.

4 MedEvalKit: A Unified Medical Evaluation Framework

MedEvalKit provides a unified, reproducible framework for evaluating medical multimodal and textual models across diverse tasks. It combines broad benchmark coverage, standardized execution, and task-specific assessment protocols.

  • 4 MedEvalKit: A Unified Medical Evaluation Framework: MedEvalKit integrates mainstream medical benchmarks across multimodal QA, text-only QA, and report generation, supporting multiple question formats and unified model evaluation.It standardizes preprocessing, deployment, inference, and result assessment, combining rule-based evaluation with an LLM-as-a-Judge strategy and supporting vLLM acceleration.
  • 4.1 A Large-scale Medical Evaluation Benchmark: The benchmark contains 152,066 evaluation samples, 121,622 distinct medical images, and 16 datasets spanning multimodal QA, text-only QA, and report generation.Its multimodal coverage includes diverse imaging modalities, while text-only QA aggregates major medical knowledge benchmarks and report generation uses MIMIC-CXR, IU-Xray, and CheXpert Plus.
  • 4.1 A Large-scale Medical Evaluation Benchmark: The framework standardizes question inputs while following each candidate model’s officially recommended chat template, enabling consistent evaluation across question types.Question-specific prompt formats are illustrated in Figure 8.
  • 4.2 Evaluation Metrics: QA evaluation uses Accuracy, with rule-based and MMMU-codebase matching for multiple-choice questions and GPT-4.1 assessment for open-ended answers.Multiple-choice scoring falls back to similarity-based option selection when rule-based matching is insufficient.
  • 4.2 Evaluation Metrics: Report generation is assessed with Rouge-L, CIDEr, SembScore, RaTEScore, and RadCliQ-v1 to combine semantic, model-based, and composite evaluation.The multi-metric design addresses lengthy, highly open-ended reports and enables more granular assessment.

5 Experiments

Across multimodal, textual, and report-generation benchmarks, Lingshu generally leads open-source models, while ablations, modality analyses, scaling curves, and RL experiments identify data composition and quality as key determinants of performance.

  • 5.2 Performance Comparison on Medical Multimodal Benchmarks: Lingshu-32B achieves a state-of-the-art 66.6 average across seven medical multimodal benchmarks, while Lingshu-7B leads sub-10B models with 61.8 and ranks first on five benchmarks.Lingshu-7B exceeds the best sub-10B baseline by +4.5; Lingshu-32B outperforms proprietary and open-source counterparts overall.
  • 5.3 Performance Comparison on Medical Textual Benchmarks: Lingshu-32B exceeds InternVL3-38B by 3.4 percentage points on medical textual accuracy and leads six of seven benchmarks, though proprietary systems remain ahead on demanding tasks.Lingshu-7B leads sub-10B open-source models, including PubMedQA 76.6, MedQA-USMLE 63.3, Medbullets 56.2, and MedXpertQA 16.5.
  • 5.4 Performance Comparison on Report Generation Benchmarks: Lingshu-7B surpasses competing sub-10B models across MIMIC-CXR, CheXpert Plus, and IU-Xray, while Lingshu-32B leads almost all report-generation metrics.Larger models sometimes score lower on MIMIC-CXR and CheXpert Plus, and InternVL performs markedly worse on report generation than on QA benchmarks.
  • 5.5 Performance of Medical-oriented RL: Medical-oriented RLVR yields only marginal gains on MMMU-Med, PMC-VQA, and VQA-RAD, degrades other benchmarks, and leaves average performance relatively stable.The reported causes are unreliable accuracy-oriented rewards for linguistically variable medical answers and training data containing insufficiently reasoning-focused problems.
  • 5.6 Ablation Study on Data Composition: All four data categories contribute to medical multimodal solving, with removing only 173K medical-text samples causing substantial degradation across five of seven tasks.Medical multimodal data especially supports SLAKE, PathVQA, and OmniMedVQA, whereas general multimodal data more affects MMMU-Med and PMC-VQA.
  • 5.6 Ablation Study on Data Composition: Removing Stage 3 captions, early-stage caption alignment, synthetic multimodal data, or distilled reasoning texts substantially reduces performance, with reasoning-text removal affecting five of seven tasks.The findings attribute gains to high-quality captions, early alignment, modality diversity, and dense medical knowledge in chain-of-thought traces.
  • 5.7 Results per Modality on OmniMedVQA: On OmniMedVQA, Lingshu leads microscopy, MRI, dermoscopy, and OCT, remains strong on ultrasound and fundus photography, and trails the best baseline modestly on CT and X-ray.The modality results indicate strong high-resolution recognition despite Lingshu’s compact parameter budget.
  • 5.8 Analysis of Data Volume Scaling: Across seven benchmarks, accuracy rises sharply with early data additions, improves with diminishing gains toward full scale, and increases from roughly 52% to 62% overall.Image-centric tasks saturate earlier than concept-heavy MedXpertQA, suggesting later progress requires qualitative data enrichment or architectural innovation.

6 Case Studies

The case studies demonstrate Lingshu-32B across medical imaging, diagnosis, biomedical knowledge, public health, report generation, and patient–doctor dialogue, with comparisons showing practical advantages over baseline models. Across these applications, Lingshu combines clinically structured reasoning, accurate interpretation, and actionable outputs.

  • 6 Case Studies: The case studies use Lingshu-32B and compare it with equivalently sized baselines when feasible to illustrate practical benefits for healthcare professionals and patients.The applications span medical diagnosis and report generation as well as broader clinical reasoning and support.
  • 6.1 Visual Question Answering across Different Medical Imaging Modalities: Across eight medical imaging modalities, Lingshu exposes transparent reasoning by systematically examining relevant anatomy and incrementally narrowing diagnostic alternatives.Examples include ordered chest-radiograph inspection and narrowing dermoscopy hypotheses to dermatofibroma after incorporating additional visual cues.
  • 6.2 Medical Diagnosis: Lingshu outperforms InternVL3 in clinical diagnosis by considering differential diagnoses and ancillary visual evidence before reaching a cautious, clinically grounded conclusion.In the retinal case, Lingshu narrows macular edema versus macular hole to macular edema, whereas InternVL3 gives an overconfident incorrect verdict.
  • 6.3 Medical Knowledge: Lingshu accurately explains biomedical processes and classifies a synapse as noradrenergic, unlike Qwen2.5-VL, which produces incoherent content and misses the crucial classification.The case highlights sustained biomedical knowledge rather than image description alone.
  • 6.4 Public Health: Lingshu extracts cohort incidence rates of 58 per 100, 000 and 5 per 100, 000, then computes an attributable risk of 53 per 100, 000.It also articulates the population-level implication of the public-health calculation.
  • 6.5 Report Generation: Lingshu generates radiology reports with separate Findings and Impression sections, identifies cardiac and pulmonary abnormalities, and recommends follow-up radiography after diuretic therapy.The output combines radiographic detection with contextually appropriate management advice.
  • 6.6 Patient-Doctor Dialogue: In a patient–doctor consultation, Lingshu interprets concerns, proposes differential diagnoses, recommends the relevant specialty, and specifies suitable diagnostic tests.This demonstrates its potential as an assistant for consultations and broader healthcare support.

7 Related work

Medical MLLM research has progressed rapidly, with early work aligning LLMs to specialized medical visual encoders and subsequent efforts adopting similar architectures with broader training datasets.

  • Medical multimodal large language models: Early medical MLLMs combined LLMs with specialized medical visual encoders through linear transformation layers, while later models retained this paradigm and pursued more comprehensive training datasets.These efforts target medical image understanding and analysis within healthcare applications.

Public Health

This section presents Lingshu through public-health and real-life medical-support case studies, alongside its broader focus on diverse medical modalities and clinical applications.

  • Public Health: Figure 14 presents a case study of Lingshu addressing a public-health problem.
  • Public Health: Figure 15 presents a case study of Lingshu generating a medical report.
  • Public Health: Lingshu is positioned as a multimodal foundation model supporting question answering, diagnosis, report generation, and diverse clinical applications across medical modalities.This distinguishes the work from specialized medical models and tool-integrated medical agents.
  • Public Health: Figure 16 presents a case study of Lingshu providing real-life medical support.

8 Conclusion, Limitations, and Future Work · Appendix

Lingshu combines curated multimodal medical data, staged training, reinforcement-learning exploration, and MedEvalKit to deliver strong medical performance and standardized evaluation. The paper identifies limitations in data, generalization, training, and evaluation, and proposes targeted directions for broader clinical applicability.

  • 8.1 Conclusion: Lingshu unifies curated medical and general-domain data, staged training, exploratory RLVR, and MedEvalKit, consistently surpassing open-source baselines across medical VQA and report generation.The framework narrows gaps to proprietary models while providing empirically grounded guidance on data curation, training, and evaluation.
  • 8.2 Limitations: Dataset quality and diversity remain limited by inaccurate annotations, poor image resolution, uneven modalities, and hallucinations or factual errors in generated samples.These issues persist despite quality control and manual verification and may hinder downstream generalizability.
  • 8.2 Limitations: Lingshu still underperforms state-of-the-art proprietary models, while its generalization to broader and more diverse medical tasks remains insufficiently explored.Extending coverage to additional scenarios and modality types introduces substantial data-integration, preprocessing, and model-design challenges.
  • 8.2 Limitations: The optimal data mixture and training configuration remain unclear, and preliminary RLVR provides only moderate downstream gains with limited overall impact.Deeper understanding of effective reinforcement learning in medical contexts is still needed.
  • 8.3 Future Work: Future data work should build diverse, high-quality image-text datasets using specialized assessment models, fine-grained synthesis, and human-in-the-loop refinement.These systems aim to reduce the expert-evaluation burden while improving production efficiency and reliability.
  • 8.3 Future Work: Future benchmarks should better capture clinical complexity through comprehensive, application-aligned medical multimodal evaluations comparable in realism to HealthBench (Arora et al., 2025).Current benchmarks insufficiently assess practical downstream utility.
  • 8.3 Future Work: Lingshu should natively support WSI, 3D imaging, and omics data, while post-training develops medical-specific reinforcement learning with tailored rewards, reward models, and process supervision.These extensions address information loss and the knowledge-driven nature of medical reasoning.
  • 8.3 Future Work: Evaluation should add medical-specific measures and expert review, alongside a roadmap for larger models, improved benchmarks, specialized applications, expanded languages, and task-oriented clinical agents.Proposed measures include c-index, Clinical Efficacy Score, and Decision Curve Analysis for real-world medical performance.

A More Details about Data Curation · A.1 Model-based response cleaning for patient-doctor dialogue datasets

This subsection presents the prompt used for model-based response cleaning in patient-doctor dialogue datasets, with the prompt provided in Table 11.

  • A.1 Model-based response cleaning for patient-doctor dialogue datasets: The authors present the prompt used for model-based response cleaning in patient-doctor dialogue datasets.The prompt is provided in Table 11.
  • A.1 Model-based response cleaning for patient-doctor dialogue datasets: Table 11 contains the prompt used to clean responses in patient-doctor dialogue datasets.

A.2 Medical VQA Data Synthesis

This section presents the self-instruct-based prompt used to synthesize medical VQA data, provided in Table 12.

  • A.2 Medical VQA Data Synthesis: The medical VQA synthesis method uses a self-instruct-based prompt, whose contents are presented in Table 12.

A.3 Medical reasoning data distillation · A.4 Prompts for Medical Long-form Caption Synthesis

The appendix specifies prompts for distilling medical chain-of-thought reasoning from GPT4-o and for synthesizing long-form medical captions across multiple annotation stages. It also identifies the figures and tables containing these prompts and the quality-checking procedure.

  • A.3 Medical reasoning data distillation: Medical open-ended VQA reasoning traces are distilled from GPT4-o using dedicated prompts.The prompts are depicted in Figure 13.
  • A.3 Medical reasoning data distillation: Dedicated prompts are also provided for medical multiple-choice VQA datasets.These prompts are depicted in Figure 14.
  • A.3 Medical reasoning data distillation: A separate prompt checks the quality of the generated reasoning traces.The quality-checking prompt is presented in Table 15.
  • A.4 Prompts for Medical Long-form Caption Synthesis: Long-form caption synthesis uses prompts across multiple annotation stages.The appendix details prompts for the different stages of medical caption construction.
  • A.4 Prompts for Medical Long-form Caption Synthesis: Stage 3 uses a prompt for factual knowledge annotation, while Stage 4 uses image-type-specific doctor-preference prompts.These prompts appear in Tables 16–20.
  • A.4 Prompts for Medical Long-form Caption Synthesis: Stage 5 uses a summarization prompt whose {doctor_preferred_features} variable is filled from image-type-specific doctor-preferred features.The features are specified in Tables 22 and 23, and the summarization prompt appears in Table 21.

B More Details about MedEvalKit

MedEvalKit evaluates report generation with complementary semantic and model-based metrics, while its caption-synthesis prompts specify modality-specific clinical features and its benchmark statistics are summarized separately.

  • Report-generation evaluation: MedEvalKit’s report-generation evaluation integrates semantic and model-based metrics, selectively presenting a subset of the supported measures.Supported metrics include BLEU, ROUGE, METEOR, CIDEr, BERTScore, SembScore, RadGraph-F1, RadCliQ-v1, RaTEScore, and GREEN.
  • Report-generation evaluation: Semantic metrics assess n-gram precision, reference overlap, flexible unigram matching, and semantic similarity between generated reports and reference text.BLEU measures contiguous n-gram precision; ROUGE emphasizes recall-oriented overlap; METEOR uses flexible unigram matching; CIDEr measures alignment with human consensus.
  • Report-generation evaluation: Model-based metrics capture contextual similarity, clinical-content alignment, structured clinical relationships, holistic quality, entity accuracy, and clinically significant report errors.BERTScore uses BERT embedding cosine similarity; SembScore compares 14 CheXbert indicators; RadGraph-F1 evaluates entities and relations; RadCliQ-v1 combines multiple measures; RaTEScore handles synonyms and negation; GREEN uses LLMs.
  • Caption-synthesis prompting: Caption-synthesis prompts use doctor-preferred, modality-specific features, including anatomy, imaging orientation, abnormalities, tissue morphology, lesion characteristics, and ultrasound findings.Prompts cover X-ray, CT, histopathology, skin, knee X-ray, fundus, and ultrasound images, with details included when visually discernible.
Loading 2506.07044v4…