PaperlayerView exact revision on arXiv

Source-linked AI summary

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, Yu Rong

arXiv:2506.07044v4cs.CLcs.AIcs.CV

TL;DR

Medical MLLMs are limited by incomplete medical knowledge coverage, data-quality problems, and weak reasoning for complex scenarios. The paper introduces broad data curation, staged training, RLVR exploration, and MedEvalKit. Lingshu achieves strong performance across medical tasks, while dataset quality and broader generalization remain limitations.

  • Problem

    Medical MLLMs have limited coverage beyond imaging, susceptibility to hallucinations from suboptimal curation, and insufficient reasoning for complex medical scenarios.

  • Method

    The paper curates medical, textual, and general-domain data; synthesizes captions, QA, and reasoning samples; trains Lingshu progressively; explores RLVR; and develops MedEvalKit.

  • Results

    Lingshu consistently achieves state-of-the-art performance across most multimodal and textual medical VQA tasks and report generation, with Lingshu-32B averaging 7.2 accuracy points above the second-best model across seven medical VQA tasks.

  • Takeaways & Limitations

    Lingshu provides a unified medical multimodal model and evaluation framework with strong performance across multiple medical tasks and modalities.

  • Takeaways & Limitations

    Dataset quality and diversity remain limited, and generalization to broader and more diverse medical tasks remains insufficiently explored.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

1 Introduction

Existing medical MLLMs remain constrained by limited domain knowledge, noisy curation, and insufficient reasoning for complex cases. Lingshu addresses these gaps through broad data curation, staged medical training, RLVR exploration, and standardized evaluation, achieving strong results across medical tasks.

  • Motivation: Medical MLLMs remain limited because biomedical image-text data differ fundamentally from typical web content and models lack domain-specific visual knowledge.Existing models may therefore struggle with medical applications despite strong general-domain multimodal capabilities.
  • Motivation: Distilling image-text pairs from medical imaging undercovers pharmacology, public health, and clinical knowledge, while weak supervision and limited sequential reasoning constrain complex medical cases.The paper identifies both incomplete knowledge coverage and insufficient handling of decision-making beyond basic image-text correlations.
  • Approach: The data curation pipeline collects multimodal medical data, medical texts, and general-domain data, then synthesizes captions, QA pairs, and chain-of-thought reasoning samples.This approach is designed to provide broader medical knowledge and more accurate training examples.
  • Evaluation: MedEvalKit consolidates major multimodal and textual medical benchmarks into a unified framework for efficient, standardized model assessment.The framework addresses the paper’s concern that medical models are often evaluated in isolated, non-standardized environments.
  • Results: 7.2 accuracy points: Lingshu-32B exceeds the second-best model by an average across seven medical VQA tasks and also surpasses GPT-4.1 and Claude Sonnet 4.Lingshu achieves state-of-the-art performance across most multimodal and textual medical VQA tasks and in report generation for both model sizes.
  • Approach: Lingshu uses a multi-stage training paradigm in 7B and 32B models to progressively infuse medical knowledge and improve problem-solving abilities.The authors also explore reinforcement learning with verifiable rewards for multimodal medical reasoning through Lingshu-RL.

2 Data Curation

The data curation strategy combines diverse medical and general-domain sources with targeted synthesis of captions, OCR samples, VQA, and reasoning data. Rigorous cleaning and validation produce a large, deduplicated training corpus enriched for medical understanding.

  • Data collection: The curation pipeline collects multimodal medical, unimodal medical, and general-domain multimodal or textual data to broaden medical and general understanding.
  • Data synthesis: Synthetic data target diagnostic detail, medical-image text recognition, VQA, and advanced reasoning through long-form captions, OCR instructions, QA pairs, and reasoning trajectories.
  • Medical multimodal data: Medical multimodal data include caption pairs, image-instruction data, VQA, and radiology-report data supporting diverse medical tasks.
  • Medical unimodal data: Medical text instruction data cover factoid QA, distilled reasoning, and diverse medical tasks, while imaging data provide metadata for additional caption and VQA synthesis.
  • General-domain data: General-domain captioning, textual, and multimodal instruction data are incorporated to improve visual understanding, text generation, instruction following, and generalization.
  • Quality control: 3.75M high-quality open-source and 1.30M high-quality synthetic medical samples are produced after quality checks, deduplication, and benchmark-contamination prevention.

3 Model Training

Lingshu adapts Qwen2.5-VL through a shallow-to-deep, four-stage medical training framework. The first three supervised stages form Lingshu, while an additional RLVR stage produces Lingshu-RL.

  • Backbone: Lingshu uses Qwen2.5-VL 7B-Instruct and 32B-Instruct backbones with an LLM, vision encoder, and MLP-based projector.
  • Training framework: The training framework progresses through Medical Shallow Alignment, Medical Deep Alignment, Medical Instruction Tuning, and Medical-oriented Reinforcement Learning.
  • Medical-oriented Reinforcement Learning: Medical-oriented reinforcement learning applies RLVR to strengthen medical reasoning, creating the Lingshu-RL version after the first three SFT stages.
  • Medical Shallow Alignment: Medical Shallow Alignment freezes the LLM and fine-tunes the vision encoder and projector on concise medical image-caption data.
  • Medical Deep Alignment: Medical Deep Alignment unfreezes all parameters and uses more diverse, richer medical image-text data together with general-domain multimodal data.
  • Medical Instruction Tuning: Medical Instruction Tuning optimizes the full model on varied clinical directives, including diagnosis, examination, retrieval, report generation, and localization.

4 MedEvalKit: A Unified Medical Evaluation Framework

MedEvalKit addresses inconsistent medical-model comparisons by unifying multimodal and textual benchmarks, input formats, and task-specific evaluation protocols. It supports QA and report-generation assessment across broad medical tasks.

  • Framework: MedEvalKit integrates mainstream medical benchmarks and task types for systematic, comprehensive, and efficient evaluation of medical MLLMs.
  • Coverage: The framework supports multiple-choice, closed-ended, open-ended, and medical report-generation question formats for both multimodal and text-only inputs.
  • Benchmark composition: The evaluation collection contains 152,066 samples, 121,622 distinct medical images, and 16 benchmark datasets.
  • Question answering: Multimodal QA benchmarks span diverse medical imaging tasks, while text-only QA benchmarks cover multiple medical knowledge and reasoning datasets.
  • Report generation: Report generation uses MIMIC-CXR, IU-Xray, and CheXpert Plus official test splits, primarily pairing chest X-rays with radiology reports.
  • Evaluation protocols: QA evaluation uses accuracy, with rule-based and similarity-based matching for multiple-choice questions and GPT-4.1 assessment for open-ended answers.

5 Experiments

Lingshu performs strongly across medical multimodal and textual benchmarks, report generation, and eight imaging modalities. Ablations show that broad, high-quality data coverage and staged training materially support its performance, while reinforcement learning yields mixed task-level effects.

  • Multimodal benchmarks: 66.6 average score across all multimodal benchmarks makes Lingshu-32B state of the art among evaluated proprietary and open-source counterparts.Lingshu-7B achieves a 61.8 average score among open-source models with fewer than 10B parameters.
  • Medical textual benchmarks: Lingshu-7B leads open-source medical text models below 10B parameters, topping four benchmarks and ranking second on the other three.Its reported scores include PubMedQA (76.6), MedQA-USMLE (63.3), Medbullets (56.2), and MedXpertQA (16.5).
  • Medical textual benchmarks: Lingshu-32B exceeds InternVL3-38B by 3.4 percentage points on average and ranks first on six of seven textual benchmarks.A performance gap remains versus leading proprietary systems, particularly on deep clinical reasoning and extensive medical-knowledge tasks.
  • Medical report generation: Lingshu-7B consistently surpasses competing models across MIMIC-CXR, CheXpert Plus, and IU-Xray report-generation benchmarks.The passage states that it attains either the best or second-best result on these datasets.
  • Medical-oriented reinforcement learning: RLVR produces mixed effects: gains reach +0.7% on MMMU-Med and PMC-VQA, while declines reach −1.6% on OMVQA, leaving average performance stable.The authors attribute the mixed outcome to rule-based reward limitations and training data that often do not require complex reasoning.
  • Ablation studies: Removing synthetic multimodal data, distilled medical reasoning text, or medical textual data causes broad performance declines, with medical text especially critical across five of seven tasks.Medical captions remain important in both early alignment and final instruction tuning, while modality-specific data affect benchmarks differently.
  • Results per modality and data scale: Across eight imaging modalities, Lingshu matches or exceeds competitors on most fronts, leading in microscopy, MRI, dermoscopy, and OCT while trailing modestly in CT and X-ray.Accuracy rises sharply with early data additions and continues increasing with diminishing marginal gains until approximately 75–100% of the full data.

6 Case Studies

The case studies illustrate Lingshu’s medical image interpretation, diagnostic reasoning, biomedical knowledge explanation, public-health calculation, report generation, and consultation support. Across examples, the model uses structured reasoning and produces clinically oriented outputs.

  • Medical VQA: Lingshu analyzes medical VQA cases across eight imaging modalities while exposing reasoning used to reach decisions.The cases emphasize transparent reasoning during multimodal medical question answering.
  • Medical Diagnosis: Its diagnostic reasoning follows clinical workflows by examining anatomical regions systematically and narrowing differential diagnoses incrementally.Examples include ordered chest-radiograph inspection and step-wise dermoscopy differential refinement.
  • Medical Diagnosis: In a retinal case, Lingshu considers macular edema and macular hole before selecting macular edema using subretinal fluid and cystoid spaces.InternVL3 instead produces an incorrect verdict without comparable reasoning.
  • Biomedical Knowledge: Lingshu identifies noradrenergic synaptic processes, whereas Qwen2.5-VL describes the process without correctly identifying the synapse type.The distinction is attributed to Lingshu’s biomedical knowledge of neurotransmitter and receptor interactions.
  • Public Health: In a cohort-study example, Lingshu subtracts incidence rates of 5 per 100, 000 from 58 per 100, 000 to calculate attributable risk of 53 per 100, 000.The example tests extraction from a cohort-study table and application of a public-health formula.
  • Report Generation: Lingshu generates radiology reports with Findings and Impression sections, identifies cardiac and pulmonary abnormalities, and recommends follow-up imaging after diuretic therapy.The case study presents the output as aligned with conventional radiological reporting.
  • Medical Support: In a real-life consultation scenario, Lingshu interprets patient concerns, proposes differential diagnoses, directs specialty care, and suggests diagnostic tests.The case frames these capabilities as support for medical consultations and broader healthcare use.

7 Related work

Medical MLLM research has advanced alongside broader multimodal progress, with early approaches combining language models and specialized medical visual encoders. These efforts have increased interest in medical applications while retaining limitations described for existing systems.

  • Progress in Medical MLLMs: Recent MLLM advances have driven progress across multimodal domains and increased interest in medical applications.The passage cites progress in applying MLLMs within healthcare scenarios.
  • Early Approaches: Early medical approaches combine large language models with specialized medical visual encoders through a linear transformation layer.This establishes visual-language connections for medical applications.
  • Research Direction: The related-work discussion positions medical MLLMs within broader efforts to improve performance across diverse medical tasks.The supplied passage introduces this trajectory without detailing later methods.

Public Health

The supplied passages present public-health and healthcare-support examples involving cohort-study data and a real-life consultation scenario. They frame Lingshu as handling both quantitative public-health reasoning and patient-oriented support.

  • Public Health: A public-health case uses results from a prospective cohort study of smoking and lung cancer.The supplied material identifies the study context but does not provide a complete outcome statement in this section’s passages.
  • Scope: The cases span public-health analysis and consultation support rather than a single imaging-only task.The figure captions identify public-health and real-life medical-support examples.
  • Medical Support: The real-life medical-support case concerns a patient reporting persistent cough and green phlegm after earlier flu-like symptoms.The passage presents the patient’s question as consultation input.

8 Conclusion, Limitations, and Future Work

Lingshu combines broad medical data curation, multi-stage training, reinforcement-learning exploration, and standardized evaluation through MedEvalKit. The authors report strong benchmark performance but identify persistent limitations in data quality, generalization, training configuration, reinforcement learning, and benchmark realism.

  • Conclusion: Lingshu unifies medical and general-domain corpora with captions, QA pairs, and chain-of-thought annotations, then progressively improves medical understanding through multi-stage training.The work also explores reinforcement learning with verifiable rewards and introduces MedEvalKit for standardized comparisons.
  • Conclusion: MedEvalKit consolidates multimodal and textual benchmarks with standardized preprocessing, post-processing, and deployment protocols for fair, reproducible, and transparent model comparisons.The framework supports both multimodal and text-only inputs and multiple question formats, including report generation.
  • Limitations: Medical data quality and diversity remain limited because open-source data can have inaccurate annotations, poor image resolution, uneven modality distribution, and model-generated hallucinations or factual errors.These issues persist despite quality control and manual verification and may hinder downstream generalizability.
  • Limitations: Lingshu underperforms state-of-the-art proprietary models, while its generalization to broader and more diverse medical tasks remains insufficiently explored.Expanding downstream scenarios and modality types introduces challenges in data integration, preprocessing, and model design.
  • Limitations: The optimal data mixture and training configuration remain insufficiently explored, and the medical RLVR investigation is preliminary with limited overall impact.The authors state that deeper understanding is needed for effective reinforcement learning in medical contexts.
  • Future Work: Future work targets higher-quality datasets and benchmarks, broader medical modalities, task-tailored post-training, medical-specific evaluation measures, larger models, and application optimization.Planned applications include report generation, diagnosis, and staging prediction.

A.1 Model-based response cleaning for patient-doctor dialogue datasets

The appendix documents the prompt used to clean responses in patient-doctor dialogue datasets.

  • A.1 Model-based response cleaning for patient-doctor dialogue datasets: A model-based response-cleaning prompt is provided for patient-doctor dialogue datasets.The prompt is referenced in Table 11.

A.2 Medical VQA Data Synthesis

The appendix presents the prompt used to synthesize medical VQA data with a self-instruct-based method.

  • A.2 Medical VQA Data Synthesis: A self-instruct-based prompt is provided for synthesizing medical VQA data.The prompt is referenced in Table 12.

A.3 Medical reasoning data distillation

The appendix describes prompts for distilling medical reasoning traces and synthesizing long-form medical captions across multiple annotation stages.

  • A.3 Medical reasoning data distillation: Prompts are used to distill chain-of-thought reasoning from GPT4-o into open-ended and multiple-choice medical VQA datasets.The reasoning-trace quality-check prompt is provided separately.
  • A.3 Medical reasoning data distillation: The appendix provides a prompt for annotating chain-of-thought reasoning processes in open-ended VQA data.This prompt is referenced in Table 13.
  • A.3 Medical reasoning data distillation: Long-form medical caption synthesis uses staged prompts for factual knowledge annotation, doctor-preference annotation, and final summarization.Doctor-preference prompts vary by medical image type, and the summarization stage uses doctor-preferred features.

B More Details about MedEvalKit

MedEvalKit evaluates medical MLLMs across multiple dimensions using standardized benchmarks and complementary metrics for textual and radiology report generation. Its design combines semantic, clinical-content, and model-based evaluation with modality-specific annotation prompts.

  • Evaluation procedures: MedEvalKit uses GPT-4.1 as an LLM-as-a-Judge module for evaluating consistency between model outputs and reference answers in open-ended questions.The framework also includes prompts for CoT quality checking, factual knowledge annotation, doctor-preference annotation, and modality-specific image descriptions.
  • Report generation evaluation: MedEvalKit supports report generation evaluation through a multi-metric approach combining semantic and model-based metrics.The framework selectively presents a subset of supported metrics in the paper.
  • Semantic evaluation metrics: BLEU measures n-gram precision between generated and reference texts, while ROUGE emphasizes recall-oriented overlap and METEOR uses flexible unigram matching.These metrics provide complementary text-overlap and matching criteria for generated reports or descriptions.
  • Semantic evaluation metrics: CIDEr measures similarity between generated sentences and multiple human references to estimate alignment with human consensus.The metric is designed to capture semantic similarity between generated and reference descriptions.
  • Model-based evaluation metrics: BERTScore evaluates contextual textual similarity using cosine similarity between BERT embeddings of generated and reference radiology reports.It provides a semantically nuanced alternative to surface-level overlap measures.
  • Model-based evaluation metrics: SembScore, RadGraph-F1, RadCliQ-v1, RaTEScore, and GREEN assess radiology reports through pathological indicators, clinical entities, integrated metrics, medical entities, or clinically significant errors.These metrics target clinical-content alignment, structured relationships, entity sensitivity, or clinical accuracy and relevance.
Loading 2506.07044v4…