Source-linked AI summary
MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
Lai Wei, Yuchao Chen, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Bangting Wang, Wei Chen, Xiang Bai
TL;DR
Existing benchmarks do not adequately test open-ended diagnosis from combined clinical narratives and medical images. MedReaMM constructs and evaluates such cases under complete information, finding that most current LMMs remain below 50% accuracy and far from reliable clinical use.
Problem
Existing benchmarks largely emphasize factual, localized, or text-only tasks rather than integrating heterogeneous evidence for expert diagnostic synthesis.
Method
MedReaMM builds a complete-information benchmark from expert-validated case reports, combining clinical narratives, medical images, standardized ICD-11 diagnoses, and leakage screening.
Results
Most of 23 evaluated LMMs achieve below 50% diagnostic accuracy even under Top-10 evaluation, while diagnostic performance correlates with medical knowledge, image understanding, and evidence integration.
Takeaways & Limitations
Current models remain far from reliable clinical use, highlighting the need for rigorous multimodal diagnostic evaluation.
Takeaways & Limitations
The fixed-input benchmark cannot assess adaptive decision-making or information-seeking, and curated case reports may limit generalization to noisier routine hospital data.
Abstract
from arXiv · showhide
The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus on textual reasoning or isolated visual question-answering (VQA) tasks, lacking holistic integration of clinical narratives and medical imaging, and thus failing to assess the multimodal diagnostic synthesis capability central to expert clinical judgment. To bridge this gap, we introduce MedReaMM, a benchmark specifically designed to evaluate models' ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm. Constructed from case reports sourced from top-tier medical journals and curated clinical case databases, MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes. These cases predominantly represent rare, atypical, or multi-system presentations that demand expert-level evidence integration beyond routine pattern recognition. We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability. Further analysis reveals that medical knowledge proficiency, medical image understanding, and evidence integration are all highly correlated with diagnostic performance.
1 Introduction
Existing medical benchmarks often isolate factual recall, perception, or text-only reasoning rather than integrating heterogeneous evidence for open-ended diagnosis. MedReaMM addresses this gap with complete-information multimodal cases and finds substantial limitations in current LMM diagnostic synthesis.
- Motivation: Existing benchmarks emphasize factual recall, localized perception, or text-only inputs, leaving complete-information diagnostic synthesis insufficiently evaluated.They also use multiple-choice or string-matching approaches that can restrict diagnosis or misjudge semantically equivalent answers.
- Benchmark contribution: MedReaMM evaluates expert-level diagnosis by combining detailed clinical narratives with diverse medical images under a complete-information paradigm.Its cases are rare, atypical, or multi-system presentations requiring evidence-driven synthesis beyond routine feature matching.
- Benchmark contribution: 625 expert-validated cases and 1,042 standardized ICD-11-coded diagnoses provide the benchmark’s core evaluation scope.The cases are sourced from high-impact clinical journals and include supplementary reasoning paths and image captions for analysis.
- Results: Diagnostic performance strongly correlates with medical knowledge, image understanding, diagnostic reasoning, and evidence integration.Reasoning prompts offer only limited gains constrained by model scale.
2 Related Work
Medical evaluation benchmarks span factual QA, interactive information seeking, and static reasoning, but most static alternatives simplify inputs or omit multimodal diagnostic synthesis. MedReaMM positions complete-information multimodal evaluation as complementary to interactive clinical-agent benchmarks.
- Benchmark paradigms: Early medical benchmarks primarily tested factual recall through text-based question answering.Examples include MedQA, PathVQA, and RJUA-MedDQA.
- Benchmark paradigms: Dynamic benchmarks assess iterative information seeking and patient interaction through simulated clinical workflows.These include AI Hospital, Agent Hospital, and AgentClinic.
- Benchmark paradigms: Static benchmarks target reasoning over complex inputs but are often text-only or simplified VQA tasks centered on medical knowledge memorization.MedReaMM instead evaluates multimodal evidence synthesis in expert-level open-ended diagnosis.
- Positioning: MedReaMM and interactive benchmarks are presented as complementary paradigms for evaluating diagnostic synthesis and information-seeking behavior.The benchmark emphasizes complex cases with open-ended diagnostics where AI assistance is most needed.
- Model and agent developments: Recent medical LMMs improve through larger and more diverse training data, while agent-based systems add tools, visual extraction, or simulated consultation.These developments address model capability and workflow integration but do not replace complex diagnostic evaluation.
3 Benchmark Construction
MedReaMM is constructed through staged collection, multimodal extraction, leakage screening, diagnosis normalization, and clinical validation. The resulting cases preserve structured evidence while applying expert quality controls and standardized coding.
- Case collection: Diagnostic cases are collected from top-tier medical journals and curated case-report websites.The pipeline includes sources such as NEJM and The Lancet, with the full process illustrated in Figure 2.
- Multimodal extraction: Case reports are parsed into medical records, images, captions, final diagnoses, and optional reasoning paths.Filtered captions retain modality, anatomy, staining method, and magnification metadata, while image regions are cropped from PDFs using extracted coordinates.
- Label leakage detection: Three medical-major annotators screen textual and visual modalities for diagnosis mentions, interpretive captions, and post-diagnosis visual information.Cases flagged by at least two annotators are removed, and non-essential images are excluded.
- Quality control: Inter-annotator agreement for leakage judgments is 0.6809 by average pairwise Cohen’s Kappa.Three annotators independently provide binary leakage versus non-leakage judgments.
- Diagnosis encoding: Diagnoses are decomposed into disease entities and mapped to ICD-11 codes, with unmatched cases revised and reprocessed using expert input.One entity is designated primary and others secondary to support cases with multiple coexisting conditions.
- Clinical expert review: Two senior physicians review source fidelity, diagnosis faithfulness, and diagnostic sufficiency, retaining 625 high-quality cases.Cases judged insufficient or inaccurate by either expert are removed.
4 Experiment
MedReaMM evaluates multimodal diagnostic synthesis using recall and ranking-sensitive metrics, then examines capability drivers, model behavior, and input ablations. Results show substantial performance gaps, positive associations with diagnostic factors, and benefits from complete aligned inputs.
- Evaluation: Primary Diagnosis Recall counts cases where the primary diagnosis appears among top-k predictions, whereas Complete Diagnosis Recall requires all annotated diagnoses.The evaluation uses Top-5 and Top-10 settings.
- Evaluation: Top-1 Accuracy, MRR, and NDCG@10 assess ranking quality, with NDCG@10 assigning relevance scores of 2 to primary matches, 1 to other complete-set diagnoses, and 0 to mismatches.Top-1 and MRR use the primary diagnosis as ground truth; MRR uses the primary diagnosis rank.
- Main results: Top-10 Primary Diagnosis Recall exceeds 55% for Gemini-2.5-Pro and Gemini-2.5-Flash, while most evaluated models remain below 50%.Table 2 reports results for 23 leading vision-language and text-only models.
- Model comparisons: Reasoning models consistently outperform non-reasoning models across all metrics, including recall and strict ranking indices.The paper reports confidence intervals and stability across prompt variations in Appendix E.
- Drivers of diagnostic performance: Primary diagnosis recall correlates with medical image understanding, disease knowledge proficiency, and evidence integration capability.The authors evaluate nine representative models using auxiliary tasks scored on a 0–100 scale and analyze their impact on Top-5 recall.
- Drivers of diagnostic performance: Qwen-2.5-VL-72B illustrates a vision-knowledge mismatch: image understanding scores 79.8, disease knowledge scores 59.7, and diagnostic performance scores 38.9.Its image understanding score is near Gemini-2.5-Pro’s 81.9, but lower disease knowledge is described as a bottleneck.
- Input ablations: The full records-plus-images-plus-aligned-captions input performs best across models, while removing components or shuffling captions reduces performance.The shuffled-caption configuration underperforms both the aligned setting and the image–record baseline, indicating the importance of caption-image alignment.
5 Conclusion
MedReaMM is introduced as a multimodal benchmark for evaluating LMM diagnostic capabilities, and its results indicate that current models remain far from reliable clinical use.
- MedReaMM is a multimodal benchmark designed to evaluate LMM diagnostic capabilities.
- Current models remain far from reliable clinical use according to the benchmark results.
Limitations
MedReaMM has important limitations involving workflow realism, data distribution, residual label leakage, evaluation scope, and the changing model landscape. Its statistics nevertheless document substantial multimodal and diagnostic diversity, including frequent multiple-image and multi-diagnosis cases.
- Workflow realism: Fixed-input evaluation cannot assess adaptive decision-making or information-seeking behaviors in interactive clinical workflows.
- Data distribution: Curated case reports may differ from noisier, incomplete, and inconsistently documented routine electronic health records, limiting direct generalization to hospital systems.
- Label leakage: Residual implicit cues from case-report narratives or image selection may influence human and model performance despite leakage screening and expert review.
- Evaluation scope: Open-ended diagnostic accuracy does not directly measure uncertainty calibration, risk awareness, or downstream clinical impact.
- Benchmark composition: 79% of cases have two or more diagnostic labels, and many cases contain multiple images, emphasizing multi-diagnosis and multimodal complexity.
- Benchmark composition: Figure 6 summarizes clinical-term and diagnosis-term distributions, revealing lexical richness and diagnostic variability across the benchmark.
B Dataset Scope and Limitations
MedReaMM emphasizes rare, atypical, and multi-system clinical scenarios, using expert-curated narratives and standardized ICD-11 annotations to stress-test multimodal diagnostic reasoning. Its evaluation prioritizes identifying useful differential diagnoses through ranking-sensitive metrics.
- Dataset scope: MedReaMM primarily features rare diseases and atypical multi-system pathologies requiring expert-level clinical reasoning.
- Dataset scope: Expert-curated narratives provide dense, standardized inputs for evaluating logical synthesis and evidence integration before testing noisier real-world data.
- Dataset construction: 625 expert-validated cases use ICD-11 standardized annotations and emphasize depth of multimodal evidence over common clinical presentations.
- Evaluation: Top-5 and Top-10 recall assess whether models identify correct diagnostic leads for clinician review in complex diagnostic dilemmas.
D Human vs. Model Evaluation
The human-versus-model study reformulates gastrointestinal cases as standardized multiple-choice questions and evaluates both exact accuracy and diagnostic-set overlap. Human performance is interpreted in light of individual-specialist constraints on unusually complex, interdisciplinary cases.
- Study design: 46 gastrointestinal questions were created by pairing ground-truth diagnoses with distractors sampled from incorrect model predictions.
- Study design: Humans and models received the same detailed image captions to standardize access to expert image interpretations.
- Scoring: Strict Accuracy requires exact ground-truth matching, while DSC measures overlap between predicted and reference diagnostic sets.
- Results: 10.87% human accuracy and 0.2201 DSC contrasted with AI models’ stronger capacity to identify relevant diagnostic components in these cases.
- Interpretation: The human baseline represents individual specialists without peer consultation or external search tools, whereas many benchmark cases require multidisciplinary expertise.
- Interpretation: Rare diseases and multi-system complications outside specialists’ subfields make identifying every granular diagnostic component inherently challenging for individuals.
E.1 Results With Confidence Intervals
Bootstrap confidence intervals distinguish model tiers in primary diagnosis recall, while prompt robustness testing indicates stable evaluation results under minor wording changes.
- 5000 bootstrap resamples produced 95% confidence intervals for selected models on Top-10 Primary Diagnosis Recall.The intervals are shown in Figure 7.
- Models form distinct tiers in primary diagnosis recall, although overlapping adjacent intervals may indicate non-significant differences.
- Non-overlapping confidence intervals suggest more robust distinctions in diagnostic capabilities.
- Three semantically equivalent, syntactically varied prompts were used to assess evaluation robustness.
- Performance fluctuations remained within 3.3% for key metrics including Top-5 Primary and Complete Diagnosis Recall across most tested models.The results were described as stable under minor prompt-wording variations.
F.1 Data Leakage
The analysis examines robustness across publication years and disease categories. Models perform better on common, visually distinct conditions, while category-specific weaknesses persist despite more lenient recall settings.
- Data Leakage: No notable performance drop was observed for 2025 case reports, suggesting little to no pretraining data leakage or negligible evaluation impact.
- Disease Categories: Performance was evaluated across cardiovascular, dermatologic, endocrine, gastrointestinal, congenital, infectious, neoplastic, neurological, and orthopedic categories under four recall settings.The settings were Top-5 Primary, Top-5 Complete, Top-10 Primary, and Top-10 Complete Diagnosis Recall.
- Disease Categories: Figure 8 uses radar charts to present categorywise performance for representative models.
- Disease Categories: Models generally perform better on common, visually distinct endocrine/metabolic and gastrointestinal diseases than on neoplastic and neurological cases.The latter often require nuanced reasoning or involve subtle imaging features.
- Disease Categories: Gemini-2.5-Pro shows relatively balanced performance, whereas MedGemma-4B and Lingshu-32B exhibit stronger category-specific weaknesses.Model gaps narrow as recall becomes more lenient, but category discrepancies persist.
G Human Annotation
Human annotation supported leakage screening and expert validation during dataset construction. Sequential filtering reduced 2,867 reports to 625 cases whose diagnoses could be inferred from the provided inputs.
- Annotation Process: Human annotators performed label leakage detection and clinical expert review during MedReaMM construction.
- Annotation Process: Annotators checked whether diagnoses appeared in records or captions, captions contained interpretations, or images showed post-diagnostic information.
- Annotation Process: Cases with text leakage were removed when at least two annotators agreed, while inappropriate images were excluded or cropped under the same agreement rule.
- Annotation Process: 1,527 extracted cases were reviewed over approximately 45 days, with 1,193 retained for the next stage.
- Expert Validation: Two senior clinicians verified input-diagnosis consistency with source reports and whether diagnoses were logically inferable from the inputs.
- Expert Validation: 2,867 reports were reduced to 625 high-quality cases after image-content filtering, leakage detection, encoding, and expert removal of cases lacking fully inferable diagnoses.
- Evaluation Validation: Three senior-year medical students rated seven diagnostic dimensions, with agreement reported between GPT-5 and humans and among human annotators.
J Case Study
The case study illustrates that a strong model can identify a primary pulmonary condition while missing a clinically significant co-infection. The benchmark’s prompts separately support extraction, image understanding, disease knowledge, and reasoning-path evaluation.
- Case Findings: A 35-year-old woman with recurrent pneumonia was diagnosed with intralobar pulmonary sequestration complicated by actinomycosis infection.
- Case Findings: The diagnosis was supported by a left-lower-lobe chest CT lesion and histopathology confirming characteristic Actinomyces colonies.
- Model Behavior: All models failed to recognize actinomycosis despite stronger models identifying pulmonary sequestration.The paper associates this with prioritizing prominent diagnoses and difficulty extracting fine-grained pathological image details.
- Model Behavior: Gemini-2.5-Pro listed intralobar pulmonary sequestration among its differential diagnoses but did not recognize the actinomycosis co-infection.
- Prompt Design: Dataset extraction prompts require objective, pre-treatment patient, history, examination, laboratory, imaging, and diagnostic-test information.
- Prompt Design: Caption processing removes subjective interpretations and diagnostic findings while retaining required objective information and figure or table references.
- Prompt Design: Separate evaluation prompts assess main diagnosis, image understanding, disease knowledge, and reasoning paths.