MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

• Clinical diagnostic support requires models to extract salient cues from heterogeneous text and images and synthesize them into accurate diagnoses.The paper terms this capability multimodal diagnostic synthesis and identifies medical imaging as an indispensable, high-information-density source of clinical evidence.Source: §1 · p. 1
• Existing benchmarks emphasize factual recall, localized image perception, or text-only diagnosis, while interactive evaluations assess information seeking rather than synthesis from fully assembled evidence.Multiple-choice and string-matching or single-code evaluations also risk misrepresenting open-world diagnostic difficulty and semantically equivalent diagnoses.Source: §1 · pp. 1–2
• MedReaMM comprises 625 expert-validated, diagnostically demanding cases and 1,042 standardized ICD-11-coded diagnoses, with cases often involving rare, atypical, or multisystem presentations.Cases come from high-impact clinical journals and are designed to demand evidence-driven synthesis beyond routine pattern matching.Source: §1 · p. 2
• The benchmark combines detailed clinical narratives with diverse medical images and uses ranked differential-diagnosis generation, with ICD-11 alignment plus semantic similarity to evaluate open-ended outputs.A subset provides reference reasoning paths and image captions for mechanism analysis, and systematic label-leakage screening with expert review aims to keep evaluation focused on evidence-based synthesis.Source: §1 · p. 2
• Most of 23 evaluated models score below 50% accuracy under a lenient Top-10 criterion; performance correlates with medical knowledge, image understanding, and evidence integration, while reasoning prompts offer limited gains constrained by model scale.Source: §1 · pp. 2–3

Medical Benchmarks for LLMs and LMMs

• Early medical benchmarks emphasized factual recall, while dynamic benchmarks shifted toward iterative information seeking and patient interaction.Static benchmarks instead assess reasoning under complex inputs, but have often been text-only or simplified VQA tasks.Source: Medical Benchmarks for LLMs and LMMs · p. 3
• MedReaMM fills the static-benchmark gap with multimodal, expert-level evidence synthesis for complex open-ended diagnosis, complementing dynamic benchmarks that assess information seeking and patient interaction.The benchmark aims to provide a more rigorous, clinically relevant assessment of complex cases.Source: Medical Benchmarks for LLMs and LMMs · p. 3
• Medical LMMs progressed from early fine-tuning on PubMed and textbook data, which was limited by data quality, to broader datasets that improved later models.Examples of the later models include Huatuo-GPT-Vision, Lingshu, and MedGemma.Source: Medical Benchmarks for LLMs and LMMs · p. 3
• Agent-based clinical frameworks use LLM-guided visual feature extraction, multi-agent specialized tools, or simulated multidisciplinary consultations.The passage gives MC-CoT, MMedAgent, and MDAgent as examples, respectively.Source: Medical Benchmarks for LLMs and LMMs · p. 3

Case Reports Collection

• The collection comprised 2,359 cases from medical journals and 508 from web platforms manually curated for content completeness.Journal sources included top-tier medical journals with impact factors above 40; reports typically contained patient histories, diagnostic examinations, and final diagnoses.Source: §3 · pp. 3–4

Multimodal Data Extraction

• Claude Sonnet 4 and MonkeyOCR extracted records, captions, diagnoses, reasoning paths, and image regions; GPT-5 removed diagnostic interpretations from captions before images were paired with metadata.Reasoning paths were retained for auxiliary analysis and potential future tasks.Source: §3 · p. 4

Label Leakage Detection

• Three medical students screened cases for diagnosis mentions, interpretive captions, and post-diagnosis visuals; cases flagged by at least two annotators were removed, and nonessential images were excluded.Average pairwise Cohen’s Kappa was 0.6809.Source: §3 · p. 5

Clinical Expert Review

• 625 high-quality cases remained after two senior physicians removed cases with inaccurate inputs or diagnoses, or insufficient diagnostic information; inter-expert agreement was κ=0.7917.The retained set comprised 70.7% of cases from peer-reviewed journals and 29.3% from public clinical-case databases and teaching archives.Source: §3 · p. 5

Diagnosis Encoding

• Each diagnosis was decomposed into primary and secondary disease entities, mapped to ICD-11 codes, and revised with expert input when unmatched.Source: §3 · p. 5

4.1 Implementation Details

• Models were prompted to rank ten standalone diagnoses and decoded greedily, while retaining their default reasoning settings.The same prompts were used across models in the main and supplementary experiments.Source: §4.1 · p. 5

4.2 Evaluation

• Evaluation measured Top-5/10 primary and complete-diagnosis recall, plus Top-1 Accuracy, MRR, and NDCG@10.Top-1 and MRR use only the primary diagnosis; NDCG@10 gives relevance 2 to primary matches, 1 to other annotated diagnoses, and 0 to mismatches.Source: §4.2 · p. 5

4.3 Main Results

• Gemini-2.5-Pro and Gemini-2.5-Flash exceed 55% Top-10 Primary Diagnosis Recall, while most models remain below 50% even under this lenient criterion.Ranking-sensitive metrics correlate with Top-5/10 recall, and reasoning models outperform non-reasoning models across all metrics.Source: Table 2 · p. 6, §4.3 · p. 6

4.2 Evaluation

• To accommodate linguistic variation, predictions matched via exact ICD-11 codes or Sap-BERT cosine similarity at a 0.85 threshold.The two complementary approaches were intended to avoid penalizing synonymous medical expressions.Source: §4.2 · p. 6

4.4 Drivers of Diagnostic Performance

• Disease knowledge, image understanding, and evidence integration each positively correlate with Top-5 primary-diagnosis recall across evaluated models.Qwen-2.5-VL-72B scored 79.8 in image understanding but 59.7 in disease knowledge and 38.9 in diagnostic performance, illustrating a knowledge bottleneck despite strong visual scores.Source: §4.4 · pp. 6–8, Figure 3 · p. 7

4.5 Effects of Prompt-Level Interventions

• Prior-knowledge hints and explicit reasoning prompts had uneven effects: Claude-Sonnet-4 improved most with hints, while LLaMA-4-Maverick gained most from reasoning guidance.CoT guidance improved most standard models, but Qwen2.5-VL-7B declined under both interventions; the authors attribute this to limited underlying capability.Source: Table 3 · p. 7, Table 4 · p. 8, §4.5 · p. 8

4.6 Impact of Image and Captions

• The full records-plus-images-plus-aligned-captions input performed best, while removing any component reduced performance and shuffled captions underperformed even records plus images.This indicates that each modality contributes non-redundant evidence and that image-caption alignment matters.Source: Figure 4 · p. 8, §4.6 · p. 8

• MedReaMM is introduced as a multimodal benchmark for evaluating large multimodal models’ diagnostic capabilities.Source: §5 · p. 8
• Current models remain far from reliable clinical use, according to the benchmark results.Source: §5 · p. 8

Limitations

• The fixed-input benchmark cannot assess physicians’ adaptive decision-making or information-seeking through follow-up questions, examinations, or multidisciplinary consultation.Real clinical workflows often refine hypotheses iteratively, whereas MedReaMM evaluates diagnosis from fixed inputs.Source: Limitations · p. 9
• Curated case reports may differ from noisier, incomplete, inconsistently documented electronic health records, and subtle cues in narratives or image selection may persist despite leakage controls.Curation improves data quality and diagnostic clarity but may limit generalization to hospital systems; residual cues could affect both human and model performance.Source: Limitations · p. 9
• Open-ended diagnostic accuracy omits calibration, risk awareness, and downstream impact; clinician comparisons use constrained multiple choice, and reported model performance may change as systems evolve.The authors argue that multimodal evidence integration and clinically grounded reasoning remain relevant challenges for future systems.Source: Limitations · p. 9

A.1 Image Modalities and Diagnostic Categories

• GPT-4.1 with image captions categorized images into 15 types, including 13 standard medical modalities, while diagnosis-based labeling mapped disease coverage across organ systems.The image categories include whole slide images, CT, MRI, ultrasound, endoscopy, and PET; disease labels were generated from final diagnoses.Source: §A.1 · pp. 11–12

A Dataset Statistics

• Benchmark statistics span image modalities, disease categories, and diagnostic-label distributions, portraying MedReaMM’s multimodal and multiorgan coverage.The appendix describes these statistics as highlighting clinical diversity and alignment with real-world diagnostic scenarios.Source: A Dataset Statistics · p. 11, §A.1 · p. 12

A.2 Image and Diagnosis Quantity Statistics

• 79% of cases have two or more diagnostic labels, and many contain multiple images, unlike single-image VQA benchmarks (He et al., 2020; Liu et al., 2021).The dataset comprises 1043 diagnostic terms.Source: §A.2 · p. 12, Table 5 · p. 12

A.3 Vocabulary Distribution Visualization

• Figure 6’s word clouds summarize clinical terms in structured inputs and diagnosis terms across cases, revealing lexical richness and diagnostic variability.The visualizations cover both textual inputs and diagnostic terms.Source: Figure 6 · p. 12, §A.3 · p. 13

B.1 Expert-Level Case Selection

• Rare diseases and atypical multisystem pathologies make MedReaMM a benchmark for long-tail clinical scenarios that typically require multidisciplinary expert consultation.It focuses on high-level clinical reasoning rather than routine primary-care screening.Source: §B.1 · p. 13

B.2 Information Density and Narrative Structure

• Expert-curated narratives distill clinicians’ cognitive output into high-density inputs for testing logical synthesis and evidence integration.The benchmark establishes a reasoning baseline before evaluation on noisier, fragmented real-world clinical data.Source: §B.2 · p. 13

B.3 Data Distribution and Scale

• 625 expert-validated cases with ICD-11 annotations concentrate in diagnostically uncertain fields such as internal medicine and oncology, prioritizing multimodal evidence depth over common-case breadth.This composition is intended to stress-test LMMs against expert-level requirements.Source: §B.3 · p. 13

B.4 Clinical Decision Support Alignment

• Top-5 and Top-10 recall align with decision support by measuring whether models identify correct diagnostic leads among broad differentials for clinician review.The metrics target the benchmark’s diagnostic-uncertainty setting.Source: §B.4 · p. 13

C ICD Encoding

• ICD-11 is described as the latest, comprehensive global classification standard, and the WHO API offers auto-code and flexible search, with flexible search recommended only when auto-code fails.The passages identify ICD’s broad adoption and ICD-11’s role in consistent international reporting.Source: C ICD Encoding · p. 13
• Ground-truth labels were generated using ICD-11 auto-code, then manually verified and corrected to ensure diagnostic-label accuracy and validity.Source: C ICD Encoding · p. 13
• For ICD-based evaluation matching, the procedure tries auto-code first, then flexible search, and counts a diagnosis as unmatched if neither yields a valid code.The process is summarized in Algorithm 1.Source: C ICD Encoding · pp. 13–14

D Human vs. Model Evaluation

• Human gastroenterologists achieved 10.87% accuracy and a 0.2201 Sorensen-Dice Coefficient, while LMMs showed stronger capacity to identify relevant diagnostic components.Table 6 compares diagnostic accuracy between gastroenterologists and LMMs on gastrointestinal disorders.Source: Table 6 · p. 14, D Human vs. Model Evaluation · p. 14
• The controlled comparison used 46 multiple-choice questions: 27 single-answer and 19 multiple-answer, with distractors drawn from models’ incorrect predictions.Cases came from a specialized MedReaMM gastrointestinal subset and paired each ground-truth diagnosis with distractors.Source: D Human vs. Model Evaluation · p. 14
• Humans and models received the same expert-interpreted image captions, separating diagnostic reasoning from image-interpretation expertise.Clinicians were given the original detailed image captions containing expert-level interpretations.Source: D Human vs. Model Evaluation · p. 14
• Three independent gastroenterologists were scored with strict exact-match accuracy and Sorensen-Dice overlap between predicted and reference diagnostic sets.For multiple-answer questions, strict accuracy required identifying every correct option.Source: D Human vs. Model Evaluation · p. 14
• The human-model gap was attributed partly to individual specialists working without peer consultation or external search on cases that often require multidisciplinary teams.Many benchmark cases are diagnostic dilemmas sourced from top-tier journals such as NEJM.Source: D Human vs. Model Evaluation · p. 14
• Specialists reported rare diseases and multisystem complications beyond their subspecialties, while LMMs’ broader knowledge and multimodal integration may aid complex, long-tail cases.The passage frames LMMs as a potential adjunct, not a replacement for clinical expertise.Source: D Human vs. Model Evaluation · p. 14

E.1 Results With Confidence Intervals

• The authors estimated 95% confidence intervals for selected models on Top-10 Primary Diagnosis Recall using 5,000 bootstrap resamples.They used the bootstrap method.Source: §E.1 · p. 14
• Overlapping intervals between adjacent models may indicate nonsignificant differences, while non-overlapping intervals suggest more reliable distinctions in diagnostic capability.The analysis concerns Primary Diagnosis Recall.Source: §E.1 · pp. 14–15, Figure 7 · p. 15
• The confidence intervals show a stepwise distribution in which models form distinct tiers for primary diagnosis recall.Figure 7 presents bootstrap-estimated 95% confidence intervals for Primary Diagnosis Recall.Source: §E.1 · p. 14, Figure 7 · p. 15

E.2 Prompt Robustness in Evaluation

• The prompt robustness experiment evaluated models with three semantically equivalent prompts that varied syntactically.Prompt 1 was used in the main evaluation; prompts 2 and 3 appear in the Appendix Prompts section (Figures 22 and 23).Source: §E.2 · p. 15
• Prompt robustness testing was motivated by prior studies.The experiment assessed whether evaluation outcomes were robust to prompt variation.Source: §E.2 · p. 15
• Across most tested models, performance fluctuations stayed within 3.3% for Top-5 Primary and Complete Diagnosis Recall.The authors interpret this as evidence that evaluation results are stable under minor prompt wording variations.Source: §E.2 · p. 15

F.1 Data Leakage

• The leakage analysis compared performance by case-report year, assuming 2025 cases were not in training data.The authors examined whether models might memorize cases from pretraining corpora.Source: §F.1 · p. 15

F.2 Performance Across Diagnostic Categories

• Performance was analyzed under Top-5 Primary, Top-5 Complete, Top-10 Primary, and Top-10 Complete Diagnosis Recall across nine disease categories.Cases were grouped using GPT-4.1 into cardiovascular, dermatologic, endocrine, gastrointestinal, congenital, infectious, neoplastic, neurological, and orthopedic disorders.Source: §F.2 · p. 15
• Figure 8 presents radar charts of representative-model performance across disease categories and evaluation settings.The charts cover the four recall settings and category groupings described in the analysis.Source: §F.2 · p. 15, Figure 8 · p. 16

F.1 Data Leakage

• No notable performance drop appeared on 2025 cases, suggesting little to no leakage or negligible evaluation impact.The analysis treated 2025 cases as absent from training data and examined performance by case-report year.Source: §F.1 · p. 15, Figure 9 · p. 16

F.2 Performance Across Diagnostic Categories

• Models generally performed better on common, visually distinct endocrine/metabolic and gastrointestinal disorders, but worse on neoplastic and neurological cases.The latter cases often require more nuanced clinical reasoning or involve subtle imaging features.Source: §F.2 · p. 16
• Gemini-2.5-Pro (Comanici et al., 2025) was relatively balanced, while MedGemma-4B (Sellergren et al., 2025) and Lingshu-32B (Xu et al., 2025) showed stronger category-specific weaknesses.Model performance gaps narrowed under more lenient recall settings, but category-level discrepancies persisted.Source: §F.2 · p. 16

G Human Annotation

• Over approximately 45 days, annotators reviewed 1,527 cases and retained 1,193 after screening for text and image label leakage.Text cases were removed when at least two annotators identified leakage; inappropriate image content was excluded or cropped when two or more agreed.Source: G Human Annotation · p. 16
• 625 cases formed the final MedReaMM dataset after clinical experts removed cases whose diagnoses could not be fully inferred from the inputs.Experts checked consistency with the original report and whether the diagnosis was logically inferable from the records and images.Source: G Human Annotation · pp. 16–17, Table 8 · p. 17

H Human Evaluation

• Three senior-year medical students independently rated seven diagnostic dimensions to compare their scores with GPT-5’s and measure inter-annotator agreement.The human ratings were based on a random sample of model outputs; the results are summarized in Table 9.Source: H Human Evaluation · p. 17, Table 9 · p. 17

I Prompts

• Claude Sonnet 4 extracted structured information from PDFs, while GPT-5 generated final captions that retained image metadata but removed clinical findings.The paper also used a prompt to extract reasoning paths from case reports.Source: I Prompts · p. 17
• The main evaluation prompt was also used as Prompt 1 in the robustness study, with Prompts 2 and 3 shown separately.These prompts appear in Figures 14, 22, and 23, respectively.Source: I Prompts · p. 17
• Prompts assessed disease knowledge, image understanding, and clinical reasoning, with separate corresponding evaluation prompts.The assessment prompts appear in Figures 17, 15, and 20, and their corresponding evaluation prompts in Figures 18, 16, and 21.Source: I Prompts · p. 17

Case Study

• A 35-year-old female with a four-year history of recurrent pneumonia was diagnosed with intralobar pulmonary sequestration complicated by actinomycosis, based on a left lower-lobe CT lesion and characteristic histopathological colonies.Source: J Case Study · pp. 17–18
• The missed co-infection may reflect models’ focus on obvious diagnoses and difficulty extracting fine-grained pathological details from images.These limitations constrain integration of multimodal clinical signals.Source: J Case Study · p. 18
• All models, including Gemini-2.5-Pro, missed actinomycosis despite Gemini-2.5-Pro identifying pulmonary sequestration.The case illustrates models prioritizing prominent cues while overlooking a clinically significant coexisting condition.Source: J Case Study · p. 18, Figure 11 · p. 20