PaperlayerView exact revision on arXiv

Source-linked AI summary

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, Bowen Zhou

arXiv:2501.18362v3cs.AIcs.CLcs.CVcs.LG

TL;DR

Existing medical benchmarks provide insufficient specialty coverage, difficulty, and clinical realism for evaluating expert-level medical reasoning. MedXpertQA addresses these gaps with diverse text and multimodal benchmark subsets constructed through filtering, synthesis, augmentation, and expert review. Evaluations of 18 models show limited performance, particularly on complex reasoning tasks, while its Reasoning subset targets o1-like models.

  • Problem

    Current medical benchmarks have inadequate specialty coverage, insufficient difficulty, and limited clinical relevance for evaluating expert-level knowledge and reasoning in realistic scenarios.

  • Method

    MedXpertQA combines text and multimodal benchmark subsets with rigorous filtering, data synthesis, augmentation, and expert review, including a reasoning-oriented subset.

  • Results

    18 evaluated proprietary and open-source models show limited performance on MedXpertQA, especially on complex medical reasoning tasks.

  • Takeaways & Limitations

    MedXpertQA supports comprehensive evaluation of expert-level medical knowledge and reasoning across specialties, body systems, clinical tasks, and multimodal information.

  • Takeaways & Limitations

    Direct performance comparisons between the Text and MM subsets are not informative because they differ in answer-choice counts and filtering rigor.

Abstract

from arXiv · show

We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM introduces expert-level exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions. MedXpertQA applies rigorous filtering and augmentation to address the insufficient difficulty of existing benchmarks like MedQA, and incorporates specialty board questions to improve clinical relevance and comprehensiveness. We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on \benchmark. Moreover, medicine is deeply connected to real-world decision-making, providing a rich and representative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop a reasoning-oriented subset to facilitate the assessment of o1-like models. Code and data are available at: https://github.com/TsinghuaC3I/MedXpertQA

1. Introduction

Existing medical benchmarks have limited specialty coverage, insufficient difficulty, and multimodal designs that do not reflect complex clinical scenarios. MedXpertQA addresses these gaps with a challenging benchmark built through filtering, synthesis, augmentation, and expert review, then evaluates leading models and medical reasoning.

  • Existing text benchmarks lack fine-grained coverage of diverse, specialized diagnostic scenarios and do not sufficiently challenge advanced medical AI.
  • Traditional multimodal benchmarks mainly assess basic visual perception and medical knowledge rather than expert-level reasoning across diagnostic stages.
  • MedXpertQA provides challenging Text and MM subsets covering broad specialties, body systems, clinical tasks, and diverse visual information.
  • Its construction combines authoritative data collection, difficulty and similarity filtering, question and option augmentation, synthesis, and multiple expert reviews.
  • The benchmark evaluates 18 models and includes a Reasoning subset designed for assessing medical reasoning in o1-like inference-time scaled models.

2. Related Work

Existing medical benchmarks provide specialized or broader evaluations, but remain limited in scope, image diversity, clinical realism, or difficulty. MedXpertQA is presented against these limitations as a more comprehensive benchmark design.

  • Specialized multimodal benchmarks offer extensive evaluation within intended domains but have constrained scope and limited generalizability.
  • Figure 2 contrasts MedXpertQA MM’s diverse sources, image types, and question attributes with a typical traditional benchmark example.
  • Recent general-purpose multimodal benchmarks broaden evaluation but cover limited image types and are often built automatically from image captions.
  • Existing text benchmarks emphasize general medical knowledge or closed-domain questions but do not thoroughly evaluate specialized diagnostic scenarios aligned with clinical practice.

3. MedXpertQA

MedXpertQA is a universal benchmark with challenging text and multimodal subsets designed to evaluate expert-level medical knowledge and reasoning across broad clinical coverage.

  • MedXpertQA contains 4,460 questions across 17 specialties, 11 body systems, and 3 task categories.
  • Its Text and MM subsets each include a five-question few-shot development set and a test set.
  • The MM subset contains 2,005 multimodal questions associated with 2,839 images.
  • Questions come from medical licensing-level or higher examinations, and the benchmark introduces specialty evaluations to simulate specialized medical scenarios.
  • The design prioritizes both difficulty and diversity across multiple dimensions rather than difficulty alone.

Tasks & Subtasks

MedXpertQA combines broad task, modality, and specialty coverage with multi-stage filtering, augmentation, expert review, and comparisons demonstrating its clinical and multimodal richness. It also includes a reasoning-oriented evaluation focus and reports limited annotation errors.

  • Tasks & Subtasks: The benchmark spans medical imaging, diagrams, charts, documents, tables, and semistructured clinical information.
  • Tasks & Subtasks: MedXpertQA classifies questions into Diagnosis, Treatment Planning, and Basic Medicine, with additional fine-grained subtasks.
  • Benchmark Construction: Questions are collected from authoritative examinations and textbooks, including USMLE, COMLEX, radiology board exams, and other specialty sources.
  • Benchmark Construction: Human difficulty filtering uses Brier scores over answer-choice response distributions, with adaptive thresholds based on prior expert ratings.
  • Benchmark Construction: Similarity filtering removes highly redundant questions using medical embeddings and pairwise semantic comparison.
  • Benchmark Construction: Question and option augmentation rephrases questions while preserving information, increasing difficulty and robustness while mitigating leakage risk.
  • Tasks & Subtasks: Approximately 4.3% of reviewed questions were incorrectly task-labeled, based on a random 10% review subset.

4. Experiments

The experiments evaluate leading proprietary and open-source language and multimodal models on MedXpertQA, finding that current systems remain challenged, especially on complex reasoning. GPT-4o and DeepSeek-R1 lead their respective model groups, while direct comparisons between Text and MM are not informative.

  • Experimental Setup: 18 proprietary and open-source LMMs and LLMs are evaluated on MedXpertQA, including inference-time scaled models.Evaluation uses zero-shot CoT prompting unless otherwise specified, with greedy decoding when available.
  • Overall Results: Current leading models achieve low accuracies overall, demonstrating that MedXpertQA challenges state-of-the-art systems.The main results are reported in Tables 4 and 5.
  • LMM Comparison: GPT-4o consistently performs best among vanilla LMMs across all subsets.Gemini-2.0-Flash follows GPT-4o among vanilla LMMs and performs strongly on MedXpertQA MM.
  • LLM Comparison: DeepSeek-R1 shows the strongest LLM performance, particularly on the Reasoning set, where it substantially outperforms other models.Qwen2.5-32B performs nearly equivalently to random selection on the Reasoning set.
  • Medical Reasoning Performance: Vanilla LMMs and LLMs perform worse on the Reasoning subset than on the Understanding subset, whereas this gap narrows for o1-like inference-time scaled models.Visual reasoning differences among vanilla LMMs are marked, while their textual reasoning performance is relatively close.
  • Caveat: Direct performance comparisons between MedXpertQA MM and Text are not informative because the subsets differ in answer-choice counts and filtering rigor.Models generally score higher on MM, which has fewer answer choices, while Text uses more rigorous filtering parameters.

5. Analysis

The analysis examines leakage risk, inference-time scaling, and system-specific performance. Data synthesis further reduces already low leakage risk, inference-time scaling particularly improves reasoning accuracy, and GPT-4o’s performance varies across medical systems.

  • Data Leakage Analysis: Data leakage risk is already relatively low before synthesis and is further reduced after Question and Option Augmentation.The analysis uses perplexity, ROUGE-L, and edit-distance similarity at the instance level.
  • System Analysis: Figure 5 compares system-specific accuracy with each system’s representation in the correctly answered set versus the full question set.Accuracy bars show per-system accuracy, while Percentage Change bars show the difference between these proportions.
  • Impact of Inference-Time Scaling: Inference-time scaling produces more pronounced accuracy changes on the Reasoning set than on the Understanding set across the compared model groups.Reasoning accuracy shows a clear upward trend, while Understanding accuracy fluctuates and sometimes declines.
  • Impact of Inference-Time Scaling: The scaling results suggest that inference-time scaling can improve complex medical reasoning skills without additional medical training data.The comparison includes paired text and multimodal models with their corresponding backbones, plus DeepSeek-R1 and DeepSeek-V3.
  • System Analysis: GPT-4o has comparatively stronger performance on Integumentary questions, whose proportion increases in its correctly answered set.The analysis also reports lower accuracy on some other systems, but the supplied passage truncates that comparison.

6. Conclusion

The paper introduces MedXpertQA as a challenging benchmark for expert-level medical knowledge and reasoning in realistic clinical scenarios. It combines broad coverage, multimodal clinical information, leakage mitigation, expert review, model evaluation, and a reasoning-oriented subset.

  • Conclusion: MedXpertQA evaluates expert-level medical knowledge and reasoning across diverse specialties, body systems, and clinical tasks.Its multimodal subset incorporates challenging examination questions with comprehensive clinical information.
  • Conclusion: The benchmark addresses inadequate specialty coverage, insufficient difficulty, and limited clinical relevance in existing medical benchmarks.Its construction includes data synthesis to mitigate leakage risk and expert review to support accuracy and validity.
  • Conclusion: Evaluation of 18 leading models shows that current AI systems still face challenges on complex medical reasoning tasks.The paper also analyzes system-specific medical capabilities and constructs a reasoning-oriented subset.

Impact Statement

MedXpertQA is positioned as a comprehensive benchmark for healthcare application-driven machine learning, while emphasizing responsible development, oversight, and data-use restrictions. The paper also notes that benchmark performance alone does not establish clinical readiness because MedXpertQA remains a synthetic evaluation framework.

  • MedXpertQA is introduced as a comprehensive medical benchmark intended to advance healthcare application-driven machine learning.
  • Responsible deployment requires attention to data privacy, algorithmic bias, excessive dependence on automated systems, transparency, explainability, and continuous professional validation.
  • The dataset is restricted to model or human medical evaluation and must not be used for training or publicly shared in example form.
  • MedXpertQA uses publicly accessible sources, retains only a small percentage of source data, and applies rephrasing and option shuffling to support U.S. fair-use compliance.
  • Benchmark performance does not imply clinical readiness because MedXpertQA remains synthetic and safe medical-AI use requires safeguards beyond benchmark scores.

C. Additional Implementation Details

The implementation details specify the AI experts used for question filtration and the prompting and answer-extraction adjustments applied to selected reasoning models. These choices accommodate model-specific interface constraints during evaluation.

  • Basic AI experts filter text and multimodal questions, while GPT-4o and Claude 3.5 Sonnet serve as advanced AI experts.
  • For o1 and DeepSeek-R1, the evaluation follows recommended prompting guidelines and omits the system prompt.
  • DeepSeek-R1 receives a boxed-answer instruction, while temperature is unsupported for both o1 and DeepSeek-R1.
  • QVQ-72B-Preview is evaluated in a single conversation round, with answer extraction adapted for choices inside boxed expressions.

D. Error Analysis

The error analysis classifies incorrect answers into reasoning, perception, question understanding, medical knowledge, and formatting categories. Across evaluated models, reasoning errors dominate, while perceptual errors are especially prominent in multimodal questions.

  • The analysis defines five error types: reasoning process, perceptual, question understanding, lack of medical knowledge, and formatting errors.
  • Reasoning Process Errors occur when flawed key reasoning steps lead to an incorrect answer.
  • Perceptual Errors occur only in MedXpertQA MM when image misunderstanding or misinterpretation causes the incorrect answer.
  • For each analyzed model, 200 incorrect answers are sampled separately from the Text and MM subsets and labeled using question, answer, explanation, and rationale information.
  • Across gpt-4o-2024-11-20, claude-3-5-sonnet-20241022, and gemini-1.5-pro, Reasoning Process Errors are high in both subsets, while Perceptual Errors are prominent in MM.
  • The expert review process records and statistically summarizes errors identified and corrected in augmented questions and options.

F. Full Results of Performance Comparison

This section presents model results on MedXpertQA Text alongside other text benchmarks and includes representative medical questions spanning pediatric, cardiopulmonary, neurologic, and musculoskeletal scenarios. The listed materials therefore cover both comparative evaluation and question diversity.

  • Table 8 reports different model performances on MedXpertQA Text and other text benchmarks.
  • The example questions span pediatric emergency presentations, including croup-like symptoms and brief resolved unexplained events.
  • Other examples address fatigue and dyspnea after travel, brainstem neuroanatomy, acute unilateral weakness, and cervical musculoskeletal dysfunction.

G.2. Error Cases

The error cases expose failures in multimodal medical reasoning, including selecting the first plausible diagnosis, misreading fine-grained ECG patterns, and mishandling medication effects. Other examples show gaps in age-related physiology and clinical coagulation reasoning.

  • Multimodal reasoning errors: The model chose EMG for post-dislocation shoulder abduction loss, overlooking that a rotator cuff tear better fit the patient’s demographic and symptoms and required shoulder MRI.The analysis characterizes this as failure to weigh competing diagnoses rather than a lack of basic axillary-nerve knowledge.
  • Multimodal reasoning errors: The model misclassified atrial flutter as atrial fibrillation because it failed to recognize flutter waves and the regularity of atrial activations in the ECG.The ECG showed 3:1 and 4:1 AV nodal block with characteristic flutter waves.
  • Medical knowledge errors: In the hypoglycemia case, the model attributed the condition to atenolol despite evidence that aspirin was the correct answer for the described diabetic patient.The critique notes that atenolol may mask warning signs but is less suitable as the cause of hypoglycemia, whereas aspirin can decrease glucose levels in diabetic patients.
  • Medical knowledge errors: The model selected decreased growth hormone instead of decreased melatonin for an older man with less refreshing sleep, failing to distinguish their effects on sleep.The analysis states that growth hormone is associated primarily with slow-wave sleep, while decreased melatonin more directly explains the reported lack of restfulness.
  • Clinical reasoning: The evaluation case attributes new bleeding in a cirrhotic patient receiving broad-spectrum antibiotics to vitamin K deficiency superimposed on liver dysfunction rather than improving-sepsis DIC.The cited reasoning links antibiotics to loss of gut-flora vitamin K synthesis and considers DIC less likely because sepsis was improving.
Loading 2501.18362v3…