PaperlayerView exact revision on arXiv

Source-linked AI summary

Capabilities of GPT-5 on Multimodal Medical Reasoning

Shansong Wang, Mingzhe Hu, Qiang Li, Mojtaba Safari, Xiaofeng Yang

arXiv:2508.08224v2cs.CLcs.AI

TL;DR

Medical decision support requires reasoning across heterogeneous text, structured, and imaging evidence, while existing evaluations provide limited unified evidence for such settings. This study benchmarks GPT-5 and comparison models across standardized medical QA and VQA tasks using zero-shot chain-of-thought prompting, finding consistent gains and above-human performance on controlled multimodal benchmarks.

  • Problem

    Medical decision-making requires integrating heterogeneous evidence, while prior evaluations are predominantly text-centric and heterogeneous in datasets, prompting, and scoring.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates GPT-5 and comparison models across standardized text-based and multimodal medical QA benchmarks using a unified zero-shot chain-of-thought protocol.

    Source: §2.2 · pp. 2–3, §3.1 · p. 3, §3.2 · p. 3

  • Results

    GPT-5 consistently outperformed baselines across QA and VQA benchmarks, including +29.26% multimodal reasoning and +26.18% multimodal understanding over GPT-4o on MedXpertQA MM.

    Source: §3.3 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    On controlled standardized evaluations, GPT-5 moved from human-comparable to above-human-expert performance, supporting its potential as a core component for multimodal clinical decision support.

    Source: §3.4 · p. 6, §5 · p. 8

    The evaluations use idealized standardized testing environments that do not fully encompass real-world medical complexity, uncertainty, and ethical considerations.

    Source: §4 · p. 7

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have enabled general-purpose systems to perform increasingly complex domain-specific reasoning without extensive fine-tuning. In the medical domain, decision-making often requires integrating heterogeneous information sources, including patient narratives, structured data, and medical images. This study positions GPT-5 as a generalist multimodal reasoner for medical decision support and systematically evaluates its zero-shot chain-of-thought reasoning performance on both text-based question answering and visual question answering tasks under a unified protocol. We benchmark GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4o-2024-11-20 against standardized splits of MedQA, MedXpertQA (text and multimodal), MMLU medical subsets, USMLE self-assessment exams, and VQA-RAD. Results show that GPT-5 consistently outperforms all baselines, achieving state-of-the-art accuracy across all QA benchmarks and delivering substantial gains in multimodal reasoning. On MedXpertQA MM, GPT-5 improves reasoning and understanding scores by +29.26% and +26.18% over GPT-4o, respectively, and surpasses pre-licensed human experts by +24.23% in reasoning and +29.40% in understanding. In contrast, GPT-4o remains below human expert performance in most dimensions. A representative case study demonstrates GPT-5's ability to integrate visual and textual cues into a coherent diagnostic reasoning chain, recommending appropriate high-stakes interventions. Our results show that, on these controlled multimodal reasoning benchmarks, GPT-5 moves from human-comparable to above human-expert performance. This improvement may substantially inform the design of future clinical decision-support systems.

1 Introduction

Medical decision-making combines narratives, structured indicators, and images, creating a need for reliable multimodal reasoning without extensive domain-specific fine-tuning. The study evaluates GPT-5 as a generalist reasoner under a unified protocol to address this gap.

  • Medical problems often require jointly reasoning over patient history, structured indicators, and medical images.Source: §1 · p. 1
  • Existing evaluations are predominantly text-centric and heterogeneous in datasets, prompting, and scoring.Source: §1 · p. 2
  • The study positions GPT-5 as a generalist multimodal reasoner for controlled comparisons with GPT-4.Source: §1 · p. 2
  • GPT-5 is evaluated on question answering and visual question answering using standardized splits, prompts, exemplars, and zero-shot chain-of-thought supervision.Source: §1 · p. 2

2 Methodology

The methodology benchmarks GPT-5 across text-only and multimodal medical datasets using a standardized zero-shot chain-of-thought protocol. Multimodal items present associated images with textual questions, followed by a text-only answer-convergence step.

2.1 Datasets

  • The benchmark suite spans MedQA, MMLU-Medical, USMLE Self Assessment, MedXpertQA, and VQA-RAD.Source: §2.1 · p. 2
  • MedXpertQA contains 4,460 questions across 17 specialties and 11 body systems, including text-only and multimodal subsets.Source: §2.1 · p. 2
  • MedXpertQA’s multimodal subset combines clinical questions with medical images, patient records, and examination results.Source: §2.1 · p. 2

2.2 Prompting Design

  • GPT-5 is evaluated with zero-shot chain-of-thought prompting that first elicits a rationale and then restricts the response to a discrete choice.Source: §2.2 · p. 2
  • For multimodal items, associated images are appended to the first user message, while the subsequent convergence step remains purely textual.Source: §2.2 · p. 3

3 Results

GPT-5 shows consistent gains over GPT-4o and smaller variants across text, USMLE, multimodal, and human-expert comparisons, with especially large improvements on reasoning-intensive tasks.

Text-based QA

  • 95.84% on MedQA US 4-option, a 4.80% absolute improvement over GPT-4o, shows stronger performance in clinical question contexts.Source: §3.1 · p. 3
  • 26.33% and 25.30% gains in MedXpertQA Text reasoning and understanding, respectively, mark the largest text-based improvements over GPT-4o.Source: §3.1 · p. 3

USMLE Self Assessment

  • 95.22% average performance across USMLE Steps 1–3, +2.88% versus GPT-4o, included the largest step-specific margin of +4.17% on Step 2.Source: §3.2 · p. 3

VQA

  • +29.26% reasoning and +26.18% understanding gains on MedXpertQA MM over GPT-4o demonstrate a substantial multimodal improvement.Source: §3.3 · p. 6
  • 70.92% on VQA-RAD placed GPT-5 slightly below GPT-5-mini’s 74.90%, while a MedXpertQA MM case showed integration of imaging, laboratory values, and physical signs.Source: §3.3 · p. 6

Comparison with human experts

  • GPT-5 surpassed pre-licensed human experts by +15.22% to +29.40% across text and multimodal reasoning and understanding dimensions.Source: §3.4 · p. 6

4 Disscusion

GPT-5 shows substantial gains in multimodal and reasoning-intensive medical tasks, while its human-comparable benchmark performance remains bounded by idealized evaluation settings and dataset-specific variation.

Multimodal reasoning

  • +26–36% improvements over GPT-4o were observed in multimodal settings such as MedXpertQA MM.These datasets require tight integration of image-derived evidence with textual patient data.Source: §4 · p. 7

Reasoning-intensive tasks

  • GPT-5’s gains are most pronounced on reasoning-intensive tasks, including MedXpertQA Text and USMLE Step 2.The discussion associates these results with chain-of-thought prompting and enhanced internal reasoning capacity.Source: §4 · p. 7

Human comparison and scope

  • GPT-5 surpasses pre-licensed medical professionals in controlled QA/VQA evaluations, but those settings do not capture real-world clinical complexity, uncertainty, or ethical considerations.The comparison supports clinical decision-support potential within the evaluated scope, not unrestricted real-world equivalence.Source: §4 · p. 7

Dataset-specific variation

  • GPT-5 scored slightly lower than GPT-5-mini on VQA-RAD, and future work proposes adaptive prompting or calibration for small-domain multimodal tasks.The discussion attributes the discrepancy tentatively to scaling-related differences in reasoning calibration.Source: §4 · pp. 7–8

5 Conclusion

The study reports a controlled longitudinal evaluation in which GPT-5 achieves substantial gains across diverse medical QA and VQA benchmarks, especially on reasoning-intensive and multimodal tasks. It concludes that GPT-5 has potential as a core component for multimodal clinical decision support, while noting that benchmark conditions do not fully represent real-world practice.

  • The study presents a controlled longitudinal comparison of GPT-5 with GPT-4o-2024-11-20, smaller GPT-5 variants, and human experts under standardized zero-shot CoT prompting.Source: §5 · p. 8
  • Across diverse QA and VQA benchmarks, GPT-5 demonstrates substantial and consistent gains, particularly in reasoning-intensive and multimodal tasks.Source: §5 · p. 8
  • GPT-5 surpasses trained medical professionals on MedXpertQA MM by large margins, moving from GPT-4o’s near-human performance to clear super-human proficiency.Source: §5 · p. 8
  • The results highlight GPT-5’s potential as a reliable core component for multimodal clinical decision support that integrates complex textual and visual information.Source: §5 · p. 8
  • The benchmarks reflect idealized testing conditions and may not fully capture the variability, uncertainty, and ethical considerations of real-world practice.Source: §5 · p. 8

Related papers

Loading 2508.08224v2…