Source-linked AI summary
Capabilities of GPT-5 on Multimodal Medical Reasoning
Shansong Wang, Mingzhe Hu, Qiang Li, Mojtaba Safari, Xiaofeng Yang
TL;DR
Medical decision support requires reasoning across heterogeneous text, structured, and imaging evidence, while existing evaluations provide limited unified evidence for such settings. This study benchmarks GPT-5 and comparison models across standardized medical QA and VQA tasks using zero-shot chain-of-thought prompting, finding consistent gains and above-human performance on controlled multimodal benchmarks.
Problem
Medical decision-making requires integrating heterogeneous evidence, while prior evaluations are predominantly text-centric and heterogeneous in datasets, prompting, and scoring.
Method
The study evaluates GPT-5 and comparison models across standardized text-based and multimodal medical QA benchmarks using a unified zero-shot chain-of-thought protocol.
Results
GPT-5 consistently outperformed baselines across QA and VQA benchmarks, including +29.26% multimodal reasoning and +26.18% multimodal understanding over GPT-4o on MedXpertQA MM.
Takeaways & Limitations
On controlled standardized evaluations, GPT-5 moved from human-comparable to above-human-expert performance, supporting its potential as a core component for multimodal clinical decision support.
Takeaways & Limitations
The evaluations use idealized standardized testing environments that do not fully encompass real-world medical complexity, uncertainty, and ethical considerations.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have enabled general-purpose systems to perform increasingly complex domain-specific reasoning without extensive fine-tuning. In the medical domain, decision-making often requires integrating heterogeneous information sources, including patient narratives, structured data, and medical images. This study positions GPT-5 as a generalist multimodal reasoner for medical decision support and systematically evaluates its zero-shot chain-of-thought reasoning performance on both text-based question answering and visual question answering tasks under a unified protocol. We benchmark GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4o-2024-11-20 against standardized splits of MedQA, MedXpertQA (text and multimodal), MMLU medical subsets, USMLE self-assessment exams, and VQA-RAD. Results show that GPT-5 consistently outperforms all baselines, achieving state-of-the-art accuracy across all QA benchmarks and delivering substantial gains in multimodal reasoning. On MedXpertQA MM, GPT-5 improves reasoning and understanding scores by +29.26% and +26.18% over GPT-4o, respectively, and surpasses pre-licensed human experts by +24.23% in reasoning and +29.40% in understanding. In contrast, GPT-4o remains below human expert performance in most dimensions. A representative case study demonstrates GPT-5's ability to integrate visual and textual cues into a coherent diagnostic reasoning chain, recommending appropriate high-stakes interventions. Our results show that, on these controlled multimodal reasoning benchmarks, GPT-5 moves from human-comparable to above human-expert performance. This improvement may substantially inform the design of future clinical decision-support systems.
1 Introduction
Medical decision-making combines narratives, structured indicators, and images, creating a need for reliable multimodal reasoning without extensive domain-specific fine-tuning. The study evaluates GPT-5 as a generalist reasoner under a unified protocol to address this gap.
- Medical problems often require jointly reasoning over patient history, structured indicators, and medical images.
- Existing evaluations are predominantly text-centric and heterogeneous in datasets, prompting, and scoring.
- The study positions GPT-5 as a generalist multimodal reasoner for controlled comparisons with GPT-4.
- GPT-5 is evaluated on question answering and visual question answering using standardized splits, prompts, exemplars, and zero-shot chain-of-thought supervision.
2 Methodology
The methodology benchmarks GPT-5 across text-only and multimodal medical datasets using a standardized zero-shot chain-of-thought protocol. Multimodal items present associated images with textual questions, followed by a text-only answer-convergence step.
- 2.1 Datasets: The benchmark suite spans MedQA, MMLU-Medical, USMLE Self Assessment, MedXpertQA, and VQA-RAD.
- 2.1 Datasets: MedXpertQA contains 4,460 questions across 17 specialties and 11 body systems, including text-only and multimodal subsets.
- 2.1 Datasets: MedXpertQA’s multimodal subset combines clinical questions with medical images, patient records, and examination results.
- 2.2 Prompting Design: GPT-5 is evaluated with zero-shot chain-of-thought prompting that first elicits a rationale and then restricts the response to a discrete choice.
- 2.2 Prompting Design: For multimodal items, associated images are appended to the first user message, while the subsequent convergence step remains purely textual.
3 Results
GPT-5 shows consistent gains over GPT-4o and smaller variants across text, USMLE, multimodal, and human-expert comparisons, with especially large improvements on reasoning-intensive tasks.
- Text-based QA: 95.84% on MedQA US 4-option, a 4.80% absolute improvement over GPT-4o, shows stronger performance in clinical question contexts.
- Text-based QA: 26.33% and 25.30% gains in MedXpertQA Text reasoning and understanding, respectively, mark the largest text-based improvements over GPT-4o.
- USMLE Self Assessment: 95.22% average performance across USMLE Steps 1–3, +2.88% versus GPT-4o, included the largest step-specific margin of +4.17% on Step 2.
- VQA: +29.26% reasoning and +26.18% understanding gains on MedXpertQA MM over GPT-4o demonstrate a substantial multimodal improvement.
- VQA: 70.92% on VQA-RAD placed GPT-5 slightly below GPT-5-mini’s 74.90%, while a MedXpertQA MM case showed integration of imaging, laboratory values, and physical signs.
- Comparison with human experts: GPT-5 surpassed pre-licensed human experts by +15.22% to +29.40% across text and multimodal reasoning and understanding dimensions.
4 Disscusion
GPT-5 shows substantial gains in multimodal and reasoning-intensive medical tasks, while its human-comparable benchmark performance remains bounded by idealized evaluation settings and dataset-specific variation.
- Multimodal reasoning: +26–36% improvements over GPT-4o were observed in multimodal settings such as MedXpertQA MM.These datasets require tight integration of image-derived evidence with textual patient data.
- Reasoning-intensive tasks: GPT-5’s gains are most pronounced on reasoning-intensive tasks, including MedXpertQA Text and USMLE Step 2.The discussion associates these results with chain-of-thought prompting and enhanced internal reasoning capacity.
- Human comparison and scope: GPT-5 surpasses pre-licensed medical professionals in controlled QA/VQA evaluations, but those settings do not capture real-world clinical complexity, uncertainty, or ethical considerations.The comparison supports clinical decision-support potential within the evaluated scope, not unrestricted real-world equivalence.
- Dataset-specific variation: GPT-5 scored slightly lower than GPT-5-mini on VQA-RAD, and future work proposes adaptive prompting or calibration for small-domain multimodal tasks.The discussion attributes the discrepancy tentatively to scaling-related differences in reasoning calibration.
5 Conclusion
The study reports a controlled longitudinal evaluation in which GPT-5 achieves substantial gains across diverse medical QA and VQA benchmarks, especially on reasoning-intensive and multimodal tasks. It concludes that GPT-5 has potential as a core component for multimodal clinical decision support, while noting that benchmark conditions do not fully represent real-world practice.
- The study presents a controlled longitudinal comparison of GPT-5 with GPT-4o-2024-11-20, smaller GPT-5 variants, and human experts under standardized zero-shot CoT prompting.
- Across diverse QA and VQA benchmarks, GPT-5 demonstrates substantial and consistent gains, particularly in reasoning-intensive and multimodal tasks.
- GPT-5 surpasses trained medical professionals on MedXpertQA MM by large margins, moving from GPT-4o’s near-human performance to clear super-human proficiency.
- The results highlight GPT-5’s potential as a reliable core component for multimodal clinical decision support that integrates complex textual and visual information.
- The benchmarks reflect idealized testing conditions and may not fully capture the variability, uncertainty, and ethical considerations of real-world practice.