Source-linked AI summary
Capabilities of GPT-4 on Medical Challenge Problems
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, Eric Horvitz
TL;DR
The paper addresses how well a general-purpose language model can solve medical challenge problems without specialized medical training or elaborate prompting. It evaluates GPT-4 against earlier and medically adapted models across examinations and benchmarks, while probing calibration, memorization, image-related questions, and qualitative reasoning. GPT-4 achieves strong medical benchmark performance, including substantially higher USMLE scores than GPT-3.5, and shows better calibration than its predecessor.
Problem
The paper asks whether a general-purpose GPT-4 model can perform medical problem solving without specialized training or clinical-task engineering, an important question for medical applications.
Method
The study comparatively evaluates GPT-4, GPT-3.5, and Flan-PaLM 540B on official USMLE materials and MultiMedQA-related datasets, using simple zero-shot and five-shot prompts while probing calibration, memorization, image-related questions, and qualitative reasoning.
Results
GPT-4 significantly outperforms GPT-3.5 and Flan-PaLM 540B across the medical evaluations, including USMLE averages of 86.65% and 86.7% versus GPT-3.5’s 53.61% and 58.78%.
Takeaways & Limitations
The findings support potential uses of GPT-4 in medical education, assessment, and assistance for healthcare professionals, while emphasizing accuracy, fairness, safety, and broader impacts.
Takeaways & Limitations
Calibration is evaluated only for multiple-choice question answering, while calibration of long-form generative outputs remains an open research area.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.
1 Introduction
The paper evaluates GPT-4’s general-purpose medical problem-solving capabilities against earlier and medically adapted models. It examines performance across medical examinations and benchmarks, alongside calibration, memorization, multimodal-question behavior, and qualitative educational interaction.
- The study compares GPT-4 with GPT-3.5 and Flan-PaLM 540B on medical challenge problems and benchmark datasets.GPT-4 is evaluated as a general-purpose model rather than one specialized through medical training or clinical-task engineering.
- The evaluation covers official USMLE practice materials and MultiMedQA-related medical benchmarks to assess medical competency and reasoning.
- 86.65% and 86.7% are GPT-4’s average scores on USMLE Self-Assessment and Sample Exam materials, versus 53.61% and 58.78% for GPT-3.5.
- The paper investigates text-only versus image-reliant questions, calibration, possible examination-content memorization, and qualitative medical reasoning.
- The qualitative case study examines GPT-4’s ability to explain medical reasoning, personalize explanations for students, and interactively construct counterfactual medical scenarios.
2 Methodology
The methodology benchmarks text-only GPT-4 and GPT-3.5 on six medical datasets using standardized minimal prompts, including zero-shot and randomly sampled five-shot settings. The study also uses prompt templates and constrained answer generation for multiple-choice evaluation.
- The study evaluates text-only GPT-4, explicitly excluding its vision capabilities, while examining questions that may refer to visual media.
- Six datasets cover medical knowledge and reasoning, including two NBME-sourced USMLE datasets and four publicly available benchmarks.
- Prompting: The experiments use the same prompt structure as prior work, with question-specific values substituted into a multiple-choice template.
- Prompting: Zero-shot prompts omit few-shot examples, while five-shot evaluation uses five randomly drawn exemplars from the remainder of each dataset under leave-one-out cross validation.
- Prompting: The benchmark uses minimal context and one-generation-token inference, with logit_bias restricting outputs to valid answer choices such as A-D.
- Evaluation design: The baseline deliberately avoids chain-of-thought, retrieval-augmented generation, and ensembling while comparing GPT-4 and GPT-3.5 across zero-shot and five-shot settings.
3 Performance of GPT-4 on Medical Competency Exams
GPT-4 substantially outperformed earlier models on official USMLE practice materials, clearing the passing threshold by a wide margin. Despite receiving no referenced media, it retained strong performance on visual-media questions, while alignment reduced benchmark performance relative to GPT-4-base.
- USMLE results: Over 30 percentage points: GPT-4 improved on both official USMLE exams compared with GPT-3.5.The official materials were sourced from the NBME; independently reported ChatGPT comparisons were also included for the Sample Exam.
- USMLE results: Approximately 60% correct is the usual passing benchmark, and GPT-4 cleared it by a large margin.The cited threshold varies by year, while GPT-3.5 was approaching it.
- Language- and vision-centric challenges: 70-80% prediction accuracies: GPT-4 performed well on questions referencing media that were not passed to the model.Performance was best on pure-text questions, but remained strong on media-element questions across both exams.
- Language- and vision-centric challenges: GPT-4 often selected reasonable answers without seeing relevant images by using logical reasoning and test-taking strategies.The paper illustrates this behavior through an interactive investigation of a question involving a photograph.
- Influence of alignment and safety tuning: 3-5%: GPT-4-base performed better than the publicly released GPT-4 across all 14 experimental datasets.Both models showed consistently strong performance, and the experiments connect the difference to alignment and safety or instruction-following tuning.
4 Medical Challenge Benchmarks
GPT-4 outperformed GPT-3.5 and Flan-PaLM 540B across nearly all evaluated MultiMedQA datasets using direct zero-shot prompts, with especially large gains on MedQA and MedMCQA.
- The benchmarks cover MedQA, PubMedQA, MedMCQA, and medical components of MMLU, while three MultiMedQA datasets were not tested.LiveQA, MedicationQA, and HealthSearchQA require long-answer evaluation or were unavailable.
- GPT-4 outperforms GPT-3.5 and Flan-PaLM 540B on every MultiMedQA dataset except PubMedQA.GPT-4 and GPT-3.5 were evaluated with zero-shot direct prompts.
- GPT-4 performance was reported across stratified MedQA and MMLU subcomponents, including multiple language and answer-option variants of MedQA.The English/United States MedQA data include standard five-option and simplified four-option versions.
- GPT-4 shows a large boost on MedQA and MedMCQA with a much simpler zero-shot prompt.The result continues a pattern in which less prompting effort is needed for strong performance with newer model generations.
5 Calibration
The paper evaluates whether model confidence tracks answer correctness in multiple-choice medical questions. GPT-4 is significantly better calibrated than GPT-3.5, although calibration of long-form generation remains unmeasured.
- Calibration plots bin predictions by estimated probability and compare each bin’s average probability with its true positivity rate.The multiple-choice adaptation uses the selected answer choice’s probability for each question.
- GPT-4’s calibration is significantly better than GPT-3.5’s on both official USMLE datasets.Calibration compares predicted answer probabilities with true outcome frequencies.
- GPT-4 answers assigned an average probability of 0.96 were correct 93% of the time, compared with 55% for GPT-3.5 at a similar probability.These values illustrate the difference in agreement between predicted confidence and observed correctness.
- The experiment is limited to multiple-choice question answering because long-form generation probabilities are an open research problem.The reported multiple-choice results suggest calibration may increase with model scale.
6 Directions and Limitations
The paper examines prompting, exemplar selection, memorization detection, and benchmark scope as directions and limitations. It finds limited gains from basic prompting and no detected USMLE memorization, while emphasizing important evaluation boundaries.
- Directions and Limitations: Basic two-stage chain-of-thought prompting did not improve GPT-4’s USMLE performance.The method first requests step-by-step reasoning and then asks for a final prediction using the prior generation.
- Directions and Limitations: Random versus expert-curated few-shot exemplars produced negligible performance differences across the tested datasets.The comparison involved GPT-4 and GPT-3.5.
- Directions and Limitations: MELD is a black-box Levenshtein-based detector that searches for near-exact generated matches to benchmark samples as evidence of memorization.The procedure is intended to identify potential training-data leakage.
- Directions and Limitations: MELD has high precision but unknown recall, so failing to detect a match does not show that data were absent from training.The authors therefore interpret non-detection as absence of evidence rather than proof of non-exposure.
- Directions and Limitations: MELD found no memorization evidence in the tested official USMLE datasets but found strong evidence for SQuAD 2.0 and Newsgroup Sentiment Analysis.GPT-4 regenerated SQuAD questions with 99% overlap 17% of the time, but did not reach 50% overlap on either USMLE dataset.
- Directions and Limitations: The benchmarks focus mainly on multiple-choice questions and exclude quantitative evaluation of interactive Step 3 case simulations.Because recent live-exam questions and scoring criteria were unavailable, reported metrics may not indicate performance on an actual USMLE exam.
7 Beyond Correct Answers: Probing Capabilities
GPT-4’s interactive case-study behavior extends beyond answering medical questions to explanation, student-oriented reasoning, and minimal counterfactual rewrites.
- Explanation: GPT-4 provided rich explanations of a medical student’s errors and hypothesized why the student selected an incorrect diagnosis.
- The case study illustrates potential educational and clinical applications, but generated information requires expert review and confirmation.
- Explanation: GPT-4 accurately diagnosed the clinical situation as schistosomiasis given the patient’s history and clinical presentation.
- Understanding the student: GPT-4 personalized an educational explanation by attempting to understand the student’s reasoning.
- Counterfactual rewrites: GPT-4 created counterfactual clinical scenarios by minimally rewriting the case history or laboratory results.
8 From Benchmarks to Real World
The paper distinguishes strong benchmark performance from clinical deployment, outlining potential workflow benefits alongside substantial risks from errors, bias, and insufficient verification.
- From Benchmarks to Real World: There is a large gap between competency on medical benchmarks and successfully deploying GPT-4 in clinical applications.
- Risks of erroneous generations: Potential risks include inaccurate rankings or sequencing, factual errors, omissions, hallucinations, and harmful over-reliance on fluent recommendations.
- Implications for the future: Additional research should improve grounding, self-consistency checks, contextual error characterization, and calibration signals before deployment.
- Risks of Bias: Biases in healthcare-oriented generations remain poorly understood across race, socioeconomic background, gender, and other factors.
- Influences on workflows, tasks, and specialties: With appropriate expert oversight, GPT-4 could support differential diagnoses, test sequencing, therapy planning, analytics, reminders, and decision support.
- Influences on workflows, tasks, and specialties: GPT-4 could reduce physicians’ logistical and administrative burden, allowing more time for patient engagement and coordination.
9 Conclusion
The conclusion reports that GPT-4 outperformed earlier and medically specialized models across most evaluated medical benchmarks, while demonstrating reasoning and calibration capabilities relevant to medicine.
- GPT-4 significantly outperformed GPT-3.5 and Flan-PaLM 540B on USMLE and MultiMedQA evaluations, except on one MultiMedQA dataset.
- The evaluation also examined text-only versus image-referencing questions and calibrated output probabilities for medical applications.
- Sample outputs demonstrated explanation, counterfactual reasoning, differential diagnosis, and testing-strategy capabilities.
A Dataset descriptions
The paper evaluates GPT-4 on official USMLE materials and multiple medical question-answering datasets spanning licensing exams, biomedical abstracts, and broad subject areas.
- USMLE datasets: The USMLE Self Assessments dataset contains 2173 questions: 1197 from Step 1, 800 from Step 2, and 176 from Step 3.
- USMLE datasets: USMLE Sample Exam materials were sourced from official practice PDFs for Steps 1, 2, and 3.
- MultiMedQA datasets: MedQA contains multiple-choice medical licensing questions from the United States, Mainland China, and Taiwan.
- MultiMedQA datasets: PubMedQA tests yes, no, or maybe answers to biomedical research questions using context from PubMed abstracts.
- MultiMedQA datasets: MedMCQA contains four-choice questions from Indian medical school entrance examinations, with 6150 test questions.
- MultiMedQA datasets: MMLU spans 57 subjects, with the benchmarks reporting results for medicine-related subjects.
B Memorization analysis
The MELD method tests potential memorization by prompting a model with half of each item and measuring near-exact reproduction of the held-out half.
- MELD partitions each dataset item in half, uses the first half as context, and generates up to the length of the second half.The method then compares generated text with the held-out content using a Levenshtein distance ratio.
- A near-exact match requires a Levenshtein distance ratio of at least 0.95, meaning fewer than 5% of characters differ.
- The reported memorization metric is the percentage of dataset datapoints whose generated completions exceed the specified overlap threshold.
- The method treats a high near-exact-match rate across a dataset as empirical evidence that the model encountered the data during training.Individual samples may not regenerate exactly because of stochasticity in training.
D Interactive educational session (continued)
The case study extends GPT-4’s medical reasoning through prompts that ask for broader differential diagnoses, targeted testing, and patient-oriented treatment conversations.
- GPT-4: Formulating differential diagnosis: GPT-4 is asked to expand the differential diagnosis beyond the multiple-choice answers and rank the possibilities by likelihood.
- GPT-4: Formulating differential diagnosis: When pressed further, GPT-4 produces a more sophisticated rewrite focused on altering laboratory-result values.
- GPT-4: Information-gathering and testing: GPT-4 is prompted to identify additional patient information and laboratory tests that could narrow the differential diagnosis and sequence those tests.
- GPT-4: Schistosomiasis and Patient Conversations: In the schistosomiasis case, GPT-4 is asked to recommend ideal treatments and provide advice for engaging with the patient.