PaperlayerView exact revision on arXiv

Source-linked AI summary

Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine

Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M. Cheung, Robert Chen, Ronald M. Summers, Justin F. Rousseau, Peiyun Ni, Marc J Landsman, Sally L. Baxter, Subhi J. Al'Aref, Yijia Li, Alex Chen, Josef A. Brejt, Michael F. Chiang, Yifan Peng, Zhiyong Lu

arXiv:2401.08396v4cs.CVcs.AIcs.CL

TL;DR

Multi-choice accuracy may not reflect GPT-4V’s actual competence, and correct final choices are not guaranteed to rest on accurate rationales. The study comprehensively evaluated GPT-4V’s rationales when solving NEJM medical tasks. GPT-4V achieved higher overall accuracy (81.6%) than physicians (77.8%), while correct-answer mistakes predominantly occurred in image comprehension (27.2%).

  • Problem

    Multi-choice accuracy may not reflect GPT-4V’s actual competence, and correct final choices are not guaranteed to rest on accurate rationales.

    Source: Main · p. 5

  • Method

    The study comprehensively evaluated GPT-4V’s rationales when solving NEJM medical tasks.

    Source: Main · p. 10

  • Results

    GPT-4V achieved higher overall accuracy (81.6%) than physicians (77.8%), while correct-answer mistakes predominantly occurred in image comprehension (27.2%).

    Source: Main · pp. 7–8

  • Takeaways & Limitations

    Correct final choices can coexist with flawed rationales, so choice accuracy alone does not fully characterize performance.

    Source: Main · pp. 5, 8

    The evaluation covered a set of 207 questions.

    Source: Main · p. 9

Abstract

from arXiv · show

Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.

Main

The section examines GPT-4V’s performance on NEJM Image Challenge questions and the quality of its rationales, including when its final answers are correct. It considers closed-book accuracy, performance with open-book tools, and the relative challenges of image comprehension and medical-knowledge recall.

Main

  • The study evaluated 207 single-answer NEJM Image Challenge questions and structured GPT-4V rationales across image comprehension, medical-knowledge recall, and step-by-step reasoning.A student established a baseline, and physicians assessed the answers and rationales against NEJM’s official explanations.Source: Main · pp. 5–6
  • The reported accuracy interval was 75.7%–86.7%, compared with physicians’ 77.8% (CI: 71.5%–83.3%); open-book physicians scored 95.2%, and a medical student scored 61.4%.Open-book physicians’ CI was 91.3%–97.7%; the medical student’s CI was 54.4%–68.0%.Source: Main · p. 7
  • GPT-4V correctly answered 36 of 46 questions physicians missed in closed-book testing (78.3%) and 7 of 10 missed in open-book testing (70.0%).Only 3 of 207 questions were answered incorrectly by both GPT-4V and open-book physicians.Source: Main · p. 8
  • More than 25% of cases had flawed image-comprehension rationales; medical-knowledge recall was the most reliable of the three evaluated rationale dimensions.The evaluation also assessed step-by-step reasoning; the supplied passage does not provide the error-rate range for recall.Source: Main · p. 8
  • When GPT-4V’s final answer was correct, rationale errors occurred in image comprehension (27.2%), knowledge recall (8.9%), and reasoning (12.4%).Image comprehension was the most frequent source of error among these dimensions.Source: Main · pp. 8–9
  • The evaluation was limited to 207 questions, each annotated by one physician, and the NEJM case distribution might be biased toward specialties such as dermatology.The questions had single correct answers, unlike clinical cases that may have multiple possible diagnoses.Source: Main · p. 9
  • GPT-4V demonstrates superior closed-book multi-choice accuracy, but physicians remain superior with open-book tools, especially on hard questions.Source: Main · p. 10
  • Even when it answers correctly, GPT-4V may fail to understand or interpret the medical scenario in its individual rationales.Source: Main · p. 10
  • Over 20% image-comprehension error contrasts with medical knowledge recall as GPT-4V’s most reliable rationale component.Source: Main · p. 10
  • Evaluations beyond multi-choice accuracy are needed before these models can be integrated into clinical practice.Source: Main · p. 10

Methods

The study evaluated GPT-4V on recent NEJM Image Challenges using structured assessments of image, medical-knowledge, and reasoning rationales, physician annotations, and a zero-shot BiomedCLIP comparison. It also examined specialty-level performance, physician accuracy, and detection of hidden rationale flaws.

  • The dataset comprised 207 recent challenges from January 11, 2020, to February 29, 2024, tiered as easy, medium, or hard by NEJM-user correct-answer rates.Easy questions had 56%–88% correct answers, medium 44%–56%, and hard 28%–44%.Source: Collecting NEJM Image Challenge · p. 11
  • GPT-4V was prompted to describe images, recall relevant medical knowledge not stated in the question, then provide step-by-step reasoning and an answer.Source: Prompting GPT-4V · pp. 11–12
  • BiomedCLIP achieved 25.1% accuracy in zero-shot testing, only slightly above 20% chance, indicating the challenge for smaller vision-language models.Source: BiomedCLIP · pp. 12–13
  • A senior medical student provided closed-book answers, while specialty residents or attendings answered with and without external resources and then reviewed GPT-4V responses.Source: Annotations on MTurk · p. 13
  • Physicians rated GPT-4V’s image-comprehension, medical-knowledge-recall, and step-by-step-reasoning rationales as correct, partially correct, or incorrect, explaining partial or incorrect ratings.Source: Annotations on MTurk · p. 14
  • In additional annotations of 34 internal-medicine cases excluded from the main analysis, Cohen’s kappa was 0.523 for closed-book answers and 0.470, 0.562, and 0.698 for rationale correctness in image comprehension, knowledge recall, and reasoning, respectively.Source: Judging the Step-by-Step Reasoning… · p. 16
  • In 6 of 9 specialties GPT-4V scored higher than physicians and in 2 of 9 scored comparably; 8 of 9 physicians identified hidden flaws.Source: Specialty-level analysis · p. 17
  • Physician accuracy had a 91.7% Spearman rank correlation with the percentage of hidden flaws identified, suggesting the capabilities might correlate.Source: Specialty-level analysis · p. 17
  • This evaluation was the only one to combine quantitative performance comparisons with physicians and systematic rationale evaluation for all GPT-4V answers.Source: Related work · p. 17

Code Availability Statement

The authors identify the BiomedCLIP model repository and describe how they accessed GPT-4 Vision and collected its responses.

Code Availability Statement

Competing Interests

The authors report no competing non-financial interests and disclose competing financial interests, including royalties and research-related interests.

Competing Interests

  • The authors declare no competing non-financial interests but disclose competing financial interests.Source: Competing Interests · p. 18
  • R.S. reports royalties related to patents or software licenses from iCAD, Philips, ScanMed, PingAn, Translation Holdings, and MGB, and received research support.Source: Competing Interests · p. 18
Loading 2401.08396v4…