PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of ChatGPT-Generated Medical Responses: A Systematic Review and Meta-Analysis

Qiuhong Wei, Zhengxiong Yao, Ying Cui, Bo Wei, Zhezhen Jin, Ximing Xu

arXiv:2310.08410v1stat.MEstat.ML

TL;DR

Medical ChatGPT evaluations lack standardized methods, limiting consistency in the available evidence. This study systematically reviewed and meta-analyzed the literature, finding an overall integrated accuracy of 56% while documenting substantial methodological heterogeneity and incomplete reporting.

Abstract

from arXiv · show

Large language models such as ChatGPT are increasingly explored in medical domains. However, the absence of standard guidelines for performance evaluation has led to methodological inconsistencies. This study aims to summarize the available evidence on evaluating ChatGPT's performance in medicine and provide direction for future research. We searched ten medical literature databases on June 15, 2023, using the keyword "ChatGPT". A total of 3520 articles were identified, of which 60 were reviewed and summarized in this paper and 17 were included in the meta-analysis. The analysis showed that ChatGPT displayed an overall integrated accuracy of 56% (95% CI: 51%-60%, I2 = 87%) in addressing medical queries. However, the studies varied in question resource, question-asking process, and evaluation metrics. Moreover, many studies failed to report methodological details, including the version of ChatGPT and whether each question was used independently or repeatedly. Our findings revealed that although ChatGPT demonstrated considerable potential for application in healthcare, the heterogeneity of the studies and insufficient reporting may affect the reliability of these results. Further well-designed studies with comprehensive and transparent reporting are needed to evaluate ChatGPT's performance in medicine.

Introduction

ChatGPT has been studied across diverse medical applications, but evaluation remains inconsistent because standardized guidelines are lacking. This review addresses that gap by systematically examining performance, study methods, and evaluation metrics.

  • ChatGPT has been evaluated across diverse medical fields, with reported accuracies ranging from 36% to 90%.Source: Introduction · p. 3
  • The field lacks standardized guidelines for evaluating large language models, producing inconsistencies across studies.Source: Introduction · p. 3
  • Existing evaluations differ in question sources, question-asking processes, and performance metrics.Source: Introduction · pp. 3–4
  • The study conducts a systematic review, meta-analysis, and methodological examination of research evaluating ChatGPT on medical questions.Source: Introduction · p. 4
  • The review aims to characterize the current evidence and provide useful information for future studies.Source: Introduction · p. 4

Methods

The authors conducted a systematic literature search and independently screened, extracted, and appraised studies evaluating ChatGPT in medicine. Objective accuracy studies were synthesized using meta-analytic methods, with heterogeneity and robustness assessed statistically.

  • The literature search covered ten databases on June 15, 2023, using “ChatGPT” without publication, language, or date restrictions.Source: Data Source and Search Strategy · p. 5
  • Two authors independently screened records and extracted data, with disagreements and extraction conflicts resolved through discussion or a third reviewer.Source: Data Source and Search Strategy · pp. 5–6
  • Extracted characteristics included question sources, language, number and type of questions, ChatGPT version, inquiry mode, repetition, prompting, raters, metrics, and performance.Source: Data Source and Search Strategy · p. 6
  • Methodological quality was independently assessed with QUADAS-2 across question selection, index test, reference standard, and flow and timing.Source: Data Source and Search Strategy · p. 6
  • Objective medical-knowledge accuracy studies were meta-analyzed using confidence intervals, heterogeneity testing, subgroup analysis, sensitivity analysis, and publication-bias assessment.Source: Evidence Synthesis and Statistic… · p. 7

Results

The review identified 3520 records, narrowed them to 60 studies, and included 17 in the meta-analysis. The included literature spanned publication types and medical disciplines, with no detected publication bias.

Question/Prompt Sources

The included studies used heterogeneous question sources, formats, languages, and quantities. Author-designed and examination-bank questions were common, while most studies used open-ended or English-language questions.

Conversation Process

The reviewed studies used varied ChatGPT versions, access modes, question-sequencing practices, repetition schedules, and prompts. Reporting was often incomplete, limiting comparability across conversation processes.

  • 13 studies used GPT-3.5, 3 used GPT-3, 4 used GPT-4, and some combined multiple versions.Source: Conversation Process · p. 9
  • 43 studies accessed ChatGPT through a web interface, none reported API use, and 17 did not report the access mode.Source: Conversation Process · p. 9
  • 18 studies asked questions independently in new chat windows, while only one entered all questions in a single session.Source: Conversation Process · p. 9
  • 14 studies asked each question once, whereas 17 repeated questions two to six times and 29 did not report repetition.Repetition occurred twice in 6 studies, three times in 7, five times in 3, and six times in 1.Source: Conversation Process · p. 9
  • 40 studies simply prompted ChatGPT to answer, one used a laboratory-medicine-expert prompt, and 19 did not report prompting details.Source: Conversation Process · p. 9

Evaluation of ChatGPT's Performance

Studies evaluated ChatGPT with heterogeneous questions, raters, metrics, and methodological procedures, although formal risk-of-bias assessments were generally favorable. The meta-analysis estimated 56% integrated accuracy, with substantial heterogeneity and higher accuracy in internal medicine than surgery.

Discussion

This review and meta-analysis assessed ChatGPT’s medical question-answering performance and found considerable potential, while substantial heterogeneity and incomplete reporting constrain interpretation. Accuracy differed across specialties, with higher performance in internal medicine than surgery.

  • The study provides a comprehensive assessment of ChatGPT’s performance in answering medical questions.Source: Discussion · pp. 13, 15
  • 56% overall integrated accuracy was reported for ChatGPT in addressing medical queries.Source: Discussion · p. 13
  • 63% accuracy in internal medicine exceeded 49% in surgery.Source: Discussion · p. 13
  • Internal medicine draws on synthesis of symptoms, laboratory results, and medical literature, whereas surgery emphasizes manual skills, tactile sensation, and real-world experience.Source: Discussion · p. 13
  • Surgical scenarios may exceed text-based models’ capabilities because they require procedural nuance and tactile feedback, potentially producing less accurate responses.Source: Discussion · p. 13
  • The authors propose standardized reporting and a 20-item checklist covering task generation, LLM version, conversation structure, and evaluation procedure.Source: Discussion · p. 14

Data Sharing

The extracted data used in the meta-analysis are available in the Supplementary Material, with additional data available on request.

National Knowledge Infrastructure; CBM: Chinese BioMedical Literature Database;

The paper identifies Chinese literature resources and presents figures describing ChatGPT’s medical assessments and performance.

Related papers

Loading 2310.08410v1…