PaperlayerView exact revision on arXiv

Source-linked AI summary

Can large language models reason about medical questions?

Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, Ole Winther

arXiv:2207.08143v4cs.CLcs.AIcs.LG

TL;DR

It remains unclear whether large language models can reason reliably through difficult medical questions requiring expert knowledge. The paper evaluates prompting and retrieval strategies across three medical benchmarks, finding that LLMs can outperform finetuned BERT baselines and approach human-level performance with multiple sampled chain-of-thoughts.

  • Problem

    It remains unclear whether LLMs can reliably apply reasoning and expert knowledge to difficult real-world medical questions.

  • Method

    The study evaluates zero-shot, few-shot, chain-of-thought, and retrieval-augmented prompting across three medical question-answering benchmarks, including expert CoT annotation.

  • Results

    Zero-shot InstructGPT and Codex outperformed finetuned BERT on three medical QA datasets, while 100-sample Codex 5-shot CoT virtually passed the USMLE by 0.2 percentage points.

  • Takeaways & Limitations

    LLMs can mobilize medical knowledge and problem-solving without finetuning, while sampling many CoTs can substantially narrow their gap with human performance.

  • Takeaways & Limitations

    Clinical deployment requires more robust techniques because answer-order and training-data biases can affect predictions.

Abstract

from arXiv · show

Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether close- and open-source models (GPT-3.5, LLama-2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-USMLE, MedMCQA, and PubMedQA) and multiple prompting scenarios: Chain-of-Thought (CoT, think step-by-step), few-shot and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions, but also reaches the passing score on three datasets: MedQA-USMLE 60.2%, MedMCQA 62.7% and PubMedQA 78.2%. Open-source models are closing the gap: Llama-2 70B also passed the MedQA-USMLE with 62.5% accuracy.

1 INTRODUCTION

The paper examines whether large language models can reason through medical questions while addressing the safeguards required for healthcare deployment. It evaluates prompting strategies, interprets generated reasoning, studies inference-time scaling, and benchmarks open-source models across medical datasets.

  • Motivation: Healthcare deployment requires caution because language models can amplify bias, hallucinate facts, and lack robustness.The paper emphasizes that sensitive medical applications need additional safeguards.
  • Motivation: Medical benchmarks test LLMs on realistic clinical scenarios requiring specialized knowledge, advanced reasoning, and human-level reading comprehension.The introduction identifies USMLE as one such assessment setting.
  • Contributions: The study evaluates GPT-3.5 on MedQA-USMLE, MedMCQA, and PubMedQA using zero-shot and few-shot prompting, direct and Chain-of-Thought methods, domain-specific cues, and retrieval augmentation.These experiments target multiple-choice medical board exams and medical reading comprehension.
  • Contributions: A medical expert evaluation protocol categorizes generated Chain-of-Thoughts by reasoning, knowledge, and reading comprehension, finding that InstructGPT often reasons and uses memorized expert knowledge.The evaluation focuses on a subset of zero-shot InstructGPT generations.
  • Contributions: Inference-time compute scaling enables Codex five-shot Chain-of-Thought prompting to achieve calibrated predictions and pass all three medical datasets.The paper also benchmarks open-source models on MedQA-USMLE and MedMCQA.

2 METHOD

The method evaluates prompt-engineering strategies for medical question answering, spanning direct and chain-of-thought prompting, few-shot examples, answer-likelihood aggregation, and retrieval augmentation. It also tests whether multiple sampled completions and external Wikipedia context can improve prediction.

  • Zero-shot: The zero-shot setup compares direct one-step answering with two-step chain-of-thought prompting followed by extractive answer generation.The chain-of-thought prompt uses a cue such as “Let’s think step by step,” followed by “Therefore the answer is.”
  • Few-shot: Few-shot prompts insert question-answer pairs or question-explanation-answer triplets, constructed from the zero-shot template with reference explanations and answers.The few-shot chain-of-thought setup follows Wei et al. (2022).
  • Answer likelihood: The ensemble method samples multiple completions and estimates marginal answer likelihood by aggregating whether candidate answers appear in those completions.Multiple completions can explore alternative hypotheses and support self-consistency across sampled chain-of-thoughts.
  • Retrieval augmentation: Retrieval augmentation tests whether grounding predictions with additional context improves answering accuracy using a BM25 retriever and Wikipedia knowledge base.The approach addresses models’ potential failure to effectively reuse knowledge memorized during training.

3 EXPERIMENTS

The experiments evaluate medical question answering across three datasets using closed- and open-source models, zero-shot prompting, retrieval augmentation, and inference-time sampling. InstructGPT outperformed BERT baselines in zero-shot settings, while Codex’s few-shot ensemble reached passing scores across all three tasks.

  • Datasets and models: The study covers USMLE, MedMCQA, and PubMedQA with GPT-3.5-family models and open-source models ranging from 7B to 70B parameters.USMLE targets difficult real-world medical questions, MedMCQA draws from medical school entrance exams, and PubMedQA tests comprehension of PubMed abstracts.
  • Zero-shot reasoning: +1.4%, +1.0%, and +1.1% were InstructGPT’s zero-shot gains over the best BERT methods on USMLE, MedMCQA, and PubMedQA, respectively.These results were obtained without BM25 grounding.
  • Prompting and retrieval: 53.1% USMLE accuracy and 48.8% MedMCQA validation accuracy were achieved by ensembles combining retrieval-augmented prompts.Direct prompting remained better on average than domain-specific CoT prompts without grounding, while retrieval improved direct-prompt performance by +1.3% on USMLE and +2.7% on MedMCQA.
  • CoT analysis: 86% of incorrectly answered questions contained reasoning errors, 74% showed lack of knowledge, and 50% involved misunderstanding of the question or context.These frequencies came from analysis of CoTs generated by InstructGPT on USMLE questions.
  • Inference-time scaling: 60.2% USMLE, 62.7% MedMCQA, and 78.2% PubMedQA accuracy enabled Codex 5-shot CoT ensembles to reach the passing score on all three tasks.Increasing the sampling budget improved results, with the reported thresholds being 60%, 50%, and 78%, respectively.
  • Calibration: Codex 5-shot CoT appeared overall calibrated across the three datasets, although calibration was worse for PubMedQA.The analysis used k = 100 samples and compared probabilities for correctly and incorrectly answered questions.

4 DISCUSSION

The discussion concludes that zero-shot and few-shot LLMs can mobilize medical knowledge and produce interpretable reasoning, while retrieval, ensemble sampling, and model scale further improve performance. However, biases, imperfect calibration, and a remaining gap from human experts limit their reliability.

  • Zero-shot performance: Zero-shot InstructGPT and Codex outperformed finetuned BERT models on three challenging medical question-answering datasets.Retrieval-augmented BERT was also outperformed by several LLMs on USMLE and MedMCQA, even with Wikipedia passages.
  • Reasoning and interpretability: Single-sample zero-shot CoT was not competitive with direct prompting, but generated reasoning was human-readable and often sound.Expert evaluation found that InstructGPT could often reason, recall medical knowledge, and comprehend the problem, despite making mistakes.
  • Knowledge and retrieval: GPT-3.5 appears to memorize domain knowledge, while simple BM25 grounding still produced slight improvements in InstructGPT.These findings suggest that InstructGPT is not omniscient and can benefit from retrieved documents.
  • Bias: Direct prompts most often selected answer D, whereas CoT prompts favored A and D, indicating systematic answer-selection biases when models cannot answer.The authors speculate that CoTs may begin with option-A-related content by default in such cases.
  • Ensembling and performance: Up to 100 sampled completions combined with 5-shot CoT prompting enabled Codex and Llama-2 to reach MedMCQA and USMLE performance levels, though a large human-expert gap remained.CoTs can be combined or filtered using human or automated feedback.
  • Calibration and model factors: Codex and Llama-2 were close to well-calibrated, while Codex generally outperformed aligned InstructGPT and Llama-2 70B outperformed proprietary Codex 175B.The discussion also compares scale, code pre-training, human alignment, and few-shot learning as performance factors.

5 CONCLUSION

The study finds that prompting and inference-time scaling enable strong medical question answering, while open-source models approach proprietary performance. Clinical deployment still requires more robust methods to address biases and other limitations.

  • 5 CONCLUSION: Vicuna 1.5 13B outperformed both Llama-2 versions, underscoring the importance of high-quality datasets for instruction-based fine-tuning.Guanaco and Vicuna also performed exceptionally well among the evaluated instruction-finetuned models.
  • 5 CONCLUSION: CoT prompting improved performance and interpretability, while expert evaluation found that LLMs often comprehend complex medical questions, recall domain knowledge, and perform non-trivial reasoning.The study evaluated zero-shot, few-shot direct, and CoT prompting with and without retrieval augmentation.
  • 5 CONCLUSION: 100 samples enabled Codex 5-shot CoT to achieve unprecedented performance across three datasets, virtually passing the USMLE by 0.2% points.Sampling many chains of thought at inference time helped overcome some limitations of InstructGPT and Codex.
  • 5 CONCLUSION: 2 points: Llama-2 outperformed Codex on the USMLE despite having a much smaller parameter footprint.The conclusion also states that Llama-2 outperforms Codex with just half the parameters, illustrating the narrowing gap between open- and closed-source models.
  • 5 CONCLUSION: Answer-option ordering affected predictions, while limited BM25 experiments suggested that adding factual data to prompts improves performance.The authors note that other biases, including those hidden in training data, may also affect predictions and that robust techniques are needed for clinical deployment.

A SUMMARY OF THE RESULTS

Table S2 summarizes InstructGPT and Codex performance across four medical-question datasets under zero-shot and few-shot prompting, with and without grounding. The experiments use fixed sampling settings, while MedMCQA validation and test results have distinct evaluation procedures.

  • Evaluation setup: Table S2 reports InstructGPT and Codex performance on MMLU-USMLE, MedQA-USMLE, MedMCQA, and PubMedQA across zero-shot and few-shot settings, with and without grounding.The reported results cover both prompting variation and grounding conditions.
  • Evaluation setup: MedMCQA validation results are estimated from 1k samples, whereas test-set results require an official submission.These are distinct procedures for the validation and test evaluations.
  • Evaluation setup: Sampling temperature is τ = 0 for all experiments except k > 0 sampling with majority voting.The exception applies when drawing multiple samples and aggregating them through majority voting.

B DOMAIN-SPECIFIC COT CUES

The study evaluated 30 zero-shot chain-of-thought cues on 100 USMLE validation questions, accounting for approximately 5% accuracy uncertainty. It selected five prompts for the remaining experiments and examined their agreement patterns.

  • Prompt evaluation: 30 zero-shot CoT prompt variations were evaluated on a subset of 100 USMLE validation questions.The authors considered the first half of the results reasonable candidates because estimated accuracy uncertainty was approximately 5%.
  • Prompt selection: 5 prompts were selected: the original cue, a medical-expert variation, and the top three cues from Table S1.The medical variation was “Let’s think step by step like a medical expert.”
  • Prompt diversity and agreement: Most prompts had agreement rates above 50%, while a minority agreed less with the majority.Examples included cues about reflecting on each answer option, using a Bayesian step-by-step approach, and working by elimination.
  • Uncertainty estimation: Approximately 5% was the estimated uncertainty of the accuracy on 100 validation questions.This estimate used a Bernoulli model for correct versus incorrect predictions and the standard deviation of the mean estimator.

C DATASETS

The paper evaluates medical reasoning across three benchmarks spanning clinician-level problem solving, broad medical knowledge, and abstract-based reading comprehension. MedQA-USMLE and MedMCQA are self-contained multiple-choice datasets, whereas PubMedQA uses expert-annotated questions contextualized by PubMed abstracts.

  • MedQA-USMLE: MedQA-USMLE contains challenging USMLE-style clinical cases requiring problem-solving skills and comprehensive medical knowledge.The questions target trained medical professionals and emulate real clinical settings.
  • MedMCQA: MedMCQA comprises Indian medical entrance-exam questions covering broad medical topics and reasoning types, with greater emphasis on knowledge than USMLE.Its topics include dentistry, psychiatry, and surgery, while reasoning types include logic, factual, and comparison questions.
  • PubMedQA: PubMedQA contains expert-annotated yes/no/maybe research questions derived from PubMed abstracts and primarily evaluates reading comprehension.Unlike the self-contained USMLE and MedMCQA questions, each PubMedQA question is contextualized by a provided abstract.

D MEDQA-USMLE VERSUS MMLU-USMLE

The section compares MedQA-USMLE with MMLU-USMLE using five-shot results, finding MedQA-USMLE more challenging based on Codex performance. Codex scored 56.4% on MedQA-USMLE versus 69.6% on MMLU-USMLE, while a passing USMLE score is around 60%.

  • Dataset comparison: Table S3 compares test MedQA-USMLE with validation/test MMLU-USMLE results, including Flan-U-PaLM 540B, with all models using 5 shots.The comparison covers the USMLE datasets from MedQA and MMLU.
  • Dataset difficulty: 56.4% accuracy was achieved by Codex on MedQA-USMLE, compared with 69.6% on MMLU-USMLE in a 5-shot Direct and CoT setting.The reported difference is approximately 13.2 percentage points, indicating that MedQA-USMLE is more challenging for Codex.
  • USMLE threshold: Around 60% is the score required to succeed on the USMLE.This threshold provides context for the Codex results reported for the two datasets.

E COMPARING GPT VERSIONS ON THE USMLE DATASET

The section compares multiple GPT versions on USMLE test accuracy under direct and Chain-of-Thought prompts. It distinguishes Codex from human-feedback-finetuned InstructGPT and notes that the smallest model performed only slightly above random.

  • Prompt and evaluation setup: USMLE test accuracy was reported for multiple GPT versions using direct and CoT #1 prompts.The comparison appears in Table S4 under a zero-shot setting.
  • Model variants: Codex (code-davinci-002) was pretrained on text and code, whereas InstructGPT (text-davinci-002) was fine-tuned with human feedback to follow instructions helpfully and safely.
  • Results: The smallest model performed only slightly better than random.

F ANSWERING BIAS · G INFORMATION RETRIEVAL · H OPEN-SOURCE LLMS

The paper examines model answering bias, retrieval augmentation, and open-source medical language models. It finds label-defaulting and answer-distribution biases, specifies a composite BM25 retrieval method, and evaluates several open-source models on two benchmarks.

  • F ANSWERING BIAS: 52.9% was the best performance, achieved by the code-pre-trained code-davinici-002 model across prompts.The largest non-aligned text model, text-davinci-001, reached 40.2%, while text-davinci-002 reached 47.1%.
  • F ANSWERING BIAS: CoT prompting produced a more faithful InstructGPT label distribution, but biases toward labels A and D remained.The model also tended to avoid predicting labels B and C.
  • F ANSWERING BIAS: Label permutation preserved the same bias trend, while Codex showed similar behavior and more faithful distributions with few-shot learning.This comparison tested whether the bias originated from the data or the model.
  • F ANSWERING BIAS: Models tended to default to label D when Chain-of-Thought reasoning failed to narrow the answer options.The paper presents two mispredicted label-D examples in which GPT-3 failed to select one answer.
  • F ANSWERING BIAS: Bias was less important for Codex on MedMCQA and PubMedQA than on the USMLE dataset.The comparison used Codex 5-shot CoT predictions averaged over k = 100 samples.
  • H OPEN-SOURCE LLMS: Open-source models assessed included Vicuna, Guanaco, GPT-NeoX, MPT-instruct, Falcon, and Llama-2 on MedQA-USMLE and MedMCQA.They were evaluated in zero-shot and 5-shot settings using greedy decoding (τ = 0).

I ADDITIONAL COT SAMPLES

Additional CoT samples show that InstructGPT can use diverse problem-solving strategies, while Codex produces lower-quality, repetitive chains of thought. Expert annotations categorize reasoning, knowledge recall, and reading comprehension, which can be entangled within individual text segments.

  • Expert annotations: Highlighted CoT segments do not always map to one annotation category because reasoning and knowledge patterns are often entangled.This limitation is noted for the expert-labelled InstructGPT chains of thought.
  • InstructGPT samples: InstructGPT generated diverse problem-solving strategies on USMLE questions.The appendix presents selected remarkable chains of thought from text-davinci-002.
  • Codex samples: Codex produced lower-quality CoTs characterized by frequent repetition and less verbosity.The displayed Codex samples were randomly selected from USMLE questions.
  • Expert annotations: Expert annotations classify CoTs by correct or incorrect reasoning, knowledge recall, and reading comprehension.The annotation scheme includes six patterns spanning correct and incorrect reasoning, knowledge, and comprehension.
  • Annotated examples: One InstructGPT sample correctly identified an ACL injury, while another incorrectly concluded that no listed condition was associated with oral contraceptives.The ACL example linked the physical examination to the diagnosis, whereas the acne example ended with “none of the above.”
Loading 2207.08143v4…