PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models

Jenish Maharjan, Anurag Garikipati, Navan Preet Singh, Leo Cyrus, Mayank Sharma, Madalina Ciobanu, Gina Barnes, Rahul Thapa, Qingqing Mao, Ritankar Das

arXiv:2402.19371v1cs.CLcs.AIcs.IR

TL;DR

Medical LLMs often rely on costly fine-tuning, and proprietary weights limit further investigation; whether robust prompting alone could optimize generalist open-source models remained unshown. OpenMedLM combines prompting strategies on OS foundation models and reports OS SOTA on three evaluated benchmarks, including the first reported result above 80% accuracy by a generalist OS foundation LLM on medical MMLU.

  • Problem

    Whether robust prompting could optimize generalist OS foundation models on medical benchmarks without specialized fine-tuning had not been shown, despite fine-tuning costs and restricted access to proprietary weights.

    Source: §1 · pp. 5–6

  • Method

    OpenMedLM applies few-shot prompting, chain-of-thought, kNN selection, and ensemble voting to OS foundation models, using Yi 34B for further experimentation.

    Source: §1 · p. 6, §2.1 · p. 8, §2.5 · p. 14

  • Results

    OpenMedLM reports OS SOTA on three evaluated benchmarks, including the first reported result above 80% accuracy by a generalist OS foundation LLM on medical MMLU.

    Source: §4 · pp. 19–20

  • Takeaways & Limitations

    The findings show that medical Q&A-related emergent properties can appear in OS foundation models and that prompting alone can outperform intensive fine-tuning on specialized healthcare tasks.

    Source: §4 · p. 21

    The evaluated benchmarks are academic multiple-choice Q&A and may not represent clinical questions, where open-ended responses are often required.

    Source: §4.1 · p. 22

Abstract

from arXiv · show

LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.

1. Introduction

Medical LLMs can require costly fine-tuning and specialized data, while many leading models are proprietary. The introduction presents OpenMedLM as robust prompt engineering for open-source foundation models, aiming for strong medical-benchmark performance without specialized fine-tuning.

1. Introduction

  • Medical LLM evaluation uses Q&A benchmarks spanning licensing exams, biomedical abstracts, and life-sciences topics to assess accuracy across medical tasks.The cited benchmarks include MedQA, MedMCQA, PubMedQA, and the MMLU medical-subset.Source: §1 · pp. 4–5
  • Medical-model development often relies on fine-tuning that can require costly computation and hard-to-access task-specific data, while proprietary weights restrict further investigation.The passage also identifies transparency and compliance as benefits of open-source models in healthcare.Source: §1 · p. 5
  • Prior work showed that robust prompting could let generalist proprietary models such as GPT-4 reach state-of-the-art results on some medical Q&A benchmarks.The introduction identifies the untested question as whether robust prompting could similarly optimize generalist OS foundation models without specialized fine-tuning.Source: §1 · p. 6
  • OpenMedLM applies robust prompting to the OS Yi 34B foundation model without specialized fine-tuning and reports OS SOTA results on four medical benchmarks.The benchmarks are MedQA, MedMCQA, PubMedQA, and the medical-subset of MMLU.Source: §1 · p. 6
  • With Yi 34B, OpenMedLM reaches 72.6% accuracy on MedQA, surpassing all other OS models.The figure describes MedQA as a USMLE-style benchmark.Source: Figure 1 · p. 7

2. Methodology

The study selected Yi 34B after comparing zero-shot performance across open-source models. It then evaluated prompt strategies on medical benchmarks, from zero-shot and few-shot prompting to chain-of-thought and ensemble voting.

Model selection

  • Yi 34B, selected as the strongest zero-shot model among tested 7B–70B open-source LLMs, was used for further OpenMedLM experiments.The model has 34 billion parameters.Source: §2.1 · p. 8

Evaluation datasets

  • Evaluation covered MedQA’s English test set, MedMCQA’s dev set, reasoning-required PubMedQA, and nine medically relevant MMLU subsets.The PubMedQA evaluation used 500 test questions; the MMLU subsets contained 1,785 questions.Source: §2.2 · pp. 8–9

Prompting strategies

  • Prompt strategies ranged from zero-shot questions without examples to five-example few-shot prompts and few-shot chain-of-thought prompts.Few-shot examples included questions and correct answers; CoT examples also included explanations.Source: §2.3 · p. 10
  • CoT examples used explanations generated by GPT-4 and were selected randomly or by kNN similarity to each test question.The kNN approach selected five nearest questions from the training split.Source: §2.3 · p. 10, §2.4 · p. 12, §2.5 · p. 14
  • The kNN CoT ensemble ran each prompt five times, shuffled answer options, and selected the most common answer.Inference runs used a decoding temperature of 0.4.Source: §2.3 · pp. 10–11, §2.5 · p. 14

Evaluation procedure

  • Strategies were added sequentially as ablations; accuracy counted invalid outputs as incorrect after five tries, while prompt instructions and token limits were tuned empirically.The token limit was selected with GPU memory use and evaluation time in mind.Source: §2.4 · p. 12, §2.5 · pp. 13–15

3. Results

OpenMedLM’s combined prompting strategies give the Yi 34B foundation model OS state-of-the-art performance on three of four medical benchmarks, surpassing Meditron on MedQA and MedMCQA but not PubMedQA. The ablations show that component effects vary by benchmark, including a synergistic pattern on MMLU medical.

MedQA

  • On MedQA, Yi 34B’s zero-shot accuracy was 58.4%, below Meditron’s 65.4%, but combining all OpenMedLM prompting components enabled Yi 34B to outperform Meditron.Source: §3 · pp. 16–18

Across benchmarks

  • OpenMedLM achieved OS state-of-the-art results on three of four benchmarks; Yi 34B reached 68.3% on MedMCQA, 2.3% above Meditron, but 77.3% on PubMedQA, below Meditron’s 81.6%.The largest PubMedQA increase came from random few-shot CoT (11.2%); on MedMCQA, kNN few-shot CoT improved performance by 6.5% over random few-shot CoT.Source: §3 · pp. 15, 19, Table 2 · p. 15

MedQA

  • On MedQA, random few-shot raised Yi 34B accuracy from 58.4% to 61.5%, and random few-shot with CoT raised it to 66.6%.The few-shot examples were randomly selected and held constant across MedQA questions.Source: §3 · pp. 17–18
  • On MedQA, kNN few-shot CoT selected five similar training questions per test question and brought Yi 34B accuracy to 69.5%.This was a 2.9% accuracy improvement; the full OpenMedLM strategy reached 72.6%.Source: §3 · p. 18
  • On MedQA, self-consistency majority voting added 3.1% performance, bringing Yi 34B’s overall accuracy to 72.6%.Across the evaluated benchmarks, the ablation reports at least 9% improvement over the zero-shot baseline.Source: §3 · pp. 18–19

MMLU medical subset

  • 81.6% accuracy on the MMLU medical subset surpassed Meditron’s 77.6%, with prompting gains showing a synergistic rather than additive pattern.Random few-shot added 4.7%; adding CoT to it reduced performance by 2.1%, while subsequent kNN few-shot CoT added 3.5%.Source: §3 · p. 19

4. Discussion

OpenMedLM combines prompting strategies to produce OS state-of-the-art performance on three evaluated medical benchmarks, including the first reported generalist OS foundation LLM result above 80% on the MMLU medical subset. The authors also caution that their multiple-choice academic benchmarks do not establish readiness for real-world clinical use.

4. Discussion

  • OpenMedLM’s combination of prompting strategies achieved OS state-of-the-art on three evaluated benchmarks, including a first reported result above 80% for a generalist OS foundation LLM on MMLU medical.The authors describe the strategies’ effects as additive on MedQA, MedMCQA, and PubMedQA, but partly synergistic on MMLU; their order also matters.Source: §4 · pp. 19–20
  • Prompting alone outperformed intensive fine-tuning on specialized healthcare tasks in this study, and the authors identify medical Q&A performance as an emergent property of OS foundation models.They call for further study of combining fine-tuning with advanced prompting in specialized models derived from OS foundations.Source: §4 · pp. 21–22

4.1 Limitations

  • The evaluated benchmarks cover academic multiple-choice questions, not the open-ended, complex encounters that physicians and patients may pose in clinical settings.The authors say further research and experimentation are needed before these models can be applied in real-world scenarios.Source: §4.1 · p. 22

5. Conclusions

The authors present OpenMedLM as evidence that generalist OS foundation models can perform strongly on medical tasks. They frame accessibility and customizability as potential benefits, and encourage further research on prompt engineering and combinations of strategies.

5. Conclusions

  • OpenMedLM showcases strong medical-task performance by generalist OS foundation LLMs, with potential accessibility and customizability benefits for healthcare applications.The authors state that the prompting strategies and models are publicly available and hope the results encourage further prompt-engineering research.Source: §5 · p. 23
  • The authors propose combining prompting strategies and integrating LLMs with medical AI algorithms as directions toward practical, accessible clinical tools.They present improved health outcomes as an intended end goal, not an evaluated result.Source: §5 · p. 23

Related papers

Loading 2402.19371v1…