PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices

Jamil Zaghir, Marco Naguib, Mina Bjelogrlic, Aurélie Névéol, Xavier Tannier, Christian Lovis

arXiv:2405.01249v1cs.CLcs.LG

TL;DR

Few-shot prompting may involve annotation costs beyond the demonstrations in its prompts. This review searched medical prompt-engineering studies and extracted their methods and findings, reporting language-documentation gaps and proposing reporting guidelines.

  • Problem

    The annotation cost of few-shot prompting may extend beyond the few demonstrations provided in the prompt.

    Source: Prompt optimization · p. 16

  • Method

    The review searched multiple scholarly and preprint sources and extracted prompt paradigms, models, datasets, languages, techniques, baselines, and findings.

    Source: Prompt optimization · p. 19

  • Results

    English dominates the reviewed medical literature, yet many English-language studies do not state the study language, especially in computer science and clinical venues.

    Source: Prompt optimization · p. 17

  • Takeaways & Limitations

    The review developed reporting guidelines to clarify prompt-engineering choices and improve methodological transparency and reproducibility.

    Source: Prompt optimization · p. 18

    The search was limited to titles, abstracts, and keywords, so studies using prompt-engineering techniques without the queried expressions may have been missed.

    Source: Prompt optimization · p. 20

Abstract

from arXiv · show

Prompt engineering is crucial for harnessing the potential of large language models (LLMs), especially in the medical domain where specialized terminology and phrasing is used. However, the efficacy of prompt engineering in the medical domain remains to be explored. In this work, 114 recent studies (2022-2024) applying prompt engineering in medicine, covering prompt learning (PL), prompt tuning (PT), and prompt design (PD) are reviewed. PD is the most prevalent (78 articles). In 12 papers, PD, PL, and PT terms were used interchangeably. ChatGPT is the most commonly used LLM, with seven papers using it for processing sensitive clinical data. Chain-of-Thought emerges as the most common prompt engineering technique. While PL and PT articles typically provide a baseline for evaluating prompt-based approaches, 64% of PD studies lack non-prompt-related baselines. We provide tables and figures summarizing existing work, and reporting recommendations to guide future research contributions.

Results

The review included 114 articles after screening 398 search results, with 86.8% inter-annotator agreement during the first screening step.

Screening results

  • The search yielded 398 articles; after duplicate removal, 251 underwent title, abstract, and keyword screening, with 94 excluded, before 157 full-text studies were screened and 114 articles included.Annotators resolved 33 conflicts, with 86.8% inter-annotator agreement.Source: Results · p. 3
  • 86.8% inter-annotator agreement was reached while 33 conflicts were resolved during the first screening step.Source: Results · p. 3
  • The included studies came from clinical venues (13), medical informatics (33), computer science (31), and other sources (four); 33 were preprints.Source: Results · p. 3

Prompt paradigms and medical subfields

This section summarizes the review’s examination of prompt paradigms, their task and model patterns, and related reporting in medical applications. It also describes the title-, abstract-, and keyword-based search scope and its potential to miss studies without the queried expressions.

Prompt paradigms and medical subfields

  • PD accounted for 78 articles and covered a wider range of clinical specialties, especially psychiatry; PL had 29 articles and PT 19, with both balanced across biomedical and clinical domains.PD articles also leaned more toward clinical fields than biomedical disciplines.Source: Prompt paradigms and medical subfields · p. 3, Table 1 · p. 5
  • Terminology was inconsistent in 12 studies, while 102 adhered to commonly used definitions; PL was used for PD four times and PT once, and PT for PL five times and PD twice.The review identifies the affected papers as [13],,, and, respectively.Source: Prompt paradigms and medical subfields · p. 6
  • English accounted for 84.2% of selected articles and Chinese 15.8%; language had to be inferred from prompt figures or examples in 48 papers, all studying English.The language remained unknown in two articles [60], and Korean was not explicitly stated in one article.Source: Prompt paradigms and medical subfields · p. 6, Figure 3 · p. 7
  • ChatGPT was popular in PD; among open-source instruct models, LLaMA-2 appeared seven times, while PL was dominated by BERT-based encoders and PT used all model types, favoring encoders.Google’s closed PaLM and Bard models were also prominent; GPT-2 decoder models appeared occasionally in PL, and encoder-decoder models appeared in PT.Source: Prompt paradigms and medical subfields · p. 7, Figure 4 · p. 8
  • PL focused mainly on classification tasks, while PT covered a slightly broader task range; PD prominently addressed board-exam multiple-choice questions, cast as generation, and none of its selected articles proposed entity linking.One PL study used masked-token prediction to generate synthetic electronic health records.Source: Prompt paradigms and medical subfields · pp. 8, 11, Figure 5 · p. 9, Figure 6 · p. 10
  • Among PD articles, 49 used zero-shot, 23 few-shot, and 10 one-shot prompting; few-shot tended to outperform zero-shot on multiple-choice questions but had inconsistent advantages on other NLP tasks.Chain-of-Thought was the most common technique, followed by Persona; two studies used self-consistency, voting across multiple CoT outputs to select the most frequent answer.Source: Prompt paradigms and medical subfields · p. 11
  • Flipped Interaction supported doctor-patient simulations [60] and medical-student training, while Emotion Enhancement was used in mental-health contexts [60].Other techniques included k-nearest-neighbors few-shot prompting and pseudo-classification prompting; Tree-of-Thoughts appeared once.Source: Prompt paradigms and medical subfields · p. 11
  • Five highly cited papers highlighted prompt-engineering applications, including Med-PaLM’s improved question answering over FlanPaLM with instruction prompt tuning, and BioGPT’s state-of-the-art results on six biomedical NLP tasks using suffix-based PT.The review also highlights GPT-3 clinical-task prompt design, ChatGPT performance on the USMLE, and GPT-4 results on the USMLE and MultiMedQA surpassing prior state-of-the-art results [27].Source: Prompt paradigms and medical subfields · p. 13

Prompt optimization

  • The search was restricted to titles, abstracts, and keywords to address the large initial retrieval; studies using prompt engineering without queried expressions may have been missed.Source: Prompt optimization · p. 20

Supplementary Information

The supplementary analysis reports a chi-squared test of publication venue versus how English-language study was disclosed, rejecting independence at p=0.0237. Additional figures report p-values below 0.0001 and 0.0237.

Statistical analysis

Supplementary figures

Loading 2405.01249v1…