PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineering

Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, Hua Xu

arXiv:2303.16416v3cs.CL

TL;DR

This study assesses GPT models for clinical NER and tests whether task-specific prompts improve their performance. The prompts bring GPT-4 close to BioClinicalBERT, though it does not match the baseline on either dataset.

  • Problem

    The study assesses GPT models’ capabilities for clinical NER, where manually extracting information from clinical notes is labor-intensive and time-consuming.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The authors evaluate GPT models on two clinical NER tasks using a prompt framework with annotation guidance, error-analysis instructions, and few-shot examples.

    Source: §1 · p. 2, §2.1 · p. 2, §1.1.2 · p. 16

  • Results

    GPT-4 with the proposed prompts approached BioClinicalBERT but did not match it on the MTSamples and VAERS datasets.

    Source: §5 · p. 11

  • Takeaways & Limitations

    GPT-4’s performance with few training samples suggests potential for clinical NER, while remaining below BioClinicalBERT on both datasets.

    Source: §5 · p. 11

    The study evaluated GPT models only, leaving performance of other LLMs such as LLaMA and Falcon untested.

    Source: §4 · p. 10

Abstract

from arXiv · show

Objective: This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks and proposes task-specific prompts to improve their performance. Materials and Methods: We evaluated these models on two clinical NER tasks: (1) to extract medical problems, treatments, and tests from clinical notes in the MTSamples corpus, following the 2010 i2b2 concept extraction shared task, and (2) identifying nervous system disorder-related adverse events from safety reports in the vaccine adverse event reporting system (VAERS). To improve the GPT models' performance, we developed a clinical task-specific prompt framework that includes (1) baseline prompts with task description and format specification, (2) annotation guideline-based prompts, (3) error analysis-based instructions, and (4) annotated samples for few-shot learning. We assessed each prompt's effectiveness and compared the models to BioClinicalBERT. Results: Using baseline prompts, GPT-3.5 and GPT-4 achieved relaxed F1 scores of 0.634, 0.804 for MTSamples, and 0.301, 0.593 for VAERS. Additional prompt components consistently improved model performance. When all four components were used, GPT-3.5 and GPT-4 achieved relaxed F1 socres of 0.794, 0.861 for MTSamples and 0.676, 0.736 for VAERS, demonstrating the effectiveness of our prompt framework. Although these results trail BioClinicalBERT (F1 of 0.901 for the MTSamples dataset and 0.802 for the VAERS), it is very promising considering few training samples are needed. Conclusion: While direct application of GPT models to clinical NER tasks falls short of optimal performance, our task-specific prompt framework, incorporating medical knowledge and training samples, significantly enhances GPT models' feasibility for potential clinical applications.

1 Introduction

Clinical NER can reduce the manual effort of extracting information from clinical notes, but building supervised systems depends on large annotated corpora. This study investigates GPT models for clinical NER and proposes a task-specific prompt framework.

1 Introduction

  • Clinical notes contain valuable information for care and research, but manually extracting it is labor-intensive and time-consuming.Clinical NER identifies entity boundaries and semantic categories such as medical problems, treatments, and tests.Source: §1 · p. 1
  • Large annotated corpora are a bottleneck for machine-learning clinical NER, while domain-specific BERT models have shown improved performance with fewer annotated samples through fine-tuning.Examples include BioBERT, PubMedBERT, and ClinicalBERT.Source: §1 · pp. 1–2
  • No prior work had specifically examined prompt engineering for GPT models on clinical-text NER, motivating further investigation.Earlier prompt-engineering work covered open-domain and some biomedical tasks.Source: §1 · p. 2
  • The study proposes a clinical NER prompt framework and evaluates GPT models on two tasks, establishing a benchmark and making code and datasets publicly available.The tasks are the 2010 i2b2 concept extraction task and nervous system disorder-related event extraction.Source: §1 · p. 2

2 Methods

The methods assess GPT models and BioClinicalBERT on two clinical NER datasets, using progressively enriched prompts and exact- and relaxed-match evaluation. The datasets, model setup, prompt components, and evaluation criteria define the study workflow.

2.1 Task Overview

  • Figure 1 overviews the workflow; GPT models and BioClinicalBERT were evaluated on 100 MTSamples discharge-summary HPI sections.The GPT prompts target medical problems, treatments, and tests.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Dataset

  • The study used 163 synthetic MTSamples discharge summaries and 91 VAERS safety reports, split into training, validation, and test subsets.Prompt examples and error analysis used training data, while test subsets were reserved for final evaluation.Source: §2.2 · p. 3, Table 1 · p. 4

2.3 Models

  • BioClinicalBERT was fine-tuned as a supervised baseline, while GPT-3.5-turbo-0301 and GPT-4-0314 were run at temperature 0; a CRF was also included for comparison.The input and output datasets and prompt variants were made available in Jupyter notebooks interfacing with the OpenAI API.Source: §2.3 · pp. 3–4

2.4 Prompt engineering

  • Prompts combined task and output-format instructions, annotation guidelines, error-analysis rules, and either 1 or 5 annotated training examples.The authors incrementally added guideline prompts, error-based instructions, and examples to compare prompt components.Source: §2.4 · pp. 4–5

2.5 Evaluation

  • Models were scored for precision, recall, and F1 using exact-match and relaxed-match criteria.Relaxed match accepts an overlapping entity span when its entity type matches the gold standard.Source: §2.5 · p. 5

3 Results

Across the two datasets, BioClinicalBERT remained the strongest model, while GPT-4 performed competitively under relaxed matching. Adding prompt components improved zero-shot scores, and five-shot examples with all components yielded GPT-4’s best results; error analysis identified boundary and entity-type challenges.

3.1 Zero-shot performance with different prompts

  • Annotation-guideline prompts and error-analysis instructions improved zero-shot metrics for both models across datasets and criteria, with larger average gains for GPT-3.5 (0.09) than GPT-4 (0.06).The average gain was approximately 0.11 on VAERS versus 0.04 on MTSamples.Source: §3.1 · p. 5, Figure 2 · p. 7

3.2 Effect of N-Shot Examples on Model Performance

  • GPT-4’s five-shot prompts with all components achieved F1 scores of 0.593 and 0.861 on MTSamples and 0.542 and 0.736 on VAERS under exact- and relaxed-match criteria, respectively.More examples generally improved performance.Source: §3.2 · p. 7, Table 4 · p. 7, Figure 3 · p. 8

3.3 Performance Comparison to Supervised Learning

  • BioClinicalBERT led overall, with relaxed-match F1 scores of 0.901 on MTSamples and 0.802 on VAERS; GPT-4 scored 0.861 and 0.736, respectively.GPT models were less impressive under exact matching than relaxed matching.Source: §3.3 · p. 8, Table 5 · p. 8

3.4 Error analysis

  • In a 20-sentence post-processing sample, both models struggled with MTSamples entity boundaries, while GPT-4 also had difficulty classifying entity types in VAERS.The authors note lower annotator agreement and more specific event categories in VAERS as possible contributors to its complexity.Source: §3.4 · p. 9

4 Discussion

Task-specific prompting improved GPT performance across two clinical NER tasks, and the best GPT-4 result was competitive with BioClinicalBERT under relaxed matching. The discussion highlights reduced reliance on annotated examples while identifying unresolved issues in expert input, sample selection, evaluation, and model scope.

4 Discussion

  • GPT-4’s best prompted performance was competitive with BioClinicalBERT under relaxed matching, while prompt improvements were consistent across two clinical NER tasks.The authors also describe the best GPT-4 performance as close to BioClinicalBERT, while noting that the study did not show GPT models consistently outperforming it.Source: §4 · p. 10
  • With 1- or 5-shot examples, GPT models approached fine-tuned models using hundreds of training samples, potentially reducing some annotation costs without eliminating expert involvement.Experts remain important for writing annotation guidelines, preparing initial annotations for error analysis and examples, and validating performance; API and service costs also remain relevant.Source: §4 · p. 10
  • Specific annotation guidelines and annotated examples improved performance, but the study did not investigate how to select informative, representative samples or other few-shot algorithms.The authors say medical knowledge from a knowledge base or human experts remains critical and that its representation in prompts needs further investigation.Source: §4 · p. 10
  • The traditional NER evaluation schema can mark clinically relevant abnormal-value tests as mismatches when GPT models classify them as medical problems.The authors argue that a better evaluation schema is needed to assess LLM performance more accurately.Source: §4 · p. 10
  • The study tested only GPT models and used relatively simple few-shot approaches; the authors propose evaluating other LLMs and methods such as chain-of-thought.Other models mentioned for future study include LLaMA and Falcon.Source: §4 · p. 10

5 Conclusion

The study evaluated a clinical task-specific prompt framework on two clinical NER tasks. GPT-4 with the prompts approached BioClinicalBERT but did not match its performance on either dataset; the results suggest promise while leaving further refinement necessary.

5 Conclusion

  • GPT-4 with 5-shot prompts did not perform as well as BioClinicalBERT on either MTSamples or VAERS, despite using almost no training data.The conclusion describes the result as promising while underscoring the need for further refinement before LLMs can consistently outperform established models in these applications.Source: §5 · p. 11

Conflict of Interest

The authors disclose research-related financial interests at Melax Technologies Inc.

Conflict of Interest

1 Supplementary Materials

The supplementary materials specify entity definitions and annotation rules for two clinical NER tasks. BioClinicalBERT’s validation F1 rose quickly, plateauing around epoch 5 on MTSamples and epoch 4 on VAERS, with stable scores thereafter.

1.1.1 The 2010 i2b2 concept extraction task

  • The i2b2 task marks medical problems, treatments, and tests, including concepts beyond the corresponding UMLS semantic types.Problems are abnormal or disease-related observations; treatments are interventions or substances, and tests are procedures, panels, or measures used to investigate problems.Source: §1.1.1 · p. 14
  • Annotations include complete noun and adjective phrases with modifiers, while vital signs count as tests and specialists, facilities, and consultations do not.The guidelines also permit a qualifying prepositional phrase after a concept; examples illustrate how phrase boundaries are marked.Source: §1.1.1 · pp. 14–15

1.1.2 The nervous system disorder-related event extraction task

  • The VAERS task distinguishes investigations, nervous and other adverse events, and specific procedures, while excluding negated entities and routine care without a specific procedure.Other adverse events are problems assigned to other MedDRA SOCs; procedures are specific medical or surgical activities.Source: §1.1.2 · pp. 15–16
  • VAERS annotations cover events that occurred before GBS diagnosis, include reported histories and symptom improvement, and distinguish recurrence from duplicated text.A recurrence is annotated twice; when recurrence is unclear, multiple timestamps support two annotations, while one record is kept without them.Source: §1.1.2 · p. 16
  • Error-analysis rules classify hospital admissions, transfers, and discharges as procedures, and treat all abnormal symptoms as adverse events.The rules specifically include care facilities such as hospitals and rehabilitation centers as procedures.Source: §1.1.2 · p. 16

1.2 Learning Curve of BioClinicalBERT on the validation sets

  • BioClinicalBERT’s validation F1 improved sharply at first, plateaued around epoch 5 for MTSamples and epoch 4 for VAERS, and remained stable without observed overfitting.The learning curves are shown across epochs for both validation sets.Source: §1.2 · p. 17
Loading 2303.16416v3…