PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Few-Shot Clinical Information Extractors

Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, David Sontag

arXiv:2205.12689v2cs.CLcs.AI

TL;DR

Clinical information extraction is hindered by irregular clinical text, limited public corpora, and scarce annotations. The paper benchmarks LLMs on diverse structured extraction tasks, introduces CASI-based datasets, and finds strong GPT-3 performance despite no clinical-specific training. It also identifies schema-matching and answer-existence biases as limitations.

  • Problem

    Clinical information extraction is difficult because clinical text is irregular and public clinical corpora and annotations are scarce.

  • Method

    The paper benchmarks GPT-3 LLMs on diverse clinical extraction tasks, using guided prompts, resolvers, and newly annotated CASI-based datasets.

  • Results

    GPT-3 systems significantly outperform existing zero- and few-shot baselines across the clinical extraction tasks studied.

  • Takeaways & Limitations

    Guided LLM generations can support diverse clinical extraction tasks with light post-processing and may reduce engineering effort.

  • Takeaways & Limitations

    The systems do not always match exact clinical annotation schemas and can produce non-trivial answers when none exists.

Abstract

from arXiv · show

A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

1 Introduction

Clinical information extraction is valuable but difficult because clinical text is irregular, annotations require expertise, and public corpora are scarce. This work benchmarks LLMs on diverse clinical tasks and introduces new annotated datasets for few-shot evaluation.

  • Motivation: Clinical notes contain valuable information absent from structured electronic health-record fields, but irregular jargon and phrasing make extraction difficult.Domain expertise is required for annotation, while off-the-shelf NLP tools often perform poorly.
  • Motivation: Clinical NLP datasets are small and splintered because clinical text is sensitive and often restrictively licensed.These constraints limit the availability of shared corpora for evaluation.
  • Contributions: The paper benchmarks GPT-3-based LLMs on clinical NLP tasks despite their lack of clinical-domain-specific training.The study targets information extraction rather than only conventional classification or generation.
  • Contributions: The authors introduce three annotated datasets by re-annotating CASI for few-shot clinical information extraction benchmarking.The datasets address data-use restrictions affecting clinical corpora and LLM APIs.
  • Contributions: Guided prompts and resolvers map LLM generations to structured outputs for tasks including relation extraction.The approach replaces complex hand-curated domain knowledge with natural-language model output and light post-processing.

2 Related Work

Prompt-based learning adapts pretrained language models through natural-language prompts, but structured clinical extraction requires additional prompting strategies and output conversion. Prior work has emphasized general-domain tasks and qualitative or classification-focused evaluation.

  • Prompt-Based Learning: Prompt-based learning adapts a pretrained language model by priming it with natural-language prompts paired with an input.The model produces an output for the prompted task.
  • Prompt-Based Learning: Few-shot and zero-shot prompting have succeeded across many general-domain tasks, motivating instruction-focused language-model training.T0 and InstructGPT explicitly encourage performance on prompts through their training objectives.
  • Structured Extraction: Complex tasks such as coreference, span classification, and named-entity recognition require creative reframing or additional components beyond straightforward classification prompts.Some approaches require candidate generation, model access, labeled data, or potentially many queries.
  • Structured Extraction: Prior extraction methods can identify multiple spans, but they rely on underlying-model access or labeled data for an extraction layer.InstructGPT extraction examples had been evaluated qualitatively, while quantified results focused on classification and generation.
  • Clinical NLP: Clinical text differs from general NLP in syntax and vocabulary, prompting development of clinical and biomedical language models.Reported gains over classical methods such as logistic regression have sometimes been marginal to none.

3 Methods

The method uses query-only LLM prompting, handcrafted templates, guided structured outputs, and resolvers that map generations into task-specific label spaces. It also introduces CASI-based annotations and compares GPT-3 plus resolvers with established baselines across five tasks.

  • Setup: The experiments assume query-only access to the LLM, without gradients or log probabilities.This constrains the method to prompting and output processing rather than model-internal supervision.
  • Prompting: Prompt templates are handcrafted using five validation examples per task and applied to inputs with optional side information.The LLM output is an unobserved string that must be mapped to a task-specific structured output space.
  • Resolvers: Resolvers convert free-form LLM strings into structured labels, such as medication lists selected from the generated text.The resolver takes the input, side information, and LLM output as arguments.
  • Guided Prompting: Guided prompt design combines a one-shot structured-format example with instructions to use that format, simplifying resolvers for complex outputs.Resolver complexity is measured in lines of code as a proxy for human effort.
  • Dataset and Evaluation: The authors created three annotated datasets by re-annotating publicly available CASI clinical-note snippets because many corpora cannot be used with external LLM APIs.The paper evaluates five clinical NLP tasks and reports that guided-prompt resolvers can be less than 10 LOC.
  • Dataset and Evaluation: GPT-3 plus resolvers matches or exceeds strong few-shot, zero-shot, and supervised baselines across the studied tasks.This comparison is reported as the overall result of the task experiments.

4 Clinical Sense Disambiguation

The section evaluates GPT-3 for clinical acronym sense disambiguation on CASI and a transferred MIMIC reverse-substitution task. GPT-3 plus a resolver outperforms or matches clinical and non-clinical baselines, while weak supervision enables transfer to a private dataset.

  • Dataset: CASI contains 500 examples for each of 75 acronyms, filtered to 18,164 examples covering 41 acronyms for evaluation.
  • Dataset: The MIMIC reverse-substitution dataset has 8,912 test examples across the same 41 acronyms, but GPT-3 was not queried there because of data-use restrictions.
  • Method: The approach treats GPT-3 plus a resolver as a labeler, distills its outputs into a smaller model, and transfers that model to a private dataset.
  • Results: GPT-3 edit plus a resolver clearly outperforms the LMC model on both accuracy and macro F1 on CASI.
  • Results: Weakly supervised PubMedBERT further boosts CASI performance and performs similarly to LMC after transfer to MIMIC.

5 Biomedical Evidence Extraction

The section studies intervention extraction both as token-level classification and as identification of distinct clinical-trial arms. GPT-3 performs respectably but underperforms supervised baselines on token-level metrics, with schema-specific errors affecting extracted spans.

  • Task and dataset: Intervention extraction is evaluated as both token classification and the underlying task of identifying distinct intervention-control arms.
  • Task and dataset: The token-level evaluation uses 187 test abstracts, while manual arm annotations cover 20 abstracts with perfect annotator agreement.
  • Prompting and resolvers: The resolver removes noisy tokens, maps retained output tokens to the input, labels them, and merges fractured spans for token labeling.
  • Token-level results: Resolved GPT-3 performs respectably against supervised deep baselines but underperforms on token-level F1 metrics.
  • Token-level results: Common errors include adding dosing or administration details and including only an acronym or its expansion rather than both.

6 Coreference Resolution

The section evaluates GPT-3-based resolvers for clinical coreference and medication extraction tasks using newly annotated CASI-derived data. Guided prompting improves structured output behavior, and GPT-3 outperforms the longdoc and ScispaCy baselines in the reported comparisons.

  • Coreference resolution: The coreference dataset contains 5 prompt-design pairs and 100 evaluation pairs, emphasizing difficult pronoun cases with non-overlapping antecedents.
  • Coreference resolution: Guided one-shot prompting supplies an example and begins the response with “{pronoun} refers to”, while decoding uses temperature = 0.
  • Medication extraction: For medication extraction, Resolved GPT-3 consistently outperforms the ScispaCy linkage baseline, while one-shot prompting greatly improves precision.
  • Medication status classification: In medication-status classification, omitting the rarer Neither class from the one-shot example makes it unlikely to be output and depresses F1.
  • Coreference resolution: Resolved GPT-3 outperforms the longdoc baseline on macro unigram recall and precision, although the baseline already performs considerably well.
  • Coreference resolution: The one-shot example mainly constrains GPT-3 to quote spans rather than paraphrase, and the example answer’s correctness does not affect performance.

7 Medication Extraction

The paper evaluates GPT-3-based extraction of medications and medication attributes using newly annotated CASI snippets. Across extraction, classification, and relation-extraction framings, guided one-shot GPT-3 generally outperforms the supplied baselines, though errors remain for some attributes.

  • Medication and status extraction: 105 CASI snippets were newly annotated for medication mentions and three medication-status labels: active, discontinued, or neither.Mentions of the same medication were collapsed, and the examples were enriched for treatment changeover.
  • Medication and status extraction: GPT-3 was prompted to output medication lists and statuses jointly, with unguided zero-shot and guided one-shot evaluations using greedy decoding.The prompt-development setup used InstructGPT text-davinci-002 and five examples.
  • Medication and status extraction: Resolved GPT-3 consistently outperformed the ScispaCy medication-extraction baseline, while one-shot prompting greatly improved precision by reducing extraneous extractions.The baseline commonly misrecognized overloaded abbreviations and missed vendor-specific drug names.
  • Medication and status extraction: Status F1 fell when the one-shot example omitted the rarer Neither class, indicating that demonstrating all classes mattered for modifier prediction.Evaluation compared methods on the 241/340 medications found by all GPT-3 methods and also evaluated T-Few on that subset.
  • Medication attribute extraction: A second 105-snippet dataset annotated medications, dosages, routes, frequencies, reasons, and durations under a schema closely following the 2009 i2b2 medication challenge.The study compared phrase-level and token-level classification with a PubMedBERT-CRF baseline and end-to-end relation extraction with postprocessing.
  • Medication attribute extraction: 1-shot GPT-3+R outperformed the i2b2-supervised baseline across all three task framings, but struggled with duration and reason entities.The baseline’s end-to-end relation extraction suffered from cascading extraction errors, while disagreement for duration and reason annotations was reported as substantial.

8 Conclusion

The conclusion presents the work as evidence that large language models can support diverse clinical extraction tasks and structured outputs with light post-processing. It also identifies deployability and reliability under clinical uncertainty as important next steps.

  • Contributions: The paper introduces new annotated datasets and argues that large language models show promise across diverse clinical extraction tasks.The conclusion frames the contribution around clinical extraction rather than only classification or generation.
  • Contributions: Light post-processing can guide model generations into complex output spaces, and weak supervision from system outputs can train smaller task-specific models.The latter is presented as a route toward more deployable systems.
  • Future directions: Running locally inferable models such as OPT is identified as a next step for evaluating existing benchmarks and fine-tuning.This direction is motivated by the need for more accessible clinical experimentation.
  • Future directions: Combining outputs from multiple prompts could help identify GPT-3 uncertainty, which the conclusion describes as vital for reliability in high-stakes clinical information extraction.The proposed examples include one-shot prompts built from different demonstrations.

Limitations

The paper identifies limitations in exact schema adherence, handling cases with no answer, and generalizing beyond the CASI-based English evaluation setting.

  • Schema adherence: LLMs remain difficult to guide toward exact clinical annotation schemas, even when their outputs are qualitatively impressive.A duration output of “X weeks” omitted the guideline-required “for X weeks.”
  • Empty answers: GPT-3 tends to produce a non-trivial answer even when the requested entity type is absent.The authors suggest chaining prompts or using sequence-tagging-style output structures to reduce this bias.
  • Evaluation scope: Most tasks use re-annotated CASI data because existing clinical datasets restrict sharing with GPT-3 APIs, limiting evaluation across hospital systems and time periods.CASI includes multiple hospitals and specialties but is not representative of all clinical text.
  • Evaluation scope: The reported conclusions cannot be directly ported to other languages, hospital systems, or temporal settings because clinical text is subject to dataset shift.The CASI notes were primarily verbally dictated and transcribed, a practice that is not universal.

Ethics Statement

The ethics statement describes the use of existing public clinical text and notes potential benefits and deployment concerns for LLM-based clinical extraction.

  • Data and annotation: The new datasets add annotations to existing publicly available clinical text, produced by two paper authors with clinical backgrounds.The authors state that this avoids concerns such as annotator compensation.
  • Potential benefits: Automatic structuring of clinical variables could support studies of rarer diseases and real-world subpopulations not observed in clinical trials.The motivation is that clinical text is produced at a scale too large for manual annotation, while current cohorts are often small and hand-curated.
  • Deployment considerations: Direct GPT-3 inference has nontrivial cost, so the paper uses weak supervision to distill outputs into a smaller model and reduce deployment burden.The authors also encourage distillation to mitigate environmental impact.

A Prompts and Sample GPT-3 Outputs

The appendix illustrates prompting strategies and GPT-3 outputs across clinical extraction tasks, including abbreviation expansion, biomedical evidence extraction, coreference resolution, medication extraction, and token-level labeling.

  • A Prompts and Sample GPT-3 Outputs: Prompts and sample outputs are provided for each task to illustrate task-specific prompting strategies.
  • A.1 Clinical Sense Disambiguation: GPT-3 edit mode expands clinical abbreviations by editing the input text rather than continuing it.
  • A.1 Clinical Sense Disambiguation: For abbreviation expansion, the prompts ask GPT-3 to expand abbreviations such as CTA and PA, producing expanded clinical text.
  • A.2 Biomedical Evidence Extraction: Biomedical evidence extraction uses GPT-3 Davinci with prompts requesting bulleted lists of clinical-trial arms.
  • Example 1: One-shot guided (correct): A corrected guided example identifies “that” as referring to “her CEA” while asking the same style of antecedent question.
  • Example 1: Zero-shot guided: Medication prompts request each mentioned drug and its status as active, discontinued, or neither.
  • Example 1: One-shot guided (correct): The medication examples show that guided prompting can distinguish active, discontinued, and neither statuses, while token-level labeling targets medication attributes and ignores allergies.

C.2 Biomedical Evidence Extraction

This section describes baselines, output-resolution procedures, assumptions, and error analyses for biomedical evidence extraction and related medication and sequence-tagging tasks.

  • Resolver Details: GPT-3 outputs are resolved into binary token labels by matching output tokens to the input and filling intervening tokens or parenthetical acronyms.
  • Example of Token-level Error Modes: The token-level example correctly identifies both study arms but attains an F1-score of 0.70 because annotation-schema mismatches reduce precision and recall.
  • Oracle Details: PubMedBERT comparisons assume oracle span splitting and near-oracle coreference resolution, whereas Resolved GPT-3 receives neither oracle operation.
  • Analysis of Error Modes for Arm Identification: Resolved GPT-3 identified the correct number and content of study arms in 17 of 20 examples, and PubMedBERT missed the same three examples.
  • Medication Extraction Baseline: Medication extraction normalizes entities through UMLS linking and filters predictions to selected medication-related semantic types.
  • Status Classification: T-Few: Medication status classification uses T-Few with 20 additional annotated examples and a prompt asking for the status of Albuterol.
  • Sequence Tagging baseline: The sequence-tagging baseline represents medication attributes with typed B/I/O labels and trains on segmented notes from the 2009 i2b2 challenge.

D Experimental Cost

The reported experimentation cost was under $100, with upper-bound token and API-cost estimates based on spring 2022 OpenAI pricing.

  • Under $100 was spent on the included experiments, although token-per-example and API-cost estimates were rough upper bounds.
Loading 2205.12689v2…