PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Washington, You Chen, Lang Li, Huan Sun, Yu Su

arXiv:2203.08410v3cs.CLcs.IR

TL;DR

Biomedical IE needs effective few-shot learning because annotation is costly, so the paper systematically compares GPT-3 in-context learning with fine-tuned BERT-sized PLMs on NER and RE. Under the true few-shot setting, even optimized GPT-3 substantially underperforms fine-tuned smaller PLMs, while null-class handling remains a key difficulty.

  • Problem

    The paper addresses whether GPT-3 in-context learning is practical for biomedical NER and RE when annotations are costly and few-shot performance is evaluated without large-validation-set model selection.

  • Method

    The authors compare GPT-3 in-context learning with BERT-sized PLM fine-tuning across BLURB NER and RE datasets, optimizing GPT-3 with prompt selection, retrieval, calibration, and generation constraints.

  • Results

    Fine-tuned BERT-sized PLMs substantially outperform GPT-3 in-context learning across all biomedical IE datasets, with average F1 gaps of 15.6-16.7% for NER and 3.9-11.4% for RE.

  • Takeaways & Limitations

    Fine-tuning small PLMs remains the more cost-effective option for biomedical IE before qualitatively better in-context learning methods are discovered.

  • Takeaways & Limitations

    The study explores only a small prompt-style and prompt-example search space because its budget was limited.

Abstract

from arXiv · show

The strong few-shot in-context learning capability of large pre-trained language models (PLMs) such as GPT-3 is highly appealing for application domains such as biomedicine, which feature high and diverse demands of language technologies but also high data annotation costs. In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two highly representative biomedical information extraction tasks, named entity recognition and relation extraction. We follow the true few-shot setting to avoid overestimating models' few-shot performance by model selection over a large validation set. We also optimize GPT-3's performance with known techniques such as contextual calibration and dynamic in-context example retrieval. However, our results show that GPT-3 still significantly underperforms compared to simply fine-tuning a smaller PLM. In addition, GPT-3 in-context learning also yields smaller gains in accuracy when more training data becomes available. Our in-depth analyses further reveal issues of the in-context learning setting that may be detrimental to information extraction tasks in general. Given the high cost of experimenting with GPT-3, we hope our study provides guidance for biomedical researchers and practitioners towards more promising directions such as fine-tuning small PLMs.

1 Introduction

The paper tests whether GPT-3 in-context learning is practical for biomedical information extraction under realistic few-shot constraints. Across NER and RE, optimized GPT-3 remains substantially behind fine-tuned BERT-sized PLMs.

  • Biomedical IE is increasingly important because biomedical research and clinical text are produced at overwhelming scale.
  • High annotation costs, diverse biomedical IE applications, and complex model training make in-context learning appealing.
  • The study systematically compares GPT-3 in-context learning with BERT-sized PLM fine-tuning on biomedical NER and RE under the true few-shot setting.
  • The authors optimize GPT-3 with systematic prompt selection, dynamic example retrieval, logit biases for NER, and contextual calibration for RE.
  • Fine-tuning BERT-sized PLMs substantially outperforms GPT-3 in-context learning across biomedical IE datasets, while GPT-3 gains less as training data increases.The analysis also identifies difficulty handling null-class examples as a likely general problem for IE.

2 Approach

The approach compares standard few-shot fine-tuning with GPT-3 prompts reformulated for biomedical NER and RE. GPT-3 prompts are systematically designed, dynamically retrieved, and calibrated to reduce generation errors and label bias.

  • The study evaluates BERT-sized PLM fine-tuning and GPT-3 in-context learning under the true few-shot setting for NER and RE.
  • NER uses BIO token classification with a separate fine-tuned model for each entity type, while RE masks entities and classifies their relation from the [CLS] token.
  • GPT-3 reformulates NER as generating a separator-joined list of entity spans and RE as generating relations for verbatim subject-object entities.
  • Prompts contain task instructions, a sentence introduction, and a retrieval message, with relation-extraction labels mapped to natural-language verbalizers.
  • Prompt alternatives and in-context example counts are selected with leave-one-out cross-validation, while kNN retrieval selects similar training examples for each test input.
  • NER generation is constrained with logit biases favoring tokens from the input sentence, and contextual calibration adjusts GPT-3 output probabilities to reduce prompt-induced label bias.

3 Experiments

Experiments use BLURB biomedical NER and RE datasets to compare fine-tuned BERT-sized models with GPT-3 using matched 100-example training sets. The design includes multiple pretrained backbones and repeated training samples to assess variation.

  • The experiments use all NER and RE datasets included in the BLURB benchmark.
  • The benchmark includes biomedical entity datasets such as BC5CDR, NCBI-disease, and JNLPBA, alongside relation datasets including DDI, ChemProt, and GAD.
  • The main comparison fine-tunes PubMedBERT-base, BioBERT-large, and RoBERTa-large on 100 examples and retrieves GPT-3 prompts from the same 100 examples.
  • PubMedBERT uses biomedical-specific pretraining from scratch, whereas BioBERT starts from a general BERT checkpoint and both are pretrained on PubMed articles.
  • Experiments use the PubMedBERT base version, the original davinci model, and at most 10 NER or 5 RE in-context examples.
  • Table 2 evaluates at most 1,000 test examples per dataset across three different 100-example training sets, reporting standard deviations for data variance.

4 Results & Discussion

Across biomedical NER and RE datasets, fine-tuned BERT-sized PLMs substantially outperform GPT-3 in-context learning in the true few-shot setting. GPT-3 also benefits from calibration, retrieval, and other techniques, but remains less data-efficient and struggles particularly with null examples and relations.

  • Main Results: 15.6-16.7% for NER and 3.9-11.4% for RE in F1: fine-tuned BERT-sized PLMs outperform GPT-3 in-context learning across all datasets.The comparison uses the same small training set and often shows a large margin.
  • Main Results: GPT-3’s precision drops by an average of 10 points for NER, while recall drops by twice as much, indicating substantial entity underprediction.For RE, precision decreases more steeply, partly because GPT-3 performs poorly on the none relation class.
  • Main Results: BERT-sized PLMs reach the mid 80s on BC5CDR-chem but stall in the high and low 60s on disease and gene extraction, respectively.The differences are associated with lexical regularity in drug names versus greater lexical diversity in disease and gene names.
  • Main Results: RoBERTa-large outperforms PubMedBERT-base and BioBERT-large on RE, suggesting general-domain pretraining can offset domain-specific pretraining in low-resource semantic tasks.RE performance deteriorates for all methods, especially on DDI and ChemProt, which have more relation types and higher class imbalance.
  • Ablation Studies: Ablations show that logit bias and kNN retrieval improve GPT-3 NER performance complementarily, while calibration improves RE precision and recall.Removing logit bias can improve precision but lowers recall because invalid entities replace valid spans; removing calibration harms both RE metrics.
  • Detailed Error Analysis: Up to 50% of BC5CDR-disease sentences contain no disease, and including such null examples biases GPT-3 toward predicting fewer entities.This issue was identified as a major contributor to GPT-3’s poor in-context learning performance.
  • Detailed Error Analysis: Around 24%: GPT-3 recall improves after removing null disease sentences, compared with only 4% for RoBERTa-large.The result indicates that null examples affect GPT-3 much more strongly than they affect fine-tuning data.

5 Related Work

Prior work established GPT-3 in-context learning across many NLP tasks, but biomedical IE remained insufficiently evaluated. This study addresses that gap with a comprehensive benchmark-based assessment and finds further evidence of inadequacy for biomedical IE.

  • GPT-3 in-context learning has been competitive across diverse tasks, including classification, inference, translation, question answering, and semantic parsing.
  • This work is among the first comprehensive evaluations of GPT-3 in-context learning on information extraction tasks.
  • Prior prompt-design research reformulated tasks as language generation problems using manually constructed or generated cloze-style templates.
  • Biomedical sequence-to-sequence studies have outperformed previous methods on tasks including side-effect extraction, NER, and relation extraction.
  • True few-shot learning was introduced to address overestimation caused by using large validation sets for model and prompt selection.
  • Using recent techniques and a multi-dataset biomedical benchmark, this study provides more evidence that GPT-3 in-context learning is inadequate for biomedical IE.

6 Conclusions

The paper evaluates GPT-3 in-context learning for biomedical IE and compares it with BERT-sized PLM fine-tuning. Across biomedical NER and RE benchmarks, current techniques do not enable GPT-3 to surpass fine-tuning, while null examples emerge as a potential limitation.

  • The study explores GPT-3 in-context learning for biomedical information extraction, focusing on biomedical NER and RE benchmark tasks.
  • Current GPT-3 techniques do not surpass BERT-sized PLM fine-tuning across benchmark datasets for biomedical NER and RE.
  • Null examples, including entity-less NER and vacuous RE instances, are identified as potentially general limitations of in-context learning for biomedical IE.
  • The paper points researchers toward small PLM fine-tuning or directly fine-tuning GPT-3 as cost-effective low-resource IE directions.

Limitations

The study’s conclusions are bounded by limited prompt-search resources and unresolved questions about validation, retrieval, calibration, and the sources of GPT-3’s biomedical IE difficulties.

  • A limited budget constrained the study to a small set of prompt styles and prompt-example counts.
  • Larger prompt-search spaces or more examples per prompt might narrow the gap between GPT-3 in-context learning and small PLM fine-tuning.
  • It remains unclear how larger validation sets would affect prompt selection while compromising the true few-shot assumption.
  • Whole-sentence kNN retrieval may be insufficient for relation extraction, which requires attention around the target entity pair.

A Experimental Setup Details

The experiments compare standard fine-tuning of BERT-sized PLMs with GPT-3 in-context learning for NER and RE under few-shot evaluation. Setup details cover task formulations, validation procedures, computational costs, and model behavior across tasks.

  • NER: NER assigns BIO tags to tokens using contextual encoder representations and a linear prediction layer that are fine-tuned together.
  • RE: RE masks subject and object entities with [ENT1] and [ENT2], then predicts the relation type from the [CLS] representation.
  • Compute: Experiments used four NVIDIA GeForce RTX 2080 Ti GPUs, with GPU hours and GPT-3 costs reported in Table 8.
  • Validation and search: Five-fold cross-validation selects fine-tuning hyperparameters for each 100-sample training subset.
  • Validation and search: Leave-one-out cross-validation independently selects prompt designs for each 100-sample subset to preserve the true few-shot setting.
  • Base-model results: PubMedBERT-base performs best among base models on NER, whereas RoBERTa models perform best on RE tasks.

F.1 Confusion Matrices

Figure 4 compares GPT-3 and RoBERTa-large confusion matrices for DDI relation extraction, using 100 validation examples. The analysis highlights GPT-3’s difficulty identifying the none class.

  • Confusion matrices: GPT-3 in-context learning rarely predicts the none class, where two drugs have no relation.
  • Experimental setup: The experiments use leave-one-out cross-validation on 100 training examples to select among prompt-design options and in-context-example counts.The selection covers eight combinations of task instruction, sentence introduction, retrieval message, and example count.
  • Experimental setup: Table 11 extends the main results with BioBERT and RoBERTa-base baselines for comparison.
  • Confusion matrices: Figure 4 shows confusion matrices for GPT-3 and RoBERTa-large on 100 DDI validation examples.The matrices combine folds from 5-fold cross-validation for RoBERTa-large and leave-one-out cross-validation for GPT-3.

F.2 Qualitative Analysis

The qualitative analysis examines why GPT-3 misclassifies DDI none examples that RoBERTa-large handles correctly. LIME attributions suggest GPT-3 relies more on irrelevant or localized word-level signals, whereas RoBERTa uses broader phrase-level information.

  • Error patterns: RoBERTa-large and GPT-3 use similar surface-level signals on positive DDI examples, but differ substantially on none examples.The comparison covers three positive and three none examples, with RoBERTa-large correct and GPT-3 incorrect on the none cases.
  • Error patterns: GPT-3’s first none-example prediction is influenced by irrelevant drug-list terms and phrases about unperformed studies and metabolism.RoBERTa is unaffected by removing listed drugs and responds appropriately to removing the important “CYP3A4 (eg.” signal.
  • Error patterns: In the second none example, GPT-3 is most affected by isolated words, while RoBERTa distributes sensitivity across the surrounding phrase.This indicates broader phrase-level reliance for RoBERTa rather than word-level reliance.
  • Error patterns: Removing “with L-tryptophan” in the final example would create an interaction between the queried drugs by changing the sentence meaning.

F.3 LIME Details

The paper uses a modified LIME procedure to compare local explanations for GPT-3 in-context learning and RoBERTa relation-extraction predictions. The analysis focuses on how phrase removal changes model behavior.

  • LIME procedure: LIME provides faithful local explanations that can be directly compared between GPT-3 in-context learning and RoBERTa.
  • LIME outputs: Table 12 reports saliency scores for three positive and three none DDI examples, including cases correctly handled by RoBERTa-large but missed by GPT-3.Blue-highlighted words change the current prediction, and color intensity indicates the strength of the effect.
  • LIME procedure: The modified LIME implementation replaces random token masking with a sliding window of three tokens.This modification supports analysis of phrase-removal effects while retaining a reasonable number of neighboring examples.
  • Scope: The LIME analysis is applied to relation extraction rather than the paper’s other information-extraction task.
Loading 2203.08410v3…