Source-linked AI summary
Identifying and Extracting Rare Disease Phenotypes with Large Language Models
Cathy Shyr, Yan Hu, Paul A. Harris, Hua Xu
TL;DR
Rare-disease phenotypes are often buried in unstructured text, making extraction labor-intensive, while ChatGPT’s zero- and few-shot NER performance had not been studied. The paper benchmarks prompted ChatGPT against fine-tuned BioClinicalBERT and analyzes errors. Fine-tuning performs better overall, although ChatGPT matches or exceeds it for some entity types in few-shot evaluation.
Problem
Rare-disease phenotypes are difficult to extract from unstructured text, and ChatGPT’s NER performance in zero- and few-shot settings had not been studied.
Source: §1 · pp. 2–3Method
The study engineers ChatGPT prompts for rare-disease phenotype extraction, benchmarks zero- and few-shot performance against fine-tuned BioClinicalBERT, and analyzes errors.
Source: §1 · p. 3, §2.2.2 · p. 5Results
BioClinicalBERT achieves F1 0.689 overall, versus ChatGPT’s 0.472 zero-shot and 0.591 few-shot; ChatGPT reaches 0.776 for rare diseases and 0.725 for signs in few-shot evaluation.
Source: §3.1 · p. 7, §4 · p. 10Takeaways & Limitations
With one example, ChatGPT achieves similar or higher performance than its supervised counterpart for rare diseases and signs, suggesting potential for few-shot NER.
Source: §4 · p. 10Only 832 of the 1,041 RareDis texts were available, so results may not fully reflect ChatGPT performance across the full spectrum of rare diseases.
Source: §4 · p. 10
Abstract
from arXiv · showhide
Rare diseases (RDs) are collectively common and affect 300 million people worldwide. Accurate phenotyping is critical for informing diagnosis and treatment, but RD phenotypes are often embedded in unstructured text and time-consuming to extract manually. While natural language processing (NLP) models can perform named entity recognition (NER) to automate extraction, a major bottleneck is the development of a large, annotated corpus for model training. Recently, prompt learning emerged as an NLP paradigm that can lead to more generalizable results without any (zero-shot) or few labeled samples (few-shot). Despite growing interest in ChatGPT, a revolutionary large language model capable of following complex human prompts and generating high-quality responses, none have studied its NER performance for RDs in the zero- and few-shot settings. To this end, we engineered novel prompts aimed at extracting RD phenotypes and, to the best of our knowledge, are the first the establish a benchmark for evaluating ChatGPT's performance in these settings. We compared its performance to the traditional fine-tuning approach and conducted an in-depth error analysis. Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.591 in the zero- and few-shot settings, respectively). Despite this, ChatGPT achieved similar or higher accuracy for certain entities (i.e., rare diseases and signs) in the one-shot setting (F1 of 0.776 and 0.725). This suggests that with appropriate prompt engineering, ChatGPT has the potential to match or outperform fine-tuned language models for certain entity types with just one labeled sample. While the proliferation of large language models may provide opportunities for supporting RD diagnosis and treatment, researchers and clinicians should critically evaluate model outputs and be well-informed of their limitations.
1. Introduction
Rare-disease phenotyping can support diagnosis, treatment, and clinical research, but relevant information is often buried in text and costly to extract. This study addresses gaps in rare-disease NER and evaluates ChatGPT prompting against fine-tuning.
1. Introduction
- Rare diseases affect 300 million people worldwide and can bring diagnostic and therapeutic odysseys with serious medical, psychosocial, and economic consequences.The paper frames effective diagnostic and treatment strategies as necessary to shorten these odysseys.Source: §1 · p. 2
- Rare-disease phenotypes are often embedded in unstructured text, while manual extraction by trained experts is laborious, costly, and susceptible to bias.Phenotyping can inform diagnosis, disease management, and clinical-trial cohort recruitment, but scarce nosological guidelines leave many phenotypes unstructured.Source: §1 · p. 2
- Rare-disease NER is difficult because entities vary in synonyms, abbreviations, modifiers, boundaries, and overlap, from lay phrases to medical jargon.Descriptions may be discontinuous, nested, or overlapping.Source: §1 · p. 2
- Although transformer-based NER has advanced, rare-disease applications remain limited, and model training still relies on labor-intensive expert-annotated corpora.Prior rare-disease NER studies had not evaluated ChatGPT in zero- or few-shot settings.Source: §1 · pp. 2–3
- The study designs ChatGPT prompts for extracting rare diseases and phenotypes, benchmarks zero- and few-shot NER, compares fine-tuning, and analyzes errors.The benchmark uses an annotated rare-disease corpus and the work offers suggestions for future rare-disease NER research.Source: §1 · p. 3
2. Methods
The methods use RareDis to compare supervised fine-tuning with ChatGPT prompt learning for four overlapping entity types. The evaluation tests exact and relaxed matching on held-out texts.
2.1 Dataset
- RareDis contains 832 NORD texts annotated for rare diseases, diseases, symptoms, and signs; rare diseases and signs make up 77% of its entities.The corpus is publicly available in Brat standoff format, and inter-annotator agreement is 83.5% under exact match.Source: §2.1 · pp. 3–4
Paradigms and Models
- The study compares pre-training plus supervised fine-tuning with prompt learning, which reformulates NER as prompts for predicting the desired output.The comparison uses BERT-based models for fine-tuning and GPT-based models for prompt learning.Source: Paradigms and Models · pp. 4–5
2.2.1 Data Pre-processing and Fine-tuning BioClinicalBERT
- BioClinicalBERT is trained on an 8:1:1 training, validation, and test split, with validation data used for hyperparameter selection and test data for evaluation.Inputs are tokenized, padded or truncated, masked, and mapped to numerical labels for the four entity types.Source: §2.2.1 · p. 5
2.2.2 Prompt Learning using ChatGPT (GPT-3.5-turbo)
- ChatGPT receives zero-shot extraction instructions or, in few-shot settings, an example; prompts specify entity guidance, exact-name extraction, and output format.The study compares sentence and structured-list formats, and selects examples randomly or by spaCy embedding similarity to the test text.Source: §2.2.2 · pp. 5, 7
2.3 Evaluation
- Evaluation measures precision, recall, and F1 under exact matching of span and type or relaxed matching of overlapping spans with the same type.Stop words are removed from gold-standard and model-extracted entities before scoring.Source: §2.3 · p. 7
3. Results
BioClinicalBERT outperformed ChatGPT overall, although ChatGPT matched or exceeded it for some entity types in few-shot prompting. Error analysis found frequent rare-disease misses and type confusions amid overlapping entity meanings.
3.1 Overall Results
- 0.776 vs. 0.755 rare-disease F1 and 0.725 vs. 0.704 signs F1 favored ChatGPT over BioClinicalBERT in few-shot prompting.Simple-sentence prompts generally performed similarly or better than structured lists; similar-text example selection outperformed random selection.Source: §3.1 · p. 7
- 0.689 BioClinicalBERT F1 exceeded ChatGPT’s 0.472 zero-shot and 0.591 few-shot F1 under relaxed match.Table 3 summarizes model performance by entity type.Source: §3.1 · p. 7, Table 3 · p. 8
3.2 Detailed Error Analysis
- RareDis’s overlapping entity meanings—rare diseases versus diseases, symptoms versus signs, and context-dependent disease mentions—motivated detailed error analysis.Source: §3.2 · p. 9
- The analysis classified errors as boundary errors, entity-type errors, combined boundary-and-type errors, spurious extractions, or missed entities.Spurious extractions are false positives; missed entities are false negatives.Source: §3.2 · p. 9
- 45% of rare-disease errors were false negatives and 31% were entity-type errors; false positives and false negatives were most common for other entity types.Rare-disease/type confusions included labeling rare diseases as diseases; the authors also cite variable definitions and subtle entity distinctions as possible sources of errors.Source: §3.2 · p. 9
4. Discussion
The authors compare prompt-based ChatGPT with fine-tuned models, finding lower overall performance but similar or better results for some entity types. They recommend concise sentence prompts while noting limits in corpus coverage, model scope, and prompt stability.
4. Discussion
- The authors recommend simple sentence-based prompts because they performed similarly or better than lists and were shorter, reducing computational cost.Source: §4 · p. 10
- ChatGPT’s overall results aligned with prior findings of lower biomedical NER performance than fine-tuned models, but it matched or exceeded fine-tuned models for some entity types.The authors contrast BioClinicalBERT’s F1 of 0.689 with ClinicalBERT’s reported 0.695; the comparison involved different studies.Source: §4 · p. 10
- The study used only 832 of 1,041 RareDis texts and evaluated ChatGPT, not GPT-4 or other large language model variants.The authors state that results may not reflect ChatGPT’s performance across the full spectrum of rare diseases.Source: §4 · p. 10
- Small prompt changes can vary performance; the authors suggest automated prompt engineering and urge critical evaluation of model outputs.They describe semi-automated and embedding-space prompt-learning approaches as possible extensions.Source: §4 · p. 11
5. Data and Code
The RareDis corpus is linked through Martínez-deMiguel et al. (2022), and the study code is hosted on GitHub.
5. Data and Code
- The RareDis corpus is linked through Martínez-deMiguel et al. (2022), and study code is available in the authors’ GitHub repository.Source: §5 · p. 11