PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study of Generative Large Language Model for Medical Research and Healthcare

Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, Gloria Lipori, Duane A Mitchell, Naykky S Ospina, Mustafa M Ahmed, William R Hogan, Elizabeth A Shenkman, Yi Guo, Jiang Bian, Yonghui Wu

arXiv:2305.13523v1cs.CL

TL;DR

Healthcare research has largely relied on general-purpose LLMs, leaving clinical models insufficiently evaluated. This study develops GatorTronGPT and finds state-of-the-art performance on 4 of 6 biomedical NLP benchmarks, while physicians could not distinguish its text from human text.

  • Problem

    Clinical NLP research has faced limited access to large-scale clinical text and shared clinical NLP models.

    Source: Discussion · p. 11

  • Method

    The study develops GatorTronGPT, a clinical generative LLM trained on a large clinical and English text corpus using a GPT-3 architecture.

    Source: Discussion · p. 10, Data Source · p. 13

  • Results

    GatorTronGPT achieves state-of-the-art performance on 4 of 6 biomedical NLP benchmark datasets, and physicians cannot differentiate its text from human text.

    Source: Discussion · pp. 10, 12

  • Takeaways & Limitations

    GatorTronGPT-generated synthetic clinical text shows potential utility for biomedical NLP and medical research.

    Source: Discussion · p. 12

    The study’s clinical corpus was collected from UF Health, and its synthetic-text comparison used the same architecture for comparability.

    Source: Data Source · p. 13, Task 3 - GatorTronGPT for synthetic… · p. 16

Abstract

from arXiv · show

There is enormous enthusiasm and concerns in using large language models (LLMs) in healthcare, yet current assumptions are all based on general-purpose LLMs such as ChatGPT. This study develops a clinical generative LLM, GatorTronGPT, using 277 billion words of mixed clinical and English text with a GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical natural language processing for medical research. Synthetic NLP models trained using GatorTronGPT generated text outperform NLP models trained using real-world clinical text. Physicians Turing test using 1 (worst) to 9 (best) scale shows that there is no significant difference in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights on the opportunities and challenges of LLMs for medical research and healthcare.

Discussion

GatorTronGPT is a clinical generative LLM that advances biomedical NLP and enables synthetic clinical text generation for medical research and healthcare. Its outputs can match human notes on readability and clinical relevance, but clinical logic, hallucination control, and real-world utility remain challenges.

Study contribution

  • GatorTronGPT uses the GPT-3 architecture and 277 billion words of mixed clinical and English text, achieving state-of-the-art performance on 4 of 6 biomedical NLP benchmark datasets.These results demonstrate benefits for medical research and healthcare-focused text generation.Source: Discussion · p. 10

Synthetic clinical text

  • Synthetic clinical text generation can reduce patient-privacy exposure and improve access to large-scale clinical text and sharing of state-of-the-art NLP models.The discussion links this utility to sensitive clinical text and incomplete protected health information removal by automatic de-identification systems.Source: Discussion · p. 11
  • GatorTronS, trained on 5 billion words of synthetic clinical text, achieves better or comparable performance on 5 clinical NLP tasks than GatorTron trained on 90 billion words of real-world clinical text.The discussion suggests redundancies in real-world text and greater diversity in generated text as possible reasons.Source: Discussion · p. 11

Physician evaluation

  • Readability showed no significant difference (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human), and clinical relevance also showed no significant difference (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human).Physicians could not differentiate synthetic and human notes (p < 0.001), although evaluators generally found GatorTronGPT text lacking clinical logic.Source: Discussion · p. 12

Limitations and future work

  • Clinical applications require domain-specific LLMs because probabilistic generation can produce hallucinations that are dangerous in healthcare.Future work should control hallucinations, examine limitations and risks, and test utility in real-world clinical practice.Source: Discussion · pp. 12–13
  • GatorTronGPT was evaluated in a zero-shot setting without human instructions, motivating studies of reinforcement learning from human feedback and P-tuning to improve and control clinical text generation.The discussion also calls for careful examination of potential limitations, biases, and risks before clinical application.Source: Discussion · p. 13

Data Source

The study combined UF Health clinical narratives with diverse English text into a 277-billion-word corpus, using clinical data collected across patients, encounters, departments, and care settings. The study had institutional review board approval and de-identified UF Health notes by removing 18 HIPAA-defined PHI categories.

Corpus composition

  • 277 billion words combined 82 billion UF Health clinical words with 195 billion diverse English words from the Pile corpus.The UF Health clinical corpus and Pile dataset were merged to generate the combined corpus.Source: Data Source · pp. 13–14

Governance

  • The study was approved by the UF Institutional Review Board under protocol IRB202102223.The approval applied to the study’s use of the clinical and English text corpora.Source: Data Source · p. 13

UF Health data

  • Approximately 290 million clinical notes were collected from 2011–2021 across over 126 departments, approximately 2 million patients, and 50 million encounters.The encounters covered inpatient, outpatient, and emergency settings.Source: Data Source · p. 13

Preprocessing

  • UF Health notes underwent de-identification to remove 18 PHI categories defined under HIPAA, while the Pile dataset received minimal preprocessing.Detailed preprocessing steps were described in the Supplement.Source: Data Source · p. 14

Train GatorTronGPT from scratch

GatorTronGPT was trained from scratch using GPT-3 architecture and two model configurations, with parallelized training on 560 NVIDIA DGX A100 GPUs. Training progress was monitored using training and validation loss, with validation loss computed on 3% of the data.

Configuration

  • 5 billion and 20 billion parameters were used for the two GatorTronGPT configurations.The configurations differed in depth, hidden size, and attention-head count.Source: Train GatorTronGPT from scratch · p. 14
  • The 5 billion model had 24 layers, hidden size 4,096, and 32 attention heads, whereas the 20 billion model had 44 layers, hidden size 6,144, and 48 attention heads.These architectural settings followed proposed depth-to-width allocation guidelines and prior GatorTron experience.Source: Train GatorTronGPT from scratch · p. 14

Training from scratch

  • GatorTronGPT models were trained from scratch with the inherited GPT-3 architecture implemented in MegaTron-LM and the default GPT-3 loss function.Training leveraged both data-level and model-level parallelisms implemented by Megatron-LM.Source: Train GatorTronGPT from scratch · p. 14
  • 560 NVIDIA DGX A100 GPUs from 70 superPOD nodes at UF’s HiPerGator-AI cluster were used for training.The training infrastructure used parallelism capabilities provided by the Megatron-LM package.Source: Train GatorTronGPT from scratch · p. 14
  • Training was monitored using training loss and validation loss computed with 3% of the data, and stopped when there was sufficient convergence.The supplied passage states that training stopped when there was an unspecified condition following loss monitoring.Source: Train GatorTronGPT from scratch · p. 14

GatorTronGPT for end-to-end biomedical relation extraction and question answering

GatorTronGPT was evaluated on end-to-end biomedical relation extraction and biomedical question answering using a unified prompt-based text-generation architecture with fixed-LLM prompt tuning. Relation extraction used three benchmark datasets, while question answering used three widely used biomedical datasets.

Task 1 - End-to-end biomedical relation extraction

Task 2 - Biomedical question answering

  • Both tasks used a unified prompt-based text-generation architecture with continuous virtual-token embeddings as soft prompts, while the LLM remained unchanged during training.The input sequence was [virtual tokens; x; y].Source: GatorTronGPT for end-to-end biomedical… · p. 15

Task 1 - End-to-end biomedical relation extraction

Task 2 - Biomedical question answering

  • Three widely used datasets—PubMedQA, MedMCQA, and MedQA-USMLE—were used for biomedical question answering evaluation.PubMedQA uses yes/no/maybe answers; MedMCQA covers 2,400 healthcare topics and 21 medical subjects; MedQA-USMLE comes from professional medical board exams.Source: GatorTronGPT for end-to-end biomedical… · p. 15

Task 3 - GatorTronGPT for synthetic clinical text generation

Task 3 tested whether GatorTronGPT could generate synthetic clinical text for training medical-research NLP models. The study used zero-shot generation, trained GatorTronS models on varying synthetic-text sizes, and compared them with GatorTron-base models trained on real-world text using matched architectures.

Task 3 - GatorTronGPT for synthetic clinical text generation

Synthetic clinical text generation

Synthetic clinical text was generated iteratively using nucleus and temperature sampling to balance diversity and quality. The process used MIMIC III note prompts, multiple random seeds, and a 512-token generation limit.

Synthetic clinical text generation

  • Approximately 8 million prompts were generated by sampling the beginning 15 tokens from MIMIC III de-identified notes.The prompts came from all sections of the database’s notes.Source: Synthetic clinical text generation · p. 16
  • Several random seeds generated multiple documents from a single prompt in GatorTronGPT.Using multiple seeds expanded document generation from each sampled prompt.Source: Synthetic clinical text generation · p. 16
  • Clinical text generation was limited up to 512 tokens and stopped when the maximum length was reached.The study provides further generation details in the Supplement.Source: Synthetic clinical text generation · p. 16

Synthetic NLP model development

The study generated synthetic clinical text at four controlled scales and developed corresponding GatorTronS synthetic NLP models using the GatorTron architecture.

Synthetic text generation

Synthetic NLP model development

Comparison with existing transformer models

The study compares GatorTronS trained on varying amounts of synthetic clinical text with ClinicalBERT50 and GatorTron15 across five clinical NLP tasks.

Task 4 - Turing test of text generation for clinical practice

Task 4 compared human-authored and GatorTronGPT-generated clinical-note sections using physician ratings of readability, clinical relevance and authorship. The evaluation also measured interrater reliability with percent agreement and Gwet’s AC1.

Data availability … GatorTronGPT for synthetic text generation

The section describes dataset access, clinical-text preprocessing and de-identification, and GatorTronGPT’s context-restricted approach to synthetic clinical text generation using top-p sampling.

Data availability

  • Benchmark datasets supporting the study’s findings are available from the official websites of natural language processing challenges.Source: Data availability · p. 18
  • Access to the benchmark datasets is governed by Data Use Agreements.Source: Data availability · p. 18

Preprocessing and de-identification of clinical text

GatorTronGPT for synthetic text generation

GatorTronGPT for biomedical relation extraction and question answering · Introduction to existing transformer models for comparison

GatorTronGPT addressed biomedical relation extraction and question answering through prompt-based generation with soft prompt tuning. Existing comparison models included GPT-2, REBEL/REBEL-pt, and BioGPT, spanning general, sequence-to-sequence, and biomedical generative approaches.

GatorTronGPT for biomedical relation extraction and question answering

  • GatorTronGPT formulated biomedical relation extraction and question answering as prompt-based text-generation tasks using prompt-tuning algorithms.The approach followed a previous study and applied p-tuning to both tasks.Source: GatorTronGPT for biomedical relation… · p. 24
  • Biomedical relation extraction used sequences structured as [prompt, context, relation], with LSTM-generated prompts and gold-standard entity-relation labels.The prompt consisted of learnable soft prompt embeddings concatenated with contextual word embeddings.Source: GatorTronGPT for biomedical relation… · p. 24
  • Question answering concatenated contexts and candidate answers into source sequences and used fine-tuned continuous embeddings as soft prompts.The target sequence explicitly represented the answer to the question given possible options.Source: GatorTronGPT for biomedical relation… · p. 25

Introduction to existing transformer models for comparison

  • 8 million webpages and 1.5 billion parameters were used to train GPT-2, which demonstrated text-generation ability for complex NLP tasks.GPT-2 was described as a scale-up of the first-generation GPT model.Source: Introduction to existing transformer… · p. 25
  • 9 out of 12 NLP tasks were outperformed by the original GPT model, while GPT-2 extended transformer-based text generation toward machine reading comprehension and question answering.The passage presents GPT and GPT-2 as successive developments in general-purpose transformer models.Source: Introduction to existing transformer… · p. 25
  • REBEL used BART-based sequence-to-sequence modeling for end-to-end relation extraction, and REBEL-pt further fine-tuned it on triplets derived from Wikipedia hyperlinks.REBEL outperformed earlier classification-based relation extraction models.Source: Introduction to existing transformer… · p. 25
  • BioGPT is a biomedical-domain generative transformer based on GPT-2 and trained on PubMed literature, with reported performance in relation extraction and question answering.Its reported NLP capabilities were specifically in the biomedical domain.Source: Introduction to existing transformer… · p. 25

Related papers

Loading 2305.13523v1…