PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study of Generative Large Language Model for Medical Research and Healthcare

Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, Gloria Lipori, Duane A Mitchell, Naykky S Ospina, Mustafa M Ahmed, William R Hogan, Elizabeth A Shenkman, Yi Guo, Jiang Bian, Yonghui Wu

arXiv:2305.13523v1cs.CL

TL;DR

Healthcare research has largely relied on general-purpose LLMs, leaving clinical models insufficiently evaluated. This study develops GatorTronGPT and finds state-of-the-art performance on 4 of 6 biomedical NLP benchmarks, while physicians could not distinguish its text from human text.

  • Problem

    Clinical NLP research has faced limited access to large-scale clinical text and shared clinical NLP models.

  • Method

    The study develops GatorTronGPT, a clinical generative LLM trained on a large clinical and English text corpus using a GPT-3 architecture.

  • Results

    GatorTronGPT achieves state-of-the-art performance on 4 of 6 biomedical NLP benchmark datasets, and physicians cannot differentiate its text from human text.

  • Takeaways & Limitations

    GatorTronGPT-generated synthetic clinical text shows potential utility for biomedical NLP and medical research.

  • Takeaways & Limitations

    The study’s clinical corpus was collected from UF Health, and its synthetic-text comparison used the same architecture for comparability.

Abstract

from arXiv · show

There is enormous enthusiasm and concerns in using large language models (LLMs) in healthcare, yet current assumptions are all based on general-purpose LLMs such as ChatGPT. This study develops a clinical generative LLM, GatorTronGPT, using 277 billion words of mixed clinical and English text with a GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical natural language processing for medical research. Synthetic NLP models trained using GatorTronGPT generated text outperform NLP models trained using real-world clinical text. Physicians Turing test using 1 (worst) to 9 (best) scale shows that there is no significant difference in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights on the opportunities and challenges of LLMs for medical research and healthcare.

Discussion

GatorTronGPT is a clinical generative LLM that advances biomedical NLP and enables synthetic clinical text generation for medical research and healthcare. Its outputs can match human notes on readability and clinical relevance, but clinical logic, hallucination control, and real-world utility remain challenges.

  • Study contribution: GatorTronGPT uses the GPT-3 architecture and 277 billion words of mixed clinical and English text, achieving state-of-the-art performance on 4 of 6 biomedical NLP benchmark datasets.These results demonstrate benefits for medical research and healthcare-focused text generation.
  • Synthetic clinical text: Synthetic clinical text generation can reduce patient-privacy exposure and improve access to large-scale clinical text and sharing of state-of-the-art NLP models.The discussion links this utility to sensitive clinical text and incomplete protected health information removal by automatic de-identification systems.
  • Synthetic clinical text: GatorTronS, trained on 5 billion words of synthetic clinical text, achieves better or comparable performance on 5 clinical NLP tasks than GatorTron trained on 90 billion words of real-world clinical text.The discussion suggests redundancies in real-world text and greater diversity in generated text as possible reasons.
  • Physician evaluation: Readability showed no significant difference (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human), and clinical relevance also showed no significant difference (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human).Physicians could not differentiate synthetic and human notes (p < 0.001), although evaluators generally found GatorTronGPT text lacking clinical logic.
  • Limitations and future work: Clinical applications require domain-specific LLMs because probabilistic generation can produce hallucinations that are dangerous in healthcare.Future work should control hallucinations, examine limitations and risks, and test utility in real-world clinical practice.
  • Limitations and future work: GatorTronGPT was evaluated in a zero-shot setting without human instructions, motivating studies of reinforcement learning from human feedback and P-tuning to improve and control clinical text generation.The discussion also calls for careful examination of potential limitations, biases, and risks before clinical application.

Data Source

The study combined UF Health clinical narratives with diverse English text into a 277-billion-word corpus, using clinical data collected across patients, encounters, departments, and care settings. The study had institutional review board approval and de-identified UF Health notes by removing 18 HIPAA-defined PHI categories.

  • Corpus composition: 277 billion words combined 82 billion UF Health clinical words with 195 billion diverse English words from the Pile corpus.The UF Health clinical corpus and Pile dataset were merged to generate the combined corpus.
  • Governance: The study was approved by the UF Institutional Review Board under protocol IRB202102223.The approval applied to the study’s use of the clinical and English text corpora.
  • UF Health data: Approximately 290 million clinical notes were collected from 2011–2021 across over 126 departments, approximately 2 million patients, and 50 million encounters.The encounters covered inpatient, outpatient, and emergency settings.
  • Preprocessing: UF Health notes underwent de-identification to remove 18 PHI categories defined under HIPAA, while the Pile dataset received minimal preprocessing.Detailed preprocessing steps were described in the Supplement.

Train GatorTronGPT from scratch

GatorTronGPT was trained from scratch using GPT-3 architecture and two model configurations, with parallelized training on 560 NVIDIA DGX A100 GPUs. Training progress was monitored using training and validation loss, with validation loss computed on 3% of the data.

  • Configuration: 5 billion and 20 billion parameters were used for the two GatorTronGPT configurations.The configurations differed in depth, hidden size, and attention-head count.
  • Configuration: The 5 billion model had 24 layers, hidden size 4,096, and 32 attention heads, whereas the 20 billion model had 44 layers, hidden size 6,144, and 48 attention heads.These architectural settings followed proposed depth-to-width allocation guidelines and prior GatorTron experience.
  • Training from scratch: GatorTronGPT models were trained from scratch with the inherited GPT-3 architecture implemented in MegaTron-LM and the default GPT-3 loss function.Training leveraged both data-level and model-level parallelisms implemented by Megatron-LM.
  • Training from scratch: 560 NVIDIA DGX A100 GPUs from 70 superPOD nodes at UF’s HiPerGator-AI cluster were used for training.The training infrastructure used parallelism capabilities provided by the Megatron-LM package.
  • Training from scratch: Training was monitored using training loss and validation loss computed with 3% of the data, and stopped when there was sufficient convergence.The supplied passage states that training stopped when there was an unspecified condition following loss monitoring.

GatorTronGPT for end-to-end biomedical relation extraction and question answering

GatorTronGPT was evaluated on end-to-end biomedical relation extraction and biomedical question answering using a unified prompt-based text-generation architecture with fixed-LLM prompt tuning. Relation extraction used three benchmark datasets, while question answering used three widely used biomedical datasets.

  • Task 1 - End-to-end biomedical relation extraction: End-to-end relation extraction identifies concept-relation-concept triplets from biomedical text.
  • Task 2 - Biomedical question answering: Biomedical question answering identifies the answer to a given question using its context.
  • Both tasks used a unified prompt-based text-generation architecture with continuous virtual-token embeddings as soft prompts, while the LLM remained unchanged during training.The input sequence was [virtual tokens; x; y].
  • Task 1 - End-to-end biomedical relation extraction: Three benchmark datasets—DDI, BC5CDR, and KD-DTI—were used for end-to-end biomedical relation extraction.
  • Task 2 - Biomedical question answering: Three widely used datasets—PubMedQA, MedMCQA, and MedQA-USMLE—were used for biomedical question answering evaluation.PubMedQA uses yes/no/maybe answers; MedMCQA covers 2,400 healthcare topics and 21 medical subjects; MedQA-USMLE comes from professional medical board exams.

Task 3 - GatorTronGPT for synthetic clinical text generation

Task 3 tested whether GatorTronGPT could generate synthetic clinical text for training medical-research NLP models. The study used zero-shot generation, trained GatorTronS models on varying synthetic-text sizes, and compared them with GatorTron-base models trained on real-world text using matched architectures.

  • Task 3 - GatorTronGPT for synthetic clinical text generation: The study tested whether LLM-generated synthetic clinical text could train NLP models useful for medical research.
  • Task 3 - GatorTronGPT for synthetic clinical text generation: GatorTronGPT generated synthetic clinical text from seeds without fine-tuning in a typical zero-shot learning setting.
  • Task 3 - GatorTronGPT for synthetic clinical text generation: Researchers trained GatorTronS transformer-based NLP models on different sizes of synthetic clinical text and compared performance with GatorTron-base models trained on real-world text.
  • Task 3 - GatorTronGPT for synthetic clinical text generation: GatorTronS used the same architecture and 345 million parameters as GatorTron-base to enable comparison.

Synthetic clinical text generation

Synthetic clinical text was generated iteratively using nucleus and temperature sampling to balance diversity and quality. The process used MIMIC III note prompts, multiple random seeds, and a 512-token generation limit.

  • Synthetic clinical text generation: Approximately 8 million prompts were generated by sampling the beginning 15 tokens from MIMIC III de-identified notes.The prompts came from all sections of the database’s notes.
  • Synthetic clinical text generation: Several random seeds generated multiple documents from a single prompt in GatorTronGPT.Using multiple seeds expanded document generation from each sampled prompt.
  • Synthetic clinical text generation: Clinical text generation was limited up to 512 tokens and stopped when the maximum length was reached.The study provides further generation details in the Supplement.

Synthetic NLP model development

The study generated synthetic clinical text at four controlled scales and developed corresponding GatorTronS synthetic NLP models using the GatorTron architecture.

  • Synthetic text generation: 1 billion, 5 billion, 10 billion, and 20 billion words of clinical text were generated at controlled scales.
  • Synthetic NLP model development: The corresponding synthetic NLP models were developed and denoted as GatorTronS.
  • Synthetic NLP model development: GatorTronS used the same architecture as GatorTron: a BERT architecture with 345 million parameters.

Comparison with existing transformer models

The study compares GatorTronS trained on varying amounts of synthetic clinical text with ClinicalBERT50 and GatorTron15 across five clinical NLP tasks.

  • GatorTronS models trained with different amounts of synthetic clinical text were compared.
  • The comparison included ClinicalBERT50, trained on biomedical literature and clinical notes from MIMIC III.
  • The comparison also included GatorTron15, the current largest clinical transformer model trained using >90 billion words of text.
  • The models were evaluated on 5 clinical NLP tasks: concept extraction or NER, medical relation extraction, semantic textual similarity, natural language inference, and question answering.

Task 4 - Turing test of text generation for clinical practice

Task 4 compared human-authored and GatorTronGPT-generated clinical-note sections using physician ratings of readability, clinical relevance and authorship. The evaluation also measured interrater reliability with percent agreement and Gwet’s AC1.

  • 30 narrative sections from UF Health clinical notes were randomly sampled across past medical history, history of present illness, assessment/plan and chief complaint.
  • Each section’s first 15 tokens seeded GatorTronGPT to generate a synthetic paragraph of up to 512 tokens, which was mixed with 30 truncated real sections.
  • Two UF Health physicians reviewed 60 mixed paragraphs for readability, clinical relevance and consistency, and whether each was human- or GatorTronGPT-written.
  • Percent agreement and Gwet’s AC1 were calculated to assess interrater reliability.

Data availability … GatorTronGPT for synthetic text generation

The section describes dataset access, clinical-text preprocessing and de-identification, and GatorTronGPT’s context-restricted approach to synthetic clinical text generation using top-p sampling.

  • Data availability: Benchmark datasets supporting the study’s findings are available from the official websites of natural language processing challenges.
  • Data availability: Access to the benchmark datasets is governed by Data Use Agreements.
  • Preprocessing and de-identification of clinical text: Clinical-text preprocessing removed empty and duplicate notes, unified UTF-8 encoding, removed illegal strings, normalized special characters, and performed tokenization and sentence-boundary detection.
  • Preprocessing and de-identification of clinical text: A de-identification system removed protected health information from UF Health clinical text under IRB202100049 approval.
  • GatorTronGPT for synthetic text generation: Synthetic clinical text generation was formulated as an open-ended text-to-text task in which generated text is restricted by input context such as prompts.
  • GatorTronGPT for synthetic text generation: Top-p nucleus sampling selects words whose cumulative probability exceeds a predefined threshold p when generating the next token.
  • GatorTronGPT for synthetic text generation: The top-p vocabulary dynamically adjusts the number of candidate words at each step according to their probabilities, balancing diversity and coherence.

GatorTronGPT for biomedical relation extraction and question answering · Introduction to existing transformer models for comparison

GatorTronGPT addressed biomedical relation extraction and question answering through prompt-based generation with soft prompt tuning. Existing comparison models included GPT-2, REBEL/REBEL-pt, and BioGPT, spanning general, sequence-to-sequence, and biomedical generative approaches.

  • GatorTronGPT for biomedical relation extraction and question answering: GatorTronGPT formulated biomedical relation extraction and question answering as prompt-based text-generation tasks using prompt-tuning algorithms.The approach followed a previous study and applied p-tuning to both tasks.
  • GatorTronGPT for biomedical relation extraction and question answering: Biomedical relation extraction used sequences structured as [prompt, context, relation], with LSTM-generated prompts and gold-standard entity-relation labels.The prompt consisted of learnable soft prompt embeddings concatenated with contextual word embeddings.
  • GatorTronGPT for biomedical relation extraction and question answering: Question answering concatenated contexts and candidate answers into source sequences and used fine-tuned continuous embeddings as soft prompts.The target sequence explicitly represented the answer to the question given possible options.
  • Introduction to existing transformer models for comparison: 8 million webpages and 1.5 billion parameters were used to train GPT-2, which demonstrated text-generation ability for complex NLP tasks.GPT-2 was described as a scale-up of the first-generation GPT model.
  • Introduction to existing transformer models for comparison: 9 out of 12 NLP tasks were outperformed by the original GPT model, while GPT-2 extended transformer-based text generation toward machine reading comprehension and question answering.The passage presents GPT and GPT-2 as successive developments in general-purpose transformer models.
  • Introduction to existing transformer models for comparison: REBEL used BART-based sequence-to-sequence modeling for end-to-end relation extraction, and REBEL-pt further fine-tuned it on triplets derived from Wikipedia hyperlinks.REBEL outperformed earlier classification-based relation extraction models.
  • Introduction to existing transformer models for comparison: BioGPT is a biomedical-domain generative transformer based on GPT-2 and trained on PubMed literature, with reported performance in relation extraction and question answering.Its reported NLP capabilities were specifically in the biomedical domain.
Loading 2305.13523v1…