PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking large language models for biomedical natural language processing applications and recommendations

Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B. Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, Vipina Kuttichi Keloth, Kalpana Raja, Jiming Huang, Huan He, Fongci Lin, Jingcheng Du, Rui Zhang, W. Jim Zheng, Ron A. Adelman, Zhiyong Lu, Hua Xu

arXiv:2305.16326v5cs.CLcs.AIcs.IRcs.LG

TL;DR

Biomedical literature is difficult to curate because it is vast, specialized, and linguistically ambiguous, while LLM effectiveness in BioNLP remains insufficiently benchmarked. The study evaluates four LLMs across 12 datasets and six applications under zero-shot, few-shot, and fine-tuning settings, including manual quality and cost analyses. Traditional fine-tuning performs best in most tasks, whereas GPT models excel particularly in medical question answering; the findings support task-specific recommendations while highlighting output errors and evaluation limitations.

  • Problem

    The study addresses limited systematic evidence and practical guidance on LLM effectiveness in BioNLP despite rapidly expanding biomedical literature and limited labeled data.

  • Method

    The authors evaluate four LLMs on 12 BioNLP datasets across six applications, comparing zero-shot, few-shot, and fine-tuning settings with SOTA fine-tuning approaches and manually assessing outputs and costs.

  • Results

    SOTA fine-tuning outperformed zero- and few-shot LLMs in most tasks, while GPT-3.5 and GPT-4 outperformed SOTA approaches in reasoning-related medical question answering.

  • Takeaways & Limitations

    LLM use in BioNLP should follow task and resource conditions, with recommendations informed by performance, output reliability, and cost.

Abstract

from arXiv · show

The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs, GPT and LLaMA representatives on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here we show that traditional fine-tuning outperforms zero or few shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.

Introduction

The study addresses the difficulty of curating and interpreting rapidly expanding, ambiguous biomedical literature by systematically evaluating LLMs for BioNLP. It compares prompting and fine-tuning across tasks while examining output quality, costs, and practical use.

  • Introduction: PubMed adds approximately 5,000 articles daily and exceeded 36 million articles by March 2024, while biomedical terms can be highly ambiguous.Long COVID has 763 different terms, and AP2 can denote a gene, chemical, or cell line.
  • Introduction: BioNLP assists manual curation, interpretation, and knowledge discovery by leveraging biomedical language models trained on biomedical literature.
  • Introduction: The study evaluates GPT-3.5, GPT-4, LLaMA 2, and PMC LLaMA on 12 datasets spanning six BioNLP applications.The applications are named entity recognition, relation extraction, document classification, question answering, text summarization, and text simplification.
  • Introduction: The evaluation compares zero-shot, few-shot, and fine-tuning settings with established BERT- and BART-based approaches, while manually examining missing, inconsistent, and hallucinated outputs.
  • Introduction: SOTA fine-tuning outperformed zero- and few-shot LLMs in most BioNLP tasks, whereas closed-source models performed better on reasoning-related medical question answering.GPT-4 had the highest overall performance but cost 60 to 100 times more than GPT-3.5.
  • Introduction: The study provides downstream recommendations, identifies evaluation-paradigm and error-mitigation problems, and makes data, models, and results publicly accessible.

Results

Across 12 datasets, sophisticated fine-tuning was strongest in most extraction and classification tasks, while LLMs were more competitive in question answering and generation. The results also expose substantial output-quality variation, cost-performance trade-offs, and limitations of automatic evaluation.

  • Quantitative evaluations: 0.6536 was the SOTA fine-tuning macro-average across 12 datasets, versus 0.4561, 0.4750, 0.4862, and 0.5131 for the best LLM settings.Fine-tuning outperformed zero- and few-shot LLMs in 10 of 12 datasets.
  • Quantitative evaluations: 0.9090 was the SOTA entity-level F1-score on NCBI Disease, compared with 0.5988 for GPT-4 under zero- and one-shot settings.Fine-tuned LLaMA 2 13B reached 0.8682 on this dataset.
  • Quantitative evaluations: 0.7156 was GPT-4's zero-shot MedQA accuracy, compared with 0.4195 for the SOTA approach; GPT-4 also reached 0.7580 on PubMedQA with five shots.
  • Quantitative evaluations: GPT-4 had the highest zero- and few-shot performance, achieving a 0.4561 macro-average and the best result in nine of 12 datasets under zero-shot evaluation.Its macro-average was approximately 7% higher than GPT-3.5 and almost double LLaMA 2 13B.
  • Quantitative evaluations: Fine-tuning improved LLaMA 2 13B's macro-average from 0.2837 to 0.5131, but input contexts exceeding its 4,096-token limit constrained gains in summarization and simplification.
  • Evaluations on inconsistencies, missing information, and hallucinations: One-shot prompting substantially changed LLaMA 2 13B's NCBI Disease results, increasing correctly predicted entities from 148 under zero-shot to 449.
  • Evaluations on inconsistencies, missing information, and hallucinations: LLaMA 2 13B zero-shot outputs included 102 hallucinated and 69 inconsistent HoC cases, representing 32% and 22% of testing instances, respectively.Across ChemProt, it produced 506 hallucinated responses and 2,376 inconsistent responses.
  • Evaluations on inconsistencies, missing information, and hallucinations: GPT-3.5 and GPT-4 had similar or slightly higher summarization accuracy and higher readability than fine-tuned BART, but lower completeness under zero-shot evaluation.GPT-3.5 and GPT-4 achieved accuracy scores of 4.79 and 4.83, versus BART's 4.66.

Discussions

Across BioNLP benchmarks, fine-tuning generally outperformed zero- and few-shot LLMs, while closed-source models were strongest for reasoning-intensive tasks. The study also identifies evaluation, reliability, coverage, and cost constraints that shape practical deployment recommendations.

  • Performance across applications: SOTA fine-tuning achieved the best performance in 10 of 12 BioNLP benchmarks, especially for named entity recognition and relation extraction.These tasks require limited reasoning and extract information directly from sentence-level inputs.
  • Performance across applications: GPT-4 outperformed previous fine-tuned SOTA approaches in MedQA and PubMedQA using zero- or few-shot learning.Closed-source LLMs also showed competitive readability and accuracy for text summarization and simplification under manual evaluation.
  • Open-source models: LLaMA 2 zero-shot responses contained inconsistencies, missing elements, and hallucinations in up to 30% of testing-set instances.Fine-tuning remained necessary to bridge its performance gap with closed-source LLMs.
  • Open-source models: PMC LLaMA showed no significant performance improvement over LLaMA 2 across all 12 benchmarks despite continuous biomedical pretraining.The authors therefore call for a more effective and sustainable approach to developing biomedical domain-specific LLMs.
  • Evaluation and reliability: Automatic summarization metrics favored fine-tuned BART, whereas manual evaluations found GPT-3.5 and GPT-4 more readable with competitive accuracy.Completeness remained a primary limitation for GPT models in biomedical text generation.
  • Practical deployment: GPT-4 delivered the highest overall performance but cost 60 to 100 times more than GPT-3.5, illustrating a practical performance-cost trade-off.GPT-4's advantage was particularly pronounced in reasoning-related question answering, including over 20% improvement in MedQA versus GPT-3.5.

Methods

The study benchmarked LLMs across 12 biomedical datasets spanning six applications, comparing prompting and fine-tuning settings with established task-specific baselines. It also evaluated generated-text quality and reproducibility-oriented implementation choices.

  • Benchmark design: The benchmark covered 12 datasets across named entity recognition, relation extraction, document classification, question answering, summarization, and text simplification.Datasets included BC5CDR-chemical, NCBI-disease, ChemProt, DDI2013, HoC, LitCovid, MedQA, PubMedQA, PubMed Text Summarization, MS^2, Cochrane PLS, and PLOS Text Simplification.
  • Evaluation metrics: Named entity recognition used exact-match entity-level F1, while the benchmark included task-specific metrics across all 12 testing sets.NER identified chemical and disease entities in biomedical text using manually annotated datasets.
  • Baselines: SOTA baselines used fine-tuned biomedical BERT models for extraction and classification and BART models for summarization and simplification.The reported SOTA results represented pre-LLM best performance and required scalable manually labeled instances.
  • Model evaluation: The study compared GPT-3.5, GPT-4, LLaMA 2, and PMC LLaMA using zero-shot, static one- and five-shot, dynamic K-nearest few-shot, and fine-tuning settings.Dynamic prompting selected one, two, or five training instances most similar to each test instance using text embeddings and cosine similarity.
  • Qualitative evaluation: Qualitative summarization evaluation rated accuracy, completeness, and readability from 1 to 5 on 50 random PubMed samples.Two residents evaluated fine-tuned BART, GPT-3.5 zero-shot, GPT-4 zero-shot, and LLaMA 2 13B zero-shot outputs with model identities concealed.

Data Availability

The study reports that all supporting data, including source data, are publicly accessible through the article and supplementary information, with additional requests handled by the authors.

  • Public access: All data supporting the findings, including source data, are available in the article and Supplementary Information through a public Zenodo record.Additional data or requests can be obtained from the corresponding authors upon request.

Competing Interests Statement

Two authors report research-related financial interests at Melax Technologies Inc., while the remaining authors declare no competing interests.

  • Disclosures: Dr. Jingcheng Du and Dr. Hua Xu report research-related financial interests at Melax Technologies Inc.The remaining authors declare no competing interests.

Tables

The tables organize prior-study comparisons, benchmark datasets and metrics, quantitative model results, error examples, and qualitative summarization evaluations. Together, they document both performance and output-quality assessments.

  • Table 1: Table 1 compares biomedical and clinical LLM studies by domain, evaluated models, task scope, evaluation measures, and data or code availability.The comparison covers extractive and generative evaluation settings as of March 2024.
  • Table 2: Table 2 lists the 12 evaluation datasets, their sizes, and associated metrics across six BioNLP applications.It notes that MS^2 testing gold standards were unavailable, so the validation set was used instead.
  • Table 3: Table 3 compares LLM zero-shot, few-shot, and fine-tuned results with reported SOTA fine-tuning baselines across the 12 benchmarks.It marks the highest performance, statistical significance, and methodological details such as bootstrapping and Wilcoxon testing.
  • Table 4: Table 4 provides annotated examples of inconsistent, missing, and hallucinated model responses.Square brackets identify explanations, while unnecessary detail is omitted.
  • Table 5: Table 5 reports accuracy, completeness, and readability scores from 1 to 5 for four models evaluated on 50 PubMed summarization samples.The compared systems are fine-tuned BART, GPT-3.5 zero-shot, GPT-4 zero-shot, and LLaMA 2 zero-shot.

Figure Legends/Captions

The figures present dynamic few-shot performance and cost comparisons, qualitative error categories, qualitative summarization assessments, and recommendations for BioNLP use.

  • Figure 1: Figure 1 shows dynamic K-nearest few-shot results for K = 1, 2, and 5 in line charts, with costs per 100 instances in bar charts.Dynamic K-nearest few-shot selects the K closest training instances for each test instance; static one-shot performance is shown for comparison.
  • Figure 2: Figure 2 categorizes qualitative errors as inconsistency, missing information, hallucinations, wrong entities, and boundary issues across BioNLP benchmarks.The figure includes NCBI Disease entity errors and qualitative evaluations on ChemProt, HoC, and MedQA.
  • Figure 3: Figure 3 evaluates accuracy, completeness, and readability for four models using 50 random PubMed Text Summarization test instances on a 1-to-5 scale.Its comparison panels report winning, tying, and losing cases for GPT-4 zero-shot against GPT-3.5 zero-shot and fine-tuned BART.
  • Figure 4: Figure 4 presents recommendations for using LLMs in BioNLP applications.
Loading 2305.16326v5…