PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks

Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, Jimmy Huang

arXiv:2310.04270v3cs.CLcs.AI

TL;DR

Biomedical LLM capabilities had not been comprehensively evaluated despite scarce annotated datasets and the importance of biomedical text processing. The paper evaluates four LLMs across six tasks and 26 datasets, finding that zero-shot models can outperform fine-tuned biomedical baselines on smaller-data tasks but generally lag on larger-data tasks.

  • Problem

    The paper addresses the lack of comprehensive evidence about LLM performance in biomedical tasks, where annotated datasets are scarce and rigorous evaluation is important.

  • Method

    The study evaluates four LLMs across six benchmark biomedical tasks and 26 datasets, using task-specific prompts and zero-shot experiments.

  • Results

    Zero-shot LLMs consistently outperform fine-tuned baselines on tasks with smaller training sets but usually fail to outperform fine-tuned biomedical models on datasets with large training data.

  • Takeaways & Limitations

    LLMs may be useful for biomedical tasks lacking large annotated datasets, although performance varies across datasets and tasks and prompt variations noticeably affect results.

  • Takeaways & Limitations

    Few-shot performance depends on high-quality task examples, and constructing better examples remains an open direction for biomedical LLM experiments.

Abstract

from arXiv · show

Recently, Large Language Models (LLM) have demonstrated impressive capability to solve a wide range of tasks. However, despite their success across various tasks, no prior work has investigated their capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of LLMs on benchmark biomedical tasks. For this purpose, we conduct a comprehensive evaluation of 4 popular LLMs in 6 diverse biomedical tasks across 26 datasets. To the best of our knowledge, this is the first work that conducts an extensive evaluation and comparison of various LLMs in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot LLMs even outperform the current state-of-the-art fine-tuned biomedical models. This suggests that pretraining on large text corpora makes LLMs quite specialized even in the biomedical domain. We also find that not a single LLM can outperform other LLMs in all tasks, with the performance of different LLMs may vary depending on the task. While their performance is still quite poor in comparison to the biomedical models that were fine-tuned on large training sets, our findings demonstrate that LLMs have the potential to be a valuable tool for various biomedical tasks that lack large annotated data.

1 Introduction

The paper addresses the limited evaluation of LLMs for biomedical text processing, where annotated datasets are scarce. It evaluates zero-shot LLM capabilities and studies prompt construction for biomedical tasks.

  • Biomedical applications often lack large annotated datasets, making zero-shot learning a potentially useful alternative to task-specific fine-tuning.Fine-tuning specialized biomedical models requires substantial task-specific annotated data, whereas zero-shot learning uses no explicit task examples.
  • The study evaluates LLMs across benchmark biomedical tasks to address the lack of comprehensive evidence in this domain.
  • The evaluation covers common biomedical research tasks, including classification, named entity recognition, relation extraction, summarization, and question answering.
  • The paper constructs task-specific prompts by accounting for the complex linguistic structure of biomedical texts and prompt sensitivity.
  • The study aims to clarify LLM capabilities and limitations for biomedical text processing and information retrieval before their use in medicine and healthcare.
  • The authors plan to release preprocessing and parsing code, prompts, and LLM-generated responses for future research.

2 Related Work

Prior biomedical NLP work established strong task-specific models through domain-specific pretraining and fine-tuning, while newer general LLMs showed broad few-shot and instruction-following capabilities. The paper identifies limited investigation of these LLMs in biomedical tasks.

  • Earlier biomedical research spans text processing alongside biomedical imaging, genomics, diagnosis, drug discovery, cancer research, and vaccine development.
  • Domain-specific models such as BioBERT, BioBART, and BioGPT achieved strong biomedical NLP results through biomedical pretraining followed by task-specific fine-tuning.Their use is constrained by the need for large annotated datasets tailored to each task.
  • Decoder-based LLMs such as GPT-3, InstructGPT, PaLM, Claude, and LLaMA demonstrated few-shot or zero-shot capabilities across general NLP tasks.
  • Despite strong general NLP performance, these LLMs had not yet been investigated comprehensively in the biomedical domain.

3 Biomedical Tasks Description

The paper describes six biomedical text-processing tasks that transform biomedical text into extracted entities, linked concepts, relations, classifications, answers, or summaries. Examples illustrate the expected outputs for each task.

  • Biomedical Named Entity Recognition: Biomedical named entity recognition extracts genes, proteins, diseases, chemicals, treatments, and drugs from biomedical literature.The example identifies a rare form of cancer, chemotherapy, and Taxol as entities.
  • Biomedical Relation Extraction: Biomedical relation extraction identifies relations among entities, such as treatments for diseases, drug use in treatments, and drug-drug interactions.The example links chemotherapy to a rare form of cancer and Taxol to chemotherapy.
  • Biomedical Entity Linking: Biomedical entity linking connects named entities in unstructured text to corresponding definitions, knowledge-base entries, or ontologies.The example links Taxol to a corresponding external drug entry.
  • Biomedical Text Classification: Biomedical text classification assigns a text to a specific category, such as a hallmark-of-cancer taxonomy label.The provided example is assigned to Genomic Instability and Mutation.
  • Biomedical Question Answering: Biomedical question answering retrieves relevant answers from biomedical literature, medical records, or clinical trials.The example selects regular blood transfusions as the relevant answer for a question about thalassemia patients.
  • Biomedical Text Summarization: Biomedical text summarization generates concise summaries of biomedical documents to reduce review time for healthcare-related text.The example condenses a patient history, diagnosis of atrial fibrillation, hospitalization, and treatment.

4 Methodology

The methodology evaluates four LLMs across six biomedical tasks and 26 datasets by constructing task-specific prompts, generating responses, and applying task-appropriate evaluation. It covers NER, relation extraction, entity linking, text classification, question answering, and summarization, with sample prompts provided for the tasks.

  • 4.1 Prompt Design: For each test sample, the method concatenates task instruction T with sample X to form prompt P, which produces response R.This defines the core prompt-generation procedure used before task-specific evaluation.
  • The study covers biomedical NER, relation extraction, entity linking, text classification, question answering, and text summarization.The tasks use datasets including BC2GM and JNLPBA for NER, BC5CDR and KD-DTI for relation extraction, BC5CDR and NCBI for entity linking, HoC and LitCovid for classification, PubMedQA and MEDIQA-2019 for question answering, and multiple summarization datasets.
  • 4.1 Prompt Design: NER prompts request BIO-format entity identification, while relation-extraction prompts are tailored to chemical-disease, drug-target, and drug-drug relations.The NER prompts include a BIO-format description, and relation-extraction prompts depend on each dataset.
  • 4.1 Prompt Design: Entity-linking prompts ask LLMs to generate correct concept names for biomedical entities, evaluated on disease, chemical, and clinical-term datasets.The datasets include BC5CDR, NCBI, and COMETA, with sample prompts shown in Table 3.
  • 4.1 Prompt Design: Question-answering prompts use task-specific inputs for PubMedQA and MEDIQA-2019, while summarization prompts cover healthcare questions, medical answers, and doctor-patient dialogue.PubMedQA supplies a question, context, and answer for yes/no/maybe classification; the summarization datasets span several biomedical summarization settings.
  • Four LLMs are evaluated across six biomedical tasks and 26 datasets using a pipeline that constructs prompts, generates responses, and applies task-specific evaluation.The overview describes prompt construction for each dataset, response generation with the respective LLMs, and evaluation techniques selected by task.

5.1 Evaluation Metrics

The evaluation uses standard literature metrics selected for each biomedical task to support fair comparisons among LLMs and prior state-of-the-art results.

  • Different evaluation metrics are selected for different tasks to ensure fair comparison with prior state-of-the-art results.
  • Precision, Recall, and F1 evaluate relation extraction and named entity recognition, while Recall@1 evaluates entity linking.
  • ROUGE and BERTScore are used for biomedical summarization.

5.2 Baselines

Zero-shot LLMs are compared against task-specific fine-tuned biomedical baselines, including BioGPT, BioBART, BioBERT, disease-knowledge-infused ALBERT, FLAN-T5-XL, and PRIMERA.

  • The study uses current state-of-the-art fine-tuned models as baselines for comparing zero-shot LLM performance.
  • Fine-tuned BioGPT serves as the baseline for relation extraction, HoC classification, and PubMedQA question answering.BioGPT was additionally fine-tuned on more than 270K PubMedQA instances.
  • Fine-tuned BioBART is used as the baseline for entity linking and selected biomedical summarization datasets.Its biomedical pretraining uses PubMed abstracts, and the supplied passage identifies it as the entity-linking baseline and a summarization baseline.
  • Fine-tuned BioBERT provides the baseline for all biomedical NER datasets and the LitCovid classification dataset.BioBERT was additionally pretrained on PubMed and PMC abstracts and achieved state-of-the-art performance across biomedical NER datasets.
  • Disease-knowledge-infused ALBERT, FLAN-T5-XL, and PRIMERA serve as baselines for MediQA-2019, biomedical lay summarization, and readability-controlled summarization, respectively.Each baseline is selected for the task where the cited passage identifies it as state of the art.

5.3 Results

Across biomedical tasks, zero-shot LLM performance varied substantially by task and dataset: some models exceeded fine-tuned biomedical baselines on data-scarce settings, while others remained far behind established systems.

  • Relation Extraction: 17.61% improvement: zero-shot PaLM-2 exceeded fine-tuned BioGPT in BC5CDR relation-extraction F1.PaLM-2, Claude-2, and LLaMA-2 all outperformed BioGPT on BC5CDR F1, while results varied across KD-DTI and DDI.
  • Text Classification: Zero-shot LLMs were much weaker than fine-tuned baselines for text classification, with PaLM-2 best among LLMs and Claude-2 worst.The PaLM-2–Claude-2 difference was statistically significant at p ≤.05.
  • Question Answering: GPT-3.5 achieved 73.26 accuracy on MediQA-2019, approaching ALBERT’s 79.49 state-of-the-art result, whereas PubMedQA LLMs stayed below BioGPT.On PubMedQA, LLaMA-2 was the strongest LLM, and closed-source models remained below 60% accuracy.
  • Entity Linking: Claude-2 led all three entity-linking datasets, yet remained significantly below fine-tuned state-of-the-art models.Claude-2 outperformed the other evaluated LLMs on BC5CDR, Cometa, and NCBI entity linking.
  • NER and Prompting: Claude-2 led NER across datasets, but all LLMs were significantly below state-of-the-art results; prompt detail also materially affected classification F1.Removing class definitions reduced HoC F1 to 46.93, while omitting the ten class names reduced it to 38.20.
  • Summarization: Zero-shot GPT-3.5 outperformed BioBART on MEDIQA-ANS and MEDIQA-MAS, while LLMs remained weaker on summarization datasets with dedicated training sets.LLMs were also on par with BioBART on MEDIQA-QS, and GPT-3.5 was generally the strongest LLM in these datasets.

5.4 Analysis

The analysis examines prompt variation, few-shot learning, fine-tuning, and possible data contamination in biomedical LLM evaluation. Descriptive prompts improve results, few-shot examples often do not, and contamination signals vary across tasks and datasets.

  • Effects of Prompt Variations: 59.26 F1 with explicit HoC class definitions falls to 38.20 without class names and 46.93 with names but no definitions.The findings support using more descriptive prompts for biomedical classification tasks.
  • Effects of Few-Shot Learning: Few-shot examples do not improve performance in most biomedical tasks, although their quality may affect performance and prediction bias.The experiments use Claude-2 because it supports contexts up to 100k tokens; Recall decreases versus zero-shot learning in all relation extraction datasets.
  • Effects of Fine-Tuning: Fine-tuning experiments use LLaMA-2-7B-Chat on smaller datasets because several closed-source models do not support fine-tuning and GPT-3.5 fine-tuning increases inference cost.The selected model is trained for 3 epochs with learning rate 2e −5.
  • Data Contamination: Task-example extraction finds no contamination signal in NER datasets, while extraction is possible for KD-DTI and DDI relation-extraction datasets.The stronger-performing PaLM-2 on KD-DTI and GPT-3.5 and Claude-2 on DDI could extract task examples.
  • Data Contamination: Membership inference indicates possible contamination for PaLM-2 in 4 of 10 summarization datasets, but no evaluated LLM beats state-of-the-art models on those datasets.HealthcareMagic and MeQSum are reported as contaminated for all four LLMs, while none of the 2023-released datasets shows contamination under this analysis.

6 Conclusions and Future Work

The paper evaluates four LLMs across six biomedical tasks and 26 datasets, finding that zero-shot performance depends strongly on training-set size. LLM performance varies by task, while fine-tuning and prompt design can substantially improve results.

  • Conclusions: Zero-shot LLMs usually fail to outperform fine-tuned biomedical models on datasets with large training sets but consistently outperform them when fine-tuned baselines use smaller training sets.The comparison includes biomedical models such as BioBERT, BioGPT, and BioBART.
  • Conclusions: Fine-tuning significantly improves LLM performance even when only smaller training sets are available.This result contrasts with the weaker zero-shot performance against task-specific biomedical models on larger-data datasets.
  • Conclusions: No single LLM outperforms the others across all datasets and tasks, and prompt variations produce noticeable performance differences.The authors propose that these findings can guide future research and real-world biomedical system development.
  • Future Work: Future work will extend evaluation to medical coding, drug design, healthcare, protein sequences, low-resource languages, and open-domain information retrieval.These areas are identified as additional biomedical or information-retrieval settings for evaluating LLMs.
Loading 2310.04270v3…