Source-linked AI summary
BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, Christopher D. Manning
TL;DR
Large biomedical language models can be costly, privacy-sensitive, difficult to fine-tune, and opaque in their training data. This paper introduces and releases BioMedLM, a 2.7 billion parameter GPT-style model trained on PubMed text. Fine-tuned BioMedLM achieves competitive biomedical question-answering results and produces useful medical answers, while hallucinated numerical values remain a limitation.
Problem
Large biomedical models have high computational and privacy costs, restricted fine-tuning, and unknown training data, motivating smaller targeted alternatives.
Method
The paper builds BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles.
Results
BioMedLM competes with substantially larger models on biomedical question answering and produces useful multi-sentence answers to medical knowledge questions.
Takeaways & Limitations
Medium-sized models trained on domain-specific text can serve as open foundations for biomedical NLP applications with documented training data and local deployment options.
Takeaways & Limitations
BioMedLM can hallucinate, especially numerical values, so patient-facing applications require additional safeguards.
Abstract
from arXiv · showhide
Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.
1 Introduction
Large general language models offer strong biomedical performance but remain costly, closed, privacy-sensitive, and difficult to adapt. BioMedLM addresses these constraints with a smaller, domain-specific, openly released model that achieves competitive biomedical question-answering results.
- Large models such as GPT-4 and Med-PaLM 2 set high standards for biomedical question answering and information extraction.
- Training and using large language models incurs high financial and environmental costs, while API fees can exceed organizational budgets.Compute for training and inference has increased 10- to 100-fold since 2015.
- Remote access to large closed models requires sensitive data transmission and creates concerns about third-party access to personal information.These constraints conflict with medical privacy requirements.
- Unknown training data limits practitioners’ confidence in response quality and researchers’ ability to study links between training data and downstream performance.
- Closed models cannot be further fine-tuned for specialized tasks, despite the value of biomedical task-specific training.
- BioMedLM is a 2.7 billion parameter GPT-2-style autoregressive model trained exclusively on PubMed abstracts and full articles.Its biomedical-specific design targets domain performance with a smaller model.
- 57.4% on MedMCQA (dev) and 70.0% on MMLU Medical Genetics show BioMedLM’s strong multiple-choice biomedical question-answering performance.It also improves accuracy over the similarly sized GPT-Neo 2.7B baseline across BioASQ, PubMedQA, and MedQA.
- BioMedLM’s small size supports single-GPU fine-tuning and laptop inference, while its documented training data and open release support internal, transparent use.
2 Related Work
Related work spans large general models adapted to biomedicine, smaller biomedical models, scientific language models, and efforts toward open model and dataset construction. These studies motivate comparing domain specialization, architecture, scale, and openness.
- Large general models adapted to biomedical tasks include Med-PaLM 2, while smaller domain-targeted models such as DRAGON show strong performance for their size.
- GPT-Neo 2.7B provides a same-size, similarly architected baseline for evaluating how pre-training corpus composition affects downstream performance.It was trained on the diverse 825 GB Pile, which includes PubMed and other specialized sources.
- PubMedBERT demonstrates the potential of exclusive PubMed training through improvements over comparable biomedical models on several NLP tasks.
- BioLinkBERT and DRAGON augment PubMed-based modeling with link structure and biomedical knowledge graphs for biomedical NLP tasks.
- Galactica applies a scientific-text tokenizer and large-scale training to scientific and biomedical tasks, including PubMedQA.The model has 120 billion parameters.
- BioGPT is a concurrent PubMed-trained GPT-style model, whereas this work emphasizes question-answering rather than relation extraction and summarization.
- Flan-PaLM and Med-PaLM are fine-tuned versions of the 500 billion parameter PaLM model, illustrating the capabilities of large general-language models.
- GPT-3.5 and GPT-4 evaluations report strong performance on USMLE and biomedical benchmarks, with GPT-4 exceeding earlier GPT-3.5 and Med-PaLM results.
3 Model Design And Training
BioMedLM combines a 2.7-billion-parameter Transformer with a biomedical tokenizer trained on PubMed and extensive PubMed pre-training. Fine-tuning uses task-specific formats and classifiers for biomedical question answering.
- 3.1 Model Architecture: BioMedLM is an autoregressive, decoder-only Transformer with 2.7 billion parameters, similar in architecture to GPT-2.
- 3.2 BioMedLM Tokenizer: Its custom BPE tokenizer was trained on PubMed abstracts and represents many common biomedical terms as single tokens.For example, it preserves “thrombin” as one token instead of splitting it into “th”, “rom”, and “bin”.
- 3.2 BioMedLM Tokenizer: Using different tokenizers with otherwise identical settings produced meaningful MedQA improvements across five random seeds.
- 3.3 Pre-training: Initial 1.5-billion-parameter runs diverged under fp16/fp32 mixed precision, but switching to bf16 resolved the divergence issues.The final 2.7-billion-parameter run did not suffer from divergences.
- 3.3 Pre-training: Training continued for 300 billion tokens because validation and training perplexity improved steadily, while preliminary experiments showed improved downstream performance.The paper notes that this required dramatically more passes through the data than comparable models.
- 3.4 Fine-tuning: Fine-tuning applies task-specific formats, including multiple-choice classifiers for MedMCQA and MedQA and sequence classification for PubMedQA and BioASQ.For BioASQ and PubMedQA, performance improved when prompts placed context before the question using special tokens.
4 Biomedical Question-Answering Results
BioMedLM was evaluated on five biomedical question-answering benchmarks and produced results competitive with substantially larger models. The evaluation covered multiple-choice, yes/no/maybe, and yes/no formats across clinical and biomedical questions.
- Benchmark suite: BioMedLM was evaluated on MedMCQA, MedQA, MMLU, PubMedQA, and BioASQ.These benchmarks span multiple-choice medical exams and biomedical passage-based question answering.
- MedMCQA: MedMCQA contains 182822/4183/6150 train/dev/test questions with four multiple-choice options.The questions come from AIIMS PG and NEET PG materials and cover clinical and biochemical topics.
- Reported results: Results for MedMCQA, MedQA, MMLU, PubMedQA, and BioASQ are presented in Tables 7–11.The tables report performance across the five benchmark tasks.
- MedQA: MedQA contains USMLE-derived medical scenarios with four answer options, using 10178/1272/1273 train/dev/test questions.Each question presents a medical scenario that a physician should be able to answer.
- Other benchmarks: MMLU includes biomedical exams spanning academic levels, while PubMedQA and BioASQ use passage-based questions with yes/no or yes/no/maybe answers.Results for these datasets are reported in separate tables.
5 Free Response Question Answering
The paper explores free-response medical question answering by fine-tuning BioMedLM on health question-answer pairs. The authors present example responses but note that formal generation-quality evaluation remains future work.
- Task: The free-response setting aims to produce multi-sentence medical answers rather than only select among multiple-choice options.The intended interaction is similar to consulting an expert colleague.
- Fine-tuning: BioMedLM was fine-tuned on question-answer pairs derived from FAQ pages and Wikipedia subsections.FAQ questions were added directly, while Wikipedia was converted into additional question-answer examples.
- Fine-tuning: The base BioMedLM model needs instruction-data fine-tuning to answer questions effectively.Its pretraining objective was to replicate text in PubMed articles, not directly respond to questions.
- Evaluation: The authors demonstrate performance on example questions and defer formal evaluation of generation quality to future work.The evidence is presented through examples in Table 12.
6 Analysis
Across biomedical tasks, BioMedLM achieves results comparable to larger models and substantially outperforms the similarly sized general-English GPT-Neo 2.7B on selected tasks. Free-response examples are generally accurate but still show numerical hallucinations requiring safeguards.
- Performance across tasks: BioMedLM achieves comparable results to larger models across MedMCQA, MedQA, MMLU, PubMedQA, and BioASQ.The paper frames this as performance across all evaluated question-answering tasks.
- Performance across tasks: 57.38 versus 57.6 on MedMCQA shows BioMedLM matching Flan-PaLM despite being 200x smaller.The comparison is attributed partly to MedMCQA’s substantially larger training set.
- Free-response limitations: Example medical responses are generally accurate but include vague answers and incorrect numerical values caused by persistent hallucination.The authors state that patient-facing applications would need safeguards for incorrect numerical values.
- Comparison with GPT-Neo 2.7B: BioMedLM substantially outperforms GPT-Neo 2.7B on all three selected tasks, including a 27% accuracy increase on BioASQ.GPT-Neo 2.7B has a nearly identical parameter count and similar architecture but was trained on general English.
7 Usage of BioMedLM since release
Since its December 2022 release, BioMedLM has been evaluated and applied across biomedical question answering, clinical note classification, relation extraction, and interaction extraction. Reported results vary by task and domain, with strengths in several biology-focused applications.
- Subsequent evaluations: Since December 2022, BioMedLM has been evaluated on biomedical QA and used for clinical note section classification.Reported QA evaluations include MedQA and PubMedQA with accuracies of 50.3% and 74.4%, respectively.
- Relation extraction: In protein relation tasks, BioMedLM achieved a micro F1-score of 0.643 on STRING Task 2.STRING Task 2 classifies whether any association or interaction exists between two proteins.
- Biomedical extraction: For microbiome-disease interaction extraction, BioMedLM achieved precision of 0.822 and accuracy of 0.806.GPT-3 had higher accuracy at 0.814, while BioGPT’s accuracy was 0.732.
- Analysis: The paper describes a tension between domain-specific training helping or hindering performance across tasks.It reports better performance in biology-related tasks than clinical tasks and attributes weaker results partly to domain drift or over-optimization.
8 Conclusion
BioMedLM is presented as a 2.7 billion parameter GPT-style model trained on PubMed text, demonstrating the potential of medium-sized, domain-specific models.
- BioMedLM is a 2.7 billion parameter GPT-style model trained on PubMed text.
- BioMedLM can outperform bidirectional models with richer data sources on biomedical question-answering tasks.
- BioMedLM can compete with the few-shot performance of models with orders of magnitude more parameters.
- BioMedLM can produce multi-sentence answers to medical knowledge questions.
1 MedMCQA Example Questions
This section provides example questions for MedMCQA.
- Table A.1 presents example questions for MedMCQA.
2 MedQA Example Question
This section provides example questions for MedQA, MMLU, PubMedQA, and BioASQ.
- Table A.2 presents an example question for MedQA.
- Table A.3 presents an example question for MMLU.
- Table A.4 presents an example question for PubMedQA.
- Table A.5 presents an example question for BioASQ.