Source-linked AI summary
BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine
Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, Zaiqing Nie
TL;DR
Biomedical foundation models have limited access to specialized knowledge and face modality gaps between biological data and natural language. BioMedGPT aligns molecules, proteins, and biomedical text through a generative language model, and BioMedGPT-10B performs competitively on biomedical QA while supporting molecule and protein QA. The models and alignment datasets are open-sourced, although the authors caution that biomedical LLM outputs require professional supervision and broader evaluation.
Problem
General-purpose foundation models have restricted biomedical knowledge and struggle to comprehend diverse biological modalities that differ from natural language.
Method
BioMedGPT aligns molecular structures, protein sequences, and biomedical text in a unified feature space using BioMedGPT-LM and independent modality encoders.
Results
BioMedGPT-10B outperforms or matches human and significantly larger general-purpose models on biomedical QA and shows promising molecule and protein QA performance.
Takeaways & Limitations
Open-sourced BioMedGPT models and curated alignment datasets enable free-text interaction with biological modalities for biomedical applications.
Takeaways & Limitations
Biomedical LLM outputs may contain bias, discrimination, or harmful content and should not be used in life-endangering research without professional supervision and comprehensive experiments.
Abstract
from arXiv · showhide
Foundation models (FMs) have exhibited remarkable performance across a wide range of downstream tasks in many domains. Nevertheless, general-purpose FMs often face challenges when confronted with domain-specific problems, due to their limited access to the proprietary training data in a particular domain. In biomedicine, there are various biological modalities, such as molecules, proteins, and cells, which are encoded by the language of life and exhibit significant modality gaps with human natural language. In this paper, we introduce BioMedGPT, an open multimodal generative pre-trained transformer (GPT) for biomedicine, to bridge the gap between the language of life and human natural language. BioMedGPT allows users to easily ``communicate'' with diverse biological modalities through free text, which is the first of its kind. BioMedGPT aligns different biological modalities with natural language via a large generative language model, namely, BioMedGPT-LM. We publish BioMedGPT-10B, which unifies the feature spaces of molecules, proteins, and natural language via encoding and alignment. Through fine-tuning, BioMedGPT-10B outperforms or is on par with human and significantly larger general-purpose foundation models on the biomedical QA task. It also demonstrates promising performance in the molecule QA and protein QA tasks, which could greatly accelerate the discovery of new drugs and therapeutic targets. In addition, BioMedGPT-LM-7B is the first large generative language model based on Llama2 in the biomedical domain, therefore is commercial friendly. Both BioMedGPT-10B and BioMedGPT-LM-7B are open-sourced to the research community. In addition, we publish the datasets that are meticulously curated for the alignment of multi-modalities, i.e., PubChemQA and UniProtQA. All the models, codes, and datasets are available at \url{https://github.com/PharMolix/OpenBioMed}.
1 Introduction
General-purpose foundation models have limited depth in specialized biomedical domains and struggle to connect biological modalities with human language. BioMedGPT addresses this gap by aligning molecules, proteins, and biomedical knowledge with natural language through generative language models.
- General-purpose foundation models have restricted domain knowledge because they lack access to proprietary training data.
- BioMedGPT-10B unifies texts, molecular structures, and protein sequences, enabling natural-language queries over biological data.
- BioMedGPT bridges the language of life and human natural language using large-scale generative language models.
- Through fine-tuning, BioMedGPT-10B outperforms or matches human and much larger general-purpose models on biomedical QA and also performs promisingly on molecule and protein QA.
- BioMedGPT-LM-7B is an open, commercially friendly Llama2-based biomedical language model, accompanied by the PubChemQA and UniProtQA datasets.
2 An Overview of BioMedGPT
BioMedGPT is presented as a biomedical brain that connects natural language with diverse biological modalities. It combines multimodal encoders, neural alignment, and heterogeneous expert knowledge to support cross-modal understanding.
- BioMedGPT acts as a biomedical brain that connects natural language with molecules, proteins, transcriptomic data, and other modalities.
- The framework incorporates knowledge graphs, text documents, and experimental results to enhance biomedical knowledge.
- A feature-fusion approach uses independent pretrained encoders and neural alignment and translation methods to map multimodal data into the natural-language feature space.
- BioMedGPT supports inquiries in varied formats and strengthens connections between natural language and biological modalities.
LLaMA 2
This section introduces BioMedGPT-10B as a multimodal biomedical model that aligns molecular graphs, protein sequences, and natural language.
- BioMedGPT-10B aligns 2D molecular graphs, protein sequences, and natural language in a unified feature space.
- BioMedGPT-LM serves as the large-language-model component for jointly comprehending biological modalities through natural language.
- The model uses GraphMVP for molecular graphs, ESM2-3B for protein sequences, and a neural-network adaptor for feature-space alignment.
3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language
BioMedGPT-10B is built by incrementally training a biomedical language model and aligning molecular and protein representations with natural language. The alignment uses dedicated encoders, adaptors, curated question-answer datasets, and multimodal fine-tuning.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: BioMedGPT-10B aligns 2D molecular graphs, protein sequences, and natural language in a unified feature space.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: The language model is incrementally trained on extensive biomedical documents from S2ORC after selecting and refining biomedical articles.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: The model combines a GraphMVP-initialized molecule encoder, ESM-2-3B protein encoder, BioMedGPT-LM-7B, and independent modality adaptors.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: Multimodal fine-tuning trains the system to answer questions about given molecules or proteins using aligned features as language-model inputs.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: PubChemQA and UniProtQA provide the two large-scale datasets used for multimodal alignment and are publicly released for future research.
- 3 BioMedGPT-10B: Aligning Molecules, Proteins, and Natural Language: During multimodal fine-tuning, BioMedGPT-LM parameters are frozen while the molecule encoder, protein encoder, and modality adaptors are optimized.
4 Experiment
BioMedGPT-10B is evaluated on biomedical, molecule, and protein question answering to test joint modeling of biological and natural-language modalities. It achieves strong biomedical QA results and benefits molecule and protein QA from multimodal alignment.
- Biomedical QA: BioMedGPT-10B achieves state-of-the-art results on MedMCQA and PubMedQA, with PubMedQA accuracy on par with human experts.The model is fine-tuned on PubMedQA and MedMCQA and evaluated both in-domain and out-of-domain.
- Biomedical QA: BioMedGPT-10B surpasses the best same-size model on out-of-domain USMLE by 5.1% and exceeds 50% accuracy.The best same-parameter baseline is Llama2-Chat; only ChatGPT also exceeds 50% accuracy.
- Molecule QA: BioMedGPT-10B substantially outperforms ChatGPT and Llama2-7B-Chat on molecule QA, highlighting the importance of multimodal alignment and translation.Evaluation uses BLEU, ROUGE, and METEOR on ChEBI-20 molecule descriptions.
- Molecule QA: Cross-modal alignment lets BioMedGPT-10B comprehend molecular semantics and translate molecular language into human-readable descriptions.This provides a tool for experts and non-experts to access fundamental molecular knowledge.
- Protein QA: Protein QA requires jointly decoding protein properties and functions from amino acid sequences while understanding natural-language questions.The evaluation uses UniProtQA and compares raw-sequence prompting with aligned Llama2-Chat and BioMedGPT-10B.
- Protein QA: BioMedGPT-10B gains significantly over aligned Llama2-7B-Chat, especially on BLEU, while producing more precise protein function descriptions.Qualitative examples associate its outputs with serine/threonine phosphorylation and thymine nucleotide metabolism.
- Protein QA: Aligned models generate reasonable protein annotations, whereas the original Llama2-7B-Chat requests more information about the protein.The examples support faster and more accurate protein function annotation within the stated scope.
5 Limitations
BioMedGPT faces limitations in evaluation, interpretability, and safety that constrain reliable biomedical use.
- Language-centric BLEU and ROUGE measures may be insufficient for evaluating effectiveness in complex biomedical applications.The paper calls for more nuanced metrics and domain-expert evaluation of accuracy, relevance, and real-world aptness.
- Limited interpretability can hamper the soundness of scientific insights generated by complex black-box models.
- Probabilistic outputs can produce unforeseen bias, discrimination, or harmful content, so real-world applications require cautious professional supervision.The authors advise against research scenarios that endanger human life and state that safety risks cannot be fully eliminated.
6 Conclusions and Future Work
BioMedGPT is presented as a multimodal biomedical foundation model that bridges biological modalities and human language. Its open models align molecules, proteins, and natural language while supporting free-text interaction and future work on evaluation, interpretability, and safety.
- BioMedGPT is positioned as a pioneering multimodal foundation model for biomedicine that could deepen understanding of fundamental life codes.
- BioMedGPT-10B aligns molecules, proteins, and natural language, enabling free-text interaction with diverse biomedical modalities.The model is intended to support downstream chemistry and biomedicine applications, including discovery of novel molecules and therapeutic targets.
- The authors open-source BioMedGPT-10B and BioMedGPT-LM-7B and plan stronger evaluation metrics, interpretability, and safety measures.