Source-linked AI summary
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, Rui Yan
TL;DR
Existing biological cross-modal models face limitations including invalid molecular generation, limited use of contextual information, and insufficient distinction between structured and unstructured knowledge. BioT5 addresses these issues with SELFIES, multi-modal pre-training over text, molecules, and proteins, and modality-specific processing. After fine-tuning, it achieves strong performance across diverse biological tasks, including state-of-the-art results on 10 of 15 tasks and 100% validity in reported text-based molecule generation.
Problem
Existing models have limitations including invalid molecular SMILES, underuse of contextual information, and equal treatment of structured and unstructured knowledge.
Method
BioT5 pre-trains a shared encoder–decoder over text, molecules, and proteins using robust SELFIES, wrapped biological text, masked reconstruction, and bidirectional sequence–text translation.
Results
BioT5 achieves state-of-the-art performance on 10 of 15 fine-tuned tasks and maintains validity of 1.0 while surpassing MolT5-Large by 32.8% in exact match for text-based molecule generation.
Takeaways & Limitations
BioT5 captures relations and properties of bio-entities by integrating structured and unstructured data across molecule, protein, and text modalities.
Takeaways & Limitations
BioT5 requires full-parameter fine-tuning for each downstream task and is limited to text, molecule, and protein modalities.
Abstract
from arXiv · showhide
Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose $\mathbf{BioT5}$, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. $\mathbf{BioT5}$ utilizes SELFIES for $100%$ robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, $\mathbf{BioT5}$ distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at $\href{https://github.com/QizhiPei/BioT5}{Github}$.
1 Introduction
BioT5 integrates molecule, protein, and text modalities to address limitations in molecular representation and contextual knowledge use. Its pre-training combines robust SELFIES representations, wrapped biological text, and bidirectional sequence–description translation, followed by broad downstream evaluation.
- Molecules and proteins can be represented as sequences, enabling Transformer-based modeling and pre-training across biological modalities.Molecules use SMILES sequences, while proteins use FASTA sequences.
- Existing cross-modal models use literature and databases to supplement sequence information with properties, experimental results, and bio-entity interactions.These resources provide information that cannot be explicitly inferred from molecular or protein sequences alone.
- BioT5 uses SELFIES for small molecules because they provide a more robust, error-tolerant representation that eliminates illegitimate molecular structures.The framework covers text, molecules, and proteins.
- BioT5 collects molecular, protein, and text data, replacing biological entity mentions in PubMed text with corresponding SELFIES or FASTA sequences.This processing creates wrapped biological text and separately tokenizes text, FASTA, and SELFIES.
- BioT5 pre-trains shared encoder–decoder components with masked-span recovery across modalities and bidirectional translation between bio-sequences and textual descriptions.Translation is applied to both molecule–text and protein–text pairs.
- BioT5 achieves state-of-the-art performance on 10 of 15 fine-tuned tasks and results comparable to domain-specific large models on the remaining 5.The tasks include property prediction, interaction prediction, captioning, and text-based molecule generation.
2 Related Work
Prior biological cross-modal models jointly model text with molecular or protein representations, but BioT5 differs through multi-task integration and SELFIES-based molecular encoding. The related representation discussion motivates robust, modality-specific tokenization.
- Cross-modal models: Cross-modal biology models combine text with molecules, proteins, or additional biological modalities using architectures such as T5, GPT, contrastive learning, and translation.Examples include MolT5, MoSu, MolXPT, ProteinDT, BioTranslator, and Galactica.
- Cross-modal models: Earlier molecule-focused models use SMILES, which the paper associates with validity issues during molecule generation.BioT5 instead uses SELFIES to address this representation problem.
- BioT5's distinction: BioT5 focuses on molecule and protein modalities while using text as a knowledge base and bridge for their underlying relations and properties.Its multi-task pre-training models connections among molecules, proteins, and text.
- Molecule representation: SMILES is a common textual molecular representation, but its limited syntactic and semantic robustness can reduce the validity of generated molecules.SELFIES is introduced as a robust alternative in the paper’s representation discussion.
- Modality-specific encoding: BioT5 also separates dictionaries and tokenization across molecule, protein, and text modalities rather than sharing one vocabulary.The paper presents this as a way to distinguish modality-specific semantic spaces.
3 BioT5
BioT5 combines multi-modal data, modality-specific tokenization, and six pre-training tasks in a T5 architecture. The design preserves chemically meaningful units while learning masked reconstruction and cross-modal translation.
- Pre-training corpus: BioT5 combines data from different modalities for multi-task pre-training using a T5-based architecture.The pre-training corpus includes single-modal data, wrapped text, and paired sequence–text data.
- Pre-training corpus: The corpus includes molecule SELFIES, protein FASTA, general text, and wrapped PubMed text containing linked molecular and protein representations.The sources include ZINC20, UniRef50, C4, and 33M PubMed articles.
- Pre-training tasks: The T5 objective reconstructs masked spans in molecule SELFIES, protein FASTA, general text, and wrapped text, while translation links sequences with structured descriptions.Figure 2 distinguishes masked-span reconstruction from bidirectional translation tasks.
- Separate tokenization and embedding: MolT5 can split the bromine symbol “Br” into “B” and “r”, causing an incorrect chemical interpretation.The example connects character-based tokenization with chemically incorrect descriptions.
- Separate tokenization and embedding: BioT5 uses separate tokenization and embedding spaces because identical tokens carry different meanings in molecules, proteins, and text.The design is intended to prevent semantic conflation across modalities.
- Separate tokenization and embedding: BioT5 tokenizes each chemically meaningful SELFIES group as a complete unit and uses separate vocabularies for molecules, proteins, and text.Protein amino acids receive a special prefix, while text retains the original T5 dictionary.
- Pre-training tasks: BioT5 pre-training comprises six tasks grouped into modality-specific masked reconstruction, wrapped-text reconstruction, and bidirectional sequence–text translation.The task groups cover molecule SELFIES, protein FASTA, general text, wrapped text, and molecule- or protein-text pairs.
4 Experiments and Results
BioT5 is evaluated across 15 downstream tasks spanning molecular and protein prediction, interaction prediction, and cross-modal generation. It generally matches or exceeds competing methods, including larger or domain-specific models, while maintaining valid molecule generation.
- 15 downstream tasks cover single-instance prediction, multi-instance prediction, and cross-modal generation.
- Molecule Property Prediction: BioT5 surpasses baselines on most MoleculeNet molecule property prediction tasks under scaffold splitting.The benchmark uses AUROC and compares against pretrained graph neural networks and language-model baselines.
- Protein Property Prediction: BioT5 outperforms all baselines for protein solubility prediction and ranks second for protein localization prediction.ProtBert and ESM-1b are two to three times larger and are pretrained on comparable or larger protein-sequence corpora.
- Drug-target Interaction Prediction: BioT5 consistently outperforms other methods on BioSNAP and BindingDB, while showing a slight advantage on the Human drug-target interaction dataset.The reported metrics include AUROC, AUPRC, and accuracy.
- Protein-protein Interaction Prediction: BioT5 outperforms almost all baselines on both Yeast and Human protein-protein interaction datasets.The comparison includes ProtBert and ESM-1b with all parameters fine-tuned.
- Cross-modal Generation: BioT5 outperforms baseline models across nearly all molecule captioning and text-based molecule generation metrics.Its Text2Mol score is 0.603 for captioning, while text-based generation exceeds MolT5-Large by 32.8% in exact match and achieves validity of 1.0.
5 Conclusions and Future Work
BioT5 integrates structured and unstructured biological data to capture relations and properties of bio-entities, with reported improvements across tasks. Future work will expand biological modalities and investigate prediction interpretability.
- BioT5 captures bio-entity relations and properties by leveraging structured and unstructured data with robust molecular representations.
- Future Work: Future work will incorporate genomics and transcriptomics data to build a more holistic biological pre-training framework.
- Future Work: The authors also plan to evaluate BioT5 prediction interpretability to provide more insight into the biological systems under study.
6 Limitations
BioT5 has limitations in downstream adaptation, modality coverage, task coverage, and representation formats.
- BioT5 requires full-parameter fine-tuning separately for each downstream task because instruction tuning did not generalize across tasks.
- Combining different tasks with instructions can cause data leakage, including overlaps between BindingDB training data and BioSNAP and Human test sets.
- The demonstrated scope covers only text, molecules, and proteins, excluding biological modalities such as DNA, RNA, and cells.
- BioT5 primarily uses sequence representations, leaving 2D and 3D biological structures for future exploration.
7 Risks
BioT5's potential pharmaceutical benefits are accompanied by misuse risks, including ineffective or harmful molecule generation and deliberate creation of dangerous molecules.
- BioT5 might fail to generate effective molecules for treating specific diseases.
- Generated compounds could have adverse side effects, creating safety risks in pharmaceutical applications.
- The model could potentially be employed to create dangerous molecules, so misuse risks should be prevented.
D Molecule-Text Generation Metrics
Molecule captioning and text-based molecule generation use the same evaluation metrics, with SELFIES converted to SMILES before metric calculation for fair comparison.
- Molecule captioning and text-based molecule generation follow the same evaluation metrics as prior work by Edwards et al. (2022).
- SELFIES representations are converted to SMILES before calculating the evaluation metrics.
- The conversion is performed to ensure a fair comparison with prior evaluations.
D.1 Molecule Captioning Metrics
Molecule captioning is evaluated with language-generation metrics and molecule-aware similarity measures. These metrics assess both textual agreement and correspondence between generated descriptions and molecules.
- BLEU, ROUGE, and METEOR measure the closeness of generated molecule descriptions to ground-truth descriptions.
- Text2Mol measures similarity between the generated description and the ground-truth molecule using pretrained models.
- Molecule-aware evaluation reports MACCS, RDK, and Morgan fingerprint similarities, plus Levenshtein distance and FCD.
- Validity is the percentage of generated SMILES that can be processed by RDKit.
E Pre-training Details
BioT5 pre-training combines molecule-text and protein-text resources with downstream evaluations spanning molecular, protein, and interaction tasks. The setup uses structured task descriptions, modality-specific inputs, and comparisons against established baselines.
- Pre-training Details: BioT5 pre-training uses molecule-text and protein-text pairs extracted from PubChem and Swiss-Prot, formatted with special tokens anchoring scientific context and structure.
- Task Formulation: Prompts encode molecule and protein inputs with SELFIES and FASTA, while task outputs include binary labels or generated descriptions and sequences.
- Molecule Property Prediction: Downstream datasets use an 8:1:1 train-validation-test split, with scaffold splitting for molecular datasets.
- Molecule Property Prediction: Molecule property tasks include BBBP, Tox21, and ClinTox, covering barrier permeability, toxicity, and clinical-trial toxicity outcomes.
- Protein Property Prediction: Protein property tasks include solubility prediction and membrane-bound versus soluble localization classification.
- Drug-Target Interaction Prediction: The DTI evaluation uses BioSNAP and BindingDB datasets and compares BioT5 with six models including SVM, Random Forest, DeepConv-DTI, GraphDTA, and MolTrans.
F.2.2 Protein-protein Interaction Prediction Datasets
The paper evaluates protein-protein interaction prediction using yeast and human datasets, with sequence-similarity-aware splitting for yeast and binary interaction prompts. Molecule captioning and text-based molecule generation use the ChEBI-20 benchmark and shared baselines.
- Protein-protein Interaction Prediction Datasets: The yeast dataset evaluates whether two yeast proteins interact, with negative pairs drawn from distinct subcellular locations.
- Protein-protein Interaction Prediction Datasets: Yeast data are split after removing redundancy by protein-sequence similarity, enabling evaluation across dissimilar protein sequences.
- Protein-protein Interaction Prediction Datasets: Protein-protein interaction prompts require a Yes or No prediction for whether two proteins interact.
- Molecule Captioning and Generation: The ChEBI-20 benchmark contains 33,010 molecule-text pairs and is split 8:1:1 for training, validation, and testing.
- Molecule Captioning and Generation: Molecule captioning compares RNN, Transformer, T5, MolT5, and other baselines, including models trained with molecular or text data.
- Molecule Captioning and Generation: Text-based molecule generation reverses captioning by mapping English molecule descriptions to molecule SELFIES.
G Case Study
Qualitative case studies show BioT5 producing more accurate and coherent molecule descriptions than comparison models in several examples. In reverse generation, SELFIES supports valid outputs, while protein-like molecules illustrate overlap between molecular and protein modeling.
- Molecule Captioning: BioT5 best matches the ground truth in a captioning example by localizing a substituent and identifying membership in pyridines and an aryl thiol.
- Molecule Captioning: BioT5 avoids MolT5’s incorrect boron description and repetitive output, producing more accurate or semantically coherent captions across examples.
- Molecule Captioning: Only BioT5 describes one complex molecule as an inhibitor of SARS coronavirus main proteinase, a result the authors associate with protein-knowledge integration.
- Text-based Molecule Generation: BioT5 more often generates molecules that exactly match the ground truth in text-based molecule generation cases.
- Text-based Molecule Generation: SELFIES prevents invalid molecule generation, especially for the complex and longer molecules shown in the cases.
- Cross-modal Interpretation: A 33-membered polypeptide example shows that molecule and protein boundaries can overlap, supporting reciprocal use of information from both modalities.