PaperlayerView exact revision on arXiv

Source-linked AI summary

BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning

Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, Rui Yan

arXiv:2402.17810v2q-bio.QMcs.AIcs.CEcs.LGq-bio.BM

TL;DR

BioT5+ addresses gaps in molecular-language understanding, task generality, and numerical processing in biological models. It combines IUPAC-aware pre-training with expanded data and multi-task instruction tuning; across 3 problem types, 15 tasks, and 21 benchmark datasets, it achieves state-of-the-art performance in most tasks.

  • Problem

    Prior biological models neglect IUPAC molecular names, often require separate task-specific models, and lack specialized numerical tokenization for regression tasks.

    Source: §1 · p. 2

  • Method

    BioT5+ extends BioT5 with IUPAC integration, expanded biological text and molecule data, multi-task instruction tuning, and numerical tokenization.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Across 3 problem types, 15 tasks, and 21 benchmark datasets, BioT5+ achieves state-of-the-art performance in most tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present BioT5+ as a tool for understanding and analyzing biological entities across a wide range of tasks.

    Source: §5 · p. 9

    BioT5+ still faces challenges generalizing across biological tasks, has limited scale, and does not integrate other modalities such as images.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including \emph{3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets}, demonstrating the remarkable performance and state-of-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at \url{https://github.com/QizhiPei/BioT5}.

1 Introduction

The introduction identifies gaps in molecular-language understanding, task generality, and numerical processing, then presents BioT5+ as an extension designed to address them. It reports state-of-the-art performance in most of its evaluated tasks.

1 Introduction

  • BioT5+ integrates IUPAC names with molecular representations to connect chemical names used in literature with formal representations such as SELFIES.The paper presents this integration as supporting molecular understanding and property or activity prediction.Source: §1 · p. 2
  • Compared with BioT5, BioT5+ expands its corpus with bio-text from bioRxiv and PubMed and molecular data from PubChem.The paper says this expansion broadens the model’s knowledge base and contextual understanding of biological entities.Source: §1 · p. 2
  • BioT5+ uses multi-task instruction tuning instead of training a separate specialized model for each downstream task.The authors describe the unified framework as integrating knowledge across tasks and improving generalization across biological and chemical domains.Source: §1 · p. 2
  • BioT5+ adopts character-based numerical tokenization to represent numerical values more consistently.The introduction describes this technique as drawing inspiration from Llama.Source: §1 · p. 2
  • Across 3 problem types, 15 tasks, and 21 benchmark datasets, BioT5+ achieves state-of-the-art performance in most tasks.The benchmarks include molecule property prediction, retrosynthesis, molecule description generation, and drug-target interaction.Source: §1 · p. 2

2 Related Work

Related work covers cross-modal modeling of text with molecules and proteins, as well as instruction tuning for biological tasks. It contrasts these approaches with BioT5+, which is pre-trained for biology and instruction-tuned to generalize across biological tasks.

2.1 Biological Cross-modal Models

2.2 Instruction Tuning for Biological Tasks

  • Unlike Mol-Instructions [Fang et al., 2023] and InstructMol [Cao et al., 2023], BioT5+ is pre-trained for biology and instruction-tuned to generalize across biological tasks.Instruction tuning trains pre-trained LLMs with specialized instruction datasets to support understanding of task-specific instructions.Source: §2.2 · p. 3

3 BioT5+ Framework

BioT5+ extends BioT5 with IUPAC integration, an expanded pre-training corpus, specialized tokenization, and multi-task pre-training and instruction-based fine-tuning.

3.1 Intuition for IUPAC Integration

  • IUPAC names describe molecular structures through standardized chemical nomenclature and resemble natural language used in scientific literature.The paper’s example, “2-acetyloxybenzoic acid,” illustrates how name components describe structural features.Source: §3.1 · p. 3

3.2 Pre-training Corpus

  • The pre-training corpus has four classes: single-modal data, entity-wrapped text, bio-text, and molecule- or protein-description pairs; downstream Mol-Instructions and ChEBI-20 entities are excluded to prevent leakage.For wrapped text, BERN2-detected entities are retained at higher confidence; molecule names are standardized to IUPAC before SELFIES is appended.Source: §3.2 · p. 4

3.3 Tokenization

  • BioT5+ tokenizes SELFIES atom groups individually, prefixes each FASTA amino-acid token with <p>, and represents each numerical digit as an individual token.Character-based digit tokenization is intended to avoid inconsistent number segmentation while retaining the original dictionary.Source: §3.3 · p. 4

3.4 Model and Training

  • BioT5+ uses eight pre-training tasks across four categories, then applies multi-task instruction-based fine-tuning across grouped downstream tasks.Its architecture follows BioT5’s T5-v1.1-base configuration with a 35,076-token vocabulary and 252M parameters; the shared tuning approach avoids repeated task-specific tuning and eases evaluation deployment.Source: §3.4 · pp. 4–5

4 Experiments and Results

BioT5+ is evaluated across 21 benchmark datasets spanning molecule- and protein-oriented tasks. Results cover property prediction, reaction tasks, generation, and interactions, while ablations examine IUPAC names, added data, and number tokenization.

4.1.1 Molecule Property Prediction

  • BioT5+ outperforms generalist baselines, including Galactica on classification and InstructMol on classification and regression, but trails some specialist models.The evaluation includes MoleculeNet classification and QM9 regression tasks.Source: Table 1 · p. 5, Table 3 · p. 6, §4.1.1 · p. 6

4.1.2 Chemical Reaction-related Tasks

  • Under the same training setting as Mol-Instructions, BioT5+ performs better across almost all reported chemical-reaction metrics.The tasks are reagent prediction, forward reaction prediction, and retrosynthesis.Source: Table 2 · p. 6, §4.1.2 · p. 6

4.1.3 Molecule Description Generation

  • On ChEBI-20 molecule-description generation, BioT5+ outperforms all compared specialist, retrieval-based, and multi-task generalist baselines.The inputs pair molecule SELFIES with IUPAC names.Source: §4.1.3 · pp. 6–7, Table 4 · p. 7

4.1.4 Description-guided Molecule Design

  • For description-guided molecule design, BioT5+ surpasses all compared baselines; its input omits IUPAC names to avoid a simplistic name-to-SELFIES mapping.Source: Table 5 · p. 7, §4.1.4 · p. 7

4.2.3 Protein-related Interaction Prediction

4.3 Ablation Studies

  • Removing IUPAC names or PubMed Central and bioRxiv data reduces molecule-generation performance, while character-based number tokenization outperforms default T5 tokenization on regression.Source: Table 7 · p. 8, Table 9 · p. 8, §4.3 · p. 9

4.2.1 Protein Description Generation

  • BioT5+ surpasses all compared baselines across four protein-description tasks, spanning function, catalytic activity, domain or motif, and functional-description generation.Source: §4.2.1 · pp. 7–8, Figure 2 · p. 8

4.2.4 Description-guided Protein Design

  • Without an established benchmark for description-guided protein design, the paper compares selected test cases with existing models using sequence-similarity scores.The cases and scores are presented in Appendix Table 20.Source: §4.2.4 · pp. 8–9

5 Conclusions

BioT5+ combines IUPAC integration, expanded data, multi-task instruction tuning, and numerical tokenization. The paper reports state-of-the-art performance in most evaluated tasks.

  • BioT5+ integrates IUPAC names, expanded bio-text and molecular data, multi-task instruction tuning, and numerical tokenization.Source: §5 · p. 9
  • The model bridges molecular representations and textual descriptions while improving its understanding of molecular structures and complex biological data.Source: §5 · p. 9
  • BioT5+ achieves state-of-the-art performance in most evaluated tasks, supporting its use as a versatile tool for analyzing biological entities.Source: §5 · p. 9

6 Limitations

BioT5+ still has difficulty generalizing across diverse biological tasks. Its scope also excludes image integration and broad general-domain question answering.

  • BioT5+ faces challenges generalizing across biological tasks because their specialized nature makes a one-size-fits-all solution difficult.Source: §6 · p. 9
  • Its current scale limits integration of modalities such as images, restricting its applicability to multimodal biological data analysis.Source: §6 · p. 9
  • BioT5+ is not designed as a universal chatbot or to answer questions outside biology.Source: §6 · p. 9

7 Ethical Considerations

The paper flags potential misuse and intellectual-property concerns associated with molecule generation and reaction-product prediction.

  • Generating molecules from text or predicting reaction products could be misused to create harmful compounds, risking public safety and environmental health.Source: §7 · p. 9
  • AI-driven design and synthesis of novel compounds may disrupt traditional research practices and raise questions about intellectual-property rights and ownership.Source: §7 · p. 9

A Additional Results

Additional results report BioT5+ performance across molecule-generation benchmarks and describe its training and data-processing procedures. The section also details IUPAC enrichment and molecule-name mapping.

A Additional Results

  • BioT5+ outperforms baseline methods on almost all metrics for molecule description generation and description-guided molecule design.Source: A Additional Results · p. 14

B Hyper-parameters

  • Pre-training uses 300K steps with batches balanced across eight tasks; fine-tuning searches dropout and learning rates, using different batch sizes for molecule- and protein-oriented tasks.Source: B Hyper-parameters · p. 14

C NER and Entity Linking

  • The entity-linking workflow retains BERN2 detections with confidence scores above 0.9 and appends SELFIES alone for recognized IUPAC names, otherwise appending both IUPAC and SELFIES.Source: C NER and Entity Linking · p. 14

D IUPAC Incorporation

  • Adding IUPAC names enriches molecules in property-prediction and description-generation tasks because those names occur in bio-text and encode structural information.Source: D IUPAC Incorporation · p. 14

D.1 Mapping Process

  • SMILES are normalized and mapped to PubChem CIDs for IUPAC retrieval, with STOUT translating sequences that lack a PubChem CID; pre-training also includes masked-span tasks across molecule, protein, general-text, and bio-text representations.Source: §D.1 · pp. 14–15, Figure 3 · p. 15

E Comparison with BioT5

BioT5+ expands BioT5’s pre-training data and molecule-text representation, adds IUPAC names to molecule descriptions, and changes numerical tokenization. It also introduces a different procedure for appending molecule names and SELFIES in wrapped text.

E Comparison with BioT5

  • 28.8M PubChem molecular records, 28.8M PubMed articles, and 2.3M bioRxiv abstracts expand BioT5+’s pre-training data.PubChem records pair molecule SELFIES with IUPAC names; bioRxiv abstracts are wrapped as biotext.Source: E Comparison with BioT5 · p. 15
  • BioT5+ includes IUPAC representations in the text description for molecule-text bidirectional translation.Source: E Comparison with BioT5 · p. 15
  • For wrapped biotext, BioT5+ appends SELFIES to detected molecule names, using a different rule for names identified as IUPAC.Names identified as IUPAC receive the corresponding SELFIES; other names receive both the IUPAC name and SELFIES, with detected bioentities processed only above a BERN2 confidence score of 0.9.Source: E Comparison with BioT5 · p. 15
  • BioT5+ tokenizes numbers in its pre-training corpus character by character.Source: E Comparison with BioT5 · p. 15

F Additional Ablation Study

The additional ablation study compares single-task and multi-task tuning across molecule classification, generation, and regression. The fine-tuning details describe the data and evaluation settings used for molecule property prediction.

F Additional Ablation Study

  • Single-task tuning performs closely or slightly better on BACE, BBBP, Clintox, and ChEBI-20, while multi-task tuning can benefit correlated QM9 predictions.The reported multi-task advantage is not uniform across tasks.Source: F Additional Ablation Study · p. 16, Table 14 · p. 18

G Fine-tuning Details

  • BioT5+ is fine-tuned jointly on molecule-oriented and protein-oriented tasks, with Mol-Instructions and an alternative dataset excluding its sub-domain used for comparison.Results are reported across differing epoch spans to accommodate dataset size differences, and are derived from three random runs.Source: G Fine-tuning Details · pp. 16–17

G.1.1 Molecule Property Prediction

  • Molecule property prediction uses four MoleculeNet classification datasets and three QM9 regression tasks; classification inputs include SELFIES with IUPAC names.The classification datasets are BACE, BBBP, HIV, and Clintox; the regression tasks are HOMO, LUMO, and HOMO-LUMO gap.Source: §G.1.1 · p. 17
  • Table 14 compares single-task and multi-task BioT5+ performance across various datasets and tasks; Mol2text means molecule description generation, and text2mol means description-guided molecule design.Source: Table 14 · p. 18

G.1.2 Chemical Reaction-related Tasks

The fine-tuning details define molecule- and protein-focused generation, design, property-prediction, and interaction tasks. The described tasks span sequence-to-text, text-to-sequence, and binary prediction settings.

G.1.2 Chemical Reaction-related Tasks

  • Chemical-reaction evaluation covers reagent prediction, forward reaction prediction, and retrosynthesis, using instruction-formatted data and splits from Mol-Instruction.Retrosynthesis works backward from a target product to identify plausible reactants.Source: §G.1.2 · p. 18

G.1.3 Molecule Description Generation

  • Molecule description generation turns a molecule into a comprehensive account of its properties, functions, and potential applications.The section identifies ChEBI-20 and Mol-Instructions as benchmark datasets.Source: §G.1.3 · p. 19

G.1.4 Description-guided Molecule Design

  • Description-guided molecule design translates a text description of desired molecular functions or properties into a molecule structure.The section uses ChEBI-20 and Mol-Instructions, with results referenced in Tables 5 and 16.Source: §G.1.4 · p. 19

G.2.1 Protein Description Generation

  • Protein description generation takes a protein sequence as input for four outputs: function, catalytic activity, domain or motif, and functional description.These instruction-formatted data and splits come from Mol-Instruction.Source: §G.2.1 · p. 20

G.3 Description-guided Protein Design

  • Description-guided protein design generates amino acid sequences to meet user-defined requirements, and the paper reports cases rather than a benchmark.Source: §G.3 · p. 20

G.4 Protein Property Prediction

  • Protein property prediction evaluates solubility and cellular localization using PEER tasks and the same data and splits as BioT5.Localization distinguishes membrane-bound from soluble proteins.Source: §G.4 · p. 20

G.5 Protein-Protein Interaction

  • Protein-protein interaction prediction classifies whether two protein sequences interact, using yeast and human datasets from PEER.Source: §G.5 · p. 21

G.6 Drug-Target Interaction

  • Drug-target interaction prediction classifies whether a SELFIES-encoded drug molecule interacts with a protein sequence, using BioT5’s data and splits.Source: §G.6 · p. 21

H.2 Description-guided Protein Design

The protein-design case gives a functional specification and compares generated sequences with a target FASTA. BioT5+ receives the highest reported normalized Smith-Waterman score among the three shown outputs.

H.2 Description-guided Protein Design

  • The case evaluates generated amino acid sequences against the ground-truth FASTA using normalized Smith-Waterman alignment, with terminal runs longer than two amino acids manually truncated.Source: §H.2 · p. 21
  • 0.916 normalized SW score for BioT5+, versus 0.328 for Mol-Instructions and 0.119 for Galactica, on the shown protein-design case.The input specifies functional and localization requirements; the target is given as a FASTA sequence.Source: §H.2 · p. 25
Loading 2402.17810v2…