PaperlayerView exact revision on arXiv

Source-linked AI summary

BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs

Zifeng Wang, Zichen Wang, Balasubramaniam Srinivasan, Vassilis N. Ioannidis, Huzefa Rangwala, Rishita Anubhai

arXiv:2310.03320v4cs.LG

TL;DR

Biomedical foundation models are largely trained and used unimodally, limiting their multimodal behavior. BioBRIDGE learns transformations between independently trained FMs from knowledge graphs while keeping the base models frozen. Its bridged models are competitive across diverse cross-modal tasks, generalize to unseen entities and relations, and perform on par with supervised specialists.

  • Problem

    Biomedical foundation models are largely trained and used unimodally, while scaling multimodal approaches beyond two modalities is challenging because they require large volumes of paired data.

    Source: §1 · pp. 1–2

  • Method

    BioBRIDGE trains knowledge-graph-supervised transformation bridges between unimodal FMs while keeping the underlying models frozen.

    Source: §1 · p. 2

  • Results

    BioBRIDGE is competitive across diverse cross-modal prediction tasks, generalizes to unseen entities and relations, and performs on par with supervised specialist models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The approach provides a parameter-efficient way to connect unimodal biomedical FMs for multimodal tasks using knowledge-graph relations.

    Source: §1 · p. 2, §6 · p. 9

    The stated bridge-module assumptions include frozen modality encoders whose output spaces are linear subspaces of the same dimension.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Foundation models (FMs) are able to leverage large volumes of unlabeled data to demonstrate superior performance across a wide range of tasks. However, FMs developed for biomedical domains have largely remained unimodal, i.e., independently trained and used for tasks on protein sequences alone, small molecule structures alone, or clinical data alone. To overcome this limitation of biomedical FMs, we present BioBridge, a novel parameter-efficient learning framework, to bridge independently trained unimodal FMs to establish multimodal behavior. BioBridge achieves it by utilizing Knowledge Graphs (KG) to learn transformations between one unimodal FM and another without fine-tuning any underlying unimodal FMs. Our empirical results demonstrate that BioBridge can beat the best baseline KG embedding methods (on average by around 76.3%) in cross-modal retrieval tasks. We also identify BioBridge demonstrates out-of-domain generalization ability by extrapolating to unseen modalities or relations. Additionally, we also show that BioBridge presents itself as a general purpose retriever that can aid biomedical multimodal question answering as well as enhance the guided generation of novel drugs.

1 INTRODUCTION

Biomedical FMs are commonly trained unimodally, while scaling joint multimodal training faces data and computational costs. BioBRIDGE uses knowledge-graph structure to learn cross-modal transformations while keeping the underlying FMs frozen.

1 INTRODUCTION

  • Joint multimodal training requires similarly sized unimodal encoders, which can make bundled models grow with the number of modalities and increase computational burden.The approach also faces a need for large pairwise cross-modal datasets, whose quantity grows combinatorially with the number of modalities and leads to data scarcity.Source: §1 · pp. 1–2
  • BioBRIDGE learns transformations between unimodal FMs from biomedical KG structure while keeping the FMs fixed.The bridge modules are supervised by relational information in the KG, supporting parameter and computational efficiency.Source: §1 · p. 2
  • Unimodal data are more available than paired data: about 250M protein sequences and 1.5B molecule structures versus 441K protein-text pairs.The paper presents bridging independently pretrained models as a more data-sufficient and efficient alternative to jointly training multimodal encoders.Source: §1 · p. 2
  • Biomedical KG triplets encode relations among entities across modalities, and BioBRIDGE uses that structure to align unimodal FM embedding spaces.The described modalities include proteins, molecules, and diseases.Source: §1 · p. 2
  • BioBRIDGE’s bridged FMs are competitive in diverse cross-modal prediction tasks and generalize to unseen KG nodes and relationships.The paper also reports comparable performance to supervised baselines for unseen nodes and improved performance on unseen relationships with further training.Source: §1 · p. 2

2 RELATED WORK

Biomedical foundation models are often trained on single modalities, while multimodal biomedical models have used contrastive learning on paired data. BioBRIDGE instead trains transformation bridges and uses KG structure without training the base FMs.

2 RELATED WORK

  • Biomedical FMs are trained on unimodal text, protein, and molecule data and used as feature extractors for supervised prediction tasks.Prior multimodal biomedical FMs use contrastive learning on paired image-text, drug-protein, drug-text, and protein-text data.Source: §2 · p. 3
  • Extending paired-data multimodal approaches beyond two modalities is challenging because it requires large volumes of multimodal pairs.Source: §2 · p. 3
  • Unlike image-centered alignment approaches, BioBRIDGE trains transformation bridge modules without involving the base FMs and leverages KG structure for cross-modal transformations.The KG structure relaxes the central-modality assumption and allows more controlled cross-modality transformation.Source: §2 · p. 3

3 METHOD

BioBRIDGE uses knowledge-graph relations and modality types to learn transformations between frozen unimodal foundation-model spaces. The transformed embeddings support cross-modal retrieval without requiring the knowledge graph at inference.

3 METHOD

  • KG-supervised transformations map a head-node embedding into the tail modality’s space, using modality types and relations while keeping the unimodal foundation models frozen.The resulting embedding can match relevant tail-modality samples by similarity search.Source: §3 · pp. 3–4

3.1 TRAINING & PREDICTION

  • Contrastive training transforms projected head embeddings using modality and relation embeddings, then learns to bring them closer to positive tails than sampled negative tails.The base FMs remain fixed; training updates the transformation module and modality-specific projection heads.Source: §3.1 · p. 4
  • At inference, BioBRIDGE encodes candidate tails and the query head, transforms the head embedding, and ranks candidates by similarity without consulting the KG.The text describes comparing normalized embeddings through a matrix inner product.Source: §3.1 · p. 4

3.2 IMPLEMENTATION

  • The experiments use PrimeKG’s six node types, grouped into protein-sequence, molecule-SMILES, and natural-language modalities, with a vanilla six-layer transformer as the bridge.The transformer takes projected head, modality, and relation embeddings; its first-position output is added to the projected head embedding.Source: §3.2 · pp. 4–5

3.3 EXISTENCE OF BRIDGE MODULE AND LEARNABILITY

  • Under the stated assumptions, a unified bridge module exists for any pair of KG-connected nodes, mapping the source representation to the target representation using their modalities and relation type.The assumptions include frozen encoders with equal-dimensional linear output subspaces.Source: §3.3 · p. 5

4 EXPERIMENT: CROSS-MODALITY PREDICTION

The experiments evaluate BioBRIDGE on cross-modal retrieval, semantic similarity inference, protein-protein interaction prediction, and cross-species protein-phenotype matching. Results report advantages over baselines and performance on tasks involving unseen relations or entities.

4.1 CROSS-MODALITY RETRIEVAL TASKS

  • BioBRIDGE ranked best among KGE methods across seven cross-modal tasks; MRR was around 3× the best baseline for Protein →MF and around 1.6× for Protein →BP.The evaluation uses Hit@K and MRR for tail-entity prediction across PrimeKG triple types.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 SEMANTIC SIMILARITY INFERENCE

  • Around 2× better than the best baseline on average, BioBRIDGE improved semantic-similarity inference across biological-process, molecular-function, and cellular-component spaces.Protein embeddings are transformed to those functional spaces, and pairwise Manhattan similarities are compared with GO-based ground truth using Spearman rank correlation.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3 PROTEIN-PROTEIN INTERACTION

  • BioBRIDGE improved over baselines in most PPI scenarios, with greater benefit on smaller SHS27K datasets; methods tended to converge as training data increased.The task predicts seven interaction types, six of which are out-of-domain relations in PrimeKG.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 CROSS-SPECIES PROTEIN-PHENOTYPE MATCHING

  • BioBRIDGE had a large margin over the paired-data supervised baseline for mouse-protein/mouse-phenotype matching and matched mouse proteins to human phenotypes where that baseline failed.Mouse proteins and phenotypes are outside the training data; the method transforms mouse protein embeddings to disease space using the in-domain “associate with” relation.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5 CASE STUDY: MULTIMODAL GENERATION

BioBRIDGE uses cross-modal retrieval to supply evidence for Galactica’s biomedical question answering and context for generating small-molecule drugs. The reported examples describe relevant answers and generated molecules with similarities to drugs used for the target condition.

5 CASE STUDY: MULTIMODAL GENERATION

  • For multimodal question answering, BioBRIDGE retrieves proteins, diseases, or gene ontology terms relevant to a molecule or protein question.The reported Vincristine example retrieves tubulin proteins and oncology-related conditions, prompting Galactica to answer that the drug inhibits cancer-cell mitosis.Source: Table 5 · p. 8, §5 · p. 9
  • BioBRIDGE retrieval prompts Galactica to generate valid drug molecules, while direct prompting without retrieval produces poor generation results.The generated molecule in the reported example is most similar to Procarbazine, used to treat stage III/IV Hodgkin’s disease.Source: §5 · p. 9, Table 6 · p. 9
  • For generation, BioBRIDGE retrieves proteins possibly associated with the target condition to enrich Galactica’s context.The study assesses generated molecules using their Tanimoto distance to candidate DrugBank drugs.Source: §5 · p. 9, Table 6 · p. 9

6 CONCLUSION

The conclusion presents BioBRIDGE as a parameter-efficient bridge between frozen unimodal biomedical foundation models, supervised by biomedical knowledge graphs. The accompanying proof section establishes bridge existence under stated assumptions and describes path aggregation and learnability.

6 CONCLUSION

  • BioBRIDGE transforms embeddings across modality and relation types while training only the bridge modules and keeping the base foundation models fixed.The conclusion reports extrapolation to in-domain and out-of-domain entities and relations, performance on par with supervised specialist models, and support for multimodal generation tasks.Source: §6 · p. 9

A PROVING EXISTENCE OF BRIDGE MODULE AND LEARNABILITY

  • A bridge module is asserted to map a source representation to a target representation for any connected node pair, given their modality types and relation type.The proof frames the construction for frozen modality-specific networks whose output spaces are linear subspaces of equal dimension, with KG edges carrying relation types.Source: A PROVING EXISTENCE OF BRIDGE MODULE… · p. 14
  • Multiple KG paths between nodes can be aggregated as a weighted combination of path transformations; positive weights summing to one keep the aggregate within the original transformations.The text identifies each weight with a path and A with a transformation matrix.Source: A PROVING EXISTENCE OF BRIDGE MODULE… · p. 14
  • The proof argues that relation-conditioned transformations can be approximated with suitable loss functions and stochastic gradient descent.It invokes convergence of linear transformations, neural-network universality, and continuity of the unimodal foundation models.Source: A PROVING EXISTENCE OF BRIDGE MODULE… · p. 15

B ESTABLISHING TRAINING DATA BASED ON PRIMEKG

The PrimeKG training graph is curated by linking entities to external sources for required properties and dropping entities without those properties. The experiment retains six modalities as a proof of concept.

B ESTABLISHING TRAINING DATA BASED ON PRIMEKG

  • The proof-of-concept KG retains six modalities, with protein, disease, drug, and gene ontology entities covering many biomedical tasks.Table 7 is described as reporting raw and processed PrimeKG node and edge statistics, including dropped samples.Source: B ESTABLISHING TRAINING DATA BASED ON… · p. 15, Table 7 · p. 15
  • Of 27,671 original PrimeKG proteins, 27,478 sequences were matched; drugs without SMILES were dropped, leaving 6,948 of 7,957.The authors say most unmapped proteins are non-protein-coding genes and are reasonably ignored for protein-centric tasks.Source: B ESTABLISHING TRAINING DATA BASED ON… · pp. 15–16
  • All 17,080 PrimeKG diseases were retained, while GO-term features were retrieved for most terms and included in training.The reported GO counts are 27,478 BP, 10,966 MF, and 4,013 MF.Source: B ESTABLISHING TRAINING DATA BASED ON… · p. 16

C DATA STATISTICS OF THE KNOWLEDGE GRAPH

The section identifies a table of training knowledge-graph data statistics drawn from PrimeKG.

C DATA STATISTICS OF THE KNOWLEDGE GRAPH

  • Table 8 presents data statistics for the training knowledge graph drawn from PrimeKG.The supplied caption does not state the table’s values or specific categories.Source: Table 8 · p. 16

D CROSS-MODALITY RETRIEVAL TASKS

The section evaluates cross-modality retrieval across seven relation-specific tasks and reports method ranks aggregated across them. It also provides train/validation/test statistics for these tasks and statistics for a cross-species retrieval dataset.

D CROSS-MODALITY RETRIEVAL TASKS

  • Table 9 summarizes mean and standard deviation of method ranks across seven cross-modality retrieval tasks, with lower rank indicating better performance.The supplied caption does not include the rank values or identify a winning method.Source: Table 9 · p. 16
  • The section includes statistics for the retrieval-task train, validation, and test splits, as well as for a cross-species retrieval dataset.Source: Table 10 · p. 17, Table 18 · p. 18
  • Three tasks evaluate protein interactions with biological process, molecular function, and cellular component.Source: Table 11 · p. 17, Table 12 · p. 17, Table 13 · p. 17
  • The remaining four tasks cover drug indication, protein targets, disease-associated proteins, and drug contraindications.Source: Table 14 · p. 17, Table 15 · p. 18, Table 16 · p. 18

E MOUSE PROTEIN AND PHENOTYPE DATASET

The dataset pairs mouse proteins with phenotypes and corresponding human proteins. The authors map phenotype descriptions and protein sequences, then filter the paired records to form the final dataset.

E MOUSE PROTEIN AND PHENOTYPE DATASET

  • The starting data contain 29,686 paired mouse protein, human protein, and mouse phenotype identifiers, plus definitions for 14,201 mouse phenotypes.Mouse protein–phenotype pairs came from the Mouse Genome Informatics report, while phenotype descriptions came from the MPheno ontology.Source: E MOUSE PROTEIN AND PHENOTYPE DATASET · p. 18
  • Mouse protein identifiers were mapped to protein sequences through UniProt, using MGI as the source database and UniProtKB/Swiss-Prot as the target.Source: E MOUSE PROTEIN AND PHENOTYPE DATASET · p. 18
  • Filtering for sequence availability, multiple phenotype links, and qualifying PrimeKG human-protein links yielded 14,024 mouse proteins, 14,200 phenotypes, 1,582 human proteins, and 3,279 human phenotypes.Source: E MOUSE PROTEIN AND PHENOTYPE DATASET · p. 19

F CASE STUDY: MULTIMODAL Q&A

The case studies describe using BioBRIDGE retrieval to support multimodal question answering and drug generation, alongside prompts for those tasks. Related-work passages contrast BioBRIDGE with prior knowledge-graph and multimodal approaches.

F CASE STUDY: MULTIMODAL Q&A

  • The multimodal Q&A case studies take a molecule and question as input, retrieve related diseases and proteins, and use an LLM to answer.The supplied examples include drug molecules paired with disease questions, but the captions do not establish answer quality.Source: Table 19 · p. 19
  • The generation case-study captions describe retrieving proteins for a target condition and intended mechanism before prompting an LLM to generate a small-molecule drug, with a separate direct-generation baseline.Source: Table 20 · p. 20, Table 21 · p. 21

G CASE STUDY: PROMPT

  • The prompts present a molecule’s SMILES structure for question answering and ask a language model to generate a likely SMILES structure from natural-language inputs and target proteins.Source: G CASE STUDY: PROMPT · p. 22

H ADDITIONAL RELATED WORKS

  • Prior KG methods are described as limited on unseen or isolated nodes, while BioBRIDGE uses base FMs and bridge modules; prior multimodal KG methods are also described as fixed-KG or visually and textually focused.The passages distinguish BioBRIDGE by its ability to process KG modalities and transform inputs to a desired target modality.Source: H ADDITIONAL RELATED WORKS · p. 22

I TRAINING SETUPS

The training setup uses deliberately simple bridge architectures and evaluates hyperparameter sensitivity and bridge-module ablations. The authors select a fixed training configuration and report that an additive residual transformation performed better than the tested alternative.

I TRAINING SETUPS

  • The authors chose simple transformer architectures and the original InfoNCE loss to test whether bridging unimodal foundation models is feasible.They describe these choices as the minimum needed for BioBRIDGE to work.Source: I TRAINING SETUPS · p. 23
  • Across the tested batch sizes, performance did not differ significantly, and training converged within 50 epochs without further improvement from more epochs.The tested batch sizes were 512, 1,024, and 4,096; the tested epoch counts were 10, 50, and 100.Source: I TRAINING SETUPS · p. 23
  • The bridge-module ablation found that removing the residual connection failed to converge, while a RotatE transformation performed worse than the additive transformation.Source: I TRAINING SETUPS · p. 23

Related papers

Loading 2310.03320v4…