Source-linked AI summary
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, Ji-Rong Wen
TL;DR
Existing molecular AI models typically learn one cognitive ability from one modality, while molecular expertise requires connecting structural and textual information. MoMu pretrains graph and text encoders on weakly related molecular graph–SCI paper text pairs, then transfers the shared representation across tasks. It improves cross-modal retrieval and property prediction, supports molecule captioning and zero-shot text-to-graph generation, and provides a basis for custom molecule design within the demonstrated scope.
Problem
Existing molecular AI models generally use a single molecular modality or cognitive ability, limiting direct connections between molecular structures and professional textual knowledge.
Method
MoMu jointly pretrains graph and text encoders on weakly related molecular graph–text pairs retrieved from SCI papers, using pretrained unimodal encoders.
Results
MoMu shows cross-modal transfer across retrieval, captioning, zero-shot molecule generation, and property prediction, with improved retrieval and graph representations on downstream evaluations.
Takeaways & Limitations
The model directly bridges molecular graphs and natural language, supporting text-conditioned molecule design and applications such as drug discovery and materials design.
Takeaways & Limitations
MoMu uses relatively few graph-text pairs, and retrieved texts may not describe a molecule’s properties or structure, introducing risks of insufficient alignment and spurious correlations.
Abstract
from arXiv · showhide
Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality. Since the hierarchy of molecular knowledge is profound, even humans learn from different modalities including both intuitive diagrams and professional texts to assist their understanding. Inspired by this, we propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data (crawled from published Scientific Citation Index papers) via contrastive learning. This AI model represents a critical attempt that directly bridges molecular graphs and natural language. Importantly, through capturing the specific and complementary information of the two modalities, our proposed model can better grasp molecular expertise. Experimental results show that our model not only exhibits promising performance in cross-modal tasks such as cross-modal retrieval and molecule caption, but also enhances molecular property prediction and possesses capability to generate meaningful molecular graphs from natural language descriptions. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine, among others.
Results
MoMu aligns molecular graphs with weakly related scientific text and transfers this joint representation across retrieval, generation, captioning, and property prediction tasks. Results show improved cross-modal retrieval, diverse text-conditioned molecule generation, and stronger graph representations for molecular property prediction.
- Model and data: MoMu constructs graph-text pairs from molecular compounds and initializes graph and text encoders with pretrained unimodal models before joint pretraining.The dataset uses molecular graphs derived from PubChem SMILES and weakly related descriptions retrieved from SCI papers; GIN and Sci-BERT or KV-PLM initialize the encoders.
- Cross-modality retrieval: Both MoMu variants generally outperform competing methods in text-to-graph retrieval, and MoMu-K retrieves four confirmed dye molecules among the top five.The zero-shot text-to-graph evaluation also reports significant gains over Sci-BERT and KV-PLM.
- Cross-modality retrieval: MoMu-S and MoMu-K obtain comparable retrieval results, indicating that graph pretraining with language supervision captures structural information not readily transferred from SMILES modeling.This comparison supports using a graph neural network directly to learn molecular structure representations under textual supervision.
- Zero-shot text-to-graph molecule generation: MoMu enables zero-shot text-to-graph generation of diverse molecules from vague, functional, and structural descriptions without additional training data.The method combines MoMu with a differentiable pretrained molecule generator; generated molecules can reflect specified functional and structural characteristics.
- Molecule property prediction: Across eight MoleculeNet datasets, MoMu initialization beats random initialization on most datasets, achieves the best results on three, and outperforms GraphCL on six.On average, both MoMu variants outperform the compared methods; t-SNE visualizations also show better separation after fine-tuning on BBBP.
Discussion
MoMu bridges molecular graphs and natural-language descriptions through joint multimodal modeling, supporting cross-modal transfer and zero-shot molecule generation. The authors also identify data and transfer limitations that constrain reliability and coverage.
- MoMu aligns molecular graph and language representations in a common space, with downstream results demonstrating cross-modality transfer across captioning, retrieval, and property prediction.
- MoMu-based zero-shot text-to-graph generation learns text-related generative seeds for a pretrained molecule generator, enabling custom design from specified conditions.
- The method may support drug discovery and materials design that require new molecules with specified properties.
- MoMu may learn spurious correlations because retrieved texts can contain molecule names without describing corresponding properties or structures.
- Reliable generation is difficult for molecular properties absent or infrequent in training texts, and performance depends on MoMu and the pretrained generator.
Methods
MoMu is pretrained by jointly encoding molecular graphs and scientific text, using graph augmentations and contrastive objectives to align their representations. Its pipeline also supports differentiable zero-shot generation of molecule graphs from text descriptions.
- Data collection: The pretraining data pair molecular graphs with weakly related biochemical descriptions retrieved from scientific literature.Texts are retrieved from abstracts, introductions, and conclusions in papers from Medicine, Biology, Chemistry, and Computer Science.
- Graph-modal self-supervision: Graph self-supervision contrasts two augmented views of each molecule against views from different molecules.The augmentations include node dropping and subgraph sampling, and the loss uses normalized temperature-scaled cross-entropy.
- Model architecture: Two encoders map molecular graphs and texts into a joint representation space for multimodal learning.The graph encoder is a GIN, while the text encoder is initialized from scientific language models including Sci-BERT and KV-PLM.
- Cross-modal supervision: Cross-modal supervision aligns representations from graph augmentations with representations of corresponding molecular descriptions.Each minibatch contains two augmented graphs and two text descriptions per molecule, with four cross-modal contrastive losses.
- Zero-shot molecule generation: Zero-shot generation optimizes a latent variable through MoMu similarity while keeping the pretrained MoMu and MoFlow parameters fixed.The optimized latent variable is passed through MoFlow, and argmax operations convert predicted atom and bond probabilities into a molecule graph.
Data availability
The project provides access information for the generation algorithm, datasets, pretrained models, and collected multimodal data.
- Datasets and models: The paper identifies public sources for PubChem, S2orc, PCdes, ChEBI-20, MoFlow, and MoleculeNet resources used across the pipeline.These sources cover pretraining data, retrieval, captioning, generation, and property-prediction experiments.
- Generation algorithm: The zero-shot text-to-graph procedure is specified as Algorithm 1 with pretrained encoders, similarity computation, MoFlow components, and optimization hyperparameters as inputs.The procedure initializes q from a Gaussian distribution, updates it with Adam, and returns a generated molecule graph after decoding.
- Pretraining data: The collected pretraining dataset is distributed as separate folders containing molecular graphs and texts.The source text identifies a download location for this dataset.