Source-linked AI summary
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, Tat-Seng Chua
TL;DR
Molecular language models largely lack 2D graph perception, despite its importance for understanding molecular topology. MolCA addresses this gap by connecting graph representations to an LM through a cross-modal projector and adapting the LM efficiently for downstream tasks, with strong results across generation and retrieval benchmarks.
Problem
Existing molecular LMs mainly process 1D SMILES and overlook 2D graph representations, while contrastive approaches are insufficient for open-ended molecule-to-text generation.
Method
MolCA uses a graph encoder, a Q-Former cross-modal projector, and a uni-modal adapter to condition an LM on 2D molecular graphs for text generation.
Results
MolCA achieves state-of-the-art or superior performance across molecule captioning, IUPAC name prediction, and molecule-text retrieval benchmarks, including gains of 2.1, 7.6, and 10.0 BLEU-2 and 20% retrieval accuracy.
Takeaways & Limitations
Combining 2D graphs with 1D SMILES improves performance across compared molecule-to-text generation and molecular property-prediction tasks.
Takeaways & Limitations
Molecule-captioning performance is not yet sufficient for practical application, partly because the 324k molecule-text pretraining dataset is smaller than datasets used in vision-language pretraining.
Abstract
from arXiv · showhide
Language Models (LMs) have demonstrated impressive molecule understanding ability on various 1D text-related tasks. However, they inherently lack 2D graph perception - a critical ability of human professionals in comprehending molecules' topological structures. To bridge this gap, we propose MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. MolCA enables an LM (e.g., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector. Specifically, the cross-modal projector is implemented as a Q-Former to connect a graph encoder's representation space and an LM's text space. Further, MolCA employs a uni-modal adapter (i.e., LoRA) for the LM's efficient adaptation to downstream tasks. Unlike previous studies that couple an LM with a graph encoder via cross-modal contrastive learning, MolCA retains the LM's ability of open-ended text generation and augments it with 2D graph information. To showcase its effectiveness, we extensively benchmark MolCA on tasks of molecule captioning, IUPAC name prediction, and molecule-text retrieval, on which MolCA significantly outperforms the baselines. Our codes and checkpoints can be found at https://github.com/acharkq/MolCA.
1 Introduction
MolCA addresses the gap between 1D molecular language modeling and the 2D graph perception needed for molecule understanding by connecting graph representations to an LM’s text space. Its benchmarks and ablations show strong performance across molecule-to-text generation, retrieval, and graph-informed molecular tasks.
- Existing molecular LMs commonly process 1D SMILES strings, overlooking 2D graph representations important for comprehending molecular structures.
- Cross-modal contrastive learning supports retrieval but is insufficient for open-ended molecule-to-text generation, including captioning and IUPAC name prediction.
- MolCA connects 2D molecular graphs to an LM through a cross-modal projector that translates graph representations into text-space soft prompts.
- MolCA uses a three-stage training pipeline whose pretraining stages develop cross-modal alignment for retrieval and captioning before downstream generation fine-tuning.
- 2.1 and 7.6 BLEU-2 gains occur for molecule captioning on CheBI-20 and PubChem324k, while IUPAC prediction improves by 10.0 BLEU-2 over baselines.
- MolCA improves PubChem324k molecule-text retrieval accuracy by 20% and achieves the best performances on PCDes and MoMu, while graph incorporation improves functional-group counting.
2 Model Architecture
MolCA combines a GNN-based graph encoder, Galactica language model, and Q-Former cross-modal projector to connect 2D molecular structure with text generation. The architecture preserves molecule and text processing pathways while enabling their representations to interact.
- MolCA’s architecture contains a graph encoder for 2D structure understanding, an LM for text generation, and a cross-modal projector connecting their representations.
- The graph encoder is a five-layer GINE pretrained by contrastive learning on 2 million ZINC15 molecules to produce structure-aware node features.
- Galactica serves as the base decoder-only LM, leveraging scientific pretraining and the ability to process 1D SMILES.
- The Q-Former maps graph-encoder outputs into the LM’s input text space and uses query tokens with cross-attention to extract molecular features.
- Q-Former query tokens and text inputs share self-attention interactions but use separate feed-forward networks to maintain molecule- and text-processing capacities.
- Q-Former is initialized from Sci-BERT, while its cross-attention modules are randomly initialized.
3 Training Pipeline
MolCA trains a cross-modal projector to translate 2D molecular graphs into soft prompts that a language model can use for text generation. Its pipeline combines molecule-text pretraining, captioning-based alignment, and efficient downstream adaptation with a uni-modal adapter.
- Training Pipeline: Pretraining uses molecule-text pairs to learn graph representations and translate 2D molecular graphs into soft prompts understandable by a frozen language model.The pipeline contains two pretraining stages before downstream fine-tuning.
- Pretrain Stage 1: Stage 1 trains the cross-modal projector with molecule-text contrasting, matching, and captioning tasks to extract text-relevant molecule features.The resulting projector is used for molecule-text retrieval, with matching providing finer-grained similarity than contrasting.
- Pretrain Stage 1: Molecule-text matching predicts whether a pair is matched or unmatched after queries and texts interact through shared attention modules.A linear classifier operates on the mean-pooled query representations, and in retrieval experiments matching reranks candidates initially retrieved by contrasting.
- Pretrain Stage 2: Stage 2 feeds graph-projector representations to a frozen language model and trains text generation, while also using 1D SMILES to guide generation.This stage aligns graph-derived representations with the language model’s text space for molecule captioning.
- Fine-tune Stage: Fine-tuning appends a task-description prompt after molecule representations and applies language-modeling loss to downstream generation tasks such as IUPAC name prediction.MolCA’s fine-tune stage is illustrated for molecule-to-text generation.
- Fine-tune Stage: LoRA adapts selected language-model weight matrices by training low-rank updates while keeping the original weights frozen, reducing gradient-storage overhead.The adapter addresses the language model’s substantially larger parameter count relative to the graph encoder and projector.
4 Experiments
MolCA is evaluated on molecule captioning, IUPAC name prediction, and molecule-text retrieval using established datasets, baselines, and metrics.
- Molecule Captioning: The experiments cover molecule captioning on PubChem324k and CheBI-20, using Galactica and MolT5-based MolCA variants.Evaluation uses BLEU, ROUGE, and METEOR, with test performance selected by validation results.
- Molecule Captioning: 7.6 BLEU-2 on PubChem324k and 2.1 BLEU-2 on CheBI-20: MolCA, Galac1.3B outperforms the baselines.It achieves the highest performance on all reported captioning metrics.
- Molecule Captioning: MolCA, Galac125M outperforms larger baselines across all captioning metrics, indicating its advantage is not limited to model scale.The comparison includes baselines with larger model sizes.
- IUPAC Name Prediction: 10.0 BLEU-2: MolCA exceeds baselines in IUPAC name prediction on PubChem324k.The task evaluates generation of standardized chemical names from molecules.
1. MolCA demonstrates superior performance
Ablation studies examine the contributions of 1D SMILES and 2D graph representations, retrieval components, dataset scale, and graph-based functional-group reasoning.
- Retrieval Components: MTM significantly improves MolCA’s retrieval performance by modeling long-range interactions between molecule features and texts through cross-attention and self-attention.The result attributes the improvement to MTM’s interaction modeling capability.
- Pretraining Data: Over 25% retrieval-accuracy improvement results when reproduced MoMu is pretrained on PubChem324k instead of 15k molecule-text pairs.This comparison indicates that the larger pretraining dataset contributes partially to MolCA’s performance.
- Representation Types: Combining 2D graphs and 1D SMILES improves performance across all compared molecule generation and property-prediction tasks.The ablation compares variants using only 1D SMILES or only 2D graphs with MolCA’s combined representation setting.
- Functional-Group Counting: RMSE is the functional-group-counting metric, and lower values indicate better performance; results average three random seeds with ± one standard deviation.The evaluation uses PubChem324k molecules.
- Functional-Group Counting: Incorporating 2D graphs significantly improves MolCA’s functional-group counting performance across 85 functional-group types.Functional groups are treated as molecule subgraphs with consistent chemical behaviors across molecules.
5 Related Works
Prior work largely models molecules as 1D text or aligns molecule and text encoders through contrastive learning, whereas MolCA targets open-ended generation from 2D graphs.
- Molecule Understanding via 1D Language Modeling: Existing language models obtain molecular and chemical understanding from biochemical literature and process molecules through 1D representations such as SMILES.Related methods include Galactica, KV-PLM, Vaucher et al., and MolT5.
- Molecule-Text Contrastive Learning: Text2Mol and subsequent methods use cross-modal contrastive learning to connect molecular graph encoders with text encoders for molecule-text retrieval.This line of work includes graph and text encoders trained jointly for cross-modal matching.
- Molecule-Text Contrastive Learning: Cross-modal contrastive learning focuses on similarity functions and is unsuitable for open-ended conditional generation tasks.MolCA addresses this problem by enabling language models to understand 2D molecular graphs for molecule-to-text generation.
6 Conclusion and Future Works
MolCA maps 2D molecular graph representations into an LM’s text space and uses a uni-modal adapter for efficient downstream adaptation.
- Conclusion: MolCA is designed to let language models perceive 2D graphs for molecule-to-text generation.Its cross-modal projector maps graph representations into the language model’s text space.
- Conclusion: MolCA achieves state-of-the-art performance on molecule captioning and molecule-text retrieval benchmarks.The paper identifies exploration of 3D molecular modeling and drug discovery tasks as future work.
Limitations
The work is limited to LM generation abilities for molecule-text tasks, leaving in-context learning and chain-of-thought reasoning for future exploration. Molecule captioning performance is improved but not yet sufficient for practical application, partly because available molecule-text pretraining data are relatively small.
- MolCA focuses on LM generation for molecule-text tasks, while in-context learning and chain-of-thought reasoning remain outside its scope.
- Molecule captioning performance is not yet sufficient for practical application despite improvements over baselines.
- The 324k-example PubChem324k dataset is smaller than the approximately 10M-example dataset used for vision-language pretraining, limiting performance.
- Mining weakly supervised data from biochemical literature is suggested as a possible remedy for the limited pretraining-data scale.
Broader Impacts
The paper reports potential benefits for chemistry professionals and people without specialized chemistry knowledge, while noting risks associated with inaccurate or biased LM-generated content. It advises strict testing before real-world application given the limited training-data scale.
- Molecule captioning and retrieval could potentially help chemistry professionals speed up their research process.
- The method could provide people without specialized chemistry knowledge with a more affordable way to access basic chemical information.
- The model can generate inaccurate information and may be abused to produce biased content, reflecting risks shared by most LMs.
- The authors advise strictly testing the model before real applications because the training data have limited scale.
B Experimental Settings
The experiments use staged pretraining, LoRA-based fine-tuning, and datasets spanning molecule captioning, IUPAC prediction, retrieval, and related molecular tasks. Settings include an eight-query-token Q-Former, specified optimization schedules, and controlled dataset preprocessing.
- Pretraining: MolCA’s pretraining uses 50 epochs for stage 1 and 10 epochs for stage 2, with a Q-Former containing 8 query tokens.
- Generation: Molecule captioning fine-tunes Galactica with LoRA applied to q_proj and v_proj, using rank r = 8 and an adapter containing 2M parameters.
- IUPAC prediction: IUPAC name prediction collects names for PubChem324k train, validation, and test molecules using PubChemPy.
- Retrieval: Retrieval uses the stage-1 MolCA checkpoint without fine-tuning on other datasets, corresponding to a zero-shot retrieval setting.
- Molecular tasks: Functional-group counting uses RDKit ground-truth counts and separate linear regressors trained with mean squared error.
- Datasets: PubChem324k descriptions use webpage description fields, with common or IUPAC names at text beginnings replaced to avoid information leakage.
C More Experimental Results
Additional experiments show that retrieval benefits consistently from MolCA and its MTM re-ranking module, while both pretraining stages contribute to captioning. Q-Former and combined 1D SMILES plus 2D graphs improve generation, and fine-tuning is computationally affordable.
- Retrieval: MolCA consistently outperforms baselines on molecule-text retrieval, and MTM re-ranking is crucial to its retrieval performance.
- Pretraining ablations: Both pretraining stages make significant contributions to MolCA’s molecule captioning performance.
- Projector ablations: Q-Former outperforms a linear cross-modal projector, which is attributed to better bridging between 2D molecular graphs and 1D text.
- Projector ablations: The linear projector slightly outperforms the SMILES-only baseline, but its gains remain limited because the projector is less effective.
- Generation results: Using both 1D SMILES and 2D graphs produces more accurate molecule-structure descriptions than using only 1D SMILES.
- Computational cost: The fine-tuning stage is affordable in computational-resource terms in experiments conducted on two NVIDIA A100 40 GB GPUs.