Source-linked AI summary
GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text
Pengfei Liu, Yiming Ren, Jun Tao, Zhixiang Ren
TL;DR
Existing language models struggle to capture molecular graphs and images alongside textual representations. GIT-Mol integrates graph, image, and text information through GIT-Former, which aligns modalities in a unified latent space; it reports higher property-prediction accuracy and molecule-generation validity than baselines.
Problem
Existing language models process textual molecular data but cannot adequately capture the rich information in complex molecular structures or images.
Method
GIT-Mol integrates graph, image, and text modalities through GIT-Former, which uses cross-attention and multimodal pretraining to map them into a unified latent space.
Results
GIT-Mol reports 5%-10% higher accuracy in molecular property prediction and a 20.2% boost in molecule-generation validity compared with baselines.
Takeaways & Limitations
Its any-to-language translation strategy supports applications including molecule generation, captioning, image recognition, property prediction, compound name recognition, and chemical reaction prediction.
Takeaways & Limitations
Training speed remains challenging, and appropriate evaluation methods are lacking for molecular captioning and generation.
Abstract
from arXiv · showhide
Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.
1. Introduction
GIT-Mol addresses the difficulty of integrating molecular graphs, images, and text by introducing a unified multimodal architecture. It reports improvements over baselines and single-modality models across molecular generation and property prediction tasks.
- Motivation: Existing molecular methods can be time-consuming, expensive, failure-prone, and computationally demanding despite computational advances.These limitations motivate scalable multimodal approaches for molecular discovery and optimization.
- Motivation: Existing language models process textual molecular data well but struggle with molecular graphs and images, while models fusing three or more modalities remain scarce.Missing or incomplete multimodal data further complicates integration.
- Approach: GIT-Mol is a 700M-parameter multimodal language model covering graph, image, and text modalities for generation, captioning, image recognition, and property prediction.The model is designed to exploit large quantities of unlabeled multimodal molecular data.
- Approach: GIT-Former uses cross-attention to fuse three molecular modalities at the molecule level, supporting flexibility and scalability.Its architecture is presented as the core multimodal component of GIT-Mol.
- Results: 20% higher molecule-generation validity and 5%-10% higher molecular-property-prediction performance are reported relative to baselines.These figures summarize the headline quantitative comparisons presented in the introduction.
2. Related Works
Prior molecular multimodal research spans molecule–caption translation, image captioning, and property prediction, but effective fusion of text, graphs, and images remains difficult. GIT-Mol responds with cross-attention, contrastive learning, and a unified latent representation.
- Prior molecular multimodal work: Molecular multimodal research includes molecule-caption translation, image captioning, and property prediction using paired text, images, graphs, and molecular representations.These task families connect molecular data across modalities for downstream prediction and generation.
- Molecule-caption translation: Molecule-caption systems learn shared semantic spaces or translate between SMILES expressions and textual descriptions using retrieval, pretraining, or contrastive learning.Examples include Text2Mol, KV-PLM, MolT5, MoleculeSTM, MoMu, and MolReGPT.
- Molecule image captioning: Molecule image-captioning systems encode molecular images and decode them into SMILES strings with rule-based or neural architectures.Representative systems use image encoders such as Vision Transformers or ResNets with recurrent or transformer decoders.
- Property prediction: Molecular property prediction uses graph-based and multimodal models, including contrastive, autoencoding, and 3D graph approaches.Examples include GraphCL, GraphMAE, Uni-mol, GraphMVP, MoMu, and MoleculeSTM.
- Open challenge: Effective fusion of text, graphs, and images remains challenging because existing approaches incompletely capture relationships between data sources.The paper identifies modal fusion and cross-source relational modeling as areas needing improvement.
- GIT-Mol response: GIT-Mol combines contrastive learning with cross-attention and modal fusion, extending BLIP2’s Q-Former-inspired design to broader molecular modality adaptation.The approach is positioned against methods such as MoMu’s graph encoder.
3. Methodology
GIT-Mol integrates molecular graphs, images, and text through GIT-Former, which aligns heterogeneous inputs in a unified latent space for molecular translation and prediction tasks.
- Data modalities: GIT-Mol combines molecular SMILES, graphs, images, and captions as internal and external representations of molecular information.SMILES and graphs encode intrinsic molecular structure, while images and captions provide more interpretable external information.
- GIT-Former architecture: GIT-Former uses modality-specific encoders and a decoder, including MolT5 for text, Swin Transformer for images, and GIN for molecular graphs.This multi-encoder setup adapts processing to the characteristics of each modality.
- Cross-modal alignment: Cross-attention aligns graph and image representations with molecular captions and SMILES strings while modeling inter-modal dependencies.Separate cross-attention weights are computed for each source modality to align it with the target text modality.
- Pre-training strategy: Pre-training combines self-supervised alignment, Xmodal-Text Matching, and Xmodal-Text Contrastive Learning to produce unified multimodal representations.XTM distinguishes matched from mismatched cross-modal text, while XTC contrasts similarities between corresponding and incorrect pairs.
- GIT-Former architecture: GIT-Former maps variable-length multimodal tensors into a fixed-length unified latent space for molecular-level alignment.The architecture is designed to integrate and adapt to different molecular modalities through cross-attention.
4. Results
GIT-Mol is evaluated across multimodal translation, generation, image recognition, property prediction, and embedding analysis. Results indicate benefits from multimodal inputs, pretraining, and prompt learning.
- Molecule captioning: The molecule captioning evaluation used BLEU and ROUGE-style NLP metrics on a dataset combining PubChem descriptions with ChEBI-20.The caption dataset was cleaned and incorporated images, SMILES strings, and captions.
- Molecule captioning: 10%-15% improvement over single-modality models was observed in molecule captioning, while the multimodal model outperformed graph- and SMILES-only variants.The results attribute the improvement to complementary information from each modality.
- Text-based molecule generation: 92.8% validity was achieved by GIT-Mol(caption), compared with 77.2% for MolT5-base, while retaining competitive molecular similarity.The reported metrics include MACCS FTS, RDK FTS, and Morgan FTS.
- Molecular image recognition: GIT-Mol outperformed SwinOCSR across all evaluated metrics in molecular image recognition, with prompt learning included as an experimental condition.The experiment also reports overfitting and slower convergence for the XTM-only pretraining strategy.
- Embedding analysis: Pretrained GIT-Former produced hierarchical separation of graph, image, and text embeddings, whereas the untrained model tended toward uniformity.The visualizations also examine atom distributions and K-means clusters across molecular data.
5. Discussion
The discussion emphasizes multimodal data, pretrained representations, and interpretability as important considerations for molecular modeling. Reported practical implications center on predictive accuracy, validity, and drug-discovery efficiency.
- Data quality: Multimodal data provides a more comprehensive perspective and can enhance model predictions, but representative benchmarks, validation subsets, and metrics remain necessary.The discussion frames data quality and diversity as central to molecular modeling.
- Model techniques: Pretrained models, distillation, and molecular-science-informed structures are presented as routes toward scalable and resource-efficient molecular modeling.The discussion mentions specially designed model layers or preprocessing steps for incorporating molecular knowledge.
- Scientific insights: Attention mechanisms and visual outputs can help reveal which data parts and structural components influence model predictions.This connects embedding visualization with interpretability of model decision-making.
- Practical implications: 5%-10% accuracy improvement in property prediction and 20.2% enhancement in molecule-generation validity are linked to the reported generalizability of the multimodal approach.The discussion identifies potential benefits for AI-aided drug discovery and time and cost efficiency.
6. Limitations
The paper identifies training speed and evaluation as limitations for GIT-Mol. These constraints become more significant as task complexity and model size increase.
- Training efficiency: Training speed remains a challenge despite the use of prompt learning, especially as tasks become more complex and the model grows.The authors consider Parameter-Efficient Fine-Tuning as a possible way to reduce training parameters and improve performance.
- Evaluation: Molecular captioning and generation lack appropriate evaluation methods, making the model’s effectiveness in these tasks difficult to assess.This is identified as a limitation of evaluating the model rather than as a claimed failure of generation itself.
7. Conclusions
GIT-Mol addresses molecular-science modeling challenges with a multimodal architecture and reports advantages across several tasks. Future work targets faster training and additional downstream applications.
- GIT-Mol is a specialized multimodal large language model for molecular science that handles scarce molecular captions and large-model training challenges.
- GIT-Former fuses multimodal molecular data, while any-to-language translation supports flexible applications across molecule captioning, generation, image captioning, and property prediction.
- Future work will explore LoRA, Adapters, prompting, compound name recognition, and chemical reaction prediction.
CRediT authorship contribution statement
The authors contributed across conceptualization, methodology, data curation, training, analysis, supervision, funding, and manuscript preparation.
- Pengfei Liu contributed conceptualization, methodology, data curation, model training, and the original draft.
- Yiming Ren contributed model training and the original draft, while Jun Tao contributed review and editing.
- Zhixiang Ren contributed conceptualization, formal analysis, supervision, funding acquisition, and review and editing.