Source-linked AI summary
MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models
Xinjian Zhao, Xiangru Jian, Yaoyao Xu, Xiaozhuang Song, Wei Pang, Lei Bai, Tianshu Yu
TL;DR
Molecular encoders commonly produce fixed-view, unconditional vectors, motivating whether MLLMs can provide reusable embeddings conditioned on molecular profiles and semantic context. MolEmb adapts an MLLM with molecule–text contrastive alignment, and the resulting route supports prediction, cross-modal retrieval, and context-aware embedding, with supervision strongly shaping context awareness.
Problem
Specialist molecular encoders usually produce fixed-view, unconditional vectors, leaving limited support for multi-view and semantically conditioned molecular representations.
Method
MolEmb adapts a pretrained MLLM by aligning multi-view molecular profiles and textual descriptions in a shared embedding space with a bidirectional contrastive objective.
Results
The MLLM-to-embedding route supports competitive molecular prediction, cross-modal molecule–text retrieval, and context-aware retrieval, while generic description alignment alone is insufficient for reliable task-conditioned routing.
Takeaways & Limitations
MLLMs provide a viable and extensible route to general molecular embedding models, but context-aware molecular embedding is primarily a property of the supervision.
Takeaways & Limitations
MolEmb uses only 2D depictions and canonical SMILES, while MolCAR does not test unseen task families or open-ended scientific search.
Abstract
from arXiv · showhide
Molecular embedding models can serve as foundational infrastructure for computational chemistry and drug discovery, where reusable vector representations support property prediction, virtual screening, and retrieval. Most molecular encoders are specialist models built around a single molecular view, producing unconditional vectors with no language interface for varying the representation. We ask whether multimodal large language models (MLLMs), which natively process images, text, and symbolic inputs, can instead serve as \emph{general molecular embedding models} that produce embeddings conditioned on both a molecular profile and a natural-language semantic context. We introduce \textbf{MolEmb}, a lightweight framework that adapts MLLMs by aligning molecular profiles with textual descriptions in a shared embedding space using a bidirectional contrastive objective. The resulting embedding model is competitive on molecular property prediction and supports cross-modal molecule--text retrieval in the same space. We further introduce \textbf{MolCAR}, a diagnostic benchmark for context-aware retrieval, and find that context-aware molecular embedding is primarily a data property of the supervision. These results suggest that MLLMs are not merely chemistry assistants or generators, but a viable and extensible route to general molecular embedding models.
1 Introduction
Molecular embedding models offer reusable infrastructure, but specialist encoders usually produce fixed-view, unconditional vectors. MolEmb adapts MLLMs to provide multi-view, semantically conditioned embeddings evaluated across prediction and retrieval tasks.
- Molecular embedding models provide reusable representations for property prediction, virtual screening, similarity search, and retrieval-augmented reasoning.
- Specialist molecular encoders typically map one molecular view to a single unconditional vector, limiting variation by view or semantic context.
- General molecular embedding models accept complementary molecular views and semantic contexts so the same molecule can receive task-relevant representations.
- MolEmb adapts MLLMs by aligning multi-view molecular profiles with textual descriptions in a shared embedding space.
- The evaluation covers molecular prediction, cross-modal molecule–text retrieval, and context-aware retrieval, including the MolCAR diagnostic benchmark.
2 Related Work
Molecular representation learning has chiefly relied on specialist graph, sequence, and geometry encoders, while recent work explores repurposing generative backbones as embedding models. Figure 1 places MolEmb within this shift toward general multimodal molecular embeddings.
- Molecular Representation Learning: Graph-based encoders represent atoms and bonds and use self-supervised objectives such as attribute masking, context prediction, and graph contrastive learning.
- Molecular Representation Learning: Sequence-based molecular language models treat SMILES as chemical text and apply scalable Transformer pretraining.
- Molecular Representation Learning: Geometry-based approaches incorporate conformers, distances, or coordinate denoising to capture spatial molecular information.
- From Generative Models to Embedding Models: Recent work repurposes generative language and vision-language backbones for retrieval and representation learning.
- From Generative Models to Embedding Models: Figure 1 presents MolEmb as an MLLM-to-embedding route supporting multi-view inputs, natural-language instructions, and reusable downstream workflows.
3 General Molecular Embedding Models
The paper defines general molecular embeddings as vectors formed from available molecular views and semantic context, then instantiates them with an MLLM. MolEmb adapts the backbone through lightweight molecule–text alignment.
- 3.1 Definition: A general molecular embedding model maps a molecular profile and semantic context to a vector in a shared embedding space.
- 3.1 Definition: The molecular profile space represents available views such as 2D depictions, symbolic strings, conformers, and molecule-specific annotations.
- 3.1 Definition: The formulation separates molecular identity, available observations, and the semantic lens used to form an embedding family.
- 3.1 Definition: In this paper, profiles use a 2D depiction and canonical SMILES, while contexts are expressed through natural-language instructions.
- 3.2 MLLMs as a Natural Instantiation: MLLMs provide a heterogeneous-input backbone in which formatted molecular views and context produce hidden states that pooling converts into fixed-length vectors.
- 3.3 MolEmb: Adapting MLLMs through Molecule–Text Alignment: MolEmb keeps the pretrained backbone fixed, adds a lightweight trainable adapter, and aligns molecule and description embeddings with a bidirectional contrastive objective.
- 3.3 MolEmb: Adapting MLLMs through Molecule–Text Alignment: The aligned model supports property prediction, shared-space cross-modal retrieval, and context-aware embeddings by varying the semantic context.
4 Experiments
The experiments test whether MLLM-derived embeddings support molecular prediction, cross-modal retrieval, and context-aware retrieval. MolEmb provides competitive prediction representations, strong molecule–text retrieval after alignment, and context sensitivity primarily when supervision includes task-diverse, outcome-grounded examples.
- Experimental scope: MolEmb is evaluated as a general molecular embedding model across prediction, shared-space retrieval, and context-aware retrieval.The evaluation follows three empirical requirements: reusable prediction, a shared molecule–text space, and non-degenerate context-conditioned embeddings.
- Molecular Property Prediction: Directly adapted MLLM embeddings reach the range of several pretraining-based GNN methods, especially on regression tasks, but direct training remains property-specific.This establishes usefulness for molecular property learning without yet demonstrating reusability beyond a single task head.
- Molecular Property Prediction: Alignment gains depend on data diversity and coverage: Mixed alignment improves some endpoints, notably ESOL and Tox21, while MolTextNet remains stronger on others.Richer scientific coverage helps transfer, but generic description corpora do not span the full diversity of task-conditioned molecular knowledge.
- Cross-Modal Retrieval: After MolTextNet contrastive alignment, all MolEmb backbones exceed 73% R@1 in both retrieval directions, with Intern-S1-mini and Qwen3.5-0.8B exceeding 83%.The off-the-shelf Qwen3-VL-Embedding-8B reaches only about 3% R@1 in both directions, whereas alignment produces a reliable chemical molecule–text space.
- Context-Aware Embedding with MolCAR: Generic molecule–text alignment leaves same-molecule contexts poorly separated, but continued alignment raises Context R@1 to 99.8% for Intern-S1-mini and 69.4% for Qwen3.5-0.8B.The baseline context-level rate is 40.4%, and continued alignment expands embeddings along task-meaningful directions.
- Context-Aware Embedding with MolCAR: Context-aware embedding is primarily a data property: task-diverse, outcome-grounded molecule–text supervision induces instruction-sensitive structure in the embedding space.MolCAR holds molecule identity fixed while varying task lenses, exposing whether instructions route retrieval to the appropriate evidence.
5 Conclusion
MolEmb adapts multimodal large language models into reusable molecular embedding models using image, SMILES, and text context representations aligned with a contrastive objective. Results across prediction, retrieval, and context-aware retrieval support the MLLM interface while showing that generic molecule-description alignment alone does not reliably route task-conditioned queries.
- MolEmb represents each molecule through an image depiction, SMILES, and text context, then aligns molecule and text embeddings with a contrastive objective.
- The MLLM interface supports molecular property prediction, molecule–text retrieval, and context-aware retrieval.
- Generic molecule-description alignment alone is insufficient for reliable task-conditioned routing.
- More diverse, outcome-grounded, and task-aware molecular text corpora may be as important as changing backbone scale or architecture.
A Limitations
MolEmb is evaluated as an adaptation route rather than a fully scaled molecular foundation model, using limited molecular views and a controlled retrieval diagnostic. Its context-aware evaluation does not test unseen task-family transfer or open-ended scientific search, and the supervision remains small relative to industrial pretraining.
- MolEmb uses only 2D depictions and canonical SMILES, leaving conformers, reactions, spectra, and assay metadata for future profile maps.
- MolCAR holds molecule identity fixed across eight property-related task families but does not test transfer to unseen task families or open-ended scientific search.
- The Mixed corpus broadens semantic coverage, but its scale and diversity remain small relative to industrial embedding pretraining.
- Broader context-aware embedding will require larger, more carefully curated molecule–text supervision.
- The study evaluates an adaptation route rather than a fully scaled molecular foundation model.
C.1 Downstream Prediction Datasets
The downstream prediction evaluation uses eight OGB/MoleculeNet datasets with standard scaffold splits. It covers regression and classification tasks, using RMSE for regression and ROC-AUC for classification, with macro-averaging for multi-label datasets.
- Eight downstream datasets are drawn from the OGB/MoleculeNet benchmark suite and use standard scaffold splits.
- Regression datasets are ESOL, Lipophilicity, and FreeSolv, evaluated with root mean squared error.
- Classification datasets are BACE, BBBP, ClinTox, Tox21, and SIDER, evaluated with ROC-AUC.
- Tox21 and SIDER use macro-averaged ROC-AUC across valid labels, while ClinTox averages ROC-AUC over two clinical trial outcome labels.
C.2 MolCAR Retrieval Metrics
MolCAR evaluates retrieval over molecule–context pairs using molecule-level, document-level, and context-level metrics under instructed or generic query variants. Context R@1 isolates whether task instructions route a query to the correct task view for an already identified molecule.
- MolCAR evaluates five retrieval metrics over a fixed candidate pool containing all evaluation molecule–context documents.
- Mol. R@1 measures whether the ground-truth molecule appears at rank 1, rather than whether the exact document is matched.
- Doc. R@1, Doc. MRR, and Doc. R@5 measure exact-document rank 1, reciprocal rank, and top-5 retrieval, respectively.
- Context R@1 ranks the correct context first among documents sharing the same molecule, isolating instruction sensitivity from molecule identification.
- The aggregate random baseline for Context R@1 is approximately 40.4% because molecules have different numbers of available task contexts.
- MolCAR-Structured holds out molecules by exact canonical-SMILES matching and evaluates 4,092 molecule–context pairs over 1,653 held-out molecules.
- MolCAR-Train uses task-conditioned textual targets from the same eight datasets, while MolCAR-Natural replaces structured documents with generated scientific notes anchored to the same outcomes and profiles.
- Queries vary task names across eight families, with NoInst replacing task-specific instructions by a generic encoding instruction while retaining the SMILES and image inputs.
D Implementation Details
MolEmb uses EOS pooling and bidirectional contrastive alignment with shared-backbone molecule and text encodings. Its experiments standardize corpus splitting, deduplication, adapter training, and continued alignment across backbones and MolCAR variants.
- Embedding extraction: EOS pooling extracts the hidden state at the end-of-sequence token as the fixed-length molecular or text embedding.No additional mean or CLS pooling is applied.
- Alignment objective: Bidirectional InfoNCE alignment uses a shared MLLM backbone, ℓ2-normalized representations, cosine similarity, and temperature τ = 0.07.Molecular queries and text targets are encoded in separate forward passes.
- Data construction: The Mixed corpus globally deduplicates canonical-SMILES overlaps after merging three alignment-pretraining corpora.The shared 98/1/1 molecule–text-pair split uses seed 42, and merging drops 5,041 duplicate pairs.
- Optimization: Alignment pretraining uses AdamW, bfloat16 precision, LoRA on language and vision components, learning rate 1 × 10^-4, and 448-pixel images.LoRA rank is 4 for Qwen3.5-0.8B and 8 for Intern-S1-mini and Qwen3-VL-8B, with α = 2r.
- Downstream adaptation: Property-prediction adaptation adds a lightweight task head, inheriting alignment-pretrained LoRA weights for MolTextNet and Mixed while Direct starts fresh.The same target groups and per-backbone rank/alpha are used across conditions.
- Continued alignment: Continued alignment initializes from an alignment-pretrained embedding model and separately trains Structured and Natural MolCAR checkpoints with matched target styles.The Structured pass supplies Table 3, while the Natural pass supplies Table 6.
E.1 MolCAR-Natural Results
MolCAR-Natural replaces structured target cards with scientist-facing notes while preserving molecule identity, task labels, and observed outcomes. Continued alignment is evaluated against the same Inst/NoInst/∆ protocol and is also examined through controlled retrieval and t-SNE comparisons.
- MolCAR-Natural construction: MolCAR-Natural broadens target-text style by replacing structured cards with scientist-facing notes generated by Intern-S1 (235B).The molecule, task label, and observed result remain fixed while the target text changes.
- Evaluation protocol: Table 6 reports Inst, NoInst, and ∆ results for Base and +Continued Alignment on the same 4,092-document, 1,653-molecule candidate pool.∆ is computed from unrounded scores, so displayed values may differ slightly through rounding.
- Retrieval preservation: Continued alignment improves MolTextNet m2t/t2m R@1 rather than degrading retrieval for both Intern-S1-mini and Qwen3.5-0.8B.The Mixed-aligned checkpoint is compared with its MolCAR-continued version, with MolTextNet-only shown as a reference.
- Controlled instruction probe: The controlled instruction probe expands each fixed molecule into all eight MolCAR task instructions so geometry changes reflect instruction conditioning rather than sample-composition drift.The probe compares generic and task-instructed queries before and after continued alignment.
- Embedding geometry: After continued alignment, the Intern-S1-mini task-instructed t-SNE panel shows clear task-wise separation while the generic-query panel remains comparatively overlapped.Before continued alignment, generic and instructed panels are visually similar, consistent with close Inst/NoInst Context R@1 scores.
E.4 Multi-view Input Ablation
The multi-view input ablation compares a full molecular profile combining a 2D depiction with canonical SMILES against SMILES-only input. The full profile yields modest gains on most property-prediction tasks, while MolCAR examples illustrate task-conditioned targets for the same molecule.
- Multi-view ablation: Using both a 2D depiction and canonical SMILES produces modest gains on most regression and classification tasks for Intern-S1-mini under direct adaptation.The comparison is against a SMILES-only baseline.
- Alignment examples: Alignment-pretraining examples preserve the multimodal molecule-side query, including an explicit image token, alongside the full text-side target.The examples pair reusable molecular-understanding instructions with structure and physicochemical descriptions.
- MolCAR construction: The MolCAR construction holds molecule identity and observed outcomes fixed while varying task lenses and target-text style between structured cards and natural notes.Both instruction and generic no-instruction queries are shown to test whether task language changes retrieval beyond molecule identity alone.
- Query conditioning: MolCAR pairs an instruction query for blood-brain barrier penetration with a generic query for the same BrC(Br)Br molecular input.The paired queries isolate the effect of semantic task conditioning.
- Structured target cards: The structured MolCAR example uses the same molecular profile with separate targets for barrier penetration, water solubility, hydration free energy, and Tox21 toxicity outcomes.Candidate cards report task-specific observations while retaining shared molecular descriptors such as molecular weight, cLogP, HBD, HBA, and formal charge.
- Natural target notes: Natural candidate notes express the same task outcomes in scientist-facing prose, including barrier penetration, solubility, hydration, and toxicity descriptions.These notes retain shared molecular descriptors while presenting task-specific interpretations in free-form text.