MolXPT: Wrapping Molecules with Text for Generative Pre-training

Motivation

• MolXPT targets models that jointly process molecules and scientific text but do not explicitly leverage molecule–text relations in surrounding literature.The paper argues that context around a molecule name can describe its properties and is useful for joint training.Source: §1 · p. 1

Approach

• The method detects molecular names in text and replaces matched mentions with corresponding SMILES to create wrapped sequences.These sequences let SMILES and surrounding scientific text appear together during pre-training.Source: §1 · p. 1, Figure 1 · p. 2

Pre-training

• MolXPT is pretrained on 8M wrapped sequences, 30M PubChem SMILES, and 30M PubMed titles and abstracts.The model uses a 24-layer, 350M-parameter architecture.Source: §1 · pp. 1–2

Results

• MolXPT outperforms strong MoleculeNet baselines and performs comparably to MolT5-large on molecule–text translation while using 44% as many parameters.The model also enables zero-shot text-to-molecule generation without finetuning.Source: §1 · p. 2

2 Our Method

• MolXPT integrates scientific text, molecular SMILES, and wrapped text–SMILES sequences in one pre-training framework.The framework is intended to support both molecular and text downstream tasks.Source: §2 · p. 2

2.1 Pre-training corpus

• The corpus contains 30M PubMed titles and abstracts, 30M randomly selected PubChem molecules, and 8M wrapped sequences.Wrapped sequences are retained only when they contain at least one molecular SMILES.Source: §2.1 · p. 2
• The detect-and-replace pipeline uses biomedical NER and entity linking to map molecular mentions to knowledge-base entities and substitute their SMILES.BERN2 detects mentions, ChEBI provides linked entities, and the corresponding SMILES are retrieved.Source: §2.1 · p. 2

2.2 Model and training

• MolXPT follows a GPT architecture with 24 layers, 1024 hidden units, 16 attention heads, a 2048-token context, and 350M parameters.Text uses BPE tokenization, while SMILES use a molecular regular-expression tokenizer with start- and end-of-molecule tokens.Source: §2.2 · p. 2, §2.1 · p. 2
• The pre-training objective is negative log-likelihood over sequences drawn from the three data types.Prompt-based finetuning converts molecular and text tasks into prompted sequence generation using language-modeling loss.Source: §2.2 · pp. 2–3

3.1 Results on MoleculeNet

• Prompt-based finetuning converts molecular property prediction into sequence generation by placing task-specific classification tags at the prompt’s final token.For BBBP, the generated tag is “true” or “false,” indicating whether a compound crosses the blood-brain barrier.Source: §3.1 · pp. 3–4
• MolXPT outperforms GNN baselines pretrained on pure molecular data and Galactica on MoleculeNet classification tasks.The evaluation uses six scaffold-split MoleculeNet tasks and ROC-AUC, with results reported in Table 1.Source: §3.1 · pp. 3–4

3.2 Results on text-molecule translation

• MolXPT performs better than MolT5-small and MolT5-base and comparably to MolT5-large on text–molecule translation.MolT5-large has 800M parameters, whereas MolXPT uses 44% as many; MolXPT also performs best on the Text2Mol metric for both directions.Source: §3.2 · p. 4
• Wrapped sequences are attributed to MolXPT’s stronger text–molecule alignment because they let the model learn relations between molecules and text explicitly.The translation evaluation includes NLP metrics for molecule-to-text generation and exactness, fingerprint similarity, FCD, and validity for text-to-molecule generation.Source: §3.2 · p. 4
• Without finetuning, pretrained MolXPT generates molecules from text descriptions and recovers 33 molecules, although performance drops relative to full-data finetuning.Zero-shot quality is assessed using top-1 and top-5 fingerprint similarity.Source: §3.2 · p. 4

4 Conclusions and Future Work

• MolXPT is pretrained on scientific text, molecular SMILES, and wrapped sequences combining the two modalities.Source: §4 · p. 4
• Table 2 reports molecule-to-text results in its top section and text-to-molecule results in its bottom section, with FCD minimized and other metrics maximized.Source: Table 2 · p. 5
• Table 3 reports zero-shot text-to-molecule generation results.Source: Table 3 · p. 5
• MolXPT, a 24-layer model with 350M parameters, improves strong MoleculeNet baselines and achieves comparable results to the best molecule–text translation model while using much fewer parameters.Source: §4 · p. 5
• Future work includes scaling MolXPT to further verify performance and zero-shot or in-context learning across tasks.The authors also identify stronger molecule–text interaction and adaptation to additional tasks as future directions.Source: §4 · p. 5

A Datasets and Baselines of MoleculeNet

• The evaluation uses BBBP, Tox21, ClinTox, HIV, BACE, and SIDER to assess diverse molecular properties.The tasks cover blood–brain barrier penetration, toxicity, clinical outcomes, HIV replication inhibition, β-secretase 1 binding, and side effects.Source: A Datasets and Baselines of MoleculeNet · p. 7
• Baselines include GROVER variants, GraphMVP, MGSSL, GEM, Galactica-120B, KV-PLM, and MoMu.These methods span graph, GPT-like, BERT-like, and joint text–molecule pretraining approaches.Source: A Datasets and Baselines of MoleculeNet · p. 7

B Pre-training hyper-parameters

• Pretraining runs for 200k steps on eight A100 GPUs with 2048 tokens per GPU and 16-step gradient accumulation.Source: B Pre-training hyper-parameters · p. 7
• Adam optimization uses a 0.0005 peak learning rate, 20,000 warm-up steps, inverse-square-root decay, and 0.1 dropout.Source: B Pre-training hyper-parameters · pp. 7–8

C Finetuning details of downstream tasks

• MoleculeNet finetuning prompts convert molecular property prediction into natural-language conclusions with task-specific true/false or active/inactive outputs.Source: §C.1 · p. 8
• The prompts cover BBB penetration, Tox21 target activity, ClinTox toxicity and FDA approval, HIV inhibition, BACE binding, and SIDER side effects.Source: §C.1 · p. 8

• Learning rates of 3 × 10−5 and 5 × 10−5, dropout rates of 0.1 and 0.3, and 30 or 50 epochs are grid-searched.Source: §C.2 · p. 8
• The model is selected according to validation performance after searching these hyper-parameters.Source: §C.2 · p. 8

C.3 Details of finetuning text-molecule generation

• Text-molecule generation uses 100 epochs, a 0.0001 learning rate, and dropout grid search over [0.1, 0.2, 0.3, 0.4, 0.5].Dropout 0.4 performs best for molecule-to-text generation, whereas 0.5 performs best for text-to-molecule generation.Source: §C.3 · p. 8
• MolXPT is finetuned for text-molecule generation for 100 steps on one P40 GPU with 1024 tokens and 16 accumulated steps per device.Source: §C.3 · p. 8

C.4 MoleculeNet finetuning strategy selection

• The two MoleculeNet finetuning strategies produce similar results, with Eqn.(1) slightly better.Source: §C.4 · p. 8

• Top-K fingerprint similarity compares K generated molecules against a reference molecule.Source: D Zero-shot text-to-molecule generation · p. 8
• 33 generated molecules exactly match reference molecules without finetuning, with Figure 2 showing three examples.Source: D Zero-shot text-to-molecule generation · p. 8, Figure 2 · p. 9
• Figure 2 presents three randomly selected cases where MolXPT successfully generates the reference molecules without finetuning.Source: Figure 2 · p. 9