PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards 3D Molecule-Text Interpretation in Language Models

Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, Qi Tian

arXiv:2401.13923v2cs.LGcs.IRq-bio.BM

TL;DR

Prior molecule-text work largely left 3D molecular structures untouched, despite their relevance to biomolecular phenomena. 3D-MoLM connects a 3D molecular encoder to a language model with a projector and 3D-MoIT instruction tuning. It outperforms baselines in retrieval, captioning, and open-text molecular QA, particularly on 3D-dependent properties.

  • Problem

    Earlier molecule-text modeling focused on 1D sequences and 2D graphs, mostly leaving 3D molecular structures untouched despite their importance to biomolecular phenomena.

    Source: §1 · p. 1

  • Method

    3D-MoLM connects a 3D molecular encoder to a language model through a molecule-text projector and uses 3D-MoIT for molecule-centric instruction tuning.

    Source: §1 · p. 2

  • Results

    20% accuracy for molecule-text retrieval and 6.47 ROUGE-L for molecule captioning are reported as improvements over PubChem baselines, with further gains over 1D/2D baselines on open-text QA, especially for 3D-dependent properties.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that 3D-MoLM performs across molecule-text retrieval, molecule captioning, and open-text molecular QA.

    Source: §4 · p. 9

    The 3D molecule-text dataset is notably constrained compared with vision-language datasets, and the study does not explore in-context learning or chain-of-thought reasoning.

    Source: §4 · p. 9

Abstract

from arXiv · show

Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We release our codes and datasets at https://github.com/lsh0520/3D-MoLM.

1 INTRODUCTION

Earlier molecule-text systems largely focused on 1D sequences and 2D graphs, leaving 3D structures unaddressed despite their importance to biomolecular phenomena. 3D-MoLM integrates a 3D molecular encoder with a language model through a projector and instruction-tunes it on 3D-MoIT.

1 INTRODUCTION

  • Earlier molecule-text work harmonized text with 1D sequences or 2D graphs but mostly left 3D structures untouched, although these matter for dynamics, protein-ligand interactions, and enzymatic functions.The introduction identifies these structures as relevant to a range of biomolecular phenomena.Source: §1 · p. 1
  • 3D-MoLM uses a 3D molecule-text projector to align a molecular encoder’s representations with the language model’s input space, enabling interpretation of 3D structures.The paper describes this alignment as bridging the modality gap between the encoder and LM.Source: §1 · pp. 1–2
  • The alignment stage uses Q-Former to convert 3D structures into tokens that serve as soft prompts, trained in representation-learning and molecule-text-alignment stages on 316K PubChem pairs.The data are enriched with GPT-3.5 descriptions, and conformations are generated from SMILES using RDKit.Source: §1 · p. 2
  • 3D-MoIT is an instruction-tuning dataset sourced from PubChem and PubChemQC to develop instruction following and perception of 3D-dependent molecular properties.PubChem descriptions are used to generate QA pairs, while PubChemQC contributes properties such as HOMO and LUMO.Source: §1 · p. 2
  • Reported results include 20% accuracy and 6.47 ROUGE-L, respectively; the model also surpasses baselines with 1D or 2D molecular perceptions on open-text QA tasks, especially on 3D-dependent properties.Source: §1 · p. 3

2 3D-MOLM: 3D MOLECULAR LANGUAGE MODELING

3D-MoLM integrates a 3D molecular encoder with a language model through a molecule-text projector. Its three-stage training pipeline uses PubChem for representation learning and alignment, then 3D-MoIT for instruction tuning.

2.1 MODEL ARCHITECTURE

  • Uni-Mol encodes 3D geometry into atom representations; Q-Former queries those features, and Llama2 processes molecular and text tokens to generate responses.The architecture comprises a 3D molecular encoder, a projector mapping its representations to the LM input space, and an LM adapted for molecular understanding.Source: §2.1 · pp. 3–4

2.2 MODEL TRAINING

  • PubChem supplies stages 1 and 2, while 3D-MoIT supplies stage 3 of the three-stage training pipeline.The stages are 3D molecule-text representation learning, generative alignment, and molecule-centric instruction tuning.Source: Figure 3 · p. 4, §2.2 · p. 4

2.2.1 3D MOLECULE-TEXT ALIGNMENT

  • Alignment uses 316K PubChem 3D molecule-text pairs: frozen-encoder, multi-objective representation learning precedes conditional generation with molecular tokens and textual prompts.Stage 1 trains the Q-Former with molecule-text matching, contrasting, and captioning; stage 2 connects it to the LM and uses SMILES plus a task description as prompts.Source: §2.2.1 · p. 5

2.2.2 3D MOLECULE-CENTRIC INSTRUCTION TUNING

  • Final tuning freezes the encoder and jointly optimizes the projector and LM with conditional text generation to improve instruction following and understanding of 3D-dependent properties.PubChemQC contributes DFT-determined 3D conformations and attributes described as mostly not inferable from 1D or 2D representations.Source: §2.2.2 · p. 5
  • 3D-MoIT combines PubChem descriptive QA and computed properties with PubChemQC's 3D-dependent HOMO, LUMO, HOMO-LUMO Gap, and SCF-energy values.PubChem descriptions are expanded into five QA pairs per molecule; Table 1 summarizes dataset statistics.Source: §2.2.2 · pp. 5–6, Table 1 · p. 5

2.2.1 3D MOLECULE-TEXT ALIGNMENT

  • Table 2 reports PubChem retrieval in molecule-to-text and text-to-molecule directions, with † marking pretraining without GPT-3.5 enrichment.The collected PubChem pairs also support downstream molecule-text retrieval and captioning.Source: §2.2.1 · p. 5, Table 2 · p. 6

3 EXPERIMENT

Experiments evaluate 3D-MoLM on molecule-text retrieval, molecule captioning, and open-text molecular QA, using 1D and 2D variants as comparisons. The reported results favor 3D-MoLM across these tasks, with qualifications for retrieval enrichment and generalist performance.

3.1 MOLECULE-TEXT RETRIEVAL

  • 3D-MoLM surpasses the 1D and 2D retrieval baselines by a large margin; the authors attribute gains to multi-objective Q-Former pretraining and a larger pretraining dataset.The curated PubChem collection has 301K molecule-text pairs, compared with 15K for the original MoMu.Source: §3.1 · p. 6
  • GPT-3.5 enrichment appears to hurt PubChem retrieval, which the authors infer may reflect a larger distribution gap, while they report generation benefits from enrichment.The retrieval texts without enrichment have word patterns more congruent with the test set; the authors also report enrichment gains for captioning and open-text QA.Source: §3.1 · pp. 6–7, §3.2 · p. 8

3.2 MOLECULE CAPTIONING

  • 3D-MoLM achieves the highest scores across all molecule-captioning evaluation metrics, while correctly identifying roles and primary structures despite one molecule-name error.The evaluated metrics are BLEU, ROUGE, and METEOR.Source: §3.2 · p. 8

3.3 OPEN-TEXT MOLECULAR QUESTION-ANSWERING (QA)

  • For descriptive-property QA, 3D-MoLM surpasses baselines and gives supplementary explanations, including proposed intermolecular-force reasons for beryllium acetate’s elevated melting and boiling points.It also correctly identifies beryllium acetate as a solid soluble in water.Source: §3.3 · p. 8
  • On computed-property QA, 3D-MoLM consistently achieves the lowest MAE, especially for properties intrinsically determined by 3D conformations.The reported lead among generalists on HOMO-LUMO Gap is 0.77 eV; advantages are more subdued for composition- and connectivity-based properties.Source: §3.3 · p. 8
  • In open-text QA, the generalist slightly underperforms specialists but still gains over other baselines; instruction tuning also improves Llama2-7B generalist by 2.46 BLEU-2 and 3.22 METEOR.The QA evaluation reports both specialists trained separately for each task and generalists trained on all generation tasks.Source: §3.3 · p. 8

4 CONCLUSION

The paper introduces 3D-MoLM, which equips an LM with a 3D molecular encoder and uses a projector to map molecular representations into the LM’s textual space. Experiments report strong performance across retrieval, captioning, and open-text molecular QA, while the authors note dataset-scale and capability-scope limitations.

4 CONCLUSION

  • 3D-MoLM equips an LM with a 3D molecular encoder and projector for interpreting 3D molecules through text.It also incorporates 3D molecule-centric instruction tuning to support instruction adherence and 3D molecular understanding.Source: §4 · p. 9
  • Experiments demonstrate strong performance across molecule-text retrieval, molecule captioning, and open-text molecular QA.Source: §4 · p. 9
  • The authors identify constrained 3D molecule-text dataset scale as a performance limitation and leave in-context learning and chain-of-thought reasoning unexplored.They also suggest invariant learning may help generalization and mitigate potential hallucinations.Source: §4 · p. 9

A RELATED WORKS

Prior molecule-text modeling mostly uses 1D sequences or 2D graphs, while multimodal instruction tuning and language models have expanded across other modalities. The paper situates its work at the intersection of these directions and the relatively unexplored 3D molecule-text domain.

Molecule-Text Modeling

  • Earlier molecule-text approaches use 1D molecular sequences, while later approaches add 2D graph representations, but spatial information in 3D conformations remains overlooked.Examples include KV-PLM, MolT5, and Mol-Instruct for 1D, and MoMu, MoleculeSTM, and MolCA for 2D.Source: A RELATED WORKS · p. 14

Multi-Modal Instruction Tuning

  • Multimodal instruction tuning has been applied to vision and 3D visual data, but its application to 3D molecule-text modeling remains largely unexplored.Examples discussed include InstructBLIP, LLaVA, MiniGPT-4, and 3D-LLM.Source: A RELATED WORKS · p. 14

Multi-Modal Language Models

  • Multimodal LMs align visual features with LM input spaces and have incorporated video, audio, and 3D point clouds, providing inspiration for 3D-MoLM.Source: A RELATED WORKS · p. 14

B PUBCHEM DATASET AND ENRICHMENT PROCESS

The PubChem preparation process enriches sparse descriptions for alignment pretraining while reserving original data for downstream evaluation. The resulting splits also depend on generated 3D conformations, which can differ from DFT-equilibrium structures.

B PUBCHEM DATASET AND ENRICHMENT PROCESS

  • Of 324,689 PubChem SMILES-text pairs, the shortest description has one word; a 15K subset with descriptions over 19 words was split into 12K/1K/2K train/validation/test pairs.The less informative remainder was reserved for alignment, and GPT-3.5 enrichment was applied only to pretraining so downstream evaluation used original real-world data.Source: B PUBCHEM DATASET AND ENRICHMENT PROCESS · p. 14
  • GPT-3.5-enriched descriptions were more comprehensive than original PubChem texts while maintaining high correctness, based on evaluations by chemistry experts and GPT-4.Human experts assessed 100 samples and GPT-4 assessed 1,000; GPT-4 enrichment was avoided because it would cost over $5,000.Source: B PUBCHEM DATASET AND ENRICHMENT PROCESS · p. 15
  • RDKit-generated conformations may be less accurate than costly DFT equilibrium conformations, and pairs for which RDKit failed were excluded.The resulting pretrain/train/validation/test splits contained 301,658/12,000/1,000/2,000 3D molecule-text pairs.Source: B PUBCHEM DATASET AND ENRICHMENT PROCESS · p. 15

C EXPERIMENT SETUPS

The experiments use staged pretraining and task-specific or generalist fine-tuning. The supplied setup passages specify training schedules and model configurations, but not downstream performance results.

C EXPERIMENT SETUPS

  • Stage 1 pretrains a Q-Former attached to a frozen 3D molecular encoder for 50 epochs with eight query tokens.The setup uses 40 A100 GPU hours.Source: C EXPERIMENT SETUPS · p. 15
  • Stage 2 continues from the Stage-1 checkpoint with 10 epochs of text-generation training and LoRA adapters comprising 19M trainable parameters, or 0.29% of Llama2-7B.Training used batch size 16, maximum token length 320, and 60 A100 GPU hours.Source: C EXPERIMENT SETUPS · p. 16
  • A generalist checkpoint is trained across captioning and descriptive, non-3D-centric, and 3D-centric property QA, sampling datasets with probabilities proportional to the fourth root of their sizes.It trains for 400,000 steps, and the lowest-validation-loss checkpoint is evaluated on all task test sets.Source: C EXPERIMENT SETUPS · pp. 16–17
  • Three open-text QA specialists are first instruction-tuned on all 3D-MoIT data, then fine-tuned on their corresponding task subsets for 5, 5, and 1 epochs.The shared instruction-tuning stage uses one epoch of all instruction data.Source: C EXPERIMENT SETUPS · p. 17

D FAILURE CASE STUDY

The failure cases expose limits in fine-grained molecular discrimination and formula generation. The authors identify stronger encoders, improved projectors, and finer-grained data as directions for refinement.

D FAILURE CASE STUDY

  • 3D-MoLM confuses Globostellatic acid C with B, which differ only in methoxy-branch placement and have remarkably similar PubChem records.The authors say finer-grained discrimination needs improvement despite accurate identification of primary functions and structures.Source: D FAILURE CASE STUDY · p. 17
  • 3D-MoLM reports Globostellatic acid C as C34H48O6 instead of C33H48O7; the passage also says GPT-4 cannot directly derive correct formulas from SMILES.The example illustrates atom counting and formula generation as a challenge for language models.Source: D FAILURE CASE STUDY · p. 17, Figure 7 · p. 18
  • The authors propose stronger 3D encoders, more effective projectors, and finer-grained alignment and instruction-tuning datasets as refinement directions.These directions target distinguishing similar structures and bridging molecular representations with LM space.Source: D FAILURE CASE STUDY · p. 17

E ABLATION STUDY ON INPUT PROMPT

The prompt ablation compares 3D molecular tokens, 1D SMILES, and their combination. The reported pattern favors using both input types together.

E ABLATION STUDY ON INPUT PROMPT

  • Combining 3D molecular tokens and 1D SMILES consistently outperforms using either input type alone.The comparison is reported in the prompt ablation table.Source: E ABLATION STUDY ON INPUT PROMPT · p. 18, Table 7 · p. 19
  • The authors attribute the combined-input pattern to SMILES retaining the LM’s 1D understanding while 3D tokens can focus on 3D-dependent properties.They liken the combination to adding features while retaining existing representation ability.Source: E ABLATION STUDY ON INPUT PROMPT · p. 18
  • The ablation compares stage-3-tuned Llama2-7B prompted with 1D SMILES against 3D-MoLM prompts containing both modalities or only 3D tokens.The table caption marks the 3D-only condition with ‡.Source: Table 7 · p. 19

F THE EFFECTIVENESS OF 3D PERCEPTION

The authors compare MolT5 models with and without 3D perception while maintaining the same model size. Across all scales, MolT5s equipped with 3D perception outperform those without it.

F THE EFFECTIVENESS OF 3D PERCEPTION

  • The comparison integrates the 3D molecular encoder and projector into MolT5, maintaining the same model size.This setup is intended to eliminate the model-scale advantage in comparing Llama2 7B with T5 780M.Source: F THE EFFECTIVENESS OF 3D PERCEPTION · p. 19
  • MolT5s equipped with 3D perception consistently surpass those lacking 3D perception across all scales.The comparison integrates the 3D molecular encoder and projector into MolT5 to maintain the same model size.Source: F THE EFFECTIVENESS OF 3D PERCEPTION · p. 19
  • The authors interpret the across-scale advantage as substantiating that 3D geometry perception enhances language models’ molecular understanding.Source: F THE EFFECTIVENESS OF 3D PERCEPTION · p. 19

G LIMITATIONS AND FUTURE DIRECTIONS

The authors identify limits in task scope, fine-grained structural understanding, dataset scale, and use of LM capabilities, alongside potential evaluation bias from pretraining-data overlap. They point to these constraints as directions for future work.

G LIMITATIONS AND FUTURE DIRECTIONS

  • Experimental-procedure prediction, retrosynthesis, and drug-drug interaction are beyond this study’s scope, partly because openly available datasets are lacking.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 19
  • 3D-MoLM has difficulty discerning fine-grained small molecular structures, a limitation relevant to tasks requiring precise 3D understanding.Suggested directions include coordinate references in text and more powerful 3D encoders or projectors.Source: G LIMITATIONS AND FUTURE DIRECTIONS · pp. 19–20
  • The datasets comprise PubChem 300K and PubChemQC 3M, a relatively limited scale that may impede models from reaching their full potential.The paper contrasts these with multimodal datasets exceeding 10 million paired samples.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 20
  • The study does not leverage in-context learning or chain-of-thought reasoning, primarily because specialized step-by-step inference datasets are lacking.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 20
  • Pretraining-corpus overlap with test datasets can introduce bias and inflate molecular-language-model performance metrics.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 20
  • The authors present addressing these constraints as a route toward a more holistic and accurate understanding of 3D molecule-text interpretation.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 20

Related papers

Loading 2401.13923v2…