PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Adapted Molecular Language Models for Efficient Search of Make-on-Demand Libraries

Henrik Wille, Luis-Finley Schütz, Felix Strieth-Kalthoff

arXiv:2608.17567v1cs.LGcs.AIcs.CL

TL;DR

How well pretrained molecular representations support discovery across libraries in different chemical domains remains unclear. The study benchmarks molecular-language-model representations and library-specific adaptation, finding substantial embedding-space contraction in non-drug libraries.

  • Problem

    How representation–domain fit relates to discovery efficiency remains unclear, as target libraries may be shifted or compressed in pretrained embedding spaces.

    Source: Introduction · p. 2

  • Method

    The study benchmarks discovery efficiency across six virtual libraries and tests domain adaptation of molecular-language-model representations using target-library structures without costly property labels.

    Source: Introduction · p. 2, Native molecular language model… · p. 5

  • Results

    Embedding-space pairwise distances contracted substantially for non-drug libraries across molecular language models, with SmiTed producing particularly narrow distributions.

    Source: Results and Discussion · p. 3

  • Takeaways & Limitations

    Embedding dispersion is a candidate diagnostic of representation–domain mismatch, but a narrow embedding region does not necessarily imply information loss for learning structure–property relationships.

    Source: Results and Discussion · p. 3

    The benchmark relies on target-property values being available for all molecules in each library.

    Source: §1.1 · p. 11

Abstract

from arXiv · show

Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practical suitability for molecular discovery within and beyond their pretraining domain remains unclear. Herein, we systematically benchmark four molecular language models across six virtual molecular libraries spanning drug discovery, organic materials, and catalysis. Native molecular language model embeddings show substantial variation in discovery performance across libraries, whereas molecular fingerprints provide a consistently strong and robust baseline. Consistent with a potential domain-representation mismatch, we show that explicit domain adaptation substantially improves representation performance. Fine-tuning molecular language model encoders on structures from the target virtual library consistently improves sample efficiency, with several adapted encoders emerging as the top-performing representations across the benchmark tasks. These results show that molecular representation quality depends strongly on the target domain and that explicit adaptation can improve the practical utility of molecular foundation models. More broadly, our findings establish domain-adapted molecular representations as a promising strategy for sample-efficient adaptive decision making in virtual screening and self-driving laboratories.

Introduction

The paper frames representation fit to the target library as a potential constraint on sample-efficient discovery, then benchmarks whether structure-only domain adaptation can help. In the supplied results, non-drug libraries show narrower embedding distributions, treated as a candidate diagnostic rather than proof of lost predictive information.

Introduction

  • Because virtual libraries can differ from pretraining domains, candidate molecules may be insufficiently differentiated, limiting structure–property learning, especially around property cliffs.A distribution shift or embedding contraction alone does not prove a representation inadequate; quality depends on support for downstream surrogate modeling and decision-making.Source: Introduction · p. 2
  • The study benchmarks whether adapting molecular encoders to library structures can improve discovery efficiency across virtual-screening and experimental budgets.The proposed adaptation uses only candidate-library molecular structures and does not require costly property labels.Source: Introduction · p. 2

Results and Discussion

  • The benchmark spans six virtual libraries across drug discovery, materials chemistry, and catalysis.The tasks are Zinc-AmpC (~1.9 million molecules), RO4-OGG1 (~1.5 million), Enamine-TMPK (~2.1 million), HCEP-PCE (~2.3 million), Laser-OSC (~180,000), and Kraken-ES (~300,000).Source: Results and Discussion · p. 2
  • Materials- and catalysis-focused libraries show an expected domain shift from drug-like ZINC15 space, while three drug-discovery libraries substantially overlap the reference.The 25,000 randomly selected ZINC15 molecules serve as a proxy for drug-like chemical space represented in common pretraining datasets.Source: Results and Discussion · p. 3, Figure 2 · p. 3
  • The study examines four pretrained molecular language models with distinct transformer architectures and pretraining datasets.The models are ChemBERTa, MolFormer, T5Chem, and SmiTed.Source: Results and Discussion · p. 3
  • Molecular language model embeddings contracted substantially for non-drug libraries, but narrow dispersion alone does not establish information loss for learning structure–property relationships.Similar dispersion appeared across all models, with particularly narrow distributions for SmiTed; the authors treat embedding dispersion as a candidate diagnostic and evaluate its relationship to discovery efficiency later.Source: Results and Discussion · p. 3, Figure 2 · p. 3

Native molecular language model representations show variable performance across tasks.

Native molecular language model embeddings vary substantially across discovery libraries, while Morgan fingerprints provide a robust baseline. Library-specific adaptation improves language-model performance, allowing adapted ChemBERTa, MolFormer, and T5Chem embeddings to exceed that baseline.

Native molecular language model representations show variable performance across tasks.

  • Morgan fingerprints consistently ranked among the strongest native representations, while all other tested representations had negative global posterior effects relative to this baseline.Native MolFormer and T5Chem approached fingerprint performance in several libraries but did not consistently outperform fingerprints; ChemBERTa and especially SmiTed had more negative effects.Source: Native molecular language model… · p. 5
  • Representation ordering was broadly consistent across virtual-screening and experimental-discovery budgets, but differences were less pronounced under the lower experimental budget.Morgan fingerprints, MolFormer, and T5Chem had overlapping global posterior effects in the experimental regime; the authors suggest initial-sample variability and exploration may contribute, while noting this hypothesis was not formally tested.Source: Native molecular language model… · p. 5

Conclusions

  • Representation–domain fit explained little of the performance variation: language models performed relatively efficiently in some libraries despite large shifts, and distance-based correlations were weak.For example, T5Chem performed relatively efficiently in Laser-OSC and MolFormer in Kraken-ES despite substantial distances from the ZINC15 reference library.Source: Native molecular language model… · p. 5, Conclusions · p. 7

Native molecular language model representations show variable performance across tasks.

  • The three adaptation strategies were continued pretraining, contrastive learning, and fingerprint-supervised auxiliary-label training, with encoder updates performed using LoRA.Contrastive adaptation improved on native embeddings but gained less than auxiliary-label fine-tuning; continued pretraining with the original objective produced significantly smaller changes.Source: Native molecular language model… · pp. 5–6
  • Auxiliary-label adaptation produced the largest average gains, with adapted ChemBERTa, MolFormer, and T5Chem embeddings showing positive global effects relative to Morgan fingerprints.The gains followed broadly similar trends across several libraries and acquisition-budget regimes.Source: Figure 4 · p. 6, Native molecular language model… · p. 6
  • Domain adaptation reduced or reversed ChemBERTa’s native negative effects and improved MolFormer and T5Chem, whereas adapted SmiTed retained a negative effect relative to Morgan fingerprints.The findings indicate that even a compact encoder can yield a useful optimization embedding after adaptation, despite an unsuitable native embedding space.Source: Native molecular language model… · p. 6

Conclusions

  • Fingerprint-supervised adaptation requires only molecular structures sampled from the target library, so it can be performed at the outset of a discovery campaign.The conclusions are limited by the range of benchmark libraries, and broader validation across molecular distributions and target properties would be valuable.Source: Conclusions · p. 7

Supplementary Information · Domain-Adapted Molecular Language Models

The supplementary-information front matter names Henrik Wille, Luis-Finley Schütz, and Felix Strieth-Kalthoff and lists their University of Wuppertal affiliations.

for Efficient Search of Make-on-Demand Libraries

  • Henrik Wille, Luis-Finley Schütz, and Felix Strieth-Kalthoff are named in the supplementary-information front matter.The author line marks Wille and Schütz with † and Strieth-Kalthoff with an asterisk; the passage does not explain these symbols.Source: for Efficient Search of Make-on-Demand… · p. 10
  • The listed affiliation a is the University of Wuppertal’s School of Mathematics and Natural Sciences in Wuppertal, Germany.The passage gives the address as Gaußstr. 20, 42119 Wuppertal.Source: for Efficient Search of Make-on-Demand… · p. 10
  • The listed affiliation b is the University of Wuppertal’s Interdisciplinary Center of Machine Learning and Data Analytics.The passage gives Gaußstr. 20 as the street address.Source: for Efficient Search of Make-on-Demand… · p. 10

1 Experimental Setup

The experimental setup describes a benchmark comparing molecular representations and iterative optimization methods on virtual libraries with known target properties, including library-specific fine-tuning. It outlines encoder adaptation, surrogate prediction, and acquisition-based candidate selection.

1.1 General Procedure: Iterative Optimization

  • Optimization repeatedly trained a surrogate on observed molecules, selected batches by acquisition value, and stopped when the observation budget was exhausted.Experiments used 20 repeats with shared initial datasets across methods; performance was assessed by top-candidate recall trajectories and normalized area under the optimization curve (AUOC).Source: §1.1 · p. 11

1.2 Heuristic Molecular Representations

  • The heuristic baselines comprised Morgan fingerprints, Mordred descriptors, and RDKit descriptors.Morgan fingerprints used radius 2 and 2048 bits; Mordred descriptors were restricted to two-dimensional structure, with zero-variance descriptors excluded.Source: §1.2 · p. 13

1.3 Molecular Language Models

  • Four pretrained molecular language models supplied embeddings, and library-specific adaptation added a projection head and LoRA fine-tuning.The tested fine-tuning tasks included masked-token language modeling on MolFormer and a weighted combination of language-modeling and contrastive losses.Source: §1.3 · pp. 13–14
  • Molecular encoders used fingerprint-similarity contrastive pairs (positive Tanimoto >0.7; negative <0.1) and supervised Morgan-fingerprint reconstruction (radius 2, length 1024).The fingerprint reconstruction targets were generated from the projection-head outputs.Source: §1.3 · p. 14

1.4 Surrogate Models

  • Surrogates included Gaussian processes with Matérn 5/2 kernels (Tanimoto kernels for fingerprints), feedforward networks with last-layer Laplace posteriors, and random forests.The random forests used 200 estimators.Source: §1.4 · pp. 14–15

1.5 Acquisition Functions and Batch Acquisition Policies

  • Candidate batches were selected using Top-K ranking or an ensemble of acquisition functions, with UCB and LogEI as the acquisition functions.Ensemble acquisition used different exploration-parameter values to select candidates.Source: §1.5 · pp. 15–16

2 Molecular Library Distributions

This section describes six virtual libraries with ground-truth target labels and characterizes their molecular distributions, including comparisons with a sampled ZINC15 reference; distribution summaries appear in supplementary figures and tables.

2 Molecular Library Distributions

  • Within-library dispersion was assessed using Morgan fingerprints and transformer embeddings, with Tanimoto or cosine similarities across 5M random molecule pairs.The analysis covered all six labeled libraries and reported distribution characteristics in Tables S1–S6.Source: §2 · p. 17

2.1 1.9M Molecules from the Zinc Library Docked against AmpC (Zinc-AmpC)

  • 1,914,566 Zinc15-library molecules (~2% of the ~96M collection) were selected as a random subsample from the AmpC docking dataset [22].The section presents docking-score and embedding-similarity distributions for Zinc-AmpC.Source: §2.1 · pp. 18–19

2.2 1.5M Molecules from the RO4 Library Docked against OGG1 (RO4-OGG1)

  • 1,470,000 molecules (~10% of the ~14.7M Rule-of-Four-filtered Zinc15 library) were sampled from the OGG1 docking dataset [23].The section presents docking-score and embedding-similarity distributions for RO4-OGG1.Source: §2.2 · pp. 22–23

2.3 2.1M Molecules from the Enamine-HTS Collection Docked against TMPK (Enamine-TMPK)

  • The Enamine-HTS library contains 2,104,318 molecules docked against thymidylate kinase (TMPK) [16].The section presents docking-score and embedding-similarity distributions for Enamine-TMPK.Source: §2.3 · pp. 26–27

2.4 2.3M Molecules from the Harvard Clean Energy Project (HCEP-PCE)

  • The Harvard Clean Energy Project dataset contains 2,320,648 molecules with DFT-computed power conversion efficiencies for organic solar cells [24].The section presents target-property and embedding-similarity distributions for HCEP-PCE.Source: §2.4 · pp. 30–31

2.5 180k Organic Laser Candidates with Computed Oscillator Strengths (Laser-OSC)

  • The Laser-OSC library comprises 182,888 conjugated organic molecules with DFT-computed emission oscillator strengths [25].The section presents target-property and embedding-similarity distributions for Laser-OSC.Source: §2.5 · pp. 34–35

2.6 300k Phosphine Ligands with Predicted Enantioselectivities (Kraken-ES)

  • The Kraken-ES database contains 300,000 phosphine ligands with predicted enantioselectivity values (ΔΔG‡) for C(sp3)–C(sp2) cross-coupling [26].A random forest fitted to experimental data [27] using ML-extrapolated phosphine descriptors generated the values used as optimization targets.Source: §2.6 · p. 38

3 Benchmarking Surrogate Models and Acquisition Policies

The section benchmarks surrogate models and acquisition strategies on MolFormer embeddings in virtual screening and experimental-discovery settings; the supplied passages describe evaluation designs, not comparative outcomes.

3.1 Surrogate Model Benchmarks

  • Surrogate-model comparisons used MolFormer embeddings in virtual screening (20,000 experiments; batch size 1,000) and experimental discovery (1,000; batch size 50).The figures report optimization trajectories over 20 campaigns, library-specific logit-scale effects, and effect ratios averaged across libraries, relative to a Laplace Neural network model.Source: §3.1 · pp. 42, 43, 46, 47

3.2 Acquisition Strategy Benchmarks

  • Acquisition strategies were compared using a Laplace Neural network on MolFormer embeddings in virtual screening (20,000 experiments; batch size 1,000) and experimental discovery (1,000; batch size 50).The figures report optimization trajectories over 20 campaigns, library-specific logit-scale effects, and effect ratios averaged across libraries, relative to Top-K acquisition with a UCB acquisition function.Source: §3.2 · pp. 50, 51, 54, 55

4 Evaluating Molecular Representations · 5 Supplementary References

Supplementary evaluations examine molecular representations and fine-tuning strategies across virtual-screening and experimental-discovery settings, reporting optimization trajectories and encoder- or library-relative effects.

4.1 Heuristic Representations and Pre-Trained Transformer Embeddings

  • Figures compare domain-specific encoders and pretrained molecular-language-model encoders with Morgan fingerprints using optimization trajectories and encoder-relative effects across libraries.Evaluations include virtual screening with 20,000 experiments and experimental discovery with 1,000 experiments; trajectories average 20 campaigns, with shaded standard errors.Source: §4.1 · pp. 58, 59, 60, 64, 65, 66

4.2 Transformer Pooling Strategies

  • MolFormer pooling strategies are compared against mean pooling through optimization trajectories and library-specific and averaged effect measures.Figures cover virtual screening at 20,000 experiments and experimental discovery at 1,000 experiments.Source: §4.2 · pp. 70, 71, 74

ChemBERTa

  • ChemBERTa fine-tuning strategies were assessed through optimization trajectories over 20,000 virtual-screening experiments (batch size 1,000; 20 campaigns) and 1,000 experimental-discovery experiments (batch size 50; 20 campaigns), plus library-differentiated logit-scale effects.The logit-scale effects used a Laplace Neural network model; means and standard deviations came from 8,000 posterior samples.Source: ChemBERTa · pp. 78, 79, 82, 83

MolFormer-XL

  • MolFormer-XL fine-tuning strategies are evaluated using optimization trajectories and library-specific and averaged effects in virtual screening and experimental discovery.The reported budgets are 20,000 experiments for virtual screening and 1,000 for experimental discovery.Source: MolFormer-XL · pp. 86, 87, 90, 91

SmiTed

  • SmiTed fine-tuning strategies were assessed through optimization trajectories averaged over 20 independent campaigns and library-differentiated logit-scale effects, using virtual screening (20,000 experiments; batch size 1,000) and experimental discovery (1,000; batch size 50).Logit-scale means and standard deviations were calculated from 8,000 posterior samples in both settings.Source: SmiTed · pp. 94, 95, 98, 99

T5Chem

  • At a 20,000-experiment virtual-screening budget, T5Chem fine-tuning strategies are compared through mean optimization trajectories across 20 campaigns, with standard-error shading.The batch size is 1,000 experiments.Source: T5Chem · p. 102
  • In virtual screening, the figures report library-specific logit-scale effects and effect ratios averaged across libraries for T5Chem fine-tuning strategies.The Laplace neural network analysis uses 8,000 posterior samples; effect-ratio violins show 95% highest-density intervals.Source: T5Chem · p. 103
  • At a 1,000-experiment experimental-discovery budget, T5Chem fine-tuning strategies are assessed with optimization trajectories, library-specific logit-scale effects, and cross-library effect ratios.The batch size is 50; trajectories average 20 campaigns, logit-scale estimates use 8,000 posterior samples, and effect-ratio violins show 95% highest-density intervals.Source: T5Chem · pp. 106–107
Loading 2608.17567v1…