PaperlayerView exact revision on arXiv

Source-linked AI summary

ChemBART: Pre-training a Reaction Language Model for Comprehensive Synthesis Planning

Kenan Li, Yijian Zhang, Jin Wang, Haipeng Gan, Zeying Sun, Xiaoguang Lei, Hao Dong

arXiv:2601.02915v2cs.LGcs.CL

TL;DR

Single-molecule pre-training may not capture comprehensive chemical-system knowledge, motivating reaction-level learning for synthesis planning. ChemBART pre-trains on masked reaction components and fine-tunes for diverse chemical tasks, including MCTS planning. Wet-lab-validated routes were shorter and showed over 20% yield improvement over established literature benchmarks.

  • Problem

    Single-molecule pre-training may be insufficient for comprehensive chemical-system knowledge, and chemical reactions encode rules governing interactions among atoms and functional groups.

    Source: Introduction · pp. 3–4

  • Method

    ChemBART pre-trains on masked components of reaction SMILES and fine-tunes a shared model for chemical prediction tasks and MCTS synthesis planning.

    Source: Introduction · p. 4, §1 · p. 6

  • Results

    Over 20% yield improvement over established literature benchmarks was reported for wet-lab-validated, shorter ChemBART-designed routes.

    Source: Introduction · p. 5

  • Takeaways & Limitations

    ChemBART combines diverse chemical tasks and synthesis planning in a unified model using shared capabilities.

    Source: Discussion · p. 16

    ChemBART’s prediction of molecular physicochemical properties, including activation energy on QM9, remains limited.

    Source: Discussion · p. 16

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have demonstrated transformative potential across diverse fields. However, in computer-aided synthesis planning (CASP), current LLM methodologies are often restricted to single-molecule representations, limiting their application to isolated, single-task scenarios such as precursor prediction. Addressing this gap, we present ChemBART, a BART-based language model that adopts a reaction-centric pre-training strategy. By leveraging a mask-filling pre-training task on holistic reaction expressions (Reactant > Reagent > Product), ChemBART effectively solves a variety of chemical problems, including precursor/reagent generation, temperature-yield regression, and molecular property classification. Building on these capabilities, we demonstrate the integration of ChemBART with Monte Carlo tree search (MCTS) for multi-step synthesis route design, where the model serves to optimize the policy and value functions within a reinforcement learning framework. Beyond single-molecule pre-trained LLMs that are constrained to specific applications, ChemBART unifies diverse chemical tasks to facilitate synthesis planning with comprehensive implementation information. Most significantly, the proposed multi-step synthesis routes and reaction conditions were validated in the wet lab, confirming the discovery of shorter synthetic pathways with a >20% yield improvement over established literature benchmarks. Our work highlights the potential of reaction-focused pre-training and showcases the broad utility of ChemBART in closing the loop between in silico prediction and wet-lab synthesis.

Introduction

ChemBART addresses the limits of single-molecule pre-training by learning from whole reactions and applying one model across diverse chemical tasks. The authors report wet-lab-validated multistep routes with shorter pathways and over 20% yield improvement over literature benchmarks.

Introduction

  • Prior approaches include single-molecule pre-training for downstream chemical tasks and reaction-focused methods whose evaluation remained limited to precursor prediction.The introduction also notes that TTL used distinct models for precursor, reagent, and product prediction.Source: Introduction · pp. 3–4
  • Single-molecule pre-training may miss comprehensive chemical-system knowledge, while ambiguous masked fragments can complicate optimization and hinder definitive representations.Chemical reactions encode rules governing interactions among atoms and functional groups.Source: Introduction · pp. 3–4
  • ChemBART aims to unify precursor prediction, yield regression, and MCTS planning in one fine-tuning framework rather than separate task-specific models.Its reaction-level mask-filling objective masks precursors, reagents, or products, while policy and value learning supports MCTS multistep planning.Source: Introduction · p. 4
  • Over 20% yield improvement over established literature benchmarks accompanied experimentally validated ChemBART routes with shorter pathways.The proposed routes and reaction conditions were validated in the wet lab.Source: Introduction · p. 5

1. Pre-training and fine-tuning workflow

ChemBART is pre-trained to reconstruct masked components of reaction SMILES, then fine-tuned for varied prediction tasks and synthesis planning.

1. Pre-training and fine-tuning workflow

  • ChemBART represents each reaction as a SMILES sentence in reactant > reagent > product order and pre-trains by masking and reconstructing one component.The model is pre-trained on USPTOfull or USPTO-MIT; ChemBART-Mol is a single-molecule SMILES control without reaction context.Source: §1 · pp. 5–6
  • The workflow uses reaction data rather than molecule-only strings as its pre-training context.ChemBART-Mol provides a same-architecture molecule-only control trained on single-molecule SMILES from USPTO-MIT.Source: §1 · p. 6
  • After pre-training, task-specific tokens and heads support temperature and yield regression, molecular property classification, and MCTS parameter estimation.The integrated model supports multistep retrosynthetic planning and prediction of associated reaction implementation information.Source: §1 · p. 6

2. Interpretability of ChemBART.

ChemBART’s learned representations show patterns consistent with chemical relationships in element embeddings, oxidation pathways, and reaction attention. The authors interpret attention in a Grignard example as most closely aligned with polar nucleophilic substitution, while noting that its mechanism remains debated.

2. Interpretability of ChemBART.

  • Embedding distances captured element similarities consistent with electronegativity and bonding patterns, including Cl/Br at 1.02 and Na/K at 0.93.C/O/N formed a distinct cluster from the ionic-leaning Cl/Br/Na/K group; lower C/Cl distance than C/Br was consistent with stronger C-Cl covalency.Source: §2 · p. 6
  • Decoder end-token projections showed consistent vectors for alcohol-to-aldehyde and aldehyde-to-acid oxidation pathways.The authors interpret this as evidence that the model encodes SMILES syntax and chemical principles.Source: §2 · p. 7
  • For masked-product prediction in a Grignard reaction, attention linked the nucleophilic carbon strongly to the Br-bearing carbon and Mg²⁺.Attention decayed along the alkyl chain from Br, while attention to chain carbons increased from C4 to C1; the authors say this aligns most closely with polar nucleophilic substitution, although the mechanism remains debated.Source: §2 · pp. 7–8, Figure 2 · p. 7
  • Attention patterns also showed strong local token relationships, which the authors connect to recognition of neighboring atomic groups.They report a similar pattern in decoder and cross-attention and describe statistical co-occurrences as chemically intuitive associations learned from reaction data.Source: §2 · p. 8

3. Single-step retrosynthesis prediction.

ChemBART uses beam search for single-step predictions and reports performance comparable to several template-free and pre-trained SMILES models, while noting a gap to state-of-the-art systems. Its multi-task reaction pre-training is presented as supporting broader chemical reasoning and transfer.

3. Single-step retrosynthesis prediction.

  • Beam search generates multiple precursor and reagent predictions, with top-k retrosynthesis accuracy compared against recent work.Additional evaluations include product and forward prediction and syntax-error rates.Source: §3 · p. 8
  • ChemBART achieved performance comparable to AutoSynRoute and pre-trained SMILES models, though a gap remained to state-of-the-art models.The authors note that RSGPT uses 20× synthetic training data and 10× model size; ChemBART’s comparison is among similarly sized models.Source: §3 · p. 9
  • The authors attribute potential advantages to jointly predicting precursors, reagents, and products, which they say enhances chemical reasoning via implicit task correlation and triples training data.They also report low overall syntax error rates and near-zero errors for reagent predictions.Source: §3 · p. 9

4. ChemBART fine-tuning: predicting reaction features and molecular

ChemBART was fine-tuned for joint reaction temperature and yield regression and molecular-property classification. Its reported results include competitive yield prediction and strong AUC performance on three biochemical benchmarks.

properties.

  • ChemBART-M was selected for joint temperature and yield regression after showing the best individual property-prediction performance among the tested initializations.The checkpoint with the lowest validation RMSE was selected; task-specific tokens and a linear layer were used for prediction.Source: properties · pp. 9–10
  • 25°C temperature MAE across −110 to 210°C and 18% yield MAE on ORD; Suzuki-Miyaura yield prediction reached R² 0.816 and MAE 7.9%.The Suzuki-Miyaura result was reported across 10 random splits and was close to Yield-BERT’s R² of 0.81.Source: properties · p. 10, Figure 3 · p. 10
  • On Buchwald-Hartwig yield prediction, ChemBART’s four-test average R² was 0.74, comparable to T5Chem at 0.79 and Yield-BERT at 0.73.The comparison is reported for the Buchwald-Hartwig yield dataset.Source: properties · p. 10
  • ChemBART achieved state-of-the-art AUC performance on HIV, BACE, and Tox21, and comparable performance on BBBP and ClinTox.LoRA fine-tuning further improved performance, while ablations supported the effectiveness of reaction-expression pre-training.Source: properties · p. 11, Table 2 · p. 11
  • Fine-tuning induced apparent clustering in the pre-training embeddings, whereas the pre-training embeddings showed poor separation.The authors attribute the initial poor separation to a lack of property supervision and describe the clustering as task-specific features distilled from the pre-trained model.Source: properties · p. 11

5. Synthetic path planning

ChemBART combines learned policy and value functions with MCTS to plan multistep synthesis routes. Evaluations report benchmark success, long-path planning, and performance boundaries on challenging structures.

5. Synthetic path planning

  • ChemBART planned a 15-step route to compound 26, with reductive selectivity at step J identified as a potentially inspiring pathway strategy.Figure 4 shows routes for intermediates 14 and 17 and the target molecule 26.Source: Figure 4 · p. 12, §5 · p. 13
  • At each MCTS node, ChemBART generates 50 precursor-and-reagent candidates, scoring feasibility with model probabilities and synthetic difficulty with fine-tuned policy and value functions.On 12,000 ReSynZ-generated policy and value data points, pre-trained ChemBART showed balanced predictions and outperformed the template-based network.Source: §5 · pp. 12–13
  • ChemBART-M/F achieved 65% and 71% success rates on Retro*-190 using eMolecules as the starting molecule set.The authors report these rates as nearly matching template-based methods while also considering route quality.Source: §5 · p. 13
  • 88.7%, 84.9%, and 83.0% success rates were reported for beam search, top-k, and top-p on JMC2025; sampling increased diversity but also hallucination risk.Beam search tended to produce high-confidence but chemically conservative routes.Source: §5 · pp. 13–14
  • ChemBART reproduced literature routes for all 11 novel FDA-approved drugs tested, but resolved only 4 of 11 molecules on the Chematica benchmark.The authors report difficulty with ring-closing transformations and highly complex topologies, likely reflecting limited coverage in the public USPTO training set.Source: §5 · p. 14

6. Wet-lab validation

This section describes ChemBART-guided synthesis-route search and wet-lab validation, including MCTS-based route selection and the application of proposed conditions in the laboratory. It reports validation of ChemBART’s route to P1 and its overall yield.

6. Wet-lab validation

  • The more convergent route coupled amine 36 and alcohol 38 by late-stage SNAr, with p-TsOH optimization raising substitution efficiency to 67%.The reaction gave only 10% yield under HCl before condition screening; amine 36 was obtained in 70% yield under Suzuki coupling conditions.Source: §6 · p. 15
  • 35% overall yield was achieved for ChemBART’s P1 route, a reported 27% improvement from ester 28 compared with the original route.P1 had previously been synthesized in six steps with an overall yield of 6.5%.Source: §6 · p. 15
  • Most ChemBART-proposed reaction conditions were readily applicable in the laboratory, while expert knowledge and further optimization helped achieve the best overall performance.Source: §6 · p. 15

Method

  • MCTS iteratively updates each molecule node’s policy and value, then uses root-child visit counts to select the best route or top-k alternatives.Greedy selection picks the highest-probability precursor at each step; beam search can provide alternative routes.Source: Method · pp. 22–23
  • Candidate precursors are canonicalized, deduplicated, reagent-annotated, and retained only when ChemBART predicts the original parent product.Probabilities of retained candidates are normalized to form the literature-informed candidate distribution.Source: Method · p. 23
  • MCTS combines literature-informed candidate confidence with ChemBART’s learned policy to prioritize synthesis routes that are both feasible and likely to complete.The final policy multiplies the normalized candidate probabilities by the fine-tuned model policy, then normalizes the result.Source: Method · p. 23
  • Target molecules are planned in separate processes that share GPU-hosted ChemBART models, and completed routes are augmented with predicted temperature, yield, and molecular properties.Source: Method · p. 24

Data Availability

The paper reports open-source datasets and publicly available experimental source data, while model weights are described as forthcoming upon acceptance.

Data Availability

supplementary information (ESI) available. See DOI: 10.1039/c7sc02664a. Chemical Science 9,

The listed references cover molecular language modeling, retrosynthesis, and Monte Carlo tree search planning.

References

Loading 2601.02915v2…