Source-linked AI summary
Mapping the Space of Chemical Reactions Using Attention-Based Neural Networks
Philippe Schwaller, Daniel Probst, Alain C. Vaucher, Vishnu H. Nair, David Kreutter, Teodoro Laino, Jean-Louis Reymond
TL;DR
The paper develops transformer-based models for reaction classification and data-driven reaction fingerprints. The classifier reaches 98.2% accuracy versus 41% for a traditional fingerprint with a 5-nearest-neighbours classifier, while the fingerprints support reaction retrieval and analysis of chemical reaction space.
Problem
The paper addresses reaction classification and the development of reaction fingerprints that support reaction retrieval.
Method
BERT-based transformer models learn reaction classification schemes from text-based reactions, with learned representations further refined as reaction fingerprints.
Results
98.2% accuracy was achieved for the BERT classifier, compared to 41% for a traditional fingerprint plus 5-nearest-neighbours classifier.
Takeaways & Limitations
The learned fingerprints capture fine-grained reaction-class differences and enable reaction retrieval without knowing the reaction class.
Takeaways & Limitations
Failure analysis was difficult and relied on inspection of a few dozen cases, including confusion between carboxylic acid–amine condensation and amide Schotten–Baumann reactions.
Abstract
from arXiv · showhide
Organic reactions are usually assigned to classes containing reactions with similar reagents and mechanisms. Reaction classes facilitate the communication of complex concepts and efficient navigation through chemical reaction space. However, the classification process is a tedious task. It requires the identification of the corresponding reaction class template via annotation of the number of molecules in the reactions, the reaction center, and the distinction between reactants and reagents. This work shows that transformer-based models can infer reaction classes from non-annotated, simple text-based representations of chemical reactions. Our best model reaches a classification accuracy of 98.2%. We also show that the learned representations can be used as reaction fingerprints that capture fine-grained differences between reaction classes better than traditional reaction fingerprints. The insights into chemical reaction space enabled by our learned fingerprints are illustrated by an interactive reaction atlas providing visual clustering and similarity searching.
Results and Discussion
Transformer models classify reaction classes directly from SMILES representations and produce fingerprints that support fine-grained classification, visualization, and retrieval.
- Classification results: 98.2% accuracy was achieved by the BERT classifier on the Pistachio test set, exceeding the encoder-decoder model’s 95.2%.The BERT classifier predicted the correct name reaction with 98.2% accuracy, while enc2-dec1 reached 95.2%.
- Classification results: The BERT classifier performed best on confusion entropy and Matthews correlation, and its rxnfp embeddings reproduced the classifier’s scores with 5-NN.The embeddings correspond to the input of the BERT classifier’s classification layer.
- Classification results: 98.9% accuracy was achieved on the USPTO 1k TPL test set by the BERT classifier and its embeddings with 5-NN.Traditional and pretrained fingerprints performed worse on USPTO 1k TPL than on Pistachio.
- Visualisation of Attention Weights: Attention concentrated on reaction centers and class-specific precursors, while the learned fingerprints formed nearly perfect class clusters and grouped related reaction types in TMAP.The BERT model’s middle layers attended to products or precursors, and later layers focused on reaction centers and important precursors.
Conclusion
Transformer-based models learned chemical reaction classification schemes from reaction representations and achieved accuracy comparable to rule-based classification. Their learned representations also function as reaction fingerprints for mapping, searching, and downstream prediction tasks.
- Reaction fingerprints: The learned representations can serve as reaction fingerprints that capture embedded information from chemical reactions.These fingerprints support chemical-reaction representation beyond the classification task.
- Classification: Transformer-based models learned classification schemes from a broad set of chemical reactions labeled by a commercial classification tool.The models learned atomic environment characteristics for each class, providing an interpretable rationale through attention weights.
- Classification: 98.2% accuracy was achieved by the BERT classifier, compared to 41% for a traditional fingerprint plus 5-nearest neighbours classifier.The BERT classifier matched the rule-based classification accuracy reported for the task.
- Reaction fingerprints: Data-driven reaction fingerprints enable reaction-space mapping without reaction-center or reactant–reagent annotations.They also support efficient nearest-neighbour searches on reaction datasets containing millions of reactions.
- Downstream applications: The fingerprints were used to estimate experimentally measured activation energies and were fine-tuned to predict chemical reaction yields.These applications were reported as subsequent uses of the learned fingerprints.
Methods
The study trained transformer models on reaction SMILES and evaluated their reaction-class representations and classification performance across curated reaction datasets. It compared autoregressive and BERT-style architectures, nearest-neighbour classification, and reaction-space visualization.
- Datasets: The data included 2.6M reactions from the Pistachio database, classified into reaction classes using NameRXN.Pistachio reactions were split into train, validation, and test sets with identical products kept together.
- Input representation: The models used reaction SMILES without enforcing a distinction between reactants and reagents, enabling application where that distinction is subtle.Reaction SMILES were tokenized for transformer processing.
- Models: The study trained an autoregressive encoder–decoder transformer and a BERT encoder pretrained with masked language modeling before classifier fine-tuning.The BERT classifier used [CLS] embeddings as input to the classification head and as reaction fingerprints.
- Evaluation and visualization: A 5-nearest-neighbour classifier used Euclidean distance, while TMAP and Faerun generated and displayed tree-like reaction-space layouts.FAISS provided brute-force nearest-neighbour searches, and TMAP was selected for large datasets and local/global structure preservation.
Data availability
The study identifies public and commercial sources for its reaction datasets and introduces USPTO 1k TPL as a new reaction-classification dataset.
- Dataset access: USPTO 1k TPL is a new reaction dataset derived from Lowe’s USPTO work and contains the 1000 most common reaction templates as classes.The dataset is accessible through the rxn4chemistry GitHub site.
- Dataset access: The Schneider 50k dataset is publicly available, while the commercial Pistachio dataset can be obtained from NextMove Software.Pistachio relies on Leadmine to text-mine patent data and includes NameRXN-assigned reaction classes.
Code availability
The rxnfp code, public-dataset experiments, and an interactive TMAP are provided as openly available resources.
- Open resources: The rxnfp code, public-data experiments, and an interactive TMAP were made available.These resources support reuse of the reported models, experiments, and reaction-space visualization.
Correspondence
Philippe Schwaller is identified as the corresponding author.
- Philippe Schwaller is the corresponding author.
- The paper identifies Philippe Schwaller as its correspondence contact.
- Correspondence for the paper is associated with Philippe Schwaller.
Contributions
The authors describe contributions spanning project conception, model training, classification experiments, fingerprint analysis, reaction atlases, supervision, and manuscript preparation.
- PS and AV conceived the initial project idea.
- PS, DP, AV, and VH trained models, conducted classification experiments, and analysed the results.
- PS investigated reaction fingerprints and wrote the code base.
- PS, DP, and DK worked on the reaction atlases.
- TL and JLR supervised the project, while all authors contributed to discussions and manuscript writing.
Supplementary Information: Mapping the Space
The supplementary mapping material is associated with the paper’s chemical-reaction mapping work and lists its authors, affiliations, and arXiv posting date.
- The supplementary material concerns mapping the space of chemical reactions using attention-based methods.
- The paper was posted as arXiv:2012.06051v1 on 9 December 2020.
- The listed authors are Philippe Schwaller, Daniel Probst, Alain C. Vaucher, Vishnu H. Nair, David Kreutter, Teodoro Laino, and Jean-Louis Reymond.
- The affiliations include IBM Research – Zurich and the Department of Chemistry and Biochemistry at the University of Bern.
Supplementary Note 1: Reaction properties atlases
The reaction-properties atlas visualizes the Schneider 50k set with rxnfp embeddings, combining 10k training reactions and 40k unseen reactions. Same-class reactions cluster together, while precursor elements and product properties show structured map patterns.
- Reaction properties atlases: Figure 1 visualizes the Schneider 50k chemical-reaction set with TMAP using rxnfp embeddings.
- Reaction properties atlases: The BERT model generating the reaction fingerprint was trained on 10k training reactions.
- Reaction properties atlases: The maps contain 10k training reactions together with 40k unseen reactions.
- Reaction properties atlases: Reactions from the same classes are well clustered together on the reaction maps.
- Reaction properties atlases: Reactions containing specific precursor elements occur in the same map branches, while product properties also form defined clusters.
- Reaction properties atlases: Different maps highlight reaction superclasses, specific metallic elements in precursors, and product properties.
Supplementary Note 2: Analysis of Pistachio predictions
The analysis characterizes BERT’s prediction errors in the Pistachio test set, with “Unrecognised” assignments dominating and frequent confusions involving closely related transformations.
- 47.9% of incorrect predictions assigned reactions to the “Unrecognised” class.
- 22.8% of errors predicted “Unrecognised” when a reaction class should have been assigned.
- 17.5% of errors predicted an incorrect name reaction, while incorrect superclass and category predictions accounted for 8.3% and 3.5%.
- For 12 of the 15 most frequently misclassified classes, the most common error was predicting “Unrecognised”.
- Bouveault-Blanc reduction was commonly confused with Ester to alcohol reduction because both reduce an ester to a primary alcohol.The distinction involves specific precursors such as sodium with ethanol or methanol.
- The “1,3-Dioxane synthesis” class reached 88.9% accuracy, with some errors involving “Dioxolane synthesis” despite a one-carbon difference in the product heterocycle.
- The analysis was constrained by the large number of “Unrecognised” reactions and by reactions containing simultaneous transformations that resist assignment to a single class.
- Data-driven predictions identified some likely name reactions and tolerated tautomeric or minor parsing differences better than rule-based approaches.
Supplementary Note 3: Analysis of 50k set predictions
The 50k benchmark compares learned reaction fingerprints with a hand-crafted fingerprint across 50 reaction classes, including pretrained and fine-tuned transformer representations.
- The benchmark contains 50 classes with 1,000 reactions per class, using 200 training/validation and 800 testing examples per class.
- The study evaluated recall, precision, and F-score averaged over the 50 classes for pretrained and fine-tuned learned fingerprints.
- The pretrained rxnfp did not match the hand-crafted fingerprint on the balanced dataset, whereas fine-tuned rxnfp (10k) performed better.
- 0.99 versus 0.97: rxnfp (10k) reached a higher reported score than the hand-crafted fingerprint using as few as 10k reactions.
- Methylation was the hardest class to predict correctly for both data-driven fingerprints.
- The pretrained fingerprint struggled with classes differing by one atom and confused several chemically related reaction classes.
- The analysis suggests that future unsupervised fingerprints may capture fine-grained reaction-class information better.