Source-linked AI summary
polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics
Christopher Kuenneth, Rampi Ramprasad
TL;DR
High-throughput polymer informatics requires efficient exploration of very large polymer spaces. The paper presents polyBERT within an ultrafast, accurate, cloud-scalable pipeline that predicts many polymer properties much faster than handcrafted fingerprints while preserving accuracy.
Problem
The large polymer chemical space makes timely, high-throughput exploration important for polymer informatics.
Method
The pipeline uses polyBERT, a Transformer-based NLP model modified for polymer chemical language and trained on 100 million polymers, to fingerprint polymers for property prediction.
Results
Polymer property prediction is over two orders of magnitude faster than the best handcrafted PG-fingerprint pipeline at the same accuracy, including accurate prediction of 29 properties for 100 million hypothetical polymers.
Takeaways & Limitations
The pipeline enables extensive exploration of polymer spaces and is suitable for high-throughput screening on scalable cloud hardware.
Takeaways & Limitations
Extending polyBERT to updated polymer structures requires retraining and structural updates, which the paper identifies as future work.
Abstract
from arXiv · showhide
Polymers are a vital part of everyday life. Their chemical universe is so large that it presents unprecedented opportunities as well as significant challenges to identify suitable application-specific candidates. We present a complete end-to-end machine-driven polymer informatics pipeline that can search this space for suitable candidates at unprecedented speed and accuracy. This pipeline includes a polymer chemical fingerprinting capability called polyBERT (inspired by Natural Language Processing concepts), and a multitask learning approach that maps the polyBERT fingerprints to a host of properties. polyBERT is a chemical linguist that treats the chemical structure of polymers as a chemical language. The present approach outstrips the best presently available concepts for polymer property prediction based on handcrafted fingerprint schemes in speed by two orders of magnitude while preserving accuracy, thus making it a strong candidate for deployment in scalable architectures including cloud infrastructures.
Results
The study builds polyBERT by pretraining on 100 million hypothetical PSMILES strings, then maps its fingerprints to polymer properties with multitask learning. The resulting pipeline combines chemically relevant representations, near-PG accuracy, and substantially faster fingerprint computation for large-scale polymer screening.
- Data and polyBERT training: 80 million PSMILES strings train polyBERT and 20 million validate it, producing a validation F1-score greater than 99.The model learns masked-token prediction from surrounding non-masked tokens in a self-supervised fill-in-the-blanks task.
- Fingerprint representation: polyBERT fingerprints have 600 fully dense components, whereas the handcrafted fingerprints have 945 components and 93.9% zeros.The polyBERT fingerprint is the sentence-average representation of the final latent space.
- Speed and property prediction: 215 times faster: polyBERT GPU fingerprints are computed over two orders of magnitude faster than Polymer Genome fingerprints.Computation scales nearly linearly with the number of PSMILES strings; the full 29-property pipeline takes 1.06 ms/polymer/GPU for hypothetical PSMILES strings.
- Speed and property prediction: R2 = 0.80 for polyBERT versus R2 = 0.81 for Polymer Genome in overall cross-validation performance.The ordering persists across category averages and properties, with polyBERT slightly outperforming PG for Xc, Xe, and ϵb.
- Large-scale screening: The pipeline predicts all 29 properties for 100 million hypothetical polymers, with predicted maxima interpreted as potential boundaries of the total polymer property space.The estimated CO2 emissions for predicting these 29 properties are 5.5 kgCO2eq.
Other Advantages of polyBERT: Beyond Speed and Accuracy
Beyond speed and accuracy, polyBERT offers interpretability through analysis of Transformer attention and saliency, while its pooled fingerprint limits unambiguous decoding back to PSMILES strings. A future decoder could support inverse design workflows.
- Decoding limitation: Averaging over the token dimension prevents unambiguous mapping of current polyBERT fingerprints back to PSMILES strings.This limitation matters for design informatics pipelines that invert prediction to meet property specifications.
- Decoding limitation: A modified future polyBERT with a dimensionality-reducing layer and fingerprint decoder could provide PSMILES encoding and fingerprint decoding.The proposed decoder is presented as a future design for inverse prediction workflows.
- Interpretability: Transformer attention and saliency methods can reveal chemical functions and interactions among structural parts of polymers.These analyses connect PSMILES inputs with polymer fingerprints and predicted properties.
Discussion
The discussion presents polyBERT as a Transformer-based replacement for handcrafted polymer fingerprints, enabling accurate property prediction at substantially higher speed. The resulting pipeline supports large-scale polymer exploration and potential high-throughput deployment.
- Applications: polyBERT may scale on cloud hardware for high-throughput screening and could accelerate existing pipelines by replacing handcrafted fingerprints.Direct polymer design using the decoder would require retraining and structural updates, so that capability remains future work.
- Method: polyBERT is a Transformer-based NLP model modified for the polymer chemical language and trained on 100 million hypothetical polymers.It produces numerical polymer representations used by the informatics pipeline.
- Performance: Polymer property prediction is over two orders of magnitude faster with polyBERT while retaining the same accuracy as the best handcrafted PG-fingerprint pipeline.This comparison is presented as the principal performance outcome.
- Applications: The pipeline predicts 29 properties for 100 million hypothetical polymers in a reasonable time, enabling extensive exploration of polymer space.The prediction scale is used to demonstrate the pipeline’s exploratory capability.
- Performance: The approach is reported as 100 times faster than state-of-the-art informatics approaches while maintaining the accuracy of slower handcrafted fingerprinting methods.The discussion attributes this speed to Transformer-based models originally developed for NLP.
- Applications: polyBERT fingerprints are dense, chemically pertinent representations that adequately measure polymer similarity and support polymer informatics tasks requiring numerical representations.The paper demonstrates property prediction and identifies structure prediction and synthesis assistance as additional potential uses.
Methods
The methods canonicalize PSMILES representations, generate numerical polymer fingerprints with polyBERT, and use multitask neural networks to predict polymer properties. Copolymer fingerprints are composition-weighted to preserve uniqueness and invariance across equivalent representations.
- PSMILES Canonicalization: Canonicalization reduces multiple equivalent PSMILES representations of a polymer to one unique canonicalized string.The package searches for repetition patterns, creates a periodic string, canonicalizes it, and breaks it into the canonical form.
- PSMILES Canonicalization: Translational, multiplicative, and permutational PSMILES variances leave the represented polymer unchanged.These variances correspond to shifting the repeat-unit window, writing multiples of the repeat unit, and syntactical permutations.
- Polymer Fingerprinting: polyBERT converts PSMILES-based chemical and geometric information into numerical vector fingerprints for prediction and similarity tasks.The fingerprints are learned representations rather than handcrafted descriptors.
- Polymer Fingerprinting: Copolymer fingerprints are composition-weighted sums of comonomer fingerprints, with each fingerprint weighted by its comonomer fraction.For N comonomers, the representation combines each Fi with its corresponding ci.
- Polymer Fingerprinting: The composition-weighted approach makes copolymer fingerprints invariant to comonomer ordering and equivalent periodic-unit specifications.This satisfies the stated demands of uniqueness and invariance, although copolymer fingerprints may not be physically interpretable.
- Multitask Neural Networks: Multitask deep neural networks jointly learn multiple polymer properties using five-fold cross-validation and a meta learner.Hyperband optimizes network parameters, and the final meta learner forecasts property values from the cross-validation ensemble.
Data and Code Availability
The authors provide the polyBERT code, model, canonicalization package, and a dataset containing predictions for 100 million hypothetical polymers. PolyBERT-based predictions are intended for access through Polymer Genome.
- Data and Code Availability: The polyBERT code and dataset of 100 million hypothetical polymers with predictions for 29 properties are available for academic use.The code and data are hosted in the project repository.
- Data and Code Availability: The trained polyBERT model is available through Hugging Face, and the canonicalization package is available through GitHub.The package supports canonicalizing PSMILES strings used by the model.
- Data and Code Availability: Polymer Genome is intended to provide access to polyBERT-based property predictions.The passage describes this as planned accessibility through the polymer informatics platform.
Declaration of Interests
The declaration reports that R.R. founded Matmerize, Inc., which intends to provide polymer informatics services, and that R.R. and C.K. filed a provisional patent on polyBERT.
- Declaration of Interests: R.R. founded Matmerize, Inc., a company intending to provide polymer informatics services.This represents a disclosed commercial interest related to the paper’s subject.
- Declaration of Interests: R.R. and C.K. filed a provisional patent on polyBERT.The patent is disclosed alongside the company affiliation.