PaperlayerView exact revision on arXiv

Source-linked AI summary

TransPolymer: a Transformer-based language model for polymer property predictions

Changwen Xu, Yuyang Wang, Amir Barati Farimani

arXiv:2209.01307v4cs.LGphysics.chem-ph

TL;DR

Polymer property prediction is important for polymer design, but Transformer models had not yet been leveraged for polymer science. The paper proposes TransPolymer and evaluates it against established baselines, while examining pretraining and sequence modeling.

  • Problem

    Accurate and efficient polymer property prediction is essential for polymer design, while Transformer models had not yet been leveraged to learn polymer representations.

  • Method

    TransPolymer is a Transformer-based language model for polymer property prediction, using polymer sequences and a Transformer encoder.

  • Results

    TransPolymer was evaluated against established fingerprinting, architecture, and recurrent-model baselines across polymer property prediction benchmarks.

  • Takeaways & Limitations

    The paper presents TransPolymer as a computational model for polymer property prediction and examines how pretraining and self-attention contribute to sequence modeling.

  • Takeaways & Limitations

    Some information cannot be explicitly obtained from input sequences, and performance may be affected by noise in the data.

Abstract

from arXiv · show

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

1 Introduction

Polymer property prediction needs representations that capture chemical structure, interactions, and polymer-specific factors while avoiding costly structural information. TransPolymer addresses this gap with a chemically aware sequence tokenizer and Transformer architecture for polymer property prediction.

  • Accurate and efficient polymer property prediction is important for polymer design, but conventional evaluation relies on expensive experiments or simulations.
  • Graph-based models can learn molecular representations, yet polymer graphs are difficult to construct because degree of polymerization and conformational information are challenging to obtain or encode.Using only repeating units as graphs may omit structural information.
  • Sequence models offer an alternative representation, but recurrent models can lose information across deeper steps and may inadequately encode chemical knowledge from polymer sequences.
  • Transformer models use self-attention to capture relationships between tokens without relying on past hidden states, and pretraining on unlabeled text can improve downstream performance.
  • Polymer Transformers remain underexplored, while standard SMILES may miss polymer structure, monomer arrangement, degree of polymerization, and measurement temperature.
  • TransPolymer represents polymers with repeating-unit SMILES and structural descriptors, tokenizes them chemically, and uses a RoBERTa-based model with an MLP regressor and MLM pretraining.The model can learn some unavailable structural information implicitly, although bond angles and overall chain configuration are not explicitly obtained from the input sequences.
  • TransPolymer achieves state-of-the-art results on all ten benchmarks and surpasses baseline models by large margins in most cases.

2 Results

TransPolymer combines chemical-aware polymer tokenization with a Transformer encoder and pretrain–finetune workflow for polymer property prediction. Across ten benchmarks, pretrained TransPolymer generally improves generalization, benefits from larger pretraining datasets, and highlights the role of attention and task-specific finetuning.

  • TransPolymer framework: TransPolymer uses a Transformer encoder with self-attention, pretrained by Masked Language Modeling and then finetuned for ten downstream property datasets.The framework uses a chemically aware tokenizer as input and evaluates properties across diverse polymer materials.
  • TransPolymer framework: Chemical-aware tokenization represents repeating units with polymer SMILES and descriptors, while copolymers and mixtures are encoded by combining component sequences.Descriptors include degree of polymerization, polydispersity, and chain conformation; the tokenizer is designed for polymers’ complex hierarchical structures and compositions.
  • Polymer property prediction results: R2 of 0.69 on PE-I test data shows that TransPolymerpretrained improves test performance while exhibiting less overfitting than baseline models.A Gaussian Process baseline reaches R2 0.90 on training data but only 0.16 on test data, whereas TransPolymerpretrained achieves the highest scores on both sets.
  • Polymer property prediction results: RMSE of 0.61 and R2 of 0.73 on PE-II cross-validation sets are the best reported results there, indicating better generalization than the baselines.TransPolymerpretrained achieves the lowest RMSE and highest R2 on the average of cross-validation sets.
  • Polymer property prediction results: Across all ten datasets, TransPolymerpretrained outperforms other models, decreasing evaluation RMSE by 7.70% and increasing evaluation R2 by 0.11 versus the best baselines.Compared with TransPolymerunpretrained, the corresponding changes are 18.5% and 0.12, respectively.
  • Ablation and pretraining analysis: Increasing pretraining size enhances downstream performance, while finetuning both Transformer encoders and the regressor head is needed for stronger task-specific results.Performance generally improves through decreasing RMSE and increasing R2; regressor-head-only training is insufficient for task-specific information.

3 Discussion

TransPolymer combines chemical-aware polymer tokenization, data augmentation, MLM pretraining, and Transformer encoders for polymer property prediction. Across the reported discussion, it outperformed baselines and unpretrained TransPolymer, with attention analyses identifying influential tokens and supporting applications to polymer design.

  • 3 Discussion: TransPolymer maps polymer instances to token sequences, enlarges representation-learning data through augmentation, and uses approximately 5M unlabeled sequences for MLM pretraining.The pretrained model is subsequently finetuned on downstream datasets.
  • 3 Discussion: TransPolymer outperformed all baselines and unpretrained TransPolymer across different downstream datasets.The discussion attributes the performance to large-scale unlabeled-data pretraining, Transformer-encoder finetuning, and data augmentation.
  • 3 Discussion: Attention scores from hidden layers provide evidence for chemically aware representations and identify influential tokens affecting final predictions.The discussion presents these analyses as evidence for the efficacy of chemical awareness in learned representations.
  • 3 Discussion: The authors report desirable performance and generalization with a small number of labeled downstream data, suggesting potential use for newly designed polymer-property prediction.They describe active-learning-guided discovery as one possible application, where predictions screen candidates and experimental data updates the model.
  • 3 Discussion: TransPolymer’s performance on copolymer datasets compared with existing baselines supports further exploration of copolymers and broader polymer-discovery frameworks.The paper’s main focus is regression, while the discussion describes several potential (co)polymer discovery applications.

4 Methods

The methods represent polymers as chemically informed token sequences that preserve repeating-unit connectivity, components, and available descriptors. TransPolymer uses a Transformer encoder with self-attention and positional encodings, MLM pretraining, and supervised finetuning for property prediction.

  • Polymer sequence design: Polymer sequences encode repeating-unit connectivity and polymer-specific information that ordinary SMILES may omit, including molecular weight and degree of polymerization when available.Polymer repeating units are converted to SMILES, with special notation indicating connectivity, copolymer constituents, and branches.
  • Polymer sequence design: Available descriptors, component information, ratios, and temperatures are appended with special tokens, while missing values receive dataset-specific NAN tokens.Adding descriptors can improve property-prediction performance, and ratio tokens are padded for varying copolymer constituents or composite components.
  • Polymer tokenizer: The chemical-aware tokenizer recognizes polymer elements and descriptor vocabulary, treating values such as temperature as single discretized tokens.For example, Si is recognized as one token rather than likely being split into S and i by the RoBERTa tokenizer.
  • Data augmentation: SMILES augmentation creates grammatically correct representations while preserving isomerism, removes duplicates, and is applied only to training sets after splitting.This ordering is used to avoid information leakage.
  • Transformer-based encoder: The Transformer encoder uses self-attention, multi-head projections, and positional encodings to model token relationships and sequence positions without recurrence.Attention maps inputs to queries, keys, and values; multi-head attention learns information from different subspaces.
  • Pretraining: MLM pretraining masks, replaces, or leaves selected tokens unchanged so TransPolymer learns contextual embeddings and the chemical grammar of polymer sequences.The pretraining strategy uses 15% candidate tokens, with 80% masked, 10% randomly replaced, and 10% unchanged among selected tokens.
  • Finetuning: The pretrained encoder is finetuned with a one-layer MLP regressor head for polymer property prediction, using separate learning-rate treatment for the encoder and head.Layer-wise learning-rate decay may decrease the learning rate from top to bottom because different layers learn different information.

Data Availability

The work uses publicly available datasets, with original datasets available in the corresponding literature and original and processed versions used in the study.

  • Data Availability: All data used in the work are publicly available, and the original datasets can be found in the corresponding literature.The study also provides original and processed datasets used in the work.

Supporting Information Available

The supporting information contains additional details on augmentation, finetuning, baselines, sequence lengths, predictions, pretraining size, and LSTM gradient behavior.

  • Supporting Information Available: Supporting information includes downstream augmentation strategies, finetuning details, and a summary of baseline models.It also covers downstream sequence-length distributions and ground-truth versus predictions with augmented data.
  • Supporting Information Available: Supporting information reports model performance with varying pretraining size and examines LSTM gradient diminishing.

Supplementary Information for TransPolymer: a

The supplementary information identifies the work as a Transformer-based language model for polymers and lists its authors and Carnegie Mellon University affiliations.

  • The work is titled “Transformer-based Language Model for Polymer.”
  • The document is an arXiv version dated 26 April 2023.
  • Changwen Xu, Yuyang Wang, and Amir Barati Farimani are listed as authors.
  • The authors are affiliated with Carnegie Mellon University departments spanning materials science, mechanical engineering, machine learning, and chemical engineering.

Supplementary Methods

The supplementary methods describe sequence-based data augmentation, validation-driven fine-tuning, reproduced and independently trained baselines, and task-specific implementation settings.

  • Data augmentation: Copolymer sequences are augmented by generating equivalent SMILES for each repeating unit.
  • Data augmentation: Augmentation limits vary by dataset and computational-resource capacity, with PE-I capped at two variants and OPV at five per SMILES.
  • Data augmentation: Small datasets with short sequences, such as Ei, are allowed to generate many augmented sequences per polymer.
  • Fine-tuning: TransPolymer fine-tuning uses validation-set hyperparameter search with cosine-annealing learning-rate schedules.
  • Baselines: Baseline implementations follow original architectures where available, reproduce reported results, and use five-fold cross-validation for the OPV baseline.
  • Baselines: Random forests with ECFP6 fingerprints and bidirectional LSTMs provide additional comparisons with fingerprinting strategies and Transformer models.

Supplementary Discussion

The supplementary discussion examines transferability, pretraining-data effects, augmented predictions, and why Transformer attention outperforms LSTM models on these polymer tasks.

  • Transferability: Varying sequence-length distributions across downstream datasets support transferability to both long and short polymer sequences.
  • Augmented predictions: Augmented examples retain the original training point’s true property value, while their predictions cluster around the original point’s prediction.
  • Pretraining: Downstream-task performance increases as the size of the pretraining dataset increases.
  • Model comparison: LSTM gives the worst performance on most downstream tasks, whereas Transformer performance highlights an advantage for attention in understanding chemical knowledge from polymer sequences.
  • Gradient flow: Gradient analysis finds diminishing mean gradients in all three LSTM architectures, while maximum gradients do not diminish significantly in smaller models.
  • Training behavior: LSTM models are early stopped in many folds before 20 epochs because test loss does not decrease, indicating susceptibility to overfitting despite regularization.
Loading 2209.01307v4…