Predicting Chemical Properties using Self-Attention Multi-task Learning based on SMILES Representation

• Regulatory agencies and pharmaceutical companies face the challenge of evaluating more than 140 million chemicals, beyond the time and cost efficiency of traditional in vivo and in vitro methods.Source: §I · p. 1
• Selecting molecular structural features and encoding them into fixed-length vectors is a significant challenge for machine-learning QSAR models.Source: §I · p. 1
• Fingerprint-based models require converting SMILES and can map different atom environments to the same Morgan-fingerprint bit.Source: §I · p. 1
• The study proposes a self-attention QSAR model that uses SMILES directly, without computing descriptors or fingerprints.Source: §I · p. 2
• The authors report state-of-the-art performance on several datasets and examine how transformer-variant structural changes influence learning.Source: §I · p. 2
• The study also presents SMILES-embedding implementation details for analyzing trained models.Source: §I · p. 2

• The experiments use HIV, SIDER, BBBP, and clintox datasets and compare SA-MTL with the best-performing models listed for each dataset by Wu et al..Source: §II · p. 2
• FP2VEC embeds selected extended-connectivity fingerprints, while SCFP learns SMILES features through character embeddings and CNN layers.Source: §II · p. 2
• Smiles Transformer [19] uses encoder-decoder pretraining on alternate SMILES of the same compound, then uses the encoder representation as a molecular fingerprint for prediction.Source: §II · p. 2
• Transformer-CNN [20] uses encoder-only pretraining followed by CNN layers, whereas SA-MTL places its CNN before self-attention.Source: §II · pp. 2–3
• BiLSTM-SA combines a BiLSTM, modified self-attention, and dense predictor; SA-MTL instead uses a discrete output layer and multi-task learning.Source: §II · p. 3

• Two-character atoms receive a single atom-embedding index rather than separate character indices, supporting correct alignment of learned weights with atoms such as Cl.Source: A. Multi-Task Learning Self-Attention… · p. 3
• A two-dimensional CNN with filter size 7 serves as a shared hidden layer, learning features across tasks despite comparatively small, class-imbalanced datasets.Source: A. Multi-Task Learning Self-Attention… · p. 3
• The self-attention component uses the Transformer encoder without a decoder, and the authors report little to no performance degradation when position embeddings are excluded.Source: A. Multi-Task Learning Self-Attention… · p. 3
• Each task has a discrete output layer of two fully connected layers, with balancing bias set to the negative-to-positive training-instance ratio.Source: A. Multi-Task Learning Self-Attention… · p. 4
• Weighted cross-entropy uses the balancing bias as a weight to address class imbalance.Source: A. Multi-Task Learning Self-Attention… · p. 4

B. Difference between our model and the Smiles Transformer

• Smiles Transformer pretrains atom embeddings on unlabeled data before fine-tuning; the paper cautions that SMILES pretraining should account for protein targets because toxicity can vary by target.A compound can be toxic for one protein target and non-toxic for another.Source: B. Difference between our model and… · p. 4

C. Difference between our model and the Transformer-CNN

• Transformer-CNN uses pretraining and a more complex predictor than Smiles Transformer, but does not implement multi-task learning.The authors associate its higher scores than Smiles Transformer with the more complex predictor.Source: C. Difference between our model and… · p. 4

D. Difference between our model and the BiLSTM-SA

• SA-MTL uses a CNN rather than BiLSTM, and its seven-layer self-attention replaces BiLSTM-SA's single attention module; RNN substitution showed no significant performance difference but ran slower.The seven-layer design followed hyperparameter optimization.Source: D. Difference between our model and… · p. 4
• The authors attribute the BiLSTM-SA performance gap mainly to absent multi-task learning; discrete output layers distinguish their design, while two-character atoms matter for weight analysis but reportedly have little performance influence.The claim about little performance influence is attributed to the Transformer-CNN authors.Source: D. Difference between our model and… · p. 4

IV. EXPERIMENTS

• Experiments covered Tox21, HIV, SIDER, BBBP, and CLINTOX; positive-negative ratios were notably imbalanced in Tox21, HIV, and CLINTOX.The dataset statistics are summarized in Table I.Source: §IV · p. 4, Table I · p. 5
• Hyperparameters were selected by validation AUC, and test performance was averaged over five repeated evaluations; comparisons included CNN, transformer-variant, DeepTox, and Wu et al. models.Table II summarizes the search ranges and selected parameters; results for comparison models came from cited references.Source: §IV · p. 4, Table II · p. 5

A. Dataset 1: Tox21

• Tox21 contains approximately 8,000 SMILES compounds across 12 target classes, and a compound can carry multiple class labels.Source: A. Dataset 1: Tox21 · p. 5
• 0.07 higher AUC than Graph convolution (GC), SA-MTL achieved the highest AUC on the Tox21 train/test random split.Table III presents the evaluation comparison.Source: Table III · p. 5, A. Dataset 1: Tox21 · p. 5
• On score data, SA-MTL without an ensemble was substandard versus DeepTox and SCFP, while its ensemble performed better than both and added 0.036 AUC over non-ensembled SA-MTL.The score data had a slightly different distribution; SA-MTL's ensemble summed five output probabilities from the same model with different initial weights.Source: A. Dataset 1: Tox21 · p. 5, Table III · p. 5

B. Dataset 2: SIDER

• SIDER contains 1,427 instances across 27 targets, uses a random split, and allows multiple class labels per compound.The adverse reactions were extracted from public documents and package inserts.Source: B. Dataset 2: SIDER · p. 5
• Despite fewer instances and more classes than Tox21, SA-MTL exhibited good performance on SIDER.Table IV presents the SIDER evaluation comparison.Source: B. Dataset 2: SIDER · p. 5

C. Dataset 3: HIV

• The evaluation used a binary target derived from the original three HIV activity categories and scaffold splitting, which separates structurally different molecules across subsets.The authors note a scope limitation: only ligand SMILES, rather than varied mechanisms, were used as input.Source: C. Dataset 3: HIV · p. 6
• Under scaffold splitting, SA-MTL achieved higher AUC than the previous best-performing KernelSVM, though its AUC was reduced versus Transformer-CNN.Table V presents the HIV evaluation results.Source: C. Dataset 3: HIV · p. 6
• For HIV, a conventional machine-learning model achieved a better result than the deep-learning models.Source: C. Dataset 3: HIV · p. 6

D. Dataset 4&5: BBBP & clintox

• 0.966 AUC on BBBP under scaffold split; the authors note its positive-to-negative ratio differs from the other datasets as one reason for the high score.The dataset was filtered from 2,052 compounds to 2,031 after compounds with abnormal atom valence were removed.Source: D. Dataset 4&5: BBBP & clintox · p. 6
• Above 0.95 AUC on both BBBP and ClinTox; the authors describe the datasets as having no significant noise and suitable prediction objectives.Table VI presents the evaluation results against other models.Source: Table VI · p. 6, D. Dataset 4&5: BBBP & clintox · p. 6
• Table VI compares BBBP and ClinTox evaluation results with other models, but the supplied passages do not give its cell values.Other models’ results were taken from their papers.Source: Table VI · p. 6, D. Dataset 4&5: BBBP & clintox · p. 6

E. Ablation Study

• .798 AUC without self-attention; removing multi-task learning also worsened performance, while two-character embeddings reached 0.9 validation performance but sometimes 0.87 on test.The occasional test degradation was attributed to random distribution of two-character atoms; the ablation results averaged five Tox21 tests.Source: E. Ablation Study · p. 6, Table VII · p. 7
• Removing the first CNN layer worsened the original result, and self-attention was insufficient to learn shared factors across imbalanced tasks.The authors suggest additional data and more explicit objective-function feedback could improve performance.Source: E. Ablation Study · pp. 6–7
• Table VII reports performance changes from modifying model features in Tox21, with ablation results averaged across five tests.Source: E. Ablation Study · p. 6, Table VII · p. 7
• Replacing the CNN with a GRU still gave appropriate results but was slower; the discrete output layer outperformed max pooling, which may suit simpler models.The authors note that the choice of dimension-reduction layer can affect complex components such as self-attention.Source: E. Ablation Study · p. 7
• Five multi-heads had no significant impact, and position embeddings left performance unchanged; the authors relate limited positional effects to SMILES lacking grammatical position meanings.The paper also describes attention-head redundancy as an over-parametrization issue, citing Kovaleva et al. [32].Source: E. Ablation Study · p. 7

V. CONCLUSION

• SA-MTL achieved state-of-the-art performance on Tox21 and several other datasets, and better performance than existing models on several datasets.Source: §V · p. 7
• The model is descriptor-free because SMILES is its direct input, and it applies self-attention with multi-task learning to predict chemical properties.Source: §V · p. 7
• The authors report that the Transformer encoder can serve as a classification predictor.Source: §V · p. 7