Deep Learning Based Regression and Multi-class Models for Acute Oral Toxicity Prediction with Automatic Chemical Feature Extraction

Youjun Xu; Jianfeng Pei; Luhua Lai

Source-linked AI summary

Deep Learning Based Regression and Multi-class Models for Acute Oral Toxicity Prediction with Automatic Chemical Feature Extraction

Youjun Xu, Jianfeng Pei, Luhua Lai

arXiv:1704.04718v3Apr 16, 2017stat.MLcs.LGq-bio.QM

TL;DR

QSPR models often struggle to achieve high predictive power while remaining interpretable, motivating improved approaches for acute oral toxicity prediction. The study uses MGE-CNN to learn molecular features automatically and builds regression, multi-classification, and multi-task AOT models. The resulting models outperformed previously reported models, with strong external-test performance across regression and classification tasks.

Problem
QSPR studies seek interpretable relationships between chemical properties and features, but predictive power and interpretability are difficult to achieve simultaneously.
Method
MGE-CNN was used to construct regression, multi-classification, and multi-task acute oral toxicity models from molecular features without manual descriptor or fingerprint selection.
Results
0.864 PCC2 and 0.195 MAE were achieved by deepAOT-R, while deepAOT-C reached 95.8% accuracy on test set I and 96.3% on test set II.
Takeaways & Limitations
The study demonstrates that MGE-CNN-based AOT regression and classification models can combine strong predictive performance with interpretable internal features.
Takeaways & Limitations
Prior-neighbor methods depend on experimental data from query neighbors, so prediction capability is associated with chemical diversity and structural coverage of the training set.

Abstract

from arXiv · show

For quantitative structure-property relationship (QSPR) studies in chemoinformatics, it is important to get interpretable relationship between chemical properties and chemical features. However, the predictive power and interpretability of QSPR models are usually two different objectives that are difficult to achieve simultaneously. A deep learning architecture using molecular graph encoding convolutional neural networks (MGE-CNN) provided a universal strategy to construct interpretable QSPR models with high predictive power. Instead of using application-specific preset molecular descriptors or fingerprints, the models can be resolved using raw and pertinent features without manual intervention or selection. In this study, we developed acute oral toxicity (AOT) models of compounds using the MGE-CNN architecture as a case study. Three types of high-level predictive models: regression model (deepAOT-R), multi-classification model (deepAOT-C) and multi-task model (deepAOT-CR) for AOT evaluation were constructed. These models highly outperformed previously reported models. For the two external datasets containing 1673 (test set I) and 375 (test set II) compounds, the R2 and mean absolute error (MAE) of deepAOT-R on the test set I were 0.864 and 0.195, and the prediction accuracy of deepAOT-C was 95.5% and 96.3% on the test set I and II, respectively. The two external prediction accuracy of deepAOT-CR is 95.0% and 94.1%, while the R2 and MAE are 0.861 and 0.204 for test set I, respectively.

Introduction

Acute oral toxicity prediction requires reliable in silico methods because animal testing is costly, time-consuming, ethically constrained, and existing QSPR models often struggle to combine predictive power with interpretability. The study applies MGE-CNN to construct regression, classification, and multi-task AOT models that learn raw molecular features automatically and support model interpretation.

Motivation: Animal experiments are required for accurate acute toxicity assessment but are complicated, costly, time-consuming, and constrained by animal rights.LD50 testing measures acute toxicity after administration of a single dose.
Motivation: Predictive power and interpretability are difficult to achieve simultaneously in QSPR models.Linear models can expose features but may have low predictive power, whereas high-performing models may appear black-box.
Prior limitations: Existing molecular representation methods depend on complicated, chemically informed descriptors or fingerprints that must be selected for the application.Appropriate molecular representations are considered crucial for accurate prediction, but automatic representation could simplify and accelerate QSPR development.
MGE-CNN approach: MGE-CNN uses raw and pertinent molecular features instead of application-specific descriptors or fingerprints, without manual intervention or feature selection.The approach represents small molecules as undirected graphs with atoms as nodes and bonds as edges, then encodes the graph for neural-network prediction.
MGE-CNN approach: MGE-CNN was designed to improve hyperparameters, training and prediction strategy, multi-output handling, and model interpretation.The framework was developed as an improved CNN-based molecular graph encoding approach.
Study design: The study constructs deepAOT-R, deepAOT-C, and deepAOT-CR using experimental rat oral LD50 data and U.S. EPA toxicity categories, with two external test datasets for evaluation.deepAOT-R is the consensus regression model, deepAOT-C is the best multi-classification model, and deepAOT-CR is the multi-task model.

MGE-CNN

MGE-CNN converts molecular graphs derived from canonical SMILES into learned fingerprints and toxicity outputs, integrating feature learning with model construction. Its forward and backward analyses support prediction and interpretation through shallow models and activation fragments.

Architecture: Canonical SMILES are converted into molecular structural graphs whose atom and bond information characterize each atom’s chemical environment.Atom features include atomic type, degree, implicit valence, attached hydrogen count, and aromatic atoms; bond features include bond type and structural status.
Architecture: Molecular graph encoding with CNN enables continuous information transmission and end-to-end differentiable optimization by gradient descent.The architecture supports automatic feature learning without manual feature selection.
Architecture: The architecture integrates feature learning and model construction, producing deep fingerprints after supervised training for toxicity prediction.The resulting system constructs regression and classification models for estimating acute oral toxicity in rats.
Interpretability: Deep fingerprints are used to build MLR and SVM models that predict acute oral toxicity values or labels and assess the fingerprints’ support for shallow decisions.Performance gains in shallow models would suggest optimized predictive features from MGE-CNN.
Interpretability: Relevant deep-fingerprint features are traced back to atoms and mapped onto activation fragments for comparison with reported toxicity alerts.Fragments with the largest contribution to chemical toxicity are compared with ToxAlerts to validate inference capability.

Training deepAOT models

The deepAOT training process addresses the time-consuming selection of MGE-CNN hyperparameters through random search, validation-based model selection, and gradient-based optimization. Adam, shuffling, normalization, and task-specific losses are used during training.

Hyperparameter optimization: Hyperparameter selection is time-consuming because MGE-CNN requires appropriate settings for a new dataset.The parameters include fingerprint length and depth, convolution-kernel width, hidden-layer size, regularization, initial-weight scale, and learning-rate step size.
Hyperparameter optimization: 500 random hyperparameter sets are evaluated on 2045 validation compounds, with the top 10 selected by lowest RMSE for regression or highest ACC for classification.The selected models proceed to further training and model selection.
Training procedure: Training examples are shuffled after each epoch to reduce optimization bias, and training-set means and standard deviations normalize validation or external-test inputs.Normalization is intended to reduce bias caused by different data distributions.
Optimization: Regression training uses mini-batch gradient descent with Adam, while the loss function includes an L2 penalty.Autograd is used for automatic differentiation during optimization.

Experimental Setup

The study uses curated acute oral toxicity data divided into training, validation, and two external test sets, then trains MGE-CNN regression and classification models. DeepAOT models are externally challenged and interpreted through fingerprint analysis and toxicity-alert comparisons.

Data Collection and Preparation: The dataset combines admetSAR, MDL Toxicity Database, and U.S. EPA TEST sources, with standardized SMILES strings prepared before modeling.Structure Checker and Standardizer modules were used to correct valence errors and standardize structures.
Data Collection and Preparation: The final training and validation sets contain 8080 and 2045 compounds with measured LD50 values, while two external datasets contain 1673 and 375 compounds.Test set II has category labels but lacks exact experimental acute oral LD50 values.
Model Construction: Regression models target log(LD50), whereas classification models target four defined toxicity categories represented by four MGE-CNN outputs.The classification objective uses the softmax loss function.
Model Construction: For regression, 500 hyperparameter combinations are trained for 750 epochs, and the 10 lowest-validation-RMSE sets are retained to construct the consensus deepAOT-R model.The consensus model averages the retained models and is challenged on external test set I because test set II lacks LD50 values.
Model Construction: Classification models are selected by validation accuracy, further trained, and challenged on both external test sets; consistency between classification and regression outcomes is analyzed.The best-trained weights are selected using validation accuracy.

Evaluation Metrics

Regression models are evaluated with error and correlation measures, while multi-class models are assessed from confusion matrices and classification metrics. A regression-based consensus model is also used to assess classification performance on external data.

Regression Metrics: RMSE, MAE, and PCC2 are used to evaluate regression models on validation and external test sets.RMSE is defined from prediction residuals, while PCC describes linear correlation between predicted and target values.
Classification Metrics: Sensitivity, positive predictive value, and accuracy are calculated from the multi-class confusion matrix to assess classification models.These metrics are calculated as shown in Figure 1C.
Classification Assessment: The consensus deepAOT-R model is used to assess classification performance in addition to the multi-class model evaluation.This provides a regression-based classification assessment alongside confusion-matrix metrics.
Classification Assessment: For external test sets, deepAOT-R predictions are mapped into category space and converted into labels using output ranges within 1 RMSE.A prediction is considered correct when its predicted-label range contains the actual target label.

Results and Discussion

MGE-CNN-based models showed strong regression, classification, and multi-task performance for acute oral toxicity, while internal activations provided interpretable toxicity-related fragments. The models also improved on prior approaches and supported simultaneous prediction of regression and classification outcomes.

Regression performance: 0.864 PCC2 was achieved by the consensus deepAOT-R model on test set I, improving the corresponding index by 0.060.The consensus model reached PCC2 values of 0.853, 0.917, and 0.864 on the training, validation, and test set I data, respectively.
Classification performance: 95.5% and 96.3% ACC were achieved by deepAOT-C on external test sets I and II, respectively.deepAOT-C also exceeded the previously reported model on the validation and two external test sets.
Multi-task performance: p-value of paired t-test < 0.001 indicated that deepAOT-CR improved category IV distinguishing capability.The deepAOT-R category IV errors were mostly lower than zero, while allowing a 1-fold ValRMSE wiggle room improved classification performance.
Interpretability: Deep fingerprints correlated with molecular-topological fingerprints and internal activations highlighted reported toxicity structural alerts.The models could infer toxicity alerts from highlighted fragments, including α,β-unsaturated nitriles, allyl (thio)phosphates, and thiocarbonyl groups.

Conclusion

MGE-CNN-based regression, classification, and multi-task models provided strong acute oral toxicity prediction while automatically learning features and supporting fragment-level interpretation. The models also showed competitive shallow-model performance using deep fingerprints and identified toxicity-related fragments without prior fragment knowledge.

Model development: MGE-CNN constructed regression and multi-classification models for acute oral toxicity prediction.A multi-task model was also developed for simultaneous assessment of regression and classification problems.
Regression performance: PCC2 0.864, RMSE 0.268, and MAE 0.195 were achieved by the consensus deepAOT-R model.These results were reported for the external prediction setting described in the conclusion.
Classification performance: ACC of 92.1% on validation, 95.8% on test set I, and 96.3% on test set II was reported for deepAOT-C.The classification performance was compared with the best reported multi-classification models.
Deep fingerprints: Deep fingerprints supported shallow models better than application-specific molecular descriptors or fingerprints.The consensus MLR model reached PCC2 0.696 and MAE 0.348 on 3718 compounds, while the best SVM reached TestIACC 86.6% and TestIIACC 93.6%.
Deep fingerprints: Deep fingerprints were highly correlated with topological structure-based fingerprints, supporting their use in AOT-related shallow decision systems.The conclusion attributes the success of deep fingerprints to their learned chemical representation.
Interpretability: Tracing learned toxicity features to atomic fragments produced highlighted fragments that were consistent with reported toxicity associations.The fragments inferred by RM4 and CM1 were compared with reported TAs, and the paper states that this consistency supports model interpretability.

ASSOCIATED CONTENT

Supporting information is available free of charge through the ACS publication site.

Access: Supporting Information Available is provided free of charge at the ACS publication website.The supplied passage gives the access statement and publication-site URL.
Access: The associated content is linked to the paper through the ACS publication site.The passage identifies the location rather than describing the files’ scientific contents.
Access: Readers can consult the supporting material through the stated publication-site address.The supplied passage provides an access route for the associated content.

Supporting Information Available

The supporting information contains figures and tables documenting data curation, model variability and errors, confusion matrices, feature comparisons, and toxicity-fragment interpretation.

Tables: Tables S1–S3 document hyper-parameter ranges, top regression models, and additional toxicity-association fragment comparisons.Table 6 separately compares toxicity associations with fragments inferred by RM4.
Data curation: The supporting files include a chemical data-curation workflow using valence analysis, structure correction, standardization, and canonical SMILES transformation.The workflow defines RVA, TEV/FEV sets, charged nitrogen notation, and the roles of the Structure Checker and Standardizer modules.
Model evaluation: Figures S2–S4 and S6 report model variability, prediction-error distributions, and confusion matrices for deepAOT-R and deepAOT-CR.Figure S5 provides the classification-task confusion matrix for deepAOT-CR.
Benchmark comparisons: Figures S7 and S8 compare MGE-CNN-derived approaches with shallow models using deep, ECFP4, FCFP4, MACCS, or other fingerprints.The comparisons include MLR, LLR, and SVMOAO models and report ACC for the SVMOAO comparison.
Fragment interpretation: Figure S9 traces a toxicity feature from flocoumafen fingerprints through model layers to identify a maximum-activation toxicity fragment.Blue arrows denote trained weights, pink arrows trace the feature, and the inferred fragment is attributed to RM4.