Source-linked AI summary
Deep Learning for Computational Chemistry
Garrett B. Goh, Nathan O. Hodas, Abhinav Vishnu
TL;DR
Computational chemistry is experiencing renewed interest in deep learning, but its distinctions from traditional methods and applications across the field require synthesis. This review introduces deep neural networks, surveys their computational-chemistry uses, and reports frequent superiority over traditional machine-learning algorithms, while noting limitations in some settings.
Problem
The review addresses the need to understand deep learning’s role in computational chemistry and how its performance compares with traditional machine-learning approaches.
Method
The paper provides an introductory overview of deep neural networks and surveys applications across computational chemistry, including structural biology, materials design, and property prediction.
Results
Deep neural-network models are frequently superior to traditional machine-learning algorithms across the reviewed computational-chemistry applications.
Takeaways & Limitations
Deep learning is presented as a broadly applicable tool for computational chemistry and as a possible hypothesis-generation device linked to chemical concepts such as toxicophores.
Takeaways & Limitations
In some applications, the performance improvement over existing models is not substantial, and small sample sizes can make incremental accuracy gains difficult to establish.
Abstract
from arXiv · showhide
The rise and fall of artificial neural networks is well documented in the scientific literature of both computer science and computational chemistry. Yet almost two decades later, we are now seeing a resurgence of interest in deep learning, a machine learning algorithm based on multilayer neural networks. Within the last few years, we have seen the transformative impact of deep learning in many domains, particularly in speech recognition and computer vision, to the extent that the majority of expert practitioners in those field are now regularly eschewing prior established models in favor of deep learning models. In this review, we provide an introductory overview into the theory of deep neural networks and their unique properties that distinguish them from traditional machine learning algorithms used in cheminformatics. By providing an overview of the variety of emerging applications of deep neural networks, we highlight its ubiquity and broad applicability to a wide range of challenges in the field, including QSAR, virtual screening, protein structure prediction, quantum chemistry, materials design and property prediction. In reviewing the performance of deep neural networks, we observed a consistent outperformance against non-neural networks state-of-the-art models across disparate research topics, and deep neural network based models often exceeded the "glass ceiling" expectations of their respective tasks. Coupled with the maturity of GPU-accelerated computing for training deep neural networks and the exponential growth of chemical data on which to train these networks on, we anticipate that deep learning algorithms will be a valuable tool for computational chemistry.
1. Introduction
Deep learning is reviewed as a multilayer neural-network approach whose hierarchical representations and automatic feature engineering distinguish it from traditional machine learning. The review introduces these differences and surveys applications across computational chemistry.
- Deep learning models won the 2012 Merck activity prediction challenge and outperformed Merck’s internal baseline without a chemist or biologist on the team.
- Deep learning models also achieved top positions in NIH’s 2014 Tox21 toxicity prediction challenge.
- Traditional shallow learning relies on expert-provided features and transformations, then learns how to balance those inputs to produce an output.
- Deep neural networks arrange processing units hierarchically and combine successive nonlinear functions to learn multiple levels of representation.
- Deep learning can use raw data directly because intermediate outputs become learned features for later layers, potentially reducing reliance on expert intervention.
- The review provides an introductory account of deep neural networks, contrasts them with traditional computational-chemistry machine learning, and surveys applications and performance across the field.
2. Deep Learning 101
Deep neural networks extend artificial neural networks with multiple hidden layers that learn increasingly complex representations through nonlinear transformations. Their practical training became feasible through backpropagation and related developments addressing vanishing gradients and overfitting, enabling strong performance in computer vision.
- Artificial neural networks: Artificial neural networks map many inputs to target outputs through layered neurons and nonlinear functions with adjustable weights.The weights are adjusted during training to minimize prediction error.
- Deep neural networks: Deep neural networks differ from traditional ANNs by using multiple hidden layers to construct increasingly complex and abstract representations.Increasing layer depth or width increases the model’s capacity to learn abstract features.
- Training: Backpropagation assigns error attribution backward through the network, while gradient descent iteratively updates neuron weights to minimize prediction error.The optimization process is analogous to steepest descent, but minimizes an output error function rather than molecular energy.
- Training challenges: The vanishing-gradient problem made deep networks difficult to train because error signals decay across layers, increasing convergence time and overfitting risk.Sigmoidal functions’ finite dynamic range contributed to error-signal decay and effectively limited network depth.
- Empirical impact: DNNs achieved a 16.4% ImageNet error rate versus 26.2% for the second-best traditional-machine-learning model, later reaching under 5.0% and exceeding human performance in 2015.The review also reports human-level image-classification accuracy of approximately 5% by 2015.
3. Computer-Aided Drug Design
Deep neural networks were applied broadly in computer-aided drug design, especially QSAR and reactivity prediction. Across several studies, they generally outperformed traditional machine-learning models and supported multi-task, mechanistic, and transferable predictions.
- QSAR: Multi-task DNNs outperformed traditional models in 14 of 19 assay predictions and matched their performance in the remaining 5.The models developed shared learned representations across tasks, which could help predict tasks difficult to solve in isolation.
- QSAR: AUC performance lifts over logistic regression and random forests ranged from 0.02 to 0.09 for multi-task DNNs.The review reports consistent outperformance across the evaluated QSAR comparisons.
- QSAR: A DNN using UGRNN molecular encoding achieved AUC 0.955, exceeding models using Mold descriptors at AUC 0.931 and PaDEL descriptors at AUC 0.895.The comparison suggests that molecular encoding can materially affect DNN performance.
- Reactivity prediction: A mechanistic DNN predicted epoxidation formation and identified specific epoxidized bonds, with AUCs of 94.9% and 79.3% for its principal classifications.It also separated aromatic or double-bond sites with AUCs of 92.5% and 95.1%.
- Reactivity prediction: DNN models identified reactive sites with 90.8% accuracy and separated reactive from unreactive molecules with 80.6% accuracy.Predictions correlated with quantitative GSH reactivity in more chemically diverse external datasets, indicating broader chemical-space generalizability.
- Toxicity prediction: In Tox21, multi-task DNNs outperformed single-task models in 10 of 12 assay predictions, while DNNs outperformed in 10 of 15 cases in another analysis.The review also reports that 99% of neurons in one first hidden layer were significantly associated with at least one known toxicophore feature.
4. Computational Structural Biology
Deep neural networks advanced computational structural biology, particularly protein contact prediction. They surpassed historical accuracy limits and improved predictions of contacts, angles, torsions, and secondary structure from sequence-derived information.
- Protein contact prediction: Protein contact prediction historically remained below 30% accuracy, but DNN-based models breached that barrier.The review identifies this as a historical glass ceiling in long-range contact prediction.
- Protein contact prediction: CMAPpro achieved 36% contact-prediction accuracy, compared with 32% for DNN and NN+CA variants and 26% for the previous state of the art.Both the deep architecture and CA features were required to reach 36%.
- Protein contact prediction: DNCON achieved 34.1% cross-validated accuracy and was approximately 3% better than state-of-the-art algorithms.Reported comparisons were 32.6% versus 29.7% for ProC_S3 and 32.9% versus 28.5% for SVMcon.
- Angles and secondary structure: Sequence-based DNN models achieved mean absolute errors of 22° and 33° for φ and ψ dihedrals, and 9° and 34° for θ and τ.Independent-test secondary-structure accuracies ranged from 81.3% to 82.3%.
- Angles and secondary structure: Using DNN-based predictions, protein 3D structures were modeled with an average RMSD of 1.9 Å.The review also reports secondary-structure accuracy gains of 2% and angle or torsion MAE reductions of 1–3° while maintaining ASA performance.
- Sequence-based prediction: DeepBind outperformed all 26 existing algorithms and ranked first among 15 teams in the DREAM5 submission.Models trained on in vitro data also performed well when scoring in vivo data.
5. Quantum Chemistry
Machine learning was explored as a substitute for, or supplement to, quantum-mechanical calculations. Results show that molecular representation strongly affects accuracy, while DNN applications remained less mature than in other computational-chemistry areas.
- Molecular representation: A randomized Coulomb matrix reduced atomization-energy MAE to 3.0 kcal/mol, highlighting the importance of molecular representation.Sorting representations by information lowered accuracy across the tested machine-learning algorithms.
- Molecular representation: The benchmarked shallow neural network achieved MAE 3.5 kcal/mol, compared with 3.0 kcal/mol for nonlinear regression.The review notes that the shallow architecture and limited dataset did not constitute a true DNN implementation.
- Multi-task prediction: A multi-task model achieved atomization-energy MAE 0.16 eV and MAEs of 0.11 to 0.17 eV for other electronic properties.These errors were similar to the corresponding errors of the QM levels used to construct the training data.
- Hybrid QM/ML methods: Δ-learning used a machine-learning correction term to predict G4MP2-level properties from DFT-calculated properties.This composite QM/ML approach combines fast approximate QM calculations with data-driven estimates trained on expensive results.
- Scope and maturity: DNN applications in quantum chemistry appeared less developed than applications in computer-aided drug design and computational structural biology.The review notes that a DNN-based Δ-learning improvement had not yet been reported.
6. Computational Material Design
Computational material design applies machine learning to property prediction and synthesis, areas constrained by incomplete theoretical understanding and experimental trial-and-error. Early results show promising prediction accuracy and synthesis success, although DNN applications remained at an early stage.
- Scope and maturity: Deep learning applications in material design were still described as being in their infancy.The review calls for comparison with traditional QSPR methods and emerging rational materials-design approaches.
- Motivation and scope: Material discovery is constrained by incomplete understanding, serendipity, and experimental synthesis and characterization bottlenecks.The review describes these factors as central obstacles to molecular materials design.
- Materials synthesis: An SVM model trained with failed reactions achieved an 89% success rate for synthesizing target vanadium selenites, versus 78% for human intuition.The failed reactions were drawn from archived laboratory notebooks.
- Materials property prediction: For organic photovoltaic materials, graph-fingerprint representations produced consistent accuracy across HOMO, LUMO, and PCE predictions.The study used molecular graphs because high-quality 3D data for Coulomb matrices were unavailable.
- Materials property prediction: Testing errors for HOMO and LUMO prediction were 0.15 eV and 0.12 eV, respectively, representing an almost 5-fold improvement over prior non-neural methods.The dataset contained 2,000,000 compounds, with another 50,000 used as the test set.
7. Reservations about Deep Learning and Of Being a Black Box
Deep learning is broadly applicable across computational chemistry and often matches or exceeds state-of-the-art non-neural models. Its gains are less pronounced than in speech recognition and computer vision, while limited data and interpretability remain important reservations.
- Applications: Deep learning has been applied broadly across computational chemistry sub-fields, including QSAR, protein structure prediction, and other property-prediction tasks.The review summarizes preliminary DNN-based publications across many applications.
- Performance: DNN-based models are almost always equivalent to existing state-of-the-art non-neural models and sometimes perform better.The comparison is framed as a meta-analysis using similar or identical datasets and appropriate published or competition-based comparisons.
- Reservations: Performance gains are less significant than the improvements associated with deep learning in speech recognition and computer vision.The review attributes one possible explanation to chemistry data being smaller and more expensive to generate because experiments or computations are required.
- Performance: 0.846 AUC was reported for DeepTox, whose MT-DNN-based model was the top-performing model.Other reported examples include 0.898 AUC for DNA-reactivity prediction and 0.949 AUC for epoxidation-site prediction.
- Reservations: Deep learning remains difficult to interpret because DNNs are black boxes whose learned representations and predictions are hard to understand.The review also notes that current implementations do not provide the conceptual elegance of first-principles models based on physical laws.
- Future direction: The review proposes that deep learning could help suggest or decide which approximations to make in future AI-assisted chemistry research.This possibility is presented as an alternative to relying only on human expert knowledge when developing computational chemistry models.
8. Conclusion
The review concludes that deep learning has broad, increasingly strong applications in computational chemistry, often outperforming traditional machine-learning models. Its automatic feature extraction and learned representations may also support interpretation and hypothesis generation, although chemistry applications remain relatively recent and preliminary.
- Deep neural networks can learn representations and extract features from raw data, so engineered molecular descriptors may not be necessary.
- The review characterizes deep learning’s computational-chemistry impact as more recent and preliminary than its impact in speech recognition and computer vision.
- Deep learning is broadly applicable across computer-aided drug design, computational structural biology, quantum chemistry, and materials design.
- Deep neural network models frequently outperform traditional machine-learning algorithms across computational-chemistry applications.
- As prediction complexity and dataset size increase, deep learning progresses from frequent to consistent outperformance of traditional models.
- Learned deep-neural-network features may correspond to chemical concepts such as toxicophores, supporting emerging efforts to improve interpretability.
- Deep learning may serve not only as a high-performance prediction tool but also as a hypothesis-generation device.