PaperlayerView exact revision on arXiv

Source-linked AI summary

Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Models

Garrett B. Goh, Charles Siegel, Abhinav Vishnu, Nathan O. Hodas, Nathan Baker

arXiv:1706.06689v1stat.MLcs.AIcs.CEcs.CVcs.LG

TL;DR

The paper asks whether deep neural networks can reasonably replace human intelligence in feature extraction for chemical-property prediction. Chemception learns molecular representations with a deep convolutional network, and the paper reports task-specific outcomes including a validation/test AUC of 0.744/0.752 for HIV activity prediction.

  • Problem

    The paper asks whether a deep neural network can reasonably replace human intelligence in feature extraction for chemical-property prediction.

    Source: §3.3 · p. 23

  • Method

    Chemception uses a deep convolutional network to learn representations and extract features for chemical-property prediction.

    Source: §1 · p. 3, §4 · p. 33

  • Results

    Across activity, solvation, and toxicity evaluations, the paper reports task-specific outcomes including a validation/test AUC of 0.744/0.752 for HIV activity prediction.

    Source: §4 · p. 33

  • Takeaways & Limitations

    The work demonstrates the plausibility of deep neural networks assisting computational chemistry research through learned feature engineering.

    Source: §1 · p. 4, §4 · p. 33

    The authors note that some network levels may learn more features than others, leaving potential for further architectural optimization.

    Source: §3.5 · p. 27

Abstract

from arXiv · show

In the last few years, we have seen the transformative impact of deep learning in many applications, particularly in speech recognition and computer vision. Inspired by Google's Inception-ResNet deep convolutional neural network (CNN) for image classification, we have developed "Chemception", a deep CNN for the prediction of chemical properties, using just the images of 2D drawings of molecules. We develop Chemception without providing any additional explicit chemistry knowledge, such as basic concepts like periodicity, or advanced features like molecular descriptors and fingerprints. We then show how Chemception can serve as a general-purpose neural network architecture for predicting toxicity, activity, and solvation properties when trained on a modest database of 600 to 40,000 compounds. When compared to multi-layer perceptron (MLP) deep neural networks trained with ECFP fingerprints, Chemception slightly outperforms in activity and solvation prediction and slightly underperforms in toxicity prediction. Having matched the performance of expert-developed QSAR/QSPR deep learning models, our work demonstrates the plausibility of using deep neural networks to assist in computational chemistry research, where the feature engineering process is performed primarily by a deep learning algorithm.

1. Introduction

Chemception learns from 2D molecular drawings with minimal chemistry knowledge and achieves performance comparable to expert-developed QSAR/QSPR models. The introduction presents it as a way to reduce feature-engineering effort and potentially generalize across small-molecule properties.

1. Introduction

  • Deep learning’s hierarchical nonlinear functions can learn representations and extract predictive features from input data, shifting feature engineering away from human experts.The introduction describes this representation-learning ability as a key capability behind deep learning’s impact in computer vision.Source: §1 · pp. 3–4
  • Chemception uses 2D molecular drawings and minimal chemical knowledge, allowing the network to learn its own representations and features.The drawings require no more than high-school-level chemistry knowledge to create, and the model does not use advanced chemistry knowledge as input.Source: §1 · p. 6
  • Chemception achieves performance on par with expert-developed QSAR/QSPR models for toxicity, activity, and solvation predictions.The comparison is to models based on molecular descriptors, fingerprints, and other engineered features.Source: §1 · pp. 6–7
  • Using Chemception as a “machine intelligence” tool required virtually zero effort to develop features and could obviate the need for molecular descriptors and fingerprints.The authors frame this as useful where existing expert-developed features do not provide sufficient performance.Source: §1 · p. 7
  • The same architecture performs similarly across different properties without property-specific network-design tuning, suggesting it may generalize to other small-molecule predictions.The studied scope includes solvation free energies, in vitro HIV activity, and in vivo toxicity.Source: §1 · pp. 6–7

2. Methods · 2.1 Computational Framework Overview

Chemception converts SMILES into 2D molecular images and trains a supervised convolutional neural network to predict chemical properties, using minimal chemistry knowledge and no descriptor- or fingerprint-based features.

2.1 Computational Framework Overview

  • Chemception’s end-to-end workflow requires minimal chemical knowledge beyond generating 2D chemical structures.The framework is illustrated in Figure 2.Source: §2.1 · p. 8, Figure 2 · p. 8
  • SMILES strings are converted into their corresponding 2D molecular structures.These structures are then mapped to an 80 x 80 image input.Source: §2.1 · p. 8, Figure 2 · p. 8
  • The mapped chemical images serve as input arrays for supervised convolutional neural network training.Training uses measured chemical properties.Source: §2.1 · p. 8
  • The framework uses no additional chemistry-inspired features, such as molecular descriptors or fingerprints.Its inputs are generated chemical images.Source: §2.1 · p. 8
  • The trained deep neural network predicts toxicity, activity, and solvation properties.These are the chemical-property targets named for the framework.Source: §2.1 · p. 8

2.2 Data Preparation

The preparation pipeline converted database SMILES into discretized molecular images for CNN training, using cheminformatics software and a grayscale encoding for atoms, bonds, and empty grid regions.

2.2 Data Preparation

  • Python bindings to OpenBabel33, Pybel34, and RDKit35 through Cinfony36 decoded database SMILES strings,37-38 into corresponding 2D molecular structures.Source: §2.2 · p. 8
  • Each molecule’s 2D coordinates were mapped to an 80 x 80 grid at 0.5 Å per pixel.Source: §2.2 · p. 8
  • Atoms were encoded by atomic-mass-unit number, bonds as 2, and empty pixels as 0.The resulting discretized molecule image was parsed into the deep CNN for training.Source: §2.2 · p. 9

2.3 Chemistry Knowledge Used

Chemception used molecule images without explicit chemistry-inspired features or even basic chemical concepts, leaving property prediction dependent on representations learned from the images.

2.3 Chemistry Knowledge Used

  • Chemception received no molecular descriptors or fingerprints and was not explicitly given basic concepts such as valency or periodicity.Cheminformatics software was used for data preparation and database compatibility, not as a source of model features.Source: §2.3 · p. 9
  • The grayscale image encoded atoms and bonds in one channel, so the network could potentially learn basic chemistry but had to distinguish atom, bond, and empty-space information.The passage notes that this shared-channel representation may make learning basic chemistry concepts nonstraightforward.Source: §2.3 · p. 9
  • Property prediction therefore depended on Chemception learning relevant representations from images and automatically engineering its own analogous molecular descriptors.The input could be photographs of consistently and correctly drawn structures requiring no more than high-school-level chemistry, equivalent to computer-generated structures from the preparation workflow.Source: §2.3 · pp. 9–10

2.4 Dataset Description

Chemception was evaluated on several publicly available datasets, mostly from MoleculeNet, spanning toxicity, hydration free energy, and HIV activity.

2.4 Dataset Description

  • The evaluation used several publicly available datasets, mostly from MoleculeNet; Table 1 summarizes the datasets.Source: Table 1 · p. 10, §2.4 · p. 10
  • Tox21 includes 8,014 compounds measured across 12 toxicity endpoints and poses a binary toxic/non-toxic classification task [40].Source: §2.4 · p. 10
  • FreeSolv contains 643 compounds with measured hydration free energies from −25.5 to 3.4 kcal/mol; alchemical simulation energies provide physics-based model comparisons [41].Source: §2.4 · p. 10
  • The HIV screen measured in-vitro inhibition of HIV replication for 41,913 compounds and was curated into binary active/inactive labels [42, 39].Source: §2.4 · pp. 10–11

2.5 Dataset Preprocessing

Chemception training used five-fold cross-validation, validation-based early stopping, and separate holdout tests to assess generalizability. Classification splits were stratified and minority-class oversampling addressed class imbalance.

2.5 Dataset Preprocessing

  • Five-fold cross-validation was used for training, with validation performance guiding early stopping and a separate test set indicating generalizability.Source: §2.5 · p. 11
  • Tox21 and HIV held out 1/6 of data for testing, while Freesolv held out 1/10; the remainder entered random five-fold validation, with classification splits stratified by class ratio.Source: §2.5 · p. 11
  • Class imbalance (1:5 to 1:34) was addressed by oversampling minority-class examples after stratification, allowing repeats within each split while keeping different splits nonoverlapping.Source: §2.5 · p. 11

2.6 Deep Neural Networks

The section explains how deep neural networks learn through nonlinear, weighted layers and introduces convolutional networks as image-feature learners. It connects these ideas to Chemception’s convolutional and Inception-ResNet architecture.

2.6 Deep Neural Networks

  • Each neuron maps its input to an output, while training adjusts its weights to minimize prediction error.The network represents inputs as vectors and layer weights as matrices, then applies matrix-vector multiplication followed by a nonlinear activation.Source: §2.6 · p. 12
  • Chemception uses ReLU activations, which enable training many layers to form a deep neural network.The section notes that ReLU is used in most modern neural networks, including Chemception.Source: §2.6 · p. 12
  • Backpropagation assigns error corrections backward through the network, while gradient descent searches for minima in the error surface.The paper likens gradient descent conceptually to steepest descent in molecular dynamics, but here the objective is the output loss and the updates are to neuron weights.Source: §2.6 · p. 12
  • CNN filters extract task-relevant image features from local windows; shared weights detect features across locations while preserving spatial structure.Unlike fully connected layers, each filter receives input from a small contiguous window in the previous layer.Source: §2.6 · p. 13
  • Chemception combines a 4x4 stride-2 stem, Inception-ResNet segments with 1–3 stacked blocks, and single-block reduction segments.The accompanying architecture caption points to a separate figure for the internal inception and reduction block details.Source: Figure 3 · p. 14, Figure 4 · p. 15

2.7 Convolutional Neural Network Design

Chemception adapts Inception-ResNet v2 [47], combining Inception modules [29] with deep residual learning [30], for molecular-image prediction. Its design largely retains the architecture’s module regularities while adjusting block counts, layer widths, and the stem.

2.7 Convolutional Neural Network Design

  • Chemception is based on Inception-ResNet v2 [47], which combines Inception modules [29] and deep residual learning [30].The paper describes these as two major CNN design advances since AlexNet’s debut in 2012.Source: §2.7 · p. 16
  • Its six segments run from a stem through Inception-ResNet A, Reduction-A, B, Reduction-B, and C blocks to global pooling and an output layer.Classification tasks use softmax, while FreeSolv regression uses a linear output layer.Source: §2.7 · p. 16
  • They retained the optimized module and reduction designs, tuning the number of inception blocks per segment and layer widths.Filter regularity was preserved using a reference layer in each block to define the convolutional filter counts of its other layers.Source: §2.7 · p. 16
  • They replaced the original seven-layer stem with one 4 × 4 convolution of stride 2; Chemception T1 was the shortest design, with nine further iterations tested.The paper reports the design details and performance of those iterations with the results.Source: §2.7 · p. 17

2.8 Chemception Training Protocol

Chemception was trained with five-fold cross-validation using a two-stage optimizer schedule, dataset-specific training volumes, early stopping, and task-specific evaluation metrics; reported performance summarizes five runs.

2.8 Chemception Training Protocol

  • Training used 50 epochs of RMSprop (learning rate 10^-3, ρ = 0.9, ε = 10^-8), then 50 epochs of momentum SGD starting at 10^-3 with exponential decay, using batch size 32.Source: §2.8 · p. 17
  • After class balancing and oversampling, Tox21 tasks used ~8,000 to ~12,000 compounds, HIV ~66,000, and FreeSolv ~580; FreeSolv received 10X more data per epoch.Source: §2.8 · pp. 17–18
  • To reduce overfitting across architectures with ~70,000 to ~2,300,000 trainable parameters, 25 epochs without validation-loss improvement triggered saving the best model for fine-tuning or final use.Source: §2.8 · p. 18
  • Tox21 and HIV were evaluated using AUC, while FreeSolv was evaluated using RMSE.Source: §2.8 · p. 18
  • Reported performance was the mean and standard deviation of each metric over five runs of the five-fold cross-validation protocol.Source: §2.8 · pp. 17–18

2.8 Data Augmentation on Input Images

Chemception used real-time image augmentation to address limited task data, while accounting for molecular images’ sparse, information-rich structure. The protocol avoided cropping and rotated images to help the model learn rotational invariance.

2.8 Data Augmentation on Input Images

  • Real-time augmentation with Keras ImageDataGenerator was used to bolster the limited number of examples available for each task.The section notes that common image operations include shearing, cropping, and rotation.Source: §2.8 · p. 18
  • Molecular images were populated mostly (>90%) by zeros, leaving usable data localized to a small fraction of each image.The authors use this sparsity to motivate caution about augmentation that might remove or distort the small regions carrying molecular information.Source: §2.8 · pp. 18–19
  • For a hydroxyl group and its local chemical environment, as little as ~10 pixels may suffice, or 0.156% of an 80 x 80 image.This illustrates why a sparse molecule image can still have high information density at individual pixels.Source: §2.8 · p. 19
  • Random cropping was not used because it can distort data fidelity and may remove part of a molecule.Source: §2.8 · p. 19
  • Each image was randomly rotated between 0 to 180 degrees before parsing, a procedure that facilitates learning rotational invariance.Source: §2.8 · p. 19

3. Results and Discussion

This section develops and explores Chemception’s architecture and evaluates its performance on toxicity, activity, and solvation tasks. It also examines architectural choices and optimization opportunities.

3.1 The Chemception Architecture

  • Chemception adapts an Inception-ResNet v2-based design, using Inception modules to combine spatial scales that may link atomic features to whole-molecule representations.The authors motivate this design as a way to learn chemical representations across levels, from atoms through functional groups and fragments to whole molecules.Source: §3.1 · p. 20

3.2 Chemception Model Exploration

  • The model exploration varied inception depth from one to three blocks and width from 16 to 64 filters, with a standardized two-stage training protocol.Training used RMSprop followed by momentum SGD with exponential learning-rate decay; data augmentation and early stopping were used to mitigate overfitting.Source: §3.2 · p. 22

3.3 Benchmark Database Selection and Controlling Performance Factors

  • Using MoleculeNet datasets, the study compared Chemception with reported results for ECFP-trained multilayer perceptrons, selecting Tox21 for architecture optimization because it contains 12 toxicity measurements for almost 10,000 compounds.Source: §3.3 · pp. 23–24

3.4 Model Optimization Results

  • Tox21 validation AUC ranged from 0.702 to 0.834, with a mean AUC of 0.760 across the dataset.The reported AUCs are for the individual toxicity measurements predicted by baseline Chemception T1.Source: §3.4 · pp. 24–25

3.5 Possible Further Optimizations

  • Further optimization could vary inception blocks and filters across spatial levels, refine inherited Inception-module and training choices, and tune SGD learning rates.The authors note that aggressive architecture and hyperparameter tuning to maximize the performance metric was outside the paper’s scope, which focused on Chemception’s plausibility as a machine-intelligence tool.Source: §3.5 · p. 27

3.6 Tox21 Results

  • 0.768/0.773 validation/test AUC for single-task Chemception compared with 0.777/0.799 for multi-task MLP DNN on Tox21, despite Chemception’s lack of engineered features or chemical knowledge.The authors describe Chemception as almost matching the multi-task model, which they expected to perform better through multi-task learning.Source: §3.6 · pp. 28–29, Figure 6 · p. 29

3.7 HIV and FreeSolv Results

  • For FreeSolv, baseline Chemception RMSE was 1.67 kcal/mol on validation and 1.84 kcal/mol on test, while the finalized model performed better.The supplied passage does not provide the finalized model’s error values.Source: §3.7 · p. 31
  • Chemception achieved validation and test RMSEs of 1.51 and 1.75 kcal/mol, respectively, outperforming the MLP’s test RMSE of ~2.0 kcal/mol with 90% of the dataset used for training.Source: §3.7 · p. 31
  • At ~1.75 kcal/mol, Chemception remained less accurate than the simulation-based method’s estimated RMSE of ~1.5 kcal/mol, but approached physics-based accuracy using ~600 compounds and minimal prior chemical knowledge.Source: §3.7 · p. 32
  • The authors suggest that a larger solvation free-energy dataset could improve Chemception enough to reach parity with physics-based models.Source: §3.7 · p. 32

4. Conclusion

Chemception predicts chemical properties from 2D molecular images without explicit chemistry knowledge or engineered features. Its performance matches engineered-feature neural networks overall, while its domain-agnostic approach may assist feature engineering and transfer to other problems.

4. Conclusion

  • Chemception predicts chemical properties from 2D molecular drawings without explicit chemistry knowledge or engineered descriptors or fingerprints.Source: §4 · p. 33
  • The general-purpose architecture covers toxicity, activity, and free-energy-of-solvation prediction using modest training databases of ~600 to ~40,000 compounds.Source: §4 · p. 33
  • Chemception matched engineered-feature MLP performance overall and outperformed counterparts on 2 of 3 properties, achieving HIV validation/test AUC 0.744/0.752 and solvation RMSE 1.51/1.74 kcal/mol.The solvation RMSE is close to the ~1.5 kcal/mol RMSE of physics-based simulation methods.Source: §4 · p. 33
  • For Tox21 toxicity prediction, Chemception's validation/test AUC of 0.768/0.773 trails the ECFP multi-task DNN's 0.777/0.799, possibly because of multi-task learning.Source: §4 · p. 33
  • The authors present deep networks as machine intelligence that can assist human-driven feature engineering, processing data faster and more consistently than humans.They anticipate that this may make neural networks valuable in computational chemistry research as chemical training data grow.Source: §4 · pp. 33–34
  • Because Chemception minimizes chemistry knowledge through a domain-agnostic approach, the authors anticipate transferability to problems with limited understanding and feature engineering.Source: §4 · p. 34

Related papers

Loading 1706.06689v1…