Machine learning of molecular properties: locality and active learning

Konstantin Gubaev; Evgeny V. Podryabinkin; Alexander V. Shapeev

Source-linked AI summary

Machine learning of molecular properties: locality and active learning

Konstantin Gubaev, Evgeny V. Podryabinkin, Alexander V. Shapeev

arXiv:1709.07082v2Sep 18, 2017physics.chem-phcond-mat.mtrl-sci

TL;DR

The paper addresses the need for accurate, computationally efficient molecular-property prediction with smaller training sets and fewer errors on outliers. It combines a local model of interatomic interactions with active selection of diverse training molecules, and compares the approach with state-of-the-art methods. The model reaches chemical accuracy with about 3200 training molecules, while active learning reduces maximal error for outliers.

Problem
Existing machine-learning models require large training datasets to reach chemical accuracy and can produce large errors for out-of-sample molecular outliers.
Method
The paper combines local atomic-environment contributions for molecular-property prediction with active learning that selects diverse molecules for training.
Results
The model reaches chemical accuracy with a training set of about 3200 molecules, while active learning significantly reduces maximal error for outliers.
Takeaways & Limitations
Locality and active selection enable accurate molecular-property prediction with fewer training samples and better treatment of out-of-sample molecules.

Abstract

from arXiv · show

In recent years the machine learning techniques have shown a great potential in various problems from a multitude of disciplines, including materials design and drug discovery. The high computational speed on the one hand and the accuracy comparable to that of DFT on another hand make machine learning algorithms efficient for high-throughput screening through chemical and configurational space. However, the machine learning algorithms available in the literature require large training datasets to reach the chemical accuracy and also show large errors for the so-called outliers - the out-of-sample molecules, not well-represented in the training set. In the present paper we propose a new machine learning algorithm for predicting molecular properties that addresses these two issues: it is based on a local model of interatomic interactions providing high accuracy when trained on relatively small training sets and an active learning algorithm of optimally choosing the training set that significantly reduces the errors for the outliers. We compare our model to the other state-of-the-art algorithms from the literature on the widely used benchmark tests.

I. INTRODUCTION

The paper introduces a local machine-learning model and active-learning strategy to reduce training-data requirements and errors on molecular outliers. It represents molecular properties through local atomic environments and selects diverse training molecules to improve coverage.

Quantum-mechanical methods are accurate but too computationally expensive for high-throughput screening of large molecular collections.
Existing machine-learning models can approach chemical accuracy but often require large training datasets and perform poorly on out-of-sample molecular outliers.
The proposed model relates molecular properties to contributions from local atomic environments, allowing predictions for molecules absent from the training set.
The active-learning algorithm adds molecules with the most different geometries and compositions, reducing errors for outliers by improving training-set coverage.
The paper evaluates the algorithm against existing methods on QM9 and QM7 benchmark datasets.

Locality

The model imposes locality by decomposing molecular properties into atomic-neighborhood contributions and represents each neighborhood with rotation- and permutation-invariant moment tensor descriptors. Multi-component extensions encode chemical identities in radial functions, while model complexity is controlled by descriptor level.

Locality: The property y is partitioned into contributions from individual atoms, with each contribution determined by the atom’s neighborhood within a cutoff radius Rcut.Each neighborhood contains relative position vectors and neighboring-atom types.
Moment tensor descriptors: Basis functions are formed by contracting moment tensors into rotation-invariant scalars, and the model fits their linear coefficients from molecular-property data.The basis functions are constructed from scalar contractions of tensors with different ranks.
Invariances and completeness: The moment tensor descriptors are rotation invariant, permutation invariant for chemically equivalent atoms, and complete for describing multi-component local neighborhoods.The multi-component completeness follows from allowing radial functions to depend arbitrarily on neighboring-atom types.
Moment tensor descriptors: The model represents atomic environments with moment tensor descriptors combining radial functions of distances and chemical types with tensorial angular factors.The descriptors use outer products of relative position vectors to encode neighborhood geometry.
Model complexity: MTMd includes all basis functions with level levBα ≤ d, so increasing d produces models with higher descriptor complexity.The level is defined from the ranks and tensor orders used to construct the basis functions.
Chemical composition: For chemically different atoms, radial functions depend on the central atom type and neighboring-atom types, extending the single-component moment tensor model.The type dependence is encoded in the radial-function coefficients.

Intensive quantities

The model distinguishes extensive from intensive molecular properties and modifies its partition relation for intensive quantities, whose scale does not grow with molecular size.

Intensive quantities: Extensive quantities scale with atom count, whereas intensive quantities remain of the same order as the number of atoms increases.Atomization energy is given as an extensive example, while HOMO/LUMO level is an intensive example.
Intensive quantities: For intensive quantities, the model modifies the locality partition relation using the number of atoms in the molecule.The atom count is denoted by #(x).

A nonlocal model

The nonlocal nlMTM extends locality by combining two local models through a nonlinear dependence on their outputs. This architecture is motivated by local orbital structure together with non-independent electron occupation.

A nonlocal model: The nonlocal construction uses two local models, V1 and V2, each with its own parameter set.The resulting model is called nlMTM.
A nonlocal model: The nlMTM combines the local-model outputs through additional nonlinear terms with parameters p1, …, p9.These parameters form an additional computational layer beyond the local model.
Motivation: The architecture assumes that molecular orbitals depend largely on local environments but that electron occupation introduces nonlinear dependence among local features.The paper considers two local features, V1 and V2.

III. ACTIVE LEARNING

Active learning addresses the need for training sets that are both computationally limited and representative of molecular diversity. The proposed selection should use only unlabeled structures while targeting samples where the model extrapolates.

Active learning: Training-set accuracy depends on selecting a limited set of molecules that represents the full molecular variety and avoids extrapolation.The available training-set size is constrained by the cost of experiments or ab initio calculations.
Active learning: Active learning selects molecules using unlabeled structures before their property labels are computed.This differs from ideal fixed-size selection methods that would require labels for all available data.
Related approaches: A D-optimality approach for linear interatomic potentials selects configurations maximizing the determinant of the model-parameter equation matrix.The method detects extrapolative samples, and training on them prevents extrapolation.

A. Generalized D-optimality criterion

The model generalizes D-optimality to nonlinear parameters and uses molecular novelty to select training examples that are informative and insufficiently represented.

Generalization to nonlinear models: The nonlinear model is handled by linearizing each loss-function term around parameters found by training.This provides the basis for extending the D-optimality criterion beyond linear models.
D-optimal selection: Training examples are selected by finding linearly independent rows in the Jacobian matrix and maximizing the volume of a square submatrix with maxvol.Each Jacobian row corresponds to a molecule in the training set.
Novelty criterion: Active learning identifies molecules whose properties may be unreliable because they differ substantially from all molecules already in the training set.Such molecules should be labeled and added to the training set.
Novelty criterion: The novelty grade γ(x*) is the maximum factor by which the determinant volume can grow when molecule x* is added to the training set.A molecule is added when γ(x*) ≥ γtrsh, where the threshold prevents selection of insufficiently novel examples.
Local atomic environments: The active-learning procedure partitions molecular contributions over individual atomic environments to avoid selecting molecules assembled from already known neighborhoods.Such molecules can produce vectors that are linearly dependent on vectors from smaller molecules.

B. Active learning scheme

The active-learning loop starts from a random training set, repeatedly retrains the model, and adds sufficiently novel molecules while controlling dataset growth.

Initialization: The procedure begins with a random initial training set and trains the model on the current set.These are the initialization and first fitting steps of the iterative loop.
Iterative expansion: At each iteration, molecules with γ ≥ γtrsh are selected, with additions limited to a training-set increase of 10% or less.The algorithm can add all qualifying molecules or those with maximal γ within the growth limit.
Stopping: The loop repeats training and selection until the current model is considered satisfactory.The stopping condition is left to the model-evaluation discussion.
Stopping: For the reported performance study, iterations stopped when the training set reached 6000 molecules.The authors note that practical stopping could instead track accuracy improvement across several iterations because gains may fluctuate between iterations.

IV. RESULTS

The experiments assess random-subset accuracy, active-learning improvement, and the training-set size needed to reach chemical accuracy, using specified model levels and optimization settings.

Evaluation goals: The evaluation asks how accurately the model performs on random subsets, how active learning improves accuracy, and how many samples reach 1 kcal/mol chemical accuracy.Errors are measured on a validation set consisting of the full database excluding the training set.

Fitting enthalpy on QM9

On QM9 enthalpy prediction, learning curves compare model capacity and training-set size, while the reported benchmarks show chemical accuracy with only a few thousand molecules.

Dataset and target: QM9 contains 130831 molecules made from C, H, O, N, and F, with up to nine heavy atoms; the fitted property is enthalpy at 300 K.The dataset excludes 3054 molecules that failed a consistency test.
Learning curves: Models with fewer parameters perform better on small random training sets, whereas models with more parameters outperform them as training size increases.The learning curves report mean absolute error averaged over three independent random training-set choices.
Chemical accuracy: At 3500 molecules, MTM24 reaches chemical accuracy, nearly matching aSLATM’s approximately 3200-molecule requirement.A separate MTM24 evaluation with 3000-molecule random samples produced a validation MAE of 1.006 ± 0.0316 kcal/mol.
Benchmark comparison: For training sets of at most 10k molecules, MTM has a better learning curve than the existing state-of-the-art methods, except for nearly equal results from Ref. 17.Deep neural-network approaches show better accuracy above 50k training samples.

Active learning

Active learning selects diverse molecules to reduce outlier errors, with benefits depending on whether unusual molecules can enter training. It also tends to select smaller-than-average molecules, while random selection can achieve lower RMS error in some regimes.

Error reduction: From 4000 training samples onward, active and random selection have similar MAE, but active selection has lower RMS and maximal errors.Four thousand samples equals twice the approximately 2000 model parameters used by MTM24.
Selection strategy: Active learning selects molecules spanning a large volume of configurational space, targeting molecules most different from those already in the training set.This strategy is intended to represent unusual molecules and reduce extrapolation errors.
Error reduction: Active learning reduces maximal error compared with random selection, especially when any molecule can be added to training.The maximal error is lower in both active scenarios than random sampling, and lowest when unusual molecules can also be selected for training.
Selection strategy: With 10k training molecules, active learning makes enthalpy errors small across samples, whereas random selection leaves large-error outliers.Figure 8 focuses on a small enthalpy region to show the error scale.
Selection strategy: At each iteration, active learning tends to select molecules with fewer atoms than the QM9 database average.The authors suggest that small molecules may contain atomic neighborhoods representative of many larger molecules.

Fitting the QM7 database

On QM7, the local model matches or exceeds existing accuracy for enthalpy and polarizability but initially performs worse for HOMO and LUMO levels. A nonlocal modification restores state-of-the-art accuracy for those electronic levels.

Local model: On QM7, enthalpy and polarizability achieve accuracy equal to or better than existing methods, while HOMO and LUMO errors are 50% larger than state of the art.The benchmark contains 7.2k small organic molecules, with 5k used for training and 2200 for validation.
Nonlocal modification: The nonlocal nlMTM modification improves HOMO and LUMO prediction to state-of-the-art accuracy.Because nlMTM behaves like a three-layer model and can overfit, the authors use early stopping during training.
Model: The moment tensor model predicts molecular properties from contributions of local atomic environments represented by moment tensor descriptors.The model’s descriptors are described as complete despite being essentially two-body, with angular dependence supplied by their tensorial structure.
Active learning: The active learning algorithm reduces maximal error by adding molecules most different from those already in the training set.This complements the QM7 fitting results by targeting outlier molecules rather than only average prediction accuracy.