PaperlayerView exact revision on arXiv

Source-linked AI summary

FCHL revisited: faster and more accurate quantum machine learning

Anders S. Christensen, Lars A. Bratholm, Felix A. Faber, O. Anatole von Lilienfeld

arXiv:1909.01946v2physics.chem-ph

TL;DR

FCHL18-based models could be computationally burdensome, limiting their applicability and making hyperparameter optimization unfeasible. This work discretizes and optimizes the representation, then reports state-of-the-art accuracy across energy and force tasks with reduced computational costs.

  • Problem

    FCHL18 models performed well, but computational performance hindered some applications and proper hyperparameter optimization was computationally unfeasible.

    Source: §I · p. 1

  • Method

    The authors discretize the FCHL18-based representation and use rigorous Monte Carlo optimization to select transferable hyperparameters for kernel-based models.

    Source: §I · p. 1

  • Results

    FCHL19 models show state-of-the-art accuracy across energy and force tasks; QM7b/QM9 training times fall 10 to 20 times versus FCHL18, and Water40 binding-energy error is below 0.1 kcal/mol/molecule.

    Source: §V · p. 13

  • Takeaways & Limitations

    The authors present FCHL19 as a practical, transferable representation intended to support routine model training and molecular dynamics simulations across chemical space.

    Source: §V · p. 13

    MD17 energy-prediction accuracy seems limited by noise in the dataset.

    Source: §V · p. 13

Abstract

from arXiv · show

We introduce the FCHL19 representation for atomic environments in molecules or condensed-phase systems. Machine learning models based on FCHL19 are able to yield predictions of atomic forces and energies of query compounds with chemical accuracy on the scale of milliseconds. FCHL19 is a revision of our previous work [Faber et al. 2018] where the representation is discretized and the individual features are rigorously optimized using Monte Carlo optimization. Combined with a Gaussian kernel function that incorporates elemental screening, chemical accuracy is reached for energy learning on the QM7b and QM9 datasets after training for minutes and hours, respectively. The model also shows good performance for non-bonded interactions in the condensed phase for a set of water clusters with an MAE binding energy error of less than 0.1 kcal/mol/molecule after training on 3,200 samples. For force learning on the MD17 dataset, our optimized model similarly displays state-of-the-art accuracy with a regressor based on Gaussian process regression. When the revised FCHL19 representation is combined with the operator quantum machine learning regressor, forces and energies can be predicted in only a few milliseconds per atom. The model presented herein is fast and lightweight enough for use in general chemistry problems as well as molecular dynamics simulations.

I. INTRODUCTION

The paper addresses FCHL18’s computational limitations by introducing a discretized, optimized representation and evaluating its accuracy and timing. FCHL19 describes atomic environments with two- and three-body terms, with an empirically selected log-normal radial basis.

I. INTRODUCTION

  • FCHL19 discretizes FCHL18 for fast L2-distance evaluation, optimizes transferable hyperparameters with Monte Carlo, and benchmarks predictive accuracy and timing.The earlier FCHL18 model performed well, but computational performance hindered some applications and proper hyperparameter optimization was computationally unfeasible.Source: §I · p. 1

II. THEORY

  • The theory section introduces the representation and kernel-based learning methods, while leaving feed-forward neural-network use outside the paper’s focus.The authors note that the representation could in principle also be used with feed-forward neural networks.Source: §II · p. 1

A. Representation

  • FCHL19 is an atom-centered, discretized FCHL18-based representation with two- and three-body terms; proper parameter optimization yields compact size without loss in predictive accuracy.The representation encodes radial distributions and mean distances and angles involving neighboring atom pairs; it has no explicit one-body term and omits terms above three-body for performance reasons.Source: A. Representation · p. 2

1. Two-body function

  • The two-body term uses log-normal radial basis functions on an equidistant grid, an empirically favored choice over the Gaussian functions used previously.For a central atom, radial basis functions are constructed for each unique element type; their parameters depend on distance and hyperparameters.Source: §1 · p. 2
  • For each atomic environment, the two-body term includes radial distributions for neighboring atoms by element type, so its size scales linearly with the number of possible elements.Figure 1 displays four unique radial basis-function types for water, with the H-atom spectra above the oxygen-atom spectra and arrows marking the distances used.Source: Figure 1 · p. 2, §1 · p. 3

2. Three-body function

The three-body function represents local triplets through their distances, angle, and neighbor element types. Its angular component uses a Fourier-style expansion, retaining only the first cosine and sine terms in this work.

2. Three-body function

  • The three-body function encodes distances from an atom to neighboring pairs, the angle between the triplet, and the neighbors’ element types.Its product includes radial and angular components, with a scaling function based on the Axilrod-Teller-Muto term with modified exponents.Source: §2 · p. 3
  • The angular term is similar to a previously introduced Fourier-series expansion, and this work uses only the n = 1 cosine and sine terms.The angular Gaussian width is controlled by ζ; with sufficiently large η3, the angular spectrum can often be nearly recovered using only the first Fourier terms.Source: §2 · p. 3
  • The three-body functions scale with the number of possible elements and comprise a much larger part of the representation than the two-body part.The radial component is similar to the radial part used in ANI-1 neural-network ACSFs.Source: §2 · p. 3

B. Machine learning

This section reviews kernel-based regressors for chemical properties, then describes KRR energy prediction as a sum of atomic contributions. KRR coefficients are fitted to training energies with regularization for numerical stability.

B. Machine learning

  • The section reviews KRR for energy learning and OQML, GPR, and GDML for learning forces and energies using kernel-based methods.Although the representation could also be used with feed-forward neural networks, the paper focuses on kernel-based methods.Source: B. Machine learning · p. 3

1. Kernel Ridge Regression (KRR)

  • KRR predicts a query compound’s energy as a sum of atomic energies represented by kernels centered on atoms in the training compounds.The compound-level kernel matrix sums pairwise kernels between atoms in two compounds.Source: §1 · pp. 3–4
  • KRR regression coefficients are fitted to training-set energies, with a diagonal regularization term used to stabilize kernel-matrix inversion.The regularization parameter λ is typically small.Source: §1 · p. 4
  • Earlier work reported that KRR with FCHL18 reached state-of-the-art accuracy across many molecule and material system classes.The authors describe systematically improving property predictions in that earlier work.Source: §1 · p. 4

2. Operator Quantum Machine Learning (OQML)

OQML trains on energies and forces together by applying a force operator to a kernel expanded over training atomic environments. It uses SVD to solve the regression problem stably, and its force operator guarantees conservative force fields.

2. Operator Quantum Machine Learning (OQML)

  • Including available forces in OQML training alongside energies improves energy and force prediction, and the force operator guarantees a conservative force field.The authors note that conservative forces are crucial for energy-conserving molecular dynamics sampling without heavy thermostat use.Source: §2 · pp. 4–5
  • OQML expands its kernel over training atomic environments, giving one regression coefficient per training atom rather than per compound.The kernel matrix is non-square, with columns for training atoms and rows for training or query molecules.Source: §2 · p. 4
  • The OQML training equation applies the force operator to the kernel so that energies and forces can be fitted simultaneously.The operator differentiates with respect to coordinates in the training or query compound, as indicated by the asterisk convention.Source: §2 · p. 4
  • The least-squares coefficient solution can be numerically unstable because it multiplies ill-conditioned kernel matrices, leading to large training and test errors.The paper instead solves the regression equation directly with SVD, optionally ignoring singular values below a threshold treated as a hyperparameter.Source: §2 · p. 4
  • OQML predicts forces by differentiating the energy expression using the same regression coefficients.Applying the force operator ensures the learned potential describes a conservative force field.Source: §2 · pp. 4–5

3. Gaussian Process Regression Including Derivatives

The GPR formulation fits energies and forces in a joint kernel matrix whose second-derivative block is its computationally heaviest part. Compared with OQML, GPR has less favorable computational and memory scaling, while sparsification can reduce its computational burden.

3. Gaussian Process Regression Including Derivatives

  • GPR’s joint energy-and-force kernel includes second-derivative terms, unlike OQML’s kernel, which contains terms only up to first order.The second-derivative block is the largest part of the GPR kernel matrix.Source: §3 · p. 5
  • The GPR kernel has rows for energy and force components, while its columns span training molecules and their gradient components, unlike OQML’s atom-based columns.The paper identifies the choice of regression basis as the main difference between the two kernel formulations.Source: §3 · p. 5
  • GPR’s heaviest kernel term scales quartically with molecule size, whereas OQML’s scales cubically; for 9–21-atom benchmarks, GPR kernel calculation is expected to be 50–200 times slower.Both methods scale quadratically with the number of training molecules, but GPR has a higher prefactor.Source: §3 · p. 5
  • GPR’s kernel is roughly three times larger in memory than OQML’s, while sparsification can remove the need for second derivatives and make its kernel much faster to compute.The paper notes that fitting more regression coefficients may slightly increase GPR regression accuracy.Source: §3 · p. 5

4. Gradient-Domain Machine Learning (GDML)

GDML learns from forces alone, using a second-derivative kernel, and uses a global representation for each molecule to reduce evaluation cost. The speed advantage comes with an accuracy trade-off, particularly for condensed-phase systems.

4. Gradient-Domain Machine Learning (GDML)

  • GDML trains on forces without energy data and uses only the second-derivative block of the GPR kernel.Its regression coefficients are obtained by minimizing a cost function that includes forces and the second-derivative kernel matrix.Source: §4 · pp. 5–6
  • GDML and sGDML use one global representation per molecule; GDML uses inverse interatomic distances, while sGDML accounts for atomic symmetry.The paper contrasts these with representations of individual atomic environments.Source: §4 · p. 6
  • Global representations reduce evaluation cost by about M^2, where M is the number of atoms per molecule, by replacing a double sum over atomic contributions with one global kernel derivative.The speedup from global representations is described as similar to the scaling-cost difference between OQML and GPR.Source: §4 · p. 6
  • Global representations generally have lower predictive accuracy, especially for condensed-phase systems, and may be unsuitable for size-extensive energy learning because kernel elements do not scale with system size.Local representations account for additive energies of separated copies of the same molecule through sums over atomic contributions.Source: §4 · p. 6

5. Kernel Function

The Gaussian kernel adds elemental screening so atomic environments are compared only for atoms of the same element. FCHL19 is among the lowest-MAE models on QM9 and reaches chemical accuracy on QM7b alongside FCHL18, SLATM, and SOAP.

5. Kernel Function

  • The Gaussian kernel's elemental screening compares atomic environments only for atoms of the same element, avoiding irrelevant pairs and reducing expensive kernel and derivative calculations.The Kronecker delta could be replaced by a function that incorporates learning across alchemical space.Source: §5 · p. 6

A. Energy Learning

  • FCHL19 uses universal hyperparameters fitted to non-equilibrium structures, which may slightly reduce accuracy on minimized QM9 and QM7b geometries; its hyperparameters were not optimized on QM9.Comparisons with literature models may also reflect different model and data selection methods.Source: A. Energy Learning · p. 6, §1 · p. 7

1. Results for QM9

  • At 20,000 training samples, FCHL18 and FCHL19 have MAEs of 0.30 and 0.47 kcal/mol; at 75,000, FCHL19 reaches 0.25 kcal/mol.Overall, FCHL18 has the lowest MAE, while SOAP, FCHL19, and aSLATM have virtually indistinguishable MAEs.Source: §1 · p. 7

2. Results for QM7b

  • FCHL19, FCHL18, SLATM, and SOAP reach chemical accuracy on QM7b with 800–1,600 training samples, while Coulomb matrix and BoB do not.At 1,000 samples, the four models are each within an MAE of ±0.3 kcal/mol of FCHL19.Source: §2 · p. 7

3. Results for QM7b-T and GDB13-T

Across the reported benchmarks, FCHL19 combines strong predictive accuracy with faster learning, including on water-cluster binding energies and MD17 forces and energies. Its performance varies by task: it does not reach chemical accuracy on the larger GDB13-T extrapolation with the available training data.

3. Results for QM7b-T and GDB13-T

  • On QM7b-T in-dataset prediction, FCHL19 and FCHL18 reach 1 kcal/mol accuracy with 400–800 training samples, while FCHL18 is slightly more accurate above 1,000 samples.MOB reaches 1 kcal/mol at about 200 samples and is consistently 2–3 times more accurate, but requires a Hartree-Fock calculation for each query.Source: §3 · p. 8, Figure 5 · p. 8
  • On GDB13-T extrapolation, neither FCHL19 nor FCHL18 reaches chemical accuracy with the available QM7b-T training data; at 1,000 samples, their MAEs are 2.7 and 2.2 kcal/mol, respectively.MOB reaches this error at around 100–200 samples, although a larger MOB training set is unavailable because large models are difficult to train.Source: §3 · p. 8, Figure 5 · p. 8

4. Results for Water40

  • At 1,000 Water40 training instances, FCHL18 has an MAE of 0.22 kcal/mol/molecule and FCHL19 has an MAE test error of 0.12 kcal/mol/molecule.FCHL19 reduces the data needed to reach a given accuracy by roughly 5 times versus FCHL18 and 10 times versus aSLATM; global representations including Coulomb matrix, BoB, and SLATM remain near 0.5 kcal/mol/molecule across training sizes.Source: §4 · p. 8, Figure 6 · p. 9

1. Results for MD17

  • For MD17, FCHL19/GPR needs about half as many samples as FCHL19/OQML to reach a given accuracy, and FCHL19/OQML needs about half as many as FCHL18/OQML.For ethanol, an MAE force error of 0.4 kcal/mol/Å is reached at roughly 200, 400, and 800 samples, respectively; FCHL19/GPR had the highest accuracy for salicylic acid, aspirin, malonaldehyde, and uracil.Source: §1 · p. 9
  • For toluene, naphthalene, and uracil, energy learning is very slow for all tested methods, with nearly flat curves around 0.1 kcal/mol error.The authors suggest this is an inherent dataset property, likely related to noise in the calculated DFT energies, for example from unconverged integration grids.Source: §1 · p. 9
  • For toluene and naphthalene, sGDML performs very well for force learning, while FCHL19 variants are within 0.1 kcal/mol energy error and 0.1 kcal/mol/Å force component error of the best method at 1,000 samples.The authors speculate that high ring symmetry may slow FCHL learning for these molecules and suggest system-specific hyperparameter reoptimization as a possible improvement.Source: §1 · p. 9

C. Timings

FCHL19 reduces kernel-generation time relative to earlier representations in the reported energy and force-learning tests. For force learning, OQML is faster than GPR, while GPR reaches a given accuracy with fewer training samples.

1. Timings for Energy Learning

  • For QM7 and QM7b, FCHL19 kernel evaluation takes 216s and 310s, respectively, compared with 3,164s and 3,286s for FCHL18.For aSLATM, the corresponding times are 4,955s and 7,727s; the timing comparison is for the three atomic representations on the reported datasets.Source: §1 · p. 10, Table I · p. 11
  • For QM9, FCHL19 takes 27 hours to calculate the kernel, versus 728 hours for aSLATM and 548 hours for FCHL18 on the same 24-core node.The authors attribute the speedup over aSLATM to the reduced representation size and element-wise kernel function.Source: §1 · pp. 10–11, Table I · p. 11

2. Timings for Force Learning

  • For FCHL19/OQML on MD17, training kernels take around 51 seconds for malonaldehyde to 527 seconds for aspirin, with force-prediction times of 5.7–25.3 milliseconds per atom.The reported prediction times exclude representation generation; FCHL19/OQML kernels are 5 to almost 20 times faster to compute than FCHL18/OQML kernels.Source: §2 · p. 11, Table II · p. 11, Figure 8 · p. 11
  • For equal accuracy, OQML models are about 10–50 times faster at prediction than GPR models, while GPR needs about half as much training data as OQML.GPR kernel calculation is 30–38 times slower for the smallest molecules and 192 times slower for aspirin.Source: §2 · p. 11, Table II · p. 11

IV. METHODOLOGY

The methodology benchmarks FCHL19 on molecular energies, non-equilibrium structures, water-cluster energies, and molecular forces. It also describes fitting transferable representation parameters using a small set of distorted molecules and a Monte Carlo optimization procedure.

A. Datasets

  • QM7b comprises 7,211 molecules with up to seven non-hydrogen atoms from CNOSCl, with PBE equilibrium geometries and 13 DFT-calculated properties.QM9 comprises 133,885 molecules with up to nine non-hydrogen atoms from CNOF; atomization energies at B3LYP equilibrium geometries are used for the learning curves, excluding 3,054 uncharacterized molecules.Source: §1 · p. 12, §2 · p. 12

3. QM7b-T and GDB13-T

  • QM7b-T and GDB13-T contain non-equilibrium geometries sampled from ab initio molecular dynamics at 350K, and provide MP2 correlation energies for testing extrapolation to larger molecules.Models are trained on QM7b-T and tested on GDB13-T, whose molecules contain 13 CNOSCl atoms before hydrogen saturation.Source: §3 · p. 12

4. Water40

  • Water40 contains 10,000 snapshots of a 40-molecule water cluster sampled at 300K, with dispersion-corrected PBEh-3c DFT single-point energies.Accurate treatment of non-bonded interactions is required to learn these energies.Source: §4 · p. 12

5. MD17

  • MD17 supplies DFT reference forces and energies for snapshots of small organic molecules; the study benchmarks ethanol, salicylic acid, aspirin, malonaldehyde, toluene, naphthalene, and uracil.Source: §5 · p. 12

B. Optimization of Representation Parameters

  • The authors use universal default parameter sets fitted a priori, rather than refitting the nine representation hyperparameters separately for every dataset.The fitting data comprise 576 distorted geometries of molecules with up to five CNO atoms, divided into 384 training and 192 test geometries; Monte Carlo greedy optimization minimizes a cost using test-set energies and forces.Source: B. Optimization of Representation… · p. 12
  • The optimization uses the same hyperparameter values across element types, fixes the angular Gaussian width ζ to π, and sets the distance cutoff to 8Å.Two parameter sets are fitted: one for energies and forces and one for energies alone, with the force term omitted for the latter.Source: B. Optimization of Representation… · pp. 12–13

C. Hyperparameter Selection

FCHL19 is evaluated with nested cross-validation and learning-curve analysis, then reported to achieve accurate energy and force predictions across molecular and water-cluster datasets with reduced training costs. The conclusion presents its transferable parameters and speed as steps toward practical models, while noting condensed-phase systems as future work.

C. Hyperparameter Selection

  • Nested five-fold cross-validation selects kernel width and regularization using inner-fold error, with the selection criterion depending on whether the dataset has energy labels alone or both force and energy labels.For force-and-energy datasets, the selected parameters minimize average L within the inner folds.Source: C. Hyperparameter Selection · p. 13

D. Learning Curves

  • Learning curves report average out-of-sample MAE across five outer folds and show log-log linear fits with bootstrap 95% confidence intervals.Source: D. Learning Curves · p. 13

E. Timings

  • Timing measurements used a 24-core Intel Xeon node with 128GB RAM and QML's default installation settings.Source: E. Timings · p. 13

V. CONCLUSION AND OUTLOOK

  • FCHL19 discretizes FCHL18 and uses two sets of universal parameters fitted to an initial training set, which showed high transferability.The conclusion describes the revised representation as delivering state-of-the-art accuracy at much reduced computational cost.Source: §V · p. 13
  • At the largest training sizes, QM7b and QM9 atomization-energy errors are below 0.5 and 0.25 kcal/mol, respectively; full QM9 training takes roughly one day versus three weeks previously.Training a model on 7K QM7 molecules takes a little over three minutes, and training times are reduced 10–20 times compared with FCHL18.Source: §V · p. 13
  • Water40 binding-energy error falls below 0.1 kcal/mol/molecule despite optimizing the representation solely for small molecules.Source: §V · p. 13
  • On MD17, FCHL19 with OQML or GPR reaches state-of-the-art force accuracy using 2–4 times less data than FCHL18 with OQML, with force predictions taking milliseconds per atom.MD17 energy prediction errors are low, though predictive accuracy appears limited by dataset noise.Source: §V · p. 13

APPENDIX A: OPTIMIZED REPRESENTATION PARAMETERS

The appendix presents optimized FCHL19 representation parameters and derives first- and second-coordinate derivatives of the kernel for implementation in QML.

APPENDIX A: OPTIMIZED REPRESENTATION PARAMETERS

  • Table III presents optimized FCHL19 representation parameters for energy-only and joint energy-and-force models.Source: Appendix A · p. 14

APPENDIX B: KERNEL DERIVATIVES

  • The kernel-derivative derivation begins with signed differences between representations and defines representation derivatives with respect to atomic coordinates.Source: Appendix B · p. 14
  • The appendix defines first- and second-derivative forms of the Gaussian kernel with respect to coordinates, and reports their analytical implementation in QML.Source: Appendix B · p. 14
Loading 1909.01946v2…