PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Machine-learned Density Functionals

Li Li, John C. Snyder, Isabelle M. Pelaschier, Jessica Huang, Uma-Naresh Niranjan, Paul Duncan, Matthias Rupp, Klaus-Robert Müller, Kieron Burke

arXiv:1404.1333v2physics.chem-phcs.LGphysics.comp-phstat.ML

TL;DR

The paper asks how machine learning can approximate the kinetic-energy functional accurately enough for self-consistent orbital-free DFT. It studies kernel ridge regression, cross-validation, constrained minimization, local PCA, and sparse density representations in a one-dimensional fermion model. Gaussian and Cauchy kernels perform best, accurate constrained densities and energies are obtained, and sparse grids do not degrade the method’s performance.

  • Problem

    Orbital-free DFT needs accurate kinetic-energy functionals and self-consistent densities, while machine-learned functionals require suitable data, kernels, and methods for functional derivatives.

  • Method

    The study applies kernel ridge regression to one-dimensional non-interacting fermion densities, comparing kernels and cross-validation while using constrained Euler–Lagrange minimization, local-PCA projected gradient descent, and sparse grids.

  • Results

    Gaussian and Cauchy kernels give the best performance; with a Gaussian kernel for N = 1, chemical accuracy is reached at NT = 60, and performance improves systematically with more training densities.

  • Takeaways & Limitations

    The results show that kernel choice matters substantially, while accurate kinetic energies and self-consistent densities can be obtained for the studied one-dimensional model, including with sparse density representations.

Abstract

from arXiv · show

Kernel ridge regression is used to approximate the kinetic energy of non-interacting fermions in a one-dimensional box as a functional of their density. The properties of different kernels and methods of cross-validation are explored, and highly accurate energies are achieved. Accurate {\em constrained optimal densities} are found via a modified Euler-Lagrange constrained minimization of the total energy. A projected gradient descent algorithm is derived using local principal component analysis. Additionally, a sparse grid representation of the density can be used without degrading the performance of the methods. The implications for machine-learned density functional approximations are discussed.

I. INTRODUCTION

Density functional theory reduces the quantum problem by using electron density instead of the wavefunction, but practical kinetic-energy approximations remain important for accurate and scalable orbital-free calculations. This paper studies machine learning for that approximation in a one-dimensional model, including kernels, validation, constrained densities, and density representations.

  • Motivation: DFT uses the one-electron density as its basic variable, reducing the complexity of the quantum-mechanical problem compared with the wavefunction.The mapping is exact in principle, but kinetic and electron-interaction energies still require approximation in practice.
  • Motivation: O(N^3) Kohn–Sham scaling motivates accurate orbital-free kinetic-energy approximations with lower computational cost.Such approximations could support calculations on very large systems while retaining useful accuracy.
  • Machine learning approach: Kernel ridge regression previously produced highly accurate, systematically improvable kinetic energies, densities, and energies for non-interacting fermions in a one-dimensional box.The present work examines this machine-learned approximation in greater detail.
  • Machine learning approach: Machine-learned functionals require reference data, may violate standard exact conditions, and interpolate only within the systems represented by their training data.Their general functional form can nevertheless avoid some limitations of conventional local approximations.
  • Scope and contributions: The study evaluates kernels and cross-validation, develops constrained optimal-density methods with projected gradient descent, and tests sparse density grids.The model system uses non-interacting same-spin fermions in one dimension and Gaussian-dip external potentials.

B. Orbital-free DFT

Orbital-free DFT replaces the Kohn–Sham kinetic energy with a density functional and obtains the ground-state density through constrained minimization. The section distinguishes functional-driven and density-driven errors, highlighting why accurate self-consistent densities are difficult.

  • Orbital-free formulation: Orbital-free DFT approximates the Kohn–Sham kinetic energy as a functional of the density and minimizes total energy under particle-number normalization.The ground-state density satisfies a constrained Euler–Lagrange equation with an adjusted chemical potential.
  • Error decomposition: The functional-driven error evaluates an approximate kinetic energy on the exact density, whereas the density-driven error measures additional error from the approximate self-consistent density.The total energy error is decomposed as ΔE = ΔTF + ΔED.
  • Self-consistency challenge: Large functional-derivative errors can make self-consistent orbital-free results much worse than energies evaluated on exact Kohn–Sham densities.Traditional functional derivatives may also have nuclear singularities that complicate all-electron convergence.
  • Approximation limits: For one particle, the exact kinetic energy has the von Weizsäcker form, while the simplest local approximation performs poorly.The local approximation has a 217 kcal/mol test-set MAE and a 1903 kcal/mol self-consistent error in the cited model.

C. Data topology and representation

Electronic densities are continuous, high-dimensional objects that must be represented finitely for machine learning. The study restricts attention to a lower-dimensional density manifold generated by parameterized external potentials and uses grid-based inner products to represent it.

  • Density representation: Although the density is a continuous function, practical machine learning expands it in a finite basis, here using a real-space grid.The density space is formally infinite-dimensional, but calculations use NG basis functions or grid points.
  • Density representation: The L2 inner product makes the model representation-independent when the basis is converged.A truncated basis can still be used, but convergence of the representation determines whether this invariance applies.
  • Density manifold: The relevant training data occupy a low-dimensional density manifold generated by a parameterized class of external potentials.For the present Gaussian-dip potentials, the external potential has nine parameters, so the manifold dimension is at most nine.
  • Density manifold: The manifold restriction reduces the variety of densities that the machine-learning model must represent.Training densities are sampled from the manifold associated with a fixed particle number.

D. The kernel trick and feature space

Kernel methods transform nonlinear data into a feature space where linear regression can be applied. The kernel trick performs this implicitly through inner products, enabling nonlinear models without explicitly constructing the feature-space coordinates.

  • Feature space: Kernel-based learning maps high-dimensional nonlinear data into a feature space where its structure can become linear.This avoids directly fitting nonlinear forms in the original input space.
  • Feature-space example: For circular data, polar-coordinate transformation reveals a linear relation that can be fitted and then transformed back into a nonlinear fit in the original space.The figure contrasts the original Cartesian representation, feature-space linear fit, and recovered nonlinear model.
  • Kernel requirements: A valid kernel must satisfy Mercer’s condition and can generally be interpreted as a measure of similarity between data.This condition supports the feature-space representation used by kernel methods.
  • Kernel trick: The kernel trick expresses a linear algorithm through inner products between transformed feature vectors, so the feature map need not be computed explicitly.Mercer’s theorem guarantees the associated reproducing-kernel Hilbert-space construction under the stated conditions.

E. Kernel ridge regression

Kernel ridge regression represents the kinetic-energy functional through kernels evaluated on density inputs, with regularization controlling overfitting and hyperparameters selected by cross-validation. Density distances, feature-space centering, and kernel length scales determine how flexibly the model captures the kinetic-energy functional.

  • Kernel ridge regression: Kernel ridge regression approximates the kinetic energy as a weighted sum of kernels evaluated between a density and training densities.The weights are obtained by minimizing a quadratic cost with regularization.
  • Regularization and hyperparameters: The regularization strength λ penalizes large weights to prevent overfitting, while kernel parameters such as σ are selected through cross-validation.
  • Density representation: The Gaussian kernel uses density distances computed on the uniform grid, with the discrete L2 approximation becoming exact as ∆x →0.The dataset’s pairwise density distances have a maximum of 0.9 for 2000 densities.
  • Feature-space preprocessing: Centering densities in feature space removes the need for an explicit bias term in the regression model.The formulation assumes centered data, although Gaussian kernels on bounded domains do not require centering.
  • Kernel scale: Very small σ yields nearly nonoverlapping kernel functions, whereas very large σ lacks flexibility to fit nonlinearities in the data.The intermediate region therefore provides the useful balance between locality and flexibility.

III. MODEL SELECTION

Model selection compares standard kernels and their hyperparameters for approximating the kinetic-energy functional. Radial basis function kernels are broadly applicable, while kernel behavior depends on the data structure and regularization properties.

  • Model selection: Model selection chooses both a kernel and its corresponding hyperparameters, including λ and kernel-specific parameters such as σ.
  • Kernel families: Gaussian, Cauchy, and Laplacian kernels are radial basis function kernels that generally behave similarly across broad classes of problems.
  • Kernel families: Kernel choice can reflect the data characteristics, while other kernels may be suited to specific data structures or regularization properties.
  • Kernel performance: The Gaussian and Cauchy kernels achieve errors below 1 kcal/mol in the middle λ–σ region, whereas the Laplacian kernel performs poorly in comparison.The authors attribute the Laplacian result likely to its cusp, which cannot fit the smooth kinetic-energy functional.
  • Dataset partitions: The training, validation, and test sets are subsets of the full dataset used for model construction, hyperparameter optimization, and final assessment.

B. Optimization of hyperparameters

Hyperparameters are selected by estimating out-of-sample error without using the test set, using repeated validation schemes to reduce dependence on random partitions. Across methods, performance is similar with 120 training densities, while differences emerge with more limited data.

  • Cross-validation principle: Cross-validation estimates generalization error without examining the test set, preventing overoptimistic performance estimates from overfitting.
  • Cross-validation principle: The available data are divided into training, validation, and test sets, with hyperparameters optimized on validation error and final performance assessed only on the test set.
  • Validation schemes: Leave-one-out cross-validation is the k-fold special case with k = N_T, so each validation bin contains one sample.
  • Cross-validation results: 120 training densities produce similar test-set MAE across cross-validation schemes despite large variations in σ and λ.Differences in MAE occur primarily for more limited data.
  • Cross-validation results: Randomized 10-fold cross-validation selects hyperparameters near the true minimum because the test-set MAE minimum lies in a relatively flat basin.The procedure is repeated 40 times for the reported optimization.

IV. RESULTS AND DISCUSSION

The Gaussian kernel performs best among the tested kernels, while sparse density grids preserve model performance down to about 10 grid points. This reflects the low dimensionality of the density data, although integral accuracy and system size constrain the usable basis.

  • Kernel performance: The Gaussian and Cauchy kernels give the best performance, with the Gaussian kernel reaching chemical accuracy at NT = 60 for N = 1.Chemical accuracy means a mean absolute error below 1 kcal/mol.
  • Kernel performance: The Gaussian-kernel performance improves systematically as the number of training densities increases, whereas the wave kernel does not improve with additional training data.The wave kernel is therefore not flexible enough to fit the kinetic-energy functional.
  • Sparse-grid representation: A sparse density grid leaves model performance unaffected until NG is reduced to about 10 grid points.The model is cross-validated after each reduction, while its hyperparameters change only slightly.
  • Sparse-grid representation: The minimum useful grid is comparable to the density manifold’s dimensionality because the model only needs enough representation degrees of freedom to distinguish densities.For this problem, the data dimensionality is about 9.
  • Sparse-grid representation: A grid that is sparse relative to the reference data can reduce computational cost, but it must still be fine enough to evaluate the required integral accurately.The dimensionality of the data may increase substantially for larger systems such as real molecules.

C. Challenge of finding density

The learned functional derivative is inaccurate away from the density manifold, so unconstrained minimization can leave the region where the model interpolates reliably. Local PCA reconstructs a tangent space and supports constrained optimization on the learned manifold.

  • Derivative challenge: The MLA functional derivative contains substantial noise and cannot reproduce the exact derivative in directions absent from the training data.The model agrees with the exact derivative only after projection onto the tangent of the density manifold.
  • Derivative challenge: Standard gradient descent can rapidly exit the interpolation region because the MLA has information mainly along the density manifold, not in orthogonal directions.The resulting errors in the functional derivative produce large errors in the density.
  • Constrained minimization: Constrained minimization reduces the search domain from the full density space to the density manifold defined implicitly by g[n] = 0.The constrained optimal density satisfies the same ground-state solution condition while remaining on the manifold.
  • Manifold reconstruction: Local PCA approximates the density manifold as locally linear using nearby training densities and a distance-based weighting function.The smooth weighting function emphasizes densities near the target density.
  • Manifold reconstruction: The first d principal components form a basis for the tangent space, and projection onto this basis supplies the local manifold approximation.Here d is the dimensionality of the density manifold; the PCA approximation defines a squared-distance constraint to the tangent plane.

E. Projected gradient descent algorithm

The projected gradient descent algorithm alternates tangent-space descent with a correction that restores the density-manifold constraint. Its step size controls the trade-off between convergence speed and stability.

  • Algorithm: The algorithm begins from a density on the manifold, evaluates the MLA functional derivative, and computes a local PCA projection operator.A training density can serve as the initial guess.
  • Algorithm: The functional derivative is projected onto the tangent space before taking a step toward lower energy.This restricts the descent direction to the locally reconstructed density manifold.
  • Step-size control: The step size satisfies 0 < ϵ ≤ 1; reducing ϵ improves stability when convergence is unstable, at the cost of speed.The algorithm therefore uses the step size to balance stability and convergence rate.
  • Constraint enforcement: After each descent step, the weighted local mean is subtracted to ensure that the manifold constraint remains satisfied.The iteration continues until the energy difference meets the specified tolerance or the maximum iteration count is reached.

F. Errors on constrained optimal densities

Constrained optimal densities substantially improve the practical evaluation of the learned kinetic-energy functional, while projected-gradient performance depends on PCA choices and manifold sampling.

  • 1 kcal/mol KE accuracy is achieved with 150 training samples for N = 1 on constrained optimal densities.Errors on constrained optimal densities decrease at the same rate with additional training data.
  • Errors on constrained optimal densities are roughly a factor of 10 worse than errors evaluated on exact densities.
  • d = 4 or 5 gives the optimal reconstruction of the density-manifold tangent space when m = 20.Increasing d further reintroduces removed noise into the projection.
  • For d = 7, many constrained searches do not converge because noise is reintroduced into the projection.
  • Table IV evaluates kinetic-energy errors for 100 test densities at NT = 40 and NT = 100 while varying PCA parameters m and d.The table also reports the percentage of converged optimal densities.
  • The method’s effectiveness depends on sampling relevant directions on the potential manifold.If those directions are absent from the training data, the model cannot learn the corresponding functional derivatives, and PCA removes them.

V. CONCLUSION

The paper demonstrates machine-learned kinetic-energy functionals in a simple one-dimensional model, finding strong performance from selected kernels, flexible validation, and sparse density representations.

  • The study uses a simple model as a proof of principle for applying standard machine-learning methods to density functional theory.
  • Accurate self-consistent densities and energies are obtained using the learned kinetic-energy functional in modified orbital-free DFT.
  • The Gaussian kernel gives the lowest errors, while the Cauchy kernel achieves similar performance.Some tested kernels perform strikingly badly, underscoring the importance of kernel choice.
  • All tested cross-validation schemes produce similar hyperparameter predictions with low generalization error on the test set.
  • A sparse density representation can be used without losing accuracy, potentially speeding calculations further.The kernel construction is basis-set independent when a complete basis is used.
Loading 1404.1333v2…