How to validate machine-learned interatomic potentials

I. INTRODUCTION

• The review covers numerical and physically guided validation for ML potentials, focusing on atomistic simulations and inorganic materials.It aims to complement broader ML-validation guidance and tutorials emphasizing molecular force fields.Source: §I · p. 1

II. WHAT ARE ML POTENTIALS?

• ML potentials learn a model of a potential-energy surface from quantum-mechanical reference energies and forces.Once developed and properly validated, they enable large-scale simulations at a similar quality level to quantum-mechanical methods with a small fraction of the computational cost.Source: §II · p. 1, §I · p. 1
• Reference databases must cover relevant chemical environments, but inadequate training coverage can lead to poor extrapolation and incorrect behavior.The quality of labels also constrains the model: an ML potential reproduces at best the level of its training data.Source: §II · p. 2
• Descriptors encode local atomic environments and should be invariant to translations, rotations, and atom permutations while remaining as complete and efficient as possible.Pairwise descriptors may suit bulk metals but not covalent systems.Source: §II · pp. 2–3
• Descriptor hyperparameters and regression choices affect potential quality, with SOAP density broadening trading structure-search robustness against prediction accuracy.Larger broadening can support robust structure searches, while smaller values are needed for highly accurate predictions; fitting frameworks also differ in flexibility, data needs, and hyperparameters.Source: §II · p. 3

III. WHAT MAKES A POTENTIAL “VALID”?

• A potential must predict qualitatively appropriate behavior and quantitatively reproduce measurable properties.Qualitative behavior is judged against experiment, high-quality quantum-mechanical predictions, or both.Source: §III · pp. 3–4
• Because potential quality is often very difficult to determine, and numerical performance is difficult to connect to physical meaning, validation tests become important; false maxima and minima affect predictions.Source: §III · pp. 3–4
• In silicon compression, the reference behavior is pressure-induced crystallization to simple-hexagonal silicon, unlike the empirical-potential and candidate-ML examples.The empirical potential does not access the very-high-density amorphous phase; the candidate ML potential instead becomes stuck in a disordered structure with coordination numbers of at least 10, and is deemed non-valid.Source: §III · p. 4

IV. NUMERICAL ERRORS

• Numerical error metrics are used to quantify a new model’s performance relative to the existing state of the art.The section reviews relevant techniques and error definitions for applying numerical validation to interatomic potentials.Source: §IV · p. 4
• The section surveys commonly used error definitions as tools for numerical validation.These techniques are considered in the context of interatomic-potential models.Source: §IV · p. 4
• Numerical-validation techniques are discussed specifically for their application to interatomic-potential models.The review first considers techniques used in machine learning more broadly.Source: §IV · p. 4

A. Error measures

• RMSE is at least as large as MAE and is skewed upward by higher errors.Both metrics attempt to measure average error, but respond differently to the error distribution.Source: A. Error measures · p. 4
• For GAP-17 predictions of total energies against ground-truth DFT total energies, normalized errors have a system-size-independent distribution when structures share type and energy distributions across all N.Source: A. Error measures · p. 5
• A robust statistic such as the median can rank a potential as better even when it performs poorly on a physically relevant part of the potential-energy surface.In the example, model B performs extremely well on 90% of points but poorly on the remaining 10%, while model A performs reasonably across the relevant configurational space.Source: A. Error measures · pp. 4–5
• Single-value metrics summarize residual errors but lose information about their distribution; violin plots show that distribution more fully than box plots.In the example, violin plots and MAE and RMSE values both indicate unfavorable errors for model B relative to A.Source: Figure 3 · p. 4, A. Error measures · p. 5

B. Scaling with system size

• Total-cell energy errors increase with system size, whereas per-atom energy errors systematically decrease.The example evaluates GAP-17 predictions against DFT total energies for structures selected to keep type and energy distributions consistent across sizes.Source: B. Scaling with system size · p. 5
• Normalized-error distributions become independent of system size under the statistically justified normalization described by the authors.The recommended comparison uses structures of the same size when possible; the normalized values differ from conventional errors in eV per atom.Source: B. Scaling with system size · p. 5
• For per-cell properties such as total energy, the authors recommend comparing datasets containing structures of the same size.If size matching is not possible, they recommend the statistically justified normalization, while noting it is not directly comparable to eV-per-atom errors.Source: B. Scaling with system size · p. 5
• Numerical errors quoted in isolation have limited use, while well-defined measures tracked across methods, databases, and hyperparameters inform systematic benchmarks.The authors emphasize careful evaluation rather than specific error values for one potential.Source: B. Scaling with system size · pp. 5–6

C. Cross-validation and external test sets

• Held-out data are needed to assess a model’s generalization, since many ML models can nearly perfectly fit their training data.K-fold cross-validation trains separate models while successively holding out each fold, then averages the error metric to reduce noise from the split choice.Source: C. Cross-validation and external test… · p. 6
• External test sets are separately generated and more time-consuming, and can be structurally related to training data or cover different data types.The figure contrasts k-fold internal testing with an independent external set, which typically derives from a different distribution.Source: C. Cross-validation and external test… · p. 6
• Random structure search offers a system-agnostic test-set approach that samples diverse configurations, including local minima on the potential-energy surface.Domain-specific sets can instead target structures such as defects, transition paths, polymorphs, distorted crystals, or high-temperature and high-pressure MD snapshots.Source: C. Cross-validation and external test… · pp. 6–7
• Test-set errors depend strongly on set composition, and nearby test points can yield optimistic estimates unless the set reflects intended applications.“External” describes how a test set is constructed, not necessarily its distance from training data; comparing several test sets can give a more comprehensive view.Source: C. Cross-validation and external test… · p. 7

V. PHYSICALLY-GUIDED VALIDATION

• Physically guided tests use domain knowledge to assess whether ML potentials behave correctly in application-relevant settings.The review presents these tests as distinct from validation techniques used in standard ML research.Source: §V · p. 7, A. Domain-specific error analysis · p. 7

A. Domain-specific error analysis

• In an MD melt–quench simulation, candidate 1—adding new structures to the existing GAP-18 database without much weight—reproduced original GAP-18 behavior, whereas candidate 2 failed entirely.26The candidates differed only in regularization hyperparameters for expected input-data error; candidate 2 used a setting tighter by a factor of 10.Source: A. Domain-specific error analysis · p. 8
• Force errors on a physically motivated test set predicted candidate 2’s failure, whereas energy errors alone did not reveal trouble.For liquid configurations, candidate 2 had a lower energy error (8.4 meV/atom) than candidate 1 (11.5 meV/atom).Source: Figure 6 · p. 7, A. Domain-specific error analysis · p. 8

B. Domain-specific structural benchmarks

• SOAP similarity to diamond-type silicon decreases as amorphous silicon collapses, while similarity to simple-hexagonal silicon rises as crystallites form.The high sh-silicon similarity at the end positively identifies those crystallites against competing phases.Source: B. Domain-specific structural benchmarks · pp. 8–9
• Pressure-adjusting the reference crystal accounts for bond-length changes and avoids an apparent similarity decline without structural rearrangement.With a fixed ambient-pressure reference, similarity decreases linearly; with a pressure-adjusted reference, it changes very little up to about 10 GPa.Source: B. Domain-specific structural benchmarks · p. 9
• SOAP analysis detected unreasonable highly coordinated false minima for candidate 2, while candidate 1 passed and could be used for large-scale MD simulations with confidence.The candidates were MTP models fitted to large (candidate 1) and comparatively small (candidate 2) structure databases.Source: B. Domain-specific structural benchmarks · p. 9
• Large compression MD simulations starting from disordered structures can sample varied configurations, while structural databases make reference crystals available for SOAP comparisons.The authors note that large systems allow frequent sampling of configurations involving close atomic approaches.Source: B. Domain-specific structural benchmarks · p. 9

C. Random search and exploration

• Relaxing random structures into local minima provides a stringent test of an interatomic potential.The RSS approach generates and relaxes large numbers of random structures subject to simple constraints, such as minimum atomic separation.Source: C. Random search and exploration · p. 9
• Both ML potentials found low-volume minima resembling high-pressure simple-hexagonal phases, but Stillinger–Weber did not; diamond-like minima numbered 112 for GAP-18, 135 for MTP, and 55 for SW.All three potentials found diamond-like minima, defined by SOAP similarity above 0.99 to the ideal crystal.Source: C. Random search and exploration · pp. 9–10
• RSS structures and minimization trajectories can provide an unbiased out-of-sample test set, though their unusual configurations may yield relatively high numerical errors that require context.The structures can be labeled with the ground-truth method and used to compare candidate potentials.Source: C. Random search and exploration · p. 10

D. Experimental data

• Experimental comparison is described as perhaps the most important and relevant validation for a potential, because a simulation must agree with and explain observations.The authors note that there is wide scope for how this comparison can be made.Source: D. Experimental data · p. 10
• For crystalline materials, relaxed lattice parameters can be compared with diffraction data, but reference choice matters: low-temperature and high-resolution measurements are preferable in the stated contexts.Powder diffraction is preferable for lattice parameters, while single-crystal diffraction yields the most accurate atomic positions, all else being equal.Source: D. Experimental data · p. 10
• Calorimetry compares phase enthalpy differences, and validation should separate ground-truth computation error from ML-fit error.For SiO2, SCAN energetics compatible with experiment were reported to be inherited by the ML model.Source: D. Experimental data · p. 10
• Experimental vibrational and spectroscopic comparisons vary in accessibility and computational difficulty, from phonon dispersions to Raman, infrared, and EXAFS measurements.Full vibrational density of states requires inelastic neutron scattering, while infrared and Raman spectra require computation of absorption intensities associated with phonon modes.Source: D. Experimental data · pp. 10–11
• Liquid and amorphous structures are difficult to characterize experimentally, making accurate, reproducible benchmarks particularly challenging; large-scale simulations also require convergence checks for structural metrics.Structural comparisons often validate the structural model itself rather than nuances of the potential, although the potential influences the generated model.Source: D. Experimental data · p. 11

Introducing an ML potential

• For a new ML potential, report internal cross-validation and external out-of-sample energy and force errors, defining how each metric was obtained.The recommendations also call for comparison with available experimental data and discussion of errors or uncertainty in the literature values.Source: §VI · p. 11

Using an existing ML potential

• For an existing ML potential, describe how applicable its reported error metrics are and, if possible, benchmark it against representative small-scale DFT simulations.Compare with experimental data wherever available.Source: §VI · p. 11

• The authors hope these recommendations increase confidence among computational practitioners and experimental colleagues who read the work.Source: §VI · p. 11
• They emphasize openly sharing training and benchmark data and testing workflows, especially as workflows become more complex.Source: §VI · p. 11

• As ML potentials spread beyond specialist developers, ensuring their validity and accuracy becomes more pressing.Source: §VII · p. 11
• The authors suggest openly available packaged tests runnable through accessible code, such as a Python script or automated workflow.They cite the elemental-silicon testing framework described in Ref. 65 as an example.Source: §VII · p. 12
• Physically guided benchmark errors can convey information that simple cross-validation cannot, provided representative structural snapshots are chosen and relabeled with the new potential’s reference method.The authors identify selection of suitable out-of-sample testing data as a key challenge and envision random configurations and shared benchmark simulations.Source: §VII · p. 12
• The authors hope carefully crafted and validated ML potentials will help make them everyday materials-modeling tools and accelerate scientific discovery.Source: §VII · p. 12

• The authors state that they have no conflicts to disclose.Source: Conflict of Interest · p. 12