PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting materials properties without crystal structure: Deep representation learning from stoichiometry

Rhys E. A. Goodall, Alpha A. Lee

arXiv:1910.00617v4physics.comp-phcond-mat.mtrl-scics.LG

TL;DR

The paper addresses structure-agnostic materials-property prediction when crystal structures are unavailable and hand-engineered stoichiometric descriptors are limited. It represents compositions as dense weighted graphs and learns descriptors from data, improving sample efficiency while supporting uncertainty estimation and transferable representations.

  • Problem

    Structure-based inputs require characterised crystal structures, while structure-agnostic methods rely on hand-engineered stoichiometric descriptors; materials discovery is also data limited.

  • Method

    Roost represents each material’s composition as a dense weighted graph whose element nodes carry fractional abundances, then uses neural message passing to learn contextual element representations.

  • Results

    The graph-based formulation significantly improves sample efficiency over other structure-agnostic approaches, while learned representations transfer across data sets and uncertainty estimates decrease with restricted confidence percentiles.

  • Takeaways & Limitations

    Recasting materials problems as regression over multisets can support transferable, uncertainty-aware prediction and may extend to tasks such as predicting major inorganic-reaction products.

  • Takeaways & Limitations

    Deep Ensemble uncertainty estimates are typically not calibrated to error magnitude without post-hoc calibration, although active-learning workflows are described as robust to this miscalibration.

Abstract

from arXiv · show

Machine learning has the potential to accelerate materials discovery by accurately predicting materials properties at a low computational cost. However, the model inputs remain a key stumbling block. Current methods typically use descriptors constructed from knowledge of either the full crystal structure -- therefore only applicable to materials with already characterised structures -- or structure-agnostic fixed-length representations hand-engineered from the stoichiometry. We develop a machine learning approach that takes only the stoichiometry as input and automatically learns appropriate and systematically improvable descriptors from data. Our key insight is to treat the stoichiometric formula as a dense weighted graph between elements. Compared to the state of the art for structure-agnostic methods, our approach achieves lower errors with less data.

MODELS AND DATA

Roost represents stoichiometric compositions as dense weighted element graphs and learns material descriptors through attention-based message passing. The resulting representations are pooled into fixed-length material vectors for end-to-end property prediction.

  • Representation Learning of Inorganic Materials: Roost represents each composition as a dense weighted graph whose nodes are elements weighted by fractional abundance.This structure lets message passing incorporate the types and quantities of other elements into each node representation.
  • Representation Learning of Inorganic Materials: Element vectors are initialized with learnable representations that include each element’s fractional composition weight.Optional prior-knowledge embeddings can improve performance, particularly with scarce training data.
  • Representation Learning of Inorganic Materials: Weighted soft attention computes pairwise coefficients so element representations receive context-dependent, pair-dependent updates.The coefficients are learned from data and normalized using the elements’ fractional weights.
  • Representation Learning of Inorganic Materials: Multiple attention heads and repeated update operations are controlled by hyperparameters T and M.The updates use residual incorporation of learned pair-dependent perturbations.
  • Representation Learning of Inorganic Materials: A weighted attention pooling operation converts the element representations into a fixed-length material representation for property prediction.A feed-forward output network uses this representation, making the full model end-to-end differentiable.

Uncertainty Estimation

The model estimates uncertainty for structure-agnostic predictions by combining aleatoric measurement variability with epistemic model uncertainty. Deep ensembles provide the framework for estimating both components, while heteroskedastic outputs model input-dependent noise.

  • Uncertainty Estimation: Structure-agnostic screening can encounter materials outside the training distribution, making prediction uncertainty important to quantify.This is especially relevant when screening combinatorially generated novel compositions.
  • Uncertainty Estimation: Aleatoric uncertainty represents natural process variability, whereas epistemic uncertainty reflects variation among plausible models under sparse sampling.The Deep Ensemble approach is used to quantify both forms.
  • Uncertainty Estimation: The model predicts both a predictive mean and an input-dependent aleatoric variance under a heteroskedastic formulation.A Laplace noise distribution yields the training loss through maximum-likelihood estimation.
  • Uncertainty Estimation: Independent randomly initialized models provide samples of plausible parameter sets for estimating epistemic uncertainty.Different local minima in the non-convex loss landscape serve as ensemble members.
  • Uncertainty Estimation: Total ensemble uncertainty combines the epistemic contribution with the average aleatoric contribution across models.The ensemble marginalization also leads to lower average errors.

Baseline Model

The baseline uses Random Forests with Magpie’s fixed-length, hand-engineered elemental and stoichiometric features. Random Forest predictions average the outputs of multiple decision trees.

  • Baseline Model: Random Forests are a standard materials-science model combining multiple decision trees with fixed-length descriptors.The trees average predictions from weak regressors to reduce overfitting relative to individual decision trees.
  • Baseline Model: The baseline input is the Magpie feature vector, which contains 145 highly engineered features.These features encode prior knowledge about elements, stoichiometry, and electronic properties.

Data Sets

The evaluation uses OQMD formation enthalpies, Materials Project band gaps, and a smaller experimental band-gap dataset. Learning curves and held-out test tables assess performance across data regimes.

  • Data Sets: OQMD contains calculated average formation enthalpies per atom, evaluated here on 256,620 materials with the lowest-energy polymorph per stoichiometry.The lowest-energy-polymorph restriction makes the stoichiometry-to-property map well defined.
  • Data Sets: The Materials Project dataset contains band gaps for 43,921 non-metals, again restricted to the lowest-energy polymorph per stoichiometry.This restriction ensures a well-defined stoichiometry-to-property mapping.
  • Data Sets: Figure 2 varies training-set size while evaluating on a fixed out-of-sample OQMD test set using log-log learning curves.The plotted trends follow inverse power laws, and ElemNet results are included for comparison.
  • Data Sets: Table I reports MAE and RMSE on a randomly withheld 10% test set, with bracketed standard deviations.The table compares baseline and proposed models.
  • Data Sets: The experimental dataset contains measured band gaps for 3,895 non-metals.It is substantially smaller than the computational datasets.

RESULTS

The results evaluate sample efficiency and uncertainty quality on OQMD, showing earlier gains over baselines and more reliable confidence estimates from the full uncertainty framework.

  • Sample Efficiency: The learning-curve gradient and intercept on a log-log plot of training-set size against model error indicate sample efficiency.Average model error approximately follows an inverse power law with training-data quantity in the large-data limit.
  • Sample Efficiency: O(10^2) training points mark Roost’s crossover to outperforming the traditional machine-learning baseline, compared with O(10^3) for ElemNet.This crossover occurs at a size typical of experimental databases for novel material classes.
  • Uncertainty Evaluation: Large differences between random-ranking and uncertainty-based curves indicate that points with larger uncertainties have larger average errors.The error-based ranking curve provides a useful lower-bound comparison for uncertainty-estimate quality.
  • Uncertainty Evaluation: Uncertainty-based confidence-error curves assess how test-set error changes as the most uncertain data points are removed sequentially.Figure 3 uses the random-ranking curve as a reference and includes standard deviation from 500 random trials.
  • Uncertainty Evaluation: The full framework produces more reliable uncertainty estimates than a purely epistemic ensemble, despite comparable errors across the whole data set.The comparison concerns a framework incorporating both aleatoric and epistemic uncertainties versus an L1-loss ensemble estimating only a predictive mean.

Transfer learning

Transfer learning improves Roost’s performance on the EX dataset, with stronger benefits when the source and target tasks are cognate. Ablation results indicate that the framework’s multiset-regression formulation is more important than specific architectural choices.

  • Transfer learning: Transfer learning from OQMD to EX produces faster convergence and slightly lower errors than direct training, despite task mismatch.Transfer from MP to EX yields more pronounced benefits because the tasks are similar.
  • Ablation study: The ablation study evaluates element embeddings, robust loss, fractional element weights, pooling, residual architectures, and the output network across EX and OQMD.Ten randomly initialized models are trained for each design choice, with both individual-model and ensemble performance assessed.
  • Ablation study: All ablation models except Model 3 significantly outperform ElemNet and the Random Forest plus Magpie baseline on OQMD.Model 3 omits element weights, while the reference architecture’s design choices provide only slight additional improvements.
  • Ablation study: The Roost framework’s multiset-regression formulation, rather than specific architectural details, is identified as responsible for the observed improvements.This conclusion distinguishes the general representation strategy from individual architectural refinements.
  • Ablation study: Element embeddings that capture chemical correlations improve performance on the smaller EX dataset but make no significant difference on larger OQMD.The authors suggest that sufficient data can compensate for the absence of this domain knowledge, supporting continuously improving end-to-end featurization.
  • Ablation study: The robust loss performs comparably to L1 on EX, while on OQMD it yields lower MAE but higher RMSE for single models and lower MAE and RMSE after ensembling.The authors attribute the ensembled improvement to cancellation of errors on outlying, high squared-error data points.

CONCLUSION

The framework predicts materials properties from composition without crystal structures, improving sample efficiency and supporting uncertainty-aware and transferable workflows. Ablation results attribute the gains primarily to jointly learning descriptors and fits, while the framework also extends to other variable-size material problems.

  • CONCLUSION: Dense weighted-graph representations of compositions significantly improve sample efficiency over other structure-agnostic approaches.The framework is designed to predict properties without crystal structures.
  • CONCLUSION: Uncertainty estimates become more useful as the retained confidence percentile narrows, with steadily decreasing test-set average error.This behavior supports using the model to drive an active learning cycle.
  • CONCLUSION: Learnt representations transfer from data-abundant databases to improve performance on smaller experimental datasets.The authors identify self-supervised learning as a possible way to further bolster performance.
  • CONCLUSION: Ablations indicate that jointly learning the descriptor and fit drives improved performance rather than the specific message-passing architecture.The study examined the model’s component choices to identify the source of the improvement.
  • CONCLUSION: Roost can handle multisets of varying sizes, supporting applications such as predicting major products of inorganic reactions.The authors propose this set-regression framing as a broader direction for materials machine learning.

METHODS

The methods use fixed architecture and training choices across experiments, with a message-passing model, attention-based pooling, output network, uncertainty estimation, and transfer-learning procedures. Baselines and implementation settings are specified for reproducible comparisons.

  • METHODS: All investigated problems use the same architecture and hyperparameters, selected heuristically from graph-convolutional models.These choices were based on ideas from other graph convolution-based architectures.
  • METHODS: The reference model uses 3 message-passing layers, 3 attention heads per layer, and single-hidden-layer 256-unit LeakyReLU networks for attention pooling.The pooling networks implement the weighted soft-attention function.
  • METHODS: The output network has five hidden layers with 1024, 512, 256, 126, and 64 units, plus skip connections.Skip connections were added to help address the vanishing-gradient problem.
  • METHODS: The reference model contains 2.4 million parameters, approximately 10x the size of the OQMD training set.The network sizes were selected to make the model appropriately over-parameterised for OQMD.
  • METHODS: Aleatoric uncertainty is estimated by predicting a log quantity and exponentiating it, while epistemic uncertainty uses ensembles of W = 10.The two uncertainty contributions are combined within the Deep Ensemble procedure.
  • METHODS: Transfer-learning experiments re-optimise all model parameters after warm restarts rather than freezing message-passing layers.This procedure performed better than re-optimising only the output network.
  • METHODS: Neural models use Adam with learning rate 3 × 10^-4, batch size 128, weight decay 10^-6, and 250 training epochs.These settings were used for both main results and ablation experiments.
  • METHODS: Random Forest baselines use scikit-learn with Magpie features generated by Matminer, 0.25 maximum features, and 200 estimators.The baseline implementation combines the specified Random Forest settings with Magpie descriptors.

DATA AVAILABILITY

The study uses datasets assembled from openly available materials databases and an accompanying external dataset. OQMD and Materials Project data are identified by their respective public sources.

  • DATA AVAILABILITY: The OQMD dataset comes from the openly available Open Quantum Materials Database and uses the subset studied in prior work.The source is identified as oqmd.org.
  • DATA AVAILABILITY: The MP dataset was assembled through the Materials API from the openly available Materials Project database.The source is identified as materialsproject.org.
  • DATA AVAILABILITY: The EX dataset is available alongside the cited prior work.The passage does not provide a separate repository address.

SI: Predicting materials properties without crystal structure:

The supplementary material identifies the paper as “Deep representation learning from stoichiometry,” authored by Rhys E. A. Goodall and Alpha A. Lee at the University of Cambridge. It also notes additional prediction-distribution figures and gives the arXiv version date.

  • SI: Predicting materials properties without crystal structure:: The paper is titled “Deep representation learning from stoichiometry.”The title identifies the paper’s focus on learning representations from stoichiometric information.
  • SI: Predicting materials properties without crystal structure:: The authors are Rhys E. A. Goodall and Alpha A. Lee.The author line includes an asterisk after Lee.
  • SI: Predicting materials properties without crystal structure:: The authors are affiliated with the University of Cambridge’s Cavendish Laboratory in Cambridge, UK.The affiliation is stated directly in the supplementary material.
  • SI: Predicting materials properties without crystal structure:: Additional figures show how the model predictions are distributed.These figures are provided to improve understanding of prediction distributions.
  • SI: Predicting materials properties without crystal structure:: The cited arXiv version is v4, dated 23 Sep 2020.The identifier is arXiv:1910.00617v4.

MISCALIBRATION OF UNCERTAINTIES

Deep Ensemble uncertainty estimates are typically miscalibrated to regression error magnitude without additional calibration. This limitation is not prohibitive for materials-discovery active-learning workflows, which can remain robust to such miscalibration.

  • Deep Ensemble uncertainty estimates are typically not calibrated to the magnitude of regression error out-of-the-box.Post-hoc calibration can correct this when required by the application.
  • Materials-discovery active-learning workflows are typically robust against this kind of uncertainty miscalibration.Acquisition-function hyperparameters can tune the exploration–exploitation trade-off and act similarly to post-hoc calibration.
  • Uncertainty miscalibration is therefore not prohibitive for materials-discovery workflows.Selecting acquisition-function hyperparameters remains non-trivial even when uncertainties are calibrated.
Loading 1910.00617v4…