PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable and Explainable Machine Learning Methods for Accelerated Material Discovery

Bhavya Kailkhura, Brian Gallagher, Sookyung Kim, Anna Hiszpanski, T. Yong-Jin Han

arXiv:1901.02717v2physics.comp-phcond-mat.mtrl-scistat.ML

TL;DR

ML for materials science must handle underrepresented and distributionally skewed data, where existing evaluation and confidence measures can be unreliable. The paper proposes a general-purpose framework combining simpler models, transfer learning, trust-oriented evaluation, and rationale generation, and demonstrates it on crystalline compounds and potentially stable solar cells.

  • Problem

    Underrepresented and distributionally skewed material data expose shortcomings in existing ML training, testing, and uncertainty-quantification practices.

  • Method

    The paper proposes a general-purpose explainable and reliable framework using simpler models, transfer learning, a new evaluation metric and trust score, and rationale generation.

  • Results

    The framework is demonstrated by predicting five physically distinct crystalline-compound properties and identifying potentially stable solar cells.

  • Takeaways & Limitations

    The proposed pipeline combines reliability, explainability, and prediction of material properties for applications involving imbalanced data.

  • Takeaways & Limitations

    Applying ML without careful consideration of its assumptions and limitations can lead to missed opportunities, wasted resources, and incorrect scientific inferences.

Abstract

from arXiv · show

Material scientists are increasingly adopting the use of machine learning (ML) for making potentially important decisions, such as, discovery, development, optimization, synthesis and characterization of materials. However, despite ML's impressive performance in commercial applications, several unique challenges exist when applying ML in materials science applications. In such a context, the contributions of this work are twofold. First, we identify common pitfalls of existing ML techniques when learning from underrepresented/imbalanced material data. Specifically, we show that with imbalanced data, standard methods for assessing quality of ML models break down and lead to misleading conclusions. Furthermore, we found that the model's own confidence score cannot be trusted and model introspection methods (using simpler models) do not help as they result in loss of predictive performance (reliability-explainability trade-off). Second, to overcome these challenges, we propose a general-purpose explainable and reliable machine-learning framework. Specifically, we propose a novel pipeline that employs an ensemble of simpler models to reliably predict material properties. We also propose a transfer learning technique and show that the performance loss due to models' simplicity can be overcome by exploiting correlations among different material properties. A new evaluation metric and a trust score to better quantify the confidence in the predictions are also proposed. To improve the interpretability, we add a rationale generator component to our framework which provides both model-level and decision-level explanations. Finally, we demonstrate the versatility of our technique on two applications: 1) predicting properties of crystalline compounds, and 2) identifying novel potentially stable solar cell materials.

I. INTRODUCTION

Materials science applies ML to discovery and characterization, but its underrepresented, skewed data and need for trustworthy explanations create challenges that commercial ML settings may not share. Standard accuracy and confidence measures can therefore mislead decisions and scientific inference.

  • Materials science uses ML for tasks including material discovery, development, optimization, synthesis, and characterization.
  • Material-science datasets are often imbalanced because researchers target uncommon properties, leaving desired materials in the minority class.
  • 95% of OQMD compounds possibly have zero band gap, so maximizing overall accuracy can ignore the minority class of interest.
  • Increasing model complexity may improve accuracy while reducing interpretability, whereas simpler explanatory models can produce misleading predictions through an accuracy trade-off.
  • Cross-validation accuracy and model-reported confidence may not establish trustworthy generalization when densely sampled data are scarce and distributions are skewed.
  • Reliable materials-informatics systems therefore require additional validation and uncertainty-quantification methods to assess when predictions can be trusted.

B. Literature Survey

Prior materials-informatics work largely emphasized predictive accuracy, while imbalanced learning, explainability, and reliability received less attention. This paper addresses that gap with a general framework combining simpler models, transfer learning, rationale generation, and trust-oriented evaluation.

  • Prior work: Earlier materials-informatics studies developed ML models for melting temperature, formation enthalpy, crystal formation, band gaps, mechanical properties, and diverse material properties.
  • Research gap: Research predominantly focused on overall predictive accuracy, whereas imbalanced learning, explainability, and reliability received limited attention.
  • Contributions: The paper identifies shortcomings in training, testing, and uncertainty quantification for underrepresented and distributionally skewed data.
  • Contributions: The framework combines a novel imbalanced-data training architecture, reliable evaluation and uncertainty quantification, and rationale generation for model-level and decision-level explanations.
  • Explainability: Partitioning materials into property-based subclasses enables simpler regression models intended to enhance explainability rather than perform exhaustive accuracy-oriented grouping.
  • Transfer learning: Transfer learning exploits correlations among material properties to overcome the predictive-performance loss caused by simpler models.
  • Interpretability and applications: The rationale generator uses prototypes for individual predictions and feature importance for material subclasses, while applications cover five crystalline-compound properties and potentially stable solar cells.

II. RESULTS AND DISCUSSIONS

The proposed general-purpose framework targets reliable, explainable, and accurate ML under imbalanced data rather than optimizing accuracy for one application. It is designed to reuse existing algorithms across material-science problems through three complementary components.

  • Framework objective: The framework addresses reliable learning and inference from distributionally skewed data with generic methods for reliability, explainability, and accuracy.
  • Framework scope: The framework is agnostic to training-data type, can use various existing ML algorithms, and is reusable across material-science problems.
  • Framework components: Its three components are an imbalanced-data training procedure, a rationale generator, and reliable testing with uncertainty quantification.

1) Training Procedure:

The proposed training procedure addresses distributionally skewed material-property data by balancing property classes, training simpler class-specific models, and transferring information across correlated properties. It converts regression into classification plus regression while retaining continuous property prediction and improving performance through knowledge transfer.

  • Distributionally skewed and underrepresented material data can prevent ML models from generalizing well, motivating a training process applicable across materials-science applications.
  • Domain knowledge partitions continuous property values into material classes, such as conductors, semiconductors, and wide-bandgap materials.
  • The pipeline partitions the property space, subsamples each class into balanced datasets, trains one classifier per property, and fits class-specific regressors.
  • At test time, a property classifier selects a class-specific regressor, which predicts the continuous property value for the compound.
  • The simpler class-based design improves understanding of material subdomains and supports rationale generation, while transfer learning exploits correlations among properties.
  • Augmenting the original feature vector with independently estimated property values enables knowledge transfer and significantly improves regression performance.

2) Rationale Generator:

The rationale generator explains predictions at both the decision and model levels. It uses similar known compounds as prototypes and decomposes feature importance into class discrimination and within-class regression.

  • Model Level Explanations: The framework can incorporate other imbalanced-learning strategies, including ensemble learning, preprocessing, and cost-based learning, to further improve performance.
  • Decision Level Explanations: Decision-level explanations provide reasoning for predictions by presenting similar experimentally known compounds and their similarity to each unseen test compound.
  • Decision Level Explanations: Gower’s metric measures similarity for the heterogeneous feature space containing continuous and categorical attributes.
  • Decision Level Explanations: The classification-plus-regression structure supplies a logical sequence of decisions leading to each prediction.
  • Model Level Explanations: Model-level explanations identify which chemical attributes matter and how they are combined in the prediction.
  • Model Level Explanations: Feature importance is separated into inter-class importance for distinguishing material classes and intra-class importance for regression within a material subdomain.

3) Robust Model Performance Evaluation and Uncertainty Quantification:

Distributionally skewed material data can make aggregate evaluation metrics and model confidence misleading, especially for minority classes. The proposed pipeline combines class-specific evaluation, simpler models, transfer learning, uncertainty quantification, and rationale generation to improve reliable and explainable prediction.

  • Robust evaluation: Standard aggregate metrics can break down when train/test data are distributionally skewed and unrepresentative of the true distribution.The paper proposes partitioning the property space into classes for more reliable evaluation.
  • Applications: The framework applies the same attributes and machine-learning pipeline to predict five physically distinct crystalline-compound properties and identify potentially stable solar cells.The applications cover reliability, explanations, evaluation, accuracy assessment, and candidate-material prediction.
  • Robust evaluation: Class-specific evaluation reveals that conventional regressors perform well mainly on majority classes but poorly on minority classes, especially for bandgap energy and stability.Aggregate scores can therefore conceal minority-class failures.
  • Model design: Simpler models lose accuracy without transfer learning, but correlation-based transfer learning compensates for that loss and can outperform conventional regression, particularly on minority classes.The authors attribute the gain to exploiting correlations among properties and note positive effects from sub-sampling on minority-class regression.
  • Uncertainty quantification: Prediction uncertainty is informative because regressors perform poorly in regions with high uncertainty.The pipeline therefore quantifies uncertainty alongside its predictions.
  • Limitations: A severe data-coverage limitation remains: when a class is heavily under-represented, model-design strategies cannot improve performance.The paper also reports overfitting when transfer learning is applied to the conventional technique.
  • Explainability: The rationale generator identifies melting temperature, electronegativity, and volume per atom as important features for discriminating bandgap-energy classes.These attributes are reported as the most important for determining compound bandgap-energy levels.

2) Novel Stable Solar Cell Prediction:

The proposed pipeline searches undiscovered ternary compounds for stable solar-cell candidates with bandgap energies in the desired range, using multiclass classification, regression, confidence, and trust scores. Among 4,500 candidates, 221 were identified as likely to meet the target criteria, including compounds with experimentally known structural prototypes.

  • Data Set: The study uses 228,573 lowest-energy OQMD compounds and 145 attributes to predict bandgap energy and stability.The test set contains 4,500 as-yet-undiscovered ternary compounds not present in OQMD.
  • Method: The pipeline partitions each property into three classes, applies XGB multiclass classifiers and GBR marginal and joint regressors, and ranks candidates using confidence and trust scores.Bandgap thresholds are 0.9 and 1.7 eV; stability thresholds are 0.0 and 1.5.
  • Results: 221 of 4,500 compounds were identified as likely to have stability and bandgap energy within the desired solar-cell range.The target bandgap range is 0.9–1.7 eV.
  • Results: The predicted candidates are accompanied by experimentally known prototypes and their distances, which can provide initial guesses for three-dimensional crystal structures.A subset ranked by trust score is shown in Table IV.
  • Results: Four of six suggested compounds are I-III-VI semiconductors, while Th2CO2 and Pm1.33PtSe3 are distinctive because they contain actinide or lanthanide elements.The scarcity and radioactivity of these elements may make experimental exploration challenging.
  • Results: The scarcity and radioactivity of actinide and lanthanide elements may constrain practical experimental exploration of two identified candidates.The two compounds are Th2CO2 and Pm1.33PtSe3.

III. SOME OPEN ISSUES

The paper identifies unresolved issues involving data scarcity, verification of model-generated material recommendations, and interpretation of feature-based explanations. These issues motivate further work on sampling, synthesis-aware recommendation, and explainable ML techniques.

  • Data scarcity: When a material class is heavily underrepresented, model-design strategies may not improve performance, making new data generation potentially necessary.The paper poses how many samples are sufficient and where to sample when existing data are inadequate.
  • Verification and synthesis: Models based on chemical attributes recommend potential materials, but verifying them with DFT or experiments requires identifying suitable crystal structures or synthesis recipes.The paper suggests biasing recommendations toward compounds with favorable synthesis conditions.
  • Explainability: Feature-importance explanations still require material scientists to interpret model and decision explanations using domain knowledge.This interpretation may be affected by human bias.
  • Future directions: Addressing these issues will require significant advances in current explainable ML techniques, with interactive ML and causal inference identified as possible directions.These techniques are presented as potential ways to resolve some open problems.

IV. CONCLUSIONS

The paper studies reliable and explainable ML for underrepresented and distributionally skewed materials data, identifies pitfalls in existing approaches, and proposes a general-purpose framework to address them. The framework combines improved evaluation and confidence quantification with explanations, and is demonstrated on crystalline-property prediction and stable solar-cell discovery.

  • Contributions: The paper addresses reliable and explainable ML for models trained on underrepresented and distributionally skewed materials-science data.It identifies common pitfalls in existing ML techniques for imbalanced data.
  • Problem: Applying ML without considering its assumptions and limitations can produce quantitatively and qualitatively incorrect predictive models.This motivates the proposed framework for imbalanced material data.
  • Framework: The proposed framework combines a general-purpose explainable and reliable ML design with a new evaluation metric and trust score for prediction confidence.These components target learning from imbalanced material data and quantifying confidence.
  • Explainability: A rationale generator provides model-level and decision-level explanations for the framework’s predictions.The explanations are intended to establish trust in the model and its predictions.
  • Applications: The technique is demonstrated by predicting five physically distinct crystalline-compound properties and identifying potentially stable solar-cell materials.The conclusion reports applicability across both prediction and discovery tasks.

V. MATERIALS AND METHODS

The study implements its ML models with Scikit-learn and XGBoost, computes material attributes using Magpie, and provides the software, training data, and input files in the supplementary information.

  • Implementation: The machine-learning models were created using the Scikit-learn and XGBoost libraries.
  • Implementation: Material attributes were computed with the Materials Agnostic Platform for Informatics and Exploration, or Magpie.
  • Reproducibility: The software, training datasets, and input files used in the work are provided in the supplementary information.

VII. CONTRIBUTIONS

The project was conceived, conducted, and analyzed by the listed authors, who jointly discussed the results and contributed to the manuscript. The study’s data are available in the article and supplementary files.

  • B.K. and T.Y.H. conceived the project, B.K. performed the experiments, and B.K. and B.G. analyzed the results.
  • All authors discussed the results and contributed to writing the manuscript.
  • All data generated or analyzed in the study are included in the published article and supplementary information files.

A. Attributes and Properties

The framework represents crystalline compounds with 145 chemically descriptive attributes to predict five physically distinct OQMD properties. It uses these representations to support reliable, explainable prediction and interprets formation-energy models through feature importance, where thermodynamic stability is most important across classes.

  • Attributes and Properties: The pipeline first computes chemical descriptors from material composition to represent each compound for machine-learning prediction.
  • Attributes and Properties: Each compound is represented by 145 attributes spanning stoichiometric properties, elemental statistics, electronic-structure properties, and ionic-compound attributes.
  • Attributes and Properties: The models target five physically distinct OQMD properties, including bandgap, energy, volume per atom, energy per atom, thermodynamic stability, and formation energy.
  • Outputs: The work provides feature-importance results for class-specific predictors and a detailed list of potentially stable solar-cell materials with predicted properties and explanations.Software, training datasets, and input files are also provided in the supplementary information.
  • Feature Importance: Thermodynamic stability is the most important attribute for formation-energy prediction across all three classes.The authors relate this result to the negative correlation between stability and formation energy.
Loading 1901.02717v2…