Source-linked AI summary
Explainable Machine Learning for Scientific Insights and Discoveries
Ribana Roscher, Bastian Bohn, Marco F. Duarte, Jochen Garcke
TL;DR
The paper addresses how machine learning can produce scientifically valuable insights and discoveries in the natural sciences while remaining explainable and scientifically consistent. It surveys explainable ML applications through transparency, interpretability, explainability, and domain-knowledge integration, concluding that these elements structure routes from ML models or outputs to scientific outcomes.
Problem
Scientific ML increasingly targets insights and discoveries, but understanding how models operate and why they make decisions is needed to ensure scientific value and remains conceptually unsettled.
Method
The paper surveys natural-science ML approaches and organizes their scientific-outcome chains using transparency, interpretability, explainability, and domain knowledge.
Results
The survey differentiates transparency, interpretability, and explainability, and characterizes how domain knowledge and consistency checks enter ML workflows for scientific outcomes.
Takeaways & Limitations
Scientific outcomes can be derived by explaining model outputs or transparent, interpretable models with domain knowledge, which can also promote scientific consistency.
Takeaways & Limitations
Human-centric explanations can be biased toward certain individuals and may reinforce confirmation biases in some tasks.
Abstract
from arXiv · showhide
Machine learning methods have been remarkably successful for a wide range of application areas in the extraction of essential information from data. An exciting and relatively recent development is the uptake of machine learning in the natural sciences, where the major goal is to obtain novel scientific insights and discoveries from observational or simulated data. A prerequisite for obtaining a scientific outcome is domain knowledge, which is needed to gain explainability, but also to enhance scientific consistency. In this article we review explainable machine learning in view of applications in the natural sciences and discuss three core elements which we identified as relevant in this context: transparency, interpretability, and explainability. With respect to these core elements, we provide a survey of recent scientific works that incorporate machine learning and the way that explainable machine learning is used in combination with domain knowledge from the application areas.
1 Introduction
Machine learning is increasingly used in the natural sciences to process scientific data and derive insights or discoveries, but scientifically valuable outcomes require explainability grounded in domain knowledge. The survey organizes explainable ML around transparency, interpretability, explainability, scientific consistency, and ML-based chains from models or outputs to scientific outcomes.
- Explainability is a prerequisite for ensuring the scientific value of outcomes derived from machine-learning models.
- Machine learning is increasingly adopted in the natural sciences to process large amounts of experimental, observational, and other scientific data.
- Scientific applications seek understanding, causal relationships, and novel insights or discoveries from observational or simulated data.
- Scientific discovery combines data-driven and analytical work, while new high-quality data and scalable computational infrastructure expand its reach.
- The survey distinguishes transparency, interpretability, and explainability within ML-based chains that derive scientific outcomes.
- Domain knowledge supports explainability and can foster scientifically consistent models and results.
2 Terminology
The paper distinguishes transparency, interpretability, and explainability, emphasizing that explanations require domain knowledge and an analysis goal. It also connects these concepts to scientific consistency and cautions that explanations can mislead.
- Transparency: Transparency concerns whether an ML approach’s structure, components, learning algorithm, and solution process can be described and motivated.The paper distinguishes model, design, and algorithmic transparency, while noting that complete transparency is generally unrealistic.
- Transparency: Kernel models illustrate how domain-specific design choices can make components more transparent, while optimization can connect design transparency to algorithmic transparency.A Gaussian kernel’s rationale may be unclear, whereas domain-specific distance measures and maximum-likelihood hyperparameter optimization improve transparency.
- Interpretability: Interpretability presents properties of an ML model in understandable terms by mapping abstract concepts, such as predicted classes, into domains humans can make sense of.Proxy models, prototypes, feature importance, saliency masks, and heatmaps are examples of interpretive approaches.
- Interpretability: Interpretation methods face a formal computational limitation: finding small relevant feature sets is NP-hard, supporting the practical use of heuristic methods.This result applies even when approximation is allowed within any non-trivial factor.
- Explainability: Explainability extends interpretation by combining model and data information with human context, especially domain knowledge and the analysis goal.Different questions, such as why a decision was made or why nearby data receive different decisions, require different explanations.
- Scientific consistency: Scientific consistency requires outcomes to be plausible and consistent with existing scientific principles, although explicitly restricting solutions is not mandatory for valuable scientific outcomes.Without such restrictions, mathematical optimality alone cannot guarantee a scientifically consistent and plausible result.
3 Scientific Outcomes From Machine Learning
The survey organizes scientific ML workflows by increasing use of transparency, interpretability, explainability, and domain knowledge. It reviews examples ranging from black-box prediction to models designed to yield scientifically interpretable outcomes.
- Core concepts: The survey distinguishes design and algorithmic transparency from interpretability of model components or scientific outcomes.Design transparency concerns domain-motivated model choices, while algorithmic transparency concerns traceable solution determination; interpretability concerns explainable components or outcomes.
- Core concepts: Domain knowledge supports both explanation and scientifically plausible, consistent results through model design, learning, and post-hoc checks.It may affect inputs, models, optimizers, outputs, cost functions, or feature generation.
- Survey organization: The reviewed literature is a non-exhaustive selection emphasizing natural-science applications with substantial domain knowledge, especially neural-network approaches.The authors organize representative works into groups and acknowledge that additional subgroups may exist.
- Survey organization: The survey’s four groups progress from no interpretability, through domain-informed design and output tools, to explaining the model itself for scientific insight.Table 1 summarizes this progression across the reviewed approaches.
- Group 1a: Group 1a uses black-box or model-transparent ML to predict outcomes without interpretability or clear scientific explanation.Examples include intuitive-physics prediction from videos and disease-image generation; the reviewed PDE approach forecasts future states but does not directly expose process structure.
- Group 1b: PhysNet illustrates design-transparent modeling by structuring its network around block arrangement and movement to predict collapsing-tower trajectories.It outperforms human subjects on synthetic data and achieves comparable results on real data.
- Group 1c: Group 1c incorporates domain knowledge into models, losses, or features to improve scientific consistency and plausibility.Examples include physics-informed fluid simulation losses and video models whose outputs are compared with physical laws, while standard CNN components remain insufficiently transparent.
4 Discussion
The discussion argues that explainability is central to extracting novel scientific results from ML, while noting that causal inference remains an emerging component requiring further basic research.
- Contributions: The survey distinguishes transparency, interpretability, and explainability while examining domain knowledge and consistency checks.It also characterizes different stages of explainability through recent natural-science examples.
- Discussion: Most scientific ML publications do not address interpretability or explainability, despite their relevance to extracting novel results and ideas.The authors frame explainability as a crucial part of deriving scientific outcomes from ML methods.
- Open directions: Causal inference is expected to become an additional component of future scientific data analysis workflows, but it requires more basic research.The authors place this need alongside the still-developing uptake of explainable ML in the natural sciences.