Source-linked AI summary
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
Aniek F. Markus, Jan A. Kors, Peter R. Rijnbeek
TL;DR
AI has major potential in health care, but limited adoption reflects unresolved transparency and trust challenges. This paper surveys recent explainable-AI literature, develops guidance for designing and selecting methods, and concludes that explainable modelling may be preferred while practical usefulness remains insufficiently evidenced.
Problem
Health-care AI adoption is limited, while transparency, explainability meaning, method selection, and standardized evaluation remain unresolved.
Method
The paper reviews recent conceptual and methodological explainable-AI literature to clarify terminology, evaluate methods, and guide health-care system design.
Results
Explainable modelling might be preferred over post-hoc explanations for creating trustworthy health-care AI.
Takeaways & Limitations
Explainable AI can contribute to trustworthy AI, but complementary measures might be needed in health care.
Takeaways & Limitations
Evidence of explainability’s usefulness in practice is still lacking.
Abstract
from arXiv · showhide
Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).
Highlights
The survey aims to formalize explainable AI and guide method selection, while assessing quantitative evaluation metrics and emphasizing that complementary measures may be needed for trustworthy AI.
- The paper provides a comprehensive survey to guide researchers and practitioners and formalize explainable AI.
- The survey assesses quantitative evaluation metrics for explainability.
- It offers step-by-step guidance for choosing among classes of explainable AI methods.
- Explainable AI can contribute to trustworthy AI, but complementary measures might also be needed.
1. Introduction
AI offers substantial potential in health care, yet adoption remains limited because transparency and trustworthy implementation remain unresolved. This survey reviews recent conceptual and methodological literature to clarify explainability, evaluate methods, and guide system design.
- AI could improve health and well-being through clinical augmentation, prevention opportunities, and personalized treatment recommendations.
- Clinical adoption remains limited, with lack of transparency identified as a key implementation barrier.
- Trustworthy health-care AI faces concerns about bias, robustness, generalizability, explanation, accountability, malicious attacks, and practical regulation.
- Explainable AI seeks insight into how and why models produce predictions while maintaining high predictive performance, but suitable explanations and their quality remain unclear.
- The authors review recent conceptual and methodological literature rather than applications, using searches, influential-paper selection, reference tracking, citations, and arXiv preprints.
- The survey addresses explainability’s meaning, usefulness, available methods, evaluation, and method selection to guide health-care system design and formalize the field.
2. What does explainability mean?
The paper treats explainability as a broad concept requiring both human interpretability and fidelity to model behavior. It proposes practical definitions while noting that terminology remains inconsistent and some properties are difficult to measure.
- Terminology: Existing literature uses explainability-related terms inconsistently, including different meanings for interpretability, comprehensibility, transparency, and understandability.The authors therefore do not distinguish comprehensibility, intelligibility, and understandability from interpretability in practice.
- Core definitions: Explainability requires an explanation to be both understandable to humans and faithful to the task model.Interpretability concerns human understanding, whereas fidelity concerns descriptive accuracy of model behavior.
- Design implications: Achieving interpretability and fidelity simultaneously is challenging, and the usefulness of an explanation depends on the target user and reason for requesting it.Relevant users may include developers, deployers, and clinicians, whose expertise and contextual values differ.
- Core definitions: Interpretability is a property of an explanation, whereas explainability encompasses the task model or a post-hoc explanation.The task model generates predictions; post-hoc explanations provide model insights without requiring access to its mechanisms.
- Operational properties: The paper defines interpretability through clarity and compactness, and fidelity through completeness and soundness.Clarity requires an unambiguous rationale, completeness describes the model dynamics sufficiently to compute outputs, and soundness requires truthfulness to the task model.
3. Why and when can explainability be useful?
Explainability can support verification, social interaction, and discovery of new insights in health care. Its value depends on the use case because explanations require resources and do not guarantee that other model desiderata are satisfied.
- Limits: Explainability cannot guarantee that model desiderata such as fairness, generalizability, privacy, robustness, and security are satisfied.The paper also states that the practical benefits of explainability remain unproven.
- Motivations: The paper identifies three reasons to demand explainability: verifying model desiderata, managing social interaction, and discovering new insights.These needs determine which explainability properties matter and influence system design.
- Verifying model desiderata: Explanations can help clinicians verify whether models use appropriate features for predictions.Interviews with clinicians found feature-based explanations desirable for checking whether appropriate or inappropriate features were used.
- Managing social interaction: Explanations can create shared meaning around decisions and help clinicians justify decisions to colleagues and patients.The paper also connects this motivation to the GDPR’s ‘right to explanation’.
- Discovering new insights: Explainability can support knowledge discovery by enabling comparisons between learned strategies and existing knowledge.The resulting insights may guide future research, including drug development and clinical-trial design.
- Costs and scope: Explanations can be costly and may provide less benefit when a model has little impact or has already demonstrated sufficient performance.In health care, high stakes and difficult predeployment testing can nevertheless increase the value of explanations.
4. Which explainable AI methods are available?
The paper classifies explainable AI methods by how explanations are generated and whether they describe individual predictions or the model globally. The classes differ in the information they provide and their explainability properties.
- Explainable modelling versus post-hoc explanation: Explainable modelling uses an intrinsically interpretable task model, while post-hoc methods add an explanation to a potentially opaque model.Model-based post-hoc explanations use an interpretable model to explain the task model and may not be faithful.
- Scope: Global explanations can also address individual predictions but are less accurate for them than local explanations.The distinction between global and local methods is therefore based on explanatory scope and accuracy for individual cases.
- Classification: The proposed classification distinguishes model-based, attribution-based, and example-based explanations, each available globally or locally.Global explanations address the model as a whole, whereas local explanations address individual predictions.
- Interpretable models: Interpretable model classes include sparse linear classifiers, discretization methods, and example-based models.Interpretability also depends on feature comprehensibility and model complexity; a deep decision tree may be less interpretable than a compact neural network.
- Health-care examples: Generalized additive models with pairwise interactions achieved the best overall predictive performance in the cited pneumonia-risk and readmission example while enabling error recognition and correction.The interpretable model exposed the erroneous association that asthma lowered pneumonia mortality risk.
- Attribution-based explanations: Attribution methods rank or measure the explanatory power of input features, helping clinicians identify features responsible for predicted outcomes.The paper focuses on perturbation-based attribution methods because backpropagation methods are often model-specific or require differentiability.
5. How can explainability be evaluated?
The survey reviews application-, human-, and functionality-grounded evaluation, emphasizing quantitative metrics for objective comparison. It finds that evaluation coverage differs substantially across explanation properties and method classes.
- Evaluation approaches: Evaluation approaches are application-grounded, human-grounded, or functionality-grounded, with the latter generally more objective than evaluations involving selected human participants.Application-grounded studies provide the strongest evidence of success but are expensive, time-consuming, and subjective.
- Evaluation framework: The paper focuses on quantitative metrics because they are less studied and needed for objective initial assessment and formal comparison of explanation methods.The evaluated properties are clarity and parsimony for interpretability, plus completeness and soundness for fidelity.
- Model-based explanations: Model-based explanations satisfy completeness by providing sufficient information to compute outputs, while task-model explanations also satisfy soundness.Post-hoc model-based explanations can assess soundness using the percentage of predictions matching the task model.
- Clarity: Global explanations usually satisfy clarity, but no metric was found for the clarity of local model-based explanations.Local explanations may differ substantially even for similar instances.
- Parsimony: Multiple metrics are available for parsimony in model-based explanations, including feature counts, relation complexity, and runtime operation counts.The available metrics remain model-dependent in important cases.
- Attribution-based explanations: Attribution-based explanations are partial and lack completeness metrics, although most reviewed methods can measure soundness and local attribution clarity.The paper distinguishes empirical evaluations, which directly measure performance, from axiomatic evaluations of ideal properties.
6. How to choose amongst different explainable AI methods?
Choosing an explainable AI method requires balancing predictive performance with the form and importance of explainability for the specific use case. The proposed framework guides developers through these trade-offs across modelling approaches, explanation types, and explanation scope.
- Step 1: Relative importance: The framework begins by assessing how important explainability is relative to predictive performance and the needs of end-users.When explainability is unimportant, a black-box model with the best predictive performance may be preferable because explanations can be costly.
- Step 1: Relative importance: Explainable modelling and post-hoc explanation trade off predictive performance against explainability.Explainable modelling may reduce predictive performance, whereas post-hoc explanations may sacrifice some explainability.
- Step 1: Relative importance: When interpretability and fidelity are both very important, explainable modelling is the most appropriate design choice.Post-hoc explanations approximate the task model and can present plausible but misleading explanations because they are not completely faithful.
- Step 2: Complex versus interpretable models: Developers should compare complex models with interpretable alternatives before choosing post-hoc explanations.If predictive performance decreases substantially with an interpretable model, a post-hoc explanation can be selected.
- Step 3: Fidelity versus interpretability: Post-hoc explanation introduces a trade-off between interpretability and fidelity, whose priority depends on the application.Fidelity is emphasized for verifying model desiderata or discovering mechanisms, while interpretability supports social interaction; model-based explanations suit fidelity and attribution- or example-based explanations often suit interpretability.
- Step 3: Fidelity versus interpretability: The framework recommends choosing global or local explanations according to the use case, with local explanations often more appropriate in health care.Global explanations are attractive because they usually satisfy clarity.
7. Discussion
The discussion positions explainability as one component of trustworthy health-care AI rather than a complete solution. It highlights a framework for design choices while emphasizing evaluation gaps and complementary measures such as data-quality reporting, external validation, and regulation.
- Contributions: The survey extends prior work with practical definitions, a framework for choosing explainable AI method classes, and an assessment of quantitative evaluation metrics.The framework provides concrete recommendations for designing explainable AI systems in health care.
- Explainability and trustworthy AI: When explainability supports verification of whether AI is lawful, ethical, and robust, fidelity is especially important and explainable modelling is preferred.The framework connects fidelity with identifying the model’s underlying mechanisms and assessing other model desiderata.
- Explainability and trustworthy AI: The framework recommends model-based explanations for post-hoc use because they satisfy completeness and have quantitative proxy metrics for soundness.The authors also note that post-hoc explanations can be misleading and suggest argumentative support as a possible response.
- Future research: The proposed framework itself remains a proposal that needs evaluation and possible refinement in practice.Further research is also needed on explainable models, interpretable feature engineering, and hybrid data-driven and knowledge-driven methods.
- Evaluating explanations: Explanation-quality evaluation remains underdeveloped, with no standard evaluation methods and gaps for clarity in local explanations and example-based methods.Interpretability is recognized as user dependent but is not quantified as such.
- Evaluating explanations: Quantitative proxy metrics should be complemented with human evaluation before AI systems are deployed in real-life settings.Good metric performance may not directly establish practical success, and explanations do not necessarily improve human-machine task performance.
- Complementary measures: Trustworthy health-care AI may also require reporting data quality, extensive external validation, and regulation.Real-world data may contain biases, mistakes, or incompleteness; external validation addresses concerns about robustness and generalizability.
- Complementary measures: Data quality and collection practices help users understand limitations of the resulting model, while common data structures can support large-scale external validation and reproducibility.The OHDSI network provides standards and tools for transparent development and external validation of patient-level prediction models.
8. Conclusion
The conclusion presents a survey-based framework for designing explainable AI in health care and formalizing the field. It favors explainable modelling in some trustworthy-AI contexts, but stresses that explainability’s practical value remains unproven and may require complementary safeguards.
- Conclusion: The paper surveys literature to guide health-care explainable AI design and contribute to formalizing the field.It connects different perspectives and provides concrete design recommendations.
- Conclusion: The paper proposes a framework to choose among classes of explainable AI methods.The framework is intended to support concrete design decisions.
- Conclusion: Explainable modelling might be preferred over post-hoc explanations when using explainable AI to create trustworthy health-care AI.This preference is presented as a conclusion of the survey rather than a universal requirement.
- Conclusion: Evidence for the usefulness of explainability in practice is still lacking.The value of explainable AI methods remains to be proven in practice.
- Conclusion: Complementary measures might be needed to create trustworthy AI in health care, including data-quality reporting, extensive external validation, and regulation.These measures are identified alongside explainability rather than as replacements for it.
Declaration of competing interest
The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.
- The authors declare no known competing financial interests or personal relationships influencing the reported work.
Funding
The project was supported by the European Health Data and Evidence Network (EHDEN), which received funding through the Innovative Medicines Initiative 2 Joint Undertaking.
- The project received support from the European Health Data and Evidence Network (EHDEN).
- EHDEN received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 806968.
- The Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA.