Source-linked AI summary
What do we need to build explainable AI systems for the medical domain?
Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, Douglas B. Kell
TL;DR
AI and ML achieve impressive results, but black-box models often cannot provide understandable explanations, creating a significant challenge for medical applications. The paper surveys explainable-AI approaches for medical images, *omics data, and text, including post-hoc and hybrid models. It argues that explainable-AI can support transparency and trust in medicine while noting important data and model-validation constraints.
Problem
Successful AI and ML models often lack understandable explanations, while medicine involves complex heterogeneous data and requires decisions that experts can retrace.
Method
The paper surveys explainable-AI research for medical images, *omics data, and text, including post-hoc explanations and hybrid models combining graph-based and dense representations.
Results
The paper reports interpretable concept linking that provides human-interpretable reasons, assigns general terms in context, and disambiguates terms without requiring a manually defined ontology.
Takeaways & Limitations
Explainable-AI research can facilitate medical AI implementation by supporting transparency and trust across complex medical data sources.
Takeaways & Limitations
Large-scale expert annotation can be expensive or unattainable, and activation-maximization interpretations require avoiding overfitted or underfitted experts.
Abstract
from arXiv · showhide
Artificial intelligence (AI) generally and machine learning (ML) specifically demonstrate impressive practical success in many different application domains, e.g. in autonomous driving, speech recognition, or recommender systems. Deep learning approaches, trained on extremely large data sets or using reinforcement learning methods have even exceeded human performance in visual tasks, particularly on playing games such as Atari, or mastering the game of Go. Even in the medical domain there are remarkable results. The central problem of such models is that they are regarded as black-box models and even if we understand the underlying mathematical principles, they lack an explicit declarative knowledge representation, hence have difficulty in generating the underlying explanatory structures. This calls for systems enabling to make decisions transparent, understandable and explainable. A huge motivation for our approach are rising legal and privacy aspects. The new European General Data Protection Regulation entering into force on May 25th 2018, will make black-box approaches difficult to use in business. This does not imply a ban on automatic learning approaches or an obligation to explain everything all the time, however, there must be a possibility to make the results re-traceable on demand. In this paper we outline some of our research topics in the context of the relatively new area of explainable-AI with a focus on the application in medicine, which is a very special domain. This is due to the fact that medical professionals are working mostly with distributed heterogeneous and complex sources of data. In this paper we concentrate on three sources: images, *omics data and text. We argue that research in explainable-AI would generally help to facilitate the implementation of AI/ML in the medical domain, and specifically help to facilitate transparency and trust.
1. Introduction and Motivation
AI and ML have achieved major practical successes, but their limited ability to explain results remains especially problematic in medicine. The paper motivates explainable-AI research for heterogeneous medical data and surveys explainable models for images, *omics data, and text.
- Motivation: AI and ML have advanced through practical successes, but a major challenge remains making sense of data within an application domain.The passage identifies learning, knowledge extraction, generalization, dimensionality reduction, and disentangling explanatory factors as broader AI goals.
- Medical Domain: Medical AI must integrate, fuse, and map distributed heterogeneous data in arbitrarily high-dimensional spaces.Relevant results may depend on diverse data sources, requiring medical professionals to understand how and why decisions were made.
- Medical Domain: Explainability and predictive performance are often in tension: deep learning can be less transparent, while decision trees can be less accurate.The passage also notes that algorithmic performance depends on data representations, which can require substantial engineering effort.
- Medical Domain: Transparent algorithms could enhance medical professionals’ trust in future AI systems.The paper links explainable-AI in medicine to maintaining strong learning performance alongside human-computer interaction techniques.
- Paper Scope: The paper surveys explainable models generally and selected research for images, *omics data, and text.It presents this combination of data types as needed for future medicine.
2. Explainability
Explainability is a longstanding AI problem because successful models often cannot make their decisions understandable. In medicine, the paper argues that experts need retraceable decisions, while explainable-AI must also address trust, safety, privacy, ethics, fairness, and usability.
- The Explainability Problem: AI systems can achieve practical success while remaining unable to explain their decisions in an understandable way.Even when the mathematical theories are understood, the models’ internal workings and the basis for their results can be difficult or impossible to inspect.
- Historical Context: Early symbolic AI systems supported explanations by tracing logical inference steps over human-readable symbols.The passage presents this traceability as a basis for related work on explainable systems.
- Terminology: In explainable-AI, understanding commonly means functional characterization of black-box behavior rather than elucidating internal representations.The terms understanding, interpreting, and explaining are often used synonymously, although model interpretability remains contested.
- Medical Application: Medical professionals need to understand and retrace machine decisions when AI systems complement or overrule them.The paper identifies medical education, research, and clinical decision making as purposes for explainable medicine.
- Requirements: Explainable-AI brings confidence, safety, security, privacy, ethics, fairness, trust, and usability onto the agenda for medical applicability.The passage extends these concerns to future personalized medicine.
3. Explainable Models
Explainable models can expose either local post-hoc explanations or interpretable structures by design. For neural networks, activation maximization and generative models produce representative prototypes, but the expert density model must be chosen carefully.
- Explainability types: Post-hoc explainability provides local, reproducible explanations for specific decisions, whereas ante-hoc systems are interpretable by design.LIME illustrates the post-hoc approach by fitting an interpretable model locally around an instance; linear regression, decision trees, and fuzzy inference systems exemplify ante-hoc models.
- Data constraints: Deep neural networks achieve broad practical success but depend heavily on large, carefully annotated datasets that are difficult and costly to obtain in medicine.Fine-grained or expert-level classification may require annotation that is expensive or unattainable at large scale.
- Activation maximization: Activation maximization searches for an input pattern that maximizes a neural network’s response for a selected class or quantity.The resulting prototype is obtained by optimizing class probabilities, commonly with gradient ascent and a regularizer or data-density model.
- Activation maximization: An expert data-density model replaces the basic regularizer to steer prototypes toward higher-probability regions of the input space.With a suitable expert, the prototype can correspond to the most likely input for a class; Gaussian RBMs and other density models are possible choices.
- Expert selection: Overfitted experts can hide model failure modes, while underfitted experts may produce prototypes distant from the data and unrepresentative of the class.The preferred degree of fit depends on the purpose: validation requires avoiding overfitting, whereas interpreting correctly predicted concepts requires preventing underfitting.
- Generative models: Generative adversarial networks learn a data distribution through a generator–discriminator minimax game and can decode optimized latent codes into class prototypes.The generator maps noise to data, while the discriminator estimates whether samples came from the training data; the decoded solution is x⋆ = g(z⋆).
4. Explainable Models for Image Data
For medical image data, explainability research maps neural activity back to interpretable inputs and explores image-captioning, attention, and structured causal models. These approaches aim to connect learned representations with human-understandable concepts.
- Explainable image models: Deconvolutional networks can unsupervisedly construct hierarchical image representations and visualize convolutional-network layers, a useful capability for very large digital pathology images.The paper contrasts digital pathology images with radiological images, noting that pathology images are orders of magnitude larger.
- Explainable image models: Mapping intermediate convolutional features back to input images reveals which input patterns produced particular feature-map activations.The paper identifies this mapping as an opportunity for research on causality in personalized medicine.
- Research motivation: Explainability insights from image models may contribute to building machines that learn and think more like people.This is framed as a fundamental-research motivation rather than an evaluated result.
- Image-to-text explanations: A second neural network can generate image captions or explanations, although its reasons are not guaranteed to correlate with those used by the original network.Neural attention mechanisms are proposed as a way to trace relevant image regions or features.
- Structured models: Structured and causal alternatives include Bayesian Rule Lists, Bayesian Program Learning, causal-relationship models, and stochastic grammars.These approaches are presented as ways to learn richer, more interpretable, and potentially causal models.
5. Explainable Features and Models in Images and *omics data
The paper presents AM-FM decompositions and related features as interpretable representations for medical images, then proposes integrating them into neural-network classifiers. It also identifies unmet needs for interpreting and connecting large-scale *omics data with hospital information.
- Image analysis: AM-FM decompositions represent medical images as visualizable components based on amplitude and instantaneous frequency.The decomposition exposes local texture content across frequency scales for human inspection.
- Image analysis: Multiscale AM-FM methods have been applied to diabetic-retinopathy screening, hysteroscopy, fMRI, MRI, and atherosclerotic-plaque imaging.These applications illustrate the breadth of medical-image analysis supported by the representation.
- Explainable neural models: The proposed AM-FM neural networks feed multiscale amplitude and frequency feature maps into multilayer networks or combine AM-FM decomposition with a hybrid ResNet architecture.The hybrid design aims to replace most lower-level ResNet layers and reduce the residual representation of the input image.
- Explainable neural models: Dominant Gabor filters provide a compact visualization of biomedical image content using 10 to 30 filters, compared with ResNet’s 152 difficult-to-visualize layers.The decompositions reveal directional selectivity, low-frequency concentration, and selected high-frequency components.
- *omics data: For *omics data, current platforms lack tools that jointly interpret high-throughput results and cross-link them to hospital data.The paper calls for bidirectional interaction with computational multiscale analysis and modelling.
6. Explainable Models for Text
The paper examines explainable text models that combine symbolic and neural representations for medical concept interpretation. An interpretable disambiguation system links concepts in context with human-readable reasons, while broader event-level semantic matching remains future work.
- Open challenges: Automated text understanding remains constrained because natural language is more expressive than tractable logical representations and lacks established text-to-ontology conversion methods.The paper identifies semantic matching beyond keywords and transfer from concepts to events as future work.
- Text-model background: Text models have shifted from interpretable symbolic systems toward statistical, probabilistic, and neural models with less explicit reasoning.Neural word embeddings replace input words with a few hundred uninterpretable dimensions.
- Medical text requirements: Medical text requires high-precision, human-verifiable reasons, motivating integration of knowledge-based and neural approaches.The medical domain already contains a large knowledge graph of terms in UMLS.
- Hybrid models: Hybrid distributional models combine sparse graph representations, dense vectors, lexical resources, and knowledge bases through iterative expert interaction.Experts improve the model and the resulting entity graph through successive annotation cycles.
- Contextual interpretation: Interpretable disambiguation supports accurate contextual concept linking and provides human-interpretable reasons for selecting concepts.The system can assign general terms and resolve words with multiple senses without requiring a manually defined ontology.
7. Conclusion and Future Outlook
Explainable-AI offers promising directions for medicine, but current examples do not provide a complete solution. Progress requires human-verifiable outputs, hybrid knowledge-based and neural methods, and effective user interfaces for understandable interaction.
- Current explainable-AI examples offer promising directions for medicine but do not yet provide a complete solution.
- Medical text requires human-verifiable, high-precision reasons because medical reports are legally binding.
- Hybrid models combine the interpretability of knowledge-based approaches with the efficiency of neural approaches.
- Effective user interfaces are needed to present understandable explanations and support two-way interaction.