Source-linked AI summary
Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
Melkamu Mersha, Khang Lam, Joseph Wood, Ali AlShami, Jugal Kalita
TL;DR
Complex AI models are difficult to trust because their decision processes are opaque, especially in sensitive and safety-critical applications. This paper conducts a comprehensive survey of XAI concepts, techniques, applications, evaluation, challenges, and future directions, reviewing more than two hundred studies. It concludes by organizing the field’s terminology and methods while identifying ongoing challenges and research directions.
Problem
Complex AI models operate as black boxes, creating needs for explainability, interpretability, accountability, and transparency in consequential applications.
Method
The paper performs a comprehensive literature review covering XAI terminology, beneficiaries, techniques, applications, evaluation methods, challenges, and future directions.
Results
More than two hundred research articles were surveyed in a review spanning traditional neural networks, reinforcement learning, and Transformer-based models.
Takeaways & Limitations
The survey provides a systematic overview for researchers, practitioners, developers, and other XAI beneficiaries seeking to understand the field’s methods and applications.
Takeaways & Limitations
LRP requires access to internal model structure and parameters, limiting its applicability when models are proprietary.
Abstract
from arXiv · showhide
Artificial intelligence models encounter significant challenges due to their black-box nature, particularly in safety-critical domains such as healthcare, finance, and autonomous vehicles. Explainable Artificial Intelligence (XAI) addresses these challenges by providing explanations for how these models make decisions and predictions, ensuring transparency, accountability, and fairness. Existing studies have examined the fundamental concepts of XAI, its general principles, and the scope of XAI techniques. However, there remains a gap in the literature as there are no comprehensive reviews that delve into the detailed mathematical representations, design methodologies of XAI models, and other associated aspects. This paper provides a comprehensive literature review encompassing common terminologies and definitions, the need for XAI, beneficiaries of XAI, a taxonomy of XAI methods, and the application of XAI methods in different application areas. The survey is aimed at XAI researchers, XAI practitioners, AI model developers, and XAI beneficiaries who are interested in enhancing the trustworthiness, transparency, accountability, and fairness of their AI models.
1. Introduction
The survey motivates XAI by the opacity of complex AI systems and presents a comprehensive review intended to address limitations in prior studies. It covers terminology, techniques, applications, evaluation, challenges, and future directions.
- Motivation: Complex AI systems are difficult to trust because their decision processes and internal architectures are opaque to users.This opacity is especially consequential in domains requiring security, transparent decisions, and responsibility for protecting information.
- Motivation: The black-box nature of AI raises concerns about explainability, interpretability, accountability, transparency, legality, ethics, and safety.These concerns affect scientists, the public, regulators, and politicians.
- Scope: The survey addresses open questions involving decision explanations, interpretability approaches, ethical implications, and potential biases or errors.It aims to advance XAI adoption among practitioners, educators, system designers, developers, and other beneficiaries.
- Contributions: The survey claims comprehensive coverage of explainability across traditional neural networks, reinforcement learning, and Transformer-based models, including large language and Vision Transformer models.It also covers application areas, evaluation methods, challenges, and future directions.
- Contributions: More than two hundred research articles were surveyed to provide a comprehensive review of XAI concepts and research.The review addresses limitations observed in previous survey studies.
- Contributions: The paper discusses XAI terminology, technique types and dimensions, applications, evaluation methods, challenges, and future research directions.Its eight-section organization follows these topics from background and motivation through conclusion.
2. Background and Motivation
This section defines the background and motivation for XAI, emphasizing that complex and widely deployed AI systems require understandable decision processes. It connects explainability with trust, governance, debugging, safety, collaboration, and the unresolved difficulty of defining and measuring interpretability.
- Background: XAI systems provide explanations for decisions or predictions and insights into model behavior, including where models succeed or fail.This operationalizes the question of why an AI system produced a particular outcome.
- Background: Black-box AI models provide predictions without reasonable details about how or why decisions were made.Model complexity, optimization procedures, training data, and training algorithms contribute to this opacity.
- Need for XAI: Explainability is needed because AI systems are pervasive and affect performance, reliability, trustworthiness, accountability, and transparency in real-world applications.The motivation is particularly strong when automated decisions affect people’s lives.
- Need for XAI: XAI supports governance by making AI decisions more ethical, accountable, transparent, explainable, and compatible with legal and regulatory requirements.It may also help identify the roots of biased decisions involving sensitive or protected attributes.
- Need for XAI: XAI can support model design and debugging by identifying relevant features and tuning hyperparameters such as activation functions, layer counts, and learning rates.These uses are presented as potential benefits for developing accurate and practical models.
- Need for XAI: Explainability is important for safety in applications such as self-driving cars and military drones, where understanding AI decisions concerns human security and lives.XAI can also facilitate human-AI collaboration by exposing the reasoning behind AI actions.
- Open Issues: Interpretability and explainability remain difficult to define and measure rigorously, with terminology that varies or can be contradictory.Studies generally treat them as related but distinct concepts.
3. Categories of Explainability Techniques
The survey categorizes XAI techniques by explainability scope, implementation stage, model dependence, and design methodology. It distinguishes local from global and ante-hoc from post-hoc explanations, while highlighting perturbation- and gradient-based approaches.
- Taxonomy criteria: The taxonomy uses scope, stage, result, and function as criteria for systematically categorizing XAI techniques.The reviewed literature is categorized by explainability scope and training level or stage.
- Scope: Local explanations address individual predictions, whereas global explanations describe the model more broadly and usually require access to its data, algorithm, and trained parameters.Global explanations are difficult for complex models with many dimensions, parameters, and weights.
- Implementation stage: Ante-hoc methods increase transparency during training, while post-hoc methods explain predictions after models are trained and deployed.Post-hoc methods focus on explaining models that were not designed to be inherently transparent.
- Model dependence: Post-hoc methods include model-specific approaches tied to internal architecture and model-agnostic approaches applicable without knowing that architecture.Model-specific methods can provide detailed structural insights, whereas model-agnostic methods offer flexibility across models.
- Design methodology: Perturbation-based methods modify inputs to measure feature importance, while gradient-based methods use prediction gradients with respect to inputs.Both methodologies can support local and global explanations; gradient methods are also described as scalable to high-dimensional inputs and large models.
- Representative techniques: The survey discusses LIME, SHAP, counterfactual explanations, LRP, CAM, and Integrated Gradients as foundational state-of-the-art techniques.These techniques are presented in a chronological overview and selected for detailed discussion.
4. Detailed Discussions on XAI Techniques
The survey organizes XAI techniques according to their mathematical principles, assumptions, applicability, and limitations, with a separate focus on Transformer explainability because of the architecture’s broad adoption.
- Classification dimensions: XAI techniques differ in mathematical principles, assumptions, applicability, and limitations.The survey classifies widely used methods by perturbation, gradients, and Transformer architecture.
- Transformer explainability: Transformer explainability receives separate treatment because Transformers are dominant across natural language processing, computer vision, time series, and other areas.The survey frames Transformer-based explainability as distinct from perturbation- and gradient-based categories.
4.1. Perturbation-based Techniques
Perturbation-based techniques explain black-box predictions by changing inputs and observing output effects, with LIME, SHAP, and counterfactual explanations presented through their formulations and assumptions.
- Overview: Perturbation-based methods make controlled input changes to produce local or global explanations of black-box models.The section examines LIME, SHAP, and counterfactual explanations, including their mathematical formulations and assumptions.
- LIME: LIME fits an interpretable explanation model near an instance by balancing fidelity, proximity, and explanation complexity.The black-box prediction is approximated locally using perturbed samples and an interpretable representation.
- LIME: LIME is model-agnostic and provides local explanations that identify influential features for individual black-box predictions.These explanations can support inspection and correction of biases associated with particular decisions.
- SHAP: SHAP assigns feature-contribution values by averaging each feature’s contribution across possible feature subsets.Its additive explanation model is described through local accuracy, missingness, and consistency properties.
- SHAP: In complex models, SHAP approximations may produce less accurate explanations, and its outputs reflect biases present in the explained model.SHAP explanations depend on the model’s behavior rather than independently correcting it.
- Counterfactual explanations: Counterfactual explanations generate hypothetical inputs to show how changing data could alter a model’s output.Generation may be computationally expensive, instance-specific, dependent on the data distribution, and sensitive to ethical concerns.
4.2. Gradient-based Techniques
Gradient-based techniques explain predictions by relating model outputs to input features or by propagating relevance through network representations. The section covers saliency maps, LRP, CAM, and Integrated Gradients, alongside their model requirements and limitations.
- Gradient-based techniques: Gradient-based techniques use output gradients with respect to input features to identify influential features and support scalable explanations for high-dimensional data.The section identifies saliency maps, LRP, CAM, and Integrated Gradients as common gradient-based techniques.
- 4.2.1. Saliency Map: Saliency maps rank image pixels by their effect on a class score and visualize the most significant image regions for an individual prediction.For nonlinear models, the class score is locally approximated around the analyzed image using a first-order Taylor expansion.
- 4.2.1. Saliency Map: Saliency maps are sensitive to input noise, limited to gradient-based models, local to individual predictions, and sometimes ambiguous when multiple image features are highlighted.These constraints can lead to incorrect explanations and do not establish a model’s global behavior.
- 4.2.2. LRP: LRP explains input-feature contributions by assigning relevance scores to neurons and propagating those scores backward through network layers under a conservation property.Its basic, epsilon, and gamma rules redistribute relevance according to positive contributions, weak contradictory contributions, or negative contributions and noise.
- 4.2.2. LRP: LRP is described as faithful but requires internal model structure, applies to backpropagation-based models, can be computationally expensive, and lacks standardized implementations.The paper also characterizes LRP as complex to implement and interpret, especially for large and complex models.
- 4.2.3. CAM: CAM maps a predicted class score to the preceding convolutional layer, using activation significance at spatial locations to produce class activation maps for image data.CAM is typically used with CNNs and requires access to the CNN architecture and weights.
- 4.2.4. Integrated Gradients: Integrated Gradients computes gradients along the straight-line path from a baseline input to the instance and satisfies properties including sensitivity and completeness.The method is presented as applicable to differential models and as supporting transparent interpretation of DNN input-output behavior.
4.3. XAI for Transformers
Transformer explanations adapt attribution and attention-based methods to models whose decisions depend on intricate attention mechanisms. Because standard relevance conservation can fail in Transformers, locally linear expansions and revised propagation rules are used to preserve explainability.
- Transformer architecture: Transformers rely on self-attention, multi-head attention, and co-attention, making their prediction and decision processes intricate to explain.These architectures are used across NLP, computer vision, multimodal reasoning, and related applications.
- Attribution methods: Gradient×Input, LRP, Integrated Gradients, and SHAP attribute portions of the predicted score to individual input features under the conservation axiom.Relevance scores indicate how much each feature at each layer contributes to the final prediction and decision.
- Gradient×Input: Gradient×Input computes attribution as each neuron representation multiplied by the gradient of the model output with respect to that representation.For input xi and output yj, the attributions are R(xi) = xi · (∂f/∂xi) and R(yj) = yj · (∂f/∂yj).
- Relevance propagation: Gradient propagation uses the chain rule, and inserting its rule into Gradient×Input converts gradient propagation into equivalent relevance propagation.This embeds Gradient×Input within the LRP framework for explaining Transformers.
- Propagation challenges: Transformers break the conservation rule, so attention heads and LayerNorm require ad-hoc propagation rules to restore conservation through locally linear expansions.Attention gating terms are treated as fixed weights of a locally linear layer, enabling the canonical LRP rule for linear layers.
- Improved explainability: Attention rollouts, generic attention, and conservative propagation methods provide empirical evidence that Transformer explainability can be improved.These methods aggregate or revise attention and relevance information to obtain more informative explanations.
4.4. Explainability in Reinforcement Learning
Reinforcement learning is applied in safety-critical domains, including autonomous vehicles, healthcare, and energy systems, where explaining learned decisions is especially important. XAI methods for RL support trust, safety assurance, compliance, debugging, human collaboration, accountability, and adoption.
- Applications and motivation: Reinforcement learning is used in safety-critical domains such as autonomous vehicles, healthcare, and energy systems.In autonomous vehicles, RL refines adaptive cruise control and lane-keeping by learning from simulated traffic scenarios.
- Explainability methods: RL explainability methods include saliency maps, counterfactual explanations, policy distillation, attention mechanisms, human-in-the-loop systems, query systems, and natural-language explanations.The methods address different ways of interpreting learned policies and decisions.
- Why explainability matters: Explainability in RL supports trust, safety assurance, regulatory compliance, ethical decision-making, model debugging, human-AI interaction, accountability, and model adoption.These needs are particularly pronounced when RL systems operate in safety-critical settings.
4.5. Summary
The survey organizes XAI techniques across multiple methodological dimensions and adds dedicated coverage of Transformers and reinforcement learning. It summarizes the reviewed techniques in a comparative table.
- Taxonomy: XAI techniques are categorized as local or global, post-hoc or ante-hoc, model-specific or model-agnostic, and perturbation or gradient based.The categories describe complementary ways to organize methods by explanation scope, timing, model dependence, and technical basis.
- Specialized models: The survey includes dedicated subsections on reinforcement learning and Transformers because of their popularity and impact across deep-learning applications.These sections extend the survey beyond general XAI method categories.
- Comparison: Table 2 summarizes the reviewed XAI techniques together with their methods, advantages, and disadvantages.The table provides a comparative view of the techniques discussed in the survey.
5. XAI Techniques in Application Areas
XAI application methods span NLP, neural networks, fine-tuned Transformers, and prompt-based explanations. Their effectiveness and interpretation depend on data preparation, model structure, perturbations, gradients, attention, and contextual demonstrations.
- Application areas: XAI techniques help identify model errors and biases while explaining predictions in NLP, computer vision, and time-series applications.These capabilities are linked to improving model reliability, accuracy, transparency, and trust.
- Natural language processing: NLP preprocessing and text-data quality strongly affect the success of XAI techniques and the reliability of transparent language models.Text data are often not inherently clean, making preprocessing and training essential for accurate language-model results.
- Perturbation-based explanations: LIME perturbs words by randomly removing or replacing features, then trains a simpler model to assign feature-importance scores to the original text.The scores indicate each word’s contribution to the black-box prediction.
- SHAP: SHAP estimates each word’s importance by comparing expected model outputs when the word is included versus excluded across perturbations.The resulting Shapley values are used in tasks including classification, sentiment analysis, topic modeling, entity recognition, and language generation.
- LRP: LRP propagates a model’s output backward through layers processing word embeddings to calculate normalized relevance scores for words.The method connects word-level explanations to the neural network’s internal relevance propagation.
- Gradient-based explanations: Integrated Gradients integrates prediction gradients along a path from a neutral or zero-embedding baseline to the original text to assign word attributions.It produces a heatmap of important words and is used for text classification, sentiment analysis, and summarization.
- Prompt-based explainability: Prompt-based explanations include Chain of Thought, In-Context Learning, and interactive prompts that use intermediate reasoning steps, examples, or instructions.These approaches investigate how contextual prompts influence model behavior and task outputs.
- Prompt analysis: Gradient-based and perturbation-based studies analyze token saliency and altered prompts to examine how reasoning steps and prompt elements affect LLM outputs.Counterfactual prompts change critical prompt elements, while backpropagation estimates token influence.
5.3. Explainability in Time Series
Time-series XAI applies visualization, perturbation, and attribution methods to explain predictions while addressing temporal dependencies, preprocessing requirements, and model sensitivity.
- Time-series forecasting supports decisions in business, finance, meteorology, and medicine by predicting future target values from ordered observations.
- Saliency maps: Saliency-map generation uses normalization, reshaping, windowing, and padding to represent time steps and features for model analysis.
- Saliency maps: Saliency maps can identify temporal feature importance, but their accuracy is challenged by time-dependent importance and sensitivity to model architecture.
- CAM: CAM can adapt to time series by treating time steps as image channels, stacking fixed windows before generating weighted visual explanations.
- TSViz: TSViz is a model-agnostic, post-hoc, human-in-the-loop toolkit that uses dimensionality reduction and multiple visualizations to support expert analysis.
- LIME: LIME explains temporal predictions by fitting local interpretable models to time segments and perturbing influential steps, such as prices, volume, and news.
- SHAP: SHAP provides global time-series insights by calculating feature contributions for perturbed versions of an input sequence.
5.4. Explainability in Healthcare
Healthcare XAI targets trust and understanding in high-risk medical decisions by explaining predictions across complex data, treatments, clinical support, and related governance concerns.
- Healthcare requires XAI because medical decisions involve high risks and complex data, creating a need for trust between patients, physicians, and deployed AI systems.
- Medical diagnosis: XAI supports medical diagnosis by transparently interpreting outputs and identifying influential features across imaging, records, genomic, laboratory, wearable, and behavioral data.
- Patient treatment: XAI helps personalize treatment plans by explaining why a specific medicine is suggested from a patient’s medical data and diagnosis.
- Drug discovery and development: In drug discovery, XAI explains relationships between molecular structures and biological effects.
- Clinical decision support: Clinical decision-support systems provide interpretable explanations that help clinicians understand complex cases and influential features.
- Legal and ethical considerations: Healthcare XAI must address transparency, accountability, fairness, bias mitigation, privacy, confidentiality, and ethical guidance for sensitive medical information.
- Autonomous vehicles: In autonomous vehicles, XAI techniques address diverse visual, spatial, temporal, audio, environmental, communication, genetic, and textual information.
- Autonomous vehicles: XAI can strengthen trust in autonomous vehicles and support human-AI collaboration by enabling users to reject predictions when explanations reveal flawed reasoning.
6. XAI Evaluation Methods
XAI evaluation combines human-centered assessment of user needs with computer-centered measurement of technical properties such as fidelity, consistency, robustness, efficiency, and sufficiency.
- XAI evaluation is needed to measure explanation quality, usefulness, and trustworthiness across sensitive applications.
- Human-centered approach: The human-centered approach evaluates comprehensibility, trust, satisfaction, decision support, usability, feedback, emotional response, error reduction, productivity, and cognitive load.
- Computer-centered approach: The computer-centered approach evaluates explanations using technical, objective, and quantifiable metrics, including fidelity, consistency, robustness, and efficiency.
- Fidelity: Fidelity measures how closely an explanation represents the model’s actual decision process.
- Consistency: Consistency assesses explanation stability and coherence across repeated model runs, including variance-based stability and feature-distribution uniformity.
- Robustness: Robustness measures explanation resilience to input perturbations, adversarial attacks, model updates, and changes across systems or platforms.
- Efficiency: Efficiency concerns the computational resources, time, and scalability required to generate explanations as input volume grows.
- Sufficiency: Sufficiency evaluates whether a rationale alone preserves model confidence relative to the full input, with smaller confidence differences indicating greater sufficiency.
7. Future Research Direction
Future XAI research must address complex-model limitations, competing performance and interpretability goals, evaluation diversity, ethical risks, multimodality, real-time constraints, and cultural diversity.
- Existing XAI challenges span design objectives, applications, standardization, model complexity, security, and evaluation metrics.
- Complexity: Highly complex models can reduce XAI effectiveness, motivating model simplification, hybrid models, and interactive explanations without substantial accuracy loss.
- Building models with explanations: Building accurate models with explanations remains technically, ethically, and legally complex, especially for safety-critical applications.
- Performance and interpretability: Simplifying models may reduce accuracy, so future systems must balance performance requirements with interpretability and explainability.
- Evaluation metrics: Different XAI design objectives make it difficult to apply identical evaluation metrics across systems emphasizing interpretability, accuracy, fairness, robustness, or transparency.
- Security and privacy: Explanations can create privacy and security risks, including information leakage, model inversion, adversarial attacks, and compromised explanation integrity.
- Multimodal AI: Multimodal XAI must explain modality fusion and intermodal relationships despite heterogeneity, scalability, and high dimensionality challenges.
- Real-time explanation: Real-time explanations in safety-critical systems are constrained by large models, continuous multimodal data streams, and latency requirements.
8. Conclusion
This survey organizes post-hoc XAI research by terminology, beneficiaries, applications, training stage, scope, and design methodology.
- The survey focuses on post-hoc model explainability across current XAI literature.
- It reviews XAI definitions, needs, beneficiaries, techniques, and applications across various fields.
- The taxonomy classifies XAI methods by training stage, scope, and design methodology.
- Ante-hoc and post-hoc techniques provide two perspectives on explaining AI systems’ inner workings.
- Perturbation-based and gradient-based approaches are identified as common algorithmic design methodologies for XAI.