PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Explainable Artificial Intelligence for Cybersecurity

Gaith Rjoub, Jamal Bentahar, Omar Abdel Wahab, Rabeb Mizouni, Alyssa Song, Robin Cohen, Hadi Otrok, Azzam Mourad

arXiv:2303.12942v2cs.CRcs.AIcs.NI

TL;DR

Prior XAI-cybersecurity surveys often cover only subsets of techniques, while widespread real-world use remains distant. This survey classifies cybersecurity research on XAI, compares approaches, and identifies criteria and future-work considerations.

  • Problem

    Existing XAI-cybersecurity surveys often focus on a specific technique or subset of techniques rather than covering the field comprehensively.

    Source: §IV · p. 6

  • Method

    The survey analyzes selected XAI-cybersecurity research through a classification of cybersecurity categories and a comparative synthesis of approaches.

    Source: §V · p. 7, §XI · p. 29

  • Results

    The survey provides a comparative summary of cybersecurity XAI techniques and proposes primary criteria for successful solutions.

    Source: §XI · p. 29

  • Takeaways & Limitations

    The analysis identifies key considerations for future work aimed at achieving the proposed criteria for cybersecurity XAI solutions.

    Source: §XI · p. 29

    Widespread real-world use of XAI for network and system security remains a long way off.

    Source: Motivation and Contributions · p. 2

Abstract

from arXiv · show

The black-box nature of artificial intelligence (AI) models has been the source of many concerns in their use for critical applications. Explainable Artificial Intelligence (XAI) is a rapidly growing research field that aims to create machine learning models that can provide clear and interpretable explanations for their decisions and actions. In the field of network cybersecurity, XAI has the potential to revolutionize the way we approach network security by enabling us to better understand the behavior of cyber threats and to design more effective defenses. In this survey, we review the state of the art in XAI for cybersecurity in network systems and explore the various approaches that have been proposed to address this important problem. The review follows a systematic classification of network-driven cybersecurity threats and issues. We discuss the challenges and limitations of current XAI methods in the context of cybersecurity and outline promising directions for future research.

I. Introduction

Black-box AI makes its decisions difficult to understand and explain, prompting questions about responsibility and trust. The survey frames XAI for cybersecurity as a response to these concerns and distinguishes related concepts used throughout the paper.

I. Introduction

  • AI methods can be difficult to understand and explain, raising questions about responsibility, explainability of outputs and internal structure, and trust in decisions.Source: §I · p. 1
  • XAI combines machine-learning techniques with explanations to help users understand, trust, and manage AI systems and their decisions or actions.Source: §I · p. 1

Motivation and Contributions

  • The survey reviews XAI approaches applicable to cybersecurity, analyzes their potential benefits, classifies cybersecurity sub-domains, and discusses future directions and ethical challenges.Source: Motivation and Contributions · p. 2
  • Widespread real-world use of XAI for network and system security remains a long way off, and the paper emphasizes identifying use cases and challenges.Source: Motivation and Contributions · p. 2

II. Preliminaries

  • The paper distinguishes explainability, interpretability, trustworthiness, interactivity, stability, and robustness as related but different concepts.Explainability concerns explaining decisions despite opacity, while interpretability concerns insights into decision-making,; trustworthiness can be viewed as a consequence of both.Source: §II · pp. 2–3
  • Stability concerns maintaining intended functionality across conditions, whereas robustness concerns continuing to perform well under input changes.Source: §II · p. 3

III. Research Methodology

The review searches publication databases and reference lists within a defined scope, then reports publication-type and subject-area breakdowns for XAI-based cybersecurity literature.

III. Research Methodology

  • The review excluded work outside AI or cybersecurity, non-peer-reviewed articles and technical reports, and studies focused only on transparency rather than explanation.Source: §III · p. 3
  • The review searched Scopus, IEEE, Springer, Science Direct, and Google Scholar for peer-reviewed journal articles, book chapters, and conference proceedings, then checked relevant reference lists.Source: §III · p. 4
  • The methodology reports percentage breakdowns of the XAI-based cybersecurity literature by publication subject area and publication type.Source: §III · p. 4, Figure 1 · p. 4, Figure 2 · p. 4
  • 52.9% of XAI-based security articles were conference papers and 39.6% were journal articles; review papers accounted for 4.7% and book chapters 2.8%.Source: §III · p. 4, Figure 2 · p. 4

IV. Literature Review

This section presents a systematic review of XAI for cybersecurity, using bibliometric mapping and discussion across six cybersecurity classes. It maps XAI model classes, security categories, and assessment criteria.

IV. Literature Review

  • Prior XAI cybersecurity surveys often cover only subsets of techniques, rely on potentially biased self-reports, omit stakeholder diversity, and lack standardized evaluation metrics.These limitations make it difficult to compare studies or draw definitive conclusions about technique effectiveness, particularly in network security.Source: §IV · p. 6
  • The survey targets cybersecurity applications and reviews challenges, frameworks, datasets, and successful systems, while mapping eight XAI model classes to six cybersecurity classes and nine assessment criteria.The mappings organize which models suit particular security classes and how solutions can be evaluated against cybersecurity criteria.Source: §IV · pp. 6–7

V. Classification of Scientific Articles on Explainability of AI Models for Cybersecurity

  • The survey identifies cybersecurity categories through analysis of the selected articles and presents its classification as systematic, multi-level, and based on clear criteria.Its comparisons also identify gaps in prior classifications, including absent sub-classification of challenges and unclassified techniques proposed to address them.Source: §V · pp. 7, 9, 10
  • Compared with, which uses a narrative approach, this survey classifies cybersecurity XAI methods and emphasizes recent developments, challenges, limitations, and future directions.It also covers a wider range of classification methods and techniques than the black-box, glass-box, gray-box, and white-box overview described in the comparison.Source: §V · pp. 8–9
  • Against, this survey provides a comprehensive classification of XAI models, techniques, and evaluation metrics across network and system security rather than primarily summarizing individual studies and experiments.It also addresses challenges and limitations more prescriptively, including challenges not discussed in [26], such as model-specific and model-agnostic explanations.Source: §V · p. 9
  • The review uses bibliometric analysis to map author keywords and annual scientific production, and plans state-of-the-art coverage of six cybersecurity classes with criteria for future XAI solutions.Keyword patterns place XAI in cybersecurity alongside trust, anomaly detection, and privacy; the cybersecurity map shows more distinct clusters, hypothesized to reflect diverse uses of XAI across cybersecurity areas.Source: §V · p. 11, Figure 3 · p. 12
  • XAI publications increased significantly over the past three years, while the 2023 cybersecurity publication count is incomplete because of the search date.The actual number of 2023 publications is likely higher.Source: §V · p. 11
  • The categories may be incomplete, context-dependent, outdated, or insufficient to capture the complexity and changing behavior of explainable AI models in cybersecurity.Their relevance and usefulness should be assessed for each specific context as the field evolves.Source: §V · pp. 11–12

VI. Explainability via Mathematical Models

The section introduces XAI’s mathematical concepts, contrasts XAI with interpretability, transparency, and robustness, and relates it to cybersecurity concepts. It also uses consistent notation while introducing SHAP, LIME, and LRP.

VI. Explainability via Mathematical Models

  • The section relates XAI to trust, intrusion detection, authentication, intrusion prevention, access control, and privacy.It also discusses similarities and differences between XAI, interpretability, transparency, and robustness.Source: §VI · p. 12
  • The survey uses a common notation set to make its mathematical equations and algorithms consistent and easier to read.Its equations may differ from those in the original publications.Source: §VI · p. 12
  • The section begins its explanation of XAI techniques with SHAP, LIME, and LRP.These are Shapley value, Local Interpretable Model-Agnostic Explanations, and Layer-wise Relevance Propagation.Source: §VI · p. 12

A. Shapley Value (SHAP)

SHAP uses Shapley values to quantify feature contributions to model predictions, supporting local and global explanations. Its linear approximation can be computationally demanding because it examines feature subsets.

A. Shapley Value (SHAP)

  • SHAP supports both global and local explanations and is described as satisfying consistency, efficiency, and local-accuracy properties.In a linear model, local functions θk(Xi) are Shapley values.Source: A. Shapley Value (SHAP) · pp. 14–15
  • Shapley values quantify each feature’s contribution to a model prediction by averaging its influence across possible classifications.The method also considers alternative feature subsets when estimating a feature’s impact.Source: A. Shapley Value (SHAP) · pp. 13–15
  • The additive linear formulation represents a model through feature components, with θ0 as the intercept and K as the number of explanatory variables.Its local functions are Shapley values.Source: A. Shapley Value (SHAP) · pp. 14–15
  • Linear approximations regressing responses on individual Shapley values can produce statistically testable local explanations but may be heavily parameterized and computationally demanding.The equation approximation may require examining subsets of the K explanatory variables.Source: A. Shapley Value (SHAP) · p. 15
  • The feature-significance procedure retrains models on feature subsets and compares predictions with and without a feature to estimate its importance.It averages the effects across alternative subsets.Source: A. Shapley Value (SHAP) · p. 15

B. Local Interpretable Model-Agnostic Explanations (LIME)

LIME explains a model’s prediction with an interpretable approximation around a selected instance. It seeks a locally faithful explanation while using proximity to define the relevant neighborhood.

B. Local Interpretable Model-Agnostic Explanations (LIME)

  • LIME is a model-agnostic approach for explaining model predictions or decisions and has been used in cybersecurity and other domains.The cited applications include healthcare, agriculture, and cybersecurity.Source: B. Local Interpretable Model-Agnostic… · p. 15
  • LIME selects an interpretable model g to approximate the black-box model f in the locality around an instance.The explanation model may be a decision tree, linear model, or falling rule list.Source: B. Local Interpretable Model-Agnostic… · p. 15
  • The proximity measure πx defines the neighborhood around x, while L(f, g, πx) measures how unfaithfully g approximates f there.The stated goal is to minimize this local loss and obtain an interpretable approximation.Source: B. Local Interpretable Model-Agnostic… · p. 15

C. Layer-wise Relevance Propagation (LRP)

Layer-wise Relevance Propagation explains a classifier’s prediction for a particular input by assigning relevance to input components through the trained network. Relevance is propagated backward, with a conservation condition, until input features are reached.

C. Layer-wise Relevance Propagation (LRP)

  • LRP explains a classifier’s prediction for a given data point by assigning relevance values to input components using the trained model’s topology.It has been applied to text, images, videos, and anomaly detection.Source: C. Layer-wise Relevance Propagation… · p. 16
  • LRP assigns relevance scores to neurons via backpropagation, redistributing each upper-layer score to neurons in the immediately lower layer.The relevance scores indicate each neuron’s contribution to the network output.Source: C. Layer-wise Relevance Propagation… · p. 16
  • The conservation property requires the total relevance score in each layer to equal the output.The propagation rule uses the contribution of neuron j to neuron h’s relevance and terminates at the input features.Source: C. Layer-wise Relevance Propagation… · p. 16
  • Cybersecurity use requires careful suitability evaluation because techniques may trade accuracy for interpretability or yield biased, misleading explanations when they miss complex variable interactions.The passage links misleading explanations to false-positive and false-negative attacks.Source: C. Layer-wise Relevance Propagation… · p. 16

VII. XAI Methods Classification

The paper classifies XAI methods by explanation level, implementation level, and model dependence. The categories distinguish what a method explains, when it explains a model, and whether it is tied to a particular model.

VII. XAI Methods Classification

  • Explanation level distinguishes global methods, which explain a whole model, from local methods, which explain a single instance.Some methods use global and local techniques together.Source: §VII · pp. 16–17
  • Implementation level separates post-hoc explanations made after training from pre-hoc explanations made before training.The xAI-GAN example combines both implementation levels.Source: §VII · p. 17
  • Model dependence distinguishes model-specific explainers from model-agnostic explainers designed to work regardless of model or domain knowledge.The passage notes that model-agnostic explainers are often used because of their flexibility.Source: §VII · p. 17

VIII. XAI-based Cybersecurity

The survey identifies a gap in XAI coverage of network and system security and reviews the growing use of AI/ML in cybersecurity. It frames the review around assessing work in light of a cybersecurity classification.

VIII. XAI-based Cybersecurity

  • Few surveys address XAI for cybersecurity, and none focus on network and system security.The survey positions its review as addressing that subfield-specific gap.Source: §VIII · p. 17
  • Growing use of big data, cloud computing, and IoT increases the need for AI/ML to help protect networks and data.The authors note potential uses in detecting and remediating advanced cyberattacks.Source: §VIII · p. 17
  • The survey assesses cybersecurity AI/ML proposals against its classification, amid increasing use of AI for diagnosis, risk assessment, and incident response.The passage also notes a rapidly changing threat landscape and the use of AI to find trends in large datasets.Source: §VIII · p. 17

A. Privacy

The section surveys XAI work across privacy, trust, and intrusion detection, describing privacy-protective approaches and explainability methods used to interpret cybersecurity models. It also notes that many AI-based intrusion-detection solutions lack explainability.

A. Privacy

  • Privacy-focused XAI can identify risks in data processing and uses differential privacy, federated learning, and homomorphic encryption to protect sensitive data while supporting prediction.The cited approaches aim to preserve learning about cyber threats and vulnerabilities.Source: A. Privacy · pp. 17–18
  • LEM [84] approximates complex models with locally linear regions and perturbs local classifiers with differential privacy to provide private explanations for classification decisions.The passage reports high classification accuracy on several image benchmark datasets.Source: A. Privacy · p. 18
  • The default-privacy-settings model emphasizes feature selection, evaluation metrics, and step-by-step interpretability to help users understand predictions.The authors describe these design choices as improvements to accuracy, robustness, reliability, and transparency.Source: A. Privacy · pp. 18–19

B. Trust and Reputation

  • Trust is framed through reliability, explanations, privacy, and lack of bias; TRUST XAI uses factor analysis, mutual information, and multimodal Gaussian distributions in an IIoT network-security case study.The passage also describes XAI applications to misbehaving-vehicle detection and IoB-based behavior change [95].Source: B. Trust and Reputation · p. 19

C. Intrusion Detection

  • XAI can expose decision processes in intrusion detection, helping analysts assess whether alerts reflect genuine threats or false positives.One approach learns normal behavior and explains deviations; another explains an IDS's output.Source: C. Intrusion Detection · p. 19
  • Many existing AI-based intrusion-detection solutions lack explainability, leaving insufficient evidence about how they identify attacks and which features drive a decision.The passage says AI can assist, not replace, network security guards.Source: C. Intrusion Detection · p. 19
  • Intrusion-detection examples include SHAP, LIME, and LSTM explanations for anomalous Kubernetes pods [97], and combined local/global Shapley explanations for IDS classifiers.Other examples cover supply-chain delivery-delay prediction and a decision-tree interpretable autoencoder for categorical data.Source: C. Intrusion Detection · p. 20

D. Intrusion Prevention

The survey reviews XAI applications to intrusion prevention, access control, and authentication, then sets out criteria and research directions for future cybersecurity solutions. It emphasizes transparency, interpretability, adaptability, and accountability alongside practical constraints.

D. Intrusion Prevention

  • For intrusion prevention, XAI can explain alerts, help distinguish genuine threats from false positives, and support threat prioritization and risk assessment.Examples include explainable Android malware detection and adversarial feature changes that explain IDS misclassifications.Source: D. Intrusion Prevention · p. 20

E. Access Control

  • Access-control applications include classifying attention states into four risk categories for insider-threat management and pixel-level explanations for face-morphing detection.The latter framework evaluates explanation quality by removing detected artifacts and observing decision changes.Source: E. Access Control · pp. 20–21

F. Authentication

  • Authentication examples include XAI-based fingerprint and biometric mimicry attacks [105] and continuous, non-intrusive user authentication using behavioral biometrics.The section also compares cybersecurity XAI studies by approach, explanation level, and cybersecurity class in Table III.Source: F. Authentication · p. 21

IX. Desirable Criteria for Future Solutions

  • Transparency requires disclosing system methods and decision processes, while explainability may trade off against predictive performance and exists by degree.The section also calls for openness about capabilities, limitations, impacts, operating modes, and data sources.Source: §IX · pp. 21–22
  • Understandability and cogency require explanations that domain experts and non-experts can follow, including why intrusion-detection events are judged benign or malicious.Understanding errors is presented as necessary for experts to modify a system to correct them.Source: §IX · p. 22
  • The proposed criteria also call for broad applicability, autonomy, openness, reasonableness, adaptation to changing environments, and responsibility for system actions.The authors state that systems should balance effectiveness with protection of people, privacy, and the environment.Source: §IX · pp. 22–23
  • Future work should improve insight into XAI operation and outputs, investigate cost and security/privacy interdependencies, and examine self-explanation for non-experts.The survey also calls for models that learn from their environments and sufficient access to methods and predictions for evaluation and consistency checks.Source: §IX · p. 24
  • The criteria are not exhaustive, and the authors recommend updating them as XAI technology and cybersecurity threats evolve.Figs. 8 and 9 map cybersecurity classes to XAI models, techniques, and selected desirable criteria.Source: §IX · pp. 24–25

X. Future Directions

The paper surveys key concerns for effective XAI in cybersecurity and characterizes central methods to consider and evaluate. It then identifies critical future research paths for the field.

X. Future Directions

  • The paper surveys key concerns for effective XAI in cybersecurity applications.The survey focuses on concerns relevant to achieving effective XAI in this domain.Source: §X · p. 25
  • It provides characterizations of central methods that should be considered and evaluated.Source: §X · p. 25
  • The section identifies research paths the authors consider critical for successful explainable AI in cybersecurity.Source: §X · p. 25

A. Risk of Machine Interpretation in Cybersecurity

The section frames machine interpretation in cybersecurity around threat-classification reliability and usability, then identifies transparency and uncertainty as challenges for XAI methods.

A. Risk of Machine Interpretation in Cybersecurity

  • Machine interpretation must classify cybersecurity threats accurately while minimizing false positives, and the section asks how to improve its methods, evaluation, usability, and deployment.The stated research questions also call for cybersecurity-suited evaluation metrics and a better understanding of methods’ strengths and limitations.Source: A. Risk of Machine Interpretation in… · p. 25

B. Transparency of XAI Methods in Cybersecurity

  • XAI faces an accuracy–transparency trade-off, with possible responses including risk-dependent transparency levels and sensitivity analysis.The section presents these as potential approaches to improve transparency without sacrificing accuracy.Source: B. Transparency of XAI Methods in… · pp. 25–26

C. Uncertainty Handling in XAI for Cybersecurity

  • Cybersecurity’s uncertain and dynamic conditions motivate XAI methods that can reason about uncertainty using probabilistic models or robust optimization.Bayesian networks are given as an example of probabilistic models.Source: C. Uncertainty Handling in XAI for… · p. 26

D. Secure and Efficient Algorithms for Distributed XAI Training

The section describes distributed explainable training, cybersecurity-specific evaluation needs, and future research directions. The conclusion presents the survey’s comparative synthesis and criteria for XAI cybersecurity solutions.

D. Secure and Efficient Algorithms for Distributed XAI Training

  • Federated XAI training addresses sensitive, distributed data without sharing it, but requires secure, efficient algorithms and explanations for both local models and their aggregation.Feature-importance analysis and saliency maps are suggested for local models, alongside transparent aggregation techniques.Source: D. Secure and Efficient Algorithms for… · pp. 27–29

E. Cybersecurity-Specific Evaluation Metrics

  • Existing metrics such as accuracy may poorly fit cybersecurity’s imbalanced, high-dimensional tasks and fail to reflect the differing consequences of false positives and false negatives.The section contrasts unnecessary alarms with missed attacks that can cause serious damage or loss of life.Source: E. Cybersecurity-Specific Evaluation… · p. 29
  • Future research should investigate XAI for sophisticated adversarial attacks and APTs, real-time data streams, more actionable security insights, and decentralized environments such as IoT.The section notes that heterogeneous devices in decentralized environments challenge traditional cybersecurity approaches.Source: E. Cybersecurity-Specific Evaluation… · p. 29

XI. CONCLUSION

  • The survey clarifies XAI’s cybersecurity challenges, defines key terms, distinguishes its scope from prior reviews, and explains its choices for characterizing the field.Source: §XI · p. 29
  • It compares major XAI techniques for cybersecurity systems, proposes primary criteria for successful solutions, and discusses considerations for future work.Source: §XI · p. 29
  • The conclusion calls for continued work on AI trust through XAI, informed by the methods and challenges specific to cybersecurity.Source: §XI · pp. 29–30
Loading 2303.12942v2…