PaperlayerView exact revision on arXiv

Source-linked AI summary

TRUST XAI: Model-Agnostic Explanations for AI With a Case Study on IIoT Security

Maede Zolanvari, Zebo Yang, Khaled Khan, Raj Jain, Nader Meskin

arXiv:2205.01232v1cs.AIcs.ITmath.IT

TL;DR

Opaque AI decisions and the difficulty of designing fast, accurate explainers for numerical applications motivate TRUST. It models AI outputs statistically and, in the IIoT cybersecurity case study, reports 98% accuracy for explanations of unseen samples.

  • Problem

    AI’s opaque decisions complicate trust, and designing XAI that is both fast and accurate remains challenging for numerical applications.

    Source: §I · pp. 1–2

  • Method

    TRUST is a model-agnostic statistical explainer that selects influential representatives and models their output distributions to explain new samples.

    Source: §I · p. 2, B. Determining the Representatives · p. 5, C. Density Estimation · p. 6

  • Results

    98% accuracy was reported for TRUST’s explanations of the primary AI’s outputs on unseen data.

    Source: D. Results · p. 9

  • Takeaways & Limitations

    The IIoT cybersecurity case study presents TRUST as a fast explainer for numerical applications, with the paper reporting that it is almost 25 times faster than LIME.

    Source: §I · p. 2, D. Results · p. 10

    TRUST may overfit its training set, and its output may be unreliable when the Gaussian assumption or stable data distributions do not hold; independent samples are also assumed.

    Source: §VI · p. 12

Abstract

from arXiv · show

Despite AI's significant growth, its "black box" nature creates challenges in generating adequate trust. Thus, it is seldom utilized as a standalone unit in IoT high-risk applications, such as critical industrial infrastructures, medical systems, and financial applications, etc. Explainable AI (XAI) has emerged to help with this problem. However, designing appropriately fast and accurate XAI is still challenging, especially in numerical applications. Here, we propose a universal XAI model named Transparency Relying Upon Statistical Theory (TRUST), which is model-agnostic, high-performing, and suitable for numerical applications. Simply put, TRUST XAI models the statistical behavior of the AI's outputs in an AI-based system. Factor analysis is used to transform the input features into a new set of latent variables. We use mutual information to rank these variables and pick only the most influential ones on the AI's outputs and call them "representatives" of the classes. Then we use multi-modal Gaussian distributions to determine the likelihood of any new sample belonging to each class. We demonstrate the effectiveness of TRUST in a case study on cybersecurity of the industrial Internet of things (IIoT) using three different cybersecurity datasets. As IIoT is a prominent application that deals with numerical data. The results show that TRUST XAI provides explanations for new random samples with an average success rate of 98%. Compared with LIME, a popular XAI model, TRUST is shown to be superior in the context of performance, speed, and the method of explainability. In the end, we also show how TRUST is explained to the user.

I. INTRODUCTION

The introduction frames opaque AI decisions as a barrier to trust and motivates a model-agnostic statistical explainer for numerical applications. The related work highlights trade-offs in existing explainers, including limits in speed and generalizability.

I. INTRODUCTION

  • Opaque AI decisions are difficult for users and leaders to understand, while AI-powered IoT automation requires trust without a human in the loop.The introduction argues that these conditions make transparency important for AI-based IoT systems.Source: §I · p. 1
  • The paper argues that statistical models can simplify explanations because they are more verifiable than the AI or ML models commonly used as explainers.It notes that XAI users are generally assumed to understand the explainer’s mathematics, rather than receiving lay explanations.Source: §I · p. 2
  • TRUST is presented as a model-agnostic surrogate that explains numerical-data predictions without restricting or sacrificing the primary AI model.The paper identifies network data for IoT and security systems as an area deficient in XAI.Source: §I · p. 2
  • TRUST uses primary-model features and outputs; its core is built once, and only its final two steps are needed to explain new instances.The paper evaluates it on three cybersecurity datasets and says it explains predicted labels in a few milliseconds.Source: §I · p. 2

II. RELATED WORK

  • Prior work spans feature-importance methods, tree- or gradient-specific approaches, and LIME’s per-instance local surrogates, with reported limits in real-time speed or model generality.The cited discussion says SHAP is computationally heavy, LRP does not generalize beyond DNNs, and a tree-based method applies only to tree learners.Source: §II · p. 3

III. PROPOSED TRUST EXPLAINER

TRUST formalizes explanations as class representatives derived from training-set features. Its formulation assumes representative independence and that representative observations within each class can be approximated by multimodal Gaussian distributions.

III. PROPOSED TRUST EXPLAINER

  • TRUST explains the primary AI’s labeling rationale by applying statistical techniques to its outputs and producing representative vectors from training-set features.The paper describes representatives as a means of formalizing the primary AI’s task.Source: §III · p. 3, A. Problem Formalization · p. 3

A. Problem Formalization

  • The explainer uses the primary AI’s training data to represent data characteristics, so significant data changes that require retraining also require rebuilding TRUST’s core.This is stated as Assumption 1.Source: A. Problem Formalization · p. 3
  • Representatives are assumed mutually independent, allowing their joint probability to be calculated as the product of individual probabilities.The method selects representatives so their correlations are close to zero.Source: A. Problem Formalization · p. 3
  • Within each class, observations for each representative are assumed to be approximated by a one-dimensional multimodal Gaussian distribution.The passage allows one or more modes, with a unimodal distribution represented by one mode.Source: A. Problem Formalization · p. 4

B. Determining the Representatives

TRUST transforms features into factors and selects representatives according to their relationship with the class labels, rather than how much data variance they explain. The number of representatives trades computation time against performance.

B. Determining the Representatives

  • Factor analysis of mixed data (FAMD) handles quantitative and qualitative variables together, avoiding information-loss approaches such as neglecting or encoding categorical data,.The paper describes FAMD as the general form of factor analysis and the most accurate form for mixed datasets.Source: B. Determining the Representatives · p. 4
  • For each class, factor analysis builds a feature relationship matrix and uses singular value decomposition to produce orthogonal factors that remove interdependencies,.The relationship measures described include Pearson correlation for quantitative pairs and chi-square for qualitative pairs.Source: B. Determining the Representatives · p. 4
  • A factor’s explained variance does not establish its ability to distinguish labels, so variance percentages alone cannot rank representative importance.The example shows that a feature with little within-class variation can still distinguish class labels.Source: B. Determining the Representatives · pp. 4–5
  • Mutual information (MI) ranks factors by their dependence on the class labels, and the highest-scoring factors become the representatives.The paper bins factor values when calculating MI because the factors are not discrete.Source: B. Determining the Representatives · p. 5
  • Choosing k representatives trades speed against performance: fewer factors support real-time use, while more factors can improve accuracy at greater computational cost.The paper notes that density and mode computations take longer as the number of representatives increases.Source: B. Determining the Representatives · p. 5

C. Density Estimation

TRUST fits one-dimensional multimodal Gaussian distributions to representative values for each class. It explains a new sample by comparing its representatives’ likelihoods across classes and selecting the class with the highest weighted total log-likelihood.

C. Density Estimation

  • TRUST fits one-dimensional multimodal Gaussian (MMG) distributions to the values of each representative, producing k × C distributions across k representatives and C classes.The expectation-maximization algorithm fits the representative values to a distribution.Source: C. Density Estimation · pp. 5–6
  • A new sample is projected into factor space and reduced to its representative values, whose class-conditional likelihoods are estimated from the fitted density models.The explanation process uses the representative values rather than all original feature values.Source: C. Density Estimation · p. 6
  • Log transformation turns the product of representative likelihoods into additions and avoids exponentials and arithmetic underflow or overflow.The paper also says the transformation reduces or removes skewness in the data.Source: C. Density Estimation · p. 6
  • TRUST combines representative log-likelihoods using normalized MI importance weights and explains the primary AI’s prediction with the class having the maximum total log-likelihood.The method estimates each class from its corresponding representative distributions rather than fitting one joint multidimensional class distribution.Source: C. Density Estimation · pp. 6–7

D. Selecting the Number of Modes

TRUST selects a mode count for each representative to fit its density distribution. The paper proposes a faster search strategy because exhaustive search grows exponentially with the number of classes.

D. Selecting the Number of Modes

  • Grid search selects each representative’s number of Gaussian modes by maximizing the probability of correct predictions, scored with Matthew’s correlation coefficient (MCC).The chosen mode count affects how well the fitted MMG distribution estimates density.Source: D. Selecting the Number of Modes · p. 7
  • Because exhaustive mode search grows exponentially with the number of classes, the faster sub-zone search may return an approximate mode count while achieving accuracy in the order of 99% in the experiments.The paper reports that reducing sub-zone size increases accuracy.Source: D. Selecting the Number of Modes · p. 7
  • MCC is used instead of accuracy because the authors consider it better suited to imbalanced datasets common in cybersecurity and other numerical applications.Undetected rate is also identified as a useful metric for the subsequent performance evaluation.Source: D. Selecting the Number of Modes · p. 7

IV. CASE STUDY: IIOT SECURITY

The case study applies TRUST to intrusion detection in a lab-scaled IIoT system. Its testbed uses a SCADA system to supervise liquid storage tanks in water treatment and distribution.

IV. CASE STUDY: IIOT SECURITY

  • The case study targets network-based intrusion detection, where the authors say XAI can add interpretability and trust to sophisticated AI models.Source: §IV · p. 7
  • The authors built a lab-scaled IIoT testbed to collect realistic and up-to-date datasets for network-based intrusion detection.Source: §IV · p. 7
  • The testbed’s SCADA system supervises liquid-storage tank level and turbidity in industrial reservoirs and water distribution systems.The authors refer readers to earlier papers and for further testbed information.Source: §IV · p. 7

A. Utilized Datasets

The evaluation covers three cybersecurity datasets, including traffic collected from the authors’ IIoT testbed and two other intrusion-detection datasets.

A. Utilized Datasets

  • WUSTL-IIoT traffic came from the authors’ testbed, where attacks included backdoor, command injection, denial of service, and reconnaissance.Source: A. Utilized Datasets · pp. 7–8
  • The three evaluation datasets are WUSTL-IIoT, NSL-KDD, and UNSW.NSL-KDD and UNSW are additional cybersecurity datasets used for proof of concept.Source: A. Utilized Datasets · pp. 7–8
  • NSL-KDD is a revised, cleansed KDD’99 dataset with four attack types grouped as class 1; UNSW traffic is labeled normal or attack.The passages identify UNSW as the UNSW-NB15 dataset, built by the Australian Centre for Cyber Security in 2015.Source: A. Utilized Datasets · p. 8

B. The Primary AI Model

The study uses a binary ANN as the primary AI and evaluates it on three datasets, while TRUST explains the AI’s output behavior rather than its correctness.

B. The Primary AI Model

  • The primary AI is a binary ANN with two hidden layers of 20 and 10 ReLU neurons, trained for ten epochs with Adam and a sigmoid output.Normal is class 0 and attack is class 1.Source: B. The Primary AI Model · p. 8
  • Each dataset is split 80:20, with training and test results presented separately.Source: B. The Primary AI Model · p. 8
  • TRUST explains the primary AI’s behavior from its predicted labels, not whether its predictions match the true labels.The primary AI’s performance is summarized using Accuracy, MCC, and undetected rate.Source: B. The Primary AI Model · p. 8, Table VI · p. 9

C. The XAI Model: TRUST

TRUST selects representative factors using mutual information, then estimates the number of probability-density modes for each representative.

C. The XAI Model: TRUST

  • Mutual-information scores rank factors against the primary AI’s predicted class labels, and the top k become representatives.The selection also assigns importance coefficients to the representatives.Source: C. The XAI Model: TRUST · p. 9
  • Choosing k trades explainer speed against performance: more representatives may improve performance but take longer.The tested range is k = 1 to 8, and the paper reports high MCC with k of 4 or 5.Source: C. The XAI Model: TRUST · p. 9
  • After selecting representatives, TRUST estimates the number of modes in each representative’s probability density function.The described case uses two classes and therefore two representative sets.Source: C. The XAI Model: TRUST · p. 9

D. Results

TRUST explains unseen samples by comparing class likelihoods, with reported agreement above 90% MCC on average. The authors also report faster mode search and faster processing than LIME.

D. Results

  • For a new sample, TRUST projects its features into the training-set factor space, calculates representative likelihoods, and assigns the class with the highest likelihood.The core representatives are built using training data.Source: D. Results · p. 9
  • 90.89% MCC or 98% accuracy on average: TRUST explains the primary AI’s output on unseen data.Figure 4 plots MCC against the number of representatives for training and test sets.Source: D. Results · p. 9, Figure 4 · p. 10
  • More than ten times faster: Algorithm 4 speeds up mode search relative to Algorithm 3.The search used a 20 by 20 grid, divided into sixteen 5×5 sub-zones for Algorithm 4.Source: D. Results · p. 10
  • Almost 25 times faster than LIME, TRUST’s reported run includes model building and labeling 8,000 test samples.With five representatives, TRUST’s test-label accuracy is higher than LIME’s except on UNSW, where LIME scores 94.86% versus TRUST’s 93.76%.Source: D. Results · p. 10

V. HOW TRUST EXPLAINS THE PRIMARY AI RESULTS TO THE USER

TRUST explains primary-AI labels by comparing a sample’s representative values with class-specific probability distributions. In the six-sample example, the combined likelihoods from all four representatives correctly explain every labeled class, though individual representatives sometimes err.

V. HOW TRUST EXPLAINS THE PRIMARY AI RESULTS TO THE USER

  • For the NSL-KDD example, each 40-dimensional traffic packet is reduced to four representatives, and the explanation assumes a user with basic AI, mathematics, and statistics knowledge.The section presents this dimension reduction in terms intended to be simple and interpretable for that assumed user.Source: §V · p. 10
  • TRUST transforms features to remove redundancy, selects influential factors as representatives using mutual information, and models their class-specific densities with multimodal distributions.Factorization projects features onto orthogonal dimensions; the resulting representatives are used to estimate class likelihoods for new samples.Source: §V · p. 10
  • A plotted example compares class-specific probability-density curves for a representative; the illustrated new sample has higher likelihood under the attack class.The paper says users can interpret the likelihood differences between the classes.Source: §V · p. 10, Figure 7 · p. 11
  • All four representatives together correctly explained the labeled classes for all six test samples, although individual representatives sometimes made errors.The examples include a first-representative error on sample four and a third-representative error on sample two.Source: §V · p. 11

VI. CONCLUSION AND FUTURE DIRECTION

The conclusion presents TRUST as a statistical XAI approach for numerical applications and reports that it outperforms LIME in speed and accuracy. It also identifies assumptions that constrain reliability and suggests extensions to representative selection, probability distributions, and dependence modeling.

VI. CONCLUSION AND FUTURE DIRECTION

  • TRUST addresses a stated shortage of XAI research for numerical applications and is reported to be 25 times faster and more accurate than LIME.The paper presents TRUST as a candidate for real-time and critical applications.Source: §VI · pp. 11–12
  • Reliability may be poor if information-gain-based representative selection overfits, Gaussian assumptions fail, data distributions change, or samples are not independent.The paper identifies independent sampling as an important assumption.Source: §VI · p. 12
  • Future directions include using metrics such as the Gini index, modeling distributions such as Poisson, and approximating feature-pair joint probabilities to relax independence.These are proposed extensions to representative selection and probability modeling.Source: §VI · p. 12
Loading 2205.01232v1…