PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey

Ibrahim Kok, Feyza Yildirim Okay, Ozgecan Muyanli, Suat Ozdemir

arXiv:2206.04800v1cs.AIcs.LGcs.NI

TL;DR

Black-box AI can make decision paths difficult for users to comprehend, and the authors identify a lack of a survey focused on XAI in IoT. They review IoT XAI studies by application area and taxonomy, and synthesize challenges and future directions. The paper concludes with a comprehensive perspective on XAI usage in IoT and identifies open issues for future work.

  • Problem

    The authors report that no existing survey, to their knowledge, investigated XAI techniques in IoT, despite the importance of understanding decisions made by black-box models.

    Source: §I · p. 1

  • Method

    The paper surveys XAI studies in IoT by application area and summarizes them using XAI terminology and taxonomy.

    Source: §I · p. 2, §VI · p. 11

  • Results

    The survey provides a comprehensive perspective on XAI usage areas in IoT and identifies open issues and future research directions.

    Source: §I · p. 2, §VI · p. 11

  • Takeaways & Limitations

    The paper presents IoT-specific challenges and open issues as directions for potential future XAI research.

    Source: §I · p. 2, §VI · p. 11

Abstract

from arXiv · show

Black-box nature of Artificial Intelligence (AI) models do not allow users to comprehend and sometimes trust the output created by such model. In AI applications, where not only the results but also the decision paths to the results are critical, such black-box AI models are not sufficient. Explainable Artificial Intelligence (XAI) addresses this problem and defines a set of AI models that are interpretable by the users. Recently, several number of XAI models have been to address the issues surrounding by lack of interpretability and explainability of black-box models in various application areas such as healthcare, military, energy, financial and industrial domains. Although the concept of XAI has gained great deal of attention recently, its integration into the IoT domain has not yet been fully defined. In this paper, we provide an in-depth and systematic review of recent studies using XAI models in the scope of IoT domain. We categorize the studies according to their methodology and applications areas. In addition, we aim to focus on the challenging problems and open issues and give future directions to guide the developers and researchers for prospective future investigations.

I. INTRODUCTION

IoT produces large volumes of data, and AI helps turn that data into decisions, but complex black-box models can obscure how decisions are made. The paper reviews XAI in IoT, an area the authors say lacked a dedicated survey, and identifies open challenges and future directions.

I. INTRODUCTION

  • IoT connects smart objects that communicate and share information, supporting applications including smart cities, agriculture, healthcare, finance, and the military.The paper describes IoT as transforming devices such as sensors, actuators, RFID tags, and cell phones into smart objects.Source: §I · p. 1
  • IoT generates massive amounts of data that require fine-grained analysis, and AI is used to extract information and support decisions in fields such as healthcare and autonomous systems.The paper also notes AI use in network, device, and service management and analysis of IoT data.Source: §I · p. 1
  • Black-box AI can hide decision logic from users, while XAI can expose factors such as feature importance and help users evaluate both outputs and their contributing inputs.The paper links explanations to verification, improvement, learning, and legislative compliance.Source: §I · p. 1
  • The authors report that no existing survey, to their knowledge, investigated XAI techniques in the IoT domain.They distinguish this gap from general XAI surveys and a survey focused on time-series data.Source: §I · p. 1
  • The paper reviews XAI studies in IoT and explains terminology and techniques for early researchers.The authors also identify IoT-related challenges and open issues and outline future research directions.Source: §I · p. 2

II. EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI)

The section distinguishes XAI from related terms and outlines abilities used to characterize explanations. It notes that XAI and interpretable machine learning are often used interchangeably, while treating IML as a subset of XAI.

A. Terminology and Definitions

  • Interpretable Machine Learning (IML) is a subset of XAI focused on describing the logic of machine-learning algorithms.The paper notes that practitioners and academics often use XAI and IML interchangeably, though they focus on explainability and interpretability, respectively.Source: A. Terminology and Definitions · p. 2
  • Explainability concerns information about a model’s inner workings, while interpretability concerns making a model or its predictions understandable to humans.The paper associates interpretability with transparency.Source: A. Terminology and Definitions · p. 2
  • Understandability concerns grasping a system’s function without its underlying structure, while comprehensibility concerns expressing learned information understandably.Source: A. Terminology and Definitions · p. 2
  • Transparency means a model is understandable rather than opaque, and faithfulness concerns consistently choosing truly relevant features.Source: A. Terminology and Definitions · p. 2
  • Informativeness is the ability of an explainability strategy to offer end-users meaningful information.Source: A. Terminology and Definitions · p. 2

B. XAI Taxonomy

The taxonomy classifies XAI from multiple perspectives, and categories may overlap. The section distinguishes when explanations are produced and whether methods depend on a particular model class.

B. XAI Taxonomy

  • XAI categories can overlap, so a method may belong to multiple categories and is better examined under its own sub-taxonomy.The paper gives post-hoc, model-agnostic, and local as possible classifications for one method.Source: B. XAI Taxonomy · p. 2

1) Ante-hoc vs. Post-hoc:

  • Ante-hoc methods explain before or during training, whereas post-hoc methods are applied externally after training and can supplement ante-hoc methods.Transparent models such as decision trees and sparse linear regression are cited as intrinsically explainable examples.Source: 1) Ante-hoc vs. Post-hoc · p. 2

2) Model-specific vs. Model-agnostic:

  • Model-specific methods are designed for particular model classes, while model-agnostic methods can be applied across model types.The paper notes that model-specific methods limit the choice of model, whereas model-agnostic explanations are independent of model type.Source: 2) Model-specific vs. Model-agnostic · pp. 2–3

3) Local vs. Global:

The section distinguishes local explanations of particular predictions from global explanations that characterize a model as a whole.

3) Local vs. Global:

  • Local explanations examine one or more instances to explain particular predictions, while global explanations characterize the model and its significant features and relationships.The paper associates local explanation with cases where predictions depend linearly on some features rather than complexly on all features.Source: 3) Local vs. Global · p. 3

C. XAI Methodologies

The paper groups XAI methods into five categories and describes visual, feature-based, and example-based explanations.

1) Visual Explanation:

  • Visual explanations show relationships between inputs and outputs or among input attributes, but large feature sets can make those relationships difficult to interpret.PDP gives global explanations; ICE retains the full distribution, whereas PDP averages over the marginal distribution.Source: 1) Visual Explanation · p. 3

2) Feature-based Explanation:

  • Feature-based approaches assess feature contributions to predictions using local or global methods, including SHAP, LIME, Anchors, feature interaction, PFI, and LOCO.PFI shuffles feature values to assess prediction error; LOCO drops each variable, retrains, and compares error with a baseline using all features.Source: 2) Feature-based Explanation · pp. 3–4

3) Example-based Explanation:

  • Example-based explanations use selected instances to build compact accounts of model behavior or data distributions, locally or globally.Counterfactuals ask what could change an outcome, while prototypes and criticisms identify representative and unrepresentative samples.Source: 3) Example-based Explanation · p. 4

4) Perturbation-based Explanation:

The section describes perturbation and gradient approaches, then outlines why XAI may be useful in IoT settings. It highlights explanation of decisions, fault discovery, and bias detection.

4) Perturbation-based Explanation:

  • Perturbation-based methods probe black-box models by varying inputs; occlusion replaces features with constants, while RISE uses randomly masked input portions.Meaningful Perturbation uses meta-predictors to estimate whether features are present, with prediction error serving as an explanation-accuracy metric.Source: 4) Perturbation-based Explanation · p. 4

5) Gradient/Backpropagation:

  • Gradient methods use information flow during backpropagation to relate input features to outputs, often presenting heatmaps or feature attributions as visual explanations.Examples include saliency maps, DeepLift, DeepSHAP, Deconvolution, Guided Backpropagation, CAM, and GradCAM.Source: 5) Gradient/Backpropagation · p. 4

III. USING XAI IN IOT

  • XAI is especially relevant when users need to understand, trust, or manage AI findings for commercial, ethical, or regulatory reasons, though its need is debated.The paper notes it may be unnecessary when explainability costs outweigh its need, error impacts are limited, or the problem is well researched in real-world settings.Source: §III · p. 4
  • In critical applications, explanations can help analyze why an IoT decision occurred, while examining model behavior can reveal faults linked to incomplete or erroneous sensor data.The paper uses a mission-critical aerial device and inaccurate decisions from faulty sensor data as examples.Source: §III · p. 4
  • XAI can provide local explanations for individual samples, uncover previously unnoticed patterns, and help reveal bias such as rejecting loan applications from particular groups.The paper states that explanations provide a guideline for revealing such model biases.Source: §III · pp. 4–5

IV. XAI IN IOT APPLICATION DOMAINS

The paper reviews XAI studies in IoT application domains and summarizes them by XAI terminology and taxonomy. Its robotics examples include explanation methods and reported effects on human-robot task performance, trust, and transparency.

IV. XAI IN IOT APPLICATION DOMAINS

  • The review organizes XAI studies in IoT application domains in Table I according to XAI terminology and taxonomy.Source: §IV · p. 5

A. Autonomous Systems and Robotics

  • In a simulated human-robot team task, a POMDP-based robot explanation mechanism improved task performance, trust, and transparency.The authors used four levels of natural-language explanation, ranging from none to confidence-level explanations.Source: A. Autonomous Systems and Robotics · p. 5
  • Iyer et al. used object saliency maps to visualize states and actions in an explainable object-sensitive deep reinforcement learning model for object recognition and classification.Source: A. Autonomous Systems and Robotics · p. 5
  • Kampik et al. studied text-based explanations of autonomous-agent actions with participants in human-robot ultimatum games.Source: A. Autonomous Systems and Robotics · p. 5
  • Guo provided partial explainability for a UAV-supported 5G network’s QoS, QoE, and energy-optimization DDDQN by extracting network state weights.Source: A. Autonomous Systems and Robotics · p. 5

B. Energy Management

The review describes XAI applications in energy management, environmental monitoring, and finance. The examples use varied explanation methods, including SHAP, LIME, and rule-based approaches.

B. Energy Management

  • Kim and Cho reported that their explainable autoencoder energy-demand model performed best against decision trees, MLP, LSTM, and other machine-learning models.It predicts demand 15, 30, and 45 minutes ahead from eight features; Sirmacek and Riverio used SHAP for feature contributions and local-pixel contributions in occupancy prediction.Source: B. Energy Management · p. 6

C. Environmental Monitoring

  • Environmental IoT studies apply explanations to air-pollution monitoring, biological sensing, flood imagery, species distribution, and campus monitoring.Examples include SHAP charts for random-forest features, rule-based explanations for flood-image classification, and LIME for African-elephant distribution analysis.Source: C. Environmental Monitoring · p. 6

D. Financial System

  • Financial-domain XAI studies applied rule-based reasoning to lending, TreeSHAP to loan-purchase risk, SHAP to insurance behavior, and LIME to LSTM/CNN stock-price predictions.Sachan et al. added factual and heuristic rules to a traditional IF-THEN system; Bussmann et al. used TreeSHAP to explain borrowers’ credit scores.Source: D. Financial System · pp. 6–7

E. Healthcare

The survey reviews XAI applications in healthcare and industrial IoT, spanning diagnostic and monitoring systems as well as methods for explaining predictions and processes.

E. Healthcare

  • Healthcare XAI studies address surgical-image retrieval, pandemic monitoring, heart-disease models, diagnostics, insurance risk, medical imaging, and prosthetic hand-gesture classification.Examples include user-feedback-driven image retrieval, interactive explanations, SHAP-based risk explanations, and visual saliency maps.Source: E. Healthcare · pp. 7–8
  • HihO rendered 20 times faster than GradCAM and 200 times faster than RISE on the PPMI dataset.The comparison concerns a CNN-based hierarchical occlusion model for medical imaging workflows.Source: E. Healthcare · p. 8

F. Industrial Domain

  • Industrial IoT studies use explainable models for energy-consumption forecasting, machine remaining-useful-life estimation, and factory process prediction.Reported approaches include LSTM-RNN models, rule-based Qarma algorithms, and an RNN-based factory interface offering local and global explanations.Source: F. Industrial Domain · p. 8
  • Industrial fault diagnosis and machine-health monitoring use visual explanations, including CAM localization of damaged parts without machine sensors and Grad-CAM heat maps for vibration classification.The vibration study also verified explainability using neural networks, ANFIS, and decision trees.Source: F. Industrial Domain · p. 8
  • Other industrial studies apply ELI5 and LIME to machine-life estimates, SHAP to production-quality relationships, and surrogate decision trees to predictive process monitoring.These examples span manufacturing and industrial process applications.Source: F. Industrial Domain · p. 8
  • Industrial fault detection combines unsupervised anomaly-detection and clustering algorithms, with SHAP and Local-DIFFI explanations evaluated on bearing, chance, and mechanical-failure datasets.Source: F. Industrial Domain · p. 8

G. Security and Privacy

The survey covers XAI in IoT security and smart agriculture, where explanations are used for interpreting security decisions and supporting agricultural decisions. It summarizes application-specific models and methods across both areas.

G. Security and Privacy

  • IoT security systems use XAI to interpret intrusion, anomaly, authentication, and malware decisions and support trust management.Source: G. Security and Privacy · p. 8
  • Security studies apply Voronoi diagrams, rule inference, and SHAP to explain anomaly detection and intrusion-detection models.Reported examples include a decision tree with higher accuracy and more interpretable rules than random forest and SVM on the KDD benchmark.Source: G. Security and Privacy · p. 10
  • Other security approaches explain cyber-threat and attack detection with LIME, decision trees that unify LSTM outputs, and DeepSHAP for mobile-IoT traffic classification.TRUST was reported as more successful than LIME in speed and accuracy.Source: G. Security and Privacy · p. 10

H. Smart Agriculture

  • Smart agriculture uses interpretable AI to address adoption concerns in decision-support systems.Source: H. Smart Agriculture · p. 10
  • Agriculture examples include explainable automated irrigation, fuzzy-rule-based field and crop management, and case-based grass-growth prediction for dairy farmers.The fuzzy system uses temperature, humidity, soil moisture, and water-nutrient sensor values.Source: H. Smart Agriculture · p. 10
  • Further agricultural studies explain plant-disease detection with GradCAM, land-use planning with feature-importance and dependence methods, and evaporative-water-loss prediction with decision-tree rules.Source: H. Smart Agriculture · p. 10

V. CHALLENGES, OPEN ISSUES, AND FUTURE RESEARCH DIRECTIONS

The section identifies conceptual, evaluative, security, and usability challenges for XAI, then outlines directions involving method combinations, automation, security, distributed explanation, and application-specific interfaces.

Challenges and Open Issues

  • XAI concepts can be contradictory or imprecise, with non-standardized terminology and unresolved questions about whether explanations should be faithful to the model or the data-generation process.Source: §V · p. 10
  • Most XAI methods do not measure expected error rates, making explanations difficult to test rigorously as scientific hypotheses.Source: §V · p. 11
  • Interpretable systems lack quantitative measures of completeness and accuracy, and interpretability assessments should not depend on an observer’s field knowledge.The section calls for objective measurement metrics.Source: §V · p. 11
  • Interpretability can expose model-relevant inputs to attackers, who may seek small input changes that alter outputs; commonly used XAI models are also described as vulnerable to adversarial attacks.Source: §V · p. 11
  • The section identifies a lack of proper structure for combining XAI methods to generate more complete explanations.Source: §V · p. 11
  • Explanations may be illogical for non-experts or omit feature relationships and why features matter, so domain expertise or techniques may be needed to identify those connections.Source: §V · p. 11

Future Research Directions

  • The proposed directions include model-specific methods for fidelity, combining methods to address partial explanations, and automating tasks such as feature selection and hyperparameter tuning.The section also points to improved libraries and packages for higher-level explanations.Source: §V · p. 11
  • Other directions include improving XAI security and resilience, using federated learning for local fog-server and global cloud-server explanations, and tailoring interfaces to IoT applications.Source: §V · p. 11

VI. CONCLUSION

The paper surveys XAI models in AI-powered IoT applications. It organizes the literature by application areas and XAI terminology and taxonomy, and identifies open issues and future directions.

VI. CONCLUSION

  • The paper presents a detailed survey of XAI models in AI-powered IoT applications.Source: §VI · p. 11
  • The survey examines IoT requirements and potential benefits of XAI, then reviews studies by application area and summarizes them using XAI terminology and taxonomy.Source: §VI · p. 11
  • The paper presents potential future-study ideas by focusing on open issues and future directions.Source: §VI · p. 11

Related papers

Loading 2206.04800v1…