PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera

arXiv:1910.10045v2cs.AIcs.LGcs.NE

TL;DR

AI systems increasingly require explainability for trustworthy adoption, yet objective metrics and interpretable methods remain limited. This overview defines audience-centered explainability, systematically reviews XAI research, and develops taxonomies that position XAI as a central component of responsible AI.

  • Problem

    Objective metrics for evaluating explanation quality, usefulness, audience understanding, and satisfaction remain insufficiently established.

  • Method

    The paper synthesizes XAI concepts and literature, defining audience-centered explainability and classifying transparent models and post-hoc techniques in global taxonomies.

  • Results

    The literature analysis yields a global taxonomy of XAI proposals, with rule extraction and feature relevance prominent among post-hoc approaches.

  • Takeaways & Limitations

    XAI is a central component of AI principles, appearing explicitly in 28 of 32 analyzed principles guides.

  • Takeaways & Limitations

    Objective measurement of whether explanations are understood remains difficult.

Abstract

from arXiv · show

In the last years, Artificial Intelligence (AI) has achieved a notable momentum that may deliver the best of expectations over many application sectors across the field. For this to occur, the entire community stands in front of the barrier of explainability, an inherent problem of AI techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI. Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is acknowledged as a crucial feature for the practical deployment of AI models. This overview examines the existing literature in the field of XAI, including a prospect toward what is yet to be reached. We summarize previous efforts to define explainability in Machine Learning, establishing a novel definition that covers prior conceptual propositions with a major focus on the audience for which explainability is sought. We then propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at Deep Learning methods for which a second taxonomy is built. This literature analysis serves as the background for a series of challenges faced by XAI, such as the crossroads between data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to XAI with a reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.

1. Introduction

AI’s expanding use of opaque models in critical contexts has intensified the need for explainability, motivating this overview’s unified XAI framework, taxonomies, challenge analysis, and Responsible AI perspective.

  • Motivation: AI’s growing deployment across sectors and increasingly complex tasks has made intelligent systems with learning, reasoning, and adaptation capabilities especially important.The introduction frames AI as central to many activity sectors and emphasizes the importance of its capabilities.
  • Motivation: Opaque Deep Neural Networks, with hundreds of layers and millions of parameters, have increased the explainability barrier compared with earlier interpretable AI systems.Their empirical success is linked to efficient learning algorithms and a huge parametric space.
  • Motivation: As black-box models make important predictions in critical contexts, stakeholders demand explanations that support transparency, legitimacy, trust, bias correction, robustness, and meaningful causal reasoning.Interpretability can reveal discriminatory rules, adversarial perturbations, and whether meaningful variables drive outputs.
  • XAI overview: XAI seeks ML techniques that preserve high learning performance while making models more explainable and enabling humans to understand, appropriately trust, and manage AI systems.The proposed field also draws on Social Sciences and considers the psychological aspects of explanation.
  • Contributions: The overview proposes an audience-centered definition of explainability, distinguishes model transparency from post-hoc explanation, and organizes approximately 400 contributions into two XAI taxonomies.The taxonomies cover ML transparency and post-hoc explainability, plus Deep Learning methods categorized by criteria such as layerwise explanations, representation vectors, and attention.
  • Challenges and prospects: The paper identifies unresolved needs for explainability concepts and metrics, more understandable Deep Learning, and analysis of confidentiality, adversarial robustness, data diversity, privacy, fairness, and accountability.It culminates in Responsible Artificial Intelligence as systematic adoption of principles for practical AI deployment.

2. Explainability: What, Why, What For and How?

This section distinguishes explainability from related concepts and defines it as audience-oriented details or reasons that clarify a model’s functioning. It also explains that explainability serves purposes including trust, causality, informativeness, and adoption in regulated sectors.

  • Conceptual distinctions: Interpretability is a passive, human-sense-making characteristic, whereas explainability is an active procedure intended to clarify a model’s internal functions.Transparency describes models understandable by themselves, while explainability concerns actions or procedures that communicate how a model works.
  • Purposes of XAI: Understanding and trust are important XAI goals, but a complete account also includes causality, transferability, informativeness, fairness, and confidence.The cited discussion identifies these purposes as omissions from a narrower definition centered only on understanding and trust.
  • Definition of explainability: Explainability is audience-dependent: an explainable AI system produces details or reasons that make its functioning clear or easy to understand.The definition treats the audience’s understanding and clarity as central, with application purposes such as improving trustworthiness.
  • Why XAI matters: Explainability can facilitate adoption of advanced ML in regulated sectors by addressing reluctance caused by risks to organizational assets.The research–business gap has limited uptake in banking, finance, security, health, and other sectors that have lagged in digital transformation.
  • Purposes of XAI: Explainable models can help identify relationships among variables for further causal testing and provide intuition that complements causality-inference techniques.The passage notes that causation involves correlation and that explainability can validate or initially guide causal inference results.
  • Purposes of XAI: Explainability supports decision making by providing information that helps users relate their decisions to model solutions and avoid misconceptions.The model’s problem is not identical to the human counterpart’s problem, so users need substantial information to interpret the model’s output.

3. Transparent Machine Learning Models

Transparent machine-learning models are understandable by themselves and may provide simulatability, decomposability, or algorithmic transparency. The section reviews representative models—including regression, decision trees, KNN, rule-based learners, and GAMs—while noting that transparency depends on model size, features, complexity, and intended use.

  • Transparent Machine Learning Models: Transparent models are understandable by themselves and can satisfy simulatability, decomposability, algorithmic transparency, or several of these levels.The surveyed models are classified according to the transparency levels introduced previously.
  • Logistic Regression: Logistic regression remains transparent because its stiffness follows a linear dependence assumption, although interpreting odds ratios as substantive effects can be dangerous.Comparing ratios across models with different variables may also be problematic because unobserved heterogeneity can vary.
  • Decision Trees: Decision trees offer off-the-shelf transparency and are comfortable for experts to interpret, but their poor generalization properties can limit applications.Their complexity and understandability have motivated substantial work on simplification and generation.
  • K-Nearest Neighbors: KNN explains predictions through distances and similarities between examples, resembling experience-based human decision making, but its transparency depends on features, neighbor count, and distance function.A very high K can impede human simulation, while complex features or distance functions can restrict interpretability to algorithmic transparency.
  • Rule-Based Learning: Rule-based learners support understandable knowledge representation, but increasing rule coverage and specificity creates a central tension for model analysis.Rules may be simple if-then statements, complex combinations, or verbally formulated fuzzy rules over imprecise domains.
  • Generalized Additive Models: GAM applications prioritize understandability of underlying relationships rather than accuracy for its own sake.Their purpose is to expose relationships among variables in cases selected for scrutiny.

4. Post-hoc Explainability Techniques for Machile Learning Models: Taxonomy, Shallow Models and Deep Learning

The section classifies post-hoc explainability techniques for models that are not transparent, distinguishing model-agnostic methods from model-specific approaches. It further separates model-specific techniques for shallow and deep learning models and introduces a complementary deep-learning taxonomy.

  • Post-hoc explainability techniques are applied when ML models do not satisfy criteria for transparency and require separate methods to explain their decisions.
  • Model-agnostic techniques: Model-agnostic techniques can be applied to any ML model regardless of its internal processing or representations.
  • Model-specific techniques: Model-specific post-hoc methods are divided into techniques for shallow ML models and techniques for deep learning models, including neural networks and related variants.
  • Deep learning taxonomy: A second taxonomy complements the general classification by organizing post-hoc explanations for deep learning models and linking them to the first taxonomy’s criteria.

4.1. Model-agnostic Techniques for Post-hoc Explainability

Model-agnostic post-hoc techniques extract information from any model through simplification, feature-relevance analysis, or visualization. The literature review finds rule extraction prevalent, while visualization mainly supports interpreting feature-relevance outputs.

  • Technique families: Model-agnostic post-hoc methods can simplify models into tractable proxies, extract knowledge directly, or visualize predictions and behavior.These techniques are designed to plug into any model and obtain explanatory information from its prediction procedure.
  • Explanation by simplification: Explanation by simplification is the broadest model-agnostic category and includes local explanations based on simplified models representing specific model sections.Most techniques in this category rely on rule extraction, including LIME and G-REX.
  • Feature relevance: Feature-relevance techniques explain opaque models by ranking or measuring each feature’s influence, relevance, or importance for prediction outputs.SHAP calculates an additive feature-importance score for each particular prediction.
  • Visual explanations: Visualization techniques provide model-agnostic explanations by presenting outputs and other information in forms that facilitate black-box model interpretation.Visualization methods include global sensitivity-analysis portfolios and data-based, Monte-Carlo, and cluster-based sensitivity-analysis methods.
  • Literature trends: Rule extraction prevails among model-agnostic post-hoc contributions, while visualization often displays feature-relevance outputs to ease interpretation.Visualization of other trained-model aspects, such as structure or operations, remains tightly linked to the specific model.

4.2. Post-hoc Explainability in Shallow ML Models

Post-hoc explainability in shallow ML focuses on tree ensembles and SVMs, using simplification, feature relevance, visualization, and probabilistic interpretations to make complex models more understandable.

  • Tree Ensembles: Tree-ensemble explainability primarily uses model simplification and feature-relevance techniques to balance interpretability against predictive accuracy.Simplification trains less complex surrogate models or constructs simplified ensemble learners, while feature relevance measures variable importance through permutation-based changes in forest performance.
  • Tree Ensembles: Stacking With Auxiliary Features integrates explanations about learner outputs and ensemble consensus to improve generalization.The approach supplements component predictions with information about their origins and agreement across the ensemble.
  • Support Vector Machines: SVM post-hoc explainability includes rule extraction from support vectors, visualizations of kernels and input relevance, and heatmaps for application-specific debugging.These techniques connect the SVM’s internal mathematical structure to interpretable information about rules, variables, and compound atoms.
  • Support Vector Machines: Bayesian interpretations of SVMs make hyper-parameter tuning comprehensible and enable class-probability predictions instead of binary classification.These methods interpret SVMs as MAP solutions to inference problems with Gaussian Process priors.
  • Support Vector Machines: Unlike other shallow models, SVM explainability increasingly includes local explanations, although simplification methods are older on average.The reviewed SVM explainability literature contains no methods dated beyond 2017, possibly reflecting the rise of Deep Learning or the maturity of SVM understanding.

4.3. Explainability in Deep Learning

Explainability in deep learning primarily relies on post-hoc local explanations and feature-relevance techniques across multilayer networks, CNNs, and RNNs. The literature develops model simplification, attribution, visualization, and representation-analysis methods while exposing theoretical and interpretive limitations.

  • Overview: Post-hoc local explanations and feature-relevance techniques are the most adopted approaches for explaining multilayer networks, CNNs, and RNNs.The reviewed studies focus on these three major deep-learning model families.
  • Multilayer neural networks: Multilayer neural networks remain difficult to explain because increasing depth makes model simplification more complex and reinforces their black-box perception.Proposed simplification methods include DeepRED, interpretable distillation, hierarchical feature-space partitioning, and ensemble knowledge distillation.
  • Multilayer neural networks: Feature-relevance methods decompose network decisions into input contributions, but most approaches violate sensitivity and implementation invariance; integrated gradients was created to follow these axioms.The approach treats neurons as decomposable objects and aggregates and back-propagates their contributions.
  • Convolutional neural networks: CNN explanations divide into mapping outputs back to discriminative input regions and interpreting intermediate representations independently of specific inputs.Methods include Deconvnet, class activation maps, guided backpropagation, filter-specific losses, pixel heatmaps, image reconstruction, and network dissection.
  • Convolutional neural networks: Local explanations in deep networks are strongly dominated by lower-level features, while global explanations can be built from locally identified prototypes.Deep architectures provide strong priors that limit changes to how low-level representations are captured.

4.4. Alternative Taxonomy of Post-hoc Explainability Techniques for Deep Learning

The alternative Deep Learning taxonomy complements and overlaps with the taxonomy in Figure 6 by regrouping methods and distinguishing explanations of model processing from model representation. This distinction supports selecting XAI methods according to whether information about execution traces or internal data structures is needed.

  • Taxonomy construction: Deep Learning XAI methods can be classified through multiple taxonomic perspectives beyond the common model-agnostic versus model-specific distinction.Model-agnostic methods such as SHAP are widely used to explain Deep Learning models, while some models are designed to simplify interpretation of their own behavior.
  • Taxonomy correspondence: The alternative taxonomy merges CNN and RNN feature-relevance categories into Explanation of Deep Network Processing with Salience Mapping.Methods classified separately as feature relevance for CNN and feature relevance for RNN in Figure 6 are included in one category under the alternative classification.
  • Taxonomy correspondence: The alternative taxonomy splits Explanation by simplification for Multi-Layer Neural Network into decision-tree and representation-vector explanations.Methods placed in one Figure 6 category are divided into Explanation of Deep Network Processing with Decision Trees and Explanation of Deep Network Representation with the Role of Representation Vectors.
  • Processing versus representation: Distinguishing model processing from model representation separates execution traces from internal data structures and guides XAI-method selection by the information required.The distinction is analogous to testing and debugging methods used in regular programming and can help address different failure reasons in complex models.

5. XAI: Opportunities, Challenges and Future Research Needs · 5.1. On the Tradeoff between Interpretability and Performance · 5.2. On the Concept and Metrics

The section identifies major XAI research opportunities, emphasizing the interpretability–performance tradeoff and the need for a shared concept of explainability with meaningful evaluation metrics. It also situates these challenges within broader research needs concerning Deep Learning, underexplored explainability niches, and responsible AI.

  • 5. XAI: Opportunities, Challenges and Future Research Needs: The section revisits XAI advances and identifies research paths for addressing persistent challenges in explainability and data fusion models.This prospective discussion is complemented by Section 6’s broader focus on responsible AI.
  • 5. XAI: Opportunities, Challenges and Future Research Needs: Further challenges concern Deep Learning explainability and underexplored research niches connected to model explainability, including topics addressed in Subsections 5.3 through 5.8.The section presents Deep Learning as especially prevalent in the XAI literature while noting that several related niches remain insufficiently studied.
  • 5.1. On the Tradeoff between Interpretability and Performance: Interpretability versus performance is a recurring issue surrounded by myths and misconceptions, motivating efforts to find an optimal balance between both properties.The section frames this balance as a potential opportunity for XAI developments.
  • 5.1. On the Tradeoff between Interpretability and Performance: More complex models are not inherently more accurate when data are well structured and available features are high-quality, valuable, constrained, and highly correlated.Such conditions can occur in industry environments involving controlled physical problems.
  • 5.1. On the Tradeoff between Interpretability and Performance: The interpretability–performance tradeoff may be improved by sophisticated XAI methods rather than remaining an unavoidable decline in interpretability as model complexity increases.Figure 12 represents an area where XAI techniques and tools could improve the common tradeoff.
  • 5.2. On the Concept and Metrics: XAI requires a unified concept of explainability that establishes common ground and a shared structure for XAI systems.The paper presents a new concept built upon Gunning’s proposition.
  • 5.2. On the Concept and Metrics: Meaningful metrics are needed to compare how well models satisfy explainability criteria and to provide solid grounding for claims about explainability.These metrics should assess specific explainability aspects similarly to accuracy, F1, and sensitivity for model performance.
  • 5.2. On the Concept and Metrics: The survey does not design a metric suite, instead calling for community-led agreement on explainability concepts, evaluation proposals, and quantitative comparison methods across contexts, models, and purposes.The proposed comparisons should evaluate XAI techniques under different application contexts, models, and purposes.

5.3. Challenges to achieve Explainable Deep Learning · 5.4. Explanations for AI Security: XAI and Adversarial Machine Learning · 5.5. XAI and Output Confidence

The sections identify unresolved explainability challenges in deep learning, especially definitional inconsistency, audience-dependent interpretability, evaluation, accessibility, confidentiality, and adversarial misuse. They also examine generative-model approaches and output-confidence assessment as ways to manage explanation and safety risks.

  • 5.3. Challenges to achieve Explainable Deep Learning: Deep-learning XAI lacks agreed vocabulary and definitions, including inconsistent usage of terms such as feature importance and feature relevance.
  • 5.3. Challenges to achieve Explainable Deep Learning: Explainability involves a trade-off between interpretability and accuracy, while intelligibility should vary according to whether the audience is an expert, policy-maker, or nonexpert user.
  • 5.3. Challenges to achieve Explainable Deep Learning: Neural-symbolic interpretability is presented as a route to convincing explanations while keeping or improving generic performance by combining connectionist and symbolic paradigms.Connectionist methods are described as more precise but opaque, whereas symbolic methods offer greater explainability.
  • 5.3. Challenges to achieve Explainable Deep Learning: XAI for deep learning still needs rigorous evaluation methods and explanations accessible to society, policy-makers, and the law, including non-technical explanations relevant to GDPR rights.Suggested evaluation considerations include consistent evaluation questions and population samples, drawing inspiration from the social sciences.
  • 5.4. Explanations for AI Security: XAI and Adversarial Machine Learning: XAI creates confidentiality concerns because explanations may reveal proprietary model knowledge and trade secrets.
  • 5.4. Explanations for AI Security: XAI and Adversarial Machine Learning: Information revealed by XAI can enable more effective adversarial attacks while also supporting protection against private-content exposure.
  • 5.4. Explanations for AI Security: XAI and Adversarial Machine Learning: GANs, variational autoencoders, and other generative models have been used to explain data-based decisions by generating learned instances from noise input vectors.
  • 5.5. XAI and Output Confidence: Because erroneous outputs in autonomous vehicles, surgery, medical diagnosis, insurance, and manufacturing can cause harm, regulation seeks to prevent decisions based solely on data processing.

5.6. XAI, Rationale Explanation, and Critical Data Studies · 5.7. XAI and Theory-guided Data Science · 5.8. Guidelines for ensuring Interpretable AI Models

The paper connects XAI with rationale explanation, Critical Data Studies, and Theory-guided Data Science to make data-driven decisions and model knowledge more understandable and assessable. It also presents stakeholder- and context-sensitive guidelines for designing interpretable, ethical, safe, and responsible AI systems.

  • 5.6. XAI, Rationale Explanation, and Critical Data Studies: Reproducibility in Data Science requires sharing not only data, models, and results, but also the discourse surrounding collection, assumptions, construction, and analysis.The passage presents this broader documentation as necessary for transforming data into a valuable actionable asset.
  • 5.6. XAI, Rationale Explanation, and Critical Data Studies: XAI can explain why models reach decisions in accessible ways for non-experts, supporting multidisciplinary evaluation of the ethical implications of data-based choices.This connection is framed within Critical Data Studies, which emphasizes information exchange around data science practices and ethical appraisal.
  • 5.7. XAI and Theory-guided Data Science: Theory-guided Data Science combines Data Science with theoretical principles from the application context so models generate knowledge grounded in prior domain knowledge.This synergy implies that model types should be selected according to the relations being studied in the relevant context.
  • 5.7. XAI and Theory-guided Data Science: XAI can explain the knowledge captured by Theory-guided Data Science models, enabling assessment of compliance with theoretical principles known beforehand.The paper identifies this alignment between theory-guided requirements and explainability as a major opportunity for XAI.
  • 5.8. Guidelines for ensuring Interpretable AI Models: Interpretability efforts should account for the interests, demands, and requirements of all stakeholders, from system designers and decision makers to affected users.The process is described as multidisciplinary and inclusive rather than limited to model developers.
  • 5.8. Guidelines for ensuring Interpretable AI Models: Interpretability approaches should consider contextual factors, domain-specific needs, available resources, existing knowledge, task suitability, and the requirements of the intended audience.These factors include the model’s purpose, required explanation complexity, and the performance and interpretability of existing technologies.
  • 5.8. Guidelines for ensuring Interpretable AI Models: When black-box models are selected, designers should weigh ethics-, fairness-, and safety-related impacts and supplement systems with XAI when needed to mitigate them.The required level of explainability depends on the domain in which the system is deployed.
  • 5.8. Guidelines for ensuring Interpretable AI Models: Interpretability should be adapted to human cognitive capacities and limitations, including mental models, audience vocabulary, and the expertise needed to determine useful explanations.Future methodological work is expected to strengthen these guidelines while supporting non-discrimination, sustainability, privacy, accountability, and responsible AI use.

6. Toward Responsible AI: Principles of Artificial Intelligence, Fairness, Privacy and Data Fusion

AI principles are guidelines published by private and public organizations for developing and using AI responsibly. They address potential threats to individuals and society and connect XAI with broader responsible-AI principles.

  • Principles of Artificial Intelligence, Fairness, Privacy and Data Fusion: AI principles are guidelines from private and public organizations concerning how AI should be developed and used.They address potential threats posed by AI to individuals and society as a whole.
  • Principles of Artificial Intelligence, Fairness, Privacy and Data Fusion: The section links XAI, which commonly appears as one principle, to the wider set of recognized AI principles.It presents important and widely recognized principles to situate XAI within responsible AI.

6.1. Principles of Artificial Intelligence

AI-principles reviews classify organizational guidance by its nature, content and coverage, target audience, and relationship to International Human Rights. These principles emphasize fairness, explainability, human benefit, privacy and security, while 28 of 32 reviewed guides explicitly include XAI as crucial.

  • 6.1. Principles of Artificial Intelligence: AI-principles frameworks classify organizations by nature, principle content and coverage, target audience, and grounding in International Human Rights.Content may include eight principles, such as privacy, explainability, and fairness.
  • 6.1. Principles of Artificial Intelligence: A company-oriented example proposes five principles to guide AI development while allowing adoption by other organizations and businesses.The principles aim to reinforce inclusion, equal opportunities, and the common good.
  • 6.1. Principles of Artificial Intelligence: Fair AI requires avoiding discrimination and evaluating how systems affect protected groups alongside conventional error optimization.The stated groups include race, religion, gender, sexual orientation, disability, and ethnic origin.
  • 6.1. Principles of Artificial Intelligence: Explainability principles require disclosure of AI interaction and personal-data use, with user-profile-sensitive explanations supporting understanding of AI decisions.The passage also identifies human-centric AI and privacy and security by design as principles supporting human benefit and lifecycle safeguards.
  • 6.1. Principles of Artificial Intelligence: 28 out of the 32 AI principles guides reviewed explicitly include XAI as a crucial component.The European Commission’s Trustworthy AI guidance also uses an assessment checklist based on principles including human agency, robustness, privacy, transparency, diversity, and non-discrimination.

6.2. Fairness and Accountability

Fairness and accountability are intertwined with XAI in Responsible AI: XAI can reveal, measure, and mitigate bias while supporting auditability and reporting of negative impacts.

  • Fairness: XAI can expose implicit correlations between protected and unprotected features by analyzing model outputs and generating counterfactual explanations.Such analyses help designers identify hidden relationships that may cause discrimination.
  • Fairness: Fairness addresses bias affecting protected groups, including discrimination from sensitive factors such as race, age, and gender.Bias can arise from skewed or tainted data, limited features, sample-size disparities, and proxy features correlated with sensitive attributes.
  • Fairness: Group fairness can be evaluated through independence, separation, and sufficiency, although these criteria cannot all be fulfilled simultaneously.Independence requires prediction rates to be equal across subgroups; separation requires equal TP and FP rates; sufficiency requires equal Positive Predictive Value.
  • Fairness: Bias mitigation can occur before, during, or after training through pre-processing, in-processing, and post-processing techniques.Examples include Reweighing, Adversarial Debiasing, and Equalized Odds threshold adjustment.
  • Fairness and XAI: 26 out of the 28 AI principles addressing XAI also explicitly address fairness, showing that organizations commonly consider both aspects together.XAI proposals can additionally provide visual fairness reports that quantify bias, indicate fairness, and explain its causes.
  • Accountability: XAI supports accountability by improving auditability for different audiences, including regulators, and by helping minimize and report negative impacts.This connection follows from the documented relationship between fairness and XAI.

6.3. Privacy and Data Fusion

Data fusion can improve machine-learning performance by combining heterogeneous sources, but its centralized or distributed forms create intertwined privacy, fairness, and explainability challenges. XAI can clarify how sources interact, yet explanations themselves may expose sensitive information or enable unintended knowledge discovery.

  • Data fusion foundations: Data fusion combines heterogeneous information sources to improve machine-learning performance across applications such as industrial prognosis, cyber-physical social systems, and the Internet of Things.The paper distinguishes data-, model-, and knowledge-level fusion, as well as centralized and distributed processing.
  • Privacy requirements: Privacy must be addressed throughout an AI system’s lifecycle, including data governance, access protocols, data quality, integrity, and proper use of collected information.The privacy principle covers both user-provided information and information generated about users through system interactions.
  • Privacy and fairness: Data fusion is directly intertwined with privacy and fairness regardless of the fusion technique employed.The concern is especially important when systems process personal data or sensitive features that could support discrimination or unfair treatment.
  • Explainability opportunities: XAI techniques can analyze feature correlations to explain how deep-learning models fuse sources across spatial or temporal domains, potentially improving model usability.Such analysis may reveal correlations induced by the predictive task among the contributing data sources.
  • Explainability risks: Explanations of fused models may compromise confidentiality by enabling sensitive information, such as ownership, to be inferred from relationships among protected and unprotected features.This risk applies to explanations of deep-learning fusion and to post-hoc analysis of model-level fusion.
  • Distributed fusion: Distributed fusion introduces additional privacy and explainability challenges because information may be shared or inferred across sources, models, or architectures despite not being directly disclosed.The paper contrasts these concerns with centralized fusion, which adds no further concerns beyond those already discussed.

6.4. Implementing Responsible AI Principles in an Organization

Implementing Responsible AI principles requires translating AI-specific values into organizational practice while leveraging existing processes for privacy, security, and safety. Effective implementation balances cultural and organizational change with feasibility and compliance within available IT assets, policies, and resources.

  • Principle types: Organizations distinguish AI-specific principles, such as explainability, fairness, and human agency, from end-to-end principles that also include privacy, security, and safety.The EC Guidelines for Trustworthy AI exemplify end-to-end principles, whereas Telefonica’s principles are more AI-specific.
  • Implementation methodology: Implementation should operationalize AI-specific principles while leveraging existing organizational processes for generic principles such as privacy, security, and safety.A methodology is needed to break implementation into distinct components.
  • Implementation methodology: A Responsible AI methodology combines values and boundaries, awareness and training, impact questionnaires, mitigation tools, and governance assigning responsibilities and accountabilities.Relevant tools include XAI tools, fairness tools, and model cards; governance may use committees or employee self-responsibility.
  • Organizational balance: Companies should balance major cultural and organizational changes with implementation feasibility and compliance across existing IT assets, policies, and resources.The passage frames this balance as central to implementing Responsible AI principles over processes containing AI functionalities.

7. Conclusions and Outlook

The overview establishes XAI as essential for adopting machine-learning methods in real-life applications and reviews explainability through conceptual clarification and systematic literature analysis. It extends this discussion toward Responsible AI, emphasizing fairness, transparency, privacy, accountability, and the joint consideration of interpretability with data and model constraints.

  • Conclusions and Outlook: XAI is identified as an utmost need for adopting ML methods in real-life applications.The overview centers on explainability as a prerequisite for practical ML adoption.
  • Conclusions and Outlook: The study clarifies model-explainability concepts and purposes before systematically reviewing recent literature from different perspectives.One perspective concerns ML models with some inherent transparency and interpretability.
  • Conclusions and Outlook: Responsible AI requires practical AI implementations to meet principles including fairness, transparency, and privacy.The overview also discusses how XAI in data fusion can compromise the privacy of protected data.
  • Conclusions and Outlook: Future XAI work must jointly address interpretability with data privacy, model confidentiality, fairness, and accountability.The authors stress understanding both the potentiality and caveats of XAI techniques for responsible organizational and institutional use.
Loading 1910.10045v2…