Source-linked AI summary
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Gabrielle Ras, Marcel van Gerven, Pim Haselager
TL;DR
The chapter asks what can be explained about DNNs given users’ needs, laws and regulations, and existing explanation methods. It classifies and evaluates explanation methods, finding that visual explanations can describe input influence, while lay-user interfaces and direct solutions to bias and unfair outcomes remain lacking.
Problem
DNN processes and outcomes are difficult to inspect, while users need explanations addressing functional, ethical and data-related concerns.
Method
The chapter identifies user concerns, analyzes relevant GDPR provisions, introduces a taxonomy of explanation methods, and evaluates their desirable properties.
Results
Rule-extraction and attribution methods provide visual explanations about various aspects of how inputs influence outputs, and intrinsic methods improve explainability when combined with attribution.
Takeaways & Limitations
Current methods are tailored to experts, while intuitive explanation interfaces for lay users are not available.
Takeaways & Limitations
Explanation methods do not directly solve dataset bias leading to biased DNNs or suspicions about unfair outcomes.
Abstract
from arXiv · showhide
Issues regarding explainable AI involve four components: users, laws & regulations, explanations and algorithms. Together these components provide a context in which explanation methods can be evaluated regarding their adequacy. The goal of this chapter is to bridge the gap between expert users and lay users. Different kinds of users are identified and their concerns revealed, relevant statements from the General Data Protection Regulation are analyzed in the context of Deep Neural Networks (DNNs), a taxonomy for the classification of existing explanation methods is introduced, and finally, the various classes of explanation methods are analyzed to verify if user concerns are justified. Overall, it is clear that (visual) explanations can be given about various aspects of the influence of the input on the output. However, it is noted that explanation methods or interfaces for lay users are missing and we speculate which criteria these methods / interfaces should satisfy. Finally it is noted that two important concerns are difficult to address with explanation methods: the concern about bias in datasets that leads to biased DNNs, as well as the suspicion about unfair outcomes.
1 Introduction
The chapter examines explainability for increasingly adopted DNN-based automation, aiming to bridge expert and lay users while evaluating explanations in a broader sociotechnical context.
- The chapter aims to bridge expert and lay users by identifying their explanation needs and analyzing current explainability.
- DNNs increasingly derive categorizations, predictions and decisions from large quantities of user-collected digital information.
- Explainable AI is evaluated through four interacting components: users, algorithms, laws and explanations.
- Unlike decision trees or Bayesian networks, DNN decisions cannot be explained by simply scrutinizing inference because their representations and information flows are difficult to analyze.
2 Users and their concerns
The chapter distinguishes expert and lay users whose values generate concerns and questions about DNN functionality, outcomes, ethics and legality.
- User categories and concerns: Users hold ethical and functional values that generate concerns such as discrimination or accuracy and questions about factors influencing outcomes or data reliability.
- Expert users: Expert users include DNN engineers, who seek functional explanations about network performance and debugging.
- Expert users: DNN developers focus on whether integrated applications meet their goals and on understanding DNN behavior across use cases.
- Lay users: Lay users need not understand DNN implementation or mathematical principles and include application owners, end users, data subjects and stakeholders.
- Lay users: End users seek explanations of application capabilities and behavior, while data subjects focus mainly on ethical and moral aspects of actionable outcomes.
- Lay users: Stakeholders without direct involvement may request information about DNN operations and consequences, especially concerning ethical and legal issues.
Case study: autonomous driving
The autonomous-driving case study maps the roles of different DNN users and stakeholders across perception, planning, vehicle ownership, use, affected entities and regulation.
- A DNN engineer develops object segmentation and classification from raw video, producing object types and locations.
- A DNN developer integrates the deep-learning output with other components into a planning system that determines the car’s actions.
- The owner acquires the planning system and deploys it in a vehicle, while the end user purchases and drives the car.
- Data subjects include pedestrians, houses and other cars whose information is captured along the route.
- Governmental institutions and insurance companies are stakeholders because they regulate autonomous vehicles or assess associated risks and consequences.
3 Laws and regulations
The GDPR supplies relevant categories and obligations for DNN-related personal-data processing, including profiling and automated decision-making.
- The GDPR distinguishes personal data, data subjects, data processors and data controllers in the context of personal-data processing.
- Personal data means information relating to an identified or identifiable natural person, while processors act on behalf of controllers determining processing purposes and means.
- A DNN may function as a tool for the data processor, whereas application owners or end users may serve as data controllers.
- The GDPR defines profiling as automated processing of personal data used to evaluate or predict personal aspects of a natural person.
- For automated decision-making involving collected personal data, data subjects have access rights and controllers must provide meaningful information about the processing.
4 Explanation
Explanations make DNN outcomes understandable to users and support accountability, but they can also create privacy and manipulation risks. The chapter distinguishes explanations for system dysfunction from local explanations requested about potentially undesired outcomes.
- Transparency means making actionable DNN outcomes understandable or comprehensible to particular users, without requiring them to accept those outcomes as valid.
- Transparency can support accountability by helping attribute responsibility for outcomes to legally or morally relevant agents.
- Transparency may negatively affect privacy or enable manipulation of data, processing, or training.
- Explanations for dysfunction help experts assess, predict, monitor, diagnose, improve, debug, or repair a DNN’s basic functioning.
- Local explanations address suspected misfunctioning by identifying inputs, learning factors, or system parameters that influenced a particular outcome.
5 Explanation methods
The chapter bridges user and technical perspectives by defining desirable properties for explanation methods, classifying methods in a taxonomy, and assessing those properties across method categories.
- The authors identify high-level desirable properties of explanation methods to connect users’ needs with technical evaluation.
- They introduce a taxonomy covering types of explanation methods and assess whether its categories contain the identified desirable properties.
5.1 Desirable properties of explainers
The chapter evaluates explainers using fidelity, interpretability, generalizability, and explanatory power. Interpretability is divided into clarity and parsimony, reflecting both comprehensibility and explanation complexity.
- High fidelity measures how closely an explanation agrees with the DNN’s input-output mapping and is described as the most important explainer property.
- High interpretability concerns whether users gain true insight into how actionable outcomes are obtained.
- Interpretability: High clarity requires explanations to be unambiguous, especially in safety-critical applications where ambiguity should be avoided.
- Interpretability: High parsimony means producing simple explanations, with the appropriate complexity partly depending on users’ capabilities.
- High generalizability measures the range of architectures to which an explanation method applies, with model-agnostic methods having the highest generalizability.
- High explanatory power measures how many phenomena or user questions a method can explain, including model behavior from a global perspective.
5.2 A taxonomy for explanation methods
The taxonomy identifies rule-extraction, attribution, and intrinsic methods. These classes differ in how they represent DNN behavior, their fidelity and interpretability characteristics, their generalizability, and whether interpretability is added during training or architecture design.
- The proposed taxonomy organizes explanation methods into three main classes derived from historical and contemporary trends in DNN interpretation.
- Attribution methods: Attribution methods measure component importance by changing inputs or internal components and recording effects on model performance; related terms include occlusion, perturbation, erasure, ablation, and influence.
- Rule-extraction methods: Rule-extraction methods produce human-interpretable rules approximating a DNN’s decision-making process from its inputs and outputs.
- Rule-extraction methods: Decompositional rule extraction breaks a network into parts, can extract layer-wise rules, and may have limited generalizability because it uses architecture information.
- Rule-extraction methods: Pedagogical rule extraction treats the network function as a learning target and is inherently model-agnostic.
- Rule-extraction methods: Rule-extraction methods can validate overall logic flow and explain input aspects, while local rules are more faithful than global rules and parsimony depends on ruleset size and user capabilities.
- Attribution methods: Attribution visualizations represent important features through feature, relationship, or process visualization, but reliability requires input invariance and visual explanations can remain ambiguous.
- Intrinsic methods: Intrinsic methods modify DNN architectures, losses, or modules to improve interpretability of internal representations and increase fidelity, clarity, and parsimony in attribution methods.
6 Addressing general concerns
General concerns about DNNs involve data quality and bias, opaque data use, correlational inference, value-sensitive outcomes, adversarial vulnerability, and limits of explanation methods.
- Flawed labeling, noise, censoring, and annotator characteristics can reduce data quality and introduce unwanted biases into DNN training.
- Opaque data use makes it difficult to know which personal data contributed to an outcome or how expert-user biases influenced training.
- DNN inferences are correlational rather than causal, so subtle feature correlations and associated biases can influence network outputs.
- Users may regard outcomes as undesirable when they conflict with values such as fairness or privacy, requiring stakeholder preferences to inform actionable outcomes.
- Imperceptible input distortions can induce incorrect inferences, and existing methods do not fully eliminate adversarial-attack effects.
- Explanation methods can improve model comprehensibility but do not completely resolve concerns about DNN reliability, bias, or fairness.
7 Discussion
The discussion finds that existing methods explain several aspects of input influence on DNN outputs, but primarily serve experts and cannot directly solve dataset bias or unfair-outcome concerns.
- The chapter analyzes what can be explained by relating user needs, laws and regulations, and existing explanation methods in the context of DNNs.
- Rule-extraction and attribution methods provide visual explanations of how input aspects influence DNN outputs.
- Intrinsic methods combined with attribution methods lead to more explainable DNNs, while future methods may combine rule-extraction, attribution, and intrinsic approaches.
- Current explanation methods are tailored to experts because interpreting their results requires knowledge of the DNN process; lay-user interfaces are not available.
- Explanation methods cannot directly solve dataset bias or suspected unfair outcomes, although visualized learned features can support separate bias analysis.