Source-linked AI summary
Stakeholders in Explainable AI
Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, Supriyo Chakraborty
TL;DR
Explainability and interpretability lack settled meanings because distinct stakeholder communities seek different things from AI explanations. The paper separates these motives using verification versus validation and knowns versus unknowns, then proposes layered explanations combining transparency with post-hoc representations. It also notes unresolved limits, including unstable attribution methods and the difficulty of formally specifying deep-network behaviour.
Problem
Explainability terminology and requirements vary because AI involves distinct stakeholder communities with different motivations and concerns.
Source: Introduction · pp. 1–2Method
The paper analyzes stakeholder communities through software engineering’s verification-validation distinction and epistemological categories of knowns and unknowns.
Source: Engineering and Epistemological… · p. 3, Conclusion · p. 5Results
The paper argues that stakeholders’ motives and requirements are not identical and suggests layered explanations combining transparency with local and global post-hoc representations.
Source: Conclusion · p. 5, Layered explanations · p. 5Takeaways & Limitations
Different stakeholders may require different explanation content, while a layered explanation object could package information for multiple recipients.
Source: Layered explanations · p. 5, Conclusion · p. 5Attribution methods may be unstable or unreliable, and formally specifying intended deep-neural-network behaviour remains an open question.
Source: Transparency-based explanations · p. 4, Engineering and Epistemological… · p. 4
Abstract
from arXiv · showhide
There is general consensus that it is important for artificial intelligence (AI) and machine learning systems to be explainable and/or interpretable. However, there is no general consensus over what is meant by 'explainable' and 'interpretable'. In this paper, we argue that this lack of consensus is due to there being several distinct stakeholder communities. We note that, while the concerns of the individual communities are broadly compatible, they are not identical, which gives rise to different intents and requirements for explainability/interpretability. We use the software engineering distinction between validation and verification, and the epistemological distinctions between knowns/unknowns, to tease apart the concerns of the stakeholder communities and highlight the areas where their foci overlap or diverge. It is not the purpose of the authors of this paper to 'take sides' - we count ourselves as members, to varying degrees, of multiple communities - but rather to help disambiguate what stakeholders mean when they ask 'Why?' of an AI.
Introduction
Explainability has longstanding roots in expert systems, but modern AI has revived unresolved questions about what explanations should mean and whom they should serve. The paper frames differing terminology and stakeholder goals as central sources of disagreement.
- Expert-system research identified explanation needs for system testing and for building end-user trust.Source: Introduction · p. 1
- Early symbolic execution traces were difficult for developers to interpret and largely unintelligible to subject-matter experts.Source: Introduction · p. 1
- Deep neural networks renewed interest in explainability, with modern usage often emphasizing human interpretation of learned models.Source: Introduction · p. 1
- Terminology varies across transparency, interpretability, explainability, intelligibility, and legibility, while accountability and fairness add ethical and legal concerns.Source: Introduction · pp. 1–2
- The paper distinguishes stakeholder communities and their motivations to refine the classical developer-versus-end-user distinction.Source: Introduction · p. 2
Four Stakeholder Communities
The paper identifies four stakeholder communities whose explainability needs differ: developers, theorists, ethicists, and users. Their motives range from quality assurance and advancing AI theory to evaluating societal impacts and guiding action.
- Developers seek explainability primarily for testing, debugging, evaluation, and improving application robustness.Source: Developers · p. 2
- Theorists focus on understanding fundamental properties and theoretical limits of AI, especially deep neural networks, rather than delivering applications.Source: Theorists · p. 2
- Ethicists, including policymakers, critics, and interdisciplinary professionals, focus on fairness, accountability, and transparency alongside explainability and interpretability.Source: Ethicists · pp. 2–3
- Users need explanations to decide whether and how to act on system outputs or to justify actions affected by AI systems.Source: Users · p. 3
Engineering and Epistemological Perspectives
Verification concerns whether an AI system is built right, whereas validation concerns what it does; the paper maps these concerns onto stakeholder communities and kinds of knowledge. Formal specification and robustness remain difficult, especially for adversarial and unknown cases.
Engineering and Epistemological Perspectives
- Transparent techniques mainly support verification, while post-hoc techniques are often useful for validation.Source: Engineering and Epistemological… · p. 3
- Developers and theorists tend to prioritize verification, while users and ethicists tend to prioritize validation, with corresponding preferences for transparency-based and post-hoc explanations.Source: Engineering and Epistemological… · p. 3
- Known knowns are training and test data, while known unknowns are intended queries, predictions, or behaviours evaluated through system testing.Source: Engineering and Epistemological… · p. 3
- Unknown knowns include biases outside the system’s scope, and validation is important for revealing them.Source: Engineering and Epistemological… · p. 3
- All four communities have reasons to address unknown unknowns, including robustness, theory, ethical and legal implications, and impacts on people’s livelihoods.Source: Engineering and Epistemological… · p. 3
- Formal specification of intended deep-neural-network behaviour remains an open question, and existing mitigations do not formally verify adversarial-input behaviour.Source: Engineering and Epistemological… · pp. 3–4
Explanation Types and Discussion
The paper distinguishes transparency-based explanations from purely post-hoc approaches and proposes layering them to address different stakeholder needs. It also emphasizes that attribution explanations can be unstable, unintelligible, or overwhelming.
Transparency-based explanations:
- Transparency-based explanations derive from model activations or inner states, while purely post-hoc approaches do not.Source: Transparency-based explanations · p. 4
- Figure 1’s saliency map highlights input regions most significant for classifying a traffic image as congested.Source: Transparency-based explanations · p. 4
- Attribution methods may be unstable or unreliable, limiting their immediate value despite relevance to verification.Source: Transparency-based explanations · p. 4
- Post-hoc explanations include local approximations, examples, and representations that relate input and output features.Source: Post-hoc explanations · pp. 4–5
Layered explanations:
- A layered explanation object could combine traceability, justification, and assurance for different recipients.Source: Layered explanations · p. 5
Conclusion
The paper separates stakeholder motives and requirements for explainable AI, while suggesting layered explanations that combine transparency with local and global post-hoc representations. It also emphasizes users as the most influential but least represented stakeholder community.
- Stakeholder communities have different motives and requirements for explainable AI.Source: Conclusion · p. 5
- A layered approach combining transparency with local and global post-hoc representations may serve multiple stakeholders.Source: Conclusion · p. 5
- Users are described as the most influential stakeholder community despite being barely represented in the literature.Source: Conclusion · p. 5
- Failure to satisfy users may contribute to another AI Winter through unfulfilled expectations or a smaller-than-hoped-for market.Source: Conclusion · p. 5