Source-linked AI summary
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, Richard Jiang, Hassan Khosravi, Freddy Lecue, Gianclaudio Malgieri, Andrés Páez, Wojciech Samek, Johannes Schneider, Timo Speith, Simone Stumpf
TL;DR
Opaque AI systems have made understanding model behavior increasingly important, while XAI still lacks settled approaches to several foundational and practical questions. The paper synthesizes perspectives from multiple disciplines into a manifesto of 27 open problems across nine categories, with proposed research directions. It frames this agenda as a collaborative roadmap for advancing XAI in practical applications.
Problem
Opaque AI systems are flourishing across real-world applications, creating a need to understand black-box models and address unresolved XAI challenges.
Method
The paper brings together experts from philosophy, psychology, HCI, and computer science to synthesize 27 XAI problems into nine categories and propose research directions.
Results
The paper presents a manifesto of 27 open problems, organized into nine categories, together with promising directions for future research.
Takeaways & Limitations
The manifesto offers an interdisciplinary proposal and roadmap for discussion, collaboration, and continued advancement of XAI.
Takeaways & Limitations
Human studies of XAI may suffer from unrepresentative samples, bias, poor reproducibility, and inappropriate statistical analyses.
Abstract
from arXiv · showhide
As systems based on opaque Artificial Intelligence (AI) continue to flourish in diverse real-world applications, understanding these black box models has become paramount. In response, Explainable AI (XAI) has emerged as a field of research with practical and ethical benefits across various domains. This paper not only highlights the advancements in XAI and its application in real-world scenarios but also addresses the ongoing challenges within XAI, emphasizing the need for broader perspectives and collaborative efforts. We bring together experts from diverse fields to identify open problems, striving to synchronize research agendas and accelerate XAI in practical applications. By fostering collaborative discussion and interdisciplinary cooperation, we aim to propel XAI forward, contributing to its continued success. Our goal is to put forward a comprehensive proposal for advancing XAI. To achieve this goal, we present a manifesto of 27 open problems categorized into nine categories. These challenges encapsulate the complexities and nuances of XAI and offer a road map for future research. For each problem, we provide promising research directions in the hope of harnessing the collective intelligence of interested stakeholders.
1 Introduction
XAI has grown from a niche AI topic into an active, multidisciplinary research field as machine learning succeeds across real-world applications. This paper organizes future research through an interdisciplinary manifesto of open problems.
- XAI has expanded from a niche AI topic into a highly active field with theoretical, empirical, and review contributions.
- Research on XAI now spans computer science and disciplines including engineering, chemistry, biology, education, psychology, neuroscience, and philosophy.
- Machine learning and deep learning have gained real-world use by extracting patterns from complex, non-linear data for classification, forecasting, prediction, recommendation, and generation.
- The paper brings together experts from philosophy, psychology, HCI, and computer science to formulate 27 XAI problems across nine categories.
- The manifesto is presented as an open proposal intended to foster debate about future XAI research and applications.
2 Advances and Applications of xAI Research
The paper surveys XAI advances, applications, and interpretability approaches while identifying unresolved challenges in making models and explanations useful to stakeholders. It emphasizes that interpretable modeling remains difficult when accuracy and model simplicity must be balanced.
- XAI research includes synthesized explanations, attribution methods, interpretable models, and applications across real-world domains.
- The paper presents XAI advances and applications as evidence of the field’s practical utility while noting that further research is needed.
- Attribution methods such as LIME and SHAP estimate input-feature importance to support interpretation of deep-learning models.
- Interpretable Models: Limiting a decision tree’s complexity while maintaining high accuracy through rule extraction remains an open problem.
- Attention-based explanations can improve interpretability and efficient tabular-data computation, but remain sensitive to attention variability in transformers.
2.2 Applications of XAI Methods
XAI methods are applied across finance, education, environmental and agricultural science, and medicine. These applications support domain-specific decisions, personalization, ecosystem management, and institutional or legal needs for transparency.
- XAI has been applied in finance, education, environmental science and agriculture, and medicine and health care.
- In medicine, explanations can help clinicians understand AI-supported decisions and calibrate trust and reliance when diagnoses or treatments carry significant consequences.
- Financial institutions use AI for credit-risk prediction, fraud detection, and investment-portfolio optimization, where transparency and explainability may be legally required.
- Agricultural and forest-ecosystem applications use drones and machine learning to support analysis, modeling, and management relevant to forest carbon stock.
- AI-powered educational systems can identify learners’ strengths and weaknesses and provide customized instructions and resources.
3 Challenges and Research Directions
Despite substantial progress and applications, XAI still faces open questions about evaluation, terminology, and its relationship to trustworthiness. Existing surveys address these issues but remain scattered.
- More research is needed to address open problems in XAI despite its advances, breakthroughs, and potential applications.
- It remains unclear how XAI methods should be evaluated, how key terms should be used, and how XAI relates to trustworthiness.
- Existing surveys address some XAI aspects but are described as scattered across conference proceedings and journals.
3.1 Creating Explanations for New Types of AI
New AI forms, especially generative and concept-based models, create explanation challenges that existing XAI methods do not yet adequately address. The paper proposes new directions including mechanistic interpretability, information geometry, training constraints, and hybrid concept-based approaches.
- Generative and Large Language Models: Generative models’ immense scale and polysemantic neurons challenge XAI methods through high-dimensional computation and difficult concept extraction.Existing methods have mostly targeted classification and regression, so generative models require substantially different approaches.
- Generative and Large Language Models: Mechanistic interpretability offers a route to study the functioning and scaling laws of generative models, although results remain strongest at small scales and on toy problems.Information geometry and model-design constraints are identified as complementary directions for high-dimensional and safety-related challenges.
- Concept-Based Learning Algorithms: Concept-based algorithms learn prototypical features directly, but effective XAI methods for these models do not yet exist.Examples include ProtoPNet, ProtoTree, Concept Bottleneck Models, Concept Activation Vectors, Concept Embedding Models, and Concept Atlases.
- Concept-Based Learning Algorithms: Genetically evolvable connections between identifiable concepts could combine object detection with symbolic classifiers that are interpretable and algorithmically transparent.The proposed hybridization is suited to datasets containing discriminative, concept-wise compositional information.
3.2 Improving Current XAI Methods
Current XAI methods face limitations involving attribution sensitivity, synthesis artifacts, and explanation fragility. The paper highlights portfolio methods, improved synthesis validation, and robustness evaluation as possible responses.
- Augmenting and Improving Attribution Methods: Pixel-attribution methods can be unsuitable for laypersons and sensitive to baselines, interface settings, model assumptions, perturbation ranges, sampling intervals, and network layers.These limitations affect methods such as LIME, SHAP, gradient-based approaches, and relevance propagation.
- Augmenting and Improving Attribution Methods: Combining attribution with complementary XAI approaches could hedge the weaknesses of individual methods and produce negotiated explanations or competing hypotheses.Mechanistic interpretability is presented as one orthogonal component of such a portfolio.
- Removing Artifacts in Synthesis-Based Explanations: Synthesized explanations can contain artifacts whose origin—generation defects or learned concepts—may be unclear.The issue arises when generated examples are used to represent strongly activating concepts or neurons.
- Removing Artifacts in Synthesis-Based Explanations: Diffusion models may reduce synthesis artifacts, while reconstructing the original input can provide a reference for detecting distortions.Even state-of-the-art generative models do not guarantee artifact-free explanations.
- Creating Robust Explanations: Small input perturbations and inconsistent synthesized explanations make robust explanations difficult, despite their proposed role in calibrating trust and improving models.The paper points to benchmark evaluation, aggregation, uncertainty quantification, and reveal-to-revise procedures as directions.
3.3 Evaluating XAI Methods and Explanations
XAI evaluation lacks a gold standard and often neglects how users interact with explanations. The paper calls for interdisciplinary user research, standardized evaluation, and methods that address the limitations of human studies.
- Evaluation Foundations: No gold standard defines what makes an XAI explanation good, making evaluation a complex development and deployment problem.Existing approaches may assess properties of explanation methods without considering final users.
- User-Centered Evaluation: Interdisciplinary user studies involving machine learning, HCI, psychology, and social science researchers are proposed as a foundation for XAI evaluation.Standardization is also identified as necessary to streamline evaluation.
- Evaluation Frameworks: Existing evaluation work includes psychometric measures, hierarchical taxonomies, best-practice procedures, conceptual properties, quantitative metrics, and the Quantus framework.Quantus implements more than 30 metrics across six categories.
- Limitations of Human Studies: Human evaluations may suffer from unrepresentative samples, bias, poor reproducibility, inappropriate statistics, and weak evidence for usefulness.Synthetic data and virtual participants are proposed to broaden demographic, behavioral, and preference coverage.
3.4 Clarifying the Use of Concepts in XAI
XAI research is constrained by ambiguous terminology, contested links between explainability and trustworthiness, and unresolved accounts of understanding. The paper proposes clearer terminology and continued conceptual and psychological work.
- Terminological Clarity: Ambiguity among explainability, interpretability, transparency, understanding, explicability, perspicuity, and intelligibility can hinder development of effective XAI systems.The paper recommends addressing communication challenges holistically and moving toward more uniform terminology.
- Trustworthiness: Trustworthiness-related concepts such as safety, fairness, and accountability are frequently invoked in regulatory discussions but require clearer conceptual treatment.The paper distinguishes trustworthiness as a system property, trustworthy AI as an enabler, and technical requirements for trustworthy systems.
- Trustworthiness: XAI is identified as one of seven requirements for trustworthy AI, alongside properties including human oversight, robustness, privacy, and fairness.The NIST AI RMF is also described as organizing trustworthiness around characteristics such as reliability, safety, accountability, explainability, privacy, and fairness.
- Accounts of Understanding: There is no consensus on whether understanding requires true explanations, can arise from distorted accounts, or may involve partly false information.Different philosophical positions also distinguish understanding from singular predictions, surrogate models, and functional input-output correlations.
- Accounts of Understanding: A useful account of understanding requires conceptual clarity together with responsiveness to psychological evidence about human-computer interaction.Recent work has clarified the conceptual map, but future developments must incorporate new psychological findings.
3.5 Supporting the Multi-Dimensionality of Explainability
Explainability is multi-dimensional, spanning trustworthiness requirements and multiple disciplines. The paper proposes combining trustworthiness metrics with centralized knowledge sharing and standardized terminology.
- Explainability has multiple facets and spans a variety of disciplines.
- Regulatory explanations should incorporate trustworthy-AI requirements, including safety, fairness, and accountability.The paper cautions against investing heavily in explanations for inaccurate, unlawful, or unfair models.
- Solution Ideas: Trustworthiness metrics could combine safety, fairness, and accountability to tailor explanation detail to system trustworthiness.The proposal uses thresholds to trigger more extensive explanations for less trustworthy systems.
- Interdisciplinary XAI work is hindered by publication overload and inconsistent terminology across fields.Researchers may struggle to engage across disciplines because each field has its own large literature and vocabulary.
- Solution Ideas: A centralized platform with curated research and standardized glossaries could make cross-disciplinary XAI research more manageable.
3.6 Supporting the Human-Centeredness of Explanations
Human-centered XAI must produce explanations that fit how people understand concepts, rather than relying only on technical feature attributions. The paper highlights reality drift and causality as limits on explanation usefulness and actionability.
- Creating Human-Understandable Explanations: Human-understandable explanations should be social, contrastive, and selective.
- Creating Human-Understandable Explanations: Pixel-level attribution heatmaps may be incomprehensible to laypersons and inadequate for complex concept-based distinctions.The paper notes that such heatmaps often remain within the input-data domain and may depend on expert intuition.
- Solution Ideas: Concept-based XAI can express predictions through meaningful concepts, such as whisker shape, rather than isolated pixel relevance.
- Creating Human-Understandable Explanations: AI systems may rely on features that humans find difficult to grasp, while humans typically reason with coarse-grained concepts.
- Addressing Explanations Divorced From Reality: Increasingly complex systems can create reality drift, making explanations plausible to humans yet detached from actual reality.The paper questions whether such explanations remain useful for AI safety.
- Uncovering Causality for Actionable Explanations: Causal and counterfactual explanations are needed when model outputs guide actions affecting people.Posthoc methods may not disentangle learned correlations from causation; counterfactuals instead ask what would need to change for a different prediction.
3.7 Adjusting XAI Methods and Explanations
XAI methods and explanations must be adjusted to stakeholders, domains, contexts, and goals. Proposed directions include semantic enrichment, personalization, interaction, and domain-specific models and standards.
- Adjusting Explanations to Different Stakeholders: Explanations may need different content, formats, and presentations during AI development, evaluation, and use.
- Adjusting Explanations to Different Stakeholders: Different stakeholders require explanations tailored to their preferences, abilities, experiences, and objectives.Business, technical, and financial stakeholders may need the same facts presented around different concerns.
- Solution Ideas: Semantic enrichment could combine XAI methods with training data, ontologies, and other modalities.The paper also proposes personalized and interactive explanations that users can refine through interaction.
- Adjusting Explanations to Different Domains: Explanation requirements differ across domains because assumptions, environments, expectations, and stakes differ.The paper contrasts self-driving cars, where regulation matters, with clinical systems, where patient details are crucial.
- Solution Ideas: Domain-specific models, terminology, contextual reasoning, and standards could support meaningful context-aware explanations.
- Adjusting Explanations to Different Goals: Explanation goals vary among model developers, regulators, and applicants, requiring stakeholder-centric strategies.Examples include improving accuracy, assessing fairness, and understanding loan rejection reasons.
3.8 Mitigating the Negative Impact of XAI
XAI can produce harmful or misleading support, create accountability problems, and face fundamental limits in safety-critical or adversarial settings. The paper therefore emphasizes failure detection, falsifiability, and treating explainability as only one part of safety.
- Mitigating Failed Support by XAI: Ineffective XAI support can be harmful, including when correct AI advice is paired with an explanation that misleads decision makers.
- Solution Ideas: Failure situations should be detected and labeled reliably, with XAI support withheld when it is detrimental or irrelevant.
- Devising Criteria for the Falsifiability of Explanations: Incorrect explanations can misassign accountability, while unclear falsifiability prevents reliable correctness benchmarks.
- Solution Ideas: Falsifiability criteria could support rigorous empirical examination of explanations through hypothesis testing and experimentation.
- Securing Explanations from Being Abused by Malicious Superintelligent Agents: Explainability is important for AI safety, but increasingly capable systems may become fundamentally impossible for humans to understand.The paper also raises concerns about unverifiability and deliberate deception in adversarial scenarios.
- Solution Ideas: Explainability should be only one component of a safety toolkit because its effectiveness may be limited in some scenarios.The paper conditions longer-term effectiveness on resolving alignment between technical XAI capabilities and human utility.
3.9 Improving the Societal Impact of XAI
XAI can affect society by exposing how AI systems produce outputs, but transparency alone does not resolve unfairness or structural power imbalances. The paper therefore discusses explainability challenges involving generative-model originality, data deletion, and participatory design.
- Societal impact: XAI methods and applications can affect society, making their societal impact a central concern.The paper frames explainability as relevant to real-world AI systems and their consequences for society.
- Generative AI: Generative-model explanations must address originality attribution, plagiarism detection, and intellectual-property questions surrounding AI-generated content.The proposed direction includes examining which modeled-data instances or regions are relevant to a synthesized output.
- Data governance: Large-scale generative models create a data-governance challenge because users may seek to have their data forgotten by the model.Similarity-based explanations, incremental retraining, and influence tracing are proposed to support identifying and unlearning related data.
- Fairness and participation: Explanations may support individual understanding and contestation, but they do not by themselves address unfairness or the structural power imbalance between AI controllers and affected people.The paper proposes participative design and participative impact assessment that include vulnerable impacted stakeholders.
4 A Novel Manifesto
The paper concludes with a manifesto that defines open XAI challenges and propositions for independent scientific research. It organizes 27 problems into research directions spanning new AI types, evaluation, human-centeredness, adaptation, and collaboration.
- Manifesto purpose: The manifesto defines and succinctly describes open challenges facing XAI scholars while proposing propositions governing independent scientific research.It is presented as a mechanism for shaping shared visions about science in XAI.
- Open problems: The 27 problems are organized into nine categories covering new AI types, improved methods, evaluation, concepts, multidimensionality, human-centeredness, adaptation, and related directions.The categories include generative and concept-based explanations, robust and evaluated explanations, causality, stakeholder adaptation, and interdisciplinary work.
- Interdisciplinary collaboration: The manifesto presents collaboration across scholars with different scientific backgrounds as a way to increase reliability, currency, productivity, and falsifiability.Its authors describe the manifesto as an attempt to build on diverse expertise and experiences.