Source-linked AI summary
Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda
Johannes Schneider
TL;DR
As GenAI produces increasingly complex outputs, explainability must address new challenges beyond those of pre-GenAI AI. This paper surveys existing work, derives desiderata and a taxonomy for GenXAI, and proposes a research roadmap while noting important access and faithfulness limitations.
Problem
GenAI’s complex, widespread outputs create a need to understand and explain generated content, including its correctness and potential hallucinations.
Method
The paper conducts a narrative review and meta-survey, develops a taxonomy of GenXAI dimensions, and examines XAI avenues from training data to prompting.
Results
The paper identifies novel explanation desiderata including verifiability, interactivity, security, and cost, and proposes a research agenda with multiple future directions.
Takeaways & Limitations
GenXAI research should address explanation scope and interactivity while extending techniques to multimodal and emerging GenAI applications.
Takeaways & Limitations
Some explanations may hallucinate and lack faithfulness to true model workings, while commercial models often provide only black-box access and withhold training information.
Abstract
from arXiv · showhide
Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs, objectives, and possibilities have emerged for explainability (XAI). In this work, we elaborate on why XAI has gained importance with the rise of GenAI and its challenges for explainability research. We also unveil novel and emerging desiderata that explanations should fulfill, covering aspects such as verifiability, interactivity, security, and cost. To this end, we focus on surveying existing works. Furthermore, we provide a taxonomy of relevant dimensions that allows us to better characterize existing XAI mechanisms and methods for GenAI. We discuss different avenues to ensure XAI, from training data to prompting. Our paper offers a short but concise technical background of GenAI for non-technical readers, focusing on text and images to better understand novel or adapted XAI techniques for GenAI. However, due to the vast array of works on GenAI, we decided to forego detailed aspects of XAI related to evaluation and usage of explanations. As such, the manuscript interests both technically oriented people and other disciplines, such as social scientists and information systems researchers. Our research roadmap provides more than ten directions for future investigation.
1 Introduction
GenAI’s shift from recognition to generation has increased the need for explainability, while creating new challenges and requirements for GenXAI. The paper responds with a narrative review, taxonomy, and research agenda.
- GenAI generates text, images, audio, 3D solutions, and videos controllable through human inputs such as textual prompts.
- Explanations can support understanding, trust, debugging, and verification of generated content, including efforts to combat hallucinations.
- The paper identifies key challenges and desiderata for GenXAI rather than only listing and structuring existing XAI techniques.
- It uses a narrative review methodology accompanied by a taxonomy development approach from information systems research.
- The authors build a meta-survey on pre-GenAI XAI research while incorporating interdisciplinary and social-science perspectives.
- The paper derives desiderata such as interactivity and verification, develops a taxonomy with novel dimensions, and proposes future research directions.
2 Technical Background
The technical background introduces GenAI systems, model architectures, and training procedures, focusing on text and images. It covers transformers, diffusion models, other generative models, and system-level orchestration.
- The background is restricted to text and image data to illustrate multimodality, while referring readers elsewhere for video and audio.
- GenAI systems may comprise a single model, combined generative models, or models interacting with external applications and data sources.
- In orchestrated systems, an application can search the web, retrieve content, and modify prompts before sending them to a GenAI model.
- Transformers are flexible models that often require more data than simpler models to reach comparable performance, although simpler models may not reach the same top-level performance.
- Encoders map tokens into latent-space embeddings that support nuanced similarity computation and retrieval of relevant information for prompt enhancement.
- Diffusion models generate samples by learning to reverse a noise-adding process, but their outputs are not easily controllable without additional guidance inputs.
- GANs use a generator and discriminator, while VAEs and related models produce high-quality outputs from random inputs whose customization requires additional effort.
- Text-to-image models encode text as input and can generate images from a low-dimensional space or directly from pixel space.
3 The importance, challenges and desiderata of GenXAI
GenXAI is important because GenAI outputs are widely used, difficult to verify and evaluate, and embedded in complex, high-impact systems. These conditions create challenges involving access, interaction, scale, security, accountability, ethical risk, and explanation evaluation.
- Why explainability matters: GenAI outputs require verification because they can be harmful, misleading, incorrect, and fluent without being factual.Explanations can help identify errors or failures to detect incorrect outputs.
- Why explainability matters: Widespread access and high-impact applications increase the importance of understanding GenAI outputs and their effects on users.GenAI is used across society, including by vulnerable groups, and applications can affect personal, educational, and other consequential decisions.
- Challenges: Automatic evaluation is limited because many GenAI tasks produce responses that are difficult to score and suitable benchmarks may not exist.This also makes function-grounded evaluation of explanations more challenging.
- Challenges and desiderata: GenXAI must address security, safety, ethical, and governance concerns because explanations can expose sensitive information or themselves be offensive, harmful, or biased.The paper also identifies uncertainty as important because language models may produce fluent but incorrect answers.
- Challenges: Commercial models restrict XAI research because users and researchers often cannot access model internals or training data.This lack of access rules out many explainability approaches.
- Challenges: GenAI systems are difficult to explain because they combine very large models, extensive data sources, fine-tuning, retrieval, and external tools.Their complex supply chains also complicate accountability and lineage across data providers, developers, and users.
- Challenges: Generated artifacts contain many interdependent decisions, making it impractical for users to understand every output detail and motivating tailored, interactive explanations.GenAI explanations may need to address inputs, outputs, and interactions rather than only a single model decision.
4 Taxonomy of XAI techniques for GenAI
The paper introduces a taxonomy for classifying XAI mechanisms and algorithms that support understanding of GenAI, and applies it to existing techniques.
- Taxonomy: The taxonomy provides a classification scheme for GenXAI mechanisms and algorithms and is used to classify existing techniques.It supports the paper’s analysis of explainability methods for GenAI.
4.1 Dimensions of taxonomy
The taxonomy characterizes GenXAI methods through explanation output properties, input and internal requirements, and expanded notions of scope and interaction. It distinguishes explanations of individual input-output relations from explanations of complete interactions.
- Taxonomy dimensions: The taxonomy distinguishes explanation properties from the input and internal properties required to produce explanations.Explanation properties include scope, modality, and interactivity.
- Training dependence: Chain-of-thought prompting can encourage explanations, but it depends on training data containing explanatory patterns.If explanatory terms are absent from training data, prompting may not produce them.
- Scope: GenXAI scope covers inputs, outputs, and interactions rather than only the input samples explained.Output scope ranges from one focused attribute to all output attributes.
- Interaction scope: Interaction scope separates explanations of individual input-output relations from explanations of complete interaction dynamics and outcomes.The latter considers sequences of user and system outputs and outcomes such as task abandonment or unsatisfactory results.
- Interactivity: Interactive explanations can help users explore decisions, improve interactions, and understand how human and AI actions affect one another.Prior human-AI work includes explanations intended to reduce errors and improve efficiency.
4.1.2 Explanation Modality
GenXAI explanations may be unimodal or multimodal, with multimodal explanations combining textual, visual, or other forms of justification.
- Explanation modality: Explanations can be unimodal or multimodal, and combining modalities has been associated with favorable outcomes.Prior work collected textual and visual justifications and reported that each modality improves in the presence of the other.
4.1.3 Dynamics
GenXAI methods are shaped by dynamic explanations, multiple foundational sources, and techniques that modify models, training, data, or prompts to improve interpretability and verification.
- Interactivity and dynamics: Interactive GenXAI treats explanations as adjustable dialogues, while dynamic explanations vary in structure, content, properties, and causal factors across samples.Users may control explanations or request additional information; generated explanations can differ depending on the explained object and output.
- Foundational sources: GenXAI draws on data, models, optimization, and prompts as foundational sources that can be modified or tailored to improve explainability.These sources include training-data composition and explanations, model architecture, optimization objectives, and prompting constraints.
- Model-induced methods: Model-induced methods alter model design through interpretable components, additional models, or explicit multistep reasoning to foster explainability.Examples include interpretable activation functions, LLM-GNN combinations, external knowledge, and chained LLMs.
- Optimization: Optimization-based methods support explainability through disentangled latent dimensions, explanation-relevant training criteria, and evidence citation.Some approaches train models to be right for the right reasons or to cite evidence for claims.
- Prompt-based methods: Prompting can structure reasoning and constrain responses to supplied or retrieved information, facilitating verification despite unreliable faithfulness.RAG narrows the answer’s information source, while prompted explanations may still hallucinate or fail to reflect true internal workings.
4.1.5 Required Model Access by XAI Method
The information available to an XAI method ranges from black-box to white-box access, with commercial GenAI models commonly imposing black-box and training-data restrictions.
- Access levels: Black-box access exposes only limited output information, whereas white-box access includes the model, training data, and training procedure.Grey-box access lies between these extremes and may provide partial internal information.
- Commercial-model constraints: Commercial models such as GPT-4 are typically accessible only through APIs, while vendors often withhold training data and even coarse dataset summaries.These restrictions make XAI techniques that rely on internal states or training data difficult to apply.
4.1.6 Model (Self-)Explainers
GenAI models can explain their own decisions or act as explainers, expanding beyond classical intrinsic XAI while raising concerns about explanation faithfulness.
- Self-explanation: GenAI models can generate explanations for their own outputs, unlike classical intrinsic XAI approaches centered on simple interpretable models.The model itself supplies the explanation rather than a dedicated XAI technique.
- LLMs as explainers: LLMs can explain their outputs, other machine-learning models, data patterns, self-diagnosis processes, and autonomous-driving systems.Their uses include tailored explanations, model analysis, autoprompting, and interpretable system support.
- Reported quality: On free-form coding tasks, LLM explanations often exceed crowd workers’ gold references, while mental-health analysis explanations approach human quality.These findings concern explanation quality in the cited task settings.
4.1.7 Explanation Sample Difficulty
Explanation difficulty varies across samples, becoming especially important for GenAI because inputs and outputs range from simple lookups to complex tasks.
- Difficulty variation: Some samples and interactions are harder to explain than others, and this variation matters more as GenAI supports a wider range of inputs and outputs.The paper contrasts simple factual lookups with long stories and complex solutions.
- Observed degradation: GPT-3 explanations degraded more with example hardness than human explanations when explaining data labels.This finding is presented as evidence that explanation quality can depend on sample difficulty.
- Taxonomic framing: Existing taxonomies contain established pre-GenAI categories, while the paper additionally discusses novel dimensions for classifying GenXAI techniques.Mechanistic interpretability is described in terms of investigated components and reverse-engineering techniques.
4.2 Classification of Techniques
The paper classifies GenXAI techniques into established categories and further characterizes them by dimensions such as scope, dynamics, foundational source, and access requirements.
- Technique categories: The taxonomy places techniques into four commonly used XAI categories, focusing on methods developed for GenAI, adapted to it, or usable with limited change.Existing techniques are also structured according to the paper’s novel dimensions.
- Technique categories: Feature attribution assigns relevance scores to input features, while perturbation-based methods alter features and examine resulting output changes.Examples include token removal, negation, and intensification in natural-language processing.
- Technique categories: Decomposition, attention-based, sample-based, probing-based, and mechanistic methods respectively trace layer contributions, input importance, sample effects, model knowledge, or internal model mechanisms.Mechanistic interpretability includes circuit discovery, causal tracing, and vocabulary-lens approaches.
- Novel dimensions: Existing techniques generally explain individual outputs or properties, whereas explanations generated by LLMs can be interactive, personalized, and sample-dependent.No existing technique explicitly focuses on the entire interaction, although LLM self-explanations could serve that purpose.
- Foundational sources: The concept matrix links XAI methods to foundational sources, including models, training data, datasets, prompts, and other inputs.Probing can naturally be performed through prompts, while sample-based and mechanistic methods commonly rely on data and models.
- Access requirements: Most existing techniques require white-box access, while black-box approaches such as occlusion typically provide coarser and less accurate explanations.The access requirement is summarized in the paper’s mapping of XAI categories and techniques.
5 Research Agenda of XAI for GenAI, Discussion and Conclusions
The paper presents GenXAI as an emerging interdisciplinary research area and proposes a roadmap spanning interaction, modalities, verification, mechanistic methods, and applications beyond text and images. It also bounds its contribution by emphasizing selected aspects rather than comprehensive coverage.
- Interdisciplinary scope: The paper argues that bridging AI research with cognitive science, psychology, and humanities is likely to advance many GenXAI topics.The field is described as quickly emerging, with opportunities spanning technical, economic, and psychological issues.
- Research directions: Explaining interactions requires understanding humans, models, and systems, making it particularly relevant to interdisciplinary research in information systems and human-computer interaction.The paper distinguishes interaction explanations from explanations of single input-output pairs.
- Research directions: The research agenda calls for work on interactive and real-time explanations, explanations of human-AI interactions, multimodal explanations, and GenXAI for video, 3D content, and actions.It also identifies mechanistic interpretability and methods addressing verifiability and hallucinations as important directions.
- Scope and limitations: The paper does not aim to be comprehensive and omits detailed treatment of explanation evaluation and usage, while focusing mainly on text and image modalities.It also presents the review as a time-bound snapshot because GenAI is evolving rapidly.
- Contribution: The authors describe the work as a conceptualization and research agenda that motivates GenXAI, structures existing knowledge, identifies dimensions, and highlights open issues.The conclusion frames addressing these issues as a joint effort for the research community.
7 Appendix
The appendix describes a mainly qualitative review approach combining systematic and narrative review elements rather than quantitatively assessing publication trends.
- Review methodology: The review combines systematic and narrative elements, with narrative literature review serving as the main qualitative approach.The authors state that the limited number of technical works on GenXAI motivated this emphasis.
- Review methodology: The study describes its search process but does not aim to quantify peer-reviewed works or show publication counts and trends over time.