PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs for Explainable AI: A Comprehensive Survey

Ahsan Bilal, David Ebert, Beiyu Lin

arXiv:2504.00125v1cs.AIcs.CL

TL;DR

Complex AI models are difficult to understand because their decision-making lacks transparency, limiting trust and use in important domains. This survey reviews LLM-based explainability approaches, challenges, limitations, evaluation-related issues, applications, and future directions, emphasizing integrated architectural and narrative perspectives.

  • Problem

    Complex AI models often function as black boxes, making their outputs difficult for nonexperts to understand and limiting trust in domains such as healthcare and finance.

  • Method

    The survey organizes LLM-based XAI into post-hoc explanations, intrinsic explainability, and human-centered narratives, while discussing evaluation, challenges, limitations, applications, and future directions.

  • Results

    The survey presents a comprehensive understanding of how LLMs can improve AI explainability by integrating model-architecture and natural-language narrative approaches.

  • Takeaways & Limitations

    Future LLM-based XAI should pursue more interpretable, automated, user-centric, and multidisciplinary approaches while addressing privacy, complexity, domain adaptation, and fairness challenges.

  • Takeaways & Limitations

    LLM-based XAI remains constrained by sensitive-data handling, societal diversity and norms, multisource data and algorithmic complexity, and bias and fairness issues.

Abstract

from arXiv · show

Large Language Models (LLMs) offer a promising approach to enhancing Explainable AI (XAI) by transforming complex machine learning outputs into easy-to-understand narratives, making model predictions more accessible to users, and helping bridge the gap between sophisticated model behavior and human interpretability. AI models, such as state-of-the-art neural networks and deep learning models, are often seen as "black boxes" due to a lack of transparency. As users cannot fully understand how the models reach conclusions, users have difficulty trusting decisions from AI models, which leads to less effective decision-making processes, reduced accountabilities, and unclear potential biases. A challenge arises in developing explainable AI (XAI) models to gain users' trust and provide insights into how models generate their outputs. With the development of Large Language Models, we want to explore the possibilities of using human language-based models, LLMs, for model explainabilities. This survey provides a comprehensive overview of existing approaches regarding LLMs for XAI, and evaluation techniques for LLM-generated explanation, discusses the corresponding challenges and limitations, and examines real-world applications. Finally, we discuss future directions by emphasizing the need for more interpretable, automated, user-centric, and multidisciplinary approaches for XAI via LLMs.

1 INTRODUCTION

Complex AI models are difficult to understand because their decision processes lack transparency, limiting trust and adoption. The survey examines three LLM-based explainability approaches, evaluation, challenges, applications, and future directions.

  • Deep learning models are often treated as black boxes because their complexity makes specific outputs difficult to understand.
  • Limited transparency makes it harder for nonexperts to understand, trust, and act on AI decisions in critical domains such as healthcare and finance.
  • LLMs can translate complex model behavior and outputs into natural-language explanations tailored to user questions.
  • The survey organizes LLM-based XAI into post-hoc explanations, intrinsic explainability, and human-centered narratives.
  • The survey also evaluates explanations, examines challenges and applications, and outlines future directions integrating model architecture with narrative approaches.

2 BACKGROUND

XAI seeks to make AI decisions transparent and understandable, while LLMs offer natural-language mechanisms for clarifying complex model behavior across domains. The background highlights their potential in communication, cybersecurity, and other applications.

  • XAI makes AI outputs and inner workings more understandable, supporting transparency, trust, and ethical alignment.
  • XAI can help identify model biases and limitations while supporting transparency and accountability under regulatory expectations.
  • LLMs use transformer architectures, massive training data, and billions of parameters to capture complex input features across diverse tasks.
  • In communication, LLMs can clarify how signal interference, propagation loss, multipath fading, noise, and weather conditions affect complex environments.
  • In cybersecurity, LLMs generate human-readable explanations of malware, phishing, network anomalies, and the rules or thresholds triggering alerts.

3 OVERVIEW OF EXPLAINABILITY TECHNIQUES

The survey presents post-hoc explanations, intrinsic interpretability, and human-centered explanations as three complementary approaches to LLM-based XAI. These approaches respectively analyze outputs, build explainability into models, and refine explanations around users.

  • Post-hoc Explanations: Post-hoc methods explain predictions after they are generated by analyzing feature contributions through tools such as SHAP, LIME, and Integrated Gradients.
  • Post-hoc Explanations: Global explanations describe overall model behavior, whereas local explanations identify features contributing to a single prediction.
  • Post-hoc Explanations: Post-hoc explanations can be unstable and may interpret predictions without revealing the model’s detailed internal computations.
  • Intrinsic Interpretability: Intrinsic interpretability designs models to be understandable by incorporating mechanisms such as attention-based feature displays and Chain of Thought reasoning.
  • Intrinsic Interpretability: Guided CoT templates structure multistep reasoning, while ReAct combines reasoning with iterative task-specific actions.
  • Intrinsic Interpretability: Architecture-based XAI checks explanations for both factual accuracy and logical correctness rather than relying on convincing presentation alone.

4 EVALUATING LLM-GENERATED EXPLANATIONS

Evaluating LLM-generated explanations requires both qualitative assessment of human usability and quantitative assessment of fidelity to model reasoning. The survey distinguishes comprehensibility, controllability, faithfulness, and plausibility as key evaluation concerns.

  • Evaluation is divided into qualitative and quantitative categories because capturing a model’s reasoning processes remains challenging.
  • Qualitative Evaluation: Comprehensibility measures whether explanations clearly communicate model reasoning in terms humans can follow.
  • Qualitative Evaluation: Controllability measures the interactivity and adjustability of explanations, including users’ ability to provide feedback and request improvements.
  • Human-readable explanations can appear convincing while failing to reflect the model’s actual logic, motivating objective quantitative evaluation.
  • Quantitative Evaluation: Faithfulness measures how accurately an explanation represents the model’s actual decision-making process, design, training data, and reproducible features.
  • Quantitative Evaluation: Plausibility evaluates whether explanations are logically coherent, domain-consistent, causally valid, and free of contradictions.

5 BENCHMARK DATASETS FOR EXPLAINABLE AI WITH LLMS

Benchmark datasets support the development and evaluation of LLM-generated explanations by pairing predictions or queries with human-written, structured, factual, or retrieved evidence. The survey highlights datasets spanning natural-language inference, commonsense reasoning, knowledge-graph grounding, and retrieval-augmented generation.

  • Benchmark datasets provide standardized material for training and measuring explanation quality, including accuracy, completeness, comprehensibility, and consistency.They help researchers assess whether explanations are clear, transparent, and user-friendly.
  • e-SNLI pairs natural-language inference labels with human-written explanations, supporting explanation generation for entailment, contradiction, and neutral relations.Its annotations connect sentence-pair classifications with human explanations of the stated relation.
  • CoS-E supplies multiple-choice commonsense questions with human-written reasoning steps to enable everyday reasoning explanations.The dataset links selected answers with explanations grounded in common knowledge.
  • XplainLLM grounds explanations in knowledge graphs by justifying the correct option and why alternatives are less suitable.This grounding links LLM reasoning to structured and factual knowledge.
  • RAGBench pairs domain-specific queries with retrieved evidence and generated responses, making the evidence behind retrieval-augmented outputs inspectable.The retrieved material provides transparency for contextually accurate responses in domain-specific applications.

6 REAL-WORLD APPLICATIONS OF LLM EXPLAINABILITY

The survey describes LLM explainability across healthcare, finance, drug discovery, legal analysis, manufacturing, education, and urban planning. In these applications, LLMs translate model inputs, factors, or reasoning into domain-specific explanations and recommendations.

  • In healthcare, LLMs explain AI-driven diagnoses and predictions by clarifying relationships among patient symptoms or medical-imaging findings.The survey also describes textual explanations designed to summarize patient data or imaging results for clinicians.
  • In finance, LLMs explain credit and fraud decisions through factors such as credit scores, debt-to-income ratios, missed payments, transaction amounts, locations, and historical behavior.These explanations support institutional review and help customers take corrective measures during loan applications.
  • In drug discovery, LLMs explain molecular interactions at chemical and biological levels, including why compounds bind to bacterial enzymes and differ in antibacterial effectiveness.The explanations help scientists understand molecule combinations and their impacts on the human body.
  • Legal and manufacturing systems use LLMs to explain flagged contract clauses, relevant precedents, and predictive-maintenance signals such as vibration patterns and temperature readings.These explanations support legal document review and preventive maintenance decisions.
  • In education, LLMs personalize learning recommendations by generating study plans with explanations, practice problems, and quizzes adapted to student preferences, weaknesses, and progress.The survey names GPT-4 and Gemini as examples used for these recommendations.
  • In urban planning, LLMs decompose traffic congestion into factors such as flow patterns, road capacities, and peak travel times, then describe potential interventions.Examples include optimizing traffic lights, creating bike lanes, and extending public transport routes.

7 CHALLENGES AND LIMITATIONS RELATED TO EXPLAINABLE AI (XAI)

LLM-based XAI faces challenges involving sensitive data, societal diversity, multisource inputs, model complexity, deployment constraints, and bias or fairness. These constraints can affect the privacy, accessibility, reliability, and practical use of generated explanations.

  • 7.1 Sensitive Data: Sensitive data needed for post-hoc and human-centered explanations can expose medical, financial, or social information and create privacy risks.The paper identifies encryption, anonymization, and authenticated access as data-handling safeguards.
  • 7.2 Societal Diversity and Norms: XAI systems must account for biological diversity and culturally varying norms to avoid confusing or harmful explanations.Examples include differing disease susceptibilities across populations and different interpretations of gestures across cultures.
  • 7.3 Multisource Data and Algorithms: Combining social, financial, and behavioral data can skew explanations because each source provides a different perspective and may receive inappropriate weight.
  • 7.4 Complexities of AI Model: Layered AI models are difficult to explain because users must understand how low-level features combine across successive layers into final predictions.Interfaces may need to adapt explanations to users’ expertise, autonomy, and trust.
  • 7.4 Complexities of AI Model: Deployment is constrained by limited computing resources and the storage or processing capacity of edge devices such as smartphones.Knowledge Distillation transfers knowledge from a larger model such as BERT to a smaller model such as DistilBERT.
  • 7.5 Bias and Fairness within LLMs: LLM bias can produce stereotyped, less comprehensible, or exclusionary explanations through social, language, and representation biases.The survey mentions in-context learning and prompt refinement as proposed mitigation methods.

8 FEATURE IMPORTANCE ACROSS LLMS USED FOR EXPLANATION USING SALIENCY MAPS

Saliency maps compare which features different LLMs prioritize during explanation generation and how closely those features align with ground truth. The reported values vary across models and input-context sizes.

  • Saliency-map interpretation: Saliency techniques identify words and phrases in generated explanations that align most closely with the ground truth.The survey describes gradient-based methods as measuring output changes caused by small input changes.
  • Model comparison: ChatGPT records saliency values of 0.84 for large inputs and 0.56 for small inputs, the highest values reported in the comparison.
  • Model comparison: LLAMAV2 and COHERE show substantial large-input saliency values of 0.75 and 0.71, whereas VICUNA, MISTRAL, and OLMO show lower values.The comparison is intended to assess whether models maintain attention to important features across context sizes.

9 FUTURE DIRECTIONS

The survey identifies automated feedback, multimodal explanations, knowledge graphs, and cross-disciplinary collaboration as future directions for clearer and more useful LLM-based XAI. These directions aim to adapt explanations to users, connect textual reasoning with visual evidence, and align systems with domain needs.

  • Automation with Human Feedback: Automating feedback could adapt explanations from user interactions, reducing the time and effort required for manual feedback collection.The survey describes systems detecting useful explanations and clarification requests, then adjusting responses accordingly.
  • Automation with Human Feedback: Automated feedback could improve explanation practicality across healthcare, finance, customer support, and education by tailoring content to recurring user needs.Examples include adding definitions or charts in finance and presenting mathematical solutions step by step in education.
  • Integrating Visual and Text Explanations: Combining textual explanations with visual aids such as heatmaps can make model reasoning and influential input features more accessible.The survey proposes pairing an LLM’s written explanation with visualizations such as Grad-CAM for medical images.
  • Integrating Visual and Text Explanations: Knowledge graphs could connect LLM-generated explanations with visual representations of relationships among questions, retrieved information, and generated responses.This approach is presented as a way to organize complex AI outputs for easier interpretation.
  • Cross-Disciplinary Research: Cross-disciplinary collaboration can align explanations with human information processing and professional requirements in specialized domains.The survey highlights cooperation among AI researchers, cognitive scientists, HCI researchers, and domain experts, including clinicians and lawyers.
Loading 2504.00125v1…