PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review

Anton Kuznietsov, Balint Gyevnar, Cheng Wang, Steven Peters, Stefano V. Albrecht

arXiv:2402.10086v2cs.ROcs.AIcs.CVcs.HCcs.LG

TL;DR

Safe and trustworthy autonomous driving lacks detailed XAI frameworks grounded in concrete research, and safety standards do not explicitly address data-driven AI. This systematic review organizes XAI methods into five categories and proposes SafeX, a modular framework whose combination of explanation sources may reduce risks from AI in autonomous driving.

  • Problem

    Safe and trustworthy autonomous driving lacks detailed XAI frameworks grounded in concrete research, and safety standards do not explicitly address data-driven AI.

    Source: §I · p. 1, §VI · p. 18

  • Method

    The paper systematically reviews XAI for autonomous driving, synthesizes data-, model-, and agency-related requirements, and proposes the modular SafeX framework.

    Source: B. Requirements For Safe and… · p. 3, B. Conceptual Framework: SafeX · p. 16

  • Results

    The review sorts XAI research for autonomous driving into five categories: interpretable design, surrogate models, monitoring, auxiliary explanations, and validation.

    Source: §VII · p. 19

  • Takeaways & Limitations

    The authors propose that combining multiple explanation sources in SafeX may reduce risks from using AI for autonomous driving.

    Source: §VI · p. 18

    Generative methods did not appear prominently in the review, and their efficacy in alleviating the long tail of critical scenarios remains unverified.

    Source: §VI · p. 18

Abstract

from arXiv · show

Artificial Intelligence (AI) shows promising applications for the perception and planning tasks in autonomous driving (AD) due to its superior performance compared to conventional methods. However, inscrutable AI systems exacerbate the existing challenge of safety assurance of AD. One way to mitigate this challenge is to utilize explainable AI (XAI) techniques. To this end, we present the first comprehensive systematic literature review of explainable methods for safe and trustworthy AD. We begin by analyzing the requirements for AI in the context of AD, focusing on three key aspects: data, model, and agency. We find that XAI is fundamental to meeting these requirements. Based on this, we explain the sources of explanations in AI and describe a taxonomy of XAI. We then identify five key contributions of XAI for safe and trustworthy AI in AD, which are interpretable design, interpretable surrogate models, interpretable monitoring, auxiliary explanations, and interpretable validation. Finally, we propose a modular framework called SafeX to integrate these contributions, enabling explanation delivery to users while simultaneously ensuring the safety of AI models.

I. INTRODUCTION … C. Scope and Structure

The paper argues that XAI can help address safety and trust challenges posed by opaque AI in autonomous driving (AD), then systematically reviews XAI methods and proposes SafeX. It frames this work against gaps in prior surveys and standards, while defining its scope across AD pipelines and tasks.

I. INTRODUCTION

  • Deep learning’s black-box behaviour obscures internal workings, hindering issue identification and judgments about real-world use in safety-critical AD.Existing mitigation processes, including well-justified data acquisition, may not ensure sufficient safety, leaving the adequacy of those measures an open question.Source: §I · p. 1
  • Current standards leave gaps for data-driven AI in AD, including unspecified SOTIF acceptance criteria and missing guidance for designing AI functionality.ISO 26262 was not developed explicitly for data-driven AI; ISO/AWI PAS 8800 remains under development with unclear scope and guidance.Source: §I · p. 1
  • XAI may help developers debug malfunctions, users calibrate trust, and assurance bodies assess due diligence and liability through greater traceability.The paper also notes that 68% of U.S. drivers are wary of AVs and that AI is a key factor in their non-acceptance.Source: §I · p. 1

A. Previous Reviews on XAI for AD

  • Omeiza et al. lack reproducibility and completeness; Atakishiyev et al. [18] insufficiently detail repeatable methods or framework integration; Zablocki et al. omit modular perception and planning and safety impact.The paper identifies three broader gaps in prior reviews: systematic methodology, attention to XAI’s safety and trustworthiness benefits and drawbacks, and frameworks for integrating XAI with AD.Source: A. Previous Reviews on XAI for AD · p. 2

B. Main Contributions

  • The paper sets out detailed AD AI requirements and a structured, systematic, repeatable survey of XAI for perception, planning and prediction, and control.It highlights XAI’s importance in fulfilling those requirements.Source: B. Main Contributions · p. 2
  • The review identifies five XAI paradigms for safe and trustworthy AD and proposes SafeX, a conceptual framework designed to incorporate the summarized techniques.The authors also analyze limitations of existing modular XAI frameworks and discuss the paradigms with concrete examples.Source: B. Main Contributions · p. 2

C. Scope and Structure

  • The survey covers modular and end-to-end AD pipelines across perception, planning and prediction, and control, but does not identify stakeholders or teach mathematical foundations.Its structure includes foundations and taxonomy, reproducible methodology, the five XAI paradigms, existing frameworks and SafeX, findings, and future directions.Source: C. Scope and Structure · p. 2

II. FOUNDATIONS · A. Trustworthy AI · B. Requirements For Safe and Trustworthy AI in AD

The foundations frame XAI as necessary but not sufficient for trustworthy AD: opaque neural systems challenge safety assurance, while AD requirements span data, models, human agency, and safety.

II. FOUNDATIONS

  • The foundations place XAI within a broader inquiry into trustworthy AI, AD-specific requirements, explanation sources, and an XAI taxonomy.Source: §II · p. 2, A. Trustworthy AI · p. 3

A. Trustworthy AI

  • Unlike symbolic expert systems [20], high-performing neural methods use opaque sub-symbolic representations, and safety-relevant AD cannot fully rely on black boxes not currently certifiable for safety.Source: A. Trustworthy AI · pp. 2–3
  • Explainability alone cannot deliver trustworthy AI; it is one necessary element alongside human agency, robustness and safety, privacy and data governance, fairness, and societal and environmental well-being.The paper frames these as socio-technical requirements for human-centric AI that keeps human values central across development, deployment, use, and monitoring.Source: A. Trustworthy AI · p. 3

B. Requirements For Safe and Trustworthy AI in AD

  • Because no requirements are published specifically for AI in AD, the paper starts from EU and NIST guidance, organizing requirements by data, model, and agency.The EU guidance lists seven trustworthy-AI requirements, and NIST lists seven characteristics; the paper uses these national-level proposals as its starting point.Source: B. Requirements For Safe and… · p. 3, Table I · p. 4
  • Data requirements call for privacy-aware governance, diverse training data, and stakeholder perspectives; a cited study found missed pedestrians included children and people with darker skin tones [31],.Source: B. Requirements For Safe and… · pp. 3–4
  • AD models must withstand changing driving conditions, other drivers, noise, distribution shifts, and adversarial attacks, while remaining transparent, user-centric, and accountable.Transparency supports developers’ debugging and regulators’ auditing and certification; robustness and safe decisions are needed even in uncertain environments.Source: B. Requirements For Safe and… · p. 4
  • Level 3 AV drivers must be ready to take over, while level 4 and 5 automation can reduce oversight; explanations can help users contest decisions and calibrate trust.Source: B. Requirements For Safe and… · p. 4
  • Safety assurance imposes strict behavioral constraints, and XAI supports trustworthy AD through intelligible explanations, traceability, model inspection, debugging, and auditing.XAI can provide evidence about personal-data processing, support non-discrimination and failure analysis, and contribute to robustness and better-calibrated trust.Source: B. Requirements For Safe and… · p. 4

C. Sources of Explanation in AI

The review distinguishes five related, nonexclusive sources of explanation—interpretability, explainability, justifiability, traceability, and transparency—rather than treating them as interchangeable, [38],.

C. Sources of Explanation in AI

  • Interpretability is an inherent quality: a reasonably experienced user can understand a sufficiently low-complexity system’s output and the causal process that produced it.Interpretable systems are often argued to suit safety-relevant applications because their decision-making causal chain is observable.Source: C. Sources of Explanation in AI · p. 5
  • Explainability means an output is accompanied by an intelligible explanation of its reasoning, which need not be inherent to the system or reflect its actual causal chain.Source: C. Sources of Explanation in AI · p. 5
  • Justifiability means explaining why an output was good, not necessarily how it was computed; goodness depends on the application domain and designers’ ethical framework.Source: C. Sources of Explanation in AI · p. 5
  • Traceability lets an external auditor follow the full causal chain from input to output; black-box design obscures it, and even white-box models can be too large to trace.Source: C. Sources of Explanation in AI · p. 5
  • Transparency is broader than an AI system or XAI alone: documentation, XAI, standardisation, and risk assessments can together support understanding and informed use.Source: C. Sources of Explanation in AI · p. 5

D. Taxonomy of XAI · E. Terminology of AD Components

The review frames XAI through a six-category taxonomy based on Speith and establishes terminology for AV functions from perception through control,.

D. Taxonomy of XAI

  • The XAI taxonomy organizes key concepts into six categories, providing a framework for describing the methods reviewed later.Figure 2 visualizes the taxonomy.Source: D. Taxonomy of XAI · p. 5, Figure 2 · p. 5
  • Decision-model representation may be symbolic, sub-symbolic, or mixed; sub-symbolic systems are harder to explain and may complicate safety and trustworthiness.This category is determined primarily by the decision-making system's design, rather than the XAI system.Source: D. Taxonomy of XAI · p. 5
  • Post-hoc explanations follow decisions and apply broadly, while ante-hoc explanations arise from symbolic or mixed models and are more constrained but generally more trustworthy and verifiable.Post-hoc methods are usually assumed to access only the decision system and its inputs and outputs.Source: D. Taxonomy of XAI · p. 5
  • Common explanation modes include surrogates, representative or counterfactual examples, and feature importance; interpretable ante-hoc systems can themselves serve as explanations.Surrogate fidelity can be difficult to quantify, examples depend on user interpretation, and importance is not causality [43],. Treating a complex model itself as an explanation can also impose substantial cognitive demands and may not build stakeholder trust.Source: D. Taxonomy of XAI · p. 5
  • Explanation scope ranges from local instances to cohorts and whole models, while the delivery medium should suit stakeholder needs because it affects efficacy and intelligibility.Examples and importance explanations suit local or cohort scope, whereas surrogates represent the overall process; explanations are often offered “as is” without considering communication.Source: D. Taxonomy of XAI · pp. 5–6

E. Terminology of AD Components

  • Perception derives environmental knowledge by identifying obstacles, signs, and markings and classifying semantic data; localization determines the AV's position..Source: E. Terminology of AD Components · p. 6
  • Planning and prediction use obstacle trajectories to pursue vehicle goals: mission planning selects a route, behavior planning chooses driving behavior, and motion planning finds a feasible path.Motion planning seeks a collision-free, comfortable, dynamically feasible path or trajectory given the current driving behavior.Source: E. Terminology of AD Components · p. 6
  • Control executes higher-level plans; path tracking follows a planned path without a temporal law, whereas trajectory tracking follows time-evolving feasible states.Modular architectures ease error tracing, while end-to-end neural networks—often limited to perception and planning—are less interpretable and make safety harder to ensure.Source: E. Terminology of AD Components · p. 6

III. REVIEW METHODOLOGY · A. Research Questions · B. Search Process

The review uses a structured, repeatable literature-search process to answer questions about XAI methods for AD safety or trustworthiness and frameworks integrating XAI with autonomous driving. It searches three indexing databases through progressively refined queries, then filters records to a final set of 84 publications.

III. REVIEW METHODOLOGY

  • The authors follow Kitchenham and Charters and Stepin et al. to systematically review the field and make the search process repeatable.They say this enables validity checks and future updates to the review.Source: §III · p. 6

B. Search Process

  • After removing duplicates, the authors filtered records using inclusion and exclusion criteria, then reapplied them to full texts to form the final set of 84 publications.Figure 5 presents the retrieval and two content-filtering stages.Source: §III · p. 6, B. Search Process · p. 7, Figure 5 · p. 7

A. Research Questions

  • RQ1 asks which XAI methods address AD safety or trustworthiness requirements and their contributions; RQ2 asks which general frameworks integrate XAI with autonomous driving.Source: A. Research Questions · p. 6

B. Search Process

  • The search uses Scopus, Web of Science, and IEEE Xplore, with increasingly refined query levels intended to yield relevant publications in manageable numbers.The query hierarchy is shown in Figure 4.Source: B. Search Process · pp. 6–7, Figure 4 · p. 7
  • Equivalent queries were applied to publication titles, author keywords, and abstracts in all three databases between 22 and 26 September 2023.Source: B. Search Process · p. 7
  • Queries broaden from explanation, interpretation, or XAI terms to autonomous-driving terms, then filter for safety or trust and branch by framework or AD subsystem.The subsystem branches cover sensing or perception, prediction or planning, and control.Source: B. Search Process · p. 7

C. Inclusion and Exclusion Criteria · IV. XAI FOR SAFE AND TRUSTWORTHY AD

The review selected work motivated by safer or more trustworthy technology and organized its XAI-for-autonomous-driving survey around five categories of methods. Its criteria restricted the included papers to concrete AI contributions addressing explanations and autonomous-driving tasks.

C. Inclusion and Exclusion Criteria

  • Papers had to be motivated at least partly by safer or more trustworthy technology and propose a concrete AI-related system, algorithm, framework, or novel artefact.All inclusion criteria had to be fulfilled at each filtering stage.Source: C. Inclusion and Exclusion Criteria · p. 7
  • Papers were excluded if they made no attempt to address a source of explanations described in Section II-C.Meeting any one exclusion criterion was sufficient for removal.Source: C. Inclusion and Exclusion Criteria · p. 7
  • Papers were excluded when autonomous driving was not their main application or evaluation domain, or they did not address AD perception, planning, prediction, or control.Source: C. Inclusion and Exclusion Criteria · p. 7

IV. XAI FOR SAFE AND TRUSTWORTHY AD

  • The review groups XAI methods for autonomous driving into interpretable design, surrogate models, monitoring, auxiliary explanations, and validation.Figure 6 illustrates the five categories and their roles, from inherently interpretable algorithms to interpretable agents used to validate autonomous vehicles.Source: §IV · p. 7, Figure 6 · p. 8
  • The collected publications include both intrinsic and post-hoc interpretability, unlike Du et al.’s grouping of interpretable machine learning into those two types.Source: §IV · p. 7
  • The review compares category advantages and disadvantages in Table III and summarizes methods in Tables IV–VII, except interpretable validation, which had only five works.The method tables follow the XAI taxonomy presented in Section II-D.Source: §IV · p. 8

A. Interpretable By Design … 3) Interpretable By Design – Control:

Interpretable-by-design methods make decision processes transparent through human-interpretable structures that expose input–output relationships. Across perception, planning and prediction, and control, they offer explanatory mechanisms but can depend on domain expertise and specialized data.

A. Interpretable By Design

  • Perception methods make models more transparent using extracted traffic-sign features, interpretable prototypes for pedestrian detection, semantic latent concepts,, or knowledge graphs.The prototype-based DNN supports ante-hoc analysis for safety argumentation, while semantic concepts align model channels with human-interpretable visual concepts.Source: A. Interpretable By Design · p. 8
  • Capsule networks express interpretable relationships between features and driving context, while Grad-CAM heat maps identify sun-glare regions in images.Source: A. Interpretable By Design · p. 8

2) Interpretable By Design – Planning & Prediction:

  • High-level driving maneuvers abstract lower-level variables such as acceleration and steering, making planning and prediction decisions more tractable and interpretable,,,,,.Goal and trajectory prediction also use Bayesian inverse planning,, decision trees,, and discrete-choice models; lane-change explanations include time-series motifs, rule extraction, and interpretable variational auto-encoders.Source: 2) Interpretable By Design – Planning… · pp. 8, 10
  • Interpretable planning can require extensive domain knowledge, and Bayesian networks may encode mistaken or biased assumptions about the causal processes behind driving decisions,,.Bayesian networks provide a principled mathematical framework for causal reasoning, but their reliance on expert knowledge can introduce human modeling errors and biases.Source: 2) Interpretable By Design – Planning… · p. 8
  • Monte Carlo Tree Search over high-level maneuvers can explore relevant parts of the search space while avoiding unsafe actions, but it is computationally expensive and depends on other-participant trajectory predictions,,.Source: 2) Interpretable By Design – Planning… · p. 8
  • Only three papers ran user studies,,, and just one thoroughly investigated explanation effects on users while eliciting stakeholder requirements for intelligibility, accountability, and trust.That method relied on a highly specialized dataset annotated with high-level semantic and structured explanations, which may not be readily available.Source: 2) Interpretable By Design – Planning… · p. 10

3) Interpretable By Design – Control:

  • A neural-network controller with control-barrier-function filters projects its output onto a safe set through quadratic programs and a gauge map, providing traceability of the control process.The method also discusses how interpretable control affects safety validation.Source: 3) Interpretable By Design – Control · p. 10

4) Interpretable By Design – Summary: … 2) Interpretable Surrogate Models – Planning & Prediction:

Interpretable-by-design methods make AD models more interpretable through semantic representations, high-level abstractions, or constrained control, while surrogate models approximate black-box behavior to provide intelligible explanations. The review notes potential safety and trust benefits but also limitations in stakeholder evaluation and human-understandable explanations.

4) Interpretable By Design – Summary:

  • Perception methods learn semantic concepts in an interpretable latent space or modular components, but defining concepts is difficult and often requires labeled datasets.Classifier interpretability can also be used to localize objects.Source: 4) Interpretable By Design – Summary · p. 10
  • Planning methods use hand-crafted high-level features or maneuvers in MCTS, Bayesian networks, or decision trees, while constraint-satisfaction can map DNN outputs to safe-by-construction control.Creating planning abstractions requires substantial domain knowledge and careful engineering.Source: 4) Interpretable By Design – Summary · p. 10
  • Interpretable algorithms may support safety validation through clear input-output causal links and trust calibration through understandable abstractions, but stakeholder studies and concrete intelligible explanations are scarce.Scientific analyses of interpretable components often do not scale to varied stakeholders or address trustworthy-AD requirements.Source: 4) Interpretable By Design – Summary · p. 10

B. Interpretable Surrogate Models

  • An interpretable surrogate model approximates a black-box algorithm’s behavior using an interpretable-by-design model to provide intelligible explanations [41].Source: B. Interpretable Surrogate Models · p. 10

1) Interpretable Surrogate Models – Perception:

2) Interpretable Surrogate Models – Planning & Prediction:

  • Cui et al. used SHAP-selected features and a random forest to explain DRL decisions, while Li et al. used ME-SHAP for XGBoost lane-change decisions, without substantiating human-understanding benefits.Source: 2) Interpretable Surrogate Models –… · p. 10
  • Mishra et al. found their decision-tree visual explanations effective with students and experts and more effective than text; Omeiza et al. evaluated scene-graph-derived language explanations.Omeiza et al. used an extensive user study to assess perceived accountability and understanding, but their method description limits assessment of performance in unseen scenarios.Source: 2) Interpretable Surrogate Models –… · p. 10
  • Gyevnar et al. used a cognitive model and probabilistic-planner simulations to generate counterfactual worlds and natural-language explanations, evaluated by over 200 participants against HEADD.Source: 2) Interpretable Surrogate Models –… · pp. 10–11

3) Interpretable Surrogate Models – Control: … 2) Interpretable Monitoring – Planning & Prediction:

The review describes interpretable surrogates that help explain black-box control and end-to-end decisions, and monitoring methods that assess or explain the safety of perception, planning, and prediction. It also notes limits involving reproducibility, explanation consistency, visual cues, and evaluation.

3) Interpretable Surrogate Models – Control:

  • A time-series clustering method interpreted a DNN controller’s response while it stabilized an AV’s dynamic position under environmental disturbances, but unspecified network details limit reproducibility and generalization.The authors identified one such control study.Source: 3) Interpretable Surrogate Models –… · p. 11

4) Interpretable Surrogate Models – End-to-end:

  • For end-to-end decision models, surrogate explanations use object-level counterfactual images or a separately trained self-supervised attention-mask model.The counterfactual framework changes objects in the input image to reveal their influence on decisions; the attention mask highlights pixels important to an agent’s decisions.Source: 4) Interpretable Surrogate Models –… · p. 11

5) Interpretable Surrogate Models – Summary:

  • Interpretable surrogates approximate black-box behavior; perception models such as inspectable random forests or Shapley-value methods [93] can help analyze detection errors and improve transparency and reliability.The review says SHAP is common in planning and prediction, with only three works not relying on it.Source: 5) Interpretable Surrogate Models –… · p. 11
  • SHAP explanations can be inconsistent with other saliency methods,, so their use for AV trustworthiness requires care and stakeholder user studies to validate trust claims.For control, post-hoc clustering interprets DNN outputs over time; end-to-end surrogates include a generative model for counterfactuals and a separate explanation module.Source: 5) Interpretable Surrogate Models –… · p. 11

C. Interpretable Monitoring

  • Interpretable monitoring uses an interpretable-by-design model to verify a decision-making algorithm’s output, with the aim of safer AV deployment.The review’s definition frames monitoring as verification of the algorithm’s decision output.Source: C. Interpretable Monitoring · p. 11

1) Interpretable Monitoring – Perception:

  • Perception monitors use interpretable visual concepts, saliency comparisons, concept bottlenecks, and feature contributions to assess decisions, detect anomalies or unsafe inputs, and enhance reliability and robustness.Examples include traffic-sign methods [108],, and fault-diagnosis monitoring. The review reports that no user studies were conducted in this category.Source: 1) Interpretable Monitoring – Perception · pp. 11–12, Table VI · p. 12

2) Interpretable Monitoring – Planning & Prediction:

  • Planning and prediction monitors commonly use decision trees –,, to score fixed high-level actions, but state-only methods omit visual cues.Other methods use raw perception, including GradCAM for accident prediction and a two-stage visual-attention and stochastic-MDP approach that predicts accident probability and fixation area.Source: 2) Interpretable Monitoring – Planning… · p. 12
  • A decision-tree pipeline trained by imitation learning for full motion planning was verifiable and interpretable, but evaluated only on lane changes.A separate holistic anomaly-detection framework synthesizes lower-level explanations into natural-language explanations using logic and commonsense, but lacked a user study.Source: 2) Interpretable Monitoring – Planning… · p. 12

3) Interpretable Monitoring – Summary: · D. Auxiliary Explanations · 1) Auxiliary Explanations – Perception:

Interpretable monitoring can check perception decisions and detect anomalies, while auxiliary explanations expose information about how an algorithm produced its output. Perception methods use heat maps, attribution maps, and analysis tools to explain or inspect model behavior.

3) Interpretable Monitoring – Summary:

  • Interpretable surrogates can verify perception decisions through semantic concepts, while heat-map monitors or interpretable meta-models can detect anomalies and their causes.These monitoring methods are suited to post-hoc safety concerns but are not designed to calibrate trust; their computational cost must be balanced against monitoring performance and main-system resources.Source: 3) Interpretable Monitoring – Summary · p. 12
  • For prediction and planning, most monitoring methods use decision trees to predict binary safety labels or scalar risk scores from environment state.Two methods also consider visual cues from image data.Source: 3) Interpretable Monitoring – Summary · p. 12

D. Auxiliary Explanations

1) Auxiliary Explanations – Perception:

  • Grad-CAM and related heat maps explain perception predictions by highlighting influential image regions, including in camera-based semantic segmentation.Saravanarajan et al. applied Grad-CAM under synthetically generated haze and used four heat maps from encoder and decoder layers to increase transparency in decision understanding.Source: 1) Auxiliary Explanations – Perception · p. 13
  • Abukmeil et al. generated semantic-segmentation heat maps from second-order derivatives, while Mankodiya et al. used Grad-CAM and saliency maps for road segmentation.Abukmeil et al. aggregated and fused attention maps with the last decoder layer to improve results.Source: 1) Auxiliary Explanations – Perception · p. 13
  • Nowak et al. used bus-charger heat maps to identify spurious predictions and guide data augmentation, while Schinagl et al. generated perturbation-based LiDAR attribution maps.Schinagl et al. also proposed visual analysis tools for identifying potential misbehaviour in LiDAR-based 3D object detection.Source: 1) Auxiliary Explanations – Perception · p. 13
  • Analysis frameworks and toolboxes inspect traffic-light detection, CNN visualizations, 3D-detection failures, and semantic-concept weaknesses.Vatld analyzes input-output data and intermediate representations using disentangled semantic concepts; Wang et al. combine spatiotemporal information with CNN features, and ScrutinAI distinguishes input metadata to identify weaknesses tied to object concepts.Source: 1) Auxiliary Explanations – Perception · pp. 13–14

2) Auxiliary Explanations – Planning & Prediction: · 3) Auxiliary Explanations – End-to-end: · 4) Auxiliary Explanations – Summary:

Auxiliary explanations for planning, prediction, and end-to-end driving expose model focus or intermediate representations, but their interpretability does not guarantee faithful explanations. The sections describe attention-based approaches, textual and visual outputs, and limits on evaluating or trusting them.

2) Auxiliary Explanations – Planning & Prediction:

  • Most auxiliary-explanation methods use attention scores assigned to inputs or interpretable input embeddings, sometimes visualized as heat maps or bar graphs; Kochakarn et al. apply spatial and temporal attention to scene graphs.Jiang et al. found their model significantly faster than similar methods and competitive compared with baselines; Yu et al. gave no quantitative evaluation, only one qualitative example.Source: 2) Auxiliary Explanations – Planning &… · p. 14
  • Non-attention alternatives use post-hoc map heat maps to represent potential goals and planning uncertainty, or CRF feature scores and hard and soft rules for lane-change scenarios.The CRF method’s evaluation was limited and did not qualitatively establish how it improved overall interpretability.Source: 2) Auxiliary Explanations – Planning &… · p. 14

3) Auxiliary Explanations – End-to-end:

  • End-to-end auxiliary explanations include textual action descriptions, generated from driving video using attention-linked controllers and explanation models, with datasets and architectures extended across several studies, [156],,,,.BDD-X and BDD-OIA provide driving descriptions or object-focused explanations; later approaches added throttle control, transformer features, or interrelations among traffic objects.Source: 3) Auxiliary Explanations – End-to-end · p. 14
  • Human evaluation of BDD-3AA found object-level explanations more persuasive than pixel-level ones and additional text increased trust, but decisions and explanations may not correlate.The passage says explanation reliability still needs validation; another semantic-segmentation annotation extension did not qualitatively show added benefit.Source: 3) Auxiliary Explanations – End-to-end · p. 14
  • End-to-end systems also expose trajectories, semantic maps, saliency or attention maps, and semantic occupancy forecasts as intermediate outputs,,,,,.These outputs can represent perception or prediction information; one motion planner also uses occupancy forecasting as an input to its cost function.Source: 3) Auxiliary Explanations – End-to-end · pp. 14–15
  • Quantized network activations produced heat maps with improved performance over other attention-map methods, but intermediate outputs are not guaranteed reliable and may not explain end-to-end behavior.Visual outputs may offer transparency similar to modular architectures and help identify errors; Wei et al.’s claimed safety and interpretability benefits were not analyzed in evaluation.Source: 3) Auxiliary Explanations – End-to-end · p. 15

4) Auxiliary Explanations – Summary:

  • Attention weights can be interpretable to inspect yet model-specific, inconsistent, and potentially plausible but unfaithful, risking over- or under-reliance; heat maps are fragile and difficult to evaluate [43],.End-to-end heat maps highlight input regions, text gives action reasons, and intermediate outputs reveal internal perception or prediction, but those outputs do not directly explain decisions.Source: 4) Auxiliary Explanations – Summary · p. 15

E. Interpretable Safety Validation … B. Conceptual Framework: SafeX

Interpretable safety validation uses explainable methods to generate or analyze failure scenarios for offline safety assessment. SafeX integrates concrete XAI methods across the autonomous-driving stack so an explainable monitoring system can both answer users’ queries and provide module-level safety feedback.

E. Interpretable Safety Validation

  • Interpretable safety validation uses interpretable algorithms to generate adversarial behaviors for AV validation; methods include offline failure-case generation and scenario extraction, with temporal logic used to support safe reinforcement-learning policies.Corso and Kochenderfer use signal temporal logic to generate high-likelihood failures; DeCastro et al. use parametric STL formulas to relate policy parameters to emergent behaviors and configure desired behaviors for testing.Source: E. Interpretable Safety Validation · p. 15, 1) Interpretable Safety Validation –… · p. 15

A. Existing XAI frameworks

  • Omeiza et al. place an explainer between users and AD modules, while Atakishiyev et al. [18] use action explanations to support compliance checks through simulation and real-world testing.Both frameworks describe high-level XAI roles rather than SafeX’s finer-grained integration of methods across the AD stack.Source: A. Existing XAI frameworks · p. 16, B. Conceptual Framework: SafeX · p. 16
  • Brajovic et al. [188] describe an AI development cycle from use-case definition through operation and risk-proportionate monitoring, but their framework does not address AV application or user queries.Source: A. Existing XAI frameworks · p. 16

B. Conceptual Framework: SafeX

  • SafeX integrates concrete surveyed XAI methods into the full AD stack, with a finer-grained framework that enables safety monitoring and intelligible explanation delivery.The review presents SafeX after identifying three existing XAI frameworks for AD.Source: B. Conceptual Framework: SafeX · p. 16, §V · p. 16
  • SafeX’s explainable monitoring system bridges users and the AV, answering user queries with intelligible explanations while monitoring each driving module and providing safety feedback.The two functions are intended to support user understanding and trust as well as AV safety.Source: B. Conceptual Framework: SafeX · p. 16
  • For black-box modules, SafeX can combine auxiliary explanations, interpretable monitors, and surrogate models; inherently interpretable functions can provide traceable explanations or serve as fallbacks when black-box behavior is unexpected and unverifiable.Inherently interpretable functions are suitable when they meet performance requirements.Source: B. Conceptual Framework: SafeX · p. 16
  • One SafeX variant uses black-box modules, pairing perception heat maps and surrogate explanations with motion-planning explanations and interpretable checks of module outputs.Examples include a random forest for identifying perception errors, a concept-bottleneck model for pedestrian detection, a cognitive-model surrogate for planning, and a decision tree to check planner decisions.Source: B. Conceptual Framework: SafeX · pp. 16–17
  • A second SafeX variant uses inherently interpretable modules: a concept-bottleneck perception model, Monte Carlo Tree Search for planning, and a controller with an interpretable projection mechanism.The planning search tree helps interpret the planner’s decision.Source: B. Conceptual Framework: SafeX · p. 17

VI. DISCUSSION

The discussion identifies five XAI paradigms for safe and trustworthy autonomous driving and argues that they must be combined with stronger evaluation and monitoring. SafeX provides a modular framework for integrating explanatory methods with AD modules and delivering explanations while monitoring safety.

VI. DISCUSSION

  • The review identifies interpretable design, surrogate models, monitoring, auxiliary explanations, and validation, while finding control and LiDAR or fusion perception comparatively underexplored.End-to-end methods receive substantial attention, but most are limited to auxiliary explanations. The authors recommend explainable approaches for LiDAR and Radar and for sensor-fusion architectures.Source: §VI · pp. 17–18
  • Auxiliary explanations based on Shapley values, attention, or saliency can be inconsistent or incorrect, and their evaluations often lack interpretable metrics and qualitative failure analysis.The authors recommend rigorous quantitative and qualitative testing, particularly of failure cases; they also note such methods may be gamed or misinterpreted.Source: §VI · pp. 17–18
  • SafeX modularly combines XAI techniques across AD modules, while its explainable monitoring system delivers explanations to users and monitors module safety at runtime.The framework proposes multiple independent, co-supervisory explanatory functions that monitor underlying black boxes and one another, potentially providing emergency fallbacks.Source: §VI · p. 18
  • Interpretable safety validation remains underexplored, though interpreting environmental and causal factors can support extracting varied real-world scenarios and manipulating them to generate new ones.The discussion connects this approach to the challenge of testing AVs under the heavy-tailed distribution of driving scenarios.Source: §VI · p. 18
  • The review excludes data diversity, ethics, oversight, and natural-language explanation, and notes a disconnect between driving-stack explainability research and end-user needs.Explanations should reflect user requirements and be evaluated with human stakeholders; safe and trustworthy AI also requires cross-disciplinary work, uncertainty quantification, rigorous testing, documentation, and standardisation. XAI is only a partial measure.Source: §VI · p. 18
  • Generative methods are a future research direction because they may self-explain decisions and help address critical long-tail scenarios, but their efficacy remains unverified.The review notes that existing XAI algorithms may not apply to generative methods, which can further exacerbate black-box decision-making.Source: §VI · p. 18

VII. CONCLUSION

The review surveys XAI for safe and trustworthy autonomous driving, outlining trustworthy-AI requirements, explanation sources, and an XAI taxonomy. It identifies five application categories and proposes SafeX to integrate them into AD.

VII. CONCLUSION

  • The review organizes XAI for AD into five categories: interpretable design, interpretable surrogate models, interpretable monitoring, auxiliary explanations, and interpretable validation.Source: §VII · p. 19
  • To address the lack of detailed, research-grounded frameworks for AD safety requirements, the review proposes SafeX to incorporate all five categories of XAI methods.Source: §VII · p. 19
  • The review expects XAI for AD to attract increasing attention amid growing legal and social pressures, requiring collaboration across disciplines and stakeholders.It also anticipates a stronger need for XAI as LLM-based systems complicate the black-box problem; LLMs could eventually serve as explainers, with involvement from multiple disciplines and stakeholders.Source: §VII · p. 19
Loading 2402.10086v2…