PaperlayerView exact revision on arXiv

Source-linked AI summary

Explanations in Autonomous Driving: A Survey

Daniel Omeiza, Helena Webb, Marina Jirotka, Lars Kunze

arXiv:2103.05154v4cs.HCcs.AIcs.CYcs.LGcs.RO

TL;DR

Autonomous vehicles use complex black-box AI, creating a need for explanations that support transparency, trust, accountability, and regulatory compliance. This paper surveys explainability research across AV operations and stakeholder needs, categorises existing approaches, identifies challenges, and proposes recommendations including a conceptual framework. It concludes that AV explainability faces challenges including limited faithfulness, biased and non-representative datasets, insufficient regulation, and a need for more interdisciplinary research.

  • Problem

    Black-box autonomous vehicles comprise interacting subsystems, but existing explainable AI research rarely explains the behaviour of the entire AV for its diverse stakeholders.

  • Method

    The paper conducts a comprehensive survey covering stakeholder requirements, explanation categorisations, AV operations, regulations, standards, and recommendations for AV explainability.

  • Results

    The survey identifies key explainability challenges, including low faithfulness assurance, explanation bias from finite non-representative datasets, and gaps in concrete AV explainability regulations.

  • Takeaways & Limitations

    AV explanations should be intelligible and personalised to stakeholder needs while supporting transparency, accountability, trust, accident investigation, and compliance assessment.

  • Takeaways & Limitations

    The survey focuses on autonomous vehicles rather than uninhabited aerial or autonomous underwater vehicles to cover the topic in sufficient depth.

Abstract

from arXiv · show

The automotive industry has witnessed an increasing level of development in the past decades; from manufacturing manually operated vehicles to manufacturing vehicles with a high level of automation. With the recent developments in Artificial Intelligence (AI), automotive companies now employ blackbox AI models to enable vehicles to perceive their environments and make driving decisions with little or no input from a human. With the hope to deploy autonomous vehicles (AV) on a commercial scale, the acceptance of AV by society becomes paramount and may largely depend on their degree of transparency, trustworthiness, and compliance with regulations. The assessment of the compliance of AVs to these acceptance requirements can be facilitated through the provision of explanations for AVs' behaviour. Explainability is therefore seen as an important requirement for AVs. AVs should be able to explain what they have 'seen', done, and might do in environments in which they operate. In this paper, we provide a comprehensive survey of the existing body of work around explainable autonomous driving. First, we open with a motivation for explanations by highlighting and emphasising the importance of transparency, accountability, and trust in AVs; and examining existing regulations and standards related to AVs. Second, we identify and categorise the different stakeholders involved in the development, use, and regulation of AVs and elicit their explanation requirements for AV. Third, we provide a rigorous review of previous work on explanations for the different AV operations (i.e., perception, localisation, planning, control, and system management). Finally, we identify pertinent challenges and provide recommendations, such as a conceptual framework for AV explainability. This survey aims to provide the fundamental knowledge required of researchers who are interested in explainability in AVs.

I. INTRODUCTION

Autonomous vehicles raise important questions about trust, accountability, transparency, and the ability to explain decisions made by complex, interacting systems. This survey addresses these needs by reviewing explainability research across AV operations and stakeholder requirements.

  • Advances in sensing devices and deep learning have supported the growth of autonomous vehicles, but real-world deployment depends partly on users’ perceptions of safety and trust.Accidents involving AV technologies can hamper trust, while explanations are considered one way to build confidence.
  • Explanations can provide observations, road rules, and traffic signs that help investigate the causes of AV accidents.The paper uses the Molly problem to illustrate how post-hoc explanations can inform accident investigation.
  • Black-box AV decisions may appear strange or confusing to end-users, making explainability especially relevant for highly automated vehicles.The paper connects this concern with broader emphasis on explainable AI and the GDPR’s right to explanation.
  • Existing explainable AI research often focuses on individual neurons or models, whereas autonomous vehicles are goal-based systems with interacting subsystems.The survey therefore treats explanations for the behaviour of the whole AV as essential.
  • The survey provides a structured overview of explanations in autonomous driving, covering perception, localisation, planning, vehicle control, and system management for different stakeholders.It also proposes recommendations including a conceptual framework, possible user interfaces, and regulatory considerations, while focusing on AVs rather than UAVs or AUVs.
  • Accountability in autonomous driving is difficult because multiple operations and stakeholders can create responsibility gaps.The paper frames explanations as intelligible accounts that support assigning responsibility and communicating causes and effects to relevant stakeholders.

B. Trust

Explanations can help users understand AV behaviour and support calibrated trust, while empirical studies report that explanations can influence trust. However, GDPR-related explanation requirements and current AV regulations leave explainability incompletely addressed.

  • B. Trust: Information about an AV’s operating modes and behaviour can improve users’ understanding and help construct calibrated trust.Explanations are particularly relevant when the vehicle acts outside user expectations.
  • B. Trust: Meaningful explanations to passengers, pedestrians, and other road participants are proposed as one way to build trust in AV technology.The cited empirical studies report that explanations can influence trust, which is discussed as relevant to adoption of automated driving systems.
  • B. Trust: The GDPR requires meaningful information about the logic and likely consequences of decisions made using individuals’ data.The paper calls this requirement the “right to explanation.”
  • B. Trust: AVs may collect sensitive location and travel-pattern information, so the paper argues they should not be exempt from GDPR explanation provisions where the regulation applies.The passage links AV data collection to the relevance of GDPR requirements.
  • B. Trust: Existing Scottish and other cited AV regulations do not directly address the explainability challenge.The paper identifies this as an unresolved regulatory issue.

B. AV Standards

AV standards and stakeholder needs shape how explanations should be designed and communicated. The survey distinguishes stakeholder groups and emphasizes personalisation because explanation detail, type, and communication mode vary by recipient and purpose.

  • B. AV Standards: ISO TC204, IEEE, and related organisations have established standards for AVs and intelligent transport systems, including safety and information or data exchange.The paper states that these standards directly or indirectly demonstrate the necessity of explainability in AVs.
  • Stakeholders: Explanation requirements vary across stakeholders, so the level of detail, explanation type, and communication mode should be personalised.Lay users may prefer user-friendly explanations, whereas developers and engineers may prefer technically detailed explanations.
  • Stakeholders: The survey groups stakeholders into Class A end-users and society, Class B technical groups, and Class C regulatory bodies including insurers.This categorisation is used to organise stakeholder explanation requirements.
  • Stakeholders: Passengers interact with the AV’s explanation agency without responsibility for driving operations, while auxiliary drivers may also participate in driving.Auxiliary drivers mainly occur in SAE Level 3 and 4 vehicles.
  • Stakeholders: Pedestrians, pedestrians with reduced mobility, and other road participants may communicate intentions through gestures or an external human-machine interface.Other road participants include cyclists and other vehicles.
  • Stakeholders: Technical stakeholders include AV developers and automobile technicians, while regulatory stakeholders include system auditors, regulators, accident investigators, and insurers.Their roles span development, maintenance, compliance, accident investigation, and financial protection.

IV. EXPLANATION CATEGORISATIONS

The survey categorises AV explanations by purpose, style, model dependence, interactivity, system type, scope, and design methodology. It also identifies underrepresented explanation categories in the autonomous-driving literature.

  • Explanation dimensions: Explanation methods are evaluated and designed according to context and purpose, using categories such as causal filters and explanation styles.Causal filters include factual, contrastive, and counterfactual investigatory queries.
  • Design methodologies: Explanation-design methodologies are categorised as unvalidated guidelines, empirically derived approaches, or psychological constructs from formal theories.The categories reflect whether methods rely on researcher experience, user studies, or psychological theories.
  • Explanation dimensions: Causal filters support factual, contrastive, and counterfactual questions such as why, why not, how to, and what if.These queries select causes relevant to interpreting an observation with respect to existing knowledge.
  • Explanation dimensions: Explanation styles include input influence, sensitivity, case-based, and demographic explanations, distinguished by the information they reference and present.Input influence reports variable effects, sensitivity reports changes needed to alter an output class, case-based explanations retrieve similar training cases, and demographic explanations aggregate outcomes by demographics.
  • Explanation dimensions: Further dimensions classify methods as model-agnostic or model-specific, interactive or noninteractive, data-driven or goal-driven, and local or global.The survey notes that only SHAP, among the discussed model-agnostic techniques, had been used in autonomous driving to the authors’ knowledge.
  • Surveyed literature: Sensitivity, demographic, contrastive, counterfactual, model-agnostic, and global explanations are rare in the surveyed autonomous-driving literature.The survey attributes this pattern potentially to the nascent nature of explainable autonomous driving.

V. EXPLAINABLE AUTONOMOUS DRIVING OPERATIONS

The survey reviews explanations across the main operations of an autonomous vehicle and illustrates posthoc explanation delivery through a mobile interface. These operations include perception, localisation, planning, control and navigation, and system management.

  • AV operations: The reviewed AV operations are perception, localisation, planning, control and navigation, and system management.System management includes event data recorders and human-machine interaction.
  • AV operations: The survey examines the role of explanations within these key AV operations.The operational review is organised around the vehicle’s major functional components.
  • Explanation delivery: A mobile AV explainer can provide posthoc explanations at strategic journey points through three different views.The interface records the journey and presents explanations after events have occurred.

A. Perception

The perception review covers datasets, data-driven explanation methods, and applications of those methods to autonomous-driving perception. It highlights gradient-based techniques and multimodal explanations combining saliency maps with text.

  • Perception review: The survey reviews perception datasets and data-driven explanation methods applied to autonomous-driving perception.Datasets are considered both for existing use and potential use in explanation generation.
  • Datasets: Driving datasets support posthoc explanations through annotations such as handcrafted explanations, vehicle trajectories, human-driver behaviour, and anomaly bounding boxes.The survey categorises dataset sensors into exteroception and proprioception.
  • Gradient-based explanations: Gradient-based methods explain CNN predictions by backpropagating gradients from the output layer to the input layer and commonly producing heatmaps.The survey lists CAM variants, VisualBackProp, LRP, DeepLift, and Guided-Backpropagation among these methods.
  • Vision-based explanations: VisualBackProp visualises input-image super-pixels that are most influential to CNN predictions in an end-to-end autonomous-driving system.The method was applied to PilotNet to examine influential parts of driving-scene images.
  • Vision-based explanations: One reviewed approach predicts acceleration and course-change commands from images, then generates textual explanations using an attention-based video-to-text model.The approach, trained on BDD-X, provides both saliency maps and text.
  • Connection to localisation: Localisation explanations can communicate positional errors over time to support error correction, reliability assessment, and timely intervention before accidents.The survey describes transmitting position errors to an operation centre and displaying them through dashboards or mobile applications.

C. Planning

The planning review presents explanations as a way to make complex AV decisions understandable to stakeholders. It covers interactive plan comparison, refinement-based planning, and human-machine collaborative planning, alongside interfaces for querying control decisions.

  • Planning role: Planning generates action sequences that influence an agent’s online decisions and behaviours in a dynamic environment.Planning is important for AVs because they perform complex manoeuvres in urban and cluttered settings.
  • Plan explanations: Explainable planning translates agent plans into understandable forms and uses interfaces that help stakeholders interpret them.The survey identifies explanation design and user-interface design as parts of this process.
  • Planning frameworks: XAI-PLAN lets users explore alternative actions, compares their resulting plan with the planner’s plan, and explains discrepancies.This interaction supports mixed-initiative planning.
  • Planning frameworks: Refinement-based Planning produces verbal-plan explanations on user request and uses forward and backward search to detect flaws and update or optimise plans.Its representation combines partial-order causal-link planning with hierarchical planning.
  • Planning frameworks: Why-Plan compares human and autonomous-agent objectives in path planning and explains differences in a human-friendly way.It addresses questions such as why the agent’s plan includes a particular action.
  • Interactive interfaces: In-vehicle interfaces should support stakeholder queries and return navigation or control explanations through voice, text, visual, gesture, or combinations of these forms.Investigatory queries may include why and why-not questions in contexts such as near-misses or collisions.

VI. SYSTEM MANAGEMENT

System management explainability depends on recording and sharing sufficient data to reconstruct AV behaviour, while current regulations and EDR practices remain incomplete for autonomous driving.

  • Data recording and accountability: Event data recorders support posthoc analysis of vehicle faults and accidents, but existing EDRs may not capture enough information to reconstruct behaviour before and after a crash.The paper links this gap to difficulty assigning fault among traffic participants.
  • Data recording and accountability: NHTSA has called for a uniform industry approach to data recording and sharing that could support explanations and accident investigation.
  • Data recording and accountability: Blockchain-inspired EDRs and broader data models have been proposed to improve accident forensics, verification, and reconstruction of AV behaviour.
  • Regulation and scope: Emerging EDR legislation raises whether current storage facilities can meet the data requirements for automated-vehicle accident investigations.
  • Regulation and scope: Higher automation levels increase the challenge of making AV operational modes understandable, motivating richer in-vehicle technologies and interfaces.

1) Novel Interaction Technologies:

Novel in-vehicle interaction technologies can improve driver experience, awareness, and comfort, but user preferences and causal explanations remain central design concerns.

  • Interaction technologies: Displaying uncertainty values associated with vehicle estimates can improve drivers’ experience and attitudes toward varying resource conditions.
  • Interaction technologies: Virtual traffic signs improved drivers’ concentration and reactions to road signs, while peripheral light displays enhanced driving performance and comfort.
  • User preferences: Wizard-of-Oz participants preferred shared control, clear handover alerts, and correct responses even when delayed or requests were not performed.
  • User preferences: Higher vehicle autonomy was associated with lower driver vigilance and awareness, while under-reporting hazards elicited stronger driver reactions.
  • User preferences: Individual preference alignment with AV decisions produced more positive changes in vehicle impressions than unaligned decisions.
  • Explanation evaluation: Studies found that explanation type can affect trust, whereas explanations did not influence user experience during or after one simulated ride.
  • Explanation evaluation: Existing interfaces often provide vehicle information without communicating the reasons or causal links behind decisions.

3) AV and External Agents Interaction:

AV interaction with pedestrians, cyclists, and other road participants increasingly uses status, intent, and awareness interfaces, but further research is needed to turn findings into effective explanation systems.

  • Pedestrians: Status and intent eHMI increased pedestrian trust, reduced crossing onset time, and produced rapid trust recovery after AV malfunction.Status eHMI alone could cause overtrust, so intent messages were needed to complement it.
  • Pedestrians: Studies of ghost vehicles reported conflicting pedestrian responses, with one finding little hesitation and another finding different behaviour when the driver was hidden.
  • Pedestrians with reduced mobility: Visual cues were identified as the most important interface elements for pedestrians with reduced mobility, with street infrastructure preferred for hosting them.
  • Other road participants: AV-cyclist interfaces improved rider confidence during lane merging, supporting communication of vehicle awareness and intent from sensor data.
  • Research needs: Researchers identify a need to determine how interaction-study findings can produce effective and efficient explanation interfaces for AV stakeholders.
  • Standards and regulation: Relevant standards address data quality, but AV explainability also depends on presentation style, language, and the interfaces used to provide data.
  • Standards and regulation: Current explainability regulations are generally framed and do not directly specify requirements for particular AV technologies and stakeholders.

1) Outcome-based explainability:

The proposed outcome-based explainability framework organizes AV explanations around perception, decision-making, and action, with real-time or recorded access to the data supporting each stage.

  • Outcome-based explainability: Outcome-based explainability covers what the AV perceives, how it plans and decides, and how it executes those decisions.
  • Perception: Perception explanations should expose sensor-to-world transformations, digital object representations, object states, detection processes, and confidence levels.
  • Framework: The conceptual framework combines perception and localisation inputs with planning and control data, delivering explanations immediately or through timestamped logs and a multimodal interface.
  • Perception: Authorised entities should be able to access the AV’s real-time semantic 3D world representation onboard or remotely.
  • Decision: Decision explanations should describe planning operations, environmental observations, constraints, risks, considered plans, and selected plans.
  • Decision: AVs should provide uncertainty levels for current and possible future environmental representations, including the threshold used to select plans or risk mitigation actions.
  • Action: Action explanations concern the vehicle dynamics produced by continuous control inputs in response to perceived circumstances and decisions.

2) Process-based Explainability:

Process-based explainability supports independent assessment of an AV’s operations and governance by combining operational and governance information. The proposed framework links AV operations, an event data recorder, and an explainer to support investigation, governance, and stakeholder assessment.

  • 2) Process-based Explainability:: Process-based explainability combines perception, decision, action, and governance data to enable independent assessment of the entire AV operation.It can support reconstructing an event or accident immediately after it occurs.
  • 2) Process-based Explainability:: Process explanations can support fairness, safety and performance assessment, accountability and responsibility, and impact assessment.They can describe design and implementation steps, system performance, responsible actors, and broader effects.
  • 2) Process-based Explainability:: The authors suggest authorised authorities receive real-time access to operational and process-management data and have powers to impose sanctions for failures.The proposal covers both in-vehicle and remote access.
  • 3) Conceptual Framework for Explainable AV:: The proposed framework fuses perception and localisation inputs from LiDAR, radar, GPS, and IMU sensors before passing environmental representations to planning.Planning estimates risks, identifies mitigation actions, and produces a trajectory for vehicle control and actuation.
  • 3) Conceptual Framework for Explainable AV:: An explainer receives data from AV operations and the event data recorder, generating bulk explanation logs or conversational responses.Logs support process-based accident investigation and governance, while conversational explanations support outcome-based explanations.
  • 1) Trust:: Existing AV explanation research relies heavily on trust studies using synthetic laboratory tasks or microworlds rather than progressive real-world experiments.The authors recommend real-world settings to obtain more accurate trust results and broader stakeholder-focused testing.

C. Ethics: AV Operations

The survey identifies faithfulness and bias as major challenges for explanations in autonomous driving. It recommends using intermediate operational data, transparent representations, broader testing, and more inclusive research and regulation.

  • 1) Faithfulness and Blackboxness:: Existing explanation models have low assurance of faithfulness because intermediate AV-operation data are missing from nearly all autonomous-driving datasets.Human rationalisations used to annotate datasets and train explanation models make auditing and faithfulness evaluation difficult.
  • 1) Faithfulness and Blackboxness:: Including intermediate operations such as planning could improve explanation faithfulness when AV behaviours are assessed on a non-binary scale.Navigation graphs and decision trees are suggested as transparent representations for intelligible explanations.
  • 1) Faithfulness and Blackboxness:: High-level commands such as turning right and lane changes can represent transitions between road and lane segments and interactions with other road participants.These commands are proposed for representing navigation-plan transitions intelligibly.
  • 1) Faithfulness and Blackboxness:: The SAX project collects proprioceptive vehicle data, including wheel angle, yaw, acceleration, and braking, alongside recorded driver and passenger thoughts.The spoken thoughts may inform analysis of how natural-language explanations should be provided.
  • 2) Explanation Bias:: Explanation models can inherit bias from finite, non-representative driving datasets and from model specifications that omit variables influencing AV actions.Regional differences in traffic rules and signs further complicate representation across driving scenarios.
  • 2) Explanation Bias:: The authors call for research inclusivity involving researchers, regulators, and end-users, while regulators may use the paper’s suggestions for explainability guidelines.The proposed scope is to address highlighted issues in AV explainability.
  • VIII. CONCLUSION: The survey identifies missing concrete regulations, insufficient interdisciplinary research, and dataset-related bias and faithfulness issues as key explainability challenges.It proposes AV-specific regulations, a conceptual explainability framework, and more interdisciplinary research.
Loading 2103.05154v4…