Source-linked AI summary
Explainability of deep vision-based autonomous driving systems: Review and challenges
Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, Matthieu Cord
TL;DR
Vision-based autonomous-driving models require explainability because driving is safety-critical and difficult to test exhaustively, while deep models remain difficult to interpret. This survey synthesizes definitions, application-specific challenges, post-hoc explanation methods, and interpretable-by-design approaches, concluding with open challenges and future directions.
Problem
Safety-critical autonomous driving needs explanations because deployed models cannot be exhaustively tested and deep visual models are difficult to interpret.
Method
The survey organizes explainability research for behavior-cloned, vision-based self-driving systems into post-hoc methods and interpretable-by-design approaches.
Results
The survey reviews how post-hoc explanations reveal model behavior and how auxiliary driving information or designed representations support interpretability.
Takeaways & Limitations
Post-hoc methods preserve driving performance and can expose spurious correlations, whereas interpretable designs provide more control over explanation quality.
Takeaways & Limitations
Counterfactual explanations remain challenging for high-dimensional images because imperceptible perturbations can cause dramatically different outputs.
Abstract
from arXiv · showhide
This survey reviews explainability methods for vision-based self-driving systems trained with behavior cloning. The concept of explainability has several facets and the need for explainability is strong in driving, a safety-critical application. Gathering contributions from several research fields, namely computer vision, deep learning, autonomous driving, explainable AI (X-AI), this survey tackles several points. First, it discusses definitions, context, and motivation for gaining more interpretability and explainability from self-driving systems, as well as the challenges that are specific to this application. Second, methods providing explanations to a black-box self-driving system in a post-hoc fashion are comprehensively organized and detailed. Third, approaches from the literature that aim at building more interpretable self-driving systems by design are presented and discussed in detail. Finally, remaining open-challenges and potential future research directions are identified and examined.
1.1 Explainability in the context of autonomous driving
Explainability in autonomous driving is motivated by safety, incomplete testing, stakeholder needs, and the difficulty of interpreting high-dimensional visual inputs. The survey distinguishes human comprehensibility and explanation completeness, while separating intrinsic transparency from post-hoc interpretation.
- Motivations: Safety-critical driving cannot be exhaustively tested across all situations, motivating explanations as a fallback for understanding driving decisions.
- Motivations: Explanations serve different stakeholders and purposes, including debugging poor systems, supporting trust and adoption, and checking limitations against the Operational Design Domain.
- Motivations: Autonomous-driving explainability also reflects a mismatch between training objectives and the more complex real-life driving goal.
- Definitions: Explainability combines human interpretability with completeness, while interpretability includes intrinsic model transparency and post-hoc comprehension after training.
- Application-specific challenges: Explanations for vision-based driving models are especially challenging because visual inputs are high-dimensional, multimodal, and not explicitly semantic.
- Contextual elements: Explanation timing, concision, and information volume affect trust: explanations should precede action, while excessive detail may overwhelm users.
1.2 Autonomous driving: learning-based self-driving models
Learning-based self-driving systems replace hand-crafted pipelines with neural models, improving flexibility while creating substantial explainability challenges. These challenges span dataset coverage, model robustness, learning behavior, and the difficulty of disentangling intertwined driving tasks.
- From modular pipelines to end-to-end learning: Pipeline architectures divide driving into specialized perception, planning, decision, and control modules, making sensor processing somewhat interpretable.Their hand-crafted intermediate representations can limit uncertainty handling and generalization, while errors may propagate between modules.
- From modular pipelines to end-to-end learning: Neural driving systems learn from large quantities of expert recordings through behavior cloning, rather than relying exclusively on hand-crafted driving rules.This learning-based shift was enabled by advances in deep learning and computer vision.
- Explainability hurdles: Finite training datasets cannot cover every driving situation and may under- or overrepresent cases, while dataset biases create additional explainability and fairness concerns.The cited biases include omitted-variable, cause-effect, and sampling bias.
- Explainability hurdles: Highly nonlinear models lack robustness guarantees, can change behavior under small input changes, and raise questions about generalization to new scenarios.The survey also notes their susceptibility to adversarial attacks.
- Explainability hurdles: The learning phase may produce underfitting, overfitting, or reliance on spurious correlations instead of causal signals, without guaranteeing good generalization.These issues motivate questions about which factors drive model decisions.
- From modular pipelines to end-to-end learning: End-to-end networks jointly perform perception, reasoning, and decision-making, but leave limited room for structured explanations of how each step is achieved.Explaining autonomous driving therefore requires making intertwined tasks human-interpretable and understanding their interactions.
1.3 Survey organization
The survey focuses specifically on explainability in deep, vision-based self-driving models rather than surveying autonomous driving models generally. It organizes the literature around post-hoc explanations and interpretability introduced during system design, while identifying limitations and future directions.
- Scope and contributions: The survey reviews deep and vision-based self-driving models through the lens of explainability, distinguishing its scope from general surveys of self-driving models.It draws together perspectives from autonomous driving, computer vision, deep learning, and explainable AI.
- Scope and contributions: It clarifies interpretability and explainability concepts in the autonomous-driving context according to explanation type and computation method.The survey notes that related concepts have fuzzy and evolving boundaries.
- Scope and contributions: It aims to help regulators, engineers, and business stakeholders approach explainability methods cautiously in light of their limitations.The survey also frames interpretability and explainability as relevant to broader concerns including traceability, auditability, liability, and accountability.
- Future directions: The survey highlights limitations of X-AI methods and proposes directions toward more transparent, richer, and more faithful explanations for future self-driving models.It encourages exploration of areas such as causality, fairness, privacy, and reliability.
- Survey structure: Post-hoc methods explain already-trained deep networks, whereas design-based methods add explainability constraints to make self-driving processing more transparent and interpretable.These two categories structure the survey’s main technical sections.
2 Explaining a deep driving model
The survey organizes post-hoc explanations for deep driving models into local approximations, saliency methods, counterfactual analyses, and representation-level inspections, while emphasizing substantial fidelity, realism, and usability challenges.
- Local approximation methods: Local approximation methods build an interpretable proxy around an instance to approximate the trained model’s local input/output behavior.LIME uses decision trees or linear models as proxies.
- Post-hoc saliency methods: VisualBackprop saliency maps predicted PilotNet failures better than raw images, especially under adverse conditions.A student model was trained over saliency maps to predict PilotNet errors.
- Challenges: Post-hoc saliency methods are difficult to evaluate because persuasive human-facing explanations may be plausible without being faithful to the model.Automated evaluation can remove salient pixels, retrain models, or use annotated causal regions, but each approach has constraints.
- Challenges: Saliency maps can be model- and data-independent, vulnerable to attacks, and inconsistent across methods, motivating research on combining explanations.Some methods behave like edge detectors on randomly initialized models, while attacks can preserve predictions without highlighting important regions.
- Counterfactual explanations: Counterfactual explanations seek input changes that switch a model decision, but driving interventions must remain realistic, causally coherent, and minimal.High-dimensional visual inputs make naive counterfactual generation especially difficult because imperceptible changes can cause large output shifts.
- Counterfactual explanations: Causal interventions identify risk-objects by removing objects and observing whether the driving behavior changes, using only high-level behavior labels rather than strong localization supervision.Removing non-causal objects should not affect ego-vehicle behavior under the proposed intervention strategy.
- Explaining representations: Representation-level testing uses synthetic but realistic scene transformations to discover erroneous corner cases, whose insights can support retraining and improve accuracy.Inspections of neurons and intermediate representations provide valuable detail for experts but are tedious and poorly suited to regular car users.
3 Designing an explainable driving model
The survey distinguishes post-hoc explanations from models designed with interpretability constraints, reviewing input-, intermediate-, and output-level approaches for vision-based driving systems.
- Design principles: Explainable driving architectures incorporate interpretability constraints during design, unlike post-hoc methods applied to already-trained black-box models.Modular designs support forensic analysis by exposing understandable intermediate quantities alongside performance outputs.
- Input-level explanations: Input-level explanations communicate which perceptual information informs decisions through attention mechanisms or semantically structured inputs.Attention-based approaches weight image regions or objects, while semantic inputs make scene dimensions directly manipulable.
- Input-level explanations: Attention mechanisms can produce sparse saliency maps focused on relevant objects, such as surrounding cars, but attention is not guaranteed to faithfully represent model reasoning.The attentional bottleneck produces tighter, sparser modes than classical visual attention, while cited work challenges the assumption that attention is inherently explanatory.
- Semantic inputs: Semantically controllable inputs facilitate fine-grained evaluation, failure discovery, robustness analysis, and counterfactual manipulation without imperceptible adversarial perturbations.Manipulable dimensions include object attributes, weather, and removing specific cars.
- Intermediate- and output-level explanations: Trajectory, spatial-grid, and action predictions provide structured intermediate or output representations that can clarify driving behavior, while auxiliary predictions expose information encoded internally.Some systems also improve action decision performance by learning to predict explanations.
4 Use case: natural language explanations
Natural-language explanation systems generate textual justifications for driving decisions, using visual scenes and predicted controls or decisions as inputs. The survey highlights gains in semantic consistency alongside dataset, faithfulness, and evaluation challenges.
- Generating explanations: Natural-language explanation models are designed to produce concise, clear sentences that justify a self-driving system’s decisions for non-technical users.Driving systems can generate explanations from video, predicted controls, and visual attention aligned with the driving model.
- Driving applications: BDD-X supports post-hoc explanation generation by training an attention-based video-to-text model after the end-to-end driving system.The captioning module’s attention is constrained to align with the driving system’s attention.
- Driving applications: At reasonably low decoding temperatures, generated explanations for the same situation remain semantically consistent while varying syntactically and in completeness.Higher temperatures increase diversity, but excessive values may cause semantic drift.
- Limitations: Explainability datasets for driving may have poor automated-driving data quality, including single-camera recordings and missing sensor calibration.The survey argues for explanation datasets built on high-quality driving datasets.
- Limitations: Explanation annotations are difficult to acquire and may encode human post-hoc rationales rather than the cues actually used by the learned driving model.Behavior cloning on explanations assumes that model and human decisions rely on the same reasons, which can prevent discovery of new cues.
- Limitations: Natural-language evaluation based on human judgments or metrics such as BLEU, METEOR, and CIDEr assesses plausibility but cannot guarantee faithfulness to model decision-making.Such protocols can encourage persuasive explanations and expose evaluators to confirmation bias.
5 Conclusion
The survey synthesizes explainability challenges for deep-learning self-driving systems and organizes methods into post-hoc explanations and interpretable-by-design architectures. It identifies evaluation, high-dimensional visual inputs, and limited use of causal and other X-AI tools as central open challenges.
- Explainability needs vary with the person seeking an explanation, their expertise, and the time available for analysis.
- Post-hoc methods explain trained driving models without compromising driving performance and can transfer across architectures to some extent.They may expose spurious correlations but do not change the underlying model.
- Interpretable-by-design models offer greater control over explanation quality but may degrade driving performance and generally lack architectural transferability.
- Explanation evaluation lacks unified methods because automated measures depend on the explanation scheme and data, while human ratings may not scale or may reward persuasive explanations.Completeness is especially difficult to assess for compositional explanations because possible fact combinations grow rapidly.
- Vision-based driving models remain difficult to explain because their inputs are high-dimensional, multimodal, and semantically implicit.These properties make many off-the-shelf X-AI methods difficult to transfer directly to autonomous driving.
- Causal modeling remains rarely applied to explain self-driving models, despite the reviewed methods seeking to expose causes behind model decisions.
6 Declarations
The authors report Valeo funding and no other funding for the manuscript, with no relevant financial or non-financial interests disclosed.
- The survey was funded by Valeo, received no other preparation funding, and reports no relevant financial or non-financial interests.