Source-linked AI summary
Textual Explanations for Self-Driving Vehicles
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, Zeynep Akata
TL;DR
End-to-end self-driving controllers need explanations that help users understand what triggered their behavior, but rationalizations may not reflect the controller’s internal state. The paper grounds video-to-text explanations in controller attention, evaluates aligned introspective models and rationalizations on BDD-X, and finds that grounded introspective explanations are accurate and attention alignment is effective.
Problem
Deep vehicle controllers are cryptic, while rationalizations are post hoc and less helpful for understanding causal behavior grounded in the vehicle’s internal state.
Method
The model trains a convolutional vehicle controller with visual attention and aligns that attention with an attention-based video-to-text explanation generator, using BDD-X annotations.
Results
Grounded introspective models generate accurate human-understandable descriptions and explanations, and attention alignment effectively combines the controller with the justification model.
Takeaways & Limitations
BDD-X enables training and automatic evaluation of interpretable driving justifications by comparison with human annotations.
Takeaways & Limitations
Human explanations of what the vehicle controller was thinking are impossible to obtain, so ground-truth explanations are often observer-generated rationalizations.
Abstract
from arXiv · showhide
Deep neural perception and control networks have become key components of self-driving vehicles. User acceptance is likely to benefit from easy-to-interpret textual explanations which allow end-users to understand what triggered a particular behavior. Explanations may be triggered by the neural controller, namely introspective explanations, or informed by the neural controller's output, namely rationalizations. We propose a new approach to introspective explanations which consists of two parts. First, we use a visual (spatial) attention model to train a convolutional network end-to-end from images to the vehicle control commands, i.e., acceleration and change of course. The controller's attention identifies image regions that potentially influence the network's output. Second, we use an attention-based video-to-text model to produce textual explanations of model actions. The attention maps of controller and explanation model are aligned so that explanations are grounded in the parts of the scene that mattered to the controller. We explore two approaches to attention alignment, strong- and weak-alignment. Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments. We evaluate these models on a novel driving dataset with ground-truth human explanations, the Berkeley DeepDrive eXplanation (BDD-X) dataset. Code is available at https://github.com/JinkyuKimUCB/explainable-deep-driving.
1 Introduction
The paper addresses the opacity of end-to-end vehicle controllers by generating textual explanations grounded in the controller’s visual attention. It distinguishes introspective explanations from rationalizations and introduces the BDD-X dataset for evaluating explainable driving models.
- Motivation: Deep neural vehicle controllers are effective but cryptic, motivating explanations for user acceptance, behavior understanding, and communication.The paper identifies the absence of explainable states or labels in typical deep networks as a central challenge.
- Approach: The model combines a vehicle controller with an explanation generator so attended regions influence textual explanations of driving actions.The controller predicts acceleration and change of course, while the explanation model produces natural-language rationales.
- Explanation types: Introspective explanations use the system’s internal state, whereas rationalizations justify behavior post hoc.The paper argues that introspective explanations better support understanding causal model behavior and communication grounded in the vehicle’s internal state.
- Grounding explanations: Visual attention can identify image regions potentially affecting controller outputs, but textual explanations are more convenient for passengers than replaying attention maps.Attention constrains possible reasons for actions, while natural language can connect actions to specific scene elements.
- Data challenge: Human explanation data are difficult to obtain because annotations are often rationalizations and cannot reveal what the vehicle controller was thinking.The authors report that attention alignment generally improves agreement with human rationalizations despite this ground-truth limitation.
- Dataset: BDD-X contains over 6,984 video clips annotated with driving descriptions and explanations, providing a test bed for explainable self-driving models.The dataset supports training and automatic evaluation against human annotations.
2 Related Work
Prior work covers end-to-end vehicle control and visual or textual explanation, but this paper connects controller attention with natural-language justification for driving decisions.
- End-to-End Learning for Self-Driving Cars: Mediated perception relies on recognizing human-designated features and generally requires demanding parameter tuning for balanced performance.Examples include lane markings, traffic lights, pedestrians, and cars.
- End-to-End Learning for Self-Driving Cars: End-to-end driving approaches learn policies directly from observation-action pairs, including networks mapping dashcam image streams to steering controls.This contrasts with mediated perception systems that depend on human-designated features such as lanes, traffic lights, pedestrians, or cars.
- Visual and Textual Explanations: The proposed driving model visualizes attended image regions and generates textual descriptions and explanations of the triggers for particular driving decisions.Its vehicle controller learns control commands, while the explanation component produces natural-language output.
- Visual and Textual Explanations: Psychological work describes explanations as aids for learning and understanding by filling information gaps according to prior knowledge and goals.This motivates explanations tailored to what end-users need to understand.
- Visual and Textual Explanations: Explainability research includes deconvolution, pixel-contribution methods, attention-based causal filtering, and textual justification models.The paper positions attention and natural language as complementary ways to expose relevant visual regions and explain decisions.
3 Explainable Driving Model
The model jointly predicts driving controls and generates textual explanations, grounding the language in visual regions attended to by the controller. Strong alignment reuses controller attention directly, while weak alignment trains a separate explanation attention map toward it.
- Vehicle Controller: The convolutional encoder preserves spatial information by representing each frame as feature vectors associated with distinct image regions.Max-pooling layers are omitted to prevent spatial information loss.
- Vehicle Controller: The vehicle controller maps dashcam images and sensor measurements to acceleration and change-of-course predictions using trainable spatial soft attention.Attention weights are applied to convolutional features and pooled into context vectors for the controller.
- Vehicle Controller: Entropy regularization encourages sparse controller attention maps that emphasize relatively few image regions.The hyperparameter λc controls the strength of this regularization.
- Attention Alignments: Strong alignment feeds controller context vectors directly to the explanation module, whereas weak alignment uses its own spatial attention constrained by KL divergence toward controller attention.Both approaches use temporal attention, but they differ in the source of spatially attended context vectors.
- Textual Explanation Generator: The explanation generator produces variable-length descriptions and justifications from video sequences, using a separator token while training on one combined sequence.Its LSTM predicts the sequence and uses temporal attention over context vectors.
- Textual Explanation Generator: The justifier combines predicted acceleration and change of course with spatially attended visual context before updating the LSTM.Training minimizes textual negative log-likelihood together with vehicle-control and attention-alignment losses.
4 Berkeley DeepDrive eXplanation Dataset (BDD-X)
BDD-X extends dashcam driving videos with time-stamped human descriptions and justifications of driving actions. It contains diverse driving conditions and large-scale annotations, with measured agreement between annotators.
- Dataset Construction: BDD-X contains dashcam videos of urban, residential, and other road types across varied weather, lighting, and driving conditions.Videos are approximately 40 seconds long and include time-stamped vehicle sensor measurements.
- Annotation: Human annotators act as driving instructors, describing driver actions and explaining why they occurred with start and end timestamps.Descriptions and justifications are entered separately, such as a lane change paired with a stopping school bus as its reason.
- Dataset Statistics: BDD-X comprises over 77 hours of driving in 6,984 videos, with over 26K activities across more than 8.4M frames.The train, validation, and test splits contain 5,588, 698, and 698 videos, respectively.
- Inter-human Agreement: Annotator agreement on 998 doubly annotated training videos included average temporal IoU 0.63 and higher CIDEr scores when descriptions matched.For IoU > 0.5, CIDEr scores were 142.60 for descriptions and 97.49 for justifications; identical descriptions yielded justification CIDEr 200.72.
- Coverage of Justifications: The dataset contains 906 action-description words and 1,668 justification words, indicating greater vocabulary diversity for justifications.Its annotations cover typical driving activities in urban driving videos.
5 Results and Discussion
The experiments evaluate the vehicle controller and textual justifier quantitatively and qualitatively. Attention alignment generally improves explanation quality, while stronger controller attention sparsity trades some control performance for better explanations.
- Vehicle controller: Higher controller entropy regularization produces sparser attention maps, improving textual explanation generation while slightly degrading control performance.The results indicate a tension between sparse, more interpretable maps and controller performance.
- Evaluation setup: The models are evaluated with controller metrics, automatic text-generation metrics, human judgments, and qualitative analysis.Controller evaluation uses mean absolute error and distance correlation; textual explanations use BLEU, METEOR, CIDEr-D, and human evaluation.
- Vehicle controller: The controller predicts acceleration and change of course using a CNN-based end-to-end architecture, with prior inputs and attention variants compared against related models.The study compares variants using identical CNNs and reports models with different entropy regularization coefficients.
- Vehicle controller: Four consecutive frames and prior inputs improve control prediction accuracy, while visual attention also provides improvements over the corresponding baseline.The model without entropy regularization performs best among the compared CNN-based approaches, especially for acceleration estimation.
- Textual explanations: Weakly aligned attention generally gives the best explanation results, and introspective models score higher than rationalization models for explanation generation.Description scores are more mixed, although most introspective model scores are higher; one example shows introspective explanations using the front vehicle while rationalization does not.
- Textual explanations: Human evaluation finds the WAA model outperforms the other two models, while rationalization can outperform SAA on a subset.The authors suggest that SAA may rely on counterfactually important regions that can confuse the explanation module.
- Textual explanations: Qualitative examples show plausible and diverse descriptions and explanations across driving forward, slowing or stopping, and turning situations.Stopping explanations include red lights, stop signs, and traffic, with attention visualizations showing the associated evidence.
6 Conclusion
The work presents an end-to-end explainable driving model with grounded introspective explanations and reports benefits from attention, prior inputs, and attention alignment. It also identifies causal filtering, stronger perception, and driver gaze as future directions.
- The model incorporates a grounded introspective explanation component into an end-to-end explainable driving system.
- Attention mechanisms and prior inputs improve vehicle control prediction accuracy compared to baselines.
- The grounded introspective model generates accurate, human-understandable textual descriptions and explanations for driving behaviors.
- Attention alignment combines the vehicle controller and justification model, while BDD-X supports training and automatic evaluation against human annotations.
- Future work includes causal filtering of attention maps, stronger perception pipelines, and incorporating driver eye gaze.The proposed directions aim to reduce spurious attention regions, improve visual grounding and explanation quality, and mimic driver behavior.
Supplemental Material: Textual Explanations for Self-Driving Vehicles
The supplemental material documents the BDD-X dataset and approach, including annotation procedures, dataset details, implementation details, and additional qualitative examples.
- The supplemental document provides more details on the BDD-X dataset and the presented approach.
- S.1 Our Amazon Mechanical Turk annotation interface: Section S.1 describes the Amazon Mechanical Turk annotation interface and data collection procedure.
- S.2 Berkeley DeepDrive eXplanation (BDD-X) Dataset Details: Section S.2 covers collected descriptions and explanations in BDD-X.
- Section S.3 gives implementation details, while Section S.4 presents qualitative attention-map and textual-explanation examples.
S.1 Our Amazon Mechanical Turk annotation interface
The annotation interface separates action descriptions from action justifications so annotators provide both what the vehicle does and why the driver performed the action.
- Annotators separately enter an action description and an action justification, such as lane movement and stopping traffic ahead.
- Preliminary studies found that separate annotation boxes helped annotators understand and perform the task better.
S.2 Berkeley DeepDrive eXplanation (BDD-X) Dataset Details
BDD-X contains action descriptions and explanations covering common driving behaviors, traffic situations, environmental conditions, and examples requiring complex reasoning or attention to detail.
- Common action descriptions concern speed changes, driving forward, and turning, with additional behaviors including merging, lane changes, reversing, parking, and wiper use.
- BDD-X dataset details include top-30 word counts for action descriptions and explanations, counting all word forms.
- Explanations reference traffic, lanes, lights, signs, pedestrians, and passing vehicles, as well as weather, vehicle types, road bumps, and safety.
- Example annotations include explanations requiring complex reasoning and others demonstrating attention to detail.
S.3 Implementation Details
The implementation predicts acceleration and change of course, using exponential smoothing for course changes and LSTM networks for control and explanation generation.
- Preprocessing: The model predicts two vehicle control commands: acceleration and change of course.The change of course is computed from the current vehicle course and a smoothed course value.
- Preprocessing: Change of course is computed as the difference between the current course and an exponentially smoothed course.The smoothing factor is αs = 0.01 in this paper.
- Preprocessing: The smoothing factor αs ranges from 0 to 1, with smaller values producing stronger smoothing and less responsiveness to recent changes.The original time series is unchanged when αs = 1.
- LSTM network: An LSTM network is used in both the vehicle controller and explanation generator.The LSTM internal states are conditioned on the hidden state and an α-weighted context vector.
- LSTM network: The LSTM updates memory and hidden states using element-wise products between its gates and states.The memory state combines the forget-gated previous memory with the input-gated candidate state, while the hidden state applies the output gate.
- LSTM network: Initial LSTM memory and hidden states are obtained from averaged convolutional feature slices passed through two additional hidden layers.The initialization uses the layers finit,c and finit,h.
S.4 Additional Examples
Additional examples demonstrate the model's visual and textual explanations, including novel explanations that were not present in the training set.
- Additional Examples: The additional-examples section presents driving decisions using attention visualization and textual justification.The examples exploit both visual attention and generated explanations.
- Additional Examples: The model generates novel explanations absent from the training set.Examples describe unobstructed lanes, slowing vehicles, turns, stopped cars, and empty intersections.
- Additional Examples: Supplementary figures provide further examples of descriptions and explanations using the paper's selected loss weights and separator-token convention.Both figures use (λc,λa) = (100,10), with a synthetic separator token replaced by '+'.