Source-linked AI summary
Explainable Object-induced Action Decision for Autonomous Vehicles
Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, Nuno Vasconcelos
TL;DR
Autonomous-driving systems need decisions that remain connected to the objects causing hazards without requiring full scene understanding or relying on opaque end-to-end mappings. The paper introduces a dataset, joint action-explanation formulation, and CNN combining object and global scene reasoning; experiments show that explanation supervision improves action prediction.
Problem
The paper addresses the need for autonomous-driving decisions that focus on hazard-relevant objects while improving interpretability beyond existing end-to-end and pipelined approaches.
Method
The paper introduces BDD-OIA, jointly predicts actions and finite explanations, and uses a multi-task CNN combining object detection with global scene context.
Results
Explanation supervision improves action prediction, with action F1all increasing from 0.675 to 0.734 when explanations receive equal training weight.
Takeaways & Limitations
The results support combining object-centric reasoning, global scene context, and explanations for autonomous-driving action prediction.
Abstract
from arXiv · showhide
A new paradigm is proposed for autonomous driving. The new paradigm lies between the end-to-end and pipelined approaches, and is inspired by how humans solve the problem. While it relies on scene understanding, the latter only considers objects that could originate hazard. These are denoted as action-inducing, since changes in their state should trigger vehicle actions. They also define a set of explanations for these actions, which should be produced jointly with the latter. An extension of the BDD100K dataset, annotated for a set of 4 actions and 21 explanations, is proposed. A new multi-task formulation of the problem, which optimizes the accuracy of both action commands and explanations, is then introduced. A CNN architecture is finally proposed to solve this problem, by combining reasoning about action inducing objects and global scene context. Experimental results show that the requirement of explanations improves the recognition of action-inducing objects, which in turn leads to better action predictions.
1. Introduction
The paper proposes an autonomous-driving paradigm between end-to-end and pipelined systems, focusing on hazard-relevant action-inducing objects and jointly predicting their associated actions and explanations. It introduces BDD-OIA, a multi-task formulation, and a CNN combining object-centric and global scene reasoning.
- Motivation: The proposed approach focuses on action-inducing objects whose state changes can trigger vehicle actions, rather than requiring full scene understanding.This intermediate strategy is motivated by how humans attend to objects that could create hazards.
- Motivation: Action-inducing objects provide a finite set of semantic explanations that can be predicted jointly with driving actions.The formulation treats joint action and explanation prediction as a multi-task problem.
- Dataset and formulation: The proposed multi-task CNN combines Faster R-CNN object detection with global scene context to identify action-inducing objects and predict action-explanation pairs.The architecture uses detailed explanation supervision alongside object and scene reasoning.
- Results: Explanation generation improves action prediction, while the combined object-centric and global reasoning benefits both actions and explanations.The paper reports state-of-the-art performance on BDD-OIA for predicting driving commands and explanations.
2. Related work
Related work spans end-to-end driving, pipelined perception, contextual reasoning, attention, textual explanations, and autonomous-driving datasets. The paper positions its architecture as combining object features, global context, and explanation supervision.
- End-to-end and pipelined driving: End-to-end driving systems map visual input directly to vehicle commands, while pipelined systems detect objects and obstacles before deciding actions.The paper notes that its approach uses object detection and contextual reasoning while remaining trained on real images.
- Global-local contextual representations: Prior work either selects critical objects without global features or uses whole-scene features while ignoring objects; the proposed architecture accounts for both.Explanation supervision is used to separate action-inducing objects from objects unimportant for driving.
- Attention mechanisms: Attention research includes neural attention mechanisms, visualizing network behavior, pixel-level contributions, and studies of human gaze in driving scenes.These approaches provide context for analyzing which scene regions contribute to predictions.
- Explanations: Prior explanation methods generate text for attention maps or predictions, including an autonomous-driving model combining textual generation with attention.The paper instead uses a finite explanation set formulated as classification.
- Datasets: Autonomous-driving datasets provide varied annotations, with BDD100K containing videos labeled for image-level attributes, objects, drivable areas, lanes, and instance segmentation.The paper extends BDD100K for object-induced action and explanation prediction.
3. Joint Action and Explanation Prediction
The paper formulates object-induced driving as joint multi-label prediction of four actions and 21 object-based explanations. BDD-OIA targets complex scenes with multiple possible actions by selecting dense, challenging clips from BDD100K.
- Definitions: Object-induced actions require reasoning about other road objects or recognized events, including abstract objects such as open lanes.
- Definitions: The task predicts four action labels and 21 explanation labels jointly as two multi-label classification problems.
- BDD-OIA Dataset: BDD-OIA contains complex scenes selected from BDD100K with at least 5 pedestrians or bicycle riders and more than 5 vehicles.
- BDD-OIA Dataset: BDD-OIA includes 22,924 five-second clips annotated for four actions and 21 explanations, with multiple action choices frequently possible.
- BDD-OIA Dataset: The dataset merges left/right lane changes with corresponding turns to avoid distribution imbalance, while explanations remain substantially more unbalanced than actions.
4. Deep Learning Architecture
The proposed CNN combines global scene context with local action-inducing object features, selecting relevant objects before jointly predicting actions and explanations. Multi-task supervision trains both outputs through a weighted loss and helps suppress irrelevant scene clutter.
- Architecture overview: The architecture uses Faster R-CNN backbone features in global and local branches to model scene context, object details, and scene-object relationships.
- Architecture overview: A selector scores object-scene feature tensors, retains the top k action-inducing objects, and concatenates their features with global features for prediction.
- Architecture overview: The network jointly predicts action and explanation vectors using L = L_A + λL_E, where λ controls the relative importance of their errors.
- Architecture overview: Explanation generation is formulated as classification, enabling explicit optimization with less data than recurrent or generative natural-language systems.
- Architecture overview: The learned selector discards low-scoring proposals, reducing clutter from objects that are unimportant to action predictions.
5. Experiments
Experiments show that jointly predicting explanations improves action recognition, while combining local object features with global scene context yields the strongest overall performance. Multiple action labels also address imbalance and better represent scenes where several actions are possible.
- 5.2. Interplay between Actions and Explanations: 0.675 to 0.734: weighting explanations equally with actions increased action F1all, while action-only training performed weakest.The λ = 1 model also achieved the best performance across all action classes.
- 5.2. Interplay between Actions and Explanations: Explanation supervision provides causal information about object-induced actions and produces larger gains for underrepresented left and right actions.The reported explanation-based regularization is especially beneficial when fewer training examples increase overfitting risk.
- 5.3. Interplay between Local and Global Features: Combining local and global features achieved the overall best results for both action and explanation prediction.Using ten selected objects outperformed using five, while global-only features produced explanations nearly equivalent to the full network.
- 5.3. Interplay between Local and Global Features: Global-only and local-only comparison models underperformed the proposed combination, supporting the need to reason about object-scene interactions.The purely global baseline had the worst performance, while the purely local selector performed similarly to random object selection.
- 5.5. Single vs. Multiple Action Predictions: Multiple action prediction mitigates class imbalance and creates more examples involving left and right turns than single driver-action labels.A single forward label does not establish that forward was the only feasible action, whereas multiple labels represent alternative possible actions.
- 5.6. Qualitative Results: Qualitative results show that the network selects small traffic signs, lights, and roadside obstacles while ignoring many non-action-inducing objects.One reported error involved incorrectly predicting a left turn in the middle of a crossroad; the compared selector missed many action-inducing objects.
6. Conclusion
The paper introduces object-induced action and explanation prediction for autonomous driving through a new dataset and architecture. Its experiments show that object-grounded explanations improve action prediction while keeping explanation generation a classification problem.
- 6. Conclusion: The proposed task detects hazardous action-inducing objects and predicts explanations for the driving actions they induce.Because explanations are grounded in objects, they reduce to one-hot-style prediction rather than generic natural-language generation.
- 6. Conclusion: The dataset and architecture require reasoning about both objects and scene context rather than object detection alone.The paper presents object reasoning that accounts for scene context as the central computer-vision challenge.
- 6. Conclusion: Adding explanations increased action-prediction accuracy in the reported experiments.The paper characterizes this as evidence that explanations can improve the performance of a deep-learning system.