Source-linked AI summary
DRAMA: Joint Risk Localization and Captioning in Driving
Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, Jiachen Li
TL;DR
The paper addresses the lack of standard benchmarks for explainable driving-risk perception by jointly localizing risky important objects and captioning their risks. It introduces DRAMA with video- and object-level annotations, benchmarks joint architectures, and reports improved important-object localization when captioning is used as a prior. The dataset and task support evaluation of visual reasoning capabilities in driving scenarios, but safety-critical deployment remains bounded by the risk of injury, death, or property loss if such models fail.
Problem
Existing methods identify important objects, but none characterizes natural-language captions explaining those objects’ risks, despite the importance of explainability for safety-critical driving.
Method
The paper introduces DRAMA, a dataset with video- and object-level closed and open-ended questions, and benchmarks jointly supervised risk-localization and captioning models.
Results
Important object localization using captioning as a prior showed improved performance against other baseline models in extensive architectural ablations.
Takeaways & Limitations
DRAMA provides a basis for evaluating visual explanations of driving risks and studying joint localization of important objects with natural-language reasoning.
Takeaways & Limitations
Applications of reasoning models trained with DRAMA in safety-critical settings may cause injury, death, or property loss if they fail to make appropriate decisions.
Abstract
from arXiv · showhide
Considering the functionality of situational awareness in safety-critical automation systems, the perception of risk in driving scenes and its explainability is of particular importance for autonomous and cooperative driving. Toward this goal, this paper proposes a new research direction of joint risk localization in driving scenes and its risk explanation as a natural language description. Due to the lack of standard benchmarks, we collected a large-scale dataset, DRAMA (Driving Risk Assessment Mechanism with A captioning module), which consists of 17,785 interactive driving scenarios collected in Tokyo, Japan. Our DRAMA dataset accommodates video- and object-level questions on driving risks with associated important objects to achieve the goal of visual captioning as a free-form language description utilizing closed and open-ended responses for multi-level questions, which can be used to evaluate a range of visual captioning capabilities in driving scenarios. We make this data available to the community for further research. Using DRAMA, we explore multiple facets of joint risk localization and captioning in interactive driving scenarios. In particular, we benchmark various multi-task prediction architectures and provide a detailed analysis of joint risk localization and risk captioning. The data set is available at https://usa.honda-ri.com/drama
1. Introduction
The paper targets explainable risk perception for safety-critical driving by jointly identifying important objects and generating natural-language descriptions of their risks. It introduces DRAMA, a dataset and benchmark designed for this joint task.
- Motivation: Situational awareness in intelligent vehicles requires perceiving driving risks and communicating them through explainable linguistic descriptions.The descriptions are intended to identify risk agents and communicate perceived risks through an AI agent.
- Research gap: Existing important-object methods support safety-critical tasks, but do not characterize captions explaining identified objects in natural language.Prior approaches use self-supervised related tasks or human-annotated object importance, while caption-based explanation remains unaddressed.
- Dataset: DRAMA provides video- and object-level question-answer annotations for free-form captions explaining important objects’ interactions and risks from the ego-driver’s perspective.The dataset uses closed and open-ended responses across sequence and motion observations.
- Dataset: Each DRAMA video depicts an ego-driver response to perceived risk that activates braking, with questions organized around what, which, where, why, and how.The design focuses on identifying important objects and captioning why they may influence future ego-vehicle behavior.
- Contributions: The paper introduces a dataset for evaluating visual captioning, an annotation strategy for video- and object-level attributes, and benchmarks joint risk localization with natural-language captioning.These contributions address both risk localization and explanations of the associated important objects.
2. Related Work
Prior driving-language and visual-grounding datasets address commands, actions, gaze, or generic region descriptions, but DRAMA focuses on explaining risks associated with important objects. Its annotations add driving-specific vocabulary and describe why scenarios are risky and how objects interact.
- Important object identification: Important-object identification methods use explicit supervision, gaze-based attention, or implicit learning through related driving tasks.The related-work categories include binary importance classification, pixel-level attention proxies, and task-driven learning.
- Visual-language methods: Visual grounding and dense captioning methods associate natural-language descriptions with image regions, while other work uses attention for captioning or driving actions.These approaches provide context for DRAMA’s connection to region-level language grounding.
- Driving-language datasets: Talk2Car, BDD-X, BDD-OIA, and HAD provide driving-language annotations, but their goals include path guidance, ego-action explanations, or gaze-based salient maps.The cited datasets differ in task focus from DRAMA’s risk-explanation setting.
- DRAMA’s distinction: DRAMA applies question answering to visual explanations of driving risks associated with important objects, extending beyond generic VQA-style annotation.Its annotations contain domain-specific terms such as lane change, cut-in, traffic congestion, and parked vehicle.
- DRAMA’s distinction: DRAMA is designed to describe why a scenario is risky and what interactions occur as free-form language.This focus supports in-depth analysis of driving scenes using domain-specific vocabulary.
3. DRAMA Dataset
DRAMA is a Tokyo driving dataset designed to support visual reasoning about risks associated with important objects through video- and object-level question answering. Its annotations combine closed-form scene questions with open-ended object attributes and risk explanations, enabling joint risk localization and captioning.
- Dataset design: DRAMA provides video- and object-level questions and answers about driving risks associated with important objects.The questions use elementary operations such as what, which, where, why, and how to represent object types, attributes, locations, and relationships to the ego-vehicle.
- Annotation schema: Video-level questions use closed-form answers, while object-level questions allow free-form responses for more complex visual reasoning.Video-level answers include yes/no or single-choice actions such as stopping or slowing down; object-level questions describe attributes and risk-related interactions.
- Dataset statistics: 17,066 scenarios, or 95.95%, are labeled risky after filtering videos by ego-vehicle braking activation.Most interactions involve an ego-vehicle traveling straight, and scenarios span wider roads, intersections, and narrow streets.
- Dataset statistics: Important objects are vehicles in 71.9% of cases, pedestrians or cyclists in 19.6%, and infrastructure in 8.5%.Object-level annotations also include 35,038 visual attributes from 17,066 objects and free-form motion, location, direction, and behavioral reasoning descriptions.
- Dataset distinction: DRAMA jointly addresses risk localization and visual reasoning with free-form language descriptions from the ego-car perspective.Unlike related datasets, its captions describe the risk of important objects while considering spatio-temporal relationships in videos.
4. Methodology
The proposed model jointly predicts an important-object bounding box and a corresponding risk caption from visual and optical-flow features. It combines localization and captioning objectives with uncertainty-based multi-task weighting and attention regularization.
- Architecture: The model uses an encoder-decoder architecture to address risk localization and risk reasoning.The decoder predicts both an important-object bounding box and a corresponding caption description.
- Encoder: Visual and optical-flow features are extracted from consecutive frames using a pretrained ResNet-101 backbone.Optical flow is generated with FlowNet 2.0 to capture object and scene motion before feature extraction.
- Decoder: Caption generation uses an LSTM-based recurrent model whose hidden states provide embeddings for self-attention.The resulting attended feature is used by an important-object localization model, implemented with a basic multilayer perceptron.
- Loss function: The joint objective combines localization loss Lb and captioning loss Lc with a doubly stochastic attention regularizer.The regularizer is adapted from SAT and uses λ as a hyperparameter over the caption attention map.
- Loss function: Homoscedastic uncertainty weighting learns relative weights σ1 and σ2 for the localization and captioning losses.The multi-task loss uses these learned weights to combine Lb and Lc.
5. Experiments
The experiments compare joint and single-task models for risk localization and captioning on DRAMA, testing optical flow, decoder design, and dual-task training. LCP generally provides stronger joint performance, while qualitative examples show risk reasoning and object descriptions.
- Model Comparison: The study compares LCP with ICL, CLP, SAT, and ResNet-101 using shared backbone and task-specific architectures.LCP uses captioning as a prior for localization; ICL predicts localization and captioning independently, while CLP uses a localization prior for caption generation.
- Ablation Analysis: Optical flow improves captioning across LCP, ICL, and CLP by capturing motion-related words unavailable from a single image.
- Quantitative Results: LCP performs well in localization, while its captioning is higher or similar to CLP and better than ICL and SAT.The paper attributes LCP’s localization performance to an additional self-attention module and links better localization with improved multi-task weighting.
- Ablation Analysis: Dual-task loss improves LCP’s overall performance, whereas ROI pooling improves CLP captioning but appears to reduce its risk localization.
- Object-Type Analysis: LCP outperforms SAT in captioning for all object types and outperforms ResNet-101 in localization for most object types.The comparison uses CIDEr and Mean-IOU metrics and supports jointly tackling captioning and important-object identification.
- Qualitative Results: Qualitative examples show correct reasoning about pedestrians, vehicles, traffic congestion, traffic lights, and infrastructure, sometimes using explanations differing from ground truth.
6. Conclusion
The paper introduces joint risk localization and natural-language explanation as a research direction and addresses the lack of suitable benchmarks with the DRAMA dataset. DRAMA supports evaluation of visual explanation capabilities and benchmarking of jointly supervised models.
- 6. Conclusion: The paper introduces joint localization of driving risks and their explanations as natural-language descriptions.
- 6. Conclusion: DRAMA contains video- and object-level questions about important objects and interpretations of ego-driver interactions.
- 6. Conclusion: DRAMA facilitates evaluation of visual explanation capabilities across driving scenarios.
- 6. Conclusion: Ablation studies show improved important-object localization when captioning is used as a prior compared with other baseline models.
7. Additional Analysis of DRAMA
The additional analysis describes DRAMA’s interactive scenarios, question-answer structure, object distributions, attributes, bounding boxes, vocabulary, and annotation procedure. The dataset emphasizes risk-related visual and motion reasoning from the ego-driver’s perspective.
- Scenario and Annotation Analysis: Each scenario includes video- and agent-level attributes describing sequence and motion observations across interactive driving environments.Examples include turning, stopping, lane changing, cyclist interactions, and pedestrian behaviors.
- Dataset Statistics: DRAMA contains 17,785 scenarios, and video-level question-answer attributes are omitted from the statistics table because every scenario has them.
- Attribute Statistics: Vehicles average 2 visual attributes per bounding box, pedestrians and cyclists 2.19, and infrastructure 2.17; vehicle motion attributes average 1.12.
- Bounding-Box Statistics: Vehicles account for 12,273 bounding boxes, or 71.91% of all boxes, with normalized mean box size of 8% of the image.
- Bounding-Box Statistics: Pedestrians and cyclists account for 19.19% of bounding boxes and have normalized mean box size of 4% of the image.
- Vocabulary Statistics: Captions use 992 unique words with 306,708 total occurrences, including 112 words shared across all object classes.
- Question Design: The dataset uses questions based on what, which, where, why, and how to elicit closed- and open-ended descriptions of risk and important agents.
8. Implementation Details
The implementation maps visual and optical-flow inputs through ResNet-101 encoders and uses decoder components for captioning and localization. Training and evaluation use defined input, optimization, checkpoint, and metric procedures.
- Localization Implementation: Bounding boxes are represented by normalized center coordinates and dimensions, then mapped through sigmoid outputs to original pixels for Mean-IOU and accuracy evaluation.
- Network Architecture: The visual and flow encoders use ResNet-101 features of size (2048,14,14), while the decoder concatenates both streams into (4096,14,14).
- Training Monitoring: Figure 9 tracks CIDEr and Mean-IOU on the validation set during training.
- Decoder Comparison: Figure 10 compares the LCP decoder with the CLP and ICL decoder baselines.
- Training Procedure: The model is trained for 120 epochs with Adam and reports the checkpoint achieving the best validation CIDEr score on the test set.
9. Additional Evaluation
Additional evaluation examines robustness, caption-length effects, multi-task training behavior, and qualitative successes and failures in risk captioning and localization.
- Robustness: The best model provides robust prediction capabilities, with a small standard deviation across three test-set runs.The standard deviation was computed from three runs of the best model.
- Caption-Length Analysis: Captioning performance generally degrades as generated captions become longer or more complex.CIDEr is plotted against caption length on the test set; increases at some lengths reflect more common caption lengths.
- Multi-Task Training: LCP (Ours) continuously improves both captioning and important-object identification, whereas LCP without DTL improves object identification while sacrificing captioning.The compared metrics are CIDEr for captioning and Mean-IOU for important-object identification.
- Successful Cases: Successful cases include correct pedestrian attributes, cyclist importance reasoning based on the absence of a bicycle lane, and traffic-light prediction despite occlusion by a vehicle.The examples cover pedestrians on a narrow road or at an intersection, a cyclist, and a partially blocked traffic light.
- Captioning Failures: Captioning can fail despite correct localization when reasoning is incomplete, the scene is difficult to interpret, or motion direction is predicted incorrectly.Examples include trucks, red traffic lights, traffic congestion, and cyclists.
- Localization Failures: Localization failures arise when multiple objects appear important, including cases where small, distant agents are harder to represent.The model may select an object closer to the ego-vehicle instead of the far annotated object.
10. Limitations
The paper cautions that failures in safety-critical applications may have severe consequences and that both models and human-labeled data have limitations.
- Safety-Critical Scope: Failures in deployed reasoning models could cause injury, death, or property loss in safety-critical driving applications.The paper advises users to consider failures beyond those explored in the reported cases.
- Annotation Limitations: The dataset’s human annotations may contain biases caused by errors in interpretation.Users are requested to assess risk when applying the dataset to safety-critical applications.