Source-linked AI summary
Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning
Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, Kate Saenko
TL;DR
Intelligent transportation systems need to understand how humans drive and interact with complex traffic scenes, beyond low-level visual recognition. The paper introduces HDD, a multimodal real-world driving dataset with a four-layer behavior annotation scheme, and evaluates baseline behavior detection. The results show that RGB-only detection is challenging, sensor data improve accuracy, and stronger representations, temporal modeling, and training strategies remain needed.
Problem
Existing driving datasets advance core perception tasks, but higher-level scene understanding of human driver behavior and interactions remains insufficient.
Method
The paper collects HDD, comprising 104 hours of real human driving, and represents behavior with Goal-oriented action, Stimulus-driven action, Cause, and Attention annotations.
Results
Baseline experiments show RGB-frame methods face a challenging detection task, while adding sensor data improves accuracy.
Takeaways & Limitations
HDD supports research on untrimmed driver behavior detection, causal interaction analysis, auxiliary behavior learning, and multimodal fusion.
Takeaways & Limitations
The work does not cover predicting traffic participants’ intentions or reasoning about overall traffic situations for motion planning and decision making.
Abstract
from arXiv · showhide
Driving Scene understanding is a key ingredient for intelligent transportation systems. To achieve systems that can operate in a complex physical and social environment, they need to understand and learn how humans drive and interact with traffic scenes. We present the Honda Research Institute Driving Dataset (HDD), a challenging dataset to enable research on learning driver behavior in real-life environments. The dataset includes 104 hours of real human driving in the San Francisco Bay Area collected using an instrumented vehicle equipped with different sensors. We provide a detailed analysis of HDD with a comparison to other driving datasets. A novel annotation methodology is introduced to enable research on driver behavior understanding from untrimmed data sequences. As the first step, baseline algorithms for driver behavior detection are trained and tested to demonstrate the feasibility of the proposed task.
1. Introduction
Existing driving datasets advance low-level visual recognition, but higher-level understanding requires modeling how human drivers act and interact with traffic scenes. HDD addresses this gap with multimodal real-world driving data, layered behavior annotations, and baseline detection experiments.
- Higher-level driving scene understanding must cover driver actions, traffic interactions, risk assessment, prediction, and causal reasoning beyond detection and tracking.
- The annotation scheme decomposes driver behavior into Goal-oriented action, Stimulus-driven action, Cause, and Attention.Bounding boxes identify traffic participants that cause stops or are attended by the driver, and multiple simultaneous scenarios can be represented.
- HDD contains 104 hours of real human driving in the San Francisco Bay Area collected with an instrumented vehicle.The data comprise 137 sessions, each representing a navigation task performed by a driver.
- The dataset supports research on imbalanced behavior detection in untrimmed video, causal interaction analysis, auxiliary behavior learning for future-action prediction, and multimodal fusion.
- The work focuses on a driver-centric view and does not address predicting traffic participants’ intentions or reasoning about overall traffic situations for motion planning and decision making.The paper identifies scalable dataset construction as a next milestone.
2. Related work
Prior driving datasets support perception, localization, mapping, and maneuver learning, whereas HDD emphasizes richer driver behavior and interactions with traffic participants. It complements existing datasets through layered annotations and multimodal vehicle signals.
- Existing driving datasets have accelerated visual scene recognition through benchmarks for object detection, scene flow, odometry, semantic segmentation, and long-term localization.
- HDD is complementary to KITTI and Oxford RobotCar because it focuses on learning driver behavior under varied traffic situations.
- Compared with datasets that mainly model turning, going straight, and lane changes, HDD adds annotations for common driver behaviors and captures CAN signals for interactions with traffic participants.
- HDD provides explicit driver-behavior definitions rather than relying on heuristics for four CAN-derived actions, supporting auxiliary behavior classification in multitask driving models.The paper notes that behavior classification can improve model explainability and permit use of human common-sense priors.
- Related datasets cover egocentric human activities, pedestrian intentions, joint attention, and visual reasoning, while HDD additionally annotates how traffic participants interact with drivers.
3. Honda Research Institute Driving Dataset
HDD is a multimodal real-world driving dataset designed to represent driver behavior, causal reasoning, and interactions with traffic participants. It contains 104 hours of Bay Area driving and uses a four-layer annotation scheme with quality-controlled labels.
- Annotation Methodology: The four-layer representation comprises Goal-oriented action, Stimulus-driven action, Cause, and Attention to capture driving behavior and causal reasoning.Goal-oriented actions describe navigation maneuvers, while Stimulus-driven actions, Causes, and Attention connect behavior to surrounding traffic participants and obstacles.
- Annotation Methodology: Annotations were produced with ELAN through independent labeling, a merging annotator, and internal expert review.This quality-control process combines external annotators’ work with expert judgment before finalizing each session.
- Annotation Quality: 98% agreement was obtained for driver behavior labels between the third external annotator and internal expert annotator on 10 sessions, while action-boundary consistency remains unresolved.The paper identifies start and end times as subjective and calls for systematic evaluation of localization consistency.
- Dataset Statistics: The released annotations cover 104 video hours and 137 navigation sessions, with a highly imbalanced distribution of behavior labels.The average session lasts 45 minutes.
4. Multimodal Fusion for Driver Behavior Detection
The driver behavior detector models temporal context with LSTMs while combining video features with vehicle-dynamics signals. Its streaming formulation uses only current input and previous observations.
- Temporal Modeling: The model uses an LSTM backbone to encode the history needed for predicting driver behaviors over time.The initial experiments focus on detecting the Goal-oriented and Cause annotation layers.
- Multimodal Fusion: The detector fuses video with CAN measurements of speed, pedal positions, yaw rate, steering angle, and steering-wheel rotation speed.The auxiliary vehicle-dynamics stream distinguishes the setup from single-modality activity detection.
- Streaming Constraint: The application requires real-time streaming predictions based solely on the current input and previous observations.The model is constrained to avoid relying on future observations.
5. Experiments
Experiments evaluate multimodal temporal representations for driver behavior detection using geographically separated routes and per-frame average precision. The setup compares visual, sensor, and fused baselines under imbalanced labels.
- Experimental Setup: Training and testing routes are split by geolocation to minimize spatial overlap and avoid testing on locations used for training.Figure 3 visualizes training traces in blue and testing traces in red.
- Experimental Setup: Baseline inputs sample video frames and CAN values at 3 Hz as a trade-off between modeling complexity and precision.Visual features come from the Conv2d 7b 1x1 layer of ImageNet-pretrained InceptionResnet-V2.
- Visual Representation: Preserving the CNN feature spatial grid substantially improves turn detection relative to pooled visual features.The comparison is described through the ‘CNN conv’ result in Table 2.
- Training: Separate LSTMs are trained for the Goal and Cause layers with sequence segments of length 90 samples and batches of size 40.The training procedure uses truncated backpropagation through time and class-imbalance-aware loss design.
- Evaluation: Goal-oriented action detection is evaluated with per-frame average precision for each behavior class and mean average precision across classes.Random serves as a lower-bound baseline, while CNN and sensor-based models provide alternative representations.
6. Results and Discussion
The baseline experiments show that multimodal fusion improves driver-behavior detection, while class imbalance, brief motion patterns, and background confusion remain important error sources.
- Goal-oriented layer: ‘CNN+Sensors’ fuses visual and vehicle-dynamics inputs, producing the best results because the modalities provide complementary information.Visual inputs support scene interpretation, while sensor measurements capture vehicle dynamics such as braking and steering.
- Goal-oriented layer: ‘CNN conv’ substantially improves over ‘CNN pool’ by replacing spatial pooling with a learnable convolutional layer that preserves more information.The pooled representation loses information before sequential prediction.
- Goal-oriented layer: Sensor-only inputs help with simple turns, whereas visual inputs better support lane-change detection and scene interpretation.Steering-wheel position can often indicate left or right turns, but lane changes require visual information.
- Goal-oriented layer: Adding sensor inputs clearly disambiguates turns from ‘intersection passing’ in the confusion matrices, although intersections also affect detection of other classes through their visual cues.The matrices omit the background class from visualization and normalize rows.
- Error analysis: Background confusion is the most frequent error source across all annotation layers.The dataset currently includes four causes for stop actions and one cause for deviate actions, so cause detection directly implies the corresponding stimulus-driven action detection.
- Cause layer: Cause detection is better for signs, congestion, and traffic lights than for pedestrians and parked cars, whose motion patterns differ from ordinary deceleration.Pedestrians often induce stops during turns, while parked cars induce trajectory deviations.
- Cause layer: Underrepresented pedestrian and parked-car examples, short cause durations, and inadequate deviation modeling limit baseline performance.The authors suggest better imbalance handling, optical-flow features, and improved motion modeling as future directions.
7. Conclusion
The paper introduces HDD and a four-layer annotation methodology for studying driver behavior in untrimmed driving videos. Baselines indicate that the task is challenging, sensor data improves accuracy, and stronger representations, temporal modeling, and training strategies are still needed.
- Conclusion: HDD is introduced to stimulate algorithms that capture driver behavior in real-world driving.The dataset is designed for driver behavior understanding rather than only conventional visual recognition.
- Conclusion: The annotation methodology decomposes driver behavior into Goal-oriented, Stimulus-driven, Cause, and Attention layers.These layers represent complementary aspects of actions and interactions with traffic situations.
- Conclusion: Baseline experiments show that RGB-frame activity-recognition methods face a challenging driver-behavior detection task, while adding sensor data improves accuracy.The conclusion identifies sensor fusion as useful but insufficient for reasonable performance on its own.
- Conclusion: The authors identify better representations, temporal modeling, and training strategies as necessary before studying relationships between behavior layers.The stated next step is understanding relationships between drivers and traffic situations.