Source-linked AI summary
Predicting Driver Attention in Critical Situations
Ye Xia, Danqing Zhang, Jinkyu Kim, Ken Nakayama, Karl Zipser, David Whitney
TL;DR
Driver-attention prediction for rare critical situations lacks sufficient in-car training data and must account for costly errors at critical moments. The paper introduces an in-lab collection protocol, the BDD-A dataset, Human Weighted Sampling, and a prediction model. The model outperforms prior models, produces human-like attention maps, and also predicts routine in-car attention data with state-of-the-art accuracy.
Problem
Rare critical driving situations are difficult to represent with conventional in-car gaze collection, while prediction errors at critical moments can have unequal costs.
Method
The paper collects multi-focus attention maps from observers viewing selected braking-event videos, builds BDD-A, and uses Human Weighted Sampling to emphasize crucial frames during training.
Results
The model outperforms state-of-the-art models, shows sophisticated pedestrian-focused behavior, and predicts in-car routine-driving attention at state-of-the-art accuracy.
Takeaways & Limitations
The dataset, collection protocol, and sampling method support driver-attention prediction across critical in-lab events and routine in-car driving data.
Takeaways & Limitations
In-lab attention may differ from in-car attention because observers may attend to driving-irrelevant objects and third-person attention differs from first-person attention.
Abstract
from arXiv · showhide
Robust driver attention prediction for critical situations is a challenging computer vision problem, yet essential for autonomous driving. Because critical driving moments are so rare, collecting enough data for these situations is difficult with the conventional in-car data collection protocol---tracking eye movements during driving. Here, we first propose a new in-lab driver attention collection protocol and introduce a new driver attention dataset, Berkeley DeepDrive Attention (BDD-A) dataset, which is built upon braking event videos selected from a large-scale, crowd-sourced driving video dataset. We further propose Human Weighted Sampling (HWS) method, which uses human gaze behavior to identify crucial frames of a driving dataset and weights them heavily during model training. With our dataset and HWS, we built a driver attention prediction model that outperforms the state-of-the-art and demonstrates sophisticated behaviors, like attending to crossing pedestrians but not giving false alarms to pedestrians safely walking on the sidewalk. Its prediction results are nearly indistinguishable from ground-truth to humans. Although only being trained with our in-lab attention data, the model also predicts in-car driver attention data of routine driving with state-of-the-art accuracy. This result not only demonstrates the performance of our model but also proves the validity and usefulness of our dataset and data collection protocol.
1 Introduction
Driver attention prediction in rare critical situations is difficult because conventional in-car gaze collection mostly captures routine driving and misses covert or multi-object attention. The paper introduces an in-lab collection protocol, the BDD-A dataset, Human Weighted Sampling, and a model that predicts attention to critical road events.
- Existing driver-attention datasets and models focus mostly on routine lane-following and car-following, leaving rare critical situations underrepresented.
- Conventional in-car collection cannot efficiently capture critical-event data because most driving is routine, eye trackers record one gaze location, and irrelevant gazes create false positives.
- The in-lab protocol averages repeated gazes from multiple observers viewing selected crowd-sourced driving videos to produce accurate multi-focus attention maps.
- Human Weighted Sampling identifies crucial frames from human eye movements and weights them during training to address unequal costs across prediction errors.
- The resulting model attends to suddenly crossing pedestrians without false alarms for pedestrians safely walking in the same direction, and its predictions are nearly indistinguishable from ground truth to human judges.
2 Related works
Prior driver-attention datasets and models are limited by sparse activities, restricted interactions, and collection settings that omit covert attention or diverse urban cues. The paper positions its dataset as addressing these limitations.
- DR(eye)VE provides six hours from 74 rides but has limited diversity and in-car collection drawbacks, including missing covert attention and false-positive gaze.
- Other datasets use coarse gaze regions or small collections of static driving images, limiting spatial detail or dataset scale.
- Models trained on DR(eye)VE tend to become vanishing-point detectors because the dataset contains sparse driving activities and limited interactions with other road users.
- BDD-A is presented as a contribution intended to capture more diverse human attention behaviors in urban driving environments.
3 Berkeley DeepDrive Attention (BDD-A) Dataset
BDD-A uses braking-event videos and an in-lab instructor task to collect multi-focus attention maps for critical driving situations. Human evaluations found these maps highlight more driving-relevant objects than in-car maps, while the HWS model’s predictions were preferred over in-car maps.
- BDD-A selects braking-event clips from the crowd-sourced BDD100k driving-video dataset to target critical situations.
- 45 experienced drivers performed a copilot-seat instructor task while watching selected videos, pressing a key when they perceived a danger.
- Aggregating independent observers’ gazes produces multi-focus maps and reduces irrelevant gaze noise from objects such as buildings and trees.
- In-car maps highlighted significantly fewer driving-relevant objects than in-lab maps, based on detected cars, pedestrians, and cyclists.
- The human evaluation supports the in-lab maps as representing where a good driver should look from a third-person perspective, without establishing general superiority over in-car maps.
- HWS predictions were chosen as better than in-lab human maps in 41% of trials and received significantly higher preference than in-car maps.
4 Attention Prediction Model
The model combines pretrained visual features, convolutional processing, and temporal modeling to predict driver attention maps. Human Weighted Sampling emphasizes frames whose attention differs from a clip’s typical pattern, increasing representation of uncommon critical moments during training.
- Network Configuration: The model predicts attention maps from current and previous frames using visual feature extraction, visual feature processing, and temporal processing modules.
- Network Configuration: A fixed ImageNet-pretrained AlexNet extracts visual features before subsequent convolutional processing.
- Network Configuration: Three fully convolutional layers with dropout process the extracted features; without dropout, training tended toward a central-bias attention map.
- Network Configuration: A convolutional LSTM processes successive frames and outputs a probability distribution over 64 × 36 spatial regions, trained with cross-entropy against ground truth.
- Human Weighted Sampling: Human Weighted Sampling addresses the prevalence of routine lane-following and car-following frames, whose errors may matter less than errors during rare critical events.
- Human Weighted Sampling: KL divergence from each frame’s clip-average attention map determines its sampling weight, increasing exposure to uncommon attention patterns.
- Human Weighted Sampling: The sampling function boosts high-divergence frames and flattens the divergence distribution over D_KL ∈ [1,3].
5 Results and Discussion
The proposed models outperform competing attention predictors across standard metrics and non-trivial frames, while HWS improves performance further on crucial moments. They also better identify driving-relevant pedestrians and cyclists, produce human-like attention maps, and generalize to in-car attention data.
- Quantitative Analysis of Attention Prediction: Both proposed models significantly outperform the DR(eye)VE, SALICON, and baseline models across all metrics on full and non-trivial testing frames.The evaluation uses DKL, CC, NSS, and AUC, with non-trivial frames defined by KL divergence greater than 2.0.
- Quantitative Analysis of Attention Prediction: HWS improves over regular sampling on the full testing set and yields a larger advantage on non-trivial frames.The authors interpret this as better handling of dataset bias and better use of information in crucial driving moments.
- Attended Object Analysis: Both proposed models outperform competing models at identifying pedestrians and cyclists, with HWS matching human attention best.Cars are easy for all models, whereas competing models miss a large proportion of pedestrians and cyclists relative to human attention.
- Attended Object Analysis: 48% of pedestrians attended by HWS were important pedestrians, versus 33% among all detected pedestrians.The HWS model therefore distinguishes pedestrians attended by humans from other detected pedestrians; its 95% bootstrapped confidence interval is [42%, 55%].
- Human Evaluation: In 41% of human-evaluation trials, participants preferred HWS predictions to in-lab human attention maps.This exceeded the 29% chosen rate previously reported for in-car DR(eye)VE attention maps, with permutation test p = 4 × 10^-5.
- Cross-Dataset Evaluation: The model trained on in-lab attention data also matches state-of-the-art performance on an existing in-car attention dataset.The paper reports this alongside sophisticated behavior in critical scenarios and predictions nearly indistinguishable from ground truth to human judges.
6 Conclusions
The paper introduces an in-lab attention protocol, the BDD-A dataset, HWS, and a prediction model that performs strongly on both in-lab and in-car attention data. The model captures safety-relevant distinctions and produces predictions nearly indistinguishable from ground truth to human observers.
- Contributions: The paper introduces an in-lab driver attention protocol, a public BDD-A dataset, and HWS to address data-collection drawbacks and driving-dataset bias.The protocol supports multi-focus attention maps from repeated laboratory gaze measurements, while HWS emphasizes crucial frames during training.
- Contributions: The resulting model predicts both in-lab and in-car driver attention data and shows sophisticated behavior in safety-relevant situations.Examples include attending to suddenly crossing pedestrians without being distracted by pedestrians safely walking in the same direction.
- Contributions: Human observers judge the model’s prediction results as nearly indistinguishable from ground truth.