PaperlayerView exact revision on arXiv

Source-linked AI summary

DADA: Driver Attention Prediction in Driving Accident Scenarios

Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, Hongkai Yu

arXiv:1912.12148v2cs.CV

TL;DR

Driver attention prediction in accident scenarios is difficult because traffic scenes are dynamic and accident categories are intricate and imbalanced. SCAFNet combines RGB features with semantic context through attentive fusion and transfers fused details across frames with Conv-LSTM. The method reports superior performance to state-of-the-art approaches across DADA-2000, DR(eye)VE, and TrafficGaze.

  • Problem

    Driver attention prediction in accident scenarios is important for learning human visual focus, but diverse drivers and dynamic, varied traffic scenes make the problem difficult.

  • Method

    SCAFNet learns RGB-frame and semantic-image features in parallel, models semantic context with a GCN, fuses features attentively, and transfers fused details across frames with Conv-LSTM.

  • Results

    SCAFNet achieves superior performance to other state-of-the-art methods across DADA-2000, DR(eye)VE, and TrafficGaze; on TrafficGaze, it attains NSS 6.10.

  • Takeaways & Limitations

    Semantic context and temporal transfer provide the paper’s supported approach for predicting attention to objects or regions in driving accident situations.

Abstract

from arXiv · show

Driver attention prediction is becoming an essential research problem in human-like driving systems. This work makes an attempt to predict the driver attention in driving accident scenarios (DADA). However, challenges tread on the heels of that because of the dynamic traffic scene, intricate and imbalanced accident categories. In this work, we design a semantic context induced attentive fusion network (SCAFNet). We first segment the RGB video frames into the images with different semantic regions (i.e., semantic images), where each region denotes one kind of semantic categories of the scene (e.g., road, trees, etc.), and learn the spatio-temporal features of RGB frames and semantic images in two parallel paths simultaneously. Then, the learned features are fused by an attentive fusion network to find the semantic-induced scene variation in driver attention prediction. The contributions are three folds. 1) With the semantic images, we introduce their semantic context features and verified the manifest promotion effect for helping the driver attention prediction, where the semantic context features are modeled by a graph convolution network (GCN) on semantic images; 2) We fuse the semantic context features of semantic images and the features of RGB frames in an attentive strategy, and the fused details are transferred over frames by a convolutional LSTM module to obtain the attention map of each video frame with the consideration of historical scene variation in driving situations; 3) The superiority of the proposed method is evaluated on our previously collected dataset (named as DADA-2000) and two other challenging datasets with state-of-the-art methods. DADA-2000 is available at https://github.com/JWFangit/LOTVS-DADA.

I. INTRODUCTION

Driver attention prediction in accident scenarios is motivated by the safety importance of human visual focus and the difficulty of modeling attention across diverse drivers and traffic conditions. SCAFNet addresses this problem by combining RGB-frame features with semantic context and temporal scene variation, and is evaluated across three datasets.

  • Driver attention is studied for assisted and autonomous driving because absent attention is associated with road fatalities and distracted, drowsy, or drunk driving.
  • Physiological attention studies are limited by subjective differences related to driving habits, experience, age, gender, and culture.
  • DADA-2000 contains 2,000 video sequences and 658,476 frames with annotated driver attention and accident intervals across diverse weather, lighting, and driving occasions.
  • SCAFNet segments RGB frames into semantic images and learns RGB and semantic spatio-temporal features in parallel before attentive fusion.
  • A GCN models relationships among semantic scene parts, while a Conv-LSTM transfers fused details across frames to account for historical scene variation.
  • The method is evaluated against state-of-the-art approaches on DADA-2000, DR(eye)VE, and TrafficGaze.

II. RELATED WORK

Driver attention prediction extends visual attention modeling to driving videos, where temporal motion and driving-specific scene cues matter. DADA-2000 targets accident scenarios with annotated attention and temporal accident windows.

  • Visual attention prediction localizes human-fixated regions, while video models additionally capture motion and object correlations across frames.
  • Driver attention prediction focuses on attention maps in driving scenes, linking visual clues to drivers’ intentions and behavior.
  • Existing datasets include sunny, unobstructed scenes, single-driver fixation data, coarse annotations, or critical situations without the transition to actual accidents.
  • DADA-2000 contains 2,000 videos and 658,476 frames, covering diverse weather, lighting, and driving occasions for accident-scenario attention prediction.
  • Each sequence is divided into before-AW, AW, and after-AW intervals, with AW boundaries manually determined from crash-object visibility and scene normalization.

B. Attention Type

DADA-2000 represents attention in accident scenarios through observer fixations converted into frame-level attention maps. SCAFNet uses semantic and RGB features with temporal fusion to predict attention while considering prior scene variation.

  • Twenty experienced volunteers contributed eye-tracking data, with each frame viewed by at least five observers twice and fixations converted into Gaussian-filtered attention maps.
  • Ground-truth examples cover nine typical driving accident scenarios, illustrating frame-level attention patterns across accident types.
  • SCAFNet segments each RGB clip into semantic images and encodes RGB and semantic clips in parallel using multi-path 3D encoding.
  • The semantic-guided attentive fusion module combines RGB and semantic features, transfers fused details across frames, and predicts the hidden map for frame T + 1.
  • The hidden attention map incorporates historical scene variation from the preceding T frames before decoding into the final driver attention map.

A. The M3DE Architecture

The M3DE architecture extracts spatio-temporal vision and semantic features simultaneously from RGB frames and semantic images. It uses parallel 3D convolutional paths with matched structures.

  • M3DE introduces semantic information because semantic clues can inform learning of driving-scene behavior and maneuver-related features.
  • Semantic images are generated from RGB frames using DeeplabV3 pretrained on Cityscapes, then passed alongside RGB frames into separate M3DE paths.
  • Each M3DE path interleaves 3D convolution, batch normalization with ReLU, and max-pooling blocks to extract spatial-temporal features.
  • Each path contains ten 3D convolution blocks, ten batch-normalization+ReLU blocks, and three 3D max-pooling blocks.
  • The encoded RGB and semantic tensors Z_v and Z_s have shape R^T×32×24×512, representing T frames with 512 channels at 32×24 resolution.

B. SAF Module

The SAF module fuses RGB features with semantic context to model category relationships and scene variation. Frame-wise graph construction supplies semantic context, while Conv-LSTM transfers fused information across time.

  • SAF exploits complementary RGB and semantic features because semantic categories have distinct importance determined by their relationships in the scene.
  • The graph is built independently for each frame, giving every frame both semantic context and RGB spatio-temporal features during fusion.
  • Conv-LSTM memorizes and transfers potentially relevant object or region features across successive frames for predicting the next attention map.
  • Semantic context is modeled by building a graph over semantic-image features and applying graph convolution to capture relationships among semantic regions.
  • For each semantic feature map, spatial locations become N = H × W graph nodes, each represented with C channels.
  • Pairwise node similarity is transformed and normalized into an affinity matrix through learned transformations and column-wise softmax.

t. So far, the affinity matrix Gs

The frame-graph GCN models relations among semantic nodes across each frame, then averages the final node outputs to obtain semantic context features.

  • The frame-graph represents semantic nodes and their relationships for the tth frame.
  • The GCN uses three graph-convolution layers to explore semantic relations among neighboring nodes.
  • Each graph-convolution layer produces node features with learned weights, followed by ReLU before the next layer.
  • Averaging the final outputs over N nodes generates the semantic context vector z_s^t.
  • The resulting semantic context representation is commonly sparse, with nonzero elements encoding preferred semantic information.

2) Transition of Spatio-temporal Scene Features:

ConvLSTM transfers spatio-temporal scene features across successive frames while preserving spatial details through recurrent memory and gated updates.

  • Transition of Spatio-temporal Scene Features: ConvLSTM extends LSTM by preserving spatial details while learning temporal features.
  • Transition of Spatio-temporal Scene Features: The module uses memory and hidden states to control memory updates and sequential scene-feature output.
  • Transition of Spatio-temporal Scene Features: The input sequence up to time t and learned weights produce the next cell and hidden states.
  • Transition of Spatio-temporal Scene Features: Its recurrence uses input, forget, and output convolution gates together with cell-state and hidden-state updates.
  • Transition of Spatio-temporal Scene Features: The transition carries spatio-temporal features from t successive frames to frame t+1.

3) Attentive Fusion:

SCAFNet fuses RGB-frame features with semantic context features through an attentive, residual strategy, then uses ConvLSTM to retain temporal scene variation.

  • Attentive Fusion: RGB-frame features and semantic context features are obtained from the vision and semantic paths before fusion.
  • Attentive Fusion: The attentive fusion strategy combines both feature streams while using a residual connection to maintain original RGB information.
  • Attentive Fusion: ConvLSTM transfers the fused details across frames to model historical scene variation in driving situations.
  • Attentive Fusion: The resulting hidden state is used to obtain the driver attention map for the next frame.

C. DAMD module

The DAMD module decodes hidden attention features into final maps, while the evaluation compares SCAFNet across accident and non-accident driving datasets under stated training settings.

  • C. DAMD module: DAMD decodes the hidden driver attention state through upsampling, convolution, and batch-normalization layers to generate the final attention map.
  • C. DAMD module: The decoded attention map for frame T+1 incorporates scene variation from the previous T frames.
  • C. DAMD module: Figure 8 compares original images, ground-truth maps, and predictions from models with and without GCN under the same configuration.
  • C. DAMD module: The figure reports that adding GCN makes predicted attention more focused than without GCN.
  • C. DAMD module: Evaluation uses DADA, DR(eye)VE, and TrafficGaze to assess the proposed method against other methods.

B. Evaluation Protocols

The evaluation uses five saliency metrics and compares SCAFNet with component ablations and established attention-prediction methods on DADA.

  • Evaluation metrics: Five metrics evaluate prediction quality: KLdiv, NSS, SIM, CC, and AUC, including AUC-Judd and shuffled AUC variants.KLdiv rewards lower information loss, while NSS, SIM, CC, AUC-Judd, and shuffled AUC prefer higher values.
  • Component ablations: The ablation study compares the full two-branch SCAFNet with an RGB-only branch and a two-branch model without GCN.The full model learns spatial-temporal features from RGB frames and semantic images, fusing them through attentive fusion; the semantic branch uses GCN.
  • External baselines: Seven state-of-the-art methods provide external comparisons, spanning video-based approaches and static-image methods.The video methods include BDDA, DR(eye)VE, TwoStream, MLNet, and ACLNet; SALICON and SalGAN operate on static images.
  • Evaluation split: Overall DADA performance is reported on the dataset’s testing set, with Table II providing the comparison results.The evaluated overall testing set contains 71,107 frames.

D. Overall Evaluation on DADA

On DADA, SCAFNet is reported to outperform the compared methods overall and across accident behaviors and driving situations, while performance varies by behavior and model design.

  • Overall evaluation: SCAFNet significantly outperforms the compared methods on the overall DADA testing set.The full model is the two-branch GCN variant, and qualitative snapshots are presented alongside the overall quantitative comparison.
  • Ablation analysis: The two-branch model improves over the RGB-only branch by complementarily fusing visual details with semantic scene context.The authors attribute the promotion to complementary RGB and semantic information, with GCN providing semantic-context inference.
  • Method behavior: Static-image methods SALICON and SalGAN show promising performance, while dynamic objects can disturb video-based prediction.The paper suggests that temporal scene variation and the complexity of attention allocation in challenging videos contribute to this pattern.
  • Accident-window evaluation: Within accident windows, most methods weaken relative to overall testing, but the full model remains the winner; BDDA adapts well to critical frames.The accident-window evaluation contains 14,771 testing frames, and BDDA’s design emphasizes frames appearing during critical events.
  • Behavior-type evaluation: Across crossing, hitting, and out-of-control behaviors, SCAFNet is best, while all methods perform worst on crossing and best on out of control.Crossing involves apparent scale variation and slower motion; out-of-control events often involve sudden, drastic movement.
  • Driving-situation evaluation: Across light, weather, and driving-occasion conditions, SCAFNet has the smallest KLdiv circle and remains superior on NSS, CC, and SIM.BDDA is closer to SCAFNet than the other approaches, which the authors associate with its consideration of critical situations.

F. Evaluation on DR(eye)VE and TrafficGaze Datasets.

SCAFNet is also evaluated on the normal-driving TrafficGaze and DR(eye)VE datasets, where it shows promising performance despite differences in scenario complexity and fixation patterns.

  • TrafficGaze: TrafficGaze evaluation compares SCAFNet with MLNet and CDNN using KLdiv, NSS, CC, SIM, and AUC-Judd.CDNN achieves a KLdiv value of 0.29, while the passage states that SCAFNet has good approximation to human fixation.
  • TrafficGaze: TrafficGaze’s simpler highway setting may explain why MLNet performs feasibly there but poorly on DADA’s complex and diverse driving situations.The passage characterizes TrafficGaze as relatively simple and DADA as containing diverse driving conditions.
  • DR(eye)VE: On DR(eye)VE, SCAFNet is compared with MLNet, RMDN, and the full DR(eye)VE model using KLdiv and CC, and shows promising performance.The dataset is described as huge, with other methods’ values reported from the DR(eye)VE work.
  • Cross-dataset comparison: TrafficGaze and DR(eye)VE commonly place fixations near the road’s vanishing point, unlike the more varied DADA accident scenarios.The passage links these fixation patterns with lower KLdiv and higher CC values compared with DADA.
  • Cross-dataset conclusion: Across DADA, DR(eye)VE, and TrafficGaze, the paper reports superior SCAFNet performance against state-of-the-art methods.The conclusion also notes analyses by accident-window behavior type and by light, weather, and driving occasion.
Loading 1912.12148v2…