PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidents

Muhammad Monjurul Karim, Yu Li, Ruwen Qin, Zhaozheng Yin

arXiv:2106.10197v2cs.CVcs.AI

TL;DR

Computational methods that turn dashcam video into perceptions of accident risk are desired for safety enhancement. The DSTA network dynamically attends to salient video segments and regions, and the authors report that it outperformed the state of the art.

  • Problem

    Accident anticipation is desired for safety enhancement, while computational methods translating dashcam video into perceptions of accident risk are largely desired.

    Source: §I · p. 1

  • Method

    The DSTA network learns to dynamically attend to salient temporal segments and spatial regions of driving-scene video for early accident anticipation.

    Source: §I · p. 1

  • Results

    DSTA outperformed the state of the art, while score-level late fusion achieved higher prediction accuracy than its constituent networks.

    Source: §I · p. 1

  • Takeaways & Limitations

    The paper identifies model fusion and dataset analysis as opportunities for further advancing early accident anticipation.

    Source: §V · p. 9

    The DSTA network shares the unexplainability of rationale common to many deep learning models.

    Source: §V · p. 10

Abstract

from arXiv · show

The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.

I. INTRODUCTION

The introduction motivates dashcam-based accident anticipation as a safety capability and presents DSTA’s dynamic spatial-temporal attention and score-level fusion alongside gaps in prior work.

I. INTRODUCTION

  • Dashcams capture accident-related factors, and anticipating risk from their video could improve preparedness for human drivers, ADAS, and autonomous vehicles.The paper notes that accident factors include human, environmental, road, traffic, temporal-spatial, and vehicle characteristics.Source: §I · p. 1
  • DSTA dynamically attends to salient temporal segments and spatial regions for early accident anticipation, and the paper reports that it outperformed the state of the art.The design is inspired by human attention to risky driving-scene regions.Source: §I · p. 1
  • Score-level late fusion is presented as applicable to complementary trained networks and as more accurate than its constituent networks.Source: §I · p. 1
  • The paper identifies analysis of representative traffic-accident datasets’ applicability and limitations as a contribution.Source: §I · p. 1

II. THE LITERATURE

  • Human-action anticipation methods – provide a foundation for predicting future events but use static-surveillance video rather than vehicle-mounted cameras.Source: §II · p. 2
  • Prior accident-anticipation attention methods – focus on spatially distributed agents, while temporal importance is ignored at frame level, object level, or both.Source: §I · p. 1, §II · p. 2

III. METHODOLOGY

DSTA processes dashcam frames into spatially attended features and uses a GRU with dynamic spatial and temporal attention to predict future accidents.

A. Dynamic Spatial-Temporal Attention Framework

  • The framework combines a GRU with DSA, DTA, and a training-only TSAA module; DSA weights object features and DTA weights past hidden representations.TSAA aggregates all hidden representations of a training video to predict its class during training.Source: §III · p. 2

1) Feature Extraction and Aggregation:

  • An object detector selects the top N detected objects per frame, and VGG-16 extracts frame-level and object-level features for aggregation.The weighted object-feature aggregation is concatenated with the frame-level feature to form each frame’s overall feature vector.Source: 1) Feature Extraction and Aggregation · pp. 2–3

2) Dynamic Spatial Attention (DSA):

  • DSA computes spatial weights using the previous weighted hidden representation and current-frame object features, then aggregates the object features.The attention weights are learned parameters normalized by softmax.Source: 2) Dynamic Spatial Attention (DSA) · p. 3

3) Dynamic Temporal Attention (DTA):

DTA weights recent hidden representations so that the GRU can emphasize temporally informative frames, with the window length balancing distraction against insufficient context.

3) Dynamic Temporal Attention (DTA):

4) Spatial-Temporal Relational Learning with GRU:

The GRU combines the current frame’s feature vector with an attended representation of past hidden states, then converts its current hidden representation into an accident probability.

4) Spatial-Temporal Relational Learning with GRU:

  • At each frame, the GRU takes the frame feature vector and an attended representation of prior hidden states as inputs.Its reset and update gates filter information from the sequence.Source: 4) Spatial-Temporal Relational… · p. 3
  • Two fully connected layers map the current hidden representation to positive- and negative-class scores, which softmax normalizes into accident-anticipation probability.Source: 4) Spatial-Temporal Relational… · p. 3

5) Temporal Self-Attention Aggregation (TSAA):

The training-only TSAA module attends over a video's GRU hidden representations to predict its video-level class. Training combines a frame-level loss that increasingly weights positive-video predictions near an accident with an auxiliary video-level loss.

5) Temporal Self-Attention Aggregation (TSAA):

  • TSAA uses learnable temporal self-attention to aggregate all GRU hidden representations into a video-level representation for predicting the video class.It is included only during training to help train the GRU hidden layers better.Source: 5) Temporal Self-Attention Aggregation… · p. 4

B. Training Procedure

  • The frame-level loss uses binary cross-entropy for negative videos and an exponentially increasing coefficient for positive videos as an accident approaches, encouraging early anticipation.After the accident begins, the positive-video loss coefficient reaches the same level as the negative-video coefficient.Source: B. Training Procedure · p. 4
  • Training also minimizes an auxiliary video-level loss, with its weight set to 15 to balance it against the primary frame-level loss.The combined loss updates DSA, DTA, GRU, and TSAA parameters by backpropagation.Source: B. Training Procedure · p. 4

C. Score-Level Late Fusion

The score-level late-fusion method combines frame-level predictions from two complementary models using separate classification thresholds. It selects the maximum, minimum, or mean score according to whether both models agree on an accident, agree on safety, or conflict.

C. Score-Level Late Fusion

  • The method is motivated by the earliness-correctness trade-off and the possibility that complementary models can improve performance, including models with the same architecture but different parameters.Models with different architectures may also complement one another.Source: C. Score-Level Late Fusion · p. 4
  • The fusion method takes the maximum score when both models exceed their respective thresholds, the minimum when both fall below them, and the mean when their classifications conflict.The resulting score is the new accident-anticipation probability for that frame.Source: C. Score-Level Late Fusion · p. 4
  • Separate thresholds are optimized on the testing dataset to select the pair that yields the best fusion performance.The thresholds are design parameters because constituent models can have different prediction behavior.Source: C. Score-Level Late Fusion · pp. 4–5

IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS

The study evaluates its early-accident-anticipation approach on two public datasets; this section describes the DAD dataset. DAD contains videos from multiple Taiwanese cities, with positive examples ending in accidents.

IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS

1) Dashcam Accident Dataset (DAD) [8]:

2) Car Crash Dataset (CCD) [23]:

The section describes the Car Crash Dataset, implementation choices, and correctness metrics for accident anticipation. It uses precision at 80% recall to assess precision at a recall level considered reasonably good in prior accident-anticipation studies.

2) Car Crash Dataset (CCD) [23]:

  • CCD contains 4,500 five-second videos at 10 fps, with accidents randomly placed in the final 2 seconds of positive videos.The dataset is split 80% for training and 20% for testing, with a 1:2 positive-to-negative ratio in both splits.Source: 2) Car Crash Dataset (CCD) [23] · p. 5

B. The Detail of Implementation

  • The study uses provided CCD features for fair comparison, with candidate objects detected using Cascade R-CNN and VGG-16 features reduced from 4,096 to 512 dimensions.The DTA temporal sliding window is 0.5 seconds; the GRU hidden representation is also 512-dimensional.Source: B. The Detail of Implementation · p. 5

C. Evaluation Metrics

  • Correctness is evaluated with recall, precision, and average precision, which is independent of the classification threshold.Average precision is the area under the precision-recall curve.Source: 1) Correctness · p. 5

1) Correctness:

  • P80R measures precision at 80% recall, a level treated as reasonably good in prior accident-anticipation studies.The passage notes that very high recall may be unrealistic when it comes at the cost of very low precision.Source: 1) Correctness · p. 5

2) Earliness:

Earliness is assessed alongside accuracy using TTA, mTTA, and TTA80R. On DAD, longer anticipation time trades off against AP, so model choice depends on users’ requirements for earliness and accuracy.

2) Earliness:

  • TTA is the period from when frame-level anticipation probability crosses the classification threshold until the accident begins; mTTA averages TTA across thresholds.TTA80R is TTA at 80% recall, representing a relatively high recall requirement.Source: 2) Earliness · p. 6

1) Experimental Evaluation:

  • 3.75 seconds of mTTA corresponds to 53.7% AP, while the best-AP efficient model reaches 72.3% AP at 1.5 seconds of mTTA.The same mTTA–AP relationship is observed on CCD.Source: Figure 2 · p. 6, 1) Experimental Evaluation · p. 6
  • Choosing a Pareto-frontier model depends on user needs, which vary with weather, time of day, and driving area; some prioritize early prediction and others accuracy.The study retains models with at least 1 second of mTTA and 50% AP as applicable to real-world use.Source: 1) Experimental Evaluation · p. 6

2) Comparison to the State-of-the-Art Models:

DSTA outperforms prior models on DAD and CCD, while the ablations show contributions from its attention modules and support GRU over LSTM and GCRNN. The experiments also expose trade-offs between AP and mTTA for some temporal-attention changes.

2) Comparison to the State-of-the-Art Models:

3) Ablation Study:

  • The full DSTA configuration has higher AP than any other ablation, while another model it provides reaches 70.5% AP and the longest mTTA in Table III, at 1.74 seconds.The table reports the best AP and corresponding mTTA for each experiment.Source: 3) Ablation Study · p. 7, Table III · p. 7
  • Reducing the temporal-attention window to the last frame lowers AP by 1.6%, which the authors attribute to losing recent-frame representations for learning temporal relations.The experiment compares a one-frame window with the several recent frames used by the temporal attention model.Source: 3) Ablation Study · p. 7
  • Replacing GRU with GCRNN lowers AP by 1.87% and mTTA by 0.28 seconds, while replacing it with LSTM lowers AP by 3.26% and mTTA by 0.02 seconds; the authors favor GRU with the attention modules.The GCRNN experiment's training time is about five times that of experiments 1 and 4.Source: 3) Ablation Study · p. 7
  • Removing TSAA lowers AP by 2.45% but raises mTTA by 0.20 seconds; removing DTA lowers AP by 2.62% and mTTA by 0.07 seconds, while removing both lowers AP by 7.31% and raises mTTA by 0.05 seconds.The authors interpret these experiments as showing that the temporal attention modules improve AP.Source: 3) Ablation Study · p. 7
  • Removing DSA lowers AP by 4.19% and mTTA by 0.17 seconds, and comparisons across other ablation pairs likewise support its contribution to both measures.The authors conclude that DSA increases both AP and mTTA.Source: 3) Ablation Study · p. 7

4) Qualitative Results:

DSTA’s attention maps shift over frames and highlight accident-related agents. Selected score-level fusions improve DAD results, while dataset differences challenge cross-dataset performance.

4) Qualitative Results:

  • Spatial attention shifts among objects across frames, while temporal attention changes its frame weights between prediction times.In the illustrated positive video, accident-involved agents were among the top-attended objects in several frames; frame #52 had the highest temporal attention at frame #58, and the weights changed by frame #60.Source: 4) Qualitative Results · p. 7, Figure 3 · p. 8
  • With a 0.5 threshold, DSTA anticipated one accident at frame #54 with 1.75 seconds TTA, correctly stayed below threshold for a negative video, and missed a distant-agent accident.The authors suggest the agents’ distance from the ego vehicle might make their spatial-temporal relational dynamics harder to extract.Source: 4) Qualitative Results · p. 7, Figure 4 · p. 9

E. Performance of the Fusion Method

  • DSTA(I) fused with GCRNN+AdaLEA achieved 74.1% AP with 1.08 seconds mTTA; DSTA(I)+DSTA(II) reached 56.4% P80R.The latter fusion exceeded DSTA(I) by 9.8% and DSTA(II) by 3.3% in P80R, and dominated all four individual models on the mTTA-AP diagram.Source: Table IV · p. 8, E. Performance of the Fusion Method · p. 8

F. Impact of the Datasets

  • CCD-small achieved 98.51% AP and 4.83 seconds mTTA despite matching DAD’s size, indicating CCD’s larger size alone was not the main reason for its stronger results.The paper reports that CCD is over 2.5 times larger than DAD, while performance on DAD was substantially lower.Source: F. Impact of the Datasets · p. 8
  • DAD’s mostly complex, non-ego accidents and brief agent appearances differ from CCD, where 53.4% of accidents involve the ego vehicle; the authors call for a more balanced, diverse dataset.The paper identifies these dataset differences as important reasons for different model performance.Source: F. Impact of the Datasets · pp. 8–9
  • Cross-testing produced AP near 36%; mixed training yielded 98.89% AP and 4.48 seconds mTTA on CCD, and 69.67% AP and 1.40 seconds mTTA on DAD, below Table I results.The four dataset-difference experiments are summarized in Table V.Source: F. Impact of the Datasets · p. 9, Table V · p. 9

V. CONCLUSION

The paper presents DSTA for dashcam-based accident anticipation and identifies model fusion and dataset analysis as opportunities to advance the task. It also points to explainability and multimodal integration as future directions.

V. CONCLUSION

  • The authors propose integrating accident anticipation with LiDAR, internet of things, and transportation safety surrogate measurements in a multimodal, multifunctional system.They also identify sensor-fusion-based, surrogate-safety-informed deep learning as a future direction.Source: §V · pp. 9–10
  • The network’s decision rationale remains unexplainable; the authors identify revealing its risk-perception mechanism as follow-up work toward human trust.The conclusion also emphasizes the need for a strong data foundation for accident anticipation.Source: §V · p. 10
  • Applying DSTA to mobile robots such as drones and remotely operated underwater vehicles faces additional challenges.Source: §V · p. 10

Related papers

Loading 2106.10197v2…