PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene-Graph Augmented Data-Driven Risk Assessment of Autonomous Vehicle Decisions

Shih-Yuan Yu, Arnav V. Malawade, Deepan Muthirayan, Pramod P. Khargonekar, Mohammad A. Al Faruque

arXiv:2009.06435v1cs.CV

TL;DR

Deep-learning risk assessment may miss relationships among traffic participants, and synthesized training data raises questions of transferability while model decisions can lack explainability. This paper uses scene graphs, graph convolution, an LSTM, and attention to assess driving-sequence risk; reported results include stronger accuracy and AUC and a smaller average accuracy drop when transferring from simulated to real-world data.

  • Problem

    Deep-learning risk-assessment methods may not capture relationships among traffic participants, and their transferability and explainability remain concerns.

    Source: §I · p. 1

  • Method

    The paper represents driving scenes as scene graphs and combines graph convolution, temporal LSTM modeling, and attention to classify driving-sequence risk.

    Source: §I · p. 2, §III · p. 3

  • Results

    95.8% vs. 91.4% accuracy and 0.978 vs. 0.958 AUC; average accuracy drop was 5.0% for the proposed approach vs. 18.1% for.

    Source: §V · p. 8

  • Takeaways & Limitations

    Attention layers support explainability by highlighting scene entities and frames associated with risk in the lane-change example.

    Source: D. Evaluation of Attention Mechanisms… · p. 8

Abstract

from arXiv · show

Despite impressive advancements in Autonomous Driving Systems (ADS), navigation in complex road conditions remains a challenging problem. There is considerable evidence that evaluating the subjective risk level of various decisions can improve ADS' safety in both normal and complex driving scenarios. However, existing deep learning-based methods often fail to model the relationships between traffic participants and can suffer when faced with complex real-world scenarios. Besides, these methods lack transferability and explainability. To address these limitations, we propose a novel data-driven approach that uses scene-graphs as intermediate representations. Our approach includes a Multi-Relation Graph Convolution Network, a Long-Short Term Memory Network, and attention layers for modeling the subjective risk of driving maneuvers. To train our model, we formulate this task as a supervised scene classification problem. We consider a typical use case to demonstrate our model's capabilities: lane changes. We show that our approach achieves a higher classification accuracy than the state-of-the-art approach on both large (96.4% vs. 91.2%) and small (91.8% vs. 71.2%) synthesized datasets, also illustrating that our approach can learn effectively even from smaller datasets. We also show that our model trained on a synthesized dataset achieves an average accuracy of 87.8% when tested on a real-world dataset compared to the 70.3% accuracy achieved by the state-of-the-art model trained on the same synthesized dataset, showing that our approach can more effectively transfer knowledge. Finally, we demonstrate that the use of spatial and temporal attention layers improves our model's performance by 2.7% and 0.7% respectively, and increases its explainability.

I. INTRODUCTION

The introduction motivates subjective risk assessment for complex driving and identifies gaps in modeling traffic relationships, transferability, and explainability. It proposes scene graphs as intermediate representations for assessing driving risk, using lane changes as a use case.

I. INTRODUCTION

  • Modeling subjective risk matters because drivers typically avoid actions they perceive as risky.The paper distinguishes subjective risk, a driver’s perceived risk, from objective accident probability.Source: §I · p. 1
  • Existing deep-learning methods may not capture relationships between traffic participants, while training data covering risky corner cases can be costly to generate.The introduction also identifies transferability from synthesized training data and explainability as challenges.Source: §I · p. 1
  • Scene graphs serve as intermediate representations that encode relationships among traffic participants for subjective risk assessment of lane changes.The paper identifies lane changes as its specific maneuver and notes that improper execution accounts for 7.62% of light-vehicle traffic accidents.Source: §I · p. 2
  • The proposed architecture combines scene-graph extraction, an MR-GCN, an LSTM, and attention layers, and frames training as supervised scene-graph sequence classification.The node-attention and LSTM-level attention layers are presented as supporting performance and explainability.Source: §I · p. 2
  • The approach is presented as outperforming existing methods across scenarios, transferring simulated knowledge to real-world assessment, and improving explainability with multi-level attention.These are stated as contributions without quantitative comparisons in the supplied introduction passages.Source: §I · p. 2

A. Design Philosophies in ADSs

The paper contrasts ADS design philosophies and prior graph-based and subjective-risk approaches before presenting its scene-graph sequence model. The model maps image sequences to safe-or-risky labels through graph processing, temporal modeling, and final classification.

A. Design Philosophies in ADSs

  • ADS design spans modular pipelines and end-to-end models; intermediate representations offer another approach, and this work uses scene graphs to encode spatial and semantic relations among traffic participants.The related-work discussion also describes direct perception and filtered-image representations as alternatives.Source: A. Design Philosophies in ADSs · pp. 2–3

B. Graph-Based Driving Scene Understanding

  • Prior graph-based driving work predicts individual object behavior, whereas this work assesses scenario-level subjective risk and adds node attention, graph pooling, and readout.The paper identifies its approach as inspired by earlier multi-relational graph convolution work.Source: B. Graph-Based Driving Scene… · p. 3

C. Risk Assessment

  • Prior risk-assessment work includes unsafe-lane-change detection and scene-level deep spatiotemporal models, while the proposed representation captures relations among traffic participants.The cited scene-level approach uses Mask-RCNN-generated image representations but does not model relations among all participants.Source: C. Risk Assessment · p. 3

III. SCENE-GRAPH AUGMENTED APPROACH FOR RISK ASSESSMENT

  • The proposed model extracts a scene graph from each frame, computes graph embeddings with graph convolution, attention pooling, and readout, then uses an LSTM and an MLP with Softmax to classify sequence risk.The task assumes driving sequences belong to mutually exclusive, exhaustive safe and risky classes.Source: §III · p. 3

A. Scene-Graph Extraction Pipeline

The Scene-Graph Extraction Pipeline provides real-image and Carla ground-truth routes for constructing driving-scene graphs. The real-image route estimates objects and their attributes from images, then assigns vehicles to lane nodes.

A. Scene-Graph Extraction Pipeline

  • The pipeline uses a partially rule-based Real Image Pipeline and a Carla Ground Truth Pipeline as a surrogate for ideally extracted object attributes.The ground-truth pipeline uses simulator information rather than image processing.Source: A. Scene-Graph Extraction Pipeline · p. 3

1) Real Image Pipeline:

  • The Real Image Pipeline detects objects, creates a birds-eye view to estimate locations relative to the road and ego vehicle, and derives scene-graph attributes from location and class.It uses Detectron2 for object detection and OpenCV perspective transformation for the top-down projection.Source: 1) Real Image Pipeline · p. 3
  • Vehicles are assigned to Left, Middle, or Right Lane nodes by horizontal displacement, with vehicles spanning two lanes linked to both.The abstraction maps all lanes on either side into a single Left Lane or Right Lane node.Source: 1) Real Image Pipeline · p. 4

1) Spatial Modeling:

The spatial model represents scenes as directed multi-relational graphs and uses MR-GCN and attention-based pooling to produce a graph-level embedding. Its graph construction uses distance and relative-position rules, while the ground-truth pipeline evaluates performance with simulator attributes.

1) Spatial Modeling:

  • The Carla GT Pipeline uses simulator-provided object classes and locations to assess the method assuming error-free attribute extraction.The paper notes that monocular object detection and location estimation can be unstable under factors such as weather and camera position.Source: 1) Spatial Modeling · p. 4

3) Graph Construction:

  • Scene-graphs are directed multi-graphs whose nodes represent scene objects and whose typed edges encode relations such as proximity, direction, and lane membership.Distance relations use fixed thresholds, while directional relations are assigned based on relative position within the Near threshold.Source: 3) Graph Construction · p. 4

1) Spatial Modeling:

  • MR-GCN captures multiple relation types, and concatenating node features across its layers forms the final node representation.The model then applies attention-based graph pooling and a graph readout to produce each scene-graph embedding.Source: B. Scene-graph Sequence Based Risk… · pp. 4–5

B. Scene-graph Sequence Based Risk Assessment Model

  • Attention-based pooling ranks nodes and retains a fraction determined by the pooling ratio; sum, mean, or max readout then yields a graph-level embedding.The candidate ratios are 0.25, 0.5, and 0.75.Source: B. Scene-graph Sequence Based Risk… · p. 5

2) Temporal Modeling:

An LSTM processes the sequence of scene-graph embeddings into a spatio-temporal representation. Temporal readout can use the final hidden state or attention over the hidden-state sequence.

2) Temporal Modeling:

  • The LSTM processes scene-graph embeddings over time, updating hidden and cell states and producing a sequence of hidden representations.Each timestamp's input is the final scene-graph embedding for that timestamp.Source: 2) Temporal Modeling · p. 5
  • Temporal readout either selects the final hidden state or uses attention to weight hidden states according to their importance for the risk output.The attention weights are computed with a Softmax over the hidden-state energy scores.Source: 2) Temporal Modeling · p. 5
  • The attention context vector is passed to an LSTM decoder to produce the clip’s final spatio-temporal embedding.The energy for each hidden state is scored in relation to the last hidden representation.Source: 2) Temporal Modeling · p. 5

3) Risk Inference:

Risk inference maps the spatio-temporal embedding to a clip-level risk prediction. Training uses cross-entropy with class weighting and dropout.

3) Risk Inference:

  • A multilayer perceptron followed by Softmax uses the spatio-temporal embedding to predict the clip’s risk assessment.Source: 3) Risk Inference · p. 5
  • The prediction is trained by minimizing cross-entropy loss.Source: 3) Risk Inference · p. 5
  • Mini-batch gradient descent updates model parameters, with class weighting for label imbalance and dropout layers to reduce overfitting.Source: 3) Risk Inference · p. 6

IV. EXPERIMENTAL RESULTS

The experiments compare the proposed model with a state-of-the-art baseline on risk-assessment accuracy and transferability, and examine the contributions of its major components.

  • The experiments compare the model and baseline on risk-assessment accuracy and transferability, and use ablation to evaluate major model components.Source: §IV · p. 6

A. Dataset Preparation

The study uses real-world and synthesized lane-change clips, assigns binary risk labels from annotator scores, and evaluates models with stratified train-test splits.

A. Dataset Preparation

  • Real-world clips came from the Honda Driving Dataset, while synthesized lane changes were generated with Carla and Scenario Runner.The simulation tool varied traffic, weather, lighting, and vehicle behaviors to create a range of urban driving scenarios.Source: A. Dataset Preparation · p. 6
  • Annotators scored each clip from -2 to 2, and the average score was labeled safe when ≤0 and risky when greater than 0.Source: A. Dataset Preparation · p. 6
  • The datasets included 271-syn and 1043-syn, balanced subsets 96-syn and 306-syn, and 571-honda with 571 real-world clips.Source: A. Dataset Preparation · p. 6
  • Each dataset was split stratified 70:30 into training and testing sets, and reported scores averaged results across ten splits.Source: A. Dataset Preparation · p. 6

B. Training and Model Specification

The experiments compare the proposed model with on synthesized lane-change datasets, test performance under dataset-size and class-balance differences, and ablate modeling components.

B. Training and Model Specification

C. Experiments on Risk Assessment

D. Evaluation of Attention Mechanisms on Risk Assessment

Spatial and temporal attention are evaluated for risk assessment, with reported performance effects and attention visualizations that highlight decision-relevant frames and scene objects.

D. Evaluation of Attention Mechanisms on Risk Assessment

  • SAGPool improved performance by 2.7% over an attention-free model, while TopkPool produced a 2.4% drop and was relatively unstable.The authors suggest TopkPool may ignore node relations when computing attention.Source: D. Evaluation of Attention Mechanisms… · pp. 7–8
  • Adding LSTM-attn to a model with one MR-GCN layer increased performance by 0.7% over the same model without temporal attention.The authors note the effect may be marginal because irrelevant lane-change frames had been manually removed.Source: D. Evaluation of Attention Mechanisms… · p. 8
  • In the illustrated risky lane change, frame attention peaked immediately before collision, while node weights rose for the ego car and yellow vehicle as collision risk increased.The authors describe these visualizations as useful for identifying factors behind a risk decision and debugging edge cases.Source: D. Evaluation of Attention Mechanisms… · p. 8

E. Transferability from Virtual To Real Driving

Testing models trained on synthesized data directly on the 571-honda real-world dataset, the authors report smaller accuracy drops for their approach than for the baseline. The conclusion also reports stronger benchmark results, better performance with less training data, and improved transfer and explainability.

E. Transferability from Virtual To Real Driving

  • 6.7% and 3.5% accuracy drops for the proposed model, versus 21.3% and 14.9% for the baseline, were reported after testing models trained on 271-syn and 1043-syn on 571-honda.The models’ weights and parameters were used directly for real-world testing.Source: E. Transferability from Virtual To… · p. 8

V. CONCLUSION

  • 5.0% average accuracy drop for the proposed model versus 18.1% for summarizes the conclusion’s transfer comparison.The conclusion also reports 95.8% versus 91.4% accuracy and 0.978 versus 0.958 AUC, and 91.8% versus 78.2% accuracy on the 96-syn dataset.Source: §V · p. 8
  • Figure 5 visualizes node projection scores and temporal attention coefficients for a risky lane-changing clip.Node scores use a light-yellow-to-red gradient, while temporal coefficients are shown as a white-to-red bar chart.Source: Figure 5 · p. 9

Related papers

Loading 2009.06435v1…