Source-linked AI summary
Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving
Jörg Gamerdinger, Victor Schwarzenberger, Philipp Schmid, Sven Teufel, Oliver Bringmann
TL;DR
The paper addresses the lack of reliable criticality metrics that generalize beyond specific scenarios and vehicle-to-vehicle interactions, especially for less predictable VRUs. It proposes a VRU-centric metric and a scenario-independent prediction framework, evaluated on a DeepAccident-based dataset. The approaches improve criticality classification, including up to 50% F1 improvement for the VRU metric and approximately 275% improvement for the prediction framework.
Problem
Existing criticality metrics are scenario-dependent, mainly designed for vehicle-to-vehicle interactions, and neglect the greater motion uncertainty of VRUs.
Method
The paper introduces a VRU-centric metric based on VRU motion uncertainty and a scenario-independent prediction framework evaluated with a DeepAccident-based criticality dataset.
Results
The VRU-centric metric improves F1-score by up to 50% over the best state-of-the-art metric, while the prediction framework improves F1-score by approximately 275%.
Takeaways & Limitations
The proposed approaches support criticality assessment for VRUs and scenario-independent classification across traffic participants.
Abstract
from arXiv · showhide
Increasing safety is the primary objective of automated vehicles. Achieving this goal requires reliable safety metrics that incorporate safety-relevant factors such as object type, velocity, and criticality. A key capability of such metrics is the distinction between critical and non-critical objects, which is addressed through criticality or relevance estimation. Existing criticality metrics are typically designed for specific scenarios and primarily focus on vehicle-to-vehicle interactions. In this paper, we therefore propose a novel criticality metric tailored to vulnerable road users (VRUs), which require special consideration due to their less predictable motion behavior. Furthermore, to avoid the complexity introduced by scenario-specific metrics, we introduce a scenario-independent criticality prediction framework applicable to all traffic participant classes. The effectiveness of both the proposed VRU-centric criticality metric and the criticality prediction framework is evaluated using the DeepAccident dataset, which contains a diverse set of safety-critical traffic scenarios. The proposed VRU-centric criticality metric improves pedestrian criticality classification performance by up to 50 %. In addition, the proposed criticality prediction framework outperforms state-of-the-art metrics by 275 %, achieving an F1-score of 0.96 and enabling scenario-independent criticality assessment across all object classes. These results demonstrate the strong potential of the proposed approaches to enhance criticality assessment for safety evaluation in automated driving systems.
I. INTRODUCTION
The paper addresses limitations of safety-criticality metrics that overlook object type, velocity, VRU motion uncertainty, and scenario variation. It introduces a VRU-focused metric and a scenario-independent prediction framework for criticality assessment.
- Existing criticality metrics are highly scenario-dependent, increase complexity, and mainly target vehicle-to-vehicle interactions while neglecting VRU-specific uncertainty.VRUs have greater freedom in movement and less predictable motion than vehicles.
- A scenario-independent criticality prediction framework is proposed to assess criticality across all traffic participants.
- The work introduces the first dataset with object-dynamics-aware labels for perception criticality.
- The paper introduces a novel criticality metric for vulnerable road users that focuses on uncertainty in their movements.
II. RELATED WORK
Prior criticality metrics use time, distance, or reachability formulations, but many are complex and constrained to particular traffic scenarios. Their applicability is therefore limited when diverse interactions must be assessed.
- Criticality metrics are grouped into time-based, distance-based, and reachability-based categories.
- Time-Based Metrics: Time-based metrics estimate the time until a specified event, commonly using TTC from distance and velocity.Related metrics include CIF, MTTC, CI, TTA, and TTB.
- Distance-based Metrics: Distance-based metrics provide thresholds for classifying criticality but often require complex calculations and scenario-specific designs.RSS formalizes driving judgment with longitudinal and lateral safety distances, times, and procedural rules.
- Reachability-based Metrics: Reachability-based metrics classify objects within possible trajectory zones, but existing examples can be restricted to narrow maneuvers or operating conditions.The cited parking method is limited to reverse parking up to 5 km/h.
III. PERCEPTION CRITICALITY DATASET
The paper constructs a perception-criticality dataset from synthetic safety-critical traffic scenarios and derives object labels using multiple criticality and trajectory-based analyses. The labeling combines established metrics with geometric and kinematic reasoning.
- The dataset is based on DeepAccident, whose safety-critical scenarios include four sensor vehicles and up to 46 additional pedestrians, cyclists, motorcyclists, or vehicles.The selected train split contains 480 scenarios, treated separately for four sensor vehicles, yielding 1920 scenarios.
- Multi-Metric-Based: Multi-metric labels combine binary outputs from TTC3 s, CIF100, and RSS0.5 to determine global object criticality.The authors note that these established metrics can still introduce label noise.
- Trajectory-Based: Trajectory-based labeling treats collisions and potential intersections with the ego trajectory as safety-critical and performs reachability analysis for other objects.
- Trajectory-Based: Reachability geometry reflects object-specific motion constraints, using pedestrian velocity and time horizon for circular zones and steering-angle limits for other participants.Vehicle steering limits depend on object type, velocity, and trajectory horizon; low-velocity steering is capped at 35°.
IV. VRU-CENTRIC CRITICALITY ASSESSMENT
The VRU-centric assessment models pedestrians’ greater motion uncertainty by combining behavior prediction with space occupancy to classify criticality.
- Pedestrians’ higher motion uncertainty motivates a VRU-specific metric because abrupt stopping and direction changes contrast with vehicle-constrained motion.Vehicle lateral acceleration and turn rates are more limited, making vehicle motion more predictable.
- The metric combines pedestrian behavior models with an adapted space occupancy index to represent possible movement during the prediction horizon.Pedestrian motion is modeled with a forward-shifted circular shape and a fast-walking velocity of 2.8 m/s.
- Criticality is determined by whether the predicted possible spaces of two objects overlap; overlap classifies the VRU as critical.Without overlap, the object is considered uncritical.
- Pedestrians in the ego vehicle’s driving direction are additionally classified as critical within twice the required braking distance, with a 6 m minimum polygon size.The minimum size preserves a criticality zone for a stationary vehicle and can include a pedestrian crossing.
- The approach incorporates pedestrian-specific movement properties that state-of-the-art criticality metrics do not capture, illustrated by example scenes in Fig. 2.The examples demonstrate the resulting VRU criticality assessment.
V. CRITICALITY PREDICTION
The paper uses simple statistical models to predict criticality independently of object class and scenario, balancing generalization with traceability.
- The framework predicts criticality across object classes and scenarios using four unsupervised and four supervised statistical methods.Methods include KMeans, DBScan, Isolation Forest, GMM, linear regression, random forest, histogram-based gradient boosting, and LightGBM.
- The model collection spans clustering, regression, and tree-based approaches while avoiding more complex models that would reduce traceability for safety applications.The paper therefore uses simpler statistical models.
- The prediction input vector contains object class, relative position, dimensions, yaw, relative velocity, LiDAR-point count, and camera visibility.These features are defined as V = (c, x, y, z, l, w, h, Ψ, vx, vy, lp, vis).
- Supervised models use an 80/20 layered train-test split, balanced class weights, and model-specific preprocessing or hyperparameter tuning.The decision threshold is scanned from 0.01 to 0.99 and selected to maximize test-set F1.
A. Experiments
The experiments evaluate both proposed approaches with precision, recall, and F1, comparing criticality prediction against selected state-of-the-art metrics.
- Both the VRU-centric metric and criticality prediction are evaluated using precision, recall, and their harmonic mean F1.The evaluation defines precision and recall from true-positive, false-positive, and false-negative classifications.
- Criticality prediction is compared with TTC, MTTC, TTA, CIF, RSS, and SACRED, selected because they performed best in prior work.Results are visualized in Figs. 3 and 4 and summarized in Tables I and II.
B. VRU-Centric Criticality Metric
On pedestrians, the VRU-centric CVRU metric outperforms the evaluated state-of-the-art metrics, achieving the best reported recall and F1-score.
- F1-score 0.57 for CVRU is the best result, about 50% higher than RSS at 0.39.CVRU also achieves precision 0.96 and recall 0.45; RSS has precision 0.61 and recall 0.28.
- TTC reaches precision 0.99 but only about 0.06 F1-score because its recall is very low, while CIF reaches about 0.08 F1-score.MTTC, TTA, and SACRED each achieve about 0.15 F1-score.
- The evaluated state-of-the-art methods are designed for vehicle-to-vehicle interaction and neglect properties specific to vulnerable road users.This limitation is reflected in their pedestrian classification results.
C. Criticality Prediction
Across all object classes, supervised prediction methods substantially outperform SOTA and unsupervised approaches, with Random Forest achieving the strongest reported performance.
- C. Criticality Prediction: SOTA methods achieved F1-scores from 0.0883 for CIF to 0.3545 for RSS, while unsupervised methods ranged from 0.0922 for GMM to 0.1678 for DBScan.
- C. Criticality Prediction: Random Forest achieved the best overall performance, with precision 0.9303, recall 0.9415, and F1-score 0.9359.Results are reported over all classes in the proposed dataset.
- C. Criticality Prediction: LightGBM slightly outperformed HGB, reaching precision 0.9005, recall 0.9317, and F1-score 0.9158 versus HGB's F1-score of 0.9117.
- C. Criticality Prediction: Logistic Regression achieved an F1-score of 0.6836, improving by more than 0.32 over the best-performing SOTA metric.
D. Discussion
The results indicate that existing SOTA metrics perform poorly for pedestrian criticality classification, whereas the proposed CVRU metric improves performance by modeling pedestrian motion characteristics.
- D. Discussion: The proposed CVRU metric significantly improves pedestrian criticality classification by incorporating pedestrians' higher freedom of movement.
- D. Discussion: SOTA metrics are not suitable for pedestrian criticality classification.
- D. Discussion: SOTA metrics exhibit considerable limitations across diverse traffic scenarios, consistent with prior observations.
VII. CONCLUSION & OUTLOOK
The paper contributes a perception-criticality dataset, a VRU-centric metric, and a scenario-independent prediction framework. The metric improves VRU classification, while the framework substantially improves classification across traffic participants.
- VII. CONCLUSION & OUTLOOK: The proposed dataset contains 161k frames and 5.46M labeled objects, with labels generated through metric aggregation, geometric properties, and kinematic constraints.
- VII. CONCLUSION & OUTLOOK: The VRU-specific metric improves F1-score by up to 50% over the best SOTA metric by modeling VRU movement and direction-change uncertainty.It combines VRU motion models with Space Occupancy Index potential-space intersections.
- VII. CONCLUSION & OUTLOOK: The scenario-independent prediction framework improves F1-score by 0.60, corresponding to an approximately 275% increase in criticality classification performance.
- VII. CONCLUSION & OUTLOOK: Future work will add prediction methods, fine-tuning, temporal information, and the VRU-centric metric, while publicly releasing the dataset and framework.