Source-linked AI summary
Misbehaviour Prediction for Autonomous Driving Systems
Andrea Stocco, Michael Weiss, Marco Calzana, Paolo Tonella
TL;DR
Autonomous-driving DNNs may fail in unexpected conditions, while existing testing largely addresses such cases offline. The paper presents SelfOracle for runtime confidence-based prediction and reports advance detection of many injected misbehaviours in simulation, outperforming DeepRoad.
Problem
The paper addresses the largely unexplored problem of monitoring DNN confidence at runtime to predict future misbehaviours under unexpected driving conditions.
Method
SelfOracle uses black-box reconstruction-based confidence estimation, Gamma-distribution threshold fitting, and time-series analysis to detect unexpected driving conditions.
Results
77% of out-of-bound episodes or crashes were anticipated up to 6 seconds in advance, with a 1% false alarm rate, and SelfOracle outperformed DeepRoad.
Takeaways & Limitations
SelfOracle can anticipate potentially safety-critical misbehaviours several seconds ahead while maintaining a low false alarm rate in the reported simulation study.
Takeaways & Limitations
The reported anticipation result may partly reflect simulator tracks designed with continuously and smoothly increasing injected anomalies.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) are the core component of modern autonomous driving systems. To date, it is still unrealistic that a DNN will generalize correctly in all driving conditions. Current testing techniques consist of offline solutions that identify adversarial or corner cases for improving the training phase, and little has been done for enabling online healing of DNN-based vehicles. In this paper, we address the problem of estimating the confidence of DNNs in response to unexpected execution contexts with the purpose of predicting potential safety-critical misbehaviours such as out of bound episodes or collisions. Our approach SelfOracle is based on a novel concept of self-assessment oracle, which monitors the DNN confidence at runtime, to predict unsupported driving scenarios in advance. SelfOracle uses autoencoder and time-series-based anomaly detection to reconstruct the driving scenarios seen by the car, and determine the confidence boundary of normal/unsupported conditions. In our empirical assessment, we evaluated the effectiveness of different variants of SelfOracle at predicting injected anomalous driving contexts, using DNN models and simulation environment from Udacity. Results show that, overall, SelfOracle can predict 77% misbehaviours, up to 6 seconds in advance, outperforming the online input validation approach of DeepRoad by a factor almost equal to 3.
1 INTRODUCTION
The paper frames runtime confidence monitoring as an unsupervised way to predict safety-critical autonomous-driving misbehaviours before they occur. SelfOracle operationalizes this idea and is evaluated against injected, previously unseen conditions in simulation.
- Unexpected driving conditions are difficult to handle because DNNs lack human-interpretable internal functioning and face a vast scenario space.
- Runtime misbehaviour prediction matters because missed unexpected conditions can cause severe consequences, while false alarms can reduce driving comfort.
- Existing autonomous-driving testing approaches primarily generate adversarial or corner cases offline to improve DNN robustness through retraining.
- SelfOracle combines confidence estimation, probability distribution fitting, and time-series analysis to detect unexpected conditions and support timely healing actions.Its reconstruction error provides a black-box confidence estimate, while a fitted Gamma distribution supports threshold selection.
- 77% of out-of-bound episodes or crashes were anticipated up to 6 seconds in advance, with a 1% false alarm rate under nominal driving.The evaluation used 72 simulations with controllably injected day/night, rain, and snow conditions disjoint from DNN training data.
- The study also contributes a dataset of 765 labeled simulation-based collision and out-of-bound episodes for evaluating prediction systems.
2 BACKGROUND
The background motivates black-box confidence estimation for complex autonomous-driving DNNs and introduces autoencoders and time-series analysis as relevant building blocks. SelfOracle connects reconstruction behavior with changing driving conditions and future misbehaviour risk.
- DNN-based Autonomous Vehicles: Autonomous-driving systems use sensor data from cameras, LIDAR, and GPS to drive real-time steering, braking, and acceleration decisions through DNNs.
- DNN-based Autonomous Vehicles: Because DNNs may not generalize across driving situations, confidence monitoring can identify low-confidence zones and support healing toward a safe state.
- Confidence Measures in DNNs: Early and accurate misbehaviour prediction is described as a precondition for safe human or automated self-healing.
- Confidence Measures in DNNs: Black-box confidence techniques monitor the relation between current images and training inputs without requiring changes to the existing DNN architecture.
- Autoencoders and Time Series: Autoencoders reconstruct their inputs, while time-series analysis identifies trends and predicts future values from DNN output sequences.AR models predict the next value from previous values, and LSTMs encode dependencies across input history.
- Confidence Measures in DNNs: Figure 1 uses SelfOracle reconstruction error as a proxy for DNN confidence, with higher error under increasingly adverse conditions.Sunny conditions produce low error, while heavy rain at night raises error above a threshold associated with driving off the road.
3 PROBLEM FORMULATION
The paper defines misbehaviour at the system level and focuses on recognizing low-confidence, unexpected driving contexts early enough to prevent collisions or out-of-bound episodes.
- For steering regression, isolated output errors are difficult to judge because optimal steering angles and acceptable deviations are generally unknown.
- A chain of inaccurate predictions is therefore evaluated by whether it ultimately produces a system misbehaviour, rather than by individual angle deviations.
- A DNN misbehaves when the containing system violates its requirements because of the DNN’s outputs.
- The evaluation focuses on collisions and out-of-bound episodes as safety-critical misbehaviours in autonomous driving.
- The approach targets low-confidence situations caused by unexpected execution contexts and aims to predict them before the vehicle crashes or leaves the road.
4 APPROACH
SelfOracle learns normal driving from visual histories, models reconstruction errors statistically, and combines thresholding with time-series filtering for online anomaly detection and self-healing.
- SelfOracle analyzes camera inputs end to end, without modifying the existing DNN or depending on its architecture.
- Model training under nominal driving behaviour: Training uses nominal visual streams to fit a driving-scenario reconstructor that operates on individual frames or sequences of preceding frames.
- Model training under nominal driving behaviour: Each reconstructed frame yields an error e_i = d(x_i,x'_i), producing a reconstruction-error set for modeling normality.
- Probability distribution fitting: A Gamma distribution is fitted to nominal reconstruction errors, and its estimated parameters determine a threshold θ for distinguishing anomalous from normal conditions.
- Probability distribution fitting: For an accepted false-alarm rate ϵ, SelfOracle sets θ = F^-1(1 − ϵ), classifying reconstruction errors at or above θ as anomalous.
- Usage scenario: During online use, an autoregressive filter smooths reconstruction-error sequences before threshold comparison; detected anomalies trigger self-healing to a safe state.
5 EMPIRICAL EVALUATION
The evaluation asks how effectively SelfOracle predicts anomalies, how prediction changes with earlier reaction periods, and how it compares with DeepRoad’s online validation.
- The study evaluates SelfOracle’s anomaly-prediction effectiveness and compares alternative reconstructors for autonomous vehicles.
- It also examines prediction performance as the reaction period increases and compares SelfOracle with DeepRoad’s online input validation.
- The evaluation uses three publicly available DNN-based self-driving cars: NVIDIA’s DAVE-2, Epoch, and Chauffeur.
5.3 Simulation Platform
The simulation platform extends Udacity’s environment with controllable unexpected contexts and automated detection and recovery for collision or out-of-bound events.
- The simulator supports training and autonomous modes, with an added third track and components for unexpected-context generation and collision/OBE detection.
- Unexpected context generator: Unexpected conditions include gradually changing illumination through a day/night cycle and variable-intensity rain, snow, and mist.
- Collision/OBE detection system: ACODS records unwanted interactions between the self-driving car and its environment to evaluate whether SelfOracle anticipates safety-critical episodes.
- Collision/OBE detection system: Collider callbacks distinguish normal road contact from crashes against objects and out-of-bound episodes.
- Collision/OBE detection system: An automatic restart mechanism returns the vehicle to a safe position after crashes or OBEs, enabling multiple simulations without manual intervention.
5.4 Procedure
The evaluation procedure builds training and evaluation data from simulated driving, compares several reconstructors, and implements DeepRoad as a baseline. Evaluation conditions include nominal, individual unexpected, and combined environmental effects.
- Data collection: 124,638 training images were collected across three tracks and two orientations during 10 laps per track.The training data were collected in training mode at 10–13 fps.
- Evaluation data: 72 evaluation simulations combined three SDC models, eight environmental conditions, and three tracks.Each simulation covered two laps at a maximum speed of 30 mph.
- Evaluation data: Evaluation conditions included nominal driving, four individual effects, and three combined day/night-plus-weather effects.The individual effects were day/night cycle, rain, snow, and fog.
- SelfOracle configurations: SelfOracle configurations used SAE, DAE, CAE, VAE, and an LSTM sequence-based reconstructor.The autoencoders ranged from a single-hidden-layer model to convolutional and variational architectures; the LSTM used two LSTM layers and one convolutional layer.
- Baseline: DeepRoad was reimplemented as the comparison baseline, using a 3,000-image randomly sampled training set and 224x224 inputs.The implementation processed five times more information than the original described implementation.
5.5 Evaluation of Simulation Results
The evaluation labels consecutive driving-frame windows according to their relationship to recorded misbehaviours. Anomaly windows precede misbehaviours, while healing periods are excluded from subsequent labels to model operational recovery and reaction time.
- Window definitions: A misbehaviour is represented by m_j ∈ {0, 1}, with m_j = 1 exactly when a misbehaviour is recorded at frame x_j.The frame sequence is denoted X = {x_1, x_2, ..., x_n}.
- Healing periods: Healing periods are misbehaviour-free frames after a misbehaviour and are ignored before later anomaly or normal windows are defined.The parameter h must be large enough for the car to be safely back on the road.
- Window definitions: An anomalous window contains consecutive misbehaviour-free frames followed by a reaction period before a misbehaviour.The reaction period consists of r misbehaviour-free frames preceding the misbehaviour and excluding healing periods.
- Window definitions: A normal window contains b consecutive misbehaviour-free frames followed by another anomaly or normal window, or remains outside healing periods.The formal rules exclude windows intersecting any healing period.
- Alarm outcomes: An alarm during an anomalous window is a true positive only if it arrives early enough to predict the misbehaviour and trigger self-healing.A missed alarm is a false negative, while an alarm outside the required anomalous-window timing is treated as a false alarm.
- Metrics: The evaluation prioritizes high TPR and low FPR, and also reports F1-score, AUC-ROC, and AUC-PRC.TPR is recall, FPR is the false positive rate, and F1 combines precision and recall.
5.6 Results
SelfOracle achieves strong misbehaviour-prediction performance across configurations, with VAE and SAE performing best on thresholded metrics and LSTM and VAE on AUC metrics. Performance remains useful several seconds before misbehaviour and exceeds DeepRoad across the reported comparisons.
- Effectiveness: LSTM and VAE were the best-performing reconstructors on AUC-PRC and AUC-ROC.Nominal-condition false positive rates were generally near zero across SelfOracle variants.
- Effectiveness: 77% TPR and 11% FPR were achieved by VAE at ϵ = 0.05, while SAE achieved 72% TPR and 10% FPR.At ϵ = 0.01, VAE achieved 55% TPR and 5% FPR, while SAE achieved 59% TPR and 5% FPR.
- Prediction over time: AUC-PRC remained quite high even 6 seconds before misbehaviour, although prediction performance degraded smoothly as anticipation increased.The authors caution that the result may partly reflect tracks with continuously increasing injected anomalies.
- Comparison with DeepRoad: 77%, 72% versus 32% TPR was reported for VAE and SAE versus DeepRoad at ϵ = 0.05, with FPR of 11%, 10% versus 9%.At ϵ = 0.01, the corresponding TPR values were 55%, 59% versus 20%, with FPR of 5% versus 6%.
- Comparison with DeepRoad: SelfOracle outperformed DeepRoad in computational cost, misbehaviour-prediction accuracy, and false-alarm minimization.The comparison used AUC-PRC, AUC-ROC, TPR, and FPR across thresholds and reaction periods.
5.7 Threats to Validity
The evaluation is reproducible, but its generalizability is constrained by the limited number of self-driving systems and tracks. Internal validity also depends on custom simulator conditions and an in-house DeepRoad implementation.
- External validity: The limited number of self-driving systems and tracks threatens the generalizability of the results.The authors attempted to mitigate this by selecting popular subjects developed with real-world frameworks.
- Internal validity: Custom unexpected-condition injections in the simulator are the main internal-validity threat.The authors state that this choice was necessary because suitable open-source simulators were unavailable.
- Internal validity: The authors’ own DeepRoad implementation is another internal-validity threat, mitigated by processing five times more information than the original implementation.All variants were compared under identical parameter settings and on the same evaluation set.
- Reproducibility: The source code, simulator, subjects, and SelfOracle results were made available for repeatable evaluation.The paper presents this availability as supporting reproducibility.
6 RELATED WORK
Prior autonomous-driving test-generation work focuses on producing challenging inputs, while SelfOracle addresses the separate problem of predicting misbehaviour and uses a precise functional oracle for evaluation.
- Adversarial Input Generation: Adversarial input generation creates scenarios that expose inconsistencies between autonomous driving systems or between original and transformed scenarios.Its main use is identifying underrepresented training scenarios, such as snowy weather, to support retraining.
- Oracle Problem: SelfOracle shares test-generation research’s challenge of validating behavior without a precise expected-behavior oracle.This challenge arises despite the different goals of test generation and misbehaviour prediction.
- Oracle Problem: Differential and metamorphic testing address the oracle problem by comparing multiple DNNs or behavior before and after input transformations.Verification-based approaches are also under investigation.
- Oracle Problem: This paper instead defines DNN misbehaviour precisely, providing an accurate functional oracle without differential testing, metamorphic testing, or verification.
7 CONCLUSIONS AND FUTURE WORK
SelfOracle anticipates potentially safety-critical autonomous-driving misbehaviours by estimating DNN confidence in unexpected contexts. The authors identify future directions including new confidence metrics, broader misbehaviour types, and confidence-guided self-healing.
- Conclusions: SelfOracle anticipated many potentially safety-critical misbehaviours, including out-of-bound episodes and collisions, several seconds in advance with a low false alarm rate.It outperformed DeepRoad’s input validator.
- Future Work: Future work includes white-box DNN-confidence metrics, prediction of additional misbehaviours, and confidence-guided self-healing in the simulator.One example of an additional target is the derivative of the steering angle.
- Future Work: A labeled crash dataset and simulation environment could support further online prediction and self-healing approaches for autonomous driving systems.