Source-linked AI summary
Safe Reinforcement Learning with Model Uncertainty Estimates
Björn Lütjens, Michael Everett, Jonathan P. How
TL;DR
The paper tackles overconfident neural-network predictions on unseen data in safety-critical pedestrian collision avoidance. It combines MC-Dropout and bootstrapping with Safe RL to estimate uncertainty and choose cautious actions, producing a simulation policy that is more robust to novel observations and safer than an uncertainty-unaware baseline.
Problem
Neural networks can be overconfident on unseen or far-from-distribution data, creating a safety gap for collision avoidance around pedestrians.
Method
The paper embeds MC-Dropout and bootstrapping in a Safe RL framework to estimate collision-prediction uncertainty and guide cautious navigation around dynamic obstacles.
Results
In simulation, the uncertainty-aware policy is more robust to novel observations and causes fewer collisions than an uncertainty-unaware baseline.
Takeaways & Limitations
Model uncertainty can identify novel obstacle observations and support cautious actions that improve robustness and safety in the demonstrated navigation scenarios.
Abstract
from arXiv · showhide
Many current autonomous systems are being designed with a strong reliance on black box predictions from deep neural networks (DNNs). However, DNNs tend to be overconfident in predictions on unseen data and can give unpredictable results for far-from-distribution test data. The importance of predictions that are robust to this distributional shift is evident for safety-critical applications, such as collision avoidance around pedestrians. Measures of model uncertainty can be used to identify unseen data, but the state-of-the-art extraction methods such as Bayesian neural networks are mostly intractable to compute. This paper uses MC-Dropout and Bootstrapping to give computationally tractable and parallelizable uncertainty estimates. The methods are embedded in a Safe Reinforcement Learning framework to form uncertainty-aware navigation around pedestrians. The result is a collision avoidance policy that knows what it does not know and cautiously avoids pedestrians that exhibit unseen behavior. The policy is demonstrated in simulation to be more robust to novel observations and take safer actions than an uncertainty-unaware baseline.
I. INTRODUCTION
The paper addresses Safe RL policies that may fail on unseen observations by estimating model uncertainty and using it to choose safer actions around dynamic obstacles.
- Motivation: DNN-based RL policies can be overconfident on out-of-distribution data, making incorrect predictions hazardous in safety-critical collision avoidance.The paper highlights pedestrian and vehicle navigation as settings where confident errors can lead to fatal failure.
- Motivation: Policies trained on collaborative pedestrians may fail to generalize to uncollaborative behavior and other far-from-training observations.The paper frames simulation-to-real-world generalization and far-from-training behavior as central challenges.
- Approach: The proposed Safe RL framework represents novel observations through model uncertainty and avoids high-uncertainty regions cautiously.The framework is designed to identify novel pedestrian observations and respond with safer actions.
- Approach: MC-Dropout and bootstrapping provide parallelizable, computationally feasible uncertainty estimates without significantly restructuring the neural network.These methods address the computational and architectural burdens associated with Bayesian uncertainty extraction.
- Contributions: The work contributes an uncertainty-aware dynamic-obstacle algorithm, an extension to more complex environments, exploration-aiding methods, and a simulation demonstration.Its stated contributions include cautious avoidance that is safer than an uncertainty-unaware baseline.
II. RELATED WORK
Related work distinguishes model uncertainty from environment uncertainty and surveys alternative Safe RL strategies for detecting or managing risk and novelty.
- Safe RL and risk sensitivity: Safe RL research has used external novelty detectors, internal model modifications, and risk-sensitive methods to identify uncertainty or avoid costly events.These approaches target either environment uncertainty or model uncertainty in different settings.
- Uncertainty concepts: Model uncertainty concerns how well a model fits possible observations, whereas environment uncertainty reflects ambiguity from imperfect or partial observations.Model uncertainty is associated with sparse training data or test data far from training data; environment uncertainty can remain even with extensive training.
- Uncertainty concepts: The paper focuses on model uncertainty because it can indicate where training data is sparse and predictions may fail to generalize.This focus differs from risk-sensitive approaches centered on known low-probability, high-cost events.
C. Measures of model uncertainty
The paper uses bootstrapped ensembles and test-time dropout to obtain prediction distributions, then incorporates their uncertainty into motion-primitive selection for dynamic collision avoidance.
- Uncertainty estimation: Bootstrapped networks agree on common data and show larger prediction variance in uncommon-data regions, providing an approximate uncertainty signal.Dropout activated during testing offers a related interpretation and approximates Bayesian inference in deep Gaussian processes.
- Uncertainty estimation: An ensemble of LSTM networks predicts collision probabilities for candidate motion primitives, while repeated dropout samples produce their sample means and variances.The system architecture evaluates uncertainty for each primitive at every time step.
- Control integration: The approach extends uncertainty-aware RL from static-obstacle settings toward dynamic obstacles and uses uncertainty information to guide cautious avoidance.The proposed algorithm targets novel dynamic-obstacle scenarios rather than only known risk events.
- Control integration: The controller selects the minimal-cost motion primitive using collision-probability estimates and time-to-goal predictions, then executes it for one step.This selection is repeated as the agent observes the environment over time.
A. Collision Prediction Network
The collision prediction network uses recurrent LSTM ensembles to estimate collision probabilities for discrete motion primitives from short histories of observations and actions.
- Prediction model: An LSTM ensemble predicts the collision probability associated with each evaluated motion primitive.Each forward pass produces one collision-probability estimate for the primitive.
- Prediction inputs: The prediction input includes recent observation history, the current observation, past actions, and the evaluated motion primitive.The observation contains pedestrian position, velocity, radius, and the agent’s relative goal position.
- Prediction inputs: The agent evaluates 11 discrete motion primitives of length h = 1, each described by a heading angle, and executes the selected primitive for one time step.The network is queried again after each executed step.
- Model choice: LSTMs are selected because prior pedestrian-path prediction work found them effective at capturing hidden temporal intentions.The paper requires accurate predictions from the first time a pedestrian enters the robot’s field of view.
B. Uncertainty Estimates with MC-Dropout and Bootstrap-
The paper combines MC-Dropout and bootstrapping to estimate model uncertainty in parallel, then uses those estimates in an MPC that selects safer motion primitives.
- Bootstrapping trains multiple networks on overlapping resampled datasets, causing predictions to differ for uncommon observations and agree for common observations.The networks can be trained and tested in parallel.
- MC-Dropout samples multiple stochastic forward passes per ensemble network to obtain a distribution over collision predictions.The method uses dropout probability p = 0.7 and collects N = ndnb forward passes.
- The framework computes the sample mean and variance from collision predictions across networks and dropout masks.
- The MPC selects the safest motion primitive by minimizing a joint cost that includes collision probability, uncertainty, and estimated time to goal.Each cost term has its own weighting factor λ.
- Regional uncertainty guides avoidance of uncertain observations instead of simply reducing velocity, because stopping or slowing is not always safe.The approach focuses on uncertain observation regions in the ground plane.
D. Adaptive variance
The training procedure increases the penalty on highly uncertain actions over time, balancing exploration early in training with uncertainty-averse execution.
- An overly uncertainty-averse model could discourage exploration and rarely find the optimal policy.
- The approach increases λv over time so the policy explores uncertain directions early and becomes uncertainty-averse during execution.The stated schedule linearly increases λv in [−50000, 200].
- At the end of each episode, the environment returns a binary collision label and stores the episode’s observation-action history in an experience dataset.
IV. RESULTS
The results show that uncertainty estimates regionally identify novel obstacle observations and support safer avoidance than an uncertainty-unaware baseline across novel scenarios.
- IV. RESULTS: The proposed model detects novel obstacle observations regionally and causes fewer collisions than an uncertainty-unaware baseline.
- A. Regional novelty detection in 1D: In the 1D experiment, MC-Dropout and bootstrapping are trained to predict collision labels from agent and obstacle headings, with training obstacles only on the right.
- A. Regional novelty detection in 1D: For unseen obstacles on the left, predictions fail to generalize, but actions toward the obstacle receive high model uncertainty.The high uncertainty indicates that the predictions should not be trusted.
- A. Regional novelty detection in 1D: On the training distribution, collision and no-collision labels are predicted accurately with low uncertainty.
- A. Regional novelty detection in 1D: Known obstacles are avoided confidently close, whereas novel obstacle appearances trigger cautious avoidance.
- A. Regional novelty detection in 1D: The MPC prefers a conservative action that is certainly safe over an uncertain action falsely predicted to be safe.
B. Novelty detection in multi-dimensional observations
The multi-dimensional experiments test dynamic-obstacle avoidance under manipulated observations and show regional uncertainty guiding low-uncertainty actions.
- B. Novelty detection in multi-dimensional observations: The experiments evaluate regional uncertainty identification and safer action selection with multi-dimensional observations.
- B. Novelty detection in multi-dimensional observations: A one-layer 16-unit LSTM is trained in simulation with one agent and one dynamic obstacle following collaborative, non-cooperative, or static policies.Training uses obstacles following an RVO policy.
- B. Novelty detection in multi-dimensional observations: With newly added observation noise, actions toward the obstacle have higher uncertainty than actions into free space, guiding the agent toward low-uncertainty avoidance actions.
- B. Novelty detection in multi-dimensional observations: The ensemble uses five networks with twenty stochastic MC-Dropout samples each, producing one hundred samples per step in an average of 32ms.The computation could be further decreased by parallelizing it on GPUs.
- B. Novelty detection in multi-dimensional observations: Test observations are manipulated with Gaussian sensor noise, 20% random dropping, or masking of obstacle position or velocity, none of which occurs during training.
2) Regional novelty detection:
The proposed model identifies novel observations regionally through increased predictive uncertainty and uses that signal to choose cautious obstacle-avoidance actions. Across noisy and other novel scenarios, uncertainty is higher than on the training distribution, while uncertainty-aware control avoids obstacles more successfully.
- Regional novelty detection: Higher uncertainty marks collision predictions for actions directed toward a noisy obstacle than for actions into free space.The MPC uses this regional difference to steer away from the obstacle without collision.
- Cautious avoidance: The uncertainty-aware agent avoids a noisy obstacle successfully, whereas the uncertainty-unaware agent collides after five time steps.Both agents have the same speed and radius in the illustrated comparison.
- Novel scenario identification: Overall model uncertainty is high in every tested novel scenario, including the scenario with added observation noise.The measured uncertainty is based on the variance of collision predictions across actions at one time step.
- Cautious avoidance: In novel cases, the uncertainty-aware model causes fewer collisions than the uncertainty-unaware baseline by preferring more cautious actions near obstacles.The regional increase of uncertainty in the obstacle’s direction guides this behavior.
- Novel scenario identification: Uncertainty is relatively low on the training distribution and higher in other scenarios, with relative magnitude indicating observation novelty compared with training.Values were averaged over 20 sessions, 50 episodes, and all time steps until episode termination.
4) Fewer collisions in novel scenarios:
The uncertainty-aware policy is more robust than the uncertainty-unaware baseline in several novel scenarios, while both perform similarly on seen training samples. Increasing the variance penalty also improves exploration during challenging training cases, although the demonstrated scenarios do not exhaust possible real-world novelty.
- Fewer collisions in novel scenarios: In added-noise, masked-position, and masked-velocity tests, the proposed model causes fewer collisions than the uncertainty-unaware baseline.Both models perform similarly on samples from the training distribution; with dropped observations, collisions are similar, but the baseline takes longer to reach the goal.
- Fewer collisions in novel scenarios: In each of four novel test scenarios, collision-prediction uncertainty is higher than on samples from the seen training distribution.Table I compares uncertainty across the training distribution and four novel scenarios.
- Generalization to other novel scenarios: The results suggest that model uncertainty identifies novel observations and that MPC selects actions with extra buffer space around them.The paper gives uncommon pedestrian behavior as an example of potentially novel real-world behavior.
- Using uncertainty to escape local minima: 80% of runs with a constant variance penalty end in collision in the challenging training case.An increasing variance penalty encourages early exploration and converges to a lower minimum, while both methods perform similarly in the easy case.
V. DISCUSSION AND FUTURE WORK
The uncertainty-aware framework showed safer, more robust behavior in novel scenarios, but one test case exposed limits in its uncertainty estimates. Increasing the uncertainty penalty also helped avoid poorer local minima in challenging training cases.
- Model uncertainty limitations: In testing on uncollaborative agents, uncertainty values did not significantly increase despite the agents being novel relative to training.The agents were trained collaboratively but evaluated on uncollaborative behavior.
- Escaping local minima: Increasing the penalty on uncertain actions enabled exploration early in training and found a better minimum in a challenging avoidance case.In an easier case, both constant- and increasing-penalty policies reached near-zero collisions.
- Model uncertainty limitations: This result may reflect either generalization to the uncollaborative behavior or overconfidence in dropout-based uncertainty estimates.The authors identify both explanations and motivate future uncertainty estimates with stronger guarantees.
- Conclusion: The uncertainty-aware framework was more robust to novelties and safer than an uncertainty-unaware baseline.Its cautious behavior supported safer dynamic-obstacle avoidance in novel scenarios.