PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Aware Reinforcement Learning for Collision Avoidance

Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, Sergey Levine

arXiv:1702.01182v1cs.LGcs.RO

TL;DR

Robots using reinforcement learning must experience collisions to learn avoidance, but high-speed training collisions can damage them. The paper introduces uncertainty-aware model-based learning that predicts collision probability and uncertainty, using these estimates to select low-speed behavior when unfamiliar and higher speeds when confident. Experiments show fewer dangerous training collisions across simulated and real-world quadrotor tasks and a real-world RC car task.

  • Problem

    Robots learning collision avoidance in unknown environments need collision experience, but high-speed collisions during training can damage the robot.

  • Method

    The method learns a collision predictor from observations and control sequences, estimates uncertainty with bootstrapped neural networks and dropout, and uses an uncertainty-dependent speed cost.

  • Results

    The method experiences substantially fewer dangerous collisions during training and is safer than methods without uncertainty estimates across simulated and real-world quadrotor and real-world RC car tasks.

  • Takeaways & Limitations

    The risk-averse cost makes the robot cautious in high-uncertainty regions and faster where collision predictions are confident, without manually engineered safety controllers or fail-safe mechanisms.

Abstract

from arXiv · show

Reinforcement learning can enable complex, adaptive behavior to be learned automatically for autonomous robotic platforms. However, practical deployment of reinforcement learning methods must contend with the fact that the training process itself can be unsafe for the robot. In this paper, we consider the specific case of a mobile robot learning to navigate an a priori unknown environment while avoiding collisions. In order to learn collision avoidance, the robot must experience collisions at training time. However, high-speed collisions, even at training time, could damage the robot. A successful learning method must therefore proceed cautiously, experiencing only low-speed collisions until it gains confidence. To this end, we present an uncertainty-aware model-based learning algorithm that estimates the probability of collision together with a statistical estimate of uncertainty. By formulating an uncertainty-dependent cost function, we show that the algorithm naturally chooses to proceed cautiously in unfamiliar environments, and increases the velocity of the robot in settings where it has high confidence. Our predictive model is based on bootstrapped neural networks using dropout, allowing it to process raw sensory inputs from high-bandwidth sensors such as cameras. Our experimental evaluation demonstrates that our method effectively minimizes dangerous collisions at training time in an obstacle avoidance task for a simulated and real-world quadrotor, and a real-world RC car. Videos of the experiments can be found at https://sites.google.com/site/probcoll.

I. INTRODUCTION

The paper addresses the safety dilemma of reinforcement-learning robots that must experience collisions to learn avoidance, while high-speed failures can be catastrophic. It proposes uncertainty-aware model-based learning that uses cautious low-speed exploration and becomes faster as confidence improves.

  • Reinforcement learning can fail during training, while methods assuming complete state and environment knowledge restrict real-world robotic deployment.
  • Robots learning collision avoidance in unknown environments must experience collisions, creating a tension between acquiring experience and avoiding catastrophic failures.
  • Uncertainty-aware interaction can help a robot avoid catastrophic failures in novel scenarios while increasing the diversity of its training distribution.
  • The proposed method learns a collision prediction model and uses uncertainty estimates with a speed-dependent cost to choose cautious behavior when predictions are uncertain.
  • The collision model predicts collision probability and uncertainty from observations and control sequences, using deep networks with bootstrapping and dropout for raw, high-dimensional sensory inputs.
  • The work adopts a model-based approach because such methods are generally more sample-efficient, while positioning uncertainty-based safety as complementary to exploration-focused methods.

III. PRELIMINARIES

The preliminaries formulate unknown-environment navigation as optimal control with an obstacle-dependent collision cost. Because collisions cannot be evaluated exactly from observations alone, the robot estimates future collision probabilities from observations and planned action sequences.

  • The navigation problem uses states, actions, known dynamics, observations, and an environment representation containing potential obstacles.
  • The objective combines an obstacle-independent task cost with an obstacle-dependent collision cost that can penalize high-speed collisions more strongly.
  • With a known environment, receding-horizon model-predictive control evaluates collisions exactly and replans after executing each action.
  • In an unknown environment, collision probability can be estimated at a future time from the current state, planned action sequence, and current observation.
  • The method simplifies prediction by estimating collision probability for any timestep within the MPC horizon, an approximation found to perform equally well in practice for relatively short horizons.
  • The collision probability is parameterized by a neural network that outputs the parameter of a Bernoulli random variable.

IV. UNCERTAINTY-AWARE COLLISION PREDICTION

The approach combines collision prediction with uncertainty estimation, using a risk-averse estimator and speed-dependent cost to support cautious exploration when uncertainty is high. As confidence improves, the robot can act more aggressively while learning from safe collisions.

  • The speed-dependent cost makes exploration cautious under high uncertainty and faster when the model is confident.This lets the robot learn about collisions without catastrophic failures and later act more aggressively.
  • In the quadrotor example, safe collisions and associated images train the model to relate observations and velocity commands to collision likelihood.As uncertainty decreases, the cost favors high-speed flight.
  • The collision prediction model estimates collision probability from the current state, observation, and a sequence of controls.
  • The risk-averse estimator increases conservatism as uncertainty in the stochastic model grows, using the variance of fθ.The hyperparameter λSTD controls how conservative the estimator is.

B. Velocity-Dependent Collision Cost

The collision cost combines task performance with a velocity-dependent collision penalty weighted by a risk-averse collision probability. This structure favors slow motion when collision risk or uncertainty is high and fast motion when confident no-collision predictions are available.

  • The collision cost is constructed from the risk-averse collision estimate and favors slow, cautious exploration in uncertain regions.
  • The full cost combines task cost with risk-averse collision probability multiplied by the velocity-dependent collision cost.The formulation is an approximation of the full cost.
  • A large risk-averse collision probability encourages slow movement to reduce the velocity-dependent collision penalty.This probability becomes large when the model predicts collision or has high variance.
  • A small risk-averse collision probability lets the robot prioritize task cost and move quickly.This occurs when the model predicts no collision and has low variance.

C. Neural Network Collision Prediction Model

The collision predictor uses deep neural networks to estimate collision probability from rich, high-dimensional sensory inputs. A deterministic discriminative network can be accurate near training data but cannot itself provide the uncertainty estimate required for risk-averse prediction.

  • Deep neural networks are used to estimate collision probability from high-dimensional inputs such as cameras or LIDAR.
  • A standard deterministic discriminative network applies a sigmoid to final-layer preactivations to produce collision probabilities.
  • Deterministic networks may predict accurately near training data but do not provide variance estimates for risk-averse collision prediction.

D. Estimating Uncertainty with Neural Networks

The method estimates neural-network uncertainty through bootstrapped models combined with dropout. Ensemble disagreement is used as an uncertainty signal, while repeated stochastic forward passes estimate the mean and variance needed by the risk-averse collision model.

  • Bootstrapping trains multiple models on datasets resampled with replacement, using their predictions to estimate uncertainty.
  • The training procedure resamples subsequence datasets, initializes separate networks, applies dropout during stochastic gradient training, and updates each model with SGD.
  • Bootstrapped models tend to agree in high-density data regions and disagree in low-density regions, corresponding to lower and higher uncertainty respectively.
  • Limited bootstrap counts in time- and resource-constrained robotics applications can produce inaccurate uncertainty estimates.
  • Dropout provides a computationally cheap way to improve uncertainty estimates and can be applied at test time through stochastic predictions.
  • At test time, repeated forward passes across bootstrapped networks and dropout realizations estimate E[fθ] and Var[fθ] for the risk-averse model.

E. Reinforcement Learning with Risk-Averse Collision Estimation

The algorithm forms a cost from the current uncertainty-aware collision model, uses MPC to sample trajectories, aggregates them, and retrains the model iteratively.

  • Each algorithm iteration forms the cost function using the current uncertainty-aware collision prediction model.
  • The model predictive controller samples trajectories according to the constructed cost.
  • Sampled trajectories are aggregated with previous trajectories into a dataset used to train the collision prediction model.

V. EXPERIMENTS

The experiments evaluate the uncertainty-aware collision model and model-based reinforcement-learning algorithm in simulation and real-world settings, including parameter variations and a conservative baseline.

  • The evaluation includes simulated and real-world experiments for the uncertainty-aware collision prediction model and model-based reinforcement-learning algorithm.
  • The experiments compare parameter settings and a model-based approach that estimates collision probability without explicitly accounting for uncertainty.
  • The collision model is a fully connected neural network that predicts collision probability from concatenated state, control-sequence, and observation inputs.
  • Uncertainty estimation uses bootstraps with dropout, with 50 bootstraps and dropout 0.2 in simulation versus 5 bootstraps and dropout 0.05 in real-world experiments.
  • The receding-horizon MPC planner selects among fixed action sequences by evaluating their costs and executing the first action of the minimum-cost sequence.

A. Quadrotor experiments

Quadrotor experiments examine uncertainty-aware collision avoidance in simulation and on a real Bebop 2, showing a safety–performance trade-off and progressively faster obstacle avoidance.

  • Simulation: The simulated quadrotor navigates a cylindrical obstacle using only a simulated monocular-camera observation while targeting 0.5 m/s.
  • Simulation: With λSTD = 0, task performance approaches 0.5 m/s but training includes high-speed collisions; increasing λSTD lowers collision speeds.
  • Simulation: The uncertainty-aware parameter λSTD trades safety against learning efficiency, since excessive conservatism reduces final task performance.
  • Simulation: The constant-penalty baseline either provides no safety improvement or substantially diminishes task performance, unlike the uncertainty-aware approach.
  • Real-world quadrotor: In real-world training, the Bebop 2 initially progresses slowly, collides while flying faster, and eventually flies around the obstacle at high speed.
  • Real-world quadrotor: With uncertainty enabled at λSTD = 2, the quadrotor experiences substantially fewer collisions, especially at higher speeds, but takes longer to approach the desired performance.

B. Real-world RC car experiments

The real-world RC car learned obstacle avoidance from raw camera observations without access to underlying state. Accounting for uncertainty reduced high-speed crashes while preserving comparable task performance.

  • Setup: The RC car used a 32 × 18 grayscale camera image as observation and speed plus steering angle as control, without access to underlying state.The car was equipped with a Logitech C920 webcam and limit-switch collision detectors.
  • Safety and task performance: 13% fewer crashes above 0.6 m/s occurred with uncertainty-aware control, while task performance remained comparable to the uncertainty-free baseline.The task objective was driving at 1.2 m/s and reaching the end of the track.
  • Training protocol: The experiments used 10 training iterations with five on-policy rollouts from four initial states per iteration.Each rollout ended after a collision or 10 time steps, and each experiment provided approximately 15 minutes of real-world experience.
  • Learning behavior: Training initially produced collisions with obstacles and side walls, but the car eventually learned to avoid them.The example training sequence shows improvement over successive iterations.
  • Performance: Almost 70% of final rollouts navigated without collision for both uncertainty settings, improving substantially over the initial policy.The comparison used λSTD = 0 without uncertainty and λSTD = 1 with uncertainty.

VI. DISCUSSION AND FUTURE WORK

The paper combines collision prediction with uncertainty-aware model-predictive control to generate cautious obstacle-avoidance behavior from raw sensory inputs. It reports safer behavior across simulated and real-world platforms, while identifying local-optimum and uncertainty-estimation limitations.

  • Method: The method predicts collision probability from raw sensory inputs and action sequences, then uses the predictor within model-predictive control to select actions.Deep neural networks implement the collision predictor.
  • Uncertainty-aware control: In high-uncertainty regions, a risk-averse cost function makes the robot adopt a cautious low-speed strategy without manually engineered safety controllers.The strategy increases speed only where prediction confidence is high.
  • Evaluation: The approach was safer than methods without uncertainty estimates in simulated and real-world quadrotor tasks and a real-world RC car task.The comparison spans both simulated and physical obstacle-avoidance experiments.
  • Limitations and future work: Because successful strategies are not explicitly sought beyond MPC optimization, the algorithm can become stuck in bad local optima.The paper gives suboptimal real-world quadrotor performance at λSTD = 2 as an example.
  • Limitations and future work: Performance depends strongly on uncertainty-estimate accuracy: optimism risks catastrophic failures, whereas pessimism can produce a perpetually scared and suboptimal policy.The paper identifies improved uncertainty estimators and characterization as future work.
  • Future directions: The uncertainty-aware prediction principle may extend beyond collision prediction to expected cost and other model-based reinforcement-learning settings.The paper suggests this could support safe exploration across broader robotic scenarios.

APPENDIX

Appendix experiments vary collision and uncertainty-cost weights in simulated quadrotor results and analyze a conservative baseline. Higher uncertainty weighting improves safety but can slow learning and reduce final task performance.

  • Appendix scope: The appendix varies λCOLL and λSTD to compare safety and task performance, and further analyzes the conservative baseline from Fig. 2.The comparisons use simulated quadrotor experiments.
  • Safety: Increasing λSTD leads to fewer training collisions, with remaining collisions occurring at lower speeds.Fig. 9 reports collision counts at or above each speed threshold across parameter settings.
  • Task performance: Higher λSTD slows progress toward optimal task performance, and sufficiently large values prevent the final performance from reaching optimum.Fig. 10 averages each setting over five experiments.
  • Conservative baseline: Increasing λCONST in the conservative baseline affects final task performance similarly to increasing λSTD, but does not necessarily increase safety.The baseline replaces uncertainty in Eqn. 1 with a constant.
Loading 1702.01182v1…