PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Mikael Henaff, Alfredo Canziani, Yann LeCun

arXiv:1901.02705v1cs.LGcs.AIstat.ML

TL;DR

Policies trained only on observational data may encounter states outside the training distribution, while real-world interaction can be costly or unsafe. The paper trains policies through multi-step learned-dynamics rollouts with an uncertainty penalty, and reports improved driving performance from observational data without environment interaction. Its evaluation environment keeps neighboring cars on recorded trajectories, so they do not react to the controlled car.

  • Problem

    Policies trained from observational data may induce states outside the training distribution, while real-world interaction can be unsafe or impractical in settings such as autonomous driving.

    Source: §1 · pp. 1–2

  • Method

    The policy is optimized over multi-step rollouts of learned stochastic dynamics using policy and uncertainty costs, with a modified posterior intended to preserve sensitivity to actions.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    The uncertainty-regularized approach substantially outperforms unregularized control and other observational-data methods, although its learned policies remain below human performance.

    Source: §1 · p. 2, §5.2 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    The released traffic dataset and environment offer a setting for evaluating generative models by the policies they produce.

    Source: §6 · p. 11

    In the evaluation environment, neighboring cars follow recorded trajectories and do not react to the controlled car.

    Source: §3 · p. 7

Abstract

from arXiv · show

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

1 INTRODUCTION

Learning from observational data is difficult when policy-induced states depart from the training distribution, while real-world interaction can be costly or unsafe. The paper proposes multi-step model-based policy training with an uncertainty penalty and reports improved driving-policy performance.

1 INTRODUCTION

  • Deep model-free reinforcement learning can require many environment interactions, which limits realistic applications when interactions are slow, expensive, or potentially dangerous.Building a simulator can mitigate this issue but requires human engineering effort that grows with environment complexity.Source: §1 · p. 1
  • Autonomous driving illustrates the need for observational-data methods: human trajectories can be collected from traffic cameras, but learning through real-environment interaction is not viable.Source: §1 · p. 2
  • A fixed observational dataset can leave the learned dynamics model unreliable outside its training domain, where optimistic prediction errors may lead the policy toward unsupported states.Unlike interactive training, a fixed dataset cannot correct the model with newly encountered states.Source: §1 · p. 2
  • Uncertainty-regularized model-based control substantially outperforms unregularized control and enables effective driving policies from observational data without environment interaction or expert labeling.The approach uses a learned dynamics model unrolled over multiple steps and penalizes both policy cost and model uncertainty.Source: §1 · p. 2

2 MODEL-PREDICTIVE POLICY LEARNING WITH UNCERTAINTY REGULARIZATION

The approach learns an action-conditional dynamics model from observational state-action data, then trains a policy over multi-step model rollouts using both policy and uncertainty costs. Dropout-based uncertainty penalizes predictions the model is less confident about, helping keep policy-induced trajectories near the training-data distribution.

2.1 ACTION-CONDITIONAL FORWARD MODEL

  • The action-conditional stochastic forward model predicts future states from past states, actions, and latent variables, with training using a posterior network and a prior-matching KL term.At evaluation, future predictions can be sampled by drawing latent variables from the prior.Source: §2.1 · pp. 2–3
  • The model’s z-dropout posterior mixes a learned Gaussian with the prior, encouraging predictions to rely on input states and actions rather than latent variables.The prior component makes the latent variable independent of the output with some probability.Source: §2.1 · p. 3

2.2 TRAINING A POLICY NETWORK WITH UNCERTAINTY MINIMIZATION

  • The policy objective combines the task cost with an uncertainty cost that measures whether predicted states remain close to the training-data manifold.The uncertainty cost is estimated from the dynamics model’s prediction uncertainty using dropout.Source: §2.2 · pp. 4–5
  • Multi-step rollouts let policy gradients penalize actions whose small immediate deviations lead to large future divergence from the training manifold.The policy objective is optimized over T steps, so actions receive gradients from multiple future time steps.Source: §2.2 · p. 5

2.3 RELATIONSHIP TO BAYESIAN NEURAL NETWORKS

  • The uncertainty decomposition distinguishes epistemic variation across model weights from aleatoric variation across latent-variable predictions, and MPUR penalizes the epistemic component.The input-dependent aleatoric component allows the approach to represent environments whose inherent uncertainty varies across inputs.Source: §2.3 · p. 6

3 DATASET AND PLANNING ENVIRONMENT

The study uses NGSIM I-80 traffic-camera data to learn and evaluate driving policies from observed trajectories. It represents each car’s surroundings and motion as states, and evaluates one controlled car while neighboring cars replay recorded trajectories.

3 DATASET AND PLANNING ENVIRONMENT

  • NGSIM I-80 contains 45 minutes of highway-camera recordings, 5,596 car trajectories, and approximately 2 million transitions split into training, validation, and testing sets.The split is 80% training, 10% validation, and 10% testing.Source: §3 · p. 6
  • Each car’s state combines a neighborhood image encoding lanes and nearby vehicles with a position-and-velocity vector; actions encode acceleration or braking and steering changes.The policy cost combines proximity and lane-overlap costs.Source: §3 · pp. 6–7
  • For evaluation, one car follows the learned policy while other cars replay dataset trajectories rather than reacting to it.The authors note that this may make evaluation harder because neighboring cars do not try to avoid collisions.Source: §3 · p. 7

4 RELATED WORK

Prior work studies learned dynamics, uncertainty-aware planning, and imitation learning, but differs in how uncertainty is used, the state representations addressed, and whether interaction is required. This work applies uncertainty penalties to high-dimensional predicted states while training policies from observational driving data.

4 RELATED WORK

  • Earlier model-based policy methods backpropagate through learned forward models, but the approaches discussed here do not penalize forward-model uncertainty during policy training.The work distinguishes its approach from deterministic methods and the stochastic SVG framework (Heess et al., 2015).Source: §4 · p. 7
  • Prior uncertainty-aware model-based RL uses dropout estimates to sample trajectories for action selection, while this work backpropagates uncertainty penalties through high-dimensional state predictions.The comparison includes trajectory sampling in (Chua et al., 2018) and uncertainty estimates for a drone collision detector in (Kahn et al., 2017).Source: §4 · p. 7
  • Uncertainty-penalized policy learning also appears in low-dimensional α-BNN settings, whereas this work combines variational autoencoders and dropout for high-dimensional video prediction.The cited α-BNN approach can yield better uncertainty estimates but requires multiple model-weight samples, increasing memory requirements and limiting scalability (Depeweg et al., 2018; Depeweg et al., 2016; Li & Gal, 2017).Source: §4 · p. 7
  • Over 200 time steps, stochastic video predictions sample distinct futures, whereas deterministic predictions average possible futures into blurred images.The figure shows two stochastic predictions generated from different latent-variable sequences.Source: Figure 5 · p. 8
  • Imitation learning addresses covariate shift from observational training data; expert-feedback methods can help, while this approach addresses the shift without querying an expert.The discussion cites,, and (Ross et al., 2011).Source: §4 · pp. 7–8
  • Prior driving work focused on lane following or static obstacles, while this paper studies dense moving traffic with high-dimensional neighborhood states.Related trajectory-level imitation work used Gaussian Processes in low-dimensional settings (Englert et al., 2013); another neural-dynamics approach used interactive discriminator feedback.Source: §4 · p. 8

5 EXPERIMENTS

Experiments compare deterministic and stochastic forward models and evaluate policies learned through them. The stochastic predictions remain sharp over time, and MPUR and MPER outperform the reported observational-data baselines in policy evaluation.

5.1 PREDICTION RESULTS

  • Stochastic predictions stay sharp far into the future, while deterministic predictions become increasingly blurry; different latent-variable sequences produce different future scenarios.The recorded future is only one of many possible futures in the dataset.Source: §5.1 · p. 8

5.2 POLICY EVALUATION RESULTS

  • MPUR and MPER significantly outperform the 1-step imitation learner, SVG, and VG, which do not beat the no-action baseline.Figure 6 summarizes performance measured by success rate and distance travelled.Source: Figure 6 · p. 9, §5.2 · p. 9
  • MPUR and MPER trajectories primarily stay within their lanes, while the single-step imitation learner produces divergent trajectories that turn into other lanes.The evaluation measures whether the car reaches the road-segment end without collision or driving off-road, and distance travelled before the episode ends.Source: §5.2 · pp. 9–10
  • Removing the uncertainty penalty makes deterministic MPUR equivalent to VG, while the uncertainty-penalized policy has lower uncertainty cost and performs better in the environment.VG has lower predicted policy cost but very high uncertainty cost; MPUR has higher policy-cost estimates and lower uncertainty cost.Source: §5.2 · p. 10
  • The standard-Gaussian stochastic model yields limited improvement over the deterministic model, whereas z-dropout significantly improves stochastic-model performance.The authors suggest z-dropout discourages latent variables from encoding action information, encouraging action-dependent output variation to be predicted from actions.Source: §5.2 · p. 10
  • Performance improves dramatically as rollout length increases, and the z-dropout stochastic model outperforms the standard stochastic and deterministic models over most rollout lengths.The authors connect longer rollouts with matching the policy-induced and training state distributions over longer time horizons.Source: §5.2 · p. 10

6 CONCLUSION

The proposed approach combines stochastic multi-step dynamics modeling, an uncertainty penalty, and action-responsive predictions to learn policies from observational data. On real-world traffic recordings, the learned policies outperform other observational-data approaches, though a sizeable gap from human performance remains.

6 CONCLUSION

  • The approach combines a learned stochastic dynamics model, multi-step policy-cost optimization, an uncertainty penalty, and a modified posterior that keeps the model responsive to actions.The uncertainty term penalizes divergence of policy-induced trajectories from the training manifold.Source: §6 · p. 10
  • Without uncertainty regularization, VG has lower predicted policy cost but higher uncertainty; adding the uncertainty cost yields better executed performance.Table 1 contrasts policy and uncertainty costs with and without regularization.Source: Table 1 · p. 11
  • The learned policies navigate dense traffic and outperform other approaches trained from observational data, but remain well below human performance.The authors identify narrowing this gap as a direction for further research.Source: §6 · p. 11
  • The authors present observational traffic recordings as a setting for evaluating generative models by their ability to produce policies.They also suggest optimizing for targeted behaviors such as lane changes while avoiding other cars.Source: §6 · p. 11

A DATASET AND PLANNING ENVIRONMENT

The section describes the NGSIM dataset, its driving-policy evaluation environment, and the representations used to model and score traffic states. It also outlines a Bayesian neural-network formulation that approximates the model posterior and relates training to a dropout-based variational autoencoder objective.

A DATASET AND PLANNING ENVIRONMENT

  • NGSIM I-80 provides 45 minutes of traffic recordings and 5,596 car trajectories, split into training, validation, and test sets at 80%, 10%, and 10%.The recordings cover six freeway lanes and an onramp across three 15-minute periods spanning uncongested to congested conditions.Source: A DATASET AND PLANNING ENVIRONMENT · p. 14
  • Each car’s state combines a neighborhood image with a position-and-velocity vector, while actions encode acceleration and change in angle.The image channels represent lane markings, neighboring cars, and the ego car; images are 3 × 117 × 24.Source: A DATASET AND PLANNING ENVIRONMENT · p. 15
  • The cost combines speed-dependent proximity to neighboring cars with lane overlap, using differentiable masks to score predicted images.The proximity mask grows in height with vehicle speed, requiring a longer safety distance to maintain low cost.Source: A DATASET AND PLANNING ENVIRONMENT · p. 15, Figure 8 · p. 15
  • Image representations avoid variable-sized or padded neighbor lists, permutation sensitivity, and the need to parameterize lane geometry.Preprocessing yields state-action pairs for training the prediction model, while the cost is used to optimize actions at planning time.Source: A DATASET AND PLANNING ENVIRONMENT · pp. 15–16
  • The evaluation environment controls one car while other cars replay recorded trajectories; collisions end episodes, and nonreactive traffic may make the task harder than real driving.The setup addresses the difficulty of evaluating policies amid other vehicles that accurately reflect human behavior.Source: A DATASET AND PLANNING ENVIRONMENT · p. 16

B BAYESIAN NEURAL NETWORK FORMULATION DETAILS

  • MPUR is framed as variational inference for a Bayesian neural network, approximating the intractable posterior over model weights and latent variables with a parameterized distribution.The latent-variable distribution may be a diagonal Gaussian or a mixture; the weight distribution uses dropout and a two-Gaussian mixture with one mean fixed at zero.Source: B BAYESIAN NEURAL NETWORK FORMULATION… · p. 16
  • The variational parameters comprise the dropout-based weight distribution and latent-variable distribution, and are optimized by maximizing the evidence lower bound.The derivation applies the KL chain rule and independence of weights and latent variables; it then ignores a small weight-prior term and approximates an integral using one sample.Source: B BAYESIAN NEURAL NETWORK FORMULATION… · p. 17
  • With a diagonal Gaussian output likelihood of constant variance, the resulting objective is equivalent to training a variational autoencoder with dropout.The passage describes this equivalence after multiplying by β, which does not change the maximum.Source: B BAYESIAN NEURAL NETWORK FORMULATION… · p. 17

C MODEL DETAILS

The forward model combines state and action encoders, a decoder, and a posterior network to predict future image and vector states. It samples a latent variable from the posterior and includes a predictor for proximity and lane costs.

C MODEL DETAILS

  • The forward model has four networks: state encoder, action encoder, decoder, and posterior network.The state encoder uses 20 previous states, while the posterior network also receives the ground-truth next state.Source: C MODEL DETAILS · p. 17, Figure 10 · p. 18
  • The model samples a latent variable from the posterior, combines its representation with encoded states and actions, and predicts the next image and state vector.The state and action representations are combined by addition before decoding.Source: C MODEL DETAILS · pp. 17–18
  • A separate cost predictor maps predicted states to proximity and lane costs, with sigmoid outputs constrained between 0 and 1.It consists of a three-layer encoder followed by a two-layer fully connected network.Source: C MODEL DETAILS · p. 18

D.1 FORWARD MODEL

The forward model is trained first deterministically and then stochastically, with the deterministic checkpoint serving as a baseline. Training uses Adam, 20-step unrolls, and dropout needed to compute epistemic uncertainty for policy learning.

D.1 FORWARD MODEL

  • The prediction model receives 200,000 deterministic updates followed by 200,000 stochastic updates, and the deterministic checkpoint is retained as a baseline.The model is trained with Adam at learning rate 0.0001 and minibatches of 64.Source: §D.1 · p. 19
  • Training uses minibatches of 64, learning rate 0.0001, and unrolls of 20 time steps.The optimizer is Adam.Source: §D.1 · p. 19
  • Dropout at every layer uses pdropout = 0.1 and is needed to compute the epistemic uncertainty cost when training the policy network.The model is trained in both deterministic and stochastic modes.Source: §D.1 · p. 19

D.2 POLICY MODELS

The policy models are evaluated on held-out testing trajectories and trained using a shared network architecture. MPUR adds a normalized uncertainty cost to the proximity-and-lane policy objective.

D.2 POLICY MODELS

  • Testing uses cars initialized at their dataset speeds and reports only trajectories excluded from forward-model and policy training.Source: §D.2 · p. 19
  • Each policy uses a 3-layer ConvNet over 20 frames and three fully connected layers; MPER and MPUR train through model rollouts, while imitation minimizes action negative log-likelihood.MPUR uses uncertainty weight λ = 0.5; gradients of predicted costs into states are detached to prevent excessive slowing.Source: §D.2 · p. 19
  • MPUR combines proximity and lane costs with an uncertainty cost, while prioritizing avoiding other cars over staying within lanes.The uncertainty cost is additionally minimized by MPUR.Source: §D.2 · p. 19
  • The uncertainty cost is normalized against the training-set mean and variance at each rollout length, and penalizes only estimates above the mean.Above-mean uncertainty pulls actions toward future states the forward model predicts with higher confidence.Source: §D.2 · p. 19

E ACTION SENSITIVITY EXPERIMENTS

The action-sensitivity experiment finds lower reported performance when latent variables are sampled from the posterior rather than the prior. The z-dropout variant reduces this problem somewhat.

E ACTION SENSITIVITY EXPERIMENTS

  • Posterior-sampled latent variables make the forward model less sensitive to input actions, while z-dropout reduces this problem somewhat.Source: E ACTION SENSITIVITY EXPERIMENTS · pp. 19–20
  • 168.2 Mean Distance and 72.1 Success Rate are reported for MPUR + z-dropout with prior sampling, versus 126.1 and 30.8 for posterior sampling.Without z-dropout, prior sampling reports 157.4 Mean Distance and 63.6 Success Rate, versus 86.0 and 19.2 with posterior sampling.Source: E ACTION SENSITIVITY EXPERIMENTS · pp. 19–20
Loading 1901.02705v1…