PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning

Jianyu Chen, Shengbo Eben Li, Masayoshi Tomizuka

arXiv:2001.08726v3cs.ROcs.CVcs.LG

TL;DR

Existing end-to-end autonomous-driving methods often lack interpretability, while learning-based driving must handle complex urban environments. The paper jointly learns a sequential latent environment model with reinforcement learning, decodes latent states into semantic bird’s-eye masks, and reports significantly better performance than baseline RL methods in crowded urban CARLA scenarios. The framework explains environmental understanding, but not model-free decision-making.

  • Problem

    Existing end-to-end autonomous-driving policies often lack interpretability, which is important for safety-critical autonomous driving.

  • Method

    The method jointly learns a sequential latent environment model and maximum-entropy reinforcement-learning policy, decoding latent states into semantic bird’s-eye masks.

  • Results

    In CARLA, the method significantly outperforms prior reinforcement-learning baselines in crowded urban scenarios.

  • Takeaways & Limitations

    The framework provides interpretable explanations of how the learned policy understands driving situations through reconstructed semantic masks.

  • Takeaways & Limitations

    Because the driving policy is model-free, the framework explains environmental understanding but does not provide intuition about how decisions are made.

Abstract

from arXiv · show

Unlike popular modularized framework, end-to-end autonomous driving seeks to solve the perception, decision and control problems in an integrated way, which can be more adapting to new scenarios and easier to generalize at scale. However, existing end-to-end approaches are often lack of interpretability, and can only deal with simple driving tasks like lane keeping. In this paper, we propose an interpretable deep reinforcement learning method for end-to-end autonomous driving, which is able to handle complex urban scenarios. A sequential latent environment model is introduced and learned jointly with the reinforcement learning process. With this latent model, a semantic birdeye mask can be generated, which is enforced to connect with a certain intermediate property in today's modularized framework for the purpose of explaining the behaviors of learned policy. The latent space also significantly reduces the sample complexity of reinforcement learning. Comparison tests with a simulated autonomous car in CARLA show that the performance of our method in urban scenarios with crowded surrounding vehicles dominates many baselines including DQN, DDPG, TD3 and SAC. Moreover, through masked outputs, the learned policy is able to provide a better explanation of how the car reasons about the driving environment. The codes and videos of this work are available at our github repo and project website.

I. INTRODUCTION

The paper motivates replacing hand-engineered modular driving systems with an end-to-end policy that remains interpretable in complex urban scenarios. It combines sequential latent environment modeling with reinforcement learning to compress sensor histories, decode semantic bird’s-eye explanations, and improve driving performance.

  • I. INTRODUCTION: Modular autonomous-driving systems rely on extensive hand-engineered components and become difficult to scale and maintain in urban scenarios.Their perception, localization, prediction, decision, and control modules are heavily entangled with human heuristics.
  • I. INTRODUCTION: End-to-end approaches learn driving policies with less hand-engineered involvement, but existing methods are criticized for weak interpretability and limited task complexity.Imitation learning uses expert driving data, whereas reinforcement learning learns through self-exploration and reinforcement.
  • I. INTRODUCTION: The proposed method jointly learns a maximum-entropy reinforcement-learning policy and a sequential latent environment model from high-dimensional historical observations.The latent space encodes visual inputs, spatial features, road conditions, and road-user states.
  • I. INTRODUCTION: The latent state can be decoded into semantic bird’s-eye masks connected to modular properties such as localization, object detection, and behavior prediction.The mask provides an explanation of how the learned policy represents the driving environment.
  • I. INTRODUCTION: In CARLA experiments, the proposed method significantly outperforms prior methods in crowded urban scenarios, while decoded masks illustrate how the vehicle understands driving situations.The evaluation compares learning curves based on average discounted cumulative rewards across testing episodes.
  • I. INTRODUCTION: The method uses a probabilistic graphical model to represent conditional dependencies and sequential latent states across observations, actions, and environment dynamics.The latent state summarizes historical information, while state transitions depend on the previous latent state and action.

C. PGM for Reinforcement Learning

The paper formulates maximum entropy reinforcement learning as probabilistic graphical model learning, combining reward optimization with entropy regularization and stochastic policy exploration.

  • The discount factor is incorporated by modifying the state-transition model rather than written explicitly in accumulated rewards.
  • Maximum entropy reinforcement learning adds the entropy term −logπφ(at|zt) to the reward objective.
  • MaxEnt RL uses a stochastic policy by default, so exploration is optimized jointly with reinforcement learning training.
  • In the PGM, zt denotes the state, at the action, and Ot a binary variable indicating whether the agent acts optimally.
  • The trajectory objective maximizes the probability of optimality, since higher reward corresponds to higher optimality.
  • MaxEnt RL reformulates reinforcement learning as maximizing the likelihood of optimality variables in the PGM.

A. PGM for Interpretable Urban Autonomous Driving

The urban-driving PGM combines latent state modeling, sensor reconstruction, semantic-mask generation, and policy learning into an interpretable end-to-end agent.

  • A. PGM for Interpretable Urban Autonomous Driving: Urban autonomous driving requires perception of the current and future situation alongside planning and control of the vehicle’s actions.
  • A. PGM for Interpretable Urban Autonomous Driving: The driving PGM represents sensor inputs, latent states, actions, and optimality variables while allowing multiple observation sources.
  • A. PGM for Interpretable Urban Autonomous Driving: The mask mt carries human-understandable semantic meaning and is decoded from the latent state at test time to show the system’s environmental interpretation.
  • A. PGM for Interpretable Urban Autonomous Driving: The inference model uses historical sensor inputs and actions to infer the current latent state, while latent dynamics predict future states.
  • A. PGM for Interpretable Urban Autonomous Driving: The generative models reconstruct raw sensor inputs and generate semantic masks from the latent state.
  • A. PGM for Interpretable Urban Autonomous Driving: The agent receives multimodal sensor inputs, outputs control commands, and simultaneously generates a semantic mask for interpretation.
  • B. Sensor Inputs and Mask: Camera inputs provide front-view road information, while lidar images provide rendered spatial information and route waypoints.
  • B. Sensor Inputs and Mask: The bird-view semantic mask represents road conditions and objects as a 64×64×3 tensor, including drivable areas and lane markings.

2) Routing:

The section describes trajectory-based joint learning of an environment model and driving policy using latent variables, variational inference, and an evidence lower bound.

  • 2) Routing:: Trajectories comprise sensor inputs, masks, actions, and rewards, forming the data structure for joint model and policy learning.
  • 2) Routing:: Variational inference introduces q(z⃗|x⃗,a⃗) to approximate latent states conditioned on historical sensor inputs and actions.
  • 2) Routing:: The trajectory distribution combines latent-state inference with policy execution and latent-state transitions.
  • 2) Routing:: The evidence lower bound combines likelihood terms for sensor inputs, masks, latent dynamics, and action priors.

B. Environment Model Learning

The environment model is learned by maximizing an ELBO that reconstructs observations and masks while regularizing latent-state inference against learned dynamics.

  • B. Environment Model Learning: The environment-model objective is the first part of the joint ELBO and is optimized separately from the driving-policy objective.
  • B. Environment Model Learning: The ELBO includes reconstruction terms for sensor inputs and masks, latent transition likelihood, and a variational posterior penalty.
  • B. Environment Model Learning: The generative model is unfolded through time using a latent-state transition function, while posterior inference conditions on successive latent states, observations, and actions.
  • B. Environment Model Learning: The posterior is approximated locally with q(z1|x1) and q(zt+1|zt,xt+1,at); exact trajectory conditioning would use bidirectional recurrent neural networks.

C. Driving Policy Learning

The driving policy is learned as a maximum-entropy reinforcement-learning problem using soft actor-critic, jointly with the latent environment model.

  • C. Driving Policy Learning: The policy-learning objective is the second part of the joint ELBO, and the uniform action prior removes the log p(at) term.
  • C. Driving Policy Learning: Soft actor-critic solves this maximum-entropy reinforcement-learning problem through function-approximation versions of soft policy evaluation and improvement.
  • C. Driving Policy Learning: The implementation optimizes a Bellman-residual loss for soft policy evaluation and a KL-divergence loss for soft policy improvement.
  • C. Driving Policy Learning: Joint training uses stochastic gradient descent to maximize the model-learning ELBO term while minimizing JQ and Jπ.

A. Simulation Setup

The method is trained and evaluated in CARLA, using a multi-agent urban simulation with randomized starts, specified rewards, and neural networks for latent modeling and control.

  • A. Simulation Setup: CARLA supplies a high-definition simulation of driving environments, vehicle dynamics, camera RGB images, and lidar point clouds.
  • A. Simulation Setup: The training town covers 400m×400m with about 6km of roads and 100 autonomous vehicles across intersections and roundabouts.
  • A. Simulation Setup: The reward combines collision, longitudinal-speed, overspeed, lane-departure, steering, and lateral-acceleration terms, with a constant discouraging stationary behavior.
  • A. Simulation Setup: The model includes observation and mask decoders, latent dynamics, filtering networks, a Q network, and a policy network, with z1t ∈ R256 and 64×64×3 inputs and masks.
  • A. Simulation Setup: Episodes use randomized feasible ego and vehicle positions, a maximum length of 500, 0.1-second frame intervals, and frame skipping of 4.

VII. EVALUATION RESULTS

The evaluation compares the proposed method and variants with baseline reinforcement-learning algorithms using recurrent encoders and learning curves of average returns. Evaluation uses stochastic policies, repeated episodes, randomized vehicle locations, and no frame skipping.

  • Evaluation protocol: Evaluation averages returns over 10 episodes at each step, with vehicles randomly relocated for every new episode and no frame skip.
  • Compared methods: The study compares the proposed method with two variants, including a version without mask supervision and a mask-input variant assuming accurate perception and localization.
  • Compared methods: The baselines are DQN, DDPG, TD3, and SAC, evaluated with the same recurrent LSTM encoding networks but without decoders.
  • Learning curves: Figure 8 plots average returns against environment steps, averaging five trials of ten episodes with shaded standard deviation.

C. Evaluation Results

In CARLA evaluation, the proposed latent method substantially outperforms the tested baseline RL algorithms while producing semantic masks that explain environmental understanding. The masks support localization, road-condition decoding, and surrounding-vehicle detection, but failures remain and decision-making itself is not explained.

  • C. Evaluation Results: All variants of the proposed method significantly outperform the baseline RL algorithms in average-return learning curves.The paper states that the baselines almost do not work at all under this comparison.
  • VIII. INTERPRETABILITY: The latent model decodes semantic masks, whereas the baseline RL algorithms lack latent spaces and cannot provide equivalent interpretable masks.
  • A. Detection & Localization Functionality: The method obtains ego-vehicle localization and road-condition information during end-to-end RL training without storing HD maps or manually designing localization algorithms.
  • A. Detection & Localization Functionality: The environment model detects surrounding vehicles by fusing camera and lidar observations.
  • A. Detection & Localization Functionality: Reconstructed bird-view masks accurately locate the ego car, decode drivable areas and road markings, and detect surrounding vehicles in sampled frames.Only camera and lidar images are observed; the ground-truth bird-view image is shown for comparison.
  • B. Quantified Evaluation: The mask interpretation metric is average pixel difference between decoded and ground-truth masks; over 10^4 simulation frames, the reported value is e = 0.032.Masks use W = H = 64 and C = 3, with RGB values scaled to [0, 1].
  • C. Failure Cases Interpretation: Failure cases include collisions when vehicles are missed at low sensor resolution or mistakenly localized in the ego lane under rare training situations.
  • IX. CONCLUSIONS: The framework explains environmental understanding but not decision-making because its driving policy is model-free.The paper identifies model-based methods as future work for improving performance and interpretability.
Loading 2001.08726v3…