PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-free Deep Reinforcement Learning for Urban Autonomous Driving

Jianyu Chen, Bodi Yuan, Masayoshi Tomizuka

arXiv:1904.09503v2cs.LGcs.AIcs.CVcs.RO

TL;DR

Urban autonomous driving needs policies that handle complex geometry and multi-agent interactions, while existing manually designed and deep-RL approaches have important limitations in complex urban scenarios. The paper proposes a bird-view and visual-encoding framework with model-free deep RL, and evaluates it in a dense-vehicle roundabout where it performs significantly better than the baseline.

Abstract

from arXiv · show

Urban autonomous driving decision making is challenging due to complex road geometry and multi-agent interactions. Current decision making methods are mostly manually designing the driving policy, which might result in sub-optimal solutions and is expensive to develop, generalize and maintain at scale. On the other hand, with reinforcement learning (RL), a policy can be learned and improved automatically without any manual designs. However, current RL methods generally do not work well on complex urban scenarios. In this paper, we propose a framework to enable model-free deep reinforcement learning in challenging urban autonomous driving scenarios. We design a specific input representation and use visual encoding to capture the low-dimensional latent states. Several state-of-the-art model-free deep RL algorithms are implemented into our framework, with several tricks to improve their performance. We evaluate our method in a challenging roundabout task with dense surrounding vehicles in a high-definition driving simulator. The result shows that our method can solve the task well and is significantly better than the baseline.

I. INTRODUCTION

Urban autonomous driving requires decision making that handles complex geometry, dense interactions, and routing while avoiding the limitations of manually designed and imitation-learned policies. The paper addresses limited success of deep RL in complex urban scenarios with a tailored representation and model-free RL framework.

  • Urban driving decision making must handle complex road geometry, multi-agent interactions, and high-level routing commands.Source: §I · p. 1
  • Manually designed policies can introduce accuracy bias in interactive environments and require redesign across scenarios and tasks.Source: §I · p. 1
  • Imitation learning requires costly expert data, covers only demonstrated skills, and cannot exceed human-level supervision.Source: §I · p. 1
  • Reinforcement learning can learn policies without expert data, simulate dangerous cases, and potentially exceed human-expert performance.Source: §I · p. 1
  • Deep RL has had limited success in complex urban driving because raw front-view inputs increase sample complexity and basic algorithms cannot solve some complex problems.Source: §I · p. 1
  • The proposed framework uses a specific input representation, visual encoding, several model-free deep RL algorithms, and training modifications to learn roundabout driving robustly.Source: §I · p. 1

III. PROBLEM FORMULATION

The framework transforms perception and routing information into a bird-view image, encodes it into low-dimensional states, and uses reinforcement learning to generate vehicle controls. This representation simplifies complex urban spatial information for learning-based policy training.

III. PROBLEM FORMULATION

  • The agent receives perception and routing information, forms a bird-view image, encodes it into low-dimensional latent states, and applies reinforcement learning.Source: §III · p. 2, Figure 1 · p. 2
  • The encoded-state policy generates control commands such as acceleration and steering angle.Source: §III · p. 2

A. Input Representation

  • Raw sensor inputs contain high-dimensional appearance, weather, lighting, and object information that complicate generalization.Source: A. Input Representation · p. 2
  • Perception can extract object positions, headings, velocities, lane features, and routing points, but these features are difficult to organize into a fixed-shape neural-network tensor.Source: A. Input Representation · p. 2
  • The framework converts perception outputs and routing information into a bird-view representation that simplifies visual and spatial information for policy input.Source: A. Input Representation · p. 2
  • The map component renders drivable roads as gray polygons and undrivable areas as black.Source: A. Input Representation · p. 2
  • The bird-view observation combines map, routing, historical detected objects, and historical ego states.Source: Figure 2 · p. 3

2) Routing:

The framework represents routing and temporal vehicle information in an aligned bird-view image, then uses visual encoding to obtain a lower-dimensional state for reinforcement learning. The encoding is evaluated by comparing original and reconstructed observations.

2) Routing:

B. Latent State Encoding

C. Reinforcement Learning

Reinforcement learning seeks an optimal policy that maximizes expected future rewards and can use model-free deep algorithms to learn from environment transitions. Q-learning and DQN estimate action values, while DDQN reduces overoptimistic estimates.

C. Reinforcement Learning

  • Reinforcement learning finds a policy that maximizes expected future total rewards from states, actions, and rewards.The resulting policy maps the current state to a control command for the vehicle.Source: C. Reinforcement Learning · p. 3
  • Three state-of-the-art model-free deep reinforcement learning algorithms are applied to learn the driving policy.Source: §IV · p. 3
  • Q-learning estimates Qπ(s, a), the expected future total reward after taking action a in state s and following policy π.The optimal policy selects the highest-valued action, and temporal-difference learning can estimate optimal Q values.Source: A. Double Deep Q-Network (DDQN) · p. 3
  • DQN parameterizes Q values with a neural network, stores transitions in a replay buffer, and samples minibatches for learning.It uses separate online and target Q networks, with target-network parameters copied every τ steps.Source: A. Double Deep Q-Network (DDQN) · pp. 3–4
  • Double DQN separates action selection from value estimation to reduce DQN’s overoptimistic Q-value estimates.Source: A. Double Deep Q-Network (DDQN) · p. 4

B. Twin Delayed Deep Deterministic Policy Gradient (TD3)

TD3 extends deterministic actor-critic learning to continuous control with replay-based Q and policy updates. It addresses value overestimation using twin Q networks, minimum-target selection, and delayed target updates.

B. Twin Delayed Deep Deterministic Policy Gradient (TD3)

C. Soft Actor Critic (SAC)

SAC uses maximum-entropy reinforcement learning to optimize both expected rewards and policy entropy. Its soft value functions, target networks, and stochastic policy support the corresponding learning updates.

C. Soft Actor Critic (SAC)

A. Simulation Environment and scenario

The method is trained and evaluated in CARLA’s busy central-roundabout scenario, where the vehicle must reach a desired exit amid dense traffic. Bird-view images are pretrained with a VAE and encoded into fixed visual features for reinforcement learning.

A. Simulation Environment and scenario

3) Soft Actor Critic (SAC):

The framework combines a five-term reward design with frame skipping to make SAC training effective while preserving maneuverability.

3) Soft Actor Critic (SAC):

  • SAC uses Q, value, and stochastic policy networks, with the policy separately representing action means and variances.The networks use visual encoding followed by dense layers and are optimized with Adam at 3 × 10−4.Source: 3) Soft Actor Critic (SAC) · p. 5

1) Rewards Design:

  • The five-term reward combines forward progress, steering smoothness, collision avoidance, lane keeping, and a stopping penalty.Forward speed is rewarded but excessive speed is penalized; collisions and lane departures receive negative rewards.Source: 1) Rewards Design · p. 5

3) Soft Actor Critic (SAC):

  • Frame skipping keeps each action unchanged for k consecutive frames, reducing training complexity by effectively reducing search depth by k.The experiments use k = 4.Source: 2) Frame Skip · p. 5
  • Frame skip must remain moderate because large k values can remove necessary actions and prevent successful turns.With k = 10, the ego vehicle had difficulty turning successfully.Source: 2) Frame Skip · p. 5

3) Exploration Strategies:

The implementations tailor exploration to each model-free RL algorithm and evaluate them using returns and roundabout success under sparse and dense traffic.

3) Exploration Strategies:

  • DDQN samples exploratory actions according to their Q values rather than uniformly, favoring actions with larger expected future rewards.The strategy is intended to make exploration more efficient by incorporating expected future reward information.Source: 3) Exploration Strategies · pp. 5–6
  • TD3 uses zero-mean Gaussian action noise whose variance changes with training time, path progress, and a periodic coefficient.The acceleration and steering noise scales are δ = 0.5 and 0.1, respectively.Source: 3) Exploration Strategies · p. 6
  • SAC needs no separate exploration mechanism because exploration is included in its learned stochastic policy.Its policy adapts exploration during training.Source: 3) Exploration Strategies · p. 6
  • Evaluation measures average return and success rates for entering and passing through the roundabout, with and without surrounding vehicles.The no-vehicle experiment uses front-view DDPG as the baseline under matched implementation settings.Source: §VI · p. 6, A. Roundabout without Vehicles · p. 6

A. Roundabout without Vehicles

  • In the no-vehicle setting, SAC performs best and reaches the goal, while the front-view DDPG baseline remains at low returns in a right-turning local optimum.Training was stopped at 140 epochs after SAC could stably reach the final goal point.Source: Figure 5 · p. 6, A. Roundabout without Vehicles · p. 6

B. Roundabout with Dense Surrounding Vehicles

With 100 surrounding vehicles, SAC remains the strongest method, frequently reaching the goal while DDQN and TD3 struggle beyond the early exits. Failures are concentrated in rear-end collisions linked to weak velocity representation.

B. Roundabout with Dense Surrounding Vehicles

  • All three proposed algorithms exceed 80% success at roundabout entry, but DDQN and TD3 success decreases dramatically as the traveled distance increases.Success is evaluated at the entrance, first exit, second exit, desired exit, and goal point.Source: B. Roundabout with Dense Surrounding… · p. 6
  • SAC reaches the goal point more often than the other methods, which the authors associate with a better balance of exploration and exploitation.The authors suggest insufficient exploration contributes to DDQN and TD3 degradation at longer distances.Source: B. Roundabout with Dense Surrounding… · p. 7
  • Almost all failures are rear-end collisions because vehicle velocity is not explicitly included in the input representation.Velocity is only implicitly represented through fading historical boxes, whose information becomes difficult to see after visual encoding.Source: B. Roundabout with Dense Surrounding… · p. 7
  • The framework solves the roundabout task better than the baseline but does not solve it perfectly.The authors identify improved learning efficiency and generalization to other scenarios as future directions.Source: §VII · p. 7

Related papers

Loading 1904.09503v2…