Source-linked AI summary
Model-free Deep Reinforcement Learning for Urban Autonomous Driving
Jianyu Chen, Bodi Yuan, Masayoshi Tomizuka
TL;DR
Urban autonomous driving needs policies that handle complex geometry and multi-agent interactions, while existing manually designed and deep-RL approaches have important limitations in complex urban scenarios. The paper proposes a bird-view and visual-encoding framework with model-free deep RL, and evaluates it in a dense-vehicle roundabout where it performs significantly better than the baseline.
Problem
Complex urban driving requires handling road geometry and multi-agent interactions, but current deep RL methods have limited success in such scenarios.
Method
The framework uses a specific bird-view input representation, visual encoding for low-dimensional latent states, and several model-free deep RL algorithms with performance-improving training techniques.
Results
The method solves the dense-vehicle roundabout task well and is significantly better than the baseline.
Takeaways & Limitations
Model-free deep RL can learn a driving policy for a challenging urban roundabout scenario with dense surrounding vehicles.
Takeaways & Limitations
Failures mainly involve rear-end collisions because vehicle velocity is not explicitly included in the input and is only implicitly represented through fading historical boxes.
Abstract
from arXiv · showhide
Urban autonomous driving decision making is challenging due to complex road geometry and multi-agent interactions. Current decision making methods are mostly manually designing the driving policy, which might result in sub-optimal solutions and is expensive to develop, generalize and maintain at scale. On the other hand, with reinforcement learning (RL), a policy can be learned and improved automatically without any manual designs. However, current RL methods generally do not work well on complex urban scenarios. In this paper, we propose a framework to enable model-free deep reinforcement learning in challenging urban autonomous driving scenarios. We design a specific input representation and use visual encoding to capture the low-dimensional latent states. Several state-of-the-art model-free deep RL algorithms are implemented into our framework, with several tricks to improve their performance. We evaluate our method in a challenging roundabout task with dense surrounding vehicles in a high-definition driving simulator. The result shows that our method can solve the task well and is significantly better than the baseline.
I. INTRODUCTION
Urban autonomous driving requires decision making that handles complex geometry, dense interactions, and routing while avoiding the limitations of manually designed and imitation-learned policies. The paper addresses limited success of deep RL in complex urban scenarios with a tailored representation and model-free RL framework.
- Urban driving decision making must handle complex road geometry, multi-agent interactions, and high-level routing commands.
- Manually designed policies can introduce accuracy bias in interactive environments and require redesign across scenarios and tasks.
- Imitation learning requires costly expert data, covers only demonstrated skills, and cannot exceed human-level supervision.
- Reinforcement learning can learn policies without expert data, simulate dangerous cases, and potentially exceed human-expert performance.
- Deep RL has had limited success in complex urban driving because raw front-view inputs increase sample complexity and basic algorithms cannot solve some complex problems.
- The proposed framework uses a specific input representation, visual encoding, several model-free deep RL algorithms, and training modifications to learn roundabout driving robustly.
III. PROBLEM FORMULATION
The framework transforms perception and routing information into a bird-view image, encodes it into low-dimensional states, and uses reinforcement learning to generate vehicle controls. This representation simplifies complex urban spatial information for learning-based policy training.
- III. PROBLEM FORMULATION: The agent receives perception and routing information, forms a bird-view image, encodes it into low-dimensional latent states, and applies reinforcement learning.
- III. PROBLEM FORMULATION: The encoded-state policy generates control commands such as acceleration and steering angle.
- A. Input Representation: Raw sensor inputs contain high-dimensional appearance, weather, lighting, and object information that complicate generalization.
- A. Input Representation: Perception can extract object positions, headings, velocities, lane features, and routing points, but these features are difficult to organize into a fixed-shape neural-network tensor.
- A. Input Representation: The framework converts perception outputs and routing information into a bird-view representation that simplifies visual and spatial information for policy input.
- A. Input Representation: The map component renders drivable roads as gray polygons and undrivable areas as black.
- A. Input Representation: The bird-view observation combines map, routing, historical detected objects, and historical ego states.
2) Routing:
The framework represents routing and temporal vehicle information in an aligned bird-view image, then uses visual encoding to obtain a lower-dimensional state for reinforcement learning. The encoding is evaluated by comparing original and reconstructed observations.
- 2) Routing:: Routing information is a route planner’s sequence of ego-vehicle waypoints rendered as a thick blue polyline.
- 2) Routing:: Historical detected-object states are rendered as green bounding boxes, with reduced brightness marking earlier timesteps.
- 2) Routing:: Historical ego states use red boxes whose reduced brightness likewise represents temporal history.
- 2) Routing:: The bird-view image is resized to 64 × 64, aligned with the ego view, and covers a (40m, 40m) field of view.
- B. Latent State Encoding: The VAE encodes high-dimensional observation o into a low-dimensional latent state x and decodes x back to o.
- B. Latent State Encoding: The VAE objective includes a KL-divergence term and a reconstruction term, with a multivariate Gaussian prior described in the method.
- B. Latent State Encoding: Reconstructed images are close to original images in road geometry and objects, indicating that the latent state preserves core input information.
C. Reinforcement Learning
Reinforcement learning seeks an optimal policy that maximizes expected future rewards and can use model-free deep algorithms to learn from environment transitions. Q-learning and DQN estimate action values, while DDQN reduces overoptimistic estimates.
- C. Reinforcement Learning: Reinforcement learning finds a policy that maximizes expected future total rewards from states, actions, and rewards.The resulting policy maps the current state to a control command for the vehicle.
- C. Reinforcement Learning: Three state-of-the-art model-free deep reinforcement learning algorithms are applied to learn the driving policy.
- C. Reinforcement Learning: Q-learning estimates Qπ(s, a), the expected future total reward after taking action a in state s and following policy π.The optimal policy selects the highest-valued action, and temporal-difference learning can estimate optimal Q values.
- C. Reinforcement Learning: DQN parameterizes Q values with a neural network, stores transitions in a replay buffer, and samples minibatches for learning.It uses separate online and target Q networks, with target-network parameters copied every τ steps.
- C. Reinforcement Learning: Double DQN separates action selection from value estimation to reduce DQN’s overoptimistic Q-value estimates.
B. Twin Delayed Deep Deterministic Policy Gradient (TD3)
TD3 extends deterministic actor-critic learning to continuous control with replay-based Q and policy updates. It addresses value overestimation using twin Q networks, minimum-target selection, and delayed target updates.
- B. Twin Delayed Deep Deterministic Policy Gradient (TD3): Continuous-control reinforcement learning introduces a policy network optimized with deterministic policy gradients over expected future rewards.
- B. Twin Delayed Deep Deterministic Policy Gradient (TD3): DDPG uses online and target Q networks and policy networks, stores transitions in a replay buffer, and updates from sampled minibatches.
- B. Twin Delayed Deep Deterministic Policy Gradient (TD3): TD3 adds a second Q network and uses the smaller Q estimate when setting the target Q value.
- B. Twin Delayed Deep Deterministic Policy Gradient (TD3): TD3 also delays target-network updates to address function-approximation errors, overestimated values, and suboptimal policies.
C. Soft Actor Critic (SAC)
SAC uses maximum-entropy reinforcement learning to optimize both expected rewards and policy entropy. Its soft value functions, target networks, and stochastic policy support the corresponding learning updates.
- C. Soft Actor Critic (SAC): SAC maximizes expected rewards together with policy entropy, weighted by the entropy coefficient α.
- C. Soft Actor Critic (SAC): The entropy-augmented objective defines a soft Q function and a soft Bellman backup operator.
- C. Soft Actor Critic (SAC): SAC uses soft Q, soft value, policy, and target value networks, updating Q and value networks through supervised targets.
- C. Soft Actor Critic (SAC): SAC represents its policy as a stochastic neural network driven by noise sampled from a fixed distribution.
A. Simulation Environment and scenario
The method is trained and evaluated in CARLA’s busy central-roundabout scenario, where the vehicle must reach a desired exit amid dense traffic. Bird-view images are pretrained with a VAE and encoded into fixed visual features for reinforcement learning.
- A. Simulation Environment and scenario: CARLA provides a 400m × 400m high-definition map with about 6km of roads and urban scenarios including intersections and roundabouts.
- A. Simulation Environment and scenario: The selected task requires entering a central roundabout, passing two exits, leaving through the desired exit, and reaching the final goal.
- A. Simulation Environment and scenario: 100 vehicles are initially sampled across the map, with nearly half around the roundabout, producing busy traffic with route choices and slowing behavior.
- A. Simulation Environment and scenario: A VAE is trained on 50k unlabeled bird-view images generated by a simple roundabout-driving controller.The dataset requires raw images only, not action commands or vehicle-position labels.
- A. Simulation Environment and scenario: During reinforcement learning, the pretrained VAE convolutional layers and first dense layer provide fixed visual encoding layers.
- A. Simulation Environment and scenario: DDQN uses discrete action outputs, while TD3 uses a policy network and two Q networks after the visual encoding layers.
3) Soft Actor Critic (SAC):
The framework combines a five-term reward design with frame skipping to make SAC training effective while preserving maneuverability.
- 3) Soft Actor Critic (SAC):: SAC uses Q, value, and stochastic policy networks, with the policy separately representing action means and variances.The networks use visual encoding followed by dense layers and are optimized with Adam at 3 × 10−4.
- 1) Rewards Design:: The five-term reward combines forward progress, steering smoothness, collision avoidance, lane keeping, and a stopping penalty.Forward speed is rewarded but excessive speed is penalized; collisions and lane departures receive negative rewards.
- 3) Soft Actor Critic (SAC):: Frame skipping keeps each action unchanged for k consecutive frames, reducing training complexity by effectively reducing search depth by k.The experiments use k = 4.
- 3) Soft Actor Critic (SAC):: Frame skip must remain moderate because large k values can remove necessary actions and prevent successful turns.With k = 10, the ego vehicle had difficulty turning successfully.
3) Exploration Strategies:
The implementations tailor exploration to each model-free RL algorithm and evaluate them using returns and roundabout success under sparse and dense traffic.
- 3) Exploration Strategies:: DDQN samples exploratory actions according to their Q values rather than uniformly, favoring actions with larger expected future rewards.The strategy is intended to make exploration more efficient by incorporating expected future reward information.
- 3) Exploration Strategies:: TD3 uses zero-mean Gaussian action noise whose variance changes with training time, path progress, and a periodic coefficient.The acceleration and steering noise scales are δ = 0.5 and 0.1, respectively.
- 3) Exploration Strategies:: SAC needs no separate exploration mechanism because exploration is included in its learned stochastic policy.Its policy adapts exploration during training.
- 3) Exploration Strategies:: Evaluation measures average return and success rates for entering and passing through the roundabout, with and without surrounding vehicles.The no-vehicle experiment uses front-view DDPG as the baseline under matched implementation settings.
- A. Roundabout without Vehicles: In the no-vehicle setting, SAC performs best and reaches the goal, while the front-view DDPG baseline remains at low returns in a right-turning local optimum.Training was stopped at 140 epochs after SAC could stably reach the final goal point.
B. Roundabout with Dense Surrounding Vehicles
With 100 surrounding vehicles, SAC remains the strongest method, frequently reaching the goal while DDQN and TD3 struggle beyond the early exits. Failures are concentrated in rear-end collisions linked to weak velocity representation.
- B. Roundabout with Dense Surrounding Vehicles: All three proposed algorithms exceed 80% success at roundabout entry, but DDQN and TD3 success decreases dramatically as the traveled distance increases.Success is evaluated at the entrance, first exit, second exit, desired exit, and goal point.
- B. Roundabout with Dense Surrounding Vehicles: SAC reaches the goal point more often than the other methods, which the authors associate with a better balance of exploration and exploitation.The authors suggest insufficient exploration contributes to DDQN and TD3 degradation at longer distances.
- B. Roundabout with Dense Surrounding Vehicles: Almost all failures are rear-end collisions because vehicle velocity is not explicitly included in the input representation.Velocity is only implicitly represented through fading historical boxes, whose information becomes difficult to see after visual encoding.
- B. Roundabout with Dense Surrounding Vehicles: The framework solves the roundabout task better than the baseline but does not solve it perfectly.The authors identify improved learning efficiency and generalization to other scenarios as future directions.