Source-linked AI summary
End-to-End Race Driving with Deep Reinforcement Learning
Maximilian Jaritz, Raoul de Charette, Marin Toromanoff, Etienne Perot, Fawzi Nashashibi
TL;DR
End-to-end driving remains difficult because prior systems relied on simplified simulators and direct learning is hard to deploy safely in real cars. This paper trains an A3C agent in realistic WRC6 using RGB images and speed, then evaluates driving performance, generalization, and real-video decisions. The reported system learns full control, improves over a smaller-network comparison, generalizes to unseen scenarios, and produces promising decisions on real images.
Problem
Prior end-to-end driving studies used simulators with simplified graphics and physics, while real-life reinforcement-learning training remains unsafe.
Method
The paper trains an A3C end-to-end driving policy in WRC6 using RGB front-view images and speed to control steering, brake, gas, and hand brake.
Results
+89.9m average distance covered (+14.3%) and −0.8 average car’s hit per kilometer (−13.0%) versus the smaller network, with generalization to real videos also reported.
Takeaways & Limitations
Simulation-trained reinforcement learning can initialize end-to-end driving decision networks that show some generalization to real images.
Abstract
from arXiv · showhide
We present research using the latest reinforcement learning algorithm for end-to-end driving without any mediated perception (object recognition, scene understanding). The newly proposed reward and learning strategies lead together to faster convergence and more robust driving using only RGB image from a forward facing camera. An Asynchronous Actor Critic (A3C) framework is used to learn the car control in a physically and graphically realistic rally game, with the agents evolving simultaneously on tracks with a variety of road structures (turns, hills), graphics (seasons, location) and physics (road adherence). A thorough evaluation is conducted and generalization is proven on unseen tracks and using legal speed limits. Open loop tests on real sequences of images show some domain adaption capability of our method.
I. INTRODUCTION
End-to-end driving replaces the traditional perception-planning-control chain with direct sensor-to-control learning, but prior driving studies used simplified simulators and limited realism. The paper addresses this gap with asynchronous reinforcement learning in a more realistic, varied rally environment and reports generalization beyond training conditions.
- Motivation: Prior end-to-end driving research showed direct sensor-to-control learning but remained limited by simplified simulator graphics, physics, and realism.The literature also included reinforcement-learning demonstrations in TORCS and supervised behavioral cloning from real-driver data.
- Contribution: The proposed method trains an end-to-end agent in WRC6 using only images and speed to predict full longitudinal and lateral control.WRC6 provides stochastic behavior, varied visual appearances, road adherence, and 29.6km of training tracks.
- Contribution: The learning strategy is reported to converge faster and generalize across unseen scenarios despite WRC6’s more complex environment.The paper also reports successful tests on real driving videos, including scenarios unseen during training.
- Related learning approaches: Behavioral cloning mimics an expert driver and is described as unable to adapt to unseen situations.Direct-perception methods instead learn high-level driving features for subsequent control.
- Related learning approaches: Reinforcement learning avoids ground-truth driving labels and can use sparse or delayed rewards for sequences of driving decisions.Asynchronous learning is presented as one way to address local optima and decorrelate experience.
III. METHOD
The method learns full car control through asynchronous reinforcement learning in a realistic rally-game simulator. At each time step, an RGB front-view state is mapped to an action, and the resulting reward supervises policy optimization.
- III. METHOD: Training occurs in WRC6, whose varied visual appearances and physics enable rally-condition learning without risking real cars.The task includes steering, braking, gas, and hand-brake control for drifting.
- III. METHOD: A3C trains the architecture by decorrelating experience across simultaneous interactions with the environment.The pipeline receives a game state, applies an action, and obtains a reward at the next iteration.
- III. METHOD: The agent-environment loop returns the next state and reward after each action, defining the reinforcement-learning supervision process.The objective is the discounted return over future rewards.
1) Policy optimization:
Policy optimization uses REINFORCE within an A3C actor-critic framework. Value-based return estimates provide advantages that adjust the policy according to whether sampled actions outperform expectations.
- 1) Policy optimization:: The policy network outputs probabilities for steering, gas, brake, and hand-brake commands, and REINFORCE estimates the gradient of expected return.The policy is parameterized by neural-network weights.
- 1) Policy optimization:: A3C estimates returns with a value function and bootstraps remaining rewards after a variable number of steps up to t_max = 5.The estimate combines observed discounted rewards with the value of the later state.
- 1) Policy optimization:: The advantage compares sampled returns with the value expected at the current state, indicating whether actions were better or worse than expected.Actor and critic functions are estimated with separate losses while sharing most network layers.
- 1) Policy optimization:: A3C supports small image encoders without experience replay and enables simultaneous training in multiple undetermined environments.This suits WRC6’s stochastic behavior and provides experience decorrelation.
B. State encoder
The state encoder combines a shallow CNN with an LSTM to extract visual features while incorporating motion-related information for control. The learning strategy treats control, reward shaping, and initialization as important components of end-to-end driving.
- B. State encoder: The encoder uses three convolutional layers with dense filtering, max pooling, and an LSTM that receives speed and the previous action.The recurrent component addresses multiple valid control decisions when motion is not considered.
- B. State encoder: Dense stride-1 filtering is intended to improve extraction of far-away visual features relevant to racing control.The architecture is slightly deeper than the comparison network from Mnih et al.
- C. End-to-end learning strategy: Naively training A3C with a fixed state encoder did not reach optimal performance, making control, reward shaping, and agent initialization important.The paper identifies agent initialization as especially neglected in prior literature.
- 1) Control:: The action space includes continuous-range steering, gas, and binary brake and hand-brake commands.The hand brake enables drifting through hairpin bends, while braking primarily slows the car.
- 1) Control:: The policy network represents control choices as 32 output classes, with gas commands occupying a prominent portion of the action design.A softmax layer assigns probabilities to these classes from the CNN-LSTM state representation.
2) Reward shaping:
The method shapes dense racing rewards with speed, heading alignment, and distance from the road center, while random checkpoint initialization improves exploration and generalization.
- Reward design: The per-frame reward combines speed and heading alignment, addressing the sparsity of end-of-track racing scores.The reward uses metadata received from the game rather than waiting for the final track score.
- Reward design: Adding distance from the road center as a penalty prevents sliding along the guard rail and helps the agent learn to stay centered.The authors report that this penalty enables rapid learning of centered track-following.
- Reward design: Two additional rewards incorporate road width, including a sigmoid distance penalty that penalizes leaving the road more smoothly.These alternatives are introduced to account for road-width variation.
- Initialization: Random checkpoint initialization replaces repeated starts at the track beginning to improve exploration and generalization.The authors note that beginning-only initialization improves training-track performance but harms generalization.
IV. EXPERIMENTS
Experiments train and evaluate the agent in WRC6, using distributed game instances and first-person visual inputs with speed, across varied tracks and realistic conditions.
- Environment: WRC6 provides more realistic physics, graphics, stochastic behavior, and road shapes than the TORCS platform used in earlier studies.The environment includes grip, drift, illuminations, animations, sharp turns, and slopes.
- Training setup: The training system runs the reinforcement-learning algorithm centrally while communicating with nine WRC6 instances across two machines.Agents communicate with game instances through TCP and a dedicated API.
- Inputs: The agent receives only normalized first-person RGB images and speed, without in-game displays such as track progress or turn indicators.Removing these usual game cues makes the task harder while supporting comparison with real-driver information.
- Evaluation protocol: Training uses three tracks, reserves two tracks for generalization testing, and starts or respawns agents at random checkpoints.Agents begin each episode at 0 km/h.
B. Performance evaluation
The trained agent learns robust track following in challenging WRC6 conditions, using road structure for control, but remains limited by missing anticipation and difficult physics.
- Quantitative performance: 72.88km/h average speed, 0.72km average distance per run, and 5.44 scene-object hits per kilometer were achieved after training.The results cover three challenging tracks over 140 million training steps.
- Driving behavior: The agent learned to navigate slopes, sharp curves, and some hairpin bends despite difficult track structures.Crashes are defined as stopping, progressing incorrectly, or traveling off road or in the wrong direction.
- Limitations: The agent drives smoothly and can drift with handbrake control, but does not achieve optimal racing trajectories because it lacks anticipation.It therefore tends to remain near the track center rather than taking turns from the inside.
- Limitations: Slippery snow-track conditions cause frequent crashes when the vehicle approaches too quickly.The authors also note that the WRC6 physics, graphics, dynamics, and tracks are more complex than those in TORCS.
- Network interpretation: Guided back-propagation indicates that road edges and curvature strongly influence actions across varied scene appearances.The analysis examines positive inner gradients associated with the selected action.
C. Comparative evaluation
The comparative evaluation isolates the state encoder, reward shaping, and respawn strategy by retraining the full network while changing one factor at a time. The proposed encoder converges more slowly but achieves better racing performance than the smaller baseline encoder.
- Each contribution was evaluated separately by retraining the whole network while changing only the state encoder, reward, or respawn strategy.
- State encoder: 80 versus 130 mega steps: the smaller encoder from Mnih et al. converged faster than the proposed encoder.
- State encoder: The two encoders produced comparable racing performance, with the proposed network exploring slightly more in sections A and B.
- State encoder: +89.9m average distance covered (+14.3%) and −0.8 average car’s hit per kilometer (−13.0%): the proposed encoder outperformed the smaller baseline after retraining.
2) Reward shaping:
Reward shaping uses road width to penalize off-lane distance, while random checkpoint respawning improves exploration. These choices reduce crashes and increase exploration, with lower-hit driving trading off against speed.
- Reward shaping: Two road-width-aware rewards penalize distance outside the lane, using either a margin or a smooth sigmoid that avoids singularity.
- Reward shaping: Mnih et al. (9.2hits/km) versus Ours (2.3hits/km): the proposed reward reduced hits and converged faster, but Mnih et al. drove faster at 106.3km/h versus Ours at 91.4km/h.
- Reward shaping: The partly constant Ours w/ margin reward performed worst among the three proposed rewards, likely because it is less suited to gradient-descent optimization.
- Respawn strategy: +32.20% and +65.19%: random checkpoint respawning improved exploration on the complex snow and mountain tracks, respectively.
- The comparative evaluation concludes that the proposed state encoder, respawn strategy, and reward shaping improve end-to-end driving performance.
V. GENERALIZATION
The agent was tested for generalization across road layouts and asked whether it could drive on unseen tracks, respect speed limits, and operate on real images. It followed varied tracks at high speeds, supporting learned driving concepts beyond memorizing one track.
- Training occurs in a stochastic game environment with animations and illumination changes, and track-completion metrics can be biased when starting positions differ.
- The evaluation asks whether the agent can drive on unseen tracks, respect speed limits, and perform on real images.
- The agent followed tracks with different road layouts at high speeds, indicating learned general driving concepts rather than memorizing a single track.
2) Racing VS Normal driving:
The racing-oriented reward encourages speed but can produce unsafe collision behavior for normal driving. Lower speeds reduce crashes, while open-loop real-video tests show promising control decisions despite commands never being applied.
- Racing VS Normal driving: Because the reward favors speed without directly penalizing collisions, the agent learned to drive fast but this behavior is dangerous for normal driving.
- Racing VS Normal driving: Lower speeds significantly reduced crashes, including collisions and off-road events, across the evaluated tracks.
- Real videos: Real-video tests produced control decisions in varied environments, including situations not encountered during training, but the commands were never applied.
- Conclusion: The authors conclude that training performance can be transposed to camera images and that the agent shows generalization capacities with ad-hoc speed limits.