Source-linked AI summary
Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning
Florian Fuchs, Yunlong Song, Elia Kaufmann, Davide Scaramuzza, Peter Duerr
TL;DR
Autonomous car racing demands minimum-time control at the vehicle’s handling limits, but sparse lap-time rewards and limited expert data complicate learning. The paper combines a high-fidelity Gran Turismo Sport simulator, a course-progress proxy reward, and deep reinforcement learning. Its policy surpasses the built-in game AI and the fastest driver in a dataset of more than 50,000 human players.
Problem
Autonomous racing requires minimum-time planning and extreme-manoeuvre control, while sparse lap-time rewards and limited expert data hinder direct learning-based solutions.
Method
The system trains a model-free deep-reinforcement-learning policy in Gran Turismo Sport using a course-progress proxy reward and a kinetic-energy-scaled wall-contact penalty.
Results
The policy achieves super-human time-trial performance, outperforming the built-in game AI and the best known human lap times across three reference settings.
Takeaways & Limitations
The approach demonstrates that autonomous racing can achieve super-human control without human intervention, expert data, or explicit path planning.
Takeaways & Limitations
The method is restricted to single-player time trials and has constrained applicability to a single track/car combination.
Abstract
from arXiv · showhide
Autonomous car racing is a major challenge in robotics. It raises fundamental problems for classical approaches such as planning minimum-time trajectories under uncertain dynamics and controlling the car at the limits of its handling. Besides, the requirement of minimizing the lap time, which is a sparse objective, and the difficulty of collecting training data from human experts have also hindered researchers from directly applying learning-based approaches to solve the problem. In the present work, we propose a learning-based system for autonomous car racing by leveraging a high-fidelity physical car simulation, a course-progress proxy reward, and deep reinforcement learning. We deploy our system in Gran Turismo Sport, a world-leading car simulator known for its realistic physics simulation of different race cars and tracks, which is even used to recruit human race car drivers. Our trained policy achieves autonomous racing performance that goes beyond what had been achieved so far by the built-in AI, and, at the same time, outperforms the fastest driver in a dataset of over 50,000 human players.
I. INTRODUCTION
Autonomous car racing requires fast, precise control at physical limits while minimizing lap time under challenging dynamics. The paper uses model-free deep reinforcement learning in Gran Turismo Sport to achieve super-human racing performance without explicit trajectory planning or human expert data.
- Autonomous racing requires minimum-time trajectory planning and safe control during extreme vehicle manoeuvres.
- Gran Turismo Sport provides realistic car and track simulation for benchmarking autonomous systems against experienced human drivers.
- The proposed controller uses model-free deep reinforcement learning, a course-progress proxy reward, and a multilayer perceptron policy.
- The policy avoids high-level trajectory planning and produces trajectories qualitatively similar to those selected by the best human drivers.
- The approach demonstrates super-human autonomous racing control performance in Gran Turismo Sport.
II. RELATED WORK
Prior autonomous-racing research includes classical planning-and-control, imitation learning, and reinforcement-learning approaches, each with distinct computational, data, or flexibility constraints.
- Prior work groups autonomous racing into classical trajectory planning, supervised learning, and reinforcement learning approaches.
- Classical Approaches: Classical MPC and MPPI methods can control vehicles at high speed but require substantial computation and may suffer from modular approximation failures.
- Imitation Learning: Imitation learning directly maps observations to actions but is performance-limited by the quality of human or algorithmic training data.
- Reinforcement Learning: Model-free reinforcement learning optimizes policies from sampled trajectories without nonlinear optimization online or labelled training data.
III. METHODOLOGY
The system learns a neural controller that maps racing observations to vehicle actions and optimizes its policy with soft actor-critic using distributed simulator-generated experience.
- The objective is to navigate a Gran Turismo Sport race car without prior vehicle-dynamics knowledge while minimizing travel time.
- A reward function formulates racing, while a neural policy maps input states to control actions and is optimized with SAC.
- The policy network maps range-finder measurements, velocity, and acceleration to steering, throttle, and brake commands.
- Samples are collected in parallel from four PlayStation 4 consoles simulating 20 cars each and stored in a FIFO replay buffer.
A. Minimum-Time Problem and Reward Function
Because lap time is sparse, the method replaces it with discounted course-progress rewards and adds a kinetic-energy-scaled wall-contact penalty to discourage unsafe wall grinding.
- The course-progress proxy reward provides more attributable feedback than sparse lap time while approximating the minimum-lap-time objective at high discount factors.
- Exponential discounting introduces short-term bias that can reduce the incentive to brake for crash prevention.
- Course progress is constructed by projecting the car’s position onto the track centerline, with reward based on progress increments.
- The wall-contact term penalizes contact relative to kinetic energy, balancing progress with collision avoidance.
- Without the additional penalty, learned policies grinded along walls; fixed penalties instead caused either inaction or full braking.
B. Policy Network
The policy uses a deep neural network whose inputs combine vehicle motion, track geometry, wall proximity, and prior control information. It outputs steering and a combined throttle-brake command.
- Network architecture: The policy is represented by a deep neural network with two hidden layers of 256 ReLU nodes in each component.The architecture includes a policy network, two Q-function networks, and a state-value network, totaling 599,566 trainable parameters.
- Input features: Input features combine velocity, acceleration, orientation relative to the track tangent, rangefinder distances, previous steering, wall contact, and future centerline curvature.These features form the observation vector s_t used at each time step.
- Input features: The orientation angle provides the policy’s direct signal for detecting whether the car faces the wrong direction.It is measured between the vehicle’s horizontal rotation and the centerline tangent at the projection point.
- Network output: The policy directly predicts a bounded steering angle and a single throttle-brake signal.The steering range is [−π/6, π/6] rad, while the combined signal ranges from full braking to full throttle.
IV. EXPERIMENTS
The experiments compare separately trained agents across three race settings with different cars and tracks against built-in AI and more than 50,000 human drivers. Evaluation also constrains action changes to approximate physical steering-wheel limitations and tests robustness under altered conditions.
- Experimental design: The study evaluates three race settings with different cars and tracks, comparing the approach with built-in AI and over 50,000 human drivers.The human data come from online time-trial competitions conducted under fixed car, course, tire, and assistance settings.
- Experimental design: Each experimental condition receives a separately trained agent using human time-trial data collected under equal racing conditions.The dataset contains personal-best lap times and trajectories ranging from beginners to World Cup contenders.
- Fair comparison: Evaluation limits frame-to-frame steering and throttle-brake changes to approximate the maximum action changes achievable with a physical steering wheel and pedals.The restrictions are 0.03 rad for steering and 80% for each throttle and brake range.
- Robustness: Robustness testing transfers the fastest setting-A agent to another car, another track, altered tire friction, noisy observations, and delayed observations without retraining.These tests probe performance under modified environments beyond the original driving condition.
V. RESULTS
The results section compares lap times with built-in GTS AI and the fastest human drivers, then examines learned driving behavior against professional human strategies. The experiments also include supplementary videos because of their dynamic nature.
- Lap-time evaluation: The evaluation compares the approach’s lap times with those of built-in GTS AI and the fastest human drivers.The comparisons use the three reference settings introduced earlier.
- Driving behavior: The study analyzes the resulting driving behavior alongside strategies used by professional human drivers.This complements the numerical lap-time comparisons with trajectory and control-behavior analysis.
- Supplementary material: The authors encourage readers to watch supplementary videos to inspect the dynamic experiments.The recommendation accompanies the evaluation because the experiments involve dynamic driving behavior.
A. Lap Time Comparison
The learned approach achieves faster lap times than the best human driver across all three settings and exceeds the built-in AI. It learns these policies from scratch, reaches human-beating performance in setting A after extended training, and produces trajectories resembling the fastest human.
- Lap-time comparison: The approach outperforms the best human lap time in all three reference settings and exceeds the built-in GTS AI.The comparison covers the fastest human drivers, built-in AI, and three race settings.
- Lap-time comparison: 0.15 seconds and 0.04 seconds are the margins by which the approach beats the fastest human in settings A and B, respectively.The reported margins apply to the first track’s two car-and-setting combinations.
- Lap-time comparison: 0.62 seconds is the margin by which the approach beats the best human time in setting C.Setting C is described as the most difficult combination because of its track layout and faster car.
- Training progress: 56 to 73 hours of training are required for each initialization to achieve lap times faster than the fastest human reference in setting A.Across three seeds, this corresponds on average to 2,151 epochs and 946,453 km driven.
- Training progress: Training from scratch surpasses the behavior-cloning policy after less than one hour, while imitation-learning warm starts do not improve final performance.The three random initializations show similar learning progress, differing mainly in when they master particular curves.
- Driving behavior: The learned controller uses the full track width for out-in-out trajectories, producing curves similar to the fastest human without demonstrations or explicit path planning.Using larger curve radii allows higher speeds before traction is lost.
B. Learned Driving Behavior
The learned policy reproduces key expert racing behaviors, including wide out-in-out trajectories, early curve anticipation, and competitive speed. Across settings, cars, and tracks, it achieves faster lap times than the human reference drivers.
- Out-in-out trajectory: The policy learned out-in-out trajectories that use the full track width to increase curve radii and maintain higher speeds before losing traction.It achieved this without human demonstrations or explicit path planning.
- Anticipating curves: The policy detected curve sharpness early enough to decelerate without overshooting, beginning braking approximately 100 meters before the setting C hairpin.This behavior balanced avoiding wall contact with avoiding excessive caution.
- Overall speed: The policy closely matched or improved on the fastest human’s speed and path, undercutting that lap time by 0.15 seconds in setting A.The reported margin is comparable to differences between top contenders in real and simulated racing championships.
- Overall speed: In the setting C hairpin, the policy compensated for a tighter incoming trajectory on the outgoing straight, producing similar exit speed and curve completion time to the human.Across most of the track, it often drove tighter and faster curves than the human expert.
- Overall result: Across different race settings, cars, and tracks, the policy achieved lap times faster than all human reference drivers while maintaining slightly higher average speed.The learned trajectories were qualitatively similar to those chosen by the best human players.
C. Robustness
Robustness testing shows that the agent adapts to some changes in car dynamics, track geometry, observation noise, and inference delay, but performance degrades under sufficiently large shifts. The strongest failures occur on unseen sharp curves, high delays, and altered dynamics that exceed the learned behavior.
- Transfer to new car: With a new car, the agent finished the track and adapted on straights and slight curves but sometimes overshot sharp curves because of unseen lower-friction dynamics.The training car had higher tire-surface friction, making it easier to control than the test car.
- Transfer to new track: On a new track, the agent drove wall-contact free on straights and slight curves but became stuck in the hairpin and could not finish.The failure was attributed to inability to extrapolate to some unseen curve shapes.
- Changed tire friction: With altered tire friction, the agent corrected its trajectory in many cases but contacted curve edges in sharper sections, losing 0.1 seconds under increased friction.Under lowered friction, it maintained out-in-out paths except on the three sharpest curves.
- Overall robustness: The robustness experiments indicate that path corrections work only to a limited degree and are insufficient for some track segments under changed dynamics.This limitation constrains transfer beyond the dynamics and geometry represented during training.
- Noise in observations: Up to 9% observation noise still allowed track driving without edge contact, although noise above 2% caused increasingly jittery actions and speed loss.At up to 2% noise, trajectories remained close to the noise-free baseline and stayed ahead of the fastest human.
- Delayed inference: Inference delays up to 20 ms preserved near-baseline performance, whereas 50 ms cost 0.3 seconds, 100 ms cost 1.4 seconds, and 150 ms caused over 6 seconds of loss.At 100 ms, late braking caused wall contact; at 150 ms, the agent could no longer drive a competitive racing line.
VI. CONCLUSION
The paper presents a model-free deep reinforcement learning racing policy that achieves super-human time-trial performance in Gran Turismo Sport without human data or explicit path planning. Its learned trajectories resemble expert driving, while the system remains computationally efficient and is limited in race and vehicle-track scope.
- The policy achieves super-human autonomous racing performance in Gran Turismo Sport time trials.It outperforms the best known human lap times in all three reference settings.
- It does not rely on human intervention, human expert data, or explicit trajectory planning.
- The learned trajectories are qualitatively similar to those chosen by the best human players.The policy learned out-in-out trajectories that increase curve radii and permit higher speeds before traction loss.
- The system is computationally efficient, generating one control command in around 0.35 ms and reaching super-human performance in under 73 hours of training.Training used four PlayStation 4 consoles and one desktop PC.
- The approach is restricted to single-player time trials and has constrained applicability to a single track-car combination.The authors propose extending observation and reward design for other cars and using data-efficient reinforcement learning for broader generalization.
- The policy uses course-progress reward design and low-dimensional observations to train vehicle control with SAC.The observation design includes vehicle motion, orientation, rangefinders, previous steering, wall contact, and future centerline curvature.
H. Reproducibility of experiments after GTS Update 1.57
Gran Turismo Sport version 1.57 changed the Audi TT Cup ’16 dynamics, making the original human comparisons for settings A and C no longer fair. The authors recommend an updated replacement dataset and report a new-setting evaluation.
- GTS version 1.57 changed Audi TT Cup ’16 dynamics and made fair comparison with the older human datasets impossible for settings A and C.
- The authors recommend kudosprime.com as a replacement human dataset for setting A under updated dynamics and tires.The replacement setting uses version 1.57+ car dynamics and RM rather than RH tires.
- 74.686 seconds was achieved after 5 days of training on the replacement setting A.