Source-linked AI summary
High-speed Autonomous Drifting with Deep Reinforcement Learning
Peide Cai, Xiaodong Mei, Lei Tai, Yuxiang Sun, Ming Liu
TL;DR
Autonomous drifting is difficult because conventional controllers require vehicle-dynamics and tire models that are hard to specify accurately. The paper formulates drift control as trajectory following and trains a model-free SAC controller with designed error-based states and rewards. After training on varied tracks, the controller drifts quickly and stably and generalizes across unseen maps and vehicle properties.
Problem
Autonomous drifting requires precise control of an unstable, high-sideslip vehicle, while prior model-based approaches depend on difficult-to-identify tire and vehicle-dynamics models.
Method
The paper trains a closed-loop model-free SAC controller for trajectory-following drift, using designed error-based states and rewards to control steering and throttle.
Results
The SAC controller achieves the shortest lap time in all tested setups and the maximum velocity among four methods, reaching 103.71 km/h in setup F3.0M1.7.
Takeaways & Limitations
The controller remains robust across varied vehicle mass and tire friction and generalizes to vehicles with diverse physical properties while drifting through complex curved tracks.
Abstract
from arXiv · showhide
Drifting is a complicated task for autonomous vehicle control. Most traditional methods in this area are based on motion equations derived by the understanding of vehicle dynamics, which is difficult to be modeled precisely. We propose a robust drift controller without explicit motion equations, which is based on the latest model-free deep reinforcement learning algorithm soft actor-critic. The drift control problem is formulated as a trajectory following task, where the errorbased state and reward are designed. After being trained on tracks with different levels of difficulty, our controller is capable of making the vehicle drift through various sharp corners quickly and stably in the unseen map. The proposed controller is further shown to have excellent generalization ability, which can directly handle unseen vehicle types with different physical properties, such as mass, tire friction, etc.
I. INTRODUCTION
Drifting enables fast cornering but requires precise control of an unstable, high-sideslip vehicle. The paper motivates model-free deep RL as a way to learn transient drift control without explicit tire models or motion equations, then evaluates generalization across environments and vehicle properties.
- Motivation: Drifting uses deliberate rear-tire saturation and high sideslip to negotiate sharp corners quickly, but increasing sideslip also increases instability and control difficulty.Compared with normal cornering, drifting can reduce racing lap time while operating beyond the vehicle’s usual handling limits.
- Motivation: High-frequency steering and throttle decisions during professional drifting expose agile-control knowledge relevant to autonomous vehicle operating envelopes.The paper connects drift behavior to reducing lap time and understanding aggressive driving techniques.
- Research gap: Traditional drift controllers rely on vehicle-dynamics models and tire parameters that are difficult and costly to identify accurately.These approaches use explicit motion equations with varying model fidelity, but tire behavior across operating points is challenging to characterize.
- Approach: The paper formulates transient drifting as trajectory following and uses model-free deep RL to learn a policy without tire models or explicit motion equations.Prior learning-based work mainly stabilized sustained drift around a single equilibrium, whereas this work targets non-steady drift states such as cornering.
- Contributions: The closed-loop controller uses deep RL to steer FWD vehicles at 80-128 km/h through sharp corners quickly and stably, while evaluations vary corner shapes, vehicle mass, and tire friction.The authors also release seven racing maps with reference drift trajectories for benchmarking and future studies.
B. Drifting control approaches
Prior drift controllers mainly use vehicle-dynamics models, while learning-based methods have often targeted steady-state drift. This paper instead uses model-free SAC and evaluates generalization across increasingly difficult maps and vehicle conditions.
- Traditional methods: Model-based drift controllers use single-track or related vehicle models to stabilize or optimize drift, but transient drift remains difficult because drifting dynamics are complex.Prior methods vary in model fidelity and include tire parameters, suspension dynamics, and optimization-based control.
- Traditional methods: Traditional methods depend on tire or road forces that cannot be known precisely in complex real-world environments, so parameter inaccuracies can impair control performance.This limitation motivates approaches that avoid explicit force and tire modeling.
- Learning-based methods: Earlier learning-based approaches used model-based RL or hybrid MPC-neural methods, primarily addressing sustained or steady-state drifting rather than transient cornering.These methods include constant-sideways-velocity drift and drifting over ranges of road radii and slip angles.
- Learning-based methods: The proposed approach adopts SAC, a model-free deep RL algorithm, to train a closed-loop drift controller.This distinguishes the method from the model-based and hybrid approaches discussed previously.
- Learning-based methods: The paper reports transient drift with deep RL and tests generalization across road structures, tire friction, and vehicle types, factors often neglected by prior work.Seven maps increase in difficulty from (a) through (g), with maps (a-f) used for training and (g) for evaluation.
III. METHODOLOGY
The drift controller treats high-speed drifting as trajectory following, using CARLA and reference maps to train and evaluate SAC-based control. Vector field guidance supplies desired heading and trajectory errors for tracking.
- Formulation: The task is to follow a reference trajectory at over 80 km/h while drifting through corners with sideslip angles above 20°.Training and validation use CARLA, an open-source simulator with realistic vehicle physics and multiple vehicle types.
- Map generation: Seven maps with different difficulty levels are designed for the drifting task.The maps are based on PopKart tracks and generated with RoadRunner.
- Reference trajectories: Reference drift trajectories are recorded from an experienced driver operating the vehicle with a steering wheel and pedals.The driver is instructed to drive as fast as possible while drifting.
- State variables: Vector field guidance determines the desired heading angle from the vehicle's position relative to the reference trajectory.The guidance uses cross-track error and heading-angle error to support trajectory following.
2) State space:
The controller uses a structured vehicle state containing motion, trajectory, and error information, then applies constrained and smoothed actions during SAC training. Terminal conditions end episodes after collisions, successful arrival, or substantial departure from the track.
- State space: The state space has dimension 42 and includes ten future reference-trajectory positions and slip angles.It also includes vehicle motion variables, trajectory errors, and time derivatives such as the cross-track-error derivative.
- Terminal state: Episodes terminate when the vehicle collides, reaches the destination, or moves more than fifteen meters from the track.These conditions are represented with an endFlag.
- Action space: Throttle is constrained to [0.6,1] and steering to [−0.8,0.8] to support high-speed driving and reduce rollover risk.The throttle range discourages slow driving, while the smaller steering range addresses rollover at high speed.
- Training: SAC training initializes the networks and iterates over episodes using a replay buffer, transition count, update threshold, and update count.The algorithm's stated result is an optimal control policy.
- Action space: Action smoothing constrains the current action using the previously deployed action to avoid shaky control output.The diagonal tuning matrices adjust smoothing separately for steering and throttle, with larger K2 producing greater similarity to the previous action.
4) Reward shaping:
The reward combines trajectory-tracking errors with vehicle speed so the controller favors accurate, fast drifting. Partial rewards peak at zero error, while low speed and large directional errors are penalized.
- Reward formulation: The reward-shaping design uses partial rewards to evaluate high-speed drifting with low cross-track, heading, and slip-angle errors.The formulation targets the stated control objective rather than speed alone.
- Reward formulation: The total reward is vehicle speed multiplied by a weighted sum of partial rewards for cross-track, heading, and slip-angle errors.The weights are selected as [40,40,20] for the three error terms.
- Partial rewards: Each partial reward is maximal when its corresponding error is zero and decreases as the error increases.The heading-error reward becomes negative when eψ exceeds 90° to penalize driving in the opposite direction.
- Speed shaping: When speed v is below 6 m/s, the total reward is halved as a punishment; otherwise, the original product is used.This speed factor directly encourages fast driving.
5) Soft actor-critic:
The controller uses SAC to learn a stochastic drift policy through actor-critic updates and repeated environment interaction. Training compares learning curves across algorithms while accounting for DDPG’s delayed evaluation.
- Soft actor-critic: SAC maximizes a trade-off between expected return and policy entropy using off-policy actor-critic learning.The policy network acts as the actor, while Q- and value networks form the critic.
- Soft actor-critic: The policy maps a 42-dimensional state to a 2-dimensional action, which is sampled, normalized with tanh, then mapped and smoothed for environment interaction.
- Soft actor-critic: Performance curves average three repeated experiments, showing mean learning behavior with shaded standard-deviation regions.
- Soft actor-critic: DDPG evaluation begins at episode 200 because it often circles near the starting location during early training.
- Soft actor-critic: Training stores each transition in a replay buffer and updates the value, twin Q, and policy networks after enough transitions are collected.
IV. EXPERIMENTS AND DISCUSSION
Training proceeds progressively across six maps, beginning with basic driving skills and continuing on maps with increasingly diverse and difficult corner geometries.
- IV. EXPERIMENTS AND DISCUSSION: Map (a) provides first-stage training for speeding up and drifting through simple corners.
- IV. EXPERIMENTS AND DISCUSSION: Maps (b–f) introduce diverse corner shapes and difficulty levels for further training.
- IV. EXPERIMENTS AND DISCUSSION: Further training initializes from map (a)’s pretrained weights so the vehicle can reuse previously learned driving knowledge.
1) Implementation:
The experiments compare SAC with DQN, DDPG, and SAC without action smoothing across varied vehicle and environment conditions, using trajectory, speed, time, smoothness, and slip metrics.
- Implementation: The comparison includes DQN, DDPG, and SAC-WOS, which removes SAC’s action-smoothing strategy.
- Implementation: SAC achieves an average velocity of 80 km/h while reaching approximately the same final heading-angle error as most baselines.
- Implementation: Tests use three tire-friction and vehicle-mass combinations on unseen map (g), whose corners range from 40° to 180°.
- Implementation: The evaluation measures cross-track error, heading-angle error, maximum and average velocity, lap time, steering smoothness, and maximum slip angle.
2) Quantitative results:
On the unseen map, SAC combines high-speed cornering with accurate trajectory heading control and smooth driving across varied environment setups. The evaluation also compares these behaviors with human driving and competing controllers.
- Quantitative results: SAC achieves the shortest lap time in all setups and the highest maximum velocity among the four compared methods.In setup F3.0M1.7, maximum velocity reaches 103.71 km/h, versus 90.59 km/h for DDPG and 84.02 km/h for SAC-WOS.
- Quantitative results: Table I reports quantitative evaluation and generalization across varied environment setups, with arrows indicating whether larger or smaller values are better.
- Quantitative results: SAC achieves the best heading-angle error, especially through corners, while SAC-WOS achieves the smallest cross-track error.The authors associate SAC’s heading performance with drifting at slip angles similar to the reference behavior.
- Quantitative results: SAC reaches the highest drifting speed and largest slip angles across all setups.Average velocity reaches 79.07 km/h in F3.5M1.8, close to HUMAN-DFT’s 79.38 km/h; slip angle reaches 29.23° in F3.0M1.7.
- Quantitative results: SAC achieves the smallest SMOS among the tested controllers, although all controllers are less smooth than the human driver.
- Quantitative results: The study includes vehicle configurations used for both training and testing, with the training vehicle identified separately.
3) Qualitative results:
The SAC controller tracks most reference trajectories smoothly through sharp corners, while performance varies with corner geometry and vehicle properties. It generalizes to unseen mass, tire friction, and vehicle types, although extreme friction and heavy-vehicle settings reduce drift quality.
- Qualitative trajectory tracking: The SAC controller tracks linear paths and most corners closely, including approximately 90° corners entered near 100 km/h.Speed decreases near corner vertices because of large slip angles, then increases quickly after the corner.
- Qualitative trajectory tracking: SAC produces smoother steering than the comparison controllers, whose steering curves are especially shaky for DDPG and SAC-WOS.The SAC steering angle concentrates in a smaller range.
- Unseen mass and friction: The controller handles unseen friction and mass combinations without fine-tuning, reaching more than 100 km/h and a maximum slip angle of 43.27° for F2.6M1.6.The result demonstrates direct operation under altered vehicle parameters.
- Unseen mass and friction: Under DF-M1.8, the vehicle completes the map quickly and smoothly, but its maximum slip angle remains below 20° because large rear-tire friction inhibits slipping.This setting changes front and rear tire friction differently.
- Unseen vehicle types: The controller generalizes to three unseen vehicle types, with Vehicle-2 reaching 128.07 km/h peak speed and 92.77 km/h average drift speed.Vehicle-4 follows the reference trajectory precisely, but its heavy weight and large size produce small slip angles and cornering speeds.
2) Unseen vehicle types:
The controller can operate with rough reference trajectories and without slip-angle information during execution, but accurate slip-angle guidance during training improves efficiency and final performance. The proposed SAC design combines error-based state and reward construction with smoothed steering and throttle control.
- Reference information: A rough centerline-based reference trajectory is sufficient for policy execution when expert drift trajectories are unavailable.The paper derives heading directions from waypoint tangents and sets reference slip angles to zero.
- Reference information: Slip-angle information is dispensable during execution, supporting application on unseen tracks with rough references derived online from maps.The paper identifies this as relevant to real-world applications using 2D or 3D maps.
- Training ablation: SAC-42, trained with slip-angle guidance, achieves higher speed, shorter lap time, smaller error, smoother driving, and lower training time than SAC-30.SAC-30 excludes slip-angle variables from the state and reward, while SAC-42 retains them.
- Controller design: The controller uses SAC with carefully designed error-based states and rewards plus action smoothing to control steering angle and throttle.Training uses maps with different driving difficulty levels.
- Overall findings: After two-stage training on six maps, SAC remains robust to varied mass and tire friction and generalizes across vehicles with diverse physical properties.The conclusion also reports non-degraded performance with a rough, accessible reference.