PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Simulation, Racing in Reality

Eugenio Chisari, Alexander Liniger, Alisa Rupenyan, Luc Van Gool, John Lygeros

arXiv:2011.13332v2cs.ROcs.AIcs.SYeess.SY

TL;DR

Direct learning of autonomous-racing policies is rarely studied, although racing requires fast control at the nonlinear limit of vehicle handling. The paper trains SAC in simulation with model randomization, lifted input-rate actions, and policy-output regularization, then refines the transferred policy using real-car interaction. The refined policy achieved an average lap time of 11.01 s and 0.44 s of constraint violations per lap, with lap time close to MPC.

Abstract

from arXiv · show

We present a reinforcement learning-based solution to autonomously race on a miniature race car platform. We show that a policy that is trained purely in simulation using a relatively simple vehicle model, including model randomization, can be successfully transferred to the real robotic setup. We achieve this by using novel policy output regularization approach and a lifted action space which enables smooth actions but still aggressive race car driving. We show that this regularized policy does outperform the Soft Actor Critic (SAC) baseline method, both in simulation and on the real car, but it is still outperformed by a Model Predictive Controller (MPC) state of the art method. The refinement of the policy with three hours of real-world interaction data allows the reinforcement learning policy to achieve lap times similar to the MPC controller while reducing track constraint violations by 50%.

I. INTRODUCTION

Autonomous racing demands fast, limit-handling control, while direct learned racing policies have received little study relative to model-predictive-control methods. This paper transfers a simulation-trained RL policy to a miniature race car using model randomization and policy-output regularization.

I. INTRODUCTION

  • Direct learning of autonomous-racing policies is rarely studied, while the most advanced methods use model-based predictive control.Prior direct-policy examples include pixel-to-action imitation learning on a 1:5-scale car and a policy learned in a racing simulation game.Source: §I · p. 1
  • The vehicle-model gap is the main sim-to-real challenge in this racing task, unlike cited autonomous-driving transfers that primarily faced a vision-domain gap.The paper notes that limited environment representations can cause simulation overfitting and that domain or model randomization can help transfer.Source: §I · p. 1
  • Policy regularization shapes learning, and prior work studied entropy maximization, deep-learning regularizers, and hand-crafted reward features for smoother driving.Prior work reported benefits from weight clipping, dropout, BatchNorm, and L2 policy-weight regularization.Source: §I · p. 1
  • The proposed regularized RL policy transfers from simulation to the miniature car, outperforms SAC in simulation and on the real platform, and after refinement approaches MPC lap times with fewer constraint violations.The introduction reports 50% fewer constraint violations after refinement compared with MPC.Source: §I · p. 1

II. AUTONOMOUS RACING

The paper formulates miniature-car racing around fast progress while respecting track limits. It uses a lifted state and action representation to support smooth control, and a dense reward that balances progress with track and tire-slip constraints.

II. AUTONOMOUS RACING

  • The 1:43 car drives an 18 m track with 13 turns at speeds above 3 m/s, and its nonlinear behavior makes dynamics difficult to model accurately.Position, heading, and velocities are estimated by an overhead infrared camera at 100 Hz.Source: §II · p. 2

A. State Space and Action Space

  • The state includes track-relative position and heading, vehicle velocities, yaw rate, and physical inputs; actions are rates of change of motor duty cycle and steering angle.Lifting the physical inputs into the state allows jerky input changes to be penalized without a noticeable learning slowdown.Source: A. State Space and Action Space · p. 2

B. Reward Function

  • The reward targets fast track progress while penalizing track-limit violations and excessive combined tire slip, which is not modeled by the simulator.Track deviation is constrained relative to half the track width, with a practical safety margin because the experimental boundaries are walls.Source: B. Reward Function · p. 2
  • Incremental progress provides a dense approximation to the sparse minimum-lap-time objective, while the reward discourages states where the simulator is imprecise.The reward uses progress from one time step to the next and includes a penalty coefficient c = 0.01.Source: B. Reward Function · p. 2

III. SIM-TO-REAL METHOD

The section introduces the steps for transferring the autonomous-racing policy from simulation to reality and presents a novel policy regularization approach to facilitate that transfer.

III. SIM-TO-REAL METHOD

  • The section sets out the necessary steps for sim-to-real transfer in autonomous racing.Its stated aim is applying a simulation-trained policy to the real racing task.Source: §III · p. 2
  • A novel policy regularization approach is introduced to facilitate sim-to-real transfer.The passage identifies the regularizer as part of the transfer method.Source: §III · p. 2
  • The method is framed specifically for the autonomous-racing task.The section connects its transfer steps and regularization approach to this task.Source: §III · p. 2

A. RL Setup

The policy is trained with SAC in a dynamic-bicycle-model simulator whose uncertainty is addressed through model randomization. The randomized dynamics are calibrated against real driving errors, and the resulting simulated tire-force uncertainty resembles the real measurements.

A. RL Setup

  • Training uses SAC for 1.8 · 10^7 simulation steps, with 600-step maximum episodes and two-hidden-layer networks of 256 neurons per layer.The run takes about 36 hours on a single CPU desktop computer.Source: A. RL Setup · p. 2

B. Simulator and Model Randomization

  • The simulator uses a computationally tractable dynamic bicycle model with Pacejka tires, but the model is over-optimistic and does not capture real-car uncertainty.The model is based on the dynamic bicycle model proposed in.Source: B. Simulator and Model Randomization · pp. 2–3
  • Model randomization multiplies nominal dynamics by (1+ε), with uncertainty applied to velocity states and bounds chosen to cover approximately 80% of errors recorded during benchmark-MPC driving.Using bounds that include all recorded errors or overestimate noise made resulting policies over-conservative.Source: B. Simulator and Model Randomization · p. 3
  • Real-data and noisy-simulation tire-force estimates show that the randomized simulator's uncertainty resembles that in recorded real driving.The figure compares estimated tire forces against the Pacejka model for real measurements and noisy simulation under the trained policy.Source: Figure 1 · p. 3, B. Simulator and Model Randomization · p. 3

C. Policy Regularization

The section motivates regularization as a way to address the physical and sim-to-real costs of jerky control inputs. It introduces reward-action and policy-weight regularization, noting that each falls short in a different way.

C. Policy Regularization

  • Excessive, jerky inputs can potentially damage actuators, while fast-changing inputs in racing can introduce drag that degrades performance.Source: C. Policy Regularization · p. 3
  • High-frequency inputs can produce behaviors that are difficult to model, including load transfer omitted from the simulation model.Source: C. Policy Regularization · p. 3

1) Reward and Policy Weight Regularization:

  • Reward-action regularization adds a quadratic action cost, while policy-weight regularization applies an L2 penalty to policy weights; the former slows learning and the latter does not effectively smooth actions.Source: 1) Reward and Policy Weight… · p. 3

2) Policy Output Regularization:

The proposed policy output regularizer adds a quadratic penalty to SAC’s objective using gradient information in the policy update. In the racing setup, applying it to input rates favors smooth changes without penalizing the physical inputs themselves.

2) Policy Output Regularization:

  • Policy output regularization directly penalizes policy outputs using gradient information in the policy update, without slowing learning.Source: 2) Policy Output Regularization · p. 3
  • The objective adds a quadratic penalty on first-step actions to SAC’s entropy term, with the positive semidefinite matrix M tuning penalties independently for drate and δrate.Source: 2) Policy Output Regularization · p. 3
  • The regularized action-value function remains related to standard SAC’s Q function, and training can use SAC with an added quadratic term in the policy-gradient step.Source: 2) Policy Output Regularization · pp. 3–4
  • Using input rates as actions penalizes how quickly physical inputs change, favoring smooth steering or torque rather than penalizing a constant input.Source: 2) Policy Output Regularization · p. 4

IV. SIM-TO-REAL RESULTS

The section presents simulation comparisons of regularizers on autonomous racing and three benchmark tasks, followed by real-platform experiments. The experiments are intended to show the proposed regularizer’s benefit for sim-to-real transfer.

IV. SIM-TO-REAL RESULTS

  • The evaluation first compares the proposed policy output regularizer with two other regularizers in simulation.The comparisons cover autonomous racing and three benchmark tasks.Source: §IV · p. 4
  • After the simulation results, the section presents experimental results for the autonomous racing problem.Source: §IV · p. 4
  • The real-platform experiments are presented to show the regularizer’s benefit in sim-to-real transfer.Source: §IV · p. 4

A. Policy Regularization for Benchmark Task

Across three benchmarks, regularization improved both action smoothness and reward, with policy output regularization performing best on the hardest task. The comparisons also show different outcomes for policy-weight and reward-action regularization across tasks.

  • The study trained MountainCarContinuous and Pendulum for 6 · 10^4 steps and PyBullet Reacher for 5 · 10^5 steps, then evaluated 100 rollouts.It reports average reward and mean RMS of action derivatives; lower derivative RMS indicates smoother actions.Source: A. Policy Regularization for Benchmark… · p. 4
  • Across all three benchmarks, regularization produced smoother actions and higher policy reward, and policy output regularization achieved the best results on Reacher.Source: A. Policy Regularization for Benchmark… · p. 4
  • On MountainCarContinuous and Pendulum, policy-weight regularization performed best, closely followed by policy output regularization; reward-action regularization often failed to learn.The passage reports that reward-action regularization appears to work across only a very small hyperparameter range.Source: A. Policy Regularization for Benchmark… · p. 4

B. Policy Regularization for Simulated Autonomous Racing

In simulation, policy output regularization produces smoother inputs and fewer constraint violations than the other RL methods. On the real car, it approaches the MPC benchmark, with fewer constraint violations but slower average lap times.

B. Policy Regularization for Simulated Autonomous Racing

C. Sim-to-Real Experimental Results

  • 30% improvement in constraint violations and only 8% slower lap times on average than the MPC benchmark were achieved by the policy output-regularized policy.It still could not match the MPC controller’s peak performance.Source: C. Sim-to-Real Experimental Results · p. 5

V. POLICY REFINEMENT ON THE CAR

The transferred policy is refined through real-car interaction using policy output-regularized SAC. After three hours, lap times and constraint violations improve, with the refined policy close to MPC in lap time and substantially reducing violations.

V. POLICY REFINEMENT ON THE CAR

  • The refinement warm-started from the sim-to-real policy and collected driving data in a replay buffer for off-policy SAC updates.The controller policy was updated after 100 SAC update steps; data collection began with roughly 70 laps using an empty buffer.Source: Figure 4 · p. 6, A. Policy Refinement Experimental… · p. 6

A. Policy Refinement Experimental Results

  • Three hours of real-car interaction improved both lap time and constraint violations after performance initially deteriorated and then recovered.The refined policy achieved very regular lap times and low constraint violations, emphasizing consistency over raw peak lap times.Source: A. Policy Refinement Experimental… · p. 6
  • Each performance point during refinement averaged the last 12 minutes of driving, corresponding to one battery interval.Figure 5 tracks the initial sim-to-real policy and the final refined policy as reference points.Source: Figure 5 · p. 6, A. Policy Refinement Experimental… · p. 6
  • 11.01 s average lap time and 0.44 s per lap of constraint violations were achieved after refinement, with violations reduced by 71% compared with the MPC.Refinement reduced mean lap time by 0.42 s and constraint violations by 0.63 s compared with the sim-to-real results; the policy remained close to MPC in lap time.Source: Figure 6 · p. 6, A. Policy Refinement Experimental… · p. 6

VI. CONCLUSION

The authors transfer SAC to miniature-car racing through model randomization and policy-output regularization, then refine the policy using real-car trajectories. The refined policy reaches lap times comparable to a state-of-the-art model-based controller while improving track-constraint violations by over three-fold.

  • Lap times became comparable to a state-of-the-art model-based controller, with an over three-fold improvement in track-constraint violations.Source: §VI · p. 6
  • Model randomization improves the learned policy’s generalization, while policy-output regularization smooths actions without affecting algorithm convergence.Source: §VI · p. 6
  • The policy trained in simulation can be further improved on the real car using asynchronously stored and sampled driving trajectories.Source: §VI · p. 6
  • The work applies the SAC algorithm to autonomous racing and validates its results experimentally on a miniature race track.Source: §VI · p. 6
Loading 2011.13332v2…