Source-linked AI summary
Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning
Yunlong Song, Angel Romero, Matthias Mueller, Vladlen Koltun, Davide Scaramuzza
TL;DR
The paper asks how to design agile-robot control systems and whether RL’s advantage over OC comes from optimization method or objective. It compares RL and state-of-the-art OC in autonomous drone racing, finding that RL’s task-level objective supports more robust control under model mismatch. The resulting policy reached greater than 12 g acceleration, 108 km/h velocity, and superhuman racing performance after minutes of simulation training.
Problem
The paper investigates why reinforcement learning has achieved successes beyond existing optimal-control systems and whether the difference lies in optimization method or optimization objective.
Method
The study compares a simulated-trained RL policy with two state-of-the-art optimal-control methods for autonomous drone racing and analyzes their optimization methods and objectives.
Results
RL outperformed state-of-the-art OC in autonomous drone racing, achieving greater than 12 g peak acceleration, 108 km/h peak velocity, and superhuman performance against three professional pilots.
Takeaways & Limitations
Directly optimizing the task goal allowed RL to avoid planning-control decomposition and produce more robust control under unmodeled dynamics.
Takeaways & Limitations
The theoretical time-optimal trajectory was computed using an approximate model, and its true physical-vehicle optimum remains unknown.
Abstract
from arXiv · showhide
A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective. OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour. Our policy achieved superhuman control within minutes of training on a standard workstation. This work presents a milestone in agile robotics and sheds light on the role of RL and OC in robot control.
INTRODUCTION
This paper systematically compares reinforcement learning and optimal control for autonomous drone racing, asking whether their performance gap arises from optimization method or objective. It argues that RL’s task-level objective enables more robust behavior under unmodeled dynamics and reports record-level racing performance.
- OC and RL: OC uses an explicit mathematical model and online optimization, whereas RL learns a policy by maximizing reward through trial-and-error interaction.Examples of OC include LQR and MPC; RL policies can be optimized offline for efficient deployment.
- Optimization objectives: OC commonly decomposes control into trajectory planning and trajectory tracking, separating the control objective from the high-level task objective.This layered interface improves interpretability and real-time pipeline simplicity but restricts how task goals reach the controller.
- Motivation: Autonomous drone racing tests control systems under extreme speed, acceleration, low error tolerance, and catastrophic crash risk.The task requires flying through ordered gates in minimum time while pushing the aircraft toward its physical limits.
- Study question: RL outperforms OC because it optimizes a better task-level objective rather than optimizing its objective more effectively.The study investigates optimization method and objective separately, with robust task performance defined under uncertainty and disturbances.
- Results: 108 km/h peak speed and greater than 12 g peak acceleration were achieved by the RL controller in an indoor flying arena.The policy was trained purely in simulation within minutes on a standard workstation and transferred zero-shot to the real world.
RESULTS
The study compares reinforcement learning with two optimal-control methods in simulation and the physical world, then isolates optimization objective as the key factor behind RL’s robustness. RL achieved high-speed, reliable racing despite model mismatch, while trajectory-based control degraded under unmodeled dynamics.
- RL versus OC: RL was compared with offline Trajectory Tracking and online Contouring Control, both using nominal-model optimization.Trajectory Tracking uses offline time-optimal planning with MPC tracking; Contouring Control jointly optimizes path progress and deviation online.
- RL versus OC: 4.92 s: Trajectory Tracking achieved the best nominal-model average lap time but only a 44% success rate.Contouring Control achieved 5.03 s and 76% success, while RL achieved 5.14 s and 100% success under the nominal model.
- RL versus OC: 0% and 20%: Trajectory Tracking and Contouring Control success rates under realistic dynamics, versus 100% for RL.The realistic model introduced a mismatch between optimization or training dynamics and test dynamics; RL also achieved the best lap time in this condition.
- Optimization Method versus Optimization Objective: 4.9 s: RL matched the optimal nominal-model lap time when trained with MPC’s trajectory-tracking objective, but did not outperform MPC on tracking loss.Both methods suffered performance drops with realistic dynamics, showing that changing the optimizer alone did not explain RL’s robustness.
- Optimization Method versus Optimization Objective: The gate-progress objective produced safer, more central gate crossings and robust performance, unlike the more aggressive trajectory-tracking objective.Gate progress enabled adaptation without a fixed reference trajectory; trajectory tracking failed with realistic dynamics despite performing well nominally.
- Physical-world performance: 108 km/h and 12.58 g: the physical drone reached these maximum velocity and acceleration values while completing 15 laps with a 100% success rate.The policy maintained control despite battery-voltage-induced thrust drops, including an especially large drop during one trial.
DISCUSSION
The study attributes RL’s advantage primarily to its task-level optimization objective, which supports adaptive behavior under uncertainty. This enabled a simulation-trained policy to reach physical performance limits in real-world drone racing.
- DISCUSSION: RL’s key advantage is optimizing a task-level objective that avoids planning-control decomposition through an explicit trajectory or path.This broader objective allows adaptive behaviors that improve robustness to unmodeled dynamics and effects.
- DISCUSSION: Domain randomization trains RL across varied dynamic environments, improving robustness to disturbances and model uncertainty.The policy becomes more adaptable by encountering simulated variations during training.
- DISCUSSION: 12 g maximum thrust-to-weight ratio supported a two-layer neural policy trained purely in simulation and transferred zero shot to the real world.The policy was trained within minutes on a standard workstation and consistently outperformed three human pilots.
- DISCUSSION: 108 km/h peak velocity and greater than 12 g peak acceleration demonstrate the RL policy’s real-world performance at the drone’s physical limits.The experiments also report 100% success across trials and throttle use reaching the platform’s physical limit.
- DISCUSSION: The work is presented as a milestone toward autonomous mobile systems achieving peak performance in the physical world.Its broader contribution is clarifying how RL and OC differ in robot control.
MATERIALS AND METHODS
The study formulates autonomous drone racing as constrained optimization and compares trajectory-, path-, and gate-progress objectives with MPC and policy-gradient methods. It examines whether RL's advantage arises from optimization method or objective, emphasizing direct task-level objectives over planner-controller decompositions.
- Problem formulation: Autonomous drone racing is modeled as a constrained discrete-time dynamical system requiring ordered gate passage, collision avoidance, and physically feasible thrust.The optimization produces feasible sequences of states and controls while respecting the system dynamics and constraints.
- Optimization objectives: Trajectory Tracking and Contouring Control first optimize an offline time-optimal trajectory, then track it online; Contouring Control instead tracks a path with flexible traversal timing.Contouring Control minimizes contouring error while maximizing traveled distance along the path, making it less restrictive than trajectory tracking.
- Optimization objectives: Gate Progress Eq. (5) removes the high-level trajectory or path planner and directly maximizes progress toward the next gate center.The objective penalizes body rate and collisions and rewards finishing the race.
- Optimization methods: MPC minimizes a finite-horizon cost online, outputs a predicted trajectory, executes only the first command, and repeats after each state transition.MPC uses an analytical dynamics model and feedback to address uncertainty and disturbances, but exact real-world models are often unattainable.
- Optimization methods: Policy gradient performs optimization offline from sampled trajectories, and its policy update does not require the transition model inside the optimization.The method can therefore cope with model uncertainty through sampled trajectories and a stochastic policy representation.
- Hypotheses: The study tests whether RL outperforms OC because it finds better solutions for the same objective or can optimize broader task-level objectives.The objective hypothesis links RL's flexibility to adaptive control sequences, whereas OC commonly decomposes planning and control around explicit reference trajectories or paths.