Source-linked AI summary
Autonomous Drone Racing with Deep Reinforcement Learning
Yunlong Song, Mats Steinweg, Elia Kaufmann, Davide Scaramuzza
TL;DR
Autonomous drone racing requires fast waypoint traversal, but existing time-optimal planners either assume advance track knowledge, specialize to layouts, simplify dynamics, or require costly computation. The paper uses deep reinforcement learning with relative gate observations, parallelized sampling, and a path-progress reward to generate adaptive trajectories. It achieves near-time-optimal performance, remains effective under track changes, and reaches speeds up to 60 km/h with a physical quadrotor.
Problem
Time-optimal quadrotor planning must handle coupled dynamics and actuator constraints, while existing approaches are layout-specialized, dynamically simplified, or computationally expensive.
Method
The method trains a neural-network policy using relative gate observations, highly parallelized sampling, and a three-dimensional path-progress reward.
Results
The system generates near-time-optimal trajectories, handles large track changes and online replanning, and reaches speeds of up to 60 km h−1 with a physical quadrotor.
Takeaways & Limitations
Deep RL can generate adaptive, computationally efficient quadrotor trajectories across deterministic, uncertain, and randomly generated racing environments.
Abstract
from arXiv · showhide
In many robotic tasks, such as autonomous drone racing, the goal is to travel through a set of waypoints as fast as possible. A key challenge for this task is planning the time-optimal trajectory, which is typically solved by assuming perfect knowledge of the waypoints to pass in advance. The resulting solution is either highly specialized for a single-track layout, or suboptimal due to simplifying assumptions about the platform dynamics. In this work, a new approach to near-time-optimal trajectory generation for quadrotors is presented. Leveraging deep reinforcement learning and relative gate observations, our approach can compute near-time-optimal trajectories and adapt the trajectory to environment changes. Our method exhibits computational advantages over approaches based on trajectory optimization for non-trivial track configurations. The proposed approach is evaluated on a set of race tracks in simulation and the real world, achieving speeds of up to 60 km/h with a physical quadrotor.
I. INTRODUCTION
Time-optimal quadrotor racing requires trajectories that respect coupled dynamics and actuator limits, while existing methods trade computational speed against modeling fidelity. The paper proposes deep reinforcement learning with relative gate observations to generate aggressive, adaptive trajectories efficiently.
- Motivation: Time-optimal racing trajectories must satisfy actuator constraints while balancing coupled linear and angular accelerations.Ignoring diminishing control authority can lead to catastrophic crashes.
- Related Work: Optimization-based planning can model true dynamics and constraints but may require hours of computation, whereas faster approximations sacrifice fidelity.Polynomial and point-mass approaches compute faster but do not fully account for quadrotor dynamics.
- Motivation: Deep reinforcement learning is proposed for complicated trajectory-planning problems that are difficult to model directly.Model-free policy search learns neural policies through environment interaction and supports flexible state representations.
- Contribution: The learned system generates aggressive trajectories close to state-of-the-art time-optimal solutions and adapts to deterministic, uncertain, and randomly generated tracks.The experiments include large track changes and online replanning, although deep RL sacrifices trajectory-optimization performance guarantees.
- Contribution: The approach uses relative gate observations, highly parallelized sampling, and a three-dimensional path-progress reward that proxies lap-time minimization.These are identified as the three key ingredients of the method.
- Related Work: Deep RL addresses quadrotor planning in a higher-dimensional state and input space than prior RL navigation work focused mainly on wheeled robots.The paper frames this larger search space as a central challenge for time-optimal quadrotor trajectories.
III. METHODOLOGY
The quadrotor is modeled as a six-degree-of-freedom rigid body whose state evolves under thrust, torque, gravity, and rotational dynamics. Rotor thrusts are converted into mass-normalized thrust and body torques for numerical simulation.
- A. Quadrotor Dynamics: The dynamics model represents the quadrotor as a 6 degree-of-freedom rigid body with mass m and diagonal inertia matrix J.The model captures translational and rotational motion.
- A. Quadrotor Dynamics: The translational and angular accelerations depend on the thrust vector, gravity, applied torque, inertia, and gyroscopic coupling.The angular dynamics include the term ωB × JωB.
- A. Quadrotor Dynamics: The mass-normalized thrust vector c and body torque η are obtained from the four individual rotor thrusts.The conversion uses the rotor torque constant κ, arm length l, and quadrotor mass m.
- A. Quadrotor Dynamics: The full state is x = [pW B, qW B, vW B, ωB], and the control input is u = [f1, f2, f3, f4].The state includes position, orientation, velocity, and body rates.
B. Task Formulation
The task is formulated as deep reinforcement learning for time-optimal quadrotor trajectory planning, using dense progress and safety-related rewards to guide flight through three-dimensional gates.
- B. Task Formulation: The task is modeled as an infinite-horizon Markov Decision Process with states, actions, transitions, rewards, an initial-state distribution, and a discount factor.
- B. Task Formulation: The neural-network policy is trained to maximize expected discounted reward, with the discount factor trading off long-term and short-term rewards.
- B. Task Formulation: The trained policy produces the trajectory by being rolled out through the environment.
- 1) Reward Function:: Total lap time is the true objective, but its terminal-only feedback is sparse, so a per-step proxy reward is used for credit assignment.
- 1) Reward Function:: The track center-line is represented by straight segments connecting adjacent three-dimensional gate centers, avoiding additional reference-path computation.
- 1) Reward Function:: At each step, the quadrotor is associated with the segment toward the next gate, and its position is projected onto that segment to compute path progress.
- 1) Reward Function:: The progress reward is the change in projected path position between consecutive quadrotor states.
- 1) Reward Function:: The optional safety reward penalizes small margins near the gate to reduce crash risk during training with large track changes.
2) Observation and Action Spaces:
The observation space combines quadrotor state information with relative observations of future gates, using representations intended to describe motion, attitude, distance, and flight direction.
- 2) Observation and Action Spaces:: The observation space contains one component for the quadrotor state and another for race-track information.
- 2) Observation and Action Spaces:: The quadrotor-state representation includes linear velocity, linear acceleration, rotation matrix, and angular velocity.
- 2) Observation and Action Spaces:: Rotation matrices represent attitude to avoid orientation ambiguities and discontinuities.
- 2) Observation and Action Spaces:: The track observation vector lists each future gate’s observation together with the angle between its gate normal and the vector from the quadrotor to the gate center.
- 2) Observation and Action Spaces:: Gate positions use spherical coordinates, separating distance to the gate from flight direction.
- 2) Observation and Action Spaces:: The next gate is expressed in a body-centered frame, while later gates are recursively expressed in the frame of the preceding gate.
C. Policy Training
The agent is trained with PPO, whose simplicity and benchmark performance motivate its use despite the high-dimensional search space and complex maneuvers required here.
- C. Policy Training: The agent is trained using Proximal Policy Optimization, a first-order policy-gradient method selected for benchmark performance and implementation simplicity.
- C. Policy Training: The task challenges PPO because quadrotor racing involves a high-dimensional search space and complex required maneuvers.
1) Parallel Sampling Scheme:
Parallel simulation accelerates data collection and supports diverse exploration, while distributed initialization prevents early crashes from restricting rollouts to the track’s starting region.
- 1) Parallel Sampling Scheme:: Simulation supports policy rollouts in up to 100 environments in parallel, speeding data collection.
- 1) Parallel Sampling Scheme:: Parallelization increases interaction diversity and can distribute rollouts across long or strongly randomized tracks.
- 1) Parallel Sampling Scheme:: Early training rollouts often crash at a gate, restricting data to a small state-space region when all quadrotors start from the same position.
- 1) Parallel Sampling Scheme:: A distributed initialization strategy ensures uniform exploration of relevant state-space areas during early learning.
3) Random Track Curriculum:
The approach trains racing policies on progressively more complex randomly generated tracks, using automatic curriculum adaptation to maintain learning progress. Parallel simulation supports large-scale training and evaluation across diverse track configurations.
- Random tracks are generated by concatenating gate primitives parameterized by relative position and orientation, allowing arbitrary complexity and length.
- Training begins with tracks close to a straight line and increases sampling complexity according to the crash ratio across parallel environments.
- This adaptive sampling acts as an automatic curriculum tailored to the agent’s current capabilities.
- Experiments cover deterministic benchmark tracks, uncertain and randomly generated layouts, ablations, and physical-quadrotor execution.
- The vectorized Flightmare environment simulates hundreds of quadrotors and collects up to 25000 environment interactions per second during training.
B. Baseline Comparison on Deterministic Tracks
On three deterministic race tracks, the learned policy approaches the trajectory-optimization reference while outperforming polynomial trajectory generation. The subsequent uncertainty experiment tests whether this performance persists when gate positions and orientations vary.
- Baseline comparison: The evaluation compares the learned policy with minimum-snap polynomial generation and CPC optimization on three deterministic tracks.
- Baseline comparison: Within 5.2 % of the theoretical limit, the proposed approach approaches CPC performance, while polynomial generation is significantly slower.
- Baseline comparison: The deterministic-track comparison uses lap time in seconds as the reported metric.
- Handling track changes: The uncertainty study randomly displaces each AlphaPilot gate’s position and horizontal-plane yaw within bounded limits.
- Handling track changes: 97.5 % success rate is achieved on the most difficult test set with maximum gate displacements, while nominal-track lap time decreases by 2.21 %.
D. Towards Learning a Universal Racing Policy
The paper evaluates whether policies trained on completely random tracks can generalize to unseen layouts. Automatic track adaptation and safety rewards improve robustness, yielding high success across 1000 randomly generated tracks.
- Universal policy: The universal-policy experiment trains policies on completely random track layouts and evaluates scalability and generalizability on unseen tracks.
- Universal policy: Three policies differ by whether automatic track adaptation and the safety reward are used.
- Universal policy: Fig. 7 visualizes evaluation trajectories spanning 110 m to 150 m with elevation changes up to 17 m.
- Universal policy: The final evaluation uses 1000 randomly sampled tracks of full complexity, with performance assessed by gate crashes.
- Universal policy: Adding automatic track adaptation reduces task complexity early and improves performance over training without adaptation or safety reward.
- Universal policy: 97.4 % success rate is achieved on 1000 randomly generated tracks, indicating strong generalizability and scalability.
E. Computation Time Comparison and Ablation Studies
The study compares computational cost and ablates observation and reward design choices. More future-gate information matters under track randomization, while safety rewards trade nominal speed for fewer crashes and larger margins.
- Computation time: Computation time is compared between reinforcement learning and trajectory optimization across fixed and randomly generated tracks with varying gate counts.
- Ablation studies: The ablations examine future-gate observation models and the safety reward.
- Gate observations: On deterministic tracks, final lap time is not significantly affected by observing one, two, or three future gates.
- Gate observations: With gate displacements, observing only one future gate is insufficient for fast lap times and low crash ratios.
- Safety reward: The safety reward increases average safety margins and consistently reduces test-set crashes, but both configurations reduce nominal-track performance.
F. Real-world Flight
The learned policy was validated on a physical quadrotor, generating aggressive Split-S trajectories at speeds up to 60 km h−1. Execution produced large tracking errors, motivating improved trajectory tracking.
- 60 km h−1 was achieved during aggressive physical flight on the Split-S track.A deterministic policy rollout over four continuous laps generated the reference trajectory, which was tracked by a model predictive controller.
- Large tracking errors occurred when the generated trajectory was executed on the physical quadrotor.The authors identify improving time-optimal trajectory tracking as future research.
- The study validated near-time-optimal, multi-gate trajectory generation with a physical quadrotor while retaining aggressive flight capability.The conclusion reports speeds up to 60 km h−1 during physical validation.