Source-linked AI summary
Deep Drone Racing: Learning Agile Flight in Dynamic Environments
Elia Kaufmann, Antonio Loquercio, Rene Ranftl, Alexey Dosovitskiy, Vladlen Koltun, Davide Scaramuzza
TL;DR
Autonomous agile drone racing is difficult because high-speed flight stresses state estimation and dynamic environments require tightly coupled perception and action. The paper combines a CNN that predicts local waypoints and speed with a classical trajectory-generation and control stack, demonstrating robust onboard navigation in simulation and on real drones. The approach remains limited by appearance changes beyond training, substantial environment-specific data requirements, and insufficient prediction frequency at high velocities.
Problem
The paper addresses robust autonomous quadrotor flight when high-speed state estimation is unreliable and environments are dynamic and resource constrained.
Method
A CNN maps raw camera images to a waypoint or local goal and desired speed, while a classical planner and controller generate and execute short trajectories.
Results
The system robustly navigates complex, dynamic, and cluttered tracks with severe occlusions in simulation and on real drones, entirely onboard.
Takeaways & Limitations
The approach demonstrates that learned perception can be integrated with classical control for agile vision-based flight without relying on global state estimates.
Takeaways & Limitations
Performance degrades when environment appearance changes substantially beyond training, training requires significant application-specific data, and 30 Hz prediction is insufficient at high velocities.
Abstract
from arXiv · showhide
Autonomous agile flight brings up fundamental challenges in robotics, such as coping with unreliable state estimation, reacting optimally to dynamically changing environments, and coupling perception and action in real time under severe resource constraints. In this paper, we consider these challenges in the context of autonomous, vision-based drone racing in dynamic environments. Our approach combines a convolutional neural network (CNN) with a state-of-the-art path-planning and control system. The CNN directly maps raw images into a robust representation in the form of a waypoint and desired speed. This information is then used by the planner to generate a short, minimum-jerk trajectory segment and corresponding motor commands to reach the desired goal. We demonstrate our method in autonomous agile flight scenarios, in which a vision-based quadrotor traverses drone-racing tracks with possibly moving gates. Our method does not require any explicit map of the environment and runs fully onboard. We extensively test the precision and robustness of the approach in simulation and in the physical world. We also evaluate our method against state-of-the-art navigation approaches and professional human drone pilots.
2 Intel Labs
The paper identifies drone racing and learning agile flight as its focus, illustrating an autonomous quadrotor navigating a track with moving gates.
- The paper focuses on drone racing and learning agile flight.
- An autonomous, vision-based quadrotor combines a CNN with trajectory generation and control to navigate moving-gate tracks with high agility.
1 Introduction
Autonomous racing drones must integrate perception, localization, trajectory generation, and control while operating at high speed in difficult environments. The paper proposes a map-free CNN and control pipeline and evaluates it in simulation and on a physical quadrotor.
- Drone-racing skills are valuable for autonomous systems operating in applications such as disaster response, aerial delivery, and complex-structure inspection.
- Developing fully autonomous racing drones is difficult because challenges span dynamics modeling, onboard perception, localization, mapping, trajectory generation, and optimal control.
- Accurate global-trajectory tracking requires precise state estimation, which can fail under changed conditions or high acceleration because of motion blur and feature loss.
- The proposed hybrid system maps raw images to a waypoint and desired speed, then uses a planning module to generate a short trajectory and motor commands without an explicit environment map.
- Experiments in simulation and on a physical quadrotor report robustness to highly dynamic environments and severe occlusions, with execution entirely onboard.
2 Related Work
Prior approaches struggle to combine agile flight, robust perception, and resource-efficient operation in dynamic, visually challenging environments. The paper combines learned image processing with classical trajectory and control methods to address this trade-off.
- High-speed flight makes state estimation vulnerable to motion blur, lighting changes, perceptual aliasing, expensive sensors, computational cost, and map-induced drift.
- Dynamic drone racing requires tightly coupling perception and action to find and localize the next gate while maintaining visual contact.
- Handcrafted gate detectors become unreliable under occlusion, partial visibility, and motion blur.
- End-to-end image-to-action methods vary from robust but low-agility discrete commands to agile but sample-intensive direct control, while also facing complexity, speed, and 3D-generalization limitations.
- The proposed combination uses deep networks for expressive raw-sensory perception and agile trajectories, while the learned policy operates without global-state information and can cope with dynamic environments.
3 Method
The method predicts a local goal direction and speed from a camera image, then converts these predictions into a minimum-jerk trajectory for low-level control. Training uses globally generated expert trajectories across varied static layouts to support dynamic-environment operation.
- Perception and control: A CNN predicts a local goal direction and normalized desired speed from a single forward-facing camera image.
- Perception and control: The control system back-projects the predicted image goal into 3D local coordinates and generates a minimum-jerk trajectory for tracking.
- Training procedure: Training uses imitation learning with globally optimal trajectories generated from known gate locations and accurate training-time state estimates, while testing uses image data only.
- Training procedure: The expert policy follows a global minimum-snap trajectory through all gates and supplies projected goal directions and normalized reference speeds.
- Training procedure: Training data pairs camera images and state estimates with expert outputs, and simulation uses a DAgger variant to recover from deviations during collection.
- Training procedure: A weighted MSE loss trains point and velocity predictions, with cross-validation selecting γ = 0.1 despite limited sensitivity to this parameter.
- Dynamic environments: Training jointly on multiple static layouts with slightly moved gates supports generalization to dynamic tracks and improves system robustness.
4 Experiments in Simulation
Simulation experiments compare the proposed approach with direct body-rate regression and VIO baselines on tracks of increasing complexity. The proposed system completes the simple track, operates reliably up to 9 ms−1 on the complex static track, and adapts to moving gates up to 1.5 times their diameter.
- Comparison to end-to-end learning approach: On the 43-meter, four-gate small track, the proposed approach always completed the track, whereas the direct body-rate regression baseline never passed more than one gate.The baseline remained unsuccessful even after training on 35K samples versus 5K for the proposed method.
- Comparison to end-to-end learning approach: The direct body-rate regression baseline was not evaluated further because it could not complete the simple track.
- Performance on a complex track: On the 116-meter, eight-gate complex track, the proposed approach reliably operated up to a maximum speed of 9 ms−1, while performance degraded at higher velocities.Task completion rate increases with passed gates and reaches 100% after five consecutive laps without crashing.
- Performance on a complex track: The VIO baseline performed worse than the proposed approach on both static and dynamic tracks, with static-track performance degrading rapidly as the success threshold increased.The reported static-track failure is attributed to accumulated drift.
- Generalization to dynamic environments: At a constant maximum velocity of 8 ms−1, the proposed approach handled moving gates with amplitudes up to 1.5 times the gates’ diameter without crashing, whereas VIO failed for tiny gate motions.Performance degraded beyond 1.5 times the gates’ diameter because shifted consecutive gates became too close in the flight direction.
5 Experiments in the Physical World
Physical-world experiments test the approach on narrow, partially occluded, cluttered, static, and moving gates, against handcrafted and state-estimation baselines as well as human pilots. The system remains robust under occlusion and dynamic gate motion, while humans achieve faster laps but incur more failures.
- Narrow-gate robustness: The approach consistently passed a narrow gate under varied initial conditions, including when the gate was only partially visible.The handcrafted detector failed when the gate was not entirely in view and sometimes lost tracking because of platform vibrations.
- Occlusion robustness: The learned system remained robust to gate occlusions covering up to 50% of the gate area, whereas the handcrafted baseline failed under moderate occlusion.Performance dropped rapidly beyond 50% occlusion because the remaining gap demanded very high precision and visual ambiguities caused overcorrection.
- Race-track evaluation: On a four-gate real track, the approach achieved higher robustness and speed than the visual-odometry baseline by generating waypoint commands in the body frame.The evaluation measured success as five consecutive laps without crashing and also recorded best lap time; visual-odometry drift caused poor high-speed performance.
- Human comparison: Professional and intermediate pilots completed the track faster than autonomous systems, but their aggressive flight produced higher failure rates.Human pilots performed more agile maneuvers in sharp turns, while the intermediate pilot frequently crashed there.
- Dynamic gates: The system generalized to moving gates on the physical track, even though training data changed the position of only one gate.Traditional high-speed approaches based on pre-planned trajectories could not be deployed when gates moved.
6 Discussion
The discussion presents a compact hybrid perception-and-control system that robustly navigates complex, dynamic, and cluttered environments without relying on global-state drift-prone behavior. It also identifies limits in environmental generalization, data requirements, and trajectory coverage.
- Approach: The system combines a CNN that predicts waypoints and speed from raw images with a classic control stack trained to follow a global reference trajectory.This combines machine-learning perception with the stability and speed of established control algorithms.
- Supported capabilities: Simulation and real-drone experiments show robust navigation of complex tracks, avoidance of global-state drift, and operation in highly dynamic and cluttered environments.The integrated system was evaluated extensively in both simulation and physical experiments.
- Broader implication: Hybrid learning systems may offer a solution for robust navigation in dynamic worlds, while geometric mapping, localization, and planning have inherent limitations there.The paper frames this as a broader implication rather than a demonstrated result beyond drone racing.
- Limitations: Performance degrades when environmental appearance differs substantially from training, and the current approach requires substantial application-specific training data.These constraints make rapid adaptation to previously unseen environments impractical in some scenarios.
- Limitations: The learner is also likely to fail when trajectory optimization cannot provide an imitable policy, such as in extremely tight turns.The paper suggests integrating learning more deeply into the control system as a possible remedy.
Supplementary Material
The supplementary material details trajectory generation, training-data collection, hyperparameter evaluation, attention analysis, baselines, and onboard deployment. The system adapts trajectory-segment length to predicted speed and uses distinct simulation and real-world data-generation procedures.
- Trajectory generation: Global trajectories use gate-center waypoints, maximum-velocity and vehicle-dynamics constraints, and generate ground-truth labels in simulation and real-world experiments.The trajectory implementation follows Mellinger et al. and permits nonconstant speed along the global trajectory.
- Trajectory generation: The navigation system repeatedly recomputes state-interception trajectory segments from the quadrotor’s current state to a CNN-predicted goal position.The current position, velocity, and acceleration define the start state, while the goal position constrains the end state.
- Trajectory generation: Long prediction horizons produce smoother high-speed flight, whereas short horizons enable more agile maneuvers near gates and in tight turns.The prediction horizon directly controls maneuver aggressiveness.
- Trajectory generation: At test time, the planning length is computed from the network’s predicted velocity, enabling both smooth high-speed flight and aggressive maneuvers without gate-position knowledge.Real-world experiments used md = 0.6s, dmin = 1.0m, and dmax = 2.0m; simulation used md = 0.5s, dmin = 2.0m, and dmax = 5.0m.
- Training data generation: Simulation training uses an automated DAgger procedure that alternates expert supervision and network actions within an expanding trajectory-deviation margin.Real-world data are instead collected in handheld mode, including 25k static-track images, 15k additional dynamic-gate images, and 23k images for narrow-gap and occlusion experiments.
- Evaluation and deployment: The evaluation compares the system with an end-to-end body-rate regressor, a handcrafted gate detector, and onboard hardware supporting approximately 10Hz network inference.The body-rate baseline failed to complete even the simplest track, while the learned system’s attention concentrates mainly on gates and ignores most background.