PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Drone Racing: Learning Agile Flight in Dynamic Environments

Elia Kaufmann, Antonio Loquercio, Rene Ranftl, Alexey Dosovitskiy, Vladlen Koltun, Davide Scaramuzza

arXiv:1806.08548v3cs.RO

TL;DR

Autonomous vision-based drone racing must handle perception, localization, and control challenges at high speed in dynamic environments. This paper pairs image-to-waypoint-and-speed prediction with a trajectory-control pipeline, and reports more robust and precise navigation than state-of-the-art engineered systems in simulation and physical experiments.

  • Problem

    Autonomous drone racing remains difficult because it spans dynamics modeling, onboard perception, localization and mapping, trajectory generation, and optimal control.

    Source: §1 · p. 2

  • Method

    A hybrid system maps raw images to waypoint and desired-speed commands with a CNN, then uses a trajectory-generation and control pipeline to execute them.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    In simulation and physical-quadrotor experiments, the system navigated complex tracks and dynamic, cluttered environments more robustly and precisely than state-of-the-art engineered systems.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The experiments show robust navigation can avoid drift inherent in systems that rely on global state estimates in complex, dynamic race-track settings.

    Source: §6 · p. 8

    The perception system requires substantial training data from the application environment, limiting fast adaptation to previously unseen environments.

    Source: §6 · p. 8

Abstract

from arXiv · show

Autonomous agile flight brings up fundamental challenges in robotics, such as coping with unreliable state estimation, reacting optimally to dynamically changing environments, and coupling perception and action in real time under severe resource constraints. In this paper, we consider these challenges in the context of autonomous, vision-based drone racing in dynamic environments. Our approach combines a convolutional neural network (CNN) with a state-of-the-art path-planning and control system. The CNN directly maps raw images into a robust representation in the form of a waypoint and desired speed. This information is then used by the planner to generate a short, minimum-jerk trajectory segment and corresponding motor commands to reach the desired goal. We demonstrate our method in autonomous agile flight scenarios, in which a vision-based quadrotor traverses drone-racing tracks with possibly moving gates. Our method does not require any explicit map of the environment and runs fully onboard. We extensively test the precision and robustness of the approach in simulation and in the physical world. We also evaluate our method against state-of-the-art navigation approaches and professional human drone pilots.

1 Introduction

The paper proposes a map-free, onboard vision-based drone-racing system that pairs CNN scene interpretation with trajectory planning and control. Experiments in simulation and on a physical quadrotor show robust, precise navigation in dynamic environments.

1 Introduction

  • Autonomous racing drones must address challenges across dynamics modeling, onboard perception, localization and mapping, trajectory generation, and optimal control.The resulting capabilities could also support disaster response, aerial delivery, and inspection of complex structures.Source: §1 · p. 2
  • Global-trajectory racing depends on accurate state estimates, while SLAM can fail under changed conditions or high acceleration and struggles with dynamic environments.Moving gates and obstacles conflict with navigation methods built around static waypoints and trajectories.Source: §1 · p. 2
  • A CNN maps raw images to a waypoint and desired speed, which a planner uses to generate a short trajectory segment and motor commands without an explicit map.All computation runs onboard, combining CNN perception with a state-of-the-art trajectory-generation and tracking pipeline.Source: §1 · p. 2
  • In simulation and on a physical quadrotor, the approach handles moving gates, racing, and severe occlusions with higher robustness and precision than state-of-the-art engineered systems.The system is compact and efficient enough to run entirely onboard.Source: §1 · p. 2

2 Related Work

The paper combines agile trajectories with learned perception to address the limits of high-speed, vision-based quadrotor flight, using image-only policies to navigate tracks that may contain moving gates.

2 Related Work

  • High-speed quadrotors face state-estimation drift from blur, lighting, and aliasing, while costly sensors, computation [22], and compressed-map drift hinder real-time use of agile trajectories.State-estimation methods are generally designed for predominantly static environments, making it difficult for resource-constrained platforms to exploit minimum-snap or minimum-jerk trajectories using onboard sensing and computing.Source: §2 · p. 2
  • Handcrafted gate detectors fail under occlusion, partial visibility, and blur [7], while visual servoing requires small image differences and remains vulnerable to occlusion and blur.Source: §2 · pp. 2–3
  • Prior end-to-end image-to-action methods avoid global maps and position estimates but face computational, speed, or 3D-generalization limits [27] [11] [24].Source: §2 · p. 3
  • Learned-control outputs trade robust but low-agility discrete commands [15] against agile direct control [14], which has high sample complexity.The passage identifies the optimal output representation for algorithms coupling perception and control as an open question.Source: §2 · p. 3
  • The paper combines agile trajectories with deep learned perception, using image-only policies supervised by global trajectory methods to fly resource-constrained quadrotors through tracks with moving gates.The learned policy does not require information about the system’s global state and can cope with dynamic environments.Source: §2 · p. 3

3 Method

The method combines an onboard CNN that predicts an image-space goal direction and desired speed from one camera image with a planner and controller that generate and track a minimum-jerk trajectory. It trains the perception network by imitation of an expert derived from global trajectories, using training data designed to support operation in dynamic environments.

3 Method

  • The onboard CNN maps a 300×200 RGB camera image to a normalized 2D goal direction and desired speed, processing roughly 10 frames per second.The network uses an efficient ResNet-based architecture.Source: §3 · p. 3
  • The controller back-projects the predicted goal along the camera ray to a depth proportional to predicted speed, then tracks a minimum-jerk interception trajectory using Faessler et al.’s control scheme.Predicted normalized speed is scaled by user-specified maximum speed to set desired speed, allowing flight aggressiveness to be adjusted without retraining.Source: §3 · p. 3

3.1 Training procedure

  • Training assumes known gate locations and accurate state estimates, but test-time flight requires no privileged information and relies on image data only.Training trajectories are globally optimal under the stated setup assumptions.Source: §3.1 · pp. 3–4
  • For imitation labels, an expert follows a minimum-snap global trajectory through all gates, projecting a goal point at prediction-horizon distance into the image and normalizing its reference speed.The global trajectory is computed with the minimum-snap implementation of Mellinger and Kumar.Source: §3.1 · p. 4
  • The training set pairs state estimates and camera images with expert labels; simulation uses DAgger-style expert recovery, while real-world data come from manually carried flights covering off-trajectory positions.The labeling procedure is agnostic to how samples are collected, and the network imitates the expert policy rather than demonstrations directly.Source: §3.1 · p. 4
  • The network is trained with weighted MSE on goal-point and speed predictions, using cross-validated γ = 0.1, although performance is mostly insensitive to that weight.The loss compares predicted image coordinates and speed with their ground-truth labels.Source: §3.1 · p. 4
  • Training jointly on varied static gate layouts lets the perception system generalize to dynamic tracks at test time and improves robustness.The layouts are generated by slightly moving gates from their initial positions, with a global reference trajectory generated for each layout.Source: §3.1 · p. 4

4 Experiments in Simulation

In simulation, the waypoint-and-speed approach was evaluated against end-to-end and visual-inertial baselines on static and moving-gate tracks. It completed a simple track that the end-to-end baseline could not, outperformed the VIO baseline on the complex track, and adapted to substantial gate motion.

4.1 Comparison to end-to-end learning approach

  • On the four-gate, 43-meter track, the approach completed the course while the image-to-body-rate baseline [21] never passed more than one gate, even with 35K versus 5K training samples.The authors attribute the approach’s sample efficiency and onboard feasibility to separating navigation from low-level stability control; they conducted no further tests of the baseline.Source: §4.1 · p. 5

4.2 Performance on a complex track

  • The approach performed better than the VIO baseline on the complex track, worked reliably up to 9 ms−1, and showed graceful degradation at higher speeds.The static-track VIO baseline failed because of accumulated drift; at higher speeds, body rates outstripped the network’s 30 Hz prediction frequency.Source: §4.2 · pp. 5–6, Figure 4 · p. 6

4.3 Generalization to dynamic environments

  • At 8 ms−1, the approach handled moving gates displaced by up to 1.5 times their diameter, while the VIO expert policy failed even for tiny gate motions.Performance degraded beyond that displacement as gates shifted into configurations requiring turns beyond the system’s navigation capabilities.Source: Figure 4 · p. 6, §4.3 · p. 6

5 Experiments in the Physical World

Physical-world experiments show robust narrow-gate navigation and stronger race-track performance than a visual-inertial baseline, while human pilots are faster but fail more often.

5 Experiments in the Physical World

  • In narrow-gate tests, the approach passed consistently across starting angles and could pass when the gate was partially visible, while the handcrafted detector failed when the gate was out of view or lost tracking under vibration.The comparison replaced the learned perception system with Falanga et al.’s handcrafted window detector [7], leaving the control system unchanged.Source: §5 · p. 7
  • The approach remained robust to gate occlusion of up to 50% in unseen configurations, while the handcrafted baseline broke down at moderate occlusion; performance dropped rapidly above 50%.Above 50% occlusion, the remaining gap was barely wider than the drone, and ambiguous partial gate edges could cause overcorrection near the gate.Source: §5 · p. 7, Figure 6 · p. 7

5.1 Experiments on a race track

  • The learning-based approach outperforms the visual-inertial baseline on the four-gate track, with higher robustness and speed because body-frame waypoints avoid state-estimation drift.The evaluation measures success rate over five consecutive laps without crashing and best lap time.Source: §5.1 · pp. 7–8, Figure 6 · p. 7
  • Both human pilots completed the track faster than autonomous systems, but their aggressive flight brought higher failure rates, including frequent crashes by the intermediate pilot on sharp turns.Human pilots also performed more agile sharp-turn maneuvers, which require environmental reasoning the autonomous system lacks.Source: Figure 6 · p. 7, §5.1 · p. 8
  • The system generalized to moving gates on the real track, although training data varied only one gate; traditional high-speed approaches assuming a static world could not be deployed.At test time, it coped with movement of any gate.Source: §5.1 · p. 8

6 Discussion

The hybrid vision-and-control system combines image-based waypoint and speed prediction with a classic control stack, enabling robust drone racing in complex, dynamic environments. The discussion highlights broader navigation potential alongside limits in appearance generalization, training data needs, and difficult trajectories.

6 Discussion

  • A compact CNN predicts waypoints and desired speed from raw images for a classic control stack to execute, following a global reference trajectory for agile flight.The approach combines machine-learning perception with the stability and speed of established control algorithms.Source: §6 · p. 8
  • Experiments on real drones and in simulation show robust navigation of complex tracks, avoidance of drift from global state estimates, and operation in dynamic, cluttered environments.The system navigated moving or partially occluded gates.Source: §6 · p. 8
  • The authors suggest hybrid learning-and-control systems could support robust robot navigation in dynamic environments where geometric mapping, localization, and planning have limitations.They argue such systems can benefit from near-optimal solutions to different subproblems.Source: §6 · p. 8
  • Scaling the approach may fail under substantial appearance shifts, require impractical application-specific training data, or lack an imitable policy for extremely tight turns.The authors propose few-shot learning for faster adaptation and deeper integration of learning into control to address these challenges.Source: §6 · p. 8

Supplementary Material

The supplementary material details trajectory generation, training-data collection, model selection, baselines, onboard hardware, and network attention. It explains how adaptive trajectory planning and task-focused visual input support the approach.

6.1 Trajectory generation

  • Global trajectories use gate-center waypoints and constrained thrust, body rates, and speed; receding-horizon state-interception segments adapt their goal distance to support smooth fast flight or agile turns.The global trajectory uses Mellinger et al.; test-time planning length is based on predicted velocity, with experiment-specific minimum and maximum lengths.Source: §6.1 · p. 11

6.2 Training data generation

  • Simulation training uses a modified DAgger procedure that alternates network control with expert recovery, widening the allowed trajectory margin as performance improves.The static-track experiment collected 20k images; the dynamic experiment generated 100k images with random gate positions.Source: §6.2 · p. 12
  • Real-world training data are collected in handheld mode by carrying the quadrotor through the track, allowing safe sampling near gates and obstacles without flying an untrained network.The experiments collected 25k static-track images, 15k additional images for dynamic gate positions, and 23k images for narrow-gap and occlusion experiments.Source: §6.2 · p. 12

6.3 Cross-Validation of loss weight γ

  • For γ = 0.1, the network completed the static simulated track for all tested γ values after 80 epochs, and γ = 0.1 gave the best results.Seven logarithmically spaced values in [0.0001,100] were tested at 8ms−1; the selected γ was used in further experiments.Source: §6.3 · p. 12, Figure 7 · p. 12

6.4 Baselines

  • The comparison includes an end-to-end body-rate regressor that could not complete even the simplest layout, a handcrafted detector baseline, and a noisy-state-estimate VIO baseline.The regressor’s labels came from the low-level controller in, while the detector baseline used [7]; the VIO baseline executes the expert policy with noisy state estimates.Source: §6.4 · p. 13

6.5 Platform

  • In real-world track experiments, an in-house quadrotor used an Intel UpBoard for flight calculations, including neural-network prediction, trajectory generation, and tracking, while a Qualcomm Snapdragon Flight Kit handled visual-inertial odometry; network inference ran at approximately 10Hz.Source: §6.5 · p. 13

6.6 Network architecture and Grad-CAM

  • The CNN maps 300 × 200 RGB images to a goal direction and desired speed, while Grad-CAM visualizations show attention focused mainly on gates rather than background.The attention visualization uses Grad-CAM [29].Source: §6.6 · p. 13, Figure 8 · p. 13

Related papers

Loading 1806.08548v3…