PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Drone Racing: From Simulation to Reality with Domain Randomization

Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, Davide Scaramuzza

arXiv:1905.09727v2cs.RO

TL;DR

Small autonomous drones must navigate dynamic tracks despite unreliable global state estimates and constrained onboard computation. This paper combines CNN-based local waypoint and speed prediction with classical planning and control, training perception with randomized simulation data for deployment on a physical platform. Experiments evaluate precision, speed, and generalization across simulation and physical flight, including complex, dynamic, and cluttered tracks.

  • Problem

    Autonomous drone racing requires navigation through complex, dynamic tracks while global state estimates can introduce drift and onboard computation is constrained.

  • Method

    A compact CNN predicts local body-frame waypoints and desired speed from onboard-camera images, after which a planner and tracker generate trajectory segments and motor commands.

  • Results

    Experiments in simulation and on a physical platform show navigation of complex race tracks, drift avoidance, and operation in highly dynamic and cluttered environments.

  • Takeaways & Limitations

    Domain-randomized simulated data can train a perception system that adapts to drastic appearance changes and deploys on a physical platform without real-track training.

  • Takeaways & Limitations

    Scaling beyond drone racing remains challenging because general applications may have unknown path characteristics, including paths that cannot be defined a priori.

Abstract

from arXiv · show

Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges that limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art planning and control system with the perceptual awareness of a convolutional neural network (CNN). The resulting modular system is both platform- and domain-independent: it is trained in simulation and deployed on a physical quadrotor without any fine-tuning. The abundance of simulated data, generated via domain randomization, makes our system robust to changes of illumination and gate appearance. To the best of our knowledge, our approach is the first to demonstrate zero-shot sim-to-real transfer on the task of agile drone flight. We extensively test the precision and robustness of our system, both in simulation and on a physical platform, and show significant improvements over the state of the art.

I. INTRODUCTION

Autonomous vision-based drone racing must handle complex tracks, dynamic gates, unreliable state estimation, and onboard resource constraints. The paper combines CNN-based local waypoint prediction with planning and control, using domain randomization to improve generalization and enable simulation-only training.

  • Autonomous racing drones must navigate complex tracks at high speed despite challenges in dynamics modeling, perception, localization, mapping, trajectory generation, and control.
  • Global trajectory tracking depends on prior track-layout knowledge and highly accurate state estimation, while drift and computational demands limit existing methods.
  • The proposed system predicts local body-frame waypoints from images, then uses a planner and tracker to generate trajectory segments and motor commands fully onboard.
  • Domain randomization replaces track-specific real-data collection by varying illumination, viewpoint, gate appearance, and background during simulated data generation.
  • Simulation-trained perception transfers to a physical quadrotor without fine-tuning, while achieving comparable real-track performance to perception trained with real data.

II. RELATED WORK

Prior approaches address autonomous flight through visual servoing, end-to-end learning, or modular perception and control, but face limitations in robustness, agility, or data requirements. This work makes neural perception compatible with fast model-based planners and trackers.

  • High-speed autonomous flight remains difficult because unreliable state estimation strains methods that depend on onboard sensing, mapping, localization, and computation.
  • Unlike prior work, the neural perception output is compatible with fast and accurate model-based trajectory planners and trackers, enabling agile flight.
  • Image-based visual servoing uses hand-crafted target detectors, which become unreliable under occlusion, partial visibility, and motion blur.
  • End-to-end imitation-learning policies can operate without state-estimation drift but generally predict low-level controls and require sufficient training data.
  • Modularization combines CNN perceptual awareness for navigation commands with classic control pipelines for fast and reliable low-level control.

C. Transfer from Simulation to Reality

The system transfers navigation policies from simulation to physical drone racing by combining domain randomization with modular perception and control. Its CNN predicts navigation direction and speed, which the control system converts into local-frame trajectory commands.

  • Simulation training addresses the high cost of collecting and annotating navigation data in the physical world.
  • Domain randomization and modularization are used to increase generalization and achieve sim-to-real transfer for drone racing.
  • The perception CNN predicts a local-image goal direction and desired navigation speed from a single forward-facing camera image.
  • 15 frames per second is the CNN inference rate while running concurrently with the full control stack.The network is designed to balance accuracy and inference time for onboard computing.
  • The control system back-projects the predicted image goal along the camera ray into three-dimensional local coordinates at a prediction-horizon depth.

A. Training Procedure

The system trains a convolutional perception module from expert trajectory supervision, using simulation procedures and visual randomization to support deployment under changing conditions.

  • Expert policy: The expert policy projects a quadrotor’s position onto a global trajectory and labels a forward goal position, image direction, and normalized speed.The desired speed is the reference-trajectory speed at the projected position, normalized by the maximum trajectory speed.
  • Generating data in simulation: Simulation training uses an automated DAgger procedure that alternates network training with expert correction when actions leave a margin around the global trajectory.The margin increases as the network completes the track with fewer than 50 expert actions.
  • Generating data in simulation: The simulation procedure collected 20k images for static-track experiments and 100k images with random gate positions for dynamic experiments.The dynamic dataset varies gate positions during data generation.
  • Generating data in the real world: Real-world data collection avoids DAgger for safety and instead uses handheld mode, with the expert policy generating samples while the quadrotor is manually carried through the track.Onboard state-estimation drift limits collection to small track portions before reinitialization.
  • Loss function: The perception network is trained with a weighted MSE loss on normalized goal-point and speed predictions, with γ = 0.1 selected by cross-validation.Performance was mostly insensitive to the weighting parameter.
  • Sim-to-real transfer: Domain randomization varies background, floor and gate textures, gate shapes, and lighting, while testing uses held-out gate configurations and common indoor or outdoor environments.Simulation uses pools of 30 background/floor textures, 10 gate textures, and 6 gate shapes.

B. Trajectory Generation

Trajectory generation converts perception outputs or trajectory-relative positions into short, receding-horizon segments whose planning length adapts to track geometry and predicted speed.

  • Global trajectory: The global trajectory is generated through gate waypoints under maximum-velocity, thrust, and body-rate constraints.Its speed is not constant.
  • Receding-horizon control: Each local trajectory segment starts from the quadrotor’s current position, velocity, and acceleration, while its end state is constrained only by the goal position.Only the first part is executed before recalculating a new segment.
  • Prediction horizon: Shorter prediction horizons reduce lateral deviation but produce more aggressive maneuvers, whereas longer horizons suit straight fast sections.Short horizons perform better in tight turns and near gates.
  • Test-time planning length: At test time, the network’s predicted direction is back-projected along the camera ray, and its velocity output determines the planning length because gate positions are unavailable.This adaptation targets smooth high-speed flight and aggressive turns.
  • Prediction horizon: During training, the prediction horizon depends on distances to the last and next gates, using their minimum to avoid jumps after gate passes.The minimum horizon is 1.5m on the simulated track and 1.0m on the real track.
  • Test-time planning length: The simulated experiments use md = 0.5s, dmin = 2.0m, and dmax = 5.0m for the planning-length function.The real-world settings are md = 0.6s, dmin = 1.0m, and dmax = 2.0m.

IV. EXPERIMENTS

The evaluation tests the system’s building blocks in simulation and then assesses real-quadrotor control, including comparisons with state-of-the-art methods and human pilots.

  • Simulation experiments: The experiments first isolate the convolutional architecture and perception-control modularization in a controlled simulated environment.This tests the main building blocks before physical-platform evaluation.
  • Physical-platform experiments: A second experiment set evaluates the approach on a physical quadrotor and compares it with state-of-the-art methods and human drone pilots.The comparisons include pilots with different levels of experience.

A. Experimental Setup

Experiments use Gazebo-based simulation of an AscTec Hummingbird in a bounded 70-meter environment containing small and large race tracks, alongside a constrained real-world quadrotor platform.

  • Simulation platform: Gazebo with RotorS models quadrotor physics for all simulation experiments.The simulated vehicle is an AscTec Hummingbird equipped with a forward-looking 300 × 200 RGB camera.
  • Simulation environment: The simulated flying space is a 70-meter cube containing the experiment-specific race track.Background and floor textures are randomized in the relevant simulation experiments.
  • Track layouts: The large track has 8 gates at different heights and spans 116 meters, while the small track has 4 gates over 43 meters.The large layout is used for most experiments; the small track supports comparison with end-to-end navigation policies.
  • Physical platform: Real-world experiments use an in-house quadrotor with an Intel UpBoard for onboard computation and a Qualcomm Snapdragon Flight Kit for visual-inertial odometry.The UpBoard runs neural-network prediction, trajectory generation, and tracking.

B. Experiments in Simulation

Simulation experiments evaluate modularization, high-speed performance, dynamic-gate handling, environmental generalization, and planning-parameter sensitivity against baselines and controlled variations.

  • Complex-track performance: The VIO baseline performs worse than the proposed approach on the complex static track because accumulated drift increasingly harms completion as the success threshold rises.Performance is comparable for one successful lap but degrades rapidly when success requires more laps.
  • Complex-track performance: The approach works reliably up to 9 ms−1 and degrades gracefully at higher velocities.At higher speeds, larger body rates and limited 30 Hz prediction frequency reduce the ability to handle rapid roll and pitch changes.
  • Dynamic environments: The proposed system handles moving gates up to 1.5 times the gate diameter at 8 ms−1, while the VIO baseline fails for motions up to 50% of the gate diameter.Performance degrades beyond 1.5 times the gate diameter because shifted consecutive gates can require extremely sharp turns.
  • Environmental generalization: Comprehensive domain randomization improves robustness to unseen backgrounds, gate appearances, and illumination conditions.Randomizing all properties simultaneously is crucial for good transfer, while omitting background randomization causes failure on even a single gate pass.
  • Simulation-to-reality transfer: The policy transfers from simulation to a real quadrotor without fine-tuning despite substantial visual differences between domains.The real-world control evaluation follows simulation training with randomized illumination, gate appearance, and background.
  • Planning-length sensitivity: Performance remains strong across a broad range of planning lengths, with sharp degradation only for very short or very long configurations.The sensitivity study varies dmin from 1.0m to 5.0m and dmax from dmin + 1.0m to dmin + 5.0m at 10.0ms−1.

C. Analysis of Accuracy and Efficiency

The analysis examines how CNN capacity and planning-length parameters affect closed-loop performance under onboard computation constraints.

  • CNN efficiency: Larger CNNs provide greater representational capacity but reduce inference rates, which can decrease agility.The neural network is the main computational bottleneck, and real-time operation requires relatively small CNNs.
  • Planning-length sensitivity: Planning-length sensitivity is evaluated by varying dmin and dmax while holding maximum speed and static track layout constant.The experiment reports average task completion over five runs for each configuration.
  • CNN efficiency: A network with capacity 1.5 outperforms capacity 2.0 at high speeds despite the lower-capacity model’s better generalization loss.Validation loss does not always predict closed-loop navigation performance.
  • CNN efficiency: A capacity factor of 0.5 is selected for new sim-to-real experiments because it eases computation while matching capacity 1.0 at 2 ms−1.The earlier conference version used capacity 1.0 as a favorable time-accuracy trade-off.

D. Experiments in the Real World

Real-world experiments evaluate precision, robustness, and multi-gate racing against handcrafted, visual-inertial, and human baselines. The system remains robust to initialization, occlusion, and dynamic gates, while human pilots retain an advantage in sharp turns.

  • Evaluation: The physical-quadrotor evaluation compares the proposed system with classic navigation methods and human pilots of different skill levels.The experiments assess the approach in real-world racing conditions.
  • Narrow gate passing: The proposed approach is not sensitive to the quadrotor’s initial position when passing through a narrow gate.Success rate was measured across different starting angles, including cases where the gate was initially invisible.
  • Narrow gate passing: The learning approach remains robust to gate occlusions up to 50%, whereas the handcrafted baseline breaks down at moderate occlusion.At more than 60% occlusion, little space remains for the platform to pass through the gap.
  • Experiments on a race track: The system outperforms the visual-inertial odometry baseline in robustness and speed on the four-gate race track.The learning-based system is insensitive to state-estimation drift because it generates navigation commands in the body frame.
  • Experiments on a race track: Human pilots perform more agile maneuvers than the autonomous systems, especially through sharp turns.The intermediate pilot experienced frequent crashes on the sharp turns, while the system still lacks comparable environmental reasoning.

E. Simulation to Real World Transfer

The navigation policy transfers directly from randomized simulation to a physical quadrotor without fine-tuning. Simulation-trained policies match real-data policies under familiar illumination and are more robust under difficult illumination.

  • Direct transfer: A policy trained in simulation with randomized illumination, gate appearance, and background controls a real quadrotor without fine-tuning.The transfer succeeds despite large visual differences between the simulated and real environments.
  • Quantitative comparison: Simulation-trained and real-data-trained policies achieve comparable task completion under easy and medium illumination.The real policy was trained under those illumination conditions.
  • Quantitative comparison: Under difficult illumination, the simulation-trained policy outperforms the real-data-trained policy.The real policy was never exposed to that degree of illumination change during training.
  • What is important for transfer?: The transfer study uses approximately 10K annotated real-world images collected across three indoor environments and varied illumination conditions.These images support the real-world evaluation of randomization factors.
  • What is important for transfer?: Illumination randomization has the greatest effect among the tested randomization factors, while gate-shape randomization has the smallest effect.The ablation evaluates prediction RMSE on real-world images; background randomization is excluded because it is treated as fundamental for transfer.

V. DISCUSSION AND CONCLUSION

The paper combines compact learned perception with classical planning and control to support agile, robust drone racing. Its main scope boundaries concern ambiguous navigation, cumulative modular errors, and scaling beyond racing.

  • Contribution: A compact CNN predicts waypoints and speed from raw images, while a classic planning and control pipeline executes those directions.This modular design combines learning-based perceptual awareness with model-based control precision and speed.
  • Results: Experiments show navigation of complex tracks, avoidance of global-state drift, and operation in highly dynamic and cluttered environments.The evaluation spans precision, speed, and generalization in simulation and on a physical platform.
  • Sim-to-real transfer: Domain randomization enables deployment on a physical platform after training entirely in simulation and improves robustness to environmental changes over real-data training.The paper reports higher robustness to changes such as illumination conditions.
  • Limitations: When ambiguous cases cannot be resolved by the learned high-level map prior, such as on an 8-shaped track, the sim-to-real approach would likely fail.Fine-tuning with real-track data or using a metric prior are proposed as possible solutions.
  • Limitations: Modularity causes errors from individual submodules to degrade full-system performance cumulatively.The authors propose reinforcement learning to improve component robustness in challenging scenarios such as moving obstacles.
  • Limitations: Scaling beyond drone racing is challenging because unknown paths hinder teacher-policy generation and sharp-turn agility remains limited.The discussion identifies few-shot learning and deeper integration of learning as possible directions.

APPENDIX

The appendix specifies the convolutional perception network, its training setup, and how its visual attention is analyzed for robustness.

  • Training Evaluation: Training sensitivity to the loss weighting factor γ was evaluated across seven logarithmically spaced values from 0.0001 to 100.Each model was trained for 100 epochs on a static simulated track and tested at 8ms−1 after each epoch.
  • Network Architecture: The network processes normalized 300 × 200 RGB images at 30Hz through seven convolutional layers, three residual blocks, and a fully connected output layer.It outputs a direction vector x⃗ and normalized desired speed v.
  • Network Architecture: The output tuple {x⃗,v} encodes the new goal direction in normalized image coordinates and the desired approach speed.x⃗ ∈[−1,1]2 and v ∈[0,1].
  • Grad-CAM: Grad-CAM analysis indicates that the network bases decisions primarily on gates while largely ignoring the background.This visualization is used to examine robustness to previously unseen environmental changes.
  • Real-World Evaluation: Zero-shot real-world generalization was quantified using approximately 10k images collected across three indoor environments with varied dimensions, appearance, and illumination.Illumination varied through natural intraday changes and artificial light sources.
Loading 1905.09727v2…