Learning High-Speed Flight in the Wild

Antonio Loquercio; Elia Kaufmann; René Ranftl; Matthias Müller; Vladlen Koltun; Davide Scaramuzza

Source-linked AI summary

Learning High-Speed Flight in the Wild

Antonio Loquercio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza

arXiv:2110.05113v1Oct 11, 2021cs.ROcs.LGeess.SY

TL;DR

Autonomous quadrotor flight through complex environments remains difficult because onboard perception and planning must operate robustly at low latency and high speeds. This paper learns an end-to-end policy in simulation that maps onboard sensory observations to collision-free trajectories, enabling zero-shot high-speed flight in challenging real-world environments without adaptation.

Problem
Autonomous quadrotors still struggle to navigate unknown, cluttered environments with the agility and safety of expert pilots because robust perception and planning must operate under noisy, partial observations and limited onboard computation.
Method
The paper trains a lightweight end-to-end policy via privileged learning in simulation to map noisy depth and inertial observations directly to dynamically feasible candidate trajectories selected in a receding horizon.
Results
The policy transfers zero-shot from simulation to diverse natural and human-made environments, while simulations show up to a 10-fold reduction in failure rate versus state-of-the-art methods.
Takeaways & Limitations
Simulation-trained end-to-end policies can support high-speed autonomous quadrotor flight in challenging real-world environments using only onboard sensing and computation.
Takeaways & Limitations
In the real world, the learned policy has low success rates at average speeds of 10 m s−1 or higher, where feasible trajectories become sparse and difficult to sample.

Abstract

from arXiv · show

Quadrotors are agile. Unlike most other machines, they can traverse extremely complex environments at high speeds. To date, only expert human pilots have been able to fully exploit their capabilities. Autonomous operation with on-board sensing and computation has been limited to low speeds. State-of-the-art methods generally separate the navigation problem into subtasks: sensing, mapping, and planning. While this approach has proven successful at low speeds, the separation it builds upon can be problematic for high-speed navigation in cluttered environments. Indeed, the subtasks are executed sequentially, leading to increased processing latency and a compounding of errors through the pipeline. Here we propose an end-to-end approach that can autonomously fly quadrotors through complex natural and man-made environments at high speeds, with purely onboard sensing and computation. The key principle is to directly map noisy sensory observations to collision-free trajectories in a receding-horizon fashion. This direct mapping drastically reduces processing latency and increases robustness to noisy and incomplete perception. The sensorimotor mapping is performed by a convolutional network that is trained exclusively in simulation via privileged learning: imitating an expert with access to privileged information. By simulating realistic sensor noise, our approach achieves zero-shot transfer from simulation to challenging real-world environments that were never experienced during training: dense forests, snow-covered terrain, derailed trains, and collapsed buildings. Our work demonstrates that end-to-end policies trained in simulation enable high-speed autonomous flight through challenging environments, outperforming traditional obstacle avoidance pipelines.

CODE AND MULTIMEDIA MATERIAL · 1. INTRODUCTION

The paper addresses the open challenge of autonomous agile flight by directly mapping onboard sensory measurements to collision-free navigation behavior, using privileged learning and zero-shot transfer from simulation. It also provides a demonstration video and publicly releases the implementation and training datasets.

CODE AND MULTIMEDIA MATERIAL: A demonstration video is available, and the authors publicly release the code and training datasets.The supplied passage provides links to the video and the project repository.
1. INTRODUCTION: Quadrotors can traverse cluttered forests and urban canyons and support applications including search and rescue, logistics, security, infrastructure, entertainment, and agriculture, but autonomous flight matching expert agility and safety remains open.Their agility is described as characteristic of among the most agile and dynamic machines.
1. INTRODUCTION: Separating mapping and planning can compound errors and add sequential-processing latency, while agile flight also requires robust perception and dynamically feasible, collision-free planning from noisy partial observations.These coupled requirements make high-speed autonomous navigation in unknown environments especially difficult.
1. INTRODUCTION: The approach predicts navigation commands directly from onboard sensor measurements, reducing perception-to-action latency while tolerating motion blur, missing data, and sensor noise.This direct mapping avoids explicit mapping and planning stages during navigation.
1. INTRODUCTION: The navigation policy is trained through privileged learning from demonstrations generated by a sampling-based expert with 3D point-cloud and perfect-state access.The simulated expert also has an unconstrained computational budget because simulation does not impose real-time constraints.
1. INTRODUCTION: Given noisy depth images and inertial measurements, the policy predicts multiple short-term, dynamically feasible polynomial trajectories and estimates their individual costs.Training uses a multi-hypothesis winner-takes-all loss to reflect the problem’s multi-modal nature.
1. INTRODUCTION: The resulting policy flies physical quadrotors through natural and human-made environments at high speeds after zero-shot transfer from simulation, without adaptation or fine-tuning.Training uses randomly generated obstacle courses built from simple schematic trees and convex shapes, followed by direct deployment in the physical world.

2. RESULTS · A. High-Speed Flight in the Wild · B. Controlled experiments

The proposed end-to-end policy enables high-speed, zero-shot quadrotor flight through diverse natural and human-made environments, while controlled simulations show it outperforms state-of-the-art navigation baselines as speed increases. Its robustness is demonstrated under challenging obstacle configurations, noisy sensing, and collision-inducing reference trajectories.

2. RESULTS: Simulation experiments reduce failure rates by up to 10 times versus state-of-the-art methods, and the policy transfers without adaptation to challenging real-world environments.The real-world platform uses onboard vision-based state estimation and depth perception, with non-collision-free reference trajectories encoding intended flight paths.
Natural environments: In natural environments, the policy completes all 31 experiments without crashes at average speeds of 3 and 5 m s−1 and successfully completes a maneuver at 7 m s−1.The natural tests include forests of varying density and steep snowy terrain, using 40 m straight-line and 6 m-radius circular reference trajectories that intersect obstacles.
Human-made environments: Across 19 human-made-environment experiments at 3–7 m s−1, the method consistently completes tasks without crashing despite obstacles and layouts never observed during training.These environments include trains, cranes, buildings, ruins, concrete slaloms, and narrow building exits.
Human-made environments: The policy handles a narrow-gap task requiring a drone to identify and pass through an approximately 0.8 m opening despite a reference trajectory directed through the wall.The comparison uses a commercial Skydio R1, whose speed could not be directly controlled and was estimated afterward.
B. Controlled experiments: At 10 m s−1, our method achieves an average success rate of 70% across all controlled environments, while no baseline completes all runs without crashing at 5 m s−1.At 3 m s−1, all methods perform similarly; baseline performance drops rapidly as speed increases.
Forest: In the simulated forest, experiments vary forward speed from 3 to 10 m s−1 through dense randomly generated trees along a collision-inducing 40 m reference trajectory.The forest occupies a 30 m × 60 m region with approximately 0.6 m diameter trees.
Narrow gap: In the simulated narrow-gap setup, the method fails in only 2 of 10 runs even at 7 m s−1 despite randomized gap widths and lateral offsets.The wall is 40 m long, the opening lies 10 m ahead, and gap widths range from 0.8 m to 1.0 m.
Disaster and urban scenario: Disaster-zone and urban-street simulations test speeds from 3 to 10 m s−1 using collision-inducing 40 m references amid collapsed buildings, crashed cars, fences, poles, vehicles, and signs.The drone starts from random map locations, and each reference intersects at least one obstacle.

C. Computational cost

The approach substantially reduces processing latency relative to both baselines, including when inference runs without a GPU or on embedded hardware. Comparisons use CPU-only timings for fairness, with desktop measurements recorded on a 6-core i7-8700 system.

C. Computational cost: All approaches are compared using CPU-only timings recorded on a desktop computer with a 6-core i7-8700 CPU.The same computer was also used for the simulation experiments.
C. Computational cost: Removing mapping substantially reduces computation time, while FastPlanner reaches 65.2 ms per frame and incurs additional effective latency from temporal filtering.FastPlanner may require two to three obstacle observations before adding an obstacle to the map.
C. Computational cost: 25.3× faster than FastPlanner and 7.4× faster than Reactive when network inference runs on the GPU.The comparison uses processing-latency measurements reported in Table 1.
C. Computational cost: Disabling GPU inference increases the network latency by only 8 ms, and the onboard Jetson TX2 forward pass requires 38.9 ms.The evaluation reports desktop CPU/GPU timings and onboard timing using the Jetson TX2.

D. The effect of latency and sensor noise · 3. DISCUSSION

In a controlled pole-avoidance experiment, the end-to-end approach remains effective at higher speeds and is less affected by noisy depth than modular baselines. The discussion attributes these gains to replacing sensing, mapping, and planning with a neural network, while noting limitations at very high real-world speeds and opportunities for faster perception.

D. The effect of latency and sensor noise: The controlled experiment varied ground-truth versus stereo-estimated depth to isolate perception latency and quantify the effect of sensing errors on pole avoidance.The theoretical maximum speed was defined as the speed at which the task became infeasible for each method, depending on sensing range.
Ground-truth depth: Up to 7 m s−1, our approach avoided the pole without failure under perfect depth, while performance decreased to 60% at 10 m s−1; all methods succeeded up to 5 m s−1.The experiment isolated latency using ground-truth depth and used a 6 m initial distance, a 1.5 m pole, and a 0.2 m quadrotor radius.
Estimated depth: With stereo-estimated depth, baseline performance dropped substantially, including a 30% drop for Reactive at 7 m s−1, whereas our approach suffered only a 10% drop.FastPlanner’s noisy-depth failure was linked to delayed obstacle detection caused by repeated observations needed to reject depth outliers.
3. DISCUSSION: Replacing modular sensing, mapping, and planning with one neural-network function reduces processing latency and improves robustness to sensor noise.The modular architecture otherwise introduces communication latency, compounds errors across modules, and creates interactions between components.
3. DISCUSSION: The approach reaches speeds up to 10 m s−1 in complex environments and reduces high-speed failure rates by up to 10 times versus the state of the art.It is trained in simulation by imitating an expert with privileged information, using a sampling-based expert, a neural-network architecture, and a training procedure that address task multi-modality.
3. DISCUSSION: At average real-world speeds of 10 m s−1 or higher, the learned policy has low success rates, and even the expert often cannot find collision-free trajectories.These speeds require temporal consistency over long horizons and strong action variations, limiting current performance.
3. DISCUSSION: Faster sensors could provide more frequent updates, reduce perception latency, improve state-estimate precision, and further reduce noise sensitivity, with event cameras suggested for dynamic obstacles.The approach is presented as a step toward onboard high-speed navigation in unseen environments, with applications including exploration, delivery, and cinematography when combined with a long-term planner.

4. MATERIALS AND METHODS · A. The privileged expert

The method trains a sensorimotor policy to predict receding-horizon trajectories from onboard observations and a long-term reference, using an expert planner with privileged simulation-only state and environment access. The expert samples collision-free, dynamically feasible trajectories by optimizing obstacle clearance and reference proximity, enabling imitation learning without dangerous real-world data collection.

4. MATERIALS AND METHODS: The sensorimotor policy maps depth, velocity, attitude, and desired flight direction to motion hypotheses with collision risks, while a model-predictive controller tracks the lowest-cost trajectory.The reference trajectory supplies a long-term goal but need not itself be collision-free.
4. MATERIALS AND METHODS: Training and demonstrations occur entirely in simulation, providing privileged expert data, unlimited trajectory samples, and no physical risk to the platform.The implementation uses Flightmare, RotorS, and Unity.
4. MATERIALS AND METHODS: Depth images from simulated stereo cameras provide an input representation with negligible simulation-to-reality domain shift, supporting zero-shot transfer to real environments.The simulated stereo images are processed with Semi-Global Matching.
A. The privileged expert: The privileged expert uses perfect platform-state and complete-3D-environment knowledge to generate one-second collision-free quadrotor trajectories conditioned on a reference trajectory.The environment is represented by a complete 3D map, while the trajectories encode desired quadrotor states over the next second.
A. The privileged expert: The expert’s trajectory distribution balances proximity to the reference with distance from obstacles through a cost function, including a collision model for a quadrotor sphere of radius 0.2 m.The collision cost is a truncated quadratic based on distance to the closest environment point, with λc = 1000.
A. The privileged expert: Because clutter can make the trajectory distribution multimodal and analytically intractable, the expert approximates it with Metropolis-Hastings sampling using score s(τ) = exp(−c(τ, τref, C)).The sampling method provides asymptotic convergence guarantees to the target distribution [41].
A. The privileged expert: Trajectories are represented as cubic B-splines with three control points, yielding efficient interpolation, continuous differentiability, bounded derivatives, and dynamically feasible paths trackable by model-predictive control.Sampling the B-spline control points reduces the sampling dimension while preserving an expressive trajectory representation.
A. The privileged expert: Conditioning samples on a global collision-free start-to-goal trajectory biases the expert toward obstacle-free regions, effectively extending its horizon and producing more conservative trajectories.The global trajectory is computed using the approach of Liu et al..

B. The student policy

The student policy maps onboard depth, state, and reference-direction measurements to multiple candidate trajectories and collision costs in real time. It is trained by imitating low-cost expert trajectories and converts selected predictions into dynamically feasible, continuously tracked motion.

Inputs and outputs: The student uses SGM-estimated depth, platform velocity and attitude, and a normalized heading toward a reference point 1 s ahead to produce collision-free trajectories in real time.The policy operates with onboard sensor measurements rather than privileged expert information.
Network architecture: A two-branch neural network encodes visual, inertial, and reference information, then outputs M = 3 trajectories with their respective collision costs.Depth features are extracted with pre-trained MobileNetV3 before multimodal processing.
Trajectory representation: Predicted trajectories are projected onto order-5 polynomials, enforcing continuity in position, velocity, and acceleration while facilitating dynamic feasibility through differential flatness.The representation avoids test-time interpolation costs associated with the sampling-based planner’s B-spline representation.
Trajectory selection: Among near-minimum-cost predictions, the policy executes the trajectory with lowest input cost and tracks it using a model-predictive controller.Selecting among trajectories with c∗/ck ≥0.95 promotes temporal continuity between successive commands.

C. Training environments

Training data were generated in Flightmare from randomized environments containing simulated trees or convex shapes, with expert trajectories supervising a student policy through DAgger. Policies were evaluated on unseen simulated environments and then deployed on physical platforms with platform-specific depth sensing.

Environment generation: The authors generated 850 randomized Flightmare environments containing simulated trees or convex shapes at intensities δ ∈ U(4, 7).Each environment included a global collision-free expert trajectory to a point 40 m ahead, while the student received only a potentially unsafe straight trajectory conveying the goal.
Training procedure: DAgger expanded state-space coverage by labeling student-visited states with the expert while preventing early divergence and crashes.Student trajectories were tracked only when the drone remained within threshold ξ of the global trajectory; otherwise, the global trajectory was followed directly.
Evaluation and deployment: Trained policies were evaluated in unseen simulation environments from the training distributions and then used to control physical platforms in real-world environments.Simulation used SGM depth from a simulated stereo pair and ground-truth platform state, whereas physical deployment used an Intel RealSense 435 for depth.

D. Method validation

An ablation study validates global-planning initialization, depth-based action representation, and multi-modal prediction as important design choices, with depth perception having a fundamental role in high-speed obstacle avoidance. RGB-only training and unimodal trajectory prediction both substantially degrade performance.

Ablation study: Figure 7 shows that global-planning initialization, depth representation, and multi-modal prediction are all important, although their impacts differ.The ablation study evaluates each component and finds that some design choices affect performance more than others.
Ablation study: Training on RGB images causes a significant performance drop, indicating that depth perception is fundamental for high-speed obstacle avoidance.This agrees with previous findings that sufficiently informative intermediate image representations improve task performance.
Ablation study: Removing multi-modal trajectory prediction with a UniModal model is detrimental to performance.The study identifies the absence of multi-modal prediction as another important source of degradation.

SUPPLEMENTARY MATERIALS … S2. PERFORMANCE ANALYSIS IN FUNCTION OF OBSTACLE DENSITY

The supplementary analyses show that the proposed approach remains robust to estimation and control noise and outperforms baseline planners as obstacle density and speed increase. They also define the evaluation setups, simulated-forest conditions, and supplementary sections covering hardware, complexity, speed limits, and sampling.

SUPPLEMENTARY MATERIALS: The supplementary materials comprise analyses of noise sensitivity, obstacle density, hardware, computational complexity, theoretical pole-avoidance speed, and Metropolis-Hastings sampling.These topics are organized as Sections S1–S6.
S1. SENSITIVITY TO ESTIMATION AND CONTROL NOISE: The noise-sensitivity experiments test increasing speeds at the highest tree density, using 10 rollouts per speed and five repetitions with different random seeds.Results report mean success rate and standard deviation across rollout sets.
S1. SENSITIVITY TO ESTIMATION AND CONTROL NOISE: Estimation and control noise do not significantly affect our approach, while Reactive degrades strongly and FastPlanner is more noise-robust but generally inferior.The comparison uses Gaussian disturbances calibrated from fast flights in an instrumented motion-capture setup.
S2. PERFORMANCE ANALYSIS IN FUNCTION OF OBSTACLE DENSITY: The obstacle-density study compares our approach with FastPlanner and Reactive using success rate, defined as reaching within 5 meters of the goal without crashing.The simulated forest uses randomly placed trees, and density controls task difficulty.
S2. PERFORMANCE ANALYSIS IN FUNCTION OF OBSTACLE DENSITY: The simulated-forest experiments vary tree density across δ1 = 1 49, δ2 = 1 36, and δ3 = 1 25 treem−2, and average forward speed from 3 m s−1 to 12 m s−1.Experiments use a 40 m straight reference trajectory and 10 random forest realizations per difficulty, with the same random seed across methods.
S2. PERFORMANCE ANALYSIS IN FUNCTION OF OBSTACLE DENSITY: At 10 m s−1, neither baseline completes a successful run at any tested obstacle density, whereas our approach is significantly more robust at higher speeds.At 3 m s−1, all methods complete every run even at high difficulty; Figure S2 reports success rates across increasing tree densities and speeds.

S3. EXPERIMENTAL PLATFORM

The experimental platform is a lightweight, racing-class quadrotor with onboard computation, stereo-inertial sensing, and a software stack designed for real-time trajectory prediction and control. Its sensing configuration and sensor tilt support obstacle avoidance during high-speed flight and simulation-to-reality transfer.

Hardware Platform: The 890 g quadrotor produces about 40 N of thrust with a 4.4 thrust-to-weight ratio, comparable in weight and power to professional drone-racing platforms.It uses an Armattan Chameleon 6” frame, Hobbywing XRotor 2306 motors, and 5” three-bladed propellers.
Computation and Control: An NVIDIA Jetson TX2 runs neural-network inference on its GPU and the remaining control framework on its CPU, sending collective-thrust and angular-rate commands to a BetaFlight controller.The flight controller converts the desired commands into single-rotor commands.
Sensing: The platform combines T265 visual-inertial odometry at 200 Hz with D435i dense VGA depth at 30 Hz, using both sensing pipelines without additional processing.Both sensing units provide stereo cameras and integrated IMUs.
Sensing: For flight at 7 m s−1 and above, the depth sensor is tilted 30° so its camera looks forward, while its approximately 90° horizontal field of view supports obstacle avoidance.The simulated stereo setup matches the real sensor characteristics to support simulation-to-reality transfer.
Software Stack: The software stack uses TensorFlow in Python for trajectory prediction and separate C++ ROS nodes for trajectory projection and tracking, with prediction running at 24.7 Hz on the physical platform.The implementation is split across Python and C++ nodes communicating through ROS.

S4. COMPUTATIONAL COMPLEXITY · S5. ROTATIONAL DYNAMICS

The proposed approach substantially reduces computation latency relative to FastPlanner and Reactive, while the rotational-dynamics analysis incorporates sensing, processing, and reorientation delays into a theoretical maximum-speed calculation. The speed analysis optimizes roll angle for obstacle avoidance but uses an approximation that omits lateral acceleration during rotation.

S4. COMPUTATIONAL COMPLEXITY: FastPlanner has 65.2 ms total latency across depth sensing, point-cloud mapping, ESDF computation, and trajectory planning.Its sequential sensing, mapping, and planning pipeline produces the largest latency among the compared baselines.
S4. COMPUTATIONAL COMPLEXITY: Reactive reduces total latency to 19.1 ms, about three times faster than FastPlanner, but limited motion primitives and sensing sensitivity weaken high-speed performance.Removing mapping accelerates computation, while the lack of temporal filtering makes the baseline sensitive to sensing errors.
S5. ROTATIONAL DYNAMICS: The rotational model treats avoidance as maximum-input pure rolling followed by pure acceleration, adding reorientation time to system latency.The analysis extends to quadrotors and estimates the maximum speed that permits avoiding a vertical cylindrical obstacle.
S5. ROTATIONAL DYNAMICS: The model selects roll angle φ by maximizing the speed that still permits avoidance, using inertia J, maximum torque T_max, and full-thrust lateral acceleration.The lateral position is expressed for a quadrotor oriented at roll angle φ and accelerating with full thrust.
S5. ROTATIONAL DYNAMICS: Maximum speed is formulated from sensing range s, obstacle-plus-drone radius r_obs, sensing latency t_s, processing latency t_p, and rotation latency t_rot.The fixed case uses a 1.5 m pole, a 0.4 m drone, r_obs = 0.95 m, c_max = 35.3 m s−2, and J = 0.007 kg m−2.
S5. ROTATIONAL DYNAMICS: Theoretical maximum speeds are obtained by grid-searching φ in Equation 11 for each method’s processing latency, but the approximation omits lateral acceleration during rotation.Table S3 states that processing latency does not affect the optimized rotation angle or rotation latency.

S6. METROPOLIS-HASTINGS SAMPLING

Metropolis-Hastings (M-H) samples an indirectly accessible distribution using a score function proportional to its density, avoiding the need to compute a high-dimensional normalization factor. It constructs a Markov chain whose equilibrium distribution is the target, favoring high-score samples while retaining nonzero probability of accepting lower-score moves.

Core formulation: M-H [41] samples a distribution P(w) through a score function d(w) proportional to P(w), avoiding computation of the normalization factor Z in high-dimensional spaces.The score need only be proportional to the target density, rather than equal to its normalized form.
Core formulation: As a Markov chain Monte Carlo method, M-H generates samples through a transition model t(w_t+1|w_t) that depends only on the current sample.The transition model is generally predefined and parametric.
Sampling behavior: M-H always accepts moves to higher-score samples, while lower-score moves are accepted with decreasing probability as the score drop increases.This concentrates samples in high-density regions while still allowing occasional exploration of lower-density regions.
Convergence: Under mild conditions, the empirical distribution converges asymptotically to P(w), and nonzero acceptance of lower-score moves prevents the chain from ultimately becoming trapped at a local extremum.Roberts et al. established the convergence result referenced in the passage.

A Forest

The simulated training environments include forests and convex-obstacle scenes, yet the approach generalizes zero-shot to complex disaster zones and urban environments. Global collision-free planning biases sampled trajectories toward obstacle-free regions beyond the immediate horizon.

Simulation Environments: Training in simulated forests and convex-obstacle environments supports zero-shot generalization to complex disaster zones and urban environments.The selected environments include complex obstacle geometries and narrow passages.
Existing Approaches: Existing drone-navigation approaches are categorized by required environmental prior knowledge and maximum achievable agility.
Global Planning: Sampling around a global collision-free trajectory biases expert trajectories toward obstacle-free regions beyond the immediate horizon, unlike raw-reference sampling.Raw-reference sampling strictly limits the expert’s sight to the immediate horizon.