PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Drone Acrobatics

Elia Kaufmann, Antonio Loquercio, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza

arXiv:2006.05768v2cs.RO

TL;DR

Acrobatic quadrotor flight is difficult because extreme accelerations challenge reliable sensing and control with onboard hardware. The paper trains a sensorimotor policy entirely in simulation from privileged-controller demonstrations, using sensory abstractions for transfer. The resulting controller performs acrobatic maneuvers on a real quadrotor without fine-tuning, reaching accelerations up to 3g.

  • Problem

    Reliable perception and control for high-acceleration acrobatic flight remain difficult with only onboard sensing and computation.

  • Method

    The policy is trained in simulation from privileged optimal-controller demonstrations and uses abstracted onboard visual and inertial inputs for sim-to-real transfer.

  • Results

    The learned controller transfers directly to a physical quadrotor without fine-tuning and performs acrobatic maneuvers with accelerations up to 3g.

  • Takeaways & Limitations

    Simulation training avoids human demonstrations and physical risk while supporting maneuvers that challenge even expert human pilots.

Abstract

from arXiv · show

Performing acrobatic maneuvers with quadrotors is extremely challenging. Acrobatic flight requires high thrust and extreme angular accelerations that push the platform to its physical limits. Professional drone pilots often measure their level of mastery by flying such maneuvers in competitions. In this paper, we propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation. We train the policy entirely in simulation by leveraging demonstrations from an optimal controller that has access to privileged information. We use appropriate abstractions of the visual input to enable transfer to a real quadrotor. We show that the resulting policy can be directly deployed in the physical world without any fine-tuning on real data. Our methodology has several favorable properties: it does not require a human expert to provide demonstrations, it cannot harm the physical system during training, and it can be used to learn maneuvers that are challenging even for the best human pilots. Our approach enables a physical quadrotor to fly maneuvers such as the Power Loop, the Barrel Roll, and the Matty Flip, during which it incurs accelerations of up to 3g.

I. INTRODUCTION

Acrobatic quadrotor flight challenges perception, state estimation, and control because high accelerations and angular rates degrade sensing while leaving little room for control errors. The paper addresses this challenge with simulation-trained sensorimotor policies that transfer directly to real hardware using only onboard sensing and computation.

  • High accelerations, motion blur, large displacements, and feature-tracking difficulty make reliable vision-based state estimation during acrobatics challenging.
  • Existing agile-flight systems often rely on external sensing or computation, while onboard-only approaches remained limited in acceleration, duration, or maneuver conservatism.
  • The proposed system enables a vision-based autonomous quadrotor using only onboard sensing and computation to perform maneuvers with accelerations up to 3g.
  • The controller is trained in simulation from privileged optimal-controller demonstrations and is evaluated on the Power Loop, Barrel Roll, and Matty Flip.
  • Visual and inertial abstractions reduce the simulation-to-reality gap, enabling an end-to-end sensorimotor mapping to transfer directly to a real quadrotor.

III. OVERVIEW

The method trains a sensorimotor controller to map onboard measurements and a reference maneuver to low-level quadrotor actions. Privileged simulation demonstrations and sensory abstraction support deployment without privileged information or real-world adaptation.

  • The controller predicts low-level thrust and body-rate actions from onboard sensor history and a user-defined reference trajectory.
  • The policy uses camera and inertial measurements despite their different update frequencies, motivating an asynchronous sensorimotor network.
  • Privileged learning trains the student from simulation demonstrations generated by an optimal controller with access to full ground-truth state.
  • Simulation provides unlimited demonstrations for desired trajectories, including maneuvers beyond even expert human pilots, without risking the physical platform.
  • The trained student uses no privileged information and transfers directly to the physical platform without adaptation.

IV. METHOD

The method formulates acrobatic flight as neural-network policy optimization against reference trajectories, using state-tracking costs and dynamically feasible trajectory primitives. Reference generation combines circular maneuver cores with constrained polynomial transitions.

  • The neural policy maps observations to continuous actions while minimizing a finite-horizon objective over trajectories induced by the policy.
  • The quadratic tracking cost compares the quadrotor state with a time-indexed reference trajectory using a positive-semidefinite cost matrix.
  • Both expert and learned policy assume a dynamically feasible reference trajectory constrained by the quadrotor’s physical limits and underactuation.
  • Reference planning uses flat outputs consisting of position coordinates and yaw, whose smooth trajectories can be tracked by the underactuated platform.
  • Circular motion primitives form maneuver cores, while constrained polynomial trajectories handle entry, transitions, and exits.
  • Position polynomials use order 7 and yaw polynomials order 2, with continuity through the third position derivative and limits on speed, thrust, and body rates.

B. Privileged Expert

The privileged expert uses an optimization-based MPC controller to generate thrust and body-rate actions while tracking a reference trajectory under quadrotor dynamics and constraints.

  • The privileged expert is an MPC that generates collective thrust and body rates through an optimization-based scheme.
  • At each step, the MPC optimizes an open-loop control problem over a receding horizon of N time steps and applies the first optimized command.
  • The optimization uses state-reference errors, system-dynamics equality constraints, and optional bounds on inputs and states.

C. Learning

The learning procedure imitates a privileged expert in simulation while using abstracted sensory inputs to reduce the simulation-to-reality gap. Iterative student-controlled data collection supports training without exposing a physical platform to partially trained policies.

  • The sensorimotor controller imitates demonstrations from an expert with ground-truth state estimates, while the student uses no privileged information.
  • The expert–student performance bound depends on action-distribution discrepancy, Wasserstein distance, and the expert-action smoothness constant.
  • DAGGER reduces this discrepancy by iteratively collecting student-controlled observations, labeling them with expert actions, and updating the student.
  • Training exclusively in simulation avoids requiring ground-truth state access on the physical platform and protects it from partially trained policies.
  • The controller operates on an intermediate representation more consistent across simulation and reality than raw visual input.
  • Feature tracks and de-biased, integrated inertial measurements are designed to retain task-relevant information while remaining invariant to nuisance factors.

D. Sensorimotor Controller

The sensorimotor controller combines histories of visual, inertial, and reference-trajectory inputs through asynchronous branches and produces low-level thrust and body-rate commands. Input abstractions are chosen to support scene- and domain-independent transfer.

  • The student must infer its state from onboard camera and IMU measurements, remain invariant across environments, and handle sensors operating at different frequencies.
  • The network uses three branches for visual input, inertial measurements, and reference trajectories, followed by a multilayer perceptron that produces actions.
  • Input abstractions are designed to preserve task-relevant information while removing environmental factors that are difficult to simulate and transfer.
  • Feature tracks encode keypoint motion and scene structure without absolute pixel intensities, while inertial measurements recover the unknown motion scale.
  • The branch outputs are concatenated and processed into body rates and collective thrust for platform control.
  • Asynchronous branches update when new sensor data arrive, while the multilayer perceptron uses their latest outputs at a fixed 100 Hz rate.

E. Implementation Details

The policies are trained in a high-fidelity Gazebo simulation and deployed on a custom quadrotor using onboard neural-network inference and visual-inertial sensing. Training alternates rollout collection with repeated optimization over the accumulated dataset.

  • Gazebo with the RotorS extension simulates an AscTec Hummingbird with a forward-facing fisheye camera in a 70-meter cubical flight space.
  • The real platform weighs 1.15 kg, has a 4:1 thrust-to-weight ratio, and runs inference on a Jetson TX2 using Intel RealSense T265 camera and inertial measurements.
  • After every 30 new rollouts, training runs for 40 epochs on the full dataset; five repetitions produce 150 rollouts using Adam with learning rate 3e −4.

V. EXPERIMENTS

The evaluation addresses whether the sensorimotor controller improves on state-estimation-and-control decomposition, whether input abstraction supports simulation-to-reality transfer, and whether ablations validate the design choices.

  • The evaluation compares the sensorimotor controller with a standard decomposition of state estimation and control.
  • It examines the role of input abstraction in facilitating transfer from simulation to reality.
  • Ablation studies are used to validate the presented design choices.

A. Experimental Setup

The experiments cover three acrobatic maneuvers and a multi-maneuver sequence, using VIO-MPC as a strong baseline and tracking error plus success rate as evaluation metrics.

  • The study learns policies for three acrobatic maneuvers and a sequence of multiple maneuvers.
  • Table I reports average tracking error and success rate over 20 runs, and adding feature tracks further improves both metrics, especially for longer maneuvers.
  • The Power Loop, Barrel Roll, and Matty Flip require high accelerations and fast angular velocities, while all maneuvers begin and end in hover.
  • The VIO-MPC baseline combines visual-inertial odometry with model predictive control and receives the same inputs as the learned controllers.
  • Tracking error is measured as average root mean square reference-position error in meters, while success rate records completion without crashes or safety-pilot intervention.

B. Experiments in Simulation

In simulation, the full learned controller outperforms VIO-MPC and reduced-input variants, while abstracted visual input improves transfer and robustness on longer sequences.

  • The full visual-inertial controller is best across all maneuvers, reducing tracking error by up to 45% versus VIO-MPC.
  • The performance gap between the learned controller and baseline widens for longer and more difficult sequences.
  • Policies without sensory input perform poorly, and visual-only policies lack sufficient information to determine absolute ego-motion scale.
  • Inertial-only policies safely fly most maneuvers but accumulate increasing drift as maneuver duration grows.
  • Combining inertial and visual data reduces drift and improves accuracy, while abstracted visual input raises Combo success rate by 10 percentage points.
  • The full controller performs repeated barrel rolls for 20 seconds without a single failure.
  • Image-based input achieves only 80% success in the training environment and has 58% higher tracking error than the feature-track approach.

C. Deployment in the Physical World

The simulation-trained controllers transfer directly to a physical quadrotor without fine-tuning, flying the evaluated maneuvers without intervention.

  • The same controllers evaluated in simulation are deployed on a physical quadrotor with no fine-tuning.
  • The abstraction scheme facilitates successful deployment despite differences between simulated and real-world appearance.
  • The physical controllers fly all evaluated maneuvers with no intervention, and feature-track visual input enhances robustness during repeated execution.
  • The approach enables autonomous acrobatics using only onboard sensing and computation, including maneuvers challenging even for expert human pilots.
Loading 2006.05768v2…