PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight

Elia Kaufmann, Leonard Bauersfeld, Davide Scaramuzza

arXiv:2202.10796v1cs.RO

TL;DR

Learned quadrotor policies are often trained in simulation because of high sample complexity, while comparing policies is difficult because published approaches rarely justify their action-space choices. The paper benchmarks learned policies with different action spaces by flight performance, peak performance under perfect model identification, and transfer to platforms with different dynamics. Collective-thrust-and-bodyrate policies are more robust to platform dynamics changes without compromising agility; separately, a simulation-trained controller performs previously unseen agile maneuvers on a real quadrotor over 45 km h−1.

  • Problem

    Learned quadrotor policies are often trained in simulation because of high sample complexity, while comparing policies is difficult because published approaches rarely justify their action-space choices.

    Source: §I · p. 1

  • Method

    The paper benchmarks learned policies with different action spaces by flight performance, peak performance under perfect model identification, and transfer to platforms with different dynamics.

    Source: §I · p. 1

  • Results

    Collective-thrust-and-bodyrate policies are more robust to platform dynamics changes without compromising agility; separately, a simulation-trained controller performs previously unseen agile maneuvers on a real quadrotor over 45 km h−1.

    Source: §I · p. 2

  • Takeaways & Limitations

    Collective-thrust-and-bodyrate commands are identified as a control input modality suited to agile flight and domain transfer.

    Source: §VI · p. 6

Abstract

from arXiv · show

Quadrotors are highly nonlinear dynamical systems that require carefully tuned controllers to be pushed to their physical limits. Recently, learning-based control policies have been proposed for quadrotors, as they would potentially allow learning direct mappings from high-dimensional raw sensory observations to actions. Due to sample inefficiency, training such learned controllers on the real platform is impractical or even impossible. Training in simulation is attractive but requires to transfer policies between domains, which demands trained policies to be robust to such domain gap. In this work, we make two contributions: (i) we perform the first benchmark comparison of existing learned control policies for agile quadrotor flight and show that training a control policy that commands body-rates and thrust results in more robust sim-to-real transfer compared to a policy that directly specifies individual rotor thrusts, (ii) we demonstrate for the first time that such a control policy trained via deep reinforcement learning can control a quadrotor in real-world experiments at speeds over 45km/h.

I. INTRODUCTION

This section motivates comparing learned quadrotor policies by action space, since prior work rarely justified its choice and comparisons were difficult. The paper benchmarks these choices in simulation and on a real platform, finding that collective-thrust-and-bodyrate policies combine robustness to dynamics changes with agility and achieve real-world speeds above 45 km h−1.

I. INTRODUCTION

  • Most published learned-control approaches do not justify their action-space choice, making performance comparisons and scientific progress difficult.Policies may output motor commands,,, collective thrust and bodyrates,, velocity commands,, or future waypoints.Source: §I · p. 1
  • Because learned policies are sample-intensive, they are often trained in simulation, making transfer robustness important; high-level action abstractions ease transfer but constrain maneuverability.Less abstract thrust-and-body-rate or individual-rotor-thrust commands may permit more agile maneuvers, but prior demonstrations were limited to near-hover trajectories or maneuver-specific policies.Source: §I · p. 1
  • The study benchmarks learned policies by peak performance under perfect model identification and transfer to a platform with potentially different dynamics, measuring average tracking error on predefined trajectories.Source: §I · p. 1
  • Collective-thrust-and-bodyrate policies are more robust to platform dynamics changes without compromising agility, and enable more agile maneuvers than high-level action parameterizations.Experiments were performed in simulation and on a real quadrotor platform.Source: §I · pp. 1–2
  • Over 45 km h−1, a simulation-trained deep-RL policy performs previously unseen agile maneuvers on a real quadrotor without fine-tuning.The policy is trained purely in simulation and transferred to the real platform.Source: §I · p. 2

II. RELATED WORK

Prior learned quadrotor controllers use linear-velocity commands, collective thrust and bodyrates, or individual rotor thrusts. The related work contrasts their agility, transfer characteristics, and real-world evidence, noting that previous deployments had not achieved agile flight.

Linear Velocity

  • High-level linear-velocity policies transfer easily but do not exploit full platform dynamics and rely on an underlying control stack with high-quality state estimation.Only among the cited approaches reached speeds beyond 3 m s−1; other policies produced near-hover flight.Source: §II · p. 2

Collective Thrust and Bodyrates

  • Collective-thrust-and-bodyrate commands enable more aggressive maneuvers than linear-velocity commands, but prior examples were not real-deployed, near-hover, or limited to one maneuver per policy.The cited acrobatic approach demonstrated real-platform flight but required a separate policy for each trajectory.Source: §II · p. 2

Single-Rotor Thrusts

  • Individual-rotor-thrust policies directly access actuators and represent platform actuation limits, with prior work spanning simulation and real-platform tasks of varying scope.Examples include real-platform stabilization, simulated racing, and a Crazyflie hover policy that flew for 6 s before crashing.Source: §II · p. 2

II. RELATED WORK

  • Despite some successful real-world deployments among these works, none achieved agile flight, with maximum speeds below 4 m s−1.Source: §II · p. 2

III. QUADROTOR DYNAMICS

The dynamics section establishes the quadrotor simulation context and notation for describing its frames, vectors, orientations, and coordinate transformations.

III. QUADROTOR DYNAMICS

  • The quadrotor dynamics are implemented as an environment in TensorFlow Agents for training agile-flight policies.Source: §III · p. 2

A. Notation

  • The notation distinguishes world and body frames, places the body frame at the quadrotor center of mass, and represents orientation with unit quaternions.Figure 2 depicts the world and body frames and propeller numbering.Source: A. Notation · pp. 2–3, Figure 2 · p. 3
  • A body-to-world SE3 transformation expresses a point as world-frame translation plus its body-frame position rotated by the body quaternion.The quaternion-vector product denotes rotation of a vector by the quaternion.Source: A. Notation · p. 3

B. Quadrotor Dynamics

The model treats the quadrotor as a rigid body with motor dynamics and computes its state evolution from gravity, propeller forces and torques, and linear drag.

B. Quadrotor Dynamics

  • The quadrotor is modeled as a 6-degree-of-freedom rigid body with mass and diagonal inertia, while propeller speeds follow first-order motor dynamics driven by commanded speeds.Source: B. Quadrotor Dynamics · p. 3
  • The 17-dimensional state evolves under gravity, propeller-generated collective force and torque, and a linear drag term.Source: B. Quadrotor Dynamics · p. 3
  • Propeller forces and torques are modeled using propeller locations, individual forces and torques, and linear drag coefficients; thrust and drag torque are proportional to squared rotational speed.Source: B. Quadrotor Dynamics · p. 3
  • The dynamics are integrated with a symplectic Euler scheme using a 1 ms step size.Source: B. Quadrotor Dynamics · p. 3

IV. METHODOLOGY

The methodology trains deep neural policies in simulation to map quadrotor observations to control actions, comparing three action-space choices. The policies share observations and network architecture while differing in their actions.

IV. METHODOLOGY

  • Policies map platform state and a reference trajectory to actions, using model-free PPO training in simulation on over 600 reference trajectories.The trajectories cover the quadrotor's full performance envelope.Source: §IV · p. 3
  • The three action spaces are linear velocity and yaw rate, collective thrust and bodyrates, and individual rotor thrusts.Each policy has a four-dimensional action space and the same observations and network architecture.Source: §IV · p. 3
  • Policy and value networks use the same state and reference inputs, while only the value network receives privileged randomized-physics information.Table I describes sliding windows of past states and current and future reference states.Source: Table I · p. 3
  • Physical simulation parameters are sampled from uniform distributions around their nominal values at each rollout.Table II summarizes the physical parameters and applied randomization.Source: Table II · p. 4

A. Observations, Actions, and Rewards

The policy receives histories of platform state and a horizon of future reference states. During training, the value network additionally receives privileged environment information, and the task uses a dense shaped reward.

A. Observations, Actions, and Rewards

  • Observations combine a 10-step history of platform state and actions with 10 future reference states, using rotation matrices for attitude.Reference position and velocity are expressed relative to the current quadrotor state, and observations are normalized.Source: A. Observations, Actions, and Rewards · p. 4
  • The value network receives randomized mass and inertia biases, sampled drag coefficients, and gravity bias unavailable to the policy network.The networks share an architecture but have separate parameters.Source: A. Observations, Actions, and Rewards · p. 4
  • A dense shaped reward penalizes state-tracking error and deviation from the reference action, with a binary crash penalty ending episodes below zero altitude.The reward uses diagonal Q and R matrices; Table III lists training hyperparameters.Source: A. Observations, Actions, and Rewards · p. 4, Table III · p. 4

B. Policy Learning

Policy learning uses PPO with parallel simulated agents, trajectory resampling, and randomized dynamics and observation noise.

B. Policy Learning

  • All policies are trained with Proximal Policy Optimization (PPO), collecting data from 50 simulated agents in parallel.At each reset, each agent samples a training trajectory and starts with a bounded perturbation.Source: B. Policy Learning · p. 4
  • At each environment reset, agents draw new trajectories from the training set and begin with bounded perturbations.This data collection runs across the 50 parallel simulated agents.Source: B. Policy Learning · p. 4
  • Training randomizes quadrotor mass, inertia, aerodynamic drag, and thrust variations, and adds Gaussian noise to policy observations.The authors describe these choices as inspired by prior simulation-to-reality transfer work.Source: B. Policy Learning · p. 4

C. Training Details

Policies are trained in a TensorFlow Agents quadrotor simulation whose nominal physical parameters come from the real platform. The reward weights tracking error and action deviation and penalizes crashes.

C. Training Details

  • The simulated quadrotor environment is implemented using TensorFlow Agents, with nominal mass and inertia identified from the real platform.The nominal parameters and training-time randomization are summarized in Table II.Source: C. Training Details · p. 4
  • Training hyperparameters are specified in Table III.Source: C. Training Details · p. 4
  • Diagonal reward weights penalize tracking error and deviation from the reference action, while crashes receive a −500 reward and terminate the episode.The crash condition is altitude p_z ≤ 0.0.Source: C. Training Details · p. 4

V. EXPERIMENTS

The experiments compare learned policies and classical controllers under nominal and mismatched simulation conditions, then examine training data requirements and sensitivity to control delay.

V. EXPERIMENTS

  • The evaluation asks how action space affects peak performance with a perfect model, robustness to mismatch, and training data requirements.Test trajectories range from hover to racing and remain within the training distribution and platform limits.Source: §V · p. 4, A. Simulation Experiments · p. 5

A. Simulation Experiments

  • With no model mismatch, SRT and CTBR perform comparably; LV performs worse, especially on agile maneuvers.The results compare average positional tracking error across test trajectories for learned policies and traditional controllers.Source: A. Simulation Experiments · p. 5, Table V · p. 5
  • Under model mismatch, CTBR policies and MPC-CTBR execute all maneuvers with modest tracking-error increases, while SRT often crashes on faster maneuvers.SRT also has significantly higher tracking error on slow trajectories; MPC-SRT crashes on RaceB and RaceC.Source: A. Simulation Experiments · p. 5, Table VI · p. 5
  • With domain randomization, CTBR and LV learning curves show robustness to dynamics changes, and randomized training accelerates early learning.Randomized SRT training has initially high variance, trains more slowly, and converges below its non-randomized performance.Source: A. Simulation Experiments · p. 5, Figure 3 · p. 6
  • For CTBR, 60 ms latency leaves hover performance without a noticeable decrease but causes a crash on the racing trajectory.Higher-abstraction LV and CTBR policies are less sensitive to control delay than SRT policies.Source: A. Simulation Experiments · p. 6, Figure 4 · p. 6

B. Real World Experiments

Real-world experiments compare learned policies on executed trajectories. CTBR flies unseen maneuvers despite control delay and outperforms LV on agile trajectories.

B. Real World Experiments

  • CTBR reaches accelerations up to 3g and speeds beyond 45 km h−1 in tested real-world trajectories.SRT could not be deployed under the real system’s 60 ms communication latency; CTBR flew unseen maneuvers despite the delay.Source: B. Real World Experiments · p. 6, Table VII · p. 6
  • On agile real-world trajectories, CTBR significantly outperforms LV, while both policy types transfer to the real platform.Real-world tracking errors exceed those in the BEM simulator, mainly due to unmodelled battery-voltage variation, imperfect motor-thrust mappings, and torque imbalances.Source: B. Real World Experiments · p. 6, Table VII · p. 6

VI. CONCLUSION

The conclusion identifies collective-thrust-and-bodyrate policies as resilient to dynamics mismatch while retaining agility.

VI. CONCLUSION

  • Across the compared learned controllers, collective-thrust-and-bodyrate policies transfer well between domains while retaining high agility.The comparison covers policies specifying single-rotor thrust, collective thrust and bodyrates, and linear velocity commands.Source: §VI · p. 6

Related papers

Loading 2202.10796v1…