Source-linked AI summary
Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors
Artem Molchanov, Tao Chen, Wolfgang Hönig, James A. Preiss, Nora Ayanian, Gaurav S. Sukhatme
TL;DR
Quadrotor stabilization often requires model-specific identification and tuning, while the paper seeks a low-level controller transferable across platforms. It trains neural-network policies in simulation and reports transfer to multiple real quadrotors, including robustness to harsh initial conditions.
Problem
Quadrotor stabilizing controllers often require careful, model-specific system identification and tuning, motivating a policy that can stabilize different quadrotors without manual parameter tuning.
Source: §I · p. 1Method
The authors use reinforcement learning to train a neural-network policy in simulation that maps quadrotor state directly to motor thrusts without an auxiliary stabilizing controller.
Source: §I · p. 1, B. Policy Representation · p. 4, §VII · p. 8Results
A single simulation-trained policy transfers to multiple real platforms, generalizes to out-of-distribution states, and shows high robustness to harsh initial conditions.
Source: §VII · p. 8Takeaways & Limitations
The authors identify the controller as potentially useful for testing custom-built quadrotors and as a backup safety controller.
Source: §I · p. 1
Abstract
from arXiv · showhide
Quadrotor stabilizing controllers often require careful, model-specific tuning for safe operation. We use reinforcement learning to train policies in simulation that transfer remarkably well to multiple different physical quadrotors. Our policies are low-level, i.e., we map the rotorcrafts' state directly to the motor outputs. The trained control policies are very robust to external disturbances and can withstand harsh initial conditions such as throws. We show how different training methodologies (change of the cost function, modeling of noise, use of domain randomization) might affect flight performance. To the best of our knowledge, this is the first work that demonstrates that a simple neural network can learn a robust stabilizing low-level quadrotor controller (without the use of a stabilizing PD controller) that is shown to generalize to multiple quadrotors.
I. INTRODUCTION
The paper targets a low-level quadrotor controller that can transfer from simulation to multiple real platforms without manual, model-specific tuning. It positions this approach against prior transfer work and investigates model parameters and domain randomization.
I. INTRODUCTION
- The controller prioritizes robustness to disturbances and collision recovery, with position control and trajectory tracking as secondary objectives.The stated motivation is to avoid careful, model-specific system identification and tuning; the authors also investigate sensor-noise modeling and cost functions.Source: §I · p. 1
- The authors report a first neural-network low-level stabilizing and trajectory-tracking policy trained entirely in simulation and shown to generalize to multiple quadrotors.Their contributions include training without auxiliary pre-tuned controllers and transferring one policy to multiple real quadrotors.Source: §I · p. 1
II. RELATED WORK
- Prior sim-to-real approaches may require real-robot data collection with untrained policies or complex parameter-estimation setups, both concerns the authors seek to avoid.They frame their target as direct actuator-force control across robots with different dynamics, including inherently unstable systems.Source: §II · p. 2
- Domain randomization varies simulated training parameters to address source-to-target discrepancies, and the paper investigates its use for low-level quadrotor transfer.The authors describe it as an approach that has also been applied to visual features and high-level policies.Source: §II · p. 2
- Related work includes low-level policies transferred to real quadrotors, while this work distinguishes itself by transferring one policy across platforms without an auxiliary pre-tuned PD controller.The authors also emphasize minimal prior knowledge of dynamics and investigate model parameters and domain randomization.Source: §II · p. 2
III. PROBLEM STATEMENT
The problem is to map a quadrotor’s state directly to rotor thrusts while stabilizing it and enabling transfer across platforms. The formulation is limited to ×-configuration quadrotors with estimated state and bounded physical parameters.
III. PROBLEM STATEMENT
- The policy maps position and velocity errors, orientation, and angular-velocity error to rotor thrusts, aiming to reduce state errors and bring the rotation axis upright quickly.The policy is intended to recover from varied initial conditions and transfer to other quadrotor platforms while retaining performance.Source: §III · p. 2
- The formulation considers only ×-configuration quadrotors; a single policy cannot control both × and + layouts without an input identifying the configuration.The paper describes × as the most widely used configuration because it permits convenient camera placement.Source: §III · p. 2, Figure 2 · p. 3
- The assumed operating scope includes reasonably accurate estimates of position, velocity, orientation, and angular velocity, plus quadrotors within a bounded range of physical parameters.The stated parameter ranges are in Table I; edge cases include very high thrust-to-weight ratios and unusual center-of-mass offsets.Source: §III · p. 2
IV. DYNAMICS SIMULATION
The simulation represents quadrotors as rigid bodies and models their motion with Newton–Euler dynamics. The model relates rotor thrusts and torques to translational and rotational motion.
A. Rigid Body Dynamics for Quadrotors
- The simulation treats each quadrotor as a rigid body with four rotors at rectangular-frame corners, producing positive thrust along the body-frame z direction.The body-frame origin is placed at the quadrotor’s center of mass.Source: A. Rigid Body Dynamics for Quadrotors · p. 3
- The translational dynamics relate mass and world-frame acceleration to gravity and rotated total thrust, with the rotation matrix mapping body-frame force into world coordinates.The passage identifies the mass, acceleration, gravity vector, rotation matrix, and total thrust force.Source: A. Rigid Body Dynamics for Quadrotors · p. 3
- The rotational model accounts for total torque, including thrust-generated torque and torque from differences in propeller rotation speeds.Rotor forces and rotation directions determine the propeller-speed-related torque contribution.Source: A. Rigid Body Dynamics for Quadrotors · p. 3
B. Normalized Motor Thrust Input
The controller uses normalized motor inputs to support transfer across quadrotors with different thrust-to-weight ratios. The simulation also models non-ideal motors and motor noise.
B. Normalized Motor Thrust Input
- Normalized thrust commands span 0 to 1, with hover input depending on thrust-to-weight ratio; this choice is intended to support transfer without system identification.The policy action is mapped to normalized thrust, and a corresponding normalized angular-velocity command is also defined.Source: B. Normalized Motor Thrust Input · p. 3
- The affine action transformation is chosen to keep the policy’s action distribution roughly zero-mean and unit-variance.Thrust is proportional to the square of rotor angular velocity.Source: B. Normalized Motor Thrust Input · p. 3
C. Simulation of Non-Ideal Motors
- Motor lag is simulated with a first-order low-pass filter, with settling time T defined as at least 4dt and by a 2% step-response threshold.The filter outputs normalized rotor angular velocities.Source: C. Simulation of Non-Ideal Motors · p. 3
- Motor noise follows a discretized Ornstein–Uhlenbeck process with zero mean, decay rate θ, scale factor σ, and four-dimensional spherical Gaussian noise.The process is introduced to make simulation more realistic and avoid policies exploiting physically implausible simulator behavior.Source: C. Simulation of Non-Ideal Motors · p. 3
D. Observation Model
The observation model adds sensor and state-estimation noise to position, orientation, linear velocity, and gyroscope angular-velocity measurements.
D. Observation Model
- Position, orientation, and linear-velocity estimation noise is modeled as zero-mean Gaussian, while gyroscope noise follows Furrer et al..Noise parameters were roughly estimated from recordings made while quadrotors rested on the ground.Source: D. Observation Model · p. 4
E. Numerical Methods
The simulation integrates the quadrotor dynamics with first-order Euler’s method and periodically restores the rotation matrix’s orthogonality using SVD.
E. Numerical Methods
- First-order Euler integration causes the rotation matrix to lose orthogonality over time, so the simulation re-orthogonalizes it every 0.5 seconds or when an orthogonality criterion fails.The SVD-based result is described as a superior orthogonalization to Gram–Schmidt.Source: E. Numerical Methods · p. 4
- The re-orthogonalization computes R⊥ = UV^T from the SVD UΣV^T = R, yielding the solution to an optimization problem over orthogonal matrices.The objective uses the Frobenius norm.Source: E. Numerical Methods · p. 4
A. Randomization
The methodology randomizes quadrotor dynamics during training and uses a neural-network policy that directly controls motor thrusts. Policy learning also specifies initial states and a weighted reinforcement-learning cost.
A. Randomization
- Dynamics parameters are sampled per trajectory either around nominal estimates or within specified limits, with geometric parameters and mass constructed from component samples in the bounded approach.The nominal-value approach uses Crazyflie 2.0 estimates.Source: A. Randomization · p. 4
B. Policy Representation
- A fully connected network with two 64-neuron tanh hidden layers maps an 18-dimensional state input directly to normalized motor-thrust commands without an auxiliary stabilizing PID controller.Its input uses current-to-goal errors except for the current rotation matrix; shifting the goal supports trajectory tracking.Source: B. Policy Representation · p. 4
C. Policy Learning
- The policy is trained with PPO, while initial states span full SO(3), a 2 m position box, velocities up to 1 m/s, and angular velocities up to 2π rad/s.The goal is hover at [0, 0, 2]^T; rotation matrices avoid the quaternion double-cover issue described in the passage.Source: C. Policy Learning · pp. 4–5
- The reinforcement-learning cost includes position, velocity, angular-velocity, and action terms with nonnegative weights, and the study investigates the influence of different cost components.The supplied cost passage also defines an orientation-angle term using the rotation matrix.Source: C. Policy Learning · p. 5
D. Sim-to-Sim Verification
The authors verify policies in a higher-fidelity simulator, then evaluate physical quadrotors using hover, trajectory-tracking, and recovery tests. Their own simulator is faster and more flexible for learning than using Gazebo with RotorS directly.
D. Sim-to-Sim Verification
- Gazebo with RotorS provides a higher-fidelity sim-to-sim check, including pre-defined quadrotor models without physical counterparts.This transfer is used to check the learning simulator's physics and policy performance in a more realistic environment.Source: D. Sim-to-Sim Verification · p. 5
- The training simulator is faster and more flexible than using Gazebo with RotorS directly.Gazebo uses ODE physics and RotorS models effects such as drag that the learning simulator neglects.Source: D. Sim-to-Sim Verification · p. 5
E. Sim-to-Real Verification
- The physical test platforms are Crazyflie-based quadrotors, including heavier builds assembled from standard parts and the Crazyflie's flight-controller board.The Crazyflie 2.0 weighs 27 g and is described as relatively crashtolerant.Source: E. Sim-to-Real Verification · p. 5
- The authors assess physical flight with hover error and oscillation measures, figure-eight tracking error, and recovery checks after disturbances.The quadrotors are tested in order of increasing size for safety.Source: E. Sim-to-Real Verification · p. 5
VI. EXPERIMENTS
The experiments use modified firmware and selected policies trained across multiple random seeds. Cost ablations show that a compact objective can train successfully, while adding rotational-error or velocity costs changes performance and tuning demands.
VI. EXPERIMENTS
- Physical experiments run at 500 Hz for onboard evaluation, EKF, and control, with the neural network evaluation taking about 0.8 ms.Motion capture provides pose at 100 Hz; firmware changes include reduced gyroscope delay and finite-difference linear-velocity estimates.Source: §VI · pp. 5–6
- Policies were trained for 3,000 iterations in most scenarios and 6,000 with randomization, then one of two top seeds was selected after simulation inspection.Five seeds were trained per scenario; selection considered trajectory cost, smoothness, and attitude oscillations.Source: §VI · p. 6
A. Ablation Analysis on Cost Components
- Successful policies use only position, angular-velocity, and action costs when αω is above 0.05 and below 1; αω = 0.1 and αa = 0.05 were chosen as the baseline.In simulation, αω = 0.25 gave the lowest position and angular errors, while higher αω could increase physical position error.Source: A. Ablation Analysis on Cost Components · p. 6
- Adding rotational-error cost improves position and angular errors in simulation but slightly increases physical position errors relative to the baseline.The authors note that it also stabilizes yaw, which may be useful for takeoff or camera use.Source: A. Ablation Analysis on Cost Components · p. 6
- The cost including linear-velocity error is harder to tune and gives slightly worse position and angular errors in simulation and on the physical quadrotor.All tested policies had no significant oscillations, with fo ≤ 1.2 Hz.Source: A. Ablation Analysis on Cost Components · p. 6
B. Sim-to-Real: Learning with Estimated Model
The experiments compare learned policies with hand-tuned controllers on a Crazyflie and across quadrotor platforms. Policies are broadly competitive across platforms, and modest model randomization can improve transfer, though full randomization is more consistent but weaker.
B. Sim-to-Real: Learning with Estimated Model
- The Mellinger controller has 0.11 m average position error, versus 0.2 m when its integral terms are removed.The trajectory task is a 5.5-second figure-eight reaching speeds up to 1.6 m/s.Source: B. Sim-to-Real: Learning with… · p. 6
- The network trained with motor settling time T = 0.15 has 0.19 m mean position error, compared with 0.21 m without motor delays and 0.24 m without sensor and motor noise.The standard deviation of position-error norm is 0.11 for the neural networks and 0.06 for the nonlinear feedback controller.Source: B. Sim-to-Real: Learning with… · pp. 6–7
C. Sim-to-Multi-Real: Learning without Model
- Across all platforms, learned policies show somewhat comparable performance to Mellinger controllers and no significant oscillations; small (20 %) randomization around nominal values can improve performance, while full randomization is more consistent but weaker.Source: C. Sim-to-Multi-Real: Learning without… · p. 7
B. Sim-to-Real: Learning with Estimated Model
- The Crazyflie-specific network was not explicitly trained for trajectory tracking, yet its tracking performance is competitive with a Crazyflie-tuned baseline controller.The multi-platform comparison tests policies trained with randomized models as well as baseline and no-delay networks.Source: B. Sim-to-Real: Learning with… · p. 7, C. Sim-to-Multi-Real: Learning without… · p. 7
C. Sim-to-Multi-Real: Learning without Model
- Policies trained for the Crazyflie work best on it and also perform well on the small quadrotor, but show large position errors on the medium quadrotor.Modeling motor delay during training has very little impact on tracking performance, and learned policies show no significant oscillations.Source: C. Sim-to-Multi-Real: Learning without… · p. 7
D. Control Policy Robustness and Recovery
A single simulation-trained policy shows recovery robustness across three quadrotor platforms, including throws and states beyond those encountered in training. The conclusion reports transfer across real platforms and moderate gains from domain randomization.
D. Control Policy Robustness and Recovery
- On the Crazyflie, the policy recovers from 80% of throws and from up to 100% with moderate attitude changes (≤ 35°); it can also recover in more than half of scenarios after hitting the ground.The hand-tuned Mellinger controller could not recover properly from deviations larger than 0.5 m.Source: D. Control Policy Robustness and… · p. 7
- On the other quadrotors, moderate-attitude throws recover in ≥ 90% of trials, while stronger disturbances yield roughly 50% recovery on average.Source: D. Control Policy Robustness and… · p. 7
- Policies develop a yaw preference despite having no yaw-related cost other than angular-velocity cost; the authors hypothesize this gives them a stable reference orientation.Mild pushes and pulls from hover do not cause failures.Source: D. Control Policy Robustness and… · p. 7
- Policies handle initial velocities often exceeding 3 m/s despite training at velocities up to 1 m/s, and can take off from the ground and fly beyond the training position-initialization box.The authors identify these conditions as out-of-distribution states.Source: D. Control Policy Robustness and… · pp. 7–8
VII. CONCLUSIONS AND FUTURE WORK
- Across the experiments, one policy transfers to multiple real platforms and out-of-distribution states; domain randomization improves results moderately over training without parameter perturbations.The conclusion also reports robustness to harsh initial conditions compared with the hand-tuned nonlinear baseline.Source: §VII · p. 8