Source-linked AI summary
Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning
Nathan O. Lambert, Daniel S. Drew, Joseph Yaconelli, Roberto Calandra, Sergey Levine, Kristofer S. J. Pister
TL;DR
The paper asks whether low-level robot controllers can be learned autonomously from limited real-world data without simulation, demonstrations, or prior dynamics knowledge. It uses model-based reinforcement learning with a learned probabilistic neural dynamics model and external MPC to control Crazyflie motors directly. The system achieved repeated stable hovering for up to 6 seconds after 3 minutes of fully autonomous training data, while remaining limited by drift, external-GPU requirements, and collision risk during learning.
Problem
The paper addresses the need for data-efficient low-level controllers that reduce reliance on manual tuning, domain knowledge, simulation, and lengthy system identification.
Method
The method learns a probabilistic neural forward-dynamics model from autonomous Crazyflie experience and uses it within GPU-parallelized model predictive control with direct PWM motor inputs.
Results
Up to 6 seconds of repeated stable hover was achieved with 3 minutes of fully autonomous training data.
Takeaways & Limitations
The results demonstrate experimental MBRL control of a rapidly dynamic quadrotor without prior dynamics knowledge, internal stability controllers, or expert demonstrations.
Takeaways & Limitations
Performance is constrained by drift and by the requirement for a high-power external GPU, with potential collisions during learning.
Abstract
from arXiv · showhide
Designing effective low-level robot controllers often entail platform-specific implementations that require manual heuristic parameter tuning, significant system knowledge, or long design times. With the rising number of robotic and mechatronic systems deployed across areas ranging from industrial automation to intelligent toys, the need for a general approach to generating low-level controllers is increasing. To address the challenge of rapidly generating low-level controllers, we argue for using model-based reinforcement learning (MBRL) trained on relatively small amounts of automatically generated (i.e., without system simulation) data. In this paper, we explore the capabilities of MBRL on a Crazyflie centimeter-scale quadrotor with rapid dynamics to predict and control at <50Hz. To our knowledge, this is the first use of MBRL for controlled hover of a quadrotor using only on-board sensors, direct motor input signals, and no initial dynamics knowledge. Our controller leverages rapid simulation of a neural network forward dynamics model on a GPU-enabled base station, which then transmits the best current action to the quadrotor firmware via radio. In our experiments, the quadrotor achieved hovering capability of up to 6 seconds with 3 minutes of experimental training data.
I. INTRODUCTION
The paper investigates whether model-based reinforcement learning can autonomously generate a competitive low-level quadrotor controller from limited real-world data, without simulation or demonstrations. It demonstrates direct motor control of a Crazyflie using a learned dynamics model and reports stable hovering for up to 6 seconds after 3 minutes of autonomous training data.
- Motivation: The paper asks whether a robot can autonomously learn a competitive low-level controller without simulation or demonstrations in limited time.The motivation is to reduce reliance on system identification effort, manual PID tuning, domain knowledge, and risky roll-outs.
- Approach: The proposed MBRL system learns forward dynamics with neural networks and uses GPU-parallelized random-shooter MPC for real-time low-level control.The controller is designed for a quadrotor with unknown analytic dynamics and sends direct motor assignments.
- Approach: The Crazyflie is controlled from onboard sensor measurements using PWM motor voltage signals, without relying on preconfigured internal stability controllers.The method is intended to learn controllers directly from real-world data.
- Findings: At control frequencies of ≤50 Hz, the learned predictive models enabled hovering where a hand-tuned PID controller failed at that frequency.The paper notes that the learned method was not yet an alternative to traditional controllers and had important performance and applicability limitations.
- Findings: Up to 6 seconds of repeated stable hover was achieved after 3 minutes of fully autonomous training data.Failures were attributed to drift of unobserved states, and the result required no prior dynamics knowledge, internal stability controller, actuator smoothing, or expert demonstration.
II. RELATED WORK
Prior quadrotor work includes classical, optimal, learning-based, and model-predictive controllers, but often relies on analytic models, onboard controllers, simulation, or external state sensing. This work instead emphasizes direct actuator-level attitude control through externally computed MPC.
- Conventional control: Classical PID, LQR, and iLQR controllers with analytic or linearized models can control quadrotor attitude and trajectory setpoints.The paper explicitly distinguishes its goal from comparing against finely tuned conventional performance.
- Conventional control: Optimal-control and specialized learning methods have demonstrated robust or extreme-case quadrotor control, including damaged-propeller configurations.These related systems address demanding control settings but are not the comparison target here.
- Positioning: The paper differs by demonstrating attitude control through real-time external MPC using a dynamics model learned fully from in-flight data and direct motor signals.Unlike implementations conditioned on existing internal controllers, the model spans sensors to actuators.
- Learning-based control: Many learning-based quadrotor systems control trajectories by sending setpoints to stable onboard attitude and thrust controllers.The paper identifies this dependence as a distinction from its direct low-level control objective.
- Learning-based control: A model-free reinforcement-learning comparison used external vision, an internal motor-speed controller, thrust commands, and substantial simulation-generated data.These design choices differ from the present work’s onboard sensing, direct motor inputs, and experimental-data focus.
- Learning-based control: Other learned quadrotor dynamics approaches have used motion-capture measurements and thrust targets for internal PIDs rather than onboard IMU measurements and motor voltages.The paper positions its implementation as a more direct sensor-to-actuator model.
C. Model-based Reinforcement Learning
Model-based reinforcement learning has been applied in low-data regimes across several robotic systems, but prior examples often involve simulation, lower-dimensional or statically stable platforms, or internal controllers. The paper extends this direction to experimental direct-motor quadrotor control with learned neural dynamics and MPC.
- Prior MBRL systems: Experimental low-level MBRL has been demonstrated on an RC car, while related helicopter and quadrotor applications often use lower-level commands or simulation.The cited systems differ in dimensionality, stability, or reliance on simulated data and existing controllers.
- Positioning: The paper’s contribution is experimental MBRL control of a rapidly dynamic quadrotor using direct motor inputs rather than internal controllers or simulation-only evaluation.The supplied related-work passages establish this as the relevant extension of prior systems.
- Prior MBRL systems: A learned-neural-network MPPI controller achieved data-efficient quadrotor trajectory control in simulation after initialization with 30 minutes of demonstration data.This contrasts with the present paper’s experimental low-level control setting.
- MBRL background: MBRL with trajectory sampling has shown sample-efficiency advantages over recent model-free algorithms in low-dimensional tasks.The paper builds on probabilistic neural-network strategies to examine real-world variability and time constraints.
- Prior MBRL systems: NN-based dynamics models with MPC have controlled an experimental under-actuated hexapod, but that platform has static stability, lower frequency requirements, and GPS state measurements.The paper describes a similar architecture adapted for higher frequencies and noisier state data.
III. EXPERIMENTAL SETUP
The experimental system uses an external ROS/GPU control loop, direct PWM motor commands, compressed onboard sensing, and a probabilistic neural dynamics model for MPC. The setup accounts for rapid nonlinear motor dynamics but remains sensitive to model mismatch and experimental variation.
- Hardware and communication: The 27 g, 9 cm2 Crazyflie requires robust control because of its rapid dynamics and normally runs onboard attitude control at 500 Hz.Euler-angle state estimation updates at 1 kHz in the default system.
- Hardware and communication: The ROS structure sends controller-generated PWM values to the Crazyflie and passes radio-derived state variables to the MPC node.The radio interface and firmware were modified to support external stability control.
- Hardware and communication: The system directly defines actions as PWM motor signals and bypasses standard firmware controller updates.Compressed IMU and Euler-angle logging reduces bitrate requirements, trading measurement precision for update frequency.
- Dynamics model: The forward model predicts the next state change from current state and action using a probabilistic neural network.The probabilistic loss models output means and variances, while the variance penalty supports stable longer-horizon predictions.
- Caveats: Lower model loss does not guarantee longer flight time, and experimental variation limits systematic verification of model accuracy between roll-outs.The authors report preliminary performance differences between probabilistic and mean-squared-error training but call for more experiments.
- Dynamics model: The dynamics model uses Euler angles, linear and angular accelerations, and four PWM values as system information.The state combines filtered Euler angles with onboard IMU-derived acceleration measurements.
- Dynamics model: A 20 ms update step is short enough for motor spin-up to affect the learned dynamics, so past state-action information is included in neural-network inputs.The Crazyflie motor step response is on the order of 250 ms.
V. LOW LEVEL MODEL-BASED CONTROL
The controller uses a learned forward dynamics model within random-shooting MPC to evaluate candidate motor actions and select the best action for hovering. Its prediction horizon and action-search computation constrain operation at higher control frequencies.
- Controller design: Random-shooting MPC simulates N randomly generated actions over horizon T and selects the action minimizing a user-designed objective.The objective evaluates predicted trajectories from the learned forward dynamics model.
- Action candidates: Candidate motor actions are constant over the prediction horizon and sampled around each motor’s stable-hover PWM value within a range controlled by σ.The action is a 4-tuple of motor PWM values sampled independently from bounded uniform intervals.
- Objective function: The hovering objective minimizes predicted pitch and roll while penalizing Euler-angle rates, with λ controlling the proportional-to-derivative cost ratio.Adding predicted-acceleration costs did not improve performance because prediction variance was high.
- Prediction and replanning: N = 50 candidate actions are visualized with the selected best action highlighted, while future predicted states may diverge because actions are replanned each step.The figure depicts candidate state trajectories rather than a single fixed open-loop trajectory.
- Computational limits: At higher frequencies, shorter feasible horizons such as T = 9 at 75 Hz or T = 6 at 100 Hz reduce the prediction span and constrain performance.Computational cost scales with model size, candidate count N, and horizon T.
VI. EXPERIMENTAL EVALUATION
This section identifies the experimental setting, learning process, and performance summary of the control algorithm, with videos and code provided online for reproduction.
- Experimental overview: The experiments describe the system setting, learning process, and performance summary of the control algorithm.The section also points readers to videos and full code for reproducing the experiments.
A. Experimental Setting
The experiments evaluate flight duration during autonomous rollouts while learning at 25 Hz and 50 Hz. Performance improves rapidly with limited data, although drift, attitude-estimation inconsistencies, and crash-related emergency stops constrain flight length.
- Experimental setting: Flight performance is measured by average flight length per rollout, with failures commonly caused by drift-induced collisions or pitch and roll angles exceeding 40°.An emergency stop is sent to the motors in either case to minimize damage.
- Experimental setting: At 25 Hz and 50 Hz, the 10-flight rollouts show a slight final-performance edge at 50 Hz and faster learning per flight at that frequency.The higher frequency provides more action changes during control.
- Experimental setting: Drift demonstrates the challenge of using attitude controllers to compensate for a velocity offset.Internal PIDs also fail regularly from drift over the same time frame without external inputs.
- Learning process: Initial training uses 124 dynamics points at 25 Hz and 394 points at 50 Hz, followed by autonomous flights with a 250 ms takeoff ramp and on-policy data collection.Early rollouts explore more extreme attitudes, providing data for later recovery from higher pitch or roll.
- Learning process: The improvement rate peaks after 1,000 trainable dynamics points; improvement appears after rollout 1 at 50 Hz and rollout 5 at 25 Hz, with longest individual flights over 5 s.Final models use 2,608 points at 25 Hz and 9,655 points at 50 Hz, although peak performance occurs earlier.
C. Performance Summary
The controller achieves multi-second hover at 25 Hz and 50 Hz, while flight duration and consistency remain constrained by drift, hardware changes, model-learning limits, and computational and safety burdens.
- About 30% of flights fail to drift at peak performance across both 25 Hz and 50 Hz operation.The controller nevertheless hovers for multiple seconds after a clean open-loop takeoff.
- Roll-out 3 already flies for > 2 seconds during 50 Hz learning, following little ability in the random and first controlled roll-outs.
- Controlled PWM values visibly change in response to pitch and roll angle oscillations during 50 Hz flight.
- Comparable but limited performance is achieved in under 10 minutes of clock time using only 3 minutes of training data.The authors frame this result as encouraging for future efforts to match or surpass basic controllers.
- PWM equilibrium values shift by over 10% after collisions, creating dynamics-model mismatch as hardware changes.The internal state estimator also does not accurately track extreme Euler-angle changes.
- Peak performance may be limited by data-constrained model-accuracy gains, changing rollout data distributions, and stochasticity in neural-network training and MPC execution.
- Computational burden and safety limit applicability because the current MPC requires a GPU-enabled base station and uses only a firmware safety killswitch.The authors propose intelligent action sampling, learned control policies, and MPC safety constraints as possible improvements.
VIII. CONCLUSIONS AND FUTURE WORK
The paper demonstrates system-agnostic MBRL for low-level control of an initially unknown, fast, and statically unstable quadrotor. The authors argue that its rapid learning and generality motivate investigation on other robots, especially microrobotic systems.
- The work explores MBRL for low-level control of an a priori unknown dynamic system with a statically unstable Crazyflie and fast dynamics.It describes the firmware, system design, and model-learning considerations needed for external MPC over radio.
- A full flight would have continued longer without drift into the wall, as shown by Euler-angle state data and corresponding video frames.
- Learned flight in only minutes of testing highlights both the generalizability and learning speed of system-agnostic MBRL.
- Microrobotics is identified as a strong candidate for MBRL because it combines under-characterized dynamics, weak or absent controllers, fast dynamics, and high testing costs.
IX. APPENDIX
The appendix examines battery voltage, hardware aging, sensor noise, and data limitations affecting Crazyflie dynamics and learned control. Battery voltage had little predictive benefit, while aging-related damage and drift constrained reliable operation.
- Experimental constraints: The quadrotor’s short battery life and low-voltage uncontrollability made long experimental data collection difficult.The battery provided about 7 minutes of flight time, while low voltage affected both the learned controllers and built-in nested PID controllers.
- Battery voltage: Less than 1% RMS error delta separated models trained with and without battery voltage, indicating its information was captured by other network inputs.This held despite clearly biased dynamics below 3,650 mV at control frequencies greater than 100 Hz.
- Battery voltage: The logged data showed an inverse relationship between battery voltage and current thrust, linked to motor impedance changes with rotor speed and drive.
- Hardware aging: Motor damage changed equilibrium PWMs, while gyroscope noise increased over the quadrotor’s operating lifetime.
- Hardware aging: Flights at 75 Hz were abandoned after inconsistent performance following collisions and drift, and future work proposed conditioning models on sensor noise or adding safety constraints.
C. Frequency Dependent Learning
Learned performance increased with the number of training points at both tested frequencies. Higher control frequency improved learning speed in wall-clock time, but equal data volumes produced similar control ability.
- Training data: Flight time showed a continuing upward trend with the logarithm of available training points at both control frequencies.
- Training data: Further data collection could improve flight performance, but collecting substantially more experimental data was considered unrealistic without safer learning.
- Control frequency: Higher control frequency allowed the controller to learn faster on wall time.
- Control frequency: Equal numbers of trained points produced no notable difference in control ability across the tested frequencies.