Source-linked AI summary
Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimization
Eivind Bøhn, Erlend M. Coates, Signe Moe, Tor Arne Johansen
TL;DR
Fixed-wing UAV autopilots have limited ability to stabilize nonlinear aircraft dynamics across challenging flight conditions. This paper develops a PPO-based DRL controller and finds that it stabilizes attitude and airspeed, often converges in more cases than PID control, and generalizes to unseen wind and turbulence.
Problem
Fixed-wing UAV autopilots struggle with nonlinear attitude stabilization across broad flight conditions, while real-world RL data collection is costly and risky.
Method
The paper trains a PPO controller in a Skywalker X8 flight simulator using compact observations augmented with several previous time steps.
Results
The RL controller stabilizes roll, pitch, and airspeed, converges in more evaluation cases than PID control, and generalizes to unseen wind and turbulence, including severe disturbances.
Takeaways & Limitations
DRL is a promising candidate for nonlinear fixed-wing flight control within the simulated conditions studied.
Abstract
from arXiv · showhide
Contemporary autopilot systems for unmanned aerial vehicles (UAVs) are far more limited in their flight envelope as compared to experienced human pilots, thereby restricting the conditions UAVs can operate in and the types of missions they can accomplish autonomously. This paper proposes a deep reinforcement learning (DRL) controller to handle the nonlinear attitude control problem, enabling extended flight envelopes for fixed-wing UAVs. A proof-of-concept controller using the proximal policy optimization (PPO) algorithm is developed, and is shown to be capable of stabilizing a fixed-wing UAV from a large set of initial conditions to reference roll, pitch and airspeed values. The training process is outlined and key factors for its progression rate are considered, with the most important factor found to be limiting the number of variables in the observation vector, and including values for several previous time steps for these variables. The trained reinforcement learning (RL) controller is compared to a proportional-integral-derivative (PID) controller, and is found to converge in more cases than the PID controller, with comparable performance. Furthermore, the RL controller is shown to generalize well to unseen disturbances in the form of wind and turbulence, even in severe disturbance conditions.
I. INTRODUCTION
The paper motivates deep reinforcement learning for fixed-wing UAV attitude control because conventional autopilots struggle with nonlinear, coupled flight dynamics. It proposes a PPO-based proof-of-concept controller and examines factors affecting training progression and performance.
- Nonlinearities, actuator saturation, aerodynamic uncertainty, and changing attitude and airspeed make low-level attitude stabilization increasingly difficult.
- Conventional fixed-wing UAV autopilots use decoupled, linear control designs despite strongly coupled and nonlinear aircraft dynamics.
- RL offers a model-free approach for complex continuous control, but neural-network training can be unstable and sample-inefficient.
- Simulation is preferred for RL training because collecting real UAV data can be unsafe, but simulator mismatch creates a reality gap.
- The study develops a PPO controller in a Skywalker X8 simulator to control roll, pitch, and airspeed, while identifying design factors affecting training and final performance.
II. RELATED WORK
Prior UAV reinforcement-learning research spans high-level planning, model-based learning, and low-level control, while this work focuses on online self-learning for fixed-wing attitude stabilization. The paper also introduces the UAV model foundations used for simulation and control.
- Related UAV reinforcement learning: UAV reinforcement learning has addressed navigation, path planning, guidance, obstacle avoidance, and trajectory planning, often alongside conventional low-level controllers.
- Low-level attitude control: Earlier low-level RL attitude-control studies largely concerned helicopter UAVs and used offline learning from expert pilot data.
- Low-level attitude control: The paper instead studies online self-learning for fixed-wing UAV attitude control using reinforcement learning.
- UAV model: The UAV is modeled as a rigid body with position, linear velocity, angular velocity, and quaternion attitude states governed by kinematic and Newton-Euler equations.
- UAV model: Euler angles are used for controller inputs and plotting, while unit quaternions represent attitude during arbitrary maneuvers.
A. Aerodynamic Forces and Moments
The aerodynamic model represents wind-relative flight and computes forces and moments from nonlinear aerodynamic coefficients. It includes steady and stochastic wind disturbances and extends the coefficient model beyond small-angle linear behavior.
- The wind field combines steady wind, stochastic gusts and turbulence, and rotational disturbances to define the UAV’s relative air motion.
- Airspeed, angle of attack, and sideslip angle are calculated from the UAV velocity relative to the surrounding air mass.
- Stochastic wind components are generated by passing white noise through Dryden velocity-spectrum shaping filters.
- Aerodynamic forces and moments use coefficients that depend nonlinearly on angles, angular rates, and control-surface deflections.
- Lift, drag, and pitch-moment coefficients are extended with nonlinear Newtonian flat-plate theory to increase validity beyond small angles.
B. Propulsion Forces and Moments
The propulsion model computes thrust from propeller discharge velocity, airspeed, motor and propeller parameters, and throttle. Propeller moments are modeled separately, with gyroscopic moments assumed negligible.
- Propeller thrust is modeled along the aircraft body x-axis.
- Thrust depends on discharge velocity, airspeed, propeller disc area, an efficiency factor, and throttle.
- The throttle is constrained to δt ∈[0, 1].
- Propeller moments are included using experimentally based parameters, while gyroscopic moments are assumed negligible.
C. Actuator Dynamics and Constraints
The controller models elevon dynamics as rate-limited, saturated second-order integrators and throttle dynamics with a first-order transfer function. Actuator deflections and rates are explicitly constrained.
- Actuator models: Elevon control surfaces are modeled as rate-limited and saturated second-order integrators.The model applies separately to the right and left elevons.
- Actuator constraints: ±30 degrees and ±200 degrees per second constrain elevon deflections and rates, respectively.
- Actuator models: Throttle dynamics are represented by a first-order transfer function with T = 0.2.
IV. METHOD
The method uses PPO to control fixed-wing UAV roll, pitch, and airspeed through actuator commands. The observation design combines sensor-feedback variables, actuator history, and several previous time steps to support a memoryless policy network.
- Algorithm choice: PPO was selected for its reported attitude-control performance, robust hyperparameters, high performance, and low computational complexity.
- Controlled variables: The controller regulates roll, pitch, and airspeed rather than directly controlling yaw angle.Yaw is controlled indirectly through yaw-rate dependence on roll, while airspeed supports energy-efficient flight, stall avoidance, and control-surface effectiveness.
- Action space: The three-dimensional action space contains commanded virtual elevator and aileron angles plus throttle.Elevator and aileron commands are mapped to commanded elevon deflections.
- Observation design: The observation vector uses standard sensor-feedback states, moving averages of actuator setpoints, and values from several previous time steps.Previous-step values provide dynamic information because the policy network has no memory.
A. The Proximal Policy Optimization Algorithm
PPO is a model-free actor-critic policy-gradient method whose policy network maps observations to continuous actuator actions. It improves sample reuse with importance sampling and constrains policy updates by clipping probability ratios.
- PPO formulation: PPO is a model-free, on-policy, actor-critic, policy-gradient method.Its policy network takes the observation vector as input and outputs elevator, aileron, and throttle setpoints.
- Policy outputs: The policy network outputs means and variances of a multivariate Gaussian from which training actions are sampled.After training, the mean is used as the action.
- Policy optimization: Policy gradients are estimated from trajectories collected by running the policy in the environment and differentiating a surrogate loss.Automatic differentiation backpropagates the gradients through the neural network to update its parameters.
- Actor-critic estimation: The critic estimates the value function, while the advantage function compares an action with alternatives available in the same state.Generalized advantage estimation and multiple simultaneous actors are used to improve advantage estimates and sample-batch size.
- Clipped updates: Importance sampling lets PPO reuse samples from an old policy for several updates of a new policy, while clipping the probability ratio to [1 − ϵ, 1 + ϵ].The clipping limits policy divergence and provides a first-order trust-region approach.
B. Action Space
The controller uses normalized, clipped actions that are later scaled to actuator ranges, while training defines initial-condition and target-setpoint constraints. Gaussian exploration can produce actuator-wearing fluctuations in online training.
- Action smoothness: High-variance Gaussian exploration can generate highly fluctuating actions that may wear actuators during online training.The authors identify this as less problematic in simulation than on a real aircraft.
- Action bounds: The action variable is clipped between supplied bounds before being used by the controller.
- Training conditions: Table I specifies constraints and ranges for the initial conditions and target setpoints used during controller training.
- Action scaling: Actions are constrained to a small normalized range and then scaled to the physical actuator ranges.This design trains the controller to output actions as fractions of minimum and maximum setpoints.
C. Training of Controller
The PPO controller is trained episodically in simulation to control roll, pitch, and airspeed, using normalized observations and a reward that penalizes tracking errors and actuator changes.
- Policy architecture: The PPO policy processes observations through a temporal convolutional layer before the default two-hidden-layer MLP policy.The convolution uses three filters spanning the time dimension for each observation component.
- Training setup: The controller assumes control of an aircraft in motion and learns to orient it toward randomized attitude and airspeed references.Episodic training allows adjustment of episode conditions and exploration of the state space.
- Reward design: The reward function uses L1 state-tracking error and penalizes changes in actuator setpoints to discourage oscillatory control.Reward components are weighted by γ factors and scaled by approximate dynamic ranges represented by ζ factors.
- Observation processing: Observation components are normalized using running means and variances so neural-network inputs share a common scale.The normalization is implemented with the VecNormalize class.
D. Evaluation
The evaluation compares fixed-gain PID and RL controllers across wind and turbulence conditions using success, timing, overshoot, and control-variation criteria.
- Baseline controller: Fixed-gain PID controllers for roll, pitch, and airspeed provide the baseline comparison for the RL controller.Throttle controls airspeed, while virtual aileron and elevator commands control roll and pitch.
- Baseline tuning: PID gains are manually tuned by trial and error for acceptable transient responses and low steady-state errors over a range of initial conditions and setpoints.Wind is disabled during tuning, and integral terms are implemented numerically using forward Euler.
- Test conditions: The controllers are evaluated with no turbulence and with light, moderate, and severe turbulence, using randomized 0, 7, 15, and 23 m/s steady-wind magnitudes.Wind orientation is randomized, and additive turbulence follows the Dryden turbulence model.
- Evaluation criteria: Success is the primary evaluation criterion, while rise time, settling time, overshoot, and control variation provide additional performance measures for successful episodes.Success requires controlling the state within a specified bound of the setpoint.
V. RESULTS AND DISCUSSION
The trained RL controller converges quickly, benefits strongly from a compact history-based observation vector, and generalizes to new setpoints and disturbances while matching PID performance overall.
- Training and inference: Around 2 million training time steps produce convergence in about one hour, while inference takes 800 microseconds on the stated hardware.The reported inference time supports operation at the assumed 100 Hertz autopilot sampling frequency.
- Key training factors: Reducing observations to essential states and adding five previous time steps significantly accelerates training and improves final performance.The essential variables include airspeed, roll and pitch angles, angular velocities, and state errors.
- Reward design: The reward-function alternatives were not studied extensively because the original L1-based design produced satisfactory results.
- Generalization: The RL controller adapts to unseen setpoints and unmodeled wind and turbulence despite training without wind estimates or wind forces.Quantitative analysis covers conditions similar to normal operation, while figures examine more challenging tasks qualitatively.
- Controller comparison: Both controllers generally converge, but RL succeeds in more situations, including all attitude-state cases, and is more robust in moderate and severe turbulence.PID has lower control variation in simple settings, whereas RL has the least control variation under severe turbulence.
- Controller comparison: The controllers have similar rise and settling times, while PID performs better on overshoot.Each controller has the edge in different states under different conditions.
- Steady-state behavior: RL eliminates steady-state errors in the illustrated cases despite using only recent states and references, possibly through learned feed-forward action or high-gain feedback.The authors propose evaluating integral-error observations in future work.
VI. CONCLUSIONS
The proof-of-concept RL controller is presented as a promising candidate for nonlinear fixed-wing flight control, including generalization to turbulent wind. Its scope remains bounded by simulation-to-reality uncertainty and by constraints on more advanced maneuvers.
- Conclusions: The authors suggest DRL is a good candidate for nonlinear flight control design based on the controller’s learning and turbulent-wind generalization.
- Conclusions: Transfer from simulation to real-world flight remains unresolved because the severity of the reality gap is unknown.Future work is identified as evaluating robustness to parametric and structural aerodynamic uncertainties before real flight experiments.
- Conclusions: The RL controller generalized to continuous setpoint tracking and severe wind and turbulence conditions despite being trained without wind or turbulence.The figure caption specifies disturbance magnitude of 20 m/s.
- Conclusions: For aerobatic flight or recovery from extreme situations, the controller may need more freedom to adjust airspeed.The proposed adjustment is to make airspeed an uncontrolled state.
- Conclusions: More complex policy architectures, training designs, reward structures, and off-policy algorithms are identified as possible directions for tighter control and greater robustness.SAC is suggested if the reality gap becomes a major obstacle because it can learn offline from gathered data.