PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for UAV Attitude Control

William Koch, Renato Mancuso, Richard West, Azer Bestavros

arXiv:1804.04154v1cs.RO

TL;DR

The paper addresses whether reinforcement learning can provide accurate, precise attitude control in the UAV inner loop, where PID performance can degrade under unpredictable dynamics. It develops the GYM FC simulation environment, trains controllers with several RL algorithms, and compares them with PID control. PPO-trained controllers outperform PID control on almost every metric, while TRPO and DDPG perform less reliably.

  • Problem

    PID control can be suboptimal under unknown dynamics, while prior RL work has focused mainly on guidance and navigation rather than time-sensitive attitude control.

  • Method

    The paper develops the open-source high-fidelity GYM FC environment with an Iris quadcopter digital twin, trains DDGP, TRPO, and PPO controllers, and compares them with PID control.

  • Results

    PPO-trained agents are the only ones reported as good enough for flight, achieving near-perfect success, while TRPO and DDPG often remain far from desired angular velocities.

  • Takeaways & Limitations

    RL can train accurate attitude controllers, and PPO-trained controllers outperform a fully tuned PID controller on almost every metric.

Abstract

from arXiv · show

Autopilot systems are typically composed of an "inner loop" providing stability and control, while an "outer loop" is responsible for mission-level objectives, e.g. way-point navigation. Autopilot systems for UAVs are predominately implemented using Proportional, Integral Derivative (PID) control systems, which have demonstrated exceptional performance in stable environments. However more sophisticated control is required to operate in unpredictable, and harsh environments. Intelligent flight control systems is an active area of research addressing limitations of PID control most recently through the use of reinforcement learning (RL) which has had success in other applications such as robotics. However previous work has focused primarily on using RL at the mission-level controller. In this work, we investigate the performance and accuracy of the inner control loop providing attitude control when using intelligent flight control systems trained with the state-of-the-art RL algorithms, Deep Deterministic Gradient Policy (DDGP), Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). To investigate these unknowns we first developed an open-source high-fidelity simulation environment to train a flight controller attitude control of a quadrotor through RL. We then use our environment to compare their performance to that of a PID controller to identify if using RL is appropriate in high-precision, time-critical flight control.

I. INTRODUCTION

UAV attitude control remains challenging because PID controllers can be suboptimal under unknown dynamics, while prior RL work has focused mainly on guidance and navigation. This paper develops and evaluates RL-based inner-loop controllers against PID control.

  • PID controllers perform close to ideally in stable environments but can be far from optimal under wind, payload, or voltage changes.
  • Prior intelligent flight-control research has focused mainly on guidance and navigation, leaving achievable accuracy for time-sensitive attitude control unclear.
  • The study develops GYM FC, an open-source high-fidelity environment using a digital twin of an Iris quadcopter for RL attitude-control training.
  • The learning architecture uses digital-twinning concepts intended to reduce effort when transferring trained controllers to hardware.
  • The evaluation compares DDGP, TRPO, and PPO controllers with a PID controller for intelligent flight-control performance.

II. BACKGROUND

Quadcopter attitude control maps rotor-speed changes into thrust and rotational effects within a time-sensitive inner loop. PID computes axis-wise feedback, then mixer geometry translates those signals into motor commands.

  • A quadcopter has six degrees of freedom but four motor inputs, making it underactuated and requiring onboard computation for stable flight.
  • Rotor speeds affect thrust and the roll, pitch, and yaw Euler angles through configuration-dependent aerodynamic effects.
  • Roll and pitch are produced through differential rotor thrust, whereas yaw is produced through torque differences from oppositely rotating rotors.
  • Attitude control computes motor signals to reach desired angular velocities for roll, pitch, and yaw in the time-sensitive inner loop.
  • For each axis, PID feedback terms are summed and translated into motor power through mixing based on frame geometry.

B. Reinforcement Learning

The RL controller observes angular-velocity errors and rotor speeds in a high-fidelity simulated environment, then selects continuous actions under partial observability. Policies are trained to maximize rewards using algorithms suited to continuous control.

  • The neural-network controller interacts with an Iris quadcopter in the high-fidelity Gazebo physics simulator.
  • At each timestep, observations include angular-velocity errors and rotor speeds, while the agent produces an action in continuous observation spaces.
  • Because the agent receives only sensor data, the environment is partially observed; the state therefore includes sequences of past observations and actions.
  • The agent-environment interaction is modeled as a Markov decision process, with a policy mapping states to actions to maximize reward over time.
  • The study uses DDPG and TRPO for continuous-action reinforcement learning, motivated by their recent use in quadcopter navigation control.

III. RELATED WORK

Prior intelligent flight-control research largely addressed navigation and guidance, while accurate and precise RL-based attitude control remained insufficiently evaluated. This work motivates physics-based simulation as a more realistic basis for attitude-controller training and benchmarking.

  • Adaptive PID approaches retain disadvantages including integral windup, mixing requirements, and reactive feedback control.
  • Online learning can adapt to disturbances but is limited by reliance on past experiences and may require aircraft mass and inertia knowledge.
  • Supervised-learning controllers can suffer from training data that inaccurately represents the aircraft’s underlying dynamics.
  • Earlier intelligent flight-control studies primarily focused on navigation and guidance rather than attitude control.
  • Evidence was limited for the accuracy and precision of neural-network-based intelligent attitude control, with no known evaluation of RL-trained controllers.
  • Physics simulations are used instead of mathematical aircraft and environment models to increase realism and provide attitude-control performance baselines.

IV. ENVIRONMENT

GYM FC is an RL training environment designed to teach aircraft attitude control through both episodic and continuous tasks. Its hierarchical architecture separates the digital twin, communication, and agent-environment interface layers.

  • GYM FC supports episodic tasks for responding to individual angular-velocity commands and continuous tasks for attitude-control learning.
  • The environment uses a three-layer hierarchy: digital twin, communication, and agent-environment interface.
  • Separating layers allows implementations such as the simulator to change without affecting other layers when interfaces remain intact.

A. Digital Twin Layer

The digital twin layer uses a high-fidelity Gazebo simulation of an Iris quadcopter, with a ball-joint setup enabling attitude control independently of guidance and navigation. Custom interfaces synchronize motor commands, simulation steps, and sensor responses.

  • The digital twin uses a high-fidelity physics simulator to provide realism and facilitate transfer to physical-platform interfaces.
  • Attitude control is isolated from guidance and navigation by linking the aircraft’s center of mass to a ball joint that permits free rotation.
  • The digital twin exposes simulation-reset and motor-update command interfaces to the communication layer.
  • The simulated Iris quadcopter is shown in Gazebo with its transparent body, center-of-mass ball joint, and model joints indicated.
  • Gazebo status messages were capped at 5 Hz, creating a simulation-and-learning bottleneck, while unprocessed messages could be silently dropped.
  • Custom plugin modifications let motor commands advance the simulation, then return IMU and ESC sensor data over UDP.

B. Communication Layer

The communication layer converts asynchronous simulator communication into a synchronized API for the agent-environment interface. It provides PWM motor writing and reset operations while blocking until simulator responses confirm synchronization.

  • The communication layer mediates between the digital twin and agent interface, exporting a synchronized API over asynchronous low-level communication.
  • Its interface provides pwm_write and reset commands to the agent-environment layer.
  • pwm_write converts actuator PWM vectors into the normalized UDP format expected by the Aircraft Plugin and blocks for a response.
  • A Gazebo socket-handle leak can cause crashes when reset commands exceed the operating system’s open-file limit during thousands of episodes.
  • Reset requests wait for world-statistics confirmation that the simulator’s time has been reset.

C. Environment Interface Layer

GYM FC exposes quadrotor attitude control through an OpenAI Gym interface, where randomized setpoints, actions, states, and rewards support RL training. The reward penalizes normalized tracking error, and timestep-by-timestep signaling performed better than sparse binary rewards.

  • Environment interface: GYM FC implements OpenAI Gym’s reset and step functions for episodic and continuous attitude-control tasks.Reset initializes the environment and computes a randomized target angular-velocity setpoint; step applies an action and returns the new state and reward.
  • Reward design: The reward converts summed absolute axis error into a clipped value in [−1, 0], so maximizing reward minimizes tracking error.The sum aggregates absolute error across axes, while clipping handles overflow; normalization supports training standardization and stabilization.
  • Reward design: Signaling performance at every timestep outperformed sparse binary rewards for the quadcopter-control task.Sparse target-reaching events were rare during early exploration and provided insufficient information for policy convergence.

V. EVALUATION

The evaluation compares RL-trained neural-network attitude controllers with a PID baseline using episodic tasks, standardized algorithm runs, physically feasible setpoints, and multiple accuracy metrics.

  • Evaluation scope: The study evaluates RL-trained neural-network attitude controllers and presents episodic-task results directly comparable with a PID baseline.The authors describe this as the first RL baseline conducted for quadcopter attitude control.
  • Algorithms: DDGP, TRPO, and PPO are evaluated using OpenAI Baselines implementations, with each algorithm trained for 10 million simulation steps.All other algorithm settings use defaults.
  • Task configuration: Episodes last at most 1 second, while target angular velocities span −5.24 to 5.24 rad/s (±300 deg/s).The duration allows controller response and possible steady-state identification; bounds were selected from PID performance to remain physically feasible.
  • Metrics: Accuracy is assessed independently for each axis using success, failure, and rise metrics based on progress toward the setpoint.Success measures settling within ±10% of the setpoint; failure measures remaining error at 500 ms for unsuccessful trials, and rise measures 10%-to-90% response time.

B. Results

PPO converged faster and accumulated higher rewards than TRPO and DDPG, while larger memory sizes reduced convergence and stability. In evaluation, PPO outperformed the other RL agents and exceeded PID performance on several attitude-control metrics.

  • Training: PPO converges faster than TRPO and DDPG and accumulates higher rewards during training.Agents trained for 10 million simulation steps, equivalent to 10,000 episodes or about 2.7 simulation hours.
  • Training: Larger memory sizes decrease convergence and stability across all trained algorithms.The authors attribute this decrease potentially to the enlarged state space slowing policy learning.
  • RL evaluation: PPO outperforms TRPO and DDPG on every reported Rise, Peak, Error, and Stability metric across 2,000 command-input evaluations.PPO is the only algorithm achieving stability for every memory size.
  • Comparison with PID: PPO agents are the only RL controllers considered good enough for flight, achieving a near-perfect success rate against the PID comparison.TRPO’s best agent misses the desired pitch target 39.2% of the time, with errors reaching upwards of 20% from the setpoint.
  • Comparison with PID: PPO with m = 1 exceeds PID in rise time, peak velocities achieved, and total error while maintaining less overshoot and reaching a stable state halfway through simulation.In a sampled step response, PPO also tracks the desired attitude without the slight roll and yaw overshoot observed for PID.

VI. FUTURE WORK AND CONCLUSION

The paper presents GYM FC and concludes that RL can train accurate attitude controllers, with PPO outperforming a fully tuned PID controller on almost every metric. It also identifies transfer to continuous tasks without retraining as a promising direction.

  • GYM FC is presented as an RL training environment for developing intelligent UAV attitude controllers.
  • PPO-trained attitude controllers outperformed a fully tuned PID controller on almost every metric.
  • PPO agents trained with episodic tasks also performed exceptionally well in continuous tasks without retraining.
  • Figure 5 compares the best trained RL agents and PID using target angular velocity Ω∗=[2.20, −5.14, −1.81] rad/s and ±10% initial-error bands.
  • Figure 6 compares PPO and PID step responses and PWM motor signals for target angular velocity Ω∗=[2.11, −1.26, 5.00] rad/s.

APPENDIX A CONTINUOUS TASK EVALUATION

The appendix evaluates a PPO agent trained only on episodic tasks in continuous command environments. The agent performs exceptionally well without retraining and achieves lower overall error than PID in one comparison.

  • A PPO agent trained with episodic tasks performed exceptionally well when evaluated in a continuous task.
  • The PPO agent achieved a 22% decrease in overall error compared with the PID agent in a continuous-task sample.
  • Continuous-task evaluation issued commands at random intervals and maintained each command for a random duration between 0.1 and 1 seconds over 60 seconds.
  • Figure 8 provides a close-up of continuous-task tracking and the corresponding PWM motor output.
Loading 1804.04154v1…