PaperlayerView exact revision on arXiv

Source-linked AI summary

Control of a Quadrotor with Reinforcement Learning

Jemin Hwangbo, Inkyu Sa, Roland Siegwart, Marco Hutter

arXiv:1707.05110v1cs.RO

TL;DR

The paper addresses quadrotor control without requiring a predefined controller structure. It trains a direct state-to-actuator neural policy with a conservative deterministic reinforcement-learning algorithm, then evaluates it in simulation and on real hardware. The policy achieves reliable recovery and waypoint tracking while remaining computationally inexpensive.

  • Problem

    Reinforcement learning had largely been used for higher-level robotic decisions rather than direct low-actuator control, motivating fully controlling an aerial vehicle with a learned policy.

  • Method

    The paper trains a model-free neural-network policy using deterministic policy optimization, natural gradient descent, and trajectory-based exploration.

  • Results

    The learned policy had a 4 % failure rate versus 71 % for a linear MPC baseline across 100 recovery rollouts.

  • Takeaways & Limitations

    The same learned policy controlled the quadrotor in simulation and on real hardware, including waypoint tracking and recovery from manual throws.

Abstract

from arXiv · show

In this paper, we present a method to control a quadrotor with a neural network trained using reinforcement learning techniques. With reinforcement learning, a common network can be trained to directly map state to actuator command making any predefined control structure obsolete for training. Moreover, we present a new learning algorithm which differs from the existing ones in certain aspects. Our algorithm is conservative but stable for complicated tasks. We found that it is more applicable to controlling a quadrotor than existing algorithms. We demonstrate the performance of the trained policy both in simulation and with a real quadrotor. Experiments show that our policy network can react to step response relatively accurately. With the same policy, we also demonstrate that we can stabilize the quadrotor in the air even under very harsh initialization (manually throwing it upside-down in the air with an initial velocity of 5 m/s). Computation time of evaluating the policy is only 7 μs per time step which is two orders of magnitude less than common trajectory optimization algorithms with an approximated model.

I. INTRODUCTION

The paper applies reinforcement learning to direct quadrotor state-to-rotor-thrust control, avoiding a predefined controller structure. It introduces a conservative learning algorithm and evaluates the resulting policy in simulation and on real hardware.

  • The policy directly maps quadrotor state to rotor thrusts, requiring few structural assumptions about the controller.
  • The authors introduce a deterministic on-policy learning algorithm designed for stable, high-performance quadrotor policy learning.The method uses zero-bias, zero-variance samples and a small number of high-quality samples.
  • The trained policy is evaluated for waypoint tracking and recovery from random states in simulation and for waypoint tracking and manual-throw recovery on real hardware.
  • The paper reports that the learned policy outperformed two established algorithms in computation time for this task.
  • 7 µs is the reported computation time for evaluating the policy on a single Intel Xeon E5-1620 core.The paper positions this low evaluation cost as leaving more computational resources for other onboard algorithms.

II. BACKGROUND

The paper develops deterministic policy optimization for reinforcement learning with continuous control, emphasizing lower-variance estimates and computational efficiency. It also identifies exploration as a central challenge because deterministic policies lack a direct exploration rule.

  • Deterministic policy optimization uses natural gradient descent and is motivated by lower-variance on-policy value estimates.The variance is zero when system dynamics are deterministic.
  • Deterministic policy gradients provide a simpler policy-gradient form that is computationally attractive.
  • Deterministic policies require a good exploration strategy because they lack the clear state-space exploration rule available to stochastic policy gradients.
  • The reinforcement-learning objective is to learn a parameterized policy that minimizes expected cumulative deterministic cost from sampled trajectories.
  • The value function represents averaged value over the stable state distribution, with discounting used to keep its values finite.
  • The action-value function evaluates an action followed by the policy, while the advantage compares that action with the current policy action.

III. METHOD

The method trains separate value and policy networks on quadrotor state representations using deterministic exploration and natural-gradient policy optimization. Long trajectories are selected to prioritize stable and reliable convergence in noiseless simulation.

  • A. Network Structure: The training procedure uses separate value and policy networks, both receiving the quadrotor state as input.
  • A. Network Structure: The quadrotor state uses nine rotation-matrix elements together with position, linear velocity, and angular velocity.
  • B. Exploration Strategy: Exploration separates trajectories into initial, junction, and branch categories, with noisy off-policy junction trajectories and on-policy initial and branch trajectories.
  • B. Exploration Strategy: Junction trajectories broaden the sample distribution, but excessive length can violate the assumption that junctions follow the policy's stable-state distribution.
  • B. Exploration Strategy: The simulations synchronize many trajectories so the policy network can evaluate batches of states, reducing network-call overhead.
  • B. Exploration Strategy: Long branch trajectories reduce advantage-estimation bias, while increasing evaluations per iteration.

C. Value Function Training

The method trains value and policy networks from on-policy trajectories using Monte-Carlo estimates, Huber loss, and deterministic natural-gradient policy optimization. It uses trust-region updates, SVD-based pseudoinversion, and manually defined exploration noise to keep optimization stable and computationally practical.

  • Value Function Training: Finite-horizon on-policy trajectories use the current value function to estimate terminal values for Monte-Carlo value targets.The value function is trained from these on-policy Monte-Carlo samples.
  • Value Function Training: The value optimizer uses all states, Huber loss, up to 200 iterations per learning step, and terminates when loss falls below 0.0001.
  • Policy Optimization: Natural-gradient policy updates constrain each sample's contribution with a trust region because small noise vectors can produce very large gradient estimates.The per-sample natural gradient is defined through the action-space Mahalanobis metric.
  • Policy Optimization: SVD computes an exact pseudoinverse for the rank-deficient policy-parameter Hessian, avoiding the explicit inverse required by the formulation.The paper contrasts this with conjugate gradient, which provides an approximate solution.
  • Computational Cost: 0.35 ms per sample is reported for Cholesky decomposition and SVD, versus 1.1 ms for 10 conjugate-gradient iterations on the benchmark Jacobian.The Cholesky decomposition and SVD account for about 20% of whole policy optimization in the given network.
  • Exploration: The action-noise covariance is manually defined for exploration, rather than optimized, and is intended to reflect the scale of actions.Automatic covariance adjustment is identified as future work.

IV. POLICY OPTIMIZATION IN SIMULATION

The authors developed a C++ robotic-AI framework and a numerically accurate C++ quadrotor simulator to support efficient training and analysis. Simulation dynamics integration took substantially less computation time than network training.

  • Simulation Infrastructure: The C++ Robotic Artificial Intelligence framework provides logging, timing, plotting, 3D animation, and video recording utilities.These utilities support debugging and analysis of computational resource consumption.
  • Simulation Infrastructure: The authors used their own C++ quadrotor simulator to improve numerical accuracy and stability.
  • Computational Cost: Dynamics integration required far less computation time than neural-network training.

A. Robot model for simulation

The simulator models the quadrotor as a floating body acted on by four thrust forces, omitting drag and thresholding negative thrust to zero. A boxplus integrator enables a 0.01 s time step for dynamic motion.

  • Robot Model: The simulation uses a simple floating-body model with four thrust forces and ignores all drag forces.
  • Robot Model: The motion equation uses rotor-center Jacobians, thrust forces, inertia, generalized acceleration, and Coriolis and gravity effects.
  • Actuation Constraint: Negative thrust is thresholded to zero because propellers can produce only positive upward force.
  • Numerical Integration: 0.01 s is the integration time step enabled by the boxplus operator for improved accuracy during dynamic motion.

B. Problem Formulation

The task is waypoint tracking without trajectory generation, together with stabilization from physically possible initial configurations. Training targets the inertial-frame origin, while waypoint-relative states allow the same policy to track arbitrary waypoints.

  • Problem Formulation: The objective is waypoint tracking without generating a trajectory.
  • Problem Formulation: The policy is trained to reach the inertial-frame origin and receives the state minus the waypoint location during operation.This avoids explicitly training separate waypoint-tracking behavior.
  • Problem Formulation: The stabilization goal includes physically possible configurations such as upside-down orientation with random linear and angular velocity.
  • Controller Design: A low-gain PD attitude controller is added to stabilize the learning process, while the final controller is intended to rely on the learned policy.The paper states that the PD controller does not aid the final controller.
  • Controller Design: The PD controller uses orientation and angular velocity, with gains of −0.2 and −0.06 except for z-direction gains set to one sixth of the others.Euler-angle PD control is considered insufficient near orientation singularities.
  • Cost Function: The cost emphasizes position error, while other cost terms are set to roughly one tenth of its magnitude and γ = 0.99.

C. Network Training

Training used conservative settings to prioritize stable, reliable convergence, while comparisons showed the proposed method was more practical than TRPO because it reduced neural-network computation burden.

  • Training configuration: 512 initial trajectories and 1024 branching trajectories with noise depth 2 yielded 1.0 million time steps per iteration.Parallelized rollouts took less than ten seconds per iteration.
  • Training configuration: The conservative advantage-estimation method was sample-expensive but designed for stable and reliable convergence rather than fast convergence.
  • Algorithm comparison: Figure 4 compares policy-optimization learning curves across three algorithms and five runs per algorithm.
  • Algorithm comparison: TRPO and the proposed method performed similarly in performance per simulation time, but the proposed method was more practical because TRPO incurred greater backpropagation and conjugate-gradient costs.The proposed method’s computational resource consumption is illustrated in Figure 5.

D. Performance in Simulation

In simulation, the learned policy recovered reliably from randomized quadrotor states and required substantially less computation per step than the linear MPC baseline.

  • Stability: 4% versus 71% failure rates were recorded for the learned policy and linear MPC controller, respectively, across 100 randomized rollouts.Failure meant that the quadrotor touched the ground.
  • Stability: Certain initial conditions, such as an upside-down quadrotor with full downward speed, were impossible to recover from, despite the policy’s overall recovery reliability.
  • Computational cost: 7 µs were required to evaluate the policy for a given state, compared with about 1,000 µs for one linear MPC time step.The policy implementation used matrix arithmetic with the Eigen library.

V. EXPERIMENTS

Experiments transferred the simulation-trained policy to a real quadrotor for waypoint tracking and recovery tests, while exposing several simulation-to-reality differences that affected performance.

  • Experimental setup: The same simulation-trained policy was evaluated on a Hummingbird quadrotor with onboard computation and Vicon-based state information.The simulation used the same model parameters, and the vehicle carried an Intel Computer Stick.
  • Experimental setup: Table I lists the physical parameters of the quadrotor used in the experiments.
  • Experimental setup: Figure 6 depicts the experimental system, linking position and orientation states, desired and estimated position, thrust, rotor speed, and IMU measurements across rates.
  • Simulation-to-reality differences: Simulation and real-vehicle dynamics differed because motor regulation was unknown, near-ground aerodynamics were omitted, parameters changed, and wireless delay and state-estimation error affected performance.The motor-speed dynamics were not identified, and battery level and weight distribution changed during operation.
  • Waypoint tracking: Waypoint tracking used four points at the vertices of a 1 m-by-1 m square and showed a minor, nonsignificant tracking error.Figure 7 shows the resulting trajectory.
  • Waypoint tracking: The policy was not trained with varied external disturbances, so its tracking error was expected to exceed that of high-gain classical controllers.
  • Recovery: Manual throws tested recovery from challenging configurations, including upside-down orientation with high linear and angular velocities, using matching experimental and simulated trajectories.
  • Recovery: A trajectory at 45 deg showed smooth, natural motion while the policy stabilized the quadrotor at high velocity.

VI. CONCLUSION

The model-free neural-network policy achieved strong quadrotor performance while remaining computationally cheap, but simulation fidelity and broader task validity remain open boundaries.

  • The model-free neural-network policy achieved outstanding quadrotor performance without requiring a sophisticated model-exploiting control structure.
  • A conservative learning algorithm avoided divergence during training and outperformed two established algorithms for this task in computation time.It uses many simulation steps while minimizing neural-network training steps.
  • 1.3 cm steady-state error was observed in simulated waypoint tracking, although tracking error exceeded that of optimization-based controllers.The authors attribute the higher error to training without disturbances present in the real environment.
  • The policy completed extended simulated waypoint tracking without failure and remained highly stable during manual-throw recovery tests.The quadrotor was reported to be more stable in the manual-throw test than in simulation, possibly because real-world air drag and gyroscopic effects helped stabilize it.
  • Future work targets more accurate simulation models, recurrent policies that adapt to modeling errors, and transfer learning on the real system.These directions aim to capture unknown system dynamics and improve policy performance.
Loading 1707.05110v1…