Source-linked AI summary
Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search
Tianhao Zhang, Gregory Kahn, Sergey Levine, Pieter Abbeel
TL;DR
The paper addresses how to train aerial-vehicle policies without relying on difficult state estimation at test time or exposing unstable vehicles to catastrophic failures during learning. It combines MPC with guided policy search to generate full-state training data for a deep neural network that uses raw onboard sensors, and reports successful, generalized control without catastrophic training failures.
Problem
MPC can be computationally demanding and require difficult state estimation, while reinforcement learning can catastrophically fail on unstable aerial vehicles before learning an effective policy.
Method
MPC generates guiding data under full-state observations in guided policy search, which trains a deep neural network policy using only raw onboard sensor observations.
Results
The method learned complex quadrotor behaviors, including high-speed obstacle avoidance, without catastrophic training failures, and the trained policies generalized effectively.
Takeaways & Limitations
The trained neural network can control the vehicle from raw sensor inputs without explicit state estimation and is substantially less expensive to evaluate than full MPC.
Takeaways & Limitations
Full state observations are required during training, potentially through an instrumented environment such as motion capture.
Abstract
from arXiv · showhide
Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.
I. INTRODUCTION
The paper combines MPC-guided policy search with deep neural networks to train aerial-vehicle policies safely using full state during training but only onboard sensor observations at test time. This addresses MPC’s computational and state-estimation demands while avoiding reinforcement learning’s catastrophic training failures.
- MPC is robust and effective for aerial-vehicle control but computationally demanding and dependent on challenging state estimation in unstructured environments.
- The approach avoids exposing unstable aerial vehicles to partially trained policies that could cause catastrophic failures during reinforcement-learning training.MPC is robust to moderate model errors, unlike learned dynamics models that are inaccurate early in training.
- During training, MPC uses full state observations from an instrumented environment while the neural network learns from raw onboard sensor observations.The policy can use IMU, laser range-finder, or other complex raw sensor information, while MPC uses the full system state.
- The proposed method replaces offline trajectory optimization with MPC-guided policy search to provide supervision for deep neural network control policies.Guided policy search converts reinforcement learning into supervised learning using an optimal control algorithm’s data.
- The resulting neural network policy is substantially cheaper to evaluate than MPC and can operate without explicit state estimation at test time.The paper evaluates simulated quadrotor tasks including obstacle avoidance with simulated laser range sensors.
III. PRELIMINARIES
Guided policy search minimizes expected trajectory cost while aligning state-based guiding controllers with an observation-based nonlinear policy. Its alternating optimization uses sampled trajectories, policy matching, and penalties that enforce behavioral agreement.
- Policy search minimizes the expected cumulative cost of trajectories generated by a policy that maps observations to actions.The trajectory includes states, actions, and a task cost, while observations are generated from the underlying state through an observation distribution.
- Guided policy search alternates trajectory optimization and policy optimization so state-based controllers and the observation-based policy converge to similar behavior.The framework trains an arbitrary nonlinear policy using samples from guiding trajectory distributions.
- The BADMM formulation adds action-matching and KL-divergence penalties to keep each guiding controller close to the learned policy.The trajectory-optimization objective uses Lagrange multipliers and a KL-divergence term to enforce agreement.
- The nonlinear policy is trained from samples collected from multiple guiding trajectory distributions, with the sampled objective reducing to weighted least squares for Gaussian policies.The sample count and trajectory length determine the empirical supervised objective, while the Gaussian covariance can be solved in closed form.
- Because the learned policy uses observations rather than the true state, it can operate after training when the full state is unavailable.Prior approaches used offline trajectory optimization or trajectory-centric reinforcement learning, which can be problematic for unstable systems.
IV. MPC-GUIDED POLICY SEARCH
The MPC-guided policy-search method repeatedly uses approximate-model MPC to generate training samples while penalizing deviation from the current neural network policy. These samples train a nonlinear policy that can generalize to new states.
- Approximate-model MPC selects actions from varied initial states to generate guiding samples for training the nonlinear policy.The resulting samples are used as supervised training data for the observation-based policy.
- The current neural network policy is included in the next sampling cost, so repeated MPC rollouts and supervised updates produce an effective nonlinear policy.The policy is retrained after each new batch of samples and can generalize to new states.
- MPC must minimize both task cost and deviation from the neural network policy when generating guiding trajectory distributions.This modification adapts guided policy search to online MPC-generated trajectories.
- Short MPC horizons require offline target trajectories and a compatible tracking objective, while changing local controllers require modifying supervised policy learning.These adaptations address structural differences between MPC-generated samples and generic guided policy search distributions.
A. Model Predictive Control with DDP
The MPC method uses iterative LQG, a DDP variant, to optimize locally approximated dynamics and costs over a short receding horizon. It repeatedly computes a control law through backward dynamic programming and forward trajectory rollouts.
- Iterative LQG uses an approximate system-dynamics model and local linear-quadratic expansions to solve the optimal control problem.
- The algorithm linearizes dynamics around a nominal trajectory and approximates the cost quadratically.
- Under these approximations, recursive Q-function and value-function computations yield the optimal policy.
- The resulting control law is g(x_t) = û_t + k_t + K_t(x_t − x̂_t), with feedback determined by Q-function derivatives.
- A forward rollout produces a new nominal trajectory, after which the backward pass repeats to take the next Gauss–Newton step.
- MPC adapts DDP by running the backward pass over a shorter horizon H from the current time step t to t + H.
B. Adapting MPC for Guided Policy Search
The method adapts MPC for guided policy search by using offline iterative LQG to define state distributions and a surrogate cost for robust online rollouts. These rollouts support long-horizon behavior and produce training data despite model errors and disturbances.
- Short-horizon MPC is supplemented with offline iterative LQG because obstacle avoidance requires long-horizon lookahead.
- The offline optimization produces a linear-Gaussian controller used to construct a surrogate cost for robust MPC trajectory generation.
- The surrogate cost encourages high-probability states, supports long-horizon behavior under short-horizon optimization, and keeps behavior near the neural policy.
- MPC computes future state distributions under approximate dynamics and the offline controller, then marginalizes them at each future time step.
- The state-focused construction preserves similar state distributions even when model errors or perturbations change the actions suggested by MPC.
- MPC samples actions from a new linear-Gaussian controller at each time step, with indices identifying trajectory distributions and samples.
- The generated samples are not exact samples from π(τ), but they still provide good training data and allow MPC to succeed with model errors and disturbances.
- Estimating log πθ(u_t|x_t) from policies defined on observations requires fitting a time-varying linear-Gaussian approximation using physical-system rollouts.
C. Training the Nonlinear Policy
The nonlinear policy is trained by supervised learning to match MPC-generated linear-Gaussian controllers. A multilayer neural network maps onboard observations to actions, enabling policies that use raw sensor inputs.
- The nonlinear policy is trained with standard supervised learning by substituting MPC controllers for the trajectory-distribution policies in the KL objective.
- For a conditionally Gaussian policy, the KL objective becomes a weighted least-squares objective on the neural network’s mean function.
- The mean function μ_π(o_t) is represented by a multilayer neural network for flexible and expressive policies.
- The policy covariance is chosen constant to obtain deterministic or nearly deterministic policies and solve it in closed form.
- The neural network mean is optimized with stochastic gradient descent using MPC-generated samples.
- Because its observation input need not match the MPC state input, the policy can operate directly on raw onboard sensor data.
- Algorithm 2 alternates offline policy optimization, MPC rollouts, nonlinear-policy training, local policy fitting, and distribution updates.
D. Algorithm Summary
Each iteration combines offline iterative LQG with MPC rollouts to generate controllers and trajectories, then trains and updates the nonlinear policy from those samples.
- Each iteration first optimizes an offline solution with iterative LQG, which initializes the MPC rollout cost.
- M MPC rollouts generate sample trajectories and MPC controllers for each trajectory distribution using a newly constructed surrogate cost.
- The resulting trajectories and controllers train the nonlinear neural network policy, completing the method’s iterative loop.
V. EXPERIMENTAL EVALUATION
The method was evaluated on simulated quadrotor obstacle-avoidance tasks.
- The evaluation used simulated quadrotor obstacle-avoidance tasks.
A. Quadrotor System
The simulated quadrotor is modeled with a 13-dimensional full state, four rotor-velocity controls, and onboard sensing for obstacle-aware control.
- Quadrotor model: The vehicle state includes position, velocity, quaternion orientation, and angular velocity in R13.
- Quadrotor model: The four controls are the quadrotor’s rotor velocities.
- Cost function: The cost penalizes deviation from target height, velocity, orientation, angular velocity, and hover rotor velocity.
- Cost function: Obstacle proximity contributes a hinge loss only when the nearest obstacle is closer than dSAFE.
- Task setting: The task requires flying around a cylindrical obstacle and down a hallway using onboard laser range-finder sensing.
C. Neural Network Policy
The learned policy uses a compact ReLU network and is trained across obstacle-avoidance environments, baselines, and model-error conditions.
- Neural network: The neural network has two fully connected hidden layers with 40 ReLU units each.
- Neural network: Training uses ADAM with 20,000 minibatches of size 50 in each guided-policy-search iteration.
- Experimental design: Policies are trained on a single cylinder or straight hallway and evaluated in corresponding generalized environments.
- Baselines and model errors: The comparison includes full MPC-guided policy search, a true-cost variant, and a no-MPC baseline under four model-error conditions.
- Evaluation metrics: Table I averages test flight duration over 20 runs and marks excessively crashing experiments as N/A.
F. Results
MPC-guided policy search trained successful obstacle-avoidance policies without catastrophic training failures and generalized effectively, while using onboard sensor inputs at deployment.
- Training safety: The surrogate-cost MPC variant trained a successful neural network policy without catastrophic failure during training.
- Generalization: With model errors, the proposed method outperformed both baselines in the majority of test scenarios.
- Overall results: The evaluation found that both obstacle-avoidance behaviors were learned without catastrophic training failures and that the policies generalized effectively.
- Method: The algorithm uses MPC to generate guiding samples for guided policy search.
- Method: Modified MPC trades off cost minimization against matching the current neural network policy to produce supervised-learning data.
- Training setup: An instrumented training setup provides full-state observations to MPC while the policy learns from raw onboard sensor readings.
- Limitation: Full-state observations are required during training even though the final policy performs the task using onboard sensors.
- Limitation: Not all aerial maneuvers can be learned in an instrumented training setup.