PaperlayerView exact revision on arXiv

Source-linked AI summary

PLATO: Policy Learning using Adaptive Trajectory Optimization

Gregory Kahn, Tianhao Zhang, Sergey Levine, Pieter Abbeel

arXiv:1603.00622v4cs.LG

TL;DR

Training high-dimensional perception-control policies is difficult, particularly when partially trained controllers may act unsafely. PLATO uses adaptive MPC to supervise policies without running them during training, and the authors report faster learning, fewer catastrophic failures, and often better final policies than prior methods.

  • Problem

    Expressive policies that combine perception and control are difficult to train, while running partially trained policies can produce unsafe actions on physical systems.

    Source: §I · p. 1

  • Method

    PLATO uses supervised learning with an adaptive MPC teacher that gradually matches the learner while preserving task cost, without executing the learner during training.

    Source: §I · pp. 1–2

  • Results

    PLATO learns faster, experiences substantially fewer catastrophic failures, and often converges to a better policy than prior methods in simulated aerial-vehicle tasks.

    Source: §I · p. 2, §VI · p. 7

  • Takeaways & Limitations

    The method combines MPC's training-time robustness with policies that can use raw onboard observations, including in uninstrumented natural environments.

    Source: §VII · p. 8

    Training requires at least a rough model of system dynamics to run MPC.

    Source: §I · p. 1

Abstract

from arXiv · show

Policy search can in principle acquire complex strategies for control of robots and other autonomous systems. When the policy is trained to process raw sensory inputs, such as images and depth maps, it can also acquire a strategy that combines perception and control. However, effectively processing such complex inputs requires an expressive policy class, such as a large neural network. These high-dimensional policies are difficult to train, especially when learning to control safety-critical systems. We propose PLATO, an algorithm that trains complex control policies with supervised learning, using model-predictive control (MPC) to generate the supervision, hence never in need of running a partially trained and potentially unsafe policy. PLATO uses an adaptive training method to modify the behavior of MPC to gradually match the learned policy in order to generate training samples at states that are likely to be visited by the learned policy. PLATO also maintains the MPC cost as an objective to avoid highly undesirable actions that would result from strictly following the learned policy before it has been fully trained. We prove that this type of adaptive MPC expert produces supervision that leads to good long-horizon performance of the resulting policy. We also empirically demonstrate that MPC can still avoid dangerous on-policy actions in unexpected situations during training. Our empirical results on a set of challenging simulated aerial vehicle tasks demonstrate that, compared to prior methods, PLATO learns faster, experiences substantially fewer catastrophic failures (crashes) during training, and often converges to a better policy.

I. INTRODUCTION

Learning policies that combine perception and control requires expressive models, but training them on physical systems is difficult because of sample demands and the risks of partially trained controllers. PLATO uses adaptive MPC to supervise policies without executing them during training, and the paper reports bounded-cost convergence and stronger simulated-quadrotor results than prior approaches.

I. INTRODUCTION

  • High-dimensional policies that process raw sensory inputs are difficult to train on physical systems because model-free learning needs extensive experience and partially trained controllers can act unsafely.These challenges are especially consequential for mobile robots and autonomous vehicles, where unsafe actions can cause damage.Source: §I · p. 1
  • PLATO trains complex perception-control policies with supervised learning, using an adaptive MPC teacher that gradually matches the learner while balancing task success and learner matching.The method is reset-free and uses MPC actions throughout training rather than running the partially trained policy.Source: §I · pp. 1–2
  • PLATO requires at least a rough system-dynamics model for MPC, but does not require an observation model and can train from raw images or depth scans.The resulting policy uses only the observations available to the robot at test time.Source: §I · pp. 1–2
  • The authors prove that PLATO's learned policy converges with bounded cost and report better final-policy performance and fewer catastrophic failures than several prior approaches.The comparisons include MPC-guided policy search, DAgger [10], DAgger with coaching, and supervised learning.Source: §I · p. 2

II. RELATED WORK

Prior neural control methods use reinforcement learning or supervised learning, but each faces practical limitations in sample demands, safety, supervision, or reset requirements. PLATO adapts an MPC teacher to the learner without executing the learner during training and supports raw observations without requiring episodic resets.

II. RELATED WORK

  • Neural-network control methods span reinforcement learning, including Q-learning and policy search,, and supervised learning, including DAgger and guided policy search.Applications cited include Atari games and camera-based robotic manipulation.Source: §II · p. 2
  • Reinforcement learning is generally applicable but requires extensive system experience and potentially dangerous exploration with partially learned policies.These demands limit direct use on real physical and safety-critical systems.Source: §II · p. 2
  • Supervised methods are more sample-efficient but need an expert; guided policy search typically requires environment resets, while DAgger assumes close expert imitation and executes the learner during training.Online learning in non-stationary environments can expose an incompletely trained policy to dangerous situations.Source: §II · p. 2
  • PLATO adapts its MPC expert to the learned policy but does not execute that policy in the real world until training ends.The paper states that this maintains a bound on the difference between learner and expert state distributions while allowing MPC to avoid dangerous learner-like actions in novel situations.Source: §II · p. 2
  • PLATO trains a raw-perception policy with an MPC teacher using true state and removes guided policy search's requirement for episodic resets,.The authors describe continuous autonomous exploration with MPC as a possible training scenario.Source: §II · p. 2

III. PRELIMINARIES AND OVERVIEW

The paper formulates control as minimizing expected cumulative cost when a policy acts from observations in a dynamical system. During training, access to true state enables MPC supervision, while the learned policy needs observations alone at test time.

III. PRELIMINARIES AND OVERVIEW

  • A policy maps observations to a conditional distribution over actions, while the system evolves through state-dependent dynamics and incurs a cost at each step.The objective is to minimize expected total cost over the policy's trajectory.Source: §III · p. 2
  • The training algorithm assumes access to true system states, enabling efficient MPC action generation, but the learned policy requires only observations at test time.The paper gives instrumented training environments or more accurate training hardware as possible ways to obtain state information.Source: §III · p. 2
  • The authors acknowledge that the true-state training assumption restricts the setup, while arguing it enables efficient and relatively safe learning.Source: §III · p. 3
  • MPC serves as the computational teacher, using true state and system dynamics to plan locally optimal trajectories and replan each time step for robustness to perturbations and model errors.The final policy is trained from observations using supervised learning rather than direct reinforcement learning.Source: §III · p. 3

IV. POLICY LEARNING USING ADAPTIVE TRAJECTORY OPTIMIZATION

Training only on an MPC expert's state distribution may leave a learned policy poorly prepared for the states it encounters at test time. PLATO adapts the MPC teacher toward the learner while retaining an objective for task performance and robust responses.

IV. POLICY LEARNING USING ADAPTIVE TRAJECTORY OPTIMIZATION

  • A fixed MPC-generated training set can mismatch the learner's test-time state distribution, undermining expected long-horizon performance.The mismatch arises because the teacher and learned policy visit different states.Source: §IV · p. 3
  • PLATO modifies MPC actions to bring training states closer to the learner's distribution while retaining robust trajectories and responses to unexpected perturbations.The adaptive teacher reoptimizes its controller at each time step.Source: §IV · p. 3
  • The weight λ sets the tradeoff between matching the learner πθ and optimizing expected return J(·) in the MPC objective.Source: §IV · p. 3
  • The described teacher uses a maximum-entropy iterative LQG variant that produces linear-Gaussian stochastic controllers.The paper refers to prior work for the variant's details,,.Source: §IV · p. 3

A. Algorithm Description

PLATO collects trajectories with an adaptive MPC teacher and trains the observation-based learner by supervised learning on those trajectories. The teacher balances long-horizon cost with matching the learner while providing near-optimal supervision at states similar to those the learner would visit.

A. Algorithm Description

  • MPC generates supervision at visited states, while the learner is trained on corresponding observations to match locally optimal policies computed without the KL-divergence term.This pairs states similar to those the learner would visit with supervision from a near-optimal policy; the learner can use raw sensory observations without true-state inputs at test time.Source: A. Algorithm Description · pp. 3–4
  • PLATO chooses actions with MPC rather than simulating the learner, because the observation model is unknown and prevents simulating the learner into the future.The teacher uses system state and requires at least a rough dynamics model, whereas the learned policy is conditioned on observations.Source: A. Algorithm Description · pp. 3–4
  • At each step, PLATO optimizes an MPC teacher against long-horizon cost and immediate disagreement with the learner, producing a sequence of locally optimized policies.The teacher's KL-divergence term is quadratic in the action and linear in controller covariance, with an entropy-maximization term; this yields the maximum-entropy iLQG objective.Source: A. Algorithm Description · p. 4
  • The teacher and learner use conditional Gaussian policies: the MPC mean is linear in state, while the learner mean is a nonlinear function of observation.The learner covariance may be learned or deterministic; in the experiments described, its neural-network mean is trained by regression with stochastic gradient descent, and its covariance is solved in closed form [11].Source: A. Algorithm Description · p. 4, Table I · p. 4

B. Relationship to previous work

PLATO relates to MPC-guided policy search, DAgger, and coaching, but uses adaptive MPC to choose training trajectories. This avoids deterministic reset requirements and avoids executing the learner during training, while the reported evaluation finds better performance than DAgger and coaching.

B. Relationship to previous work

  • Unlike MPC-GPS, PLATO does not require deterministic resets into a small set of initial states, which can be complex, time-consuming, or impossible in real-world settings.The passage also reports that PLATO empirically outperforms MPC-GPS when deterministic resets are feasible.Source: B. Relationship to previous work · p. 4
  • DAgger [10] samples trajectories from a mixture of expert and learner policies, while coaching adapts supervision to make training labels resemble the learner's actions.The passage reports that PLATO outperforms coaching.Source: B. Relationship to previous work · p. 4
  • PLATO uses adaptive MPC rather than a mixture policy to select actions, and the paper reports robust avoidance of catastrophic training failures and better policies than DAgger or coaching.Prior mixture-policy methods execute the learned policy, which may take dangerous actions before it is fully trained.Source: B. Relationship to previous work · p. 4

V. THEORETICAL ANALYSIS

The analysis extends a DAgger-style bound to PLATO's adaptive MPC policy under bounded cost-to-go difference and bounded supervised-learning error. The resulting policy-cost bound depends on horizon, imitation error, and the teacher–learner divergence; its worst-case order matches DAgger's.

V. THEORETICAL ANALYSIS

  • The bound assumes the learner's supervised objective error is bounded on the dataset sampled from the teacher policy distribution.The analysis also bounds the KL divergence between teacher and learner policies by choosing λ, and assumes the cost-to-go difference is at most δ.Source: §V · p. 5
  • Under a per-step cost-to-go difference bounded by δ, PLATO's policy cost is at most J(π*) + δ√ϵθ*O(T) + O(1).The proof extends the result of Ross et al. from mixture policies to PLATO's adaptive MPC policy.Source: §V · p. 5
  • The proof connects learner cost to its per-state 0-1 disagreement with the optimal policy, then bounds that loss under the adaptive teacher's state distribution.The derivation uses a state-distribution bound and an upper bound on 0-1 loss.Source: §V · p. 5
  • In the worst case δ is O(T), so PLATO's bound is O(T), matching DAgger's, but PLATO does not directly execute the learner during training.If the optimal policy can quickly recover from learner mistakes, δ can be O(1); the passage contrasts this with supervised learning's O(T^2) bound.Source: §V · p. 5

VI. EXPERIMENTS

Across simulated quadrotor navigation tasks, PLATO learns effective policies faster and with fewer training crashes than prior methods. Tests also examine command-conditioned policies, raw observations versus full state, and comparisons with MPC-GPS.

Experimental setting

  • Experiments use simulated quadrotor navigation with laser or camera observations, requiring policies to perform perception and control from raw sensory input.Figure 2 datapoints pool 10 random-seed policies, each evaluated with 20 scenario flights.Source: §VI · pp. 5–7, Figure 2 · p. 6

Performance of learned policies

  • PLATO learns policies faster and reaches final mean time to failure (MTTF) that is better than or comparable to baselines.Some DAgger variants reach similar final MTTFs, but always more slowly and with significantly more training crashes.Source: Figure 2 · p. 6, §VI · p. 7

Robustness during training

  • PLATO averages less than one training crash per iteration, comparable to baseline MPC, while DAgger and coaching can fail substantially when β drops.Carefully selecting the β schedule can reduce crashes for prior methods.Source: §VI · p. 7

Non-stationary environments

  • In the switching canyon/forest scenario, PLATO averages less than one crash per episode, while prior methods experience many crashes.PLATO automatically switches toward more off-policy behavior in novel scenarios.Source: Figure 2 · p. 6, §VI · p. 7

Policies with user velocity commands

  • With varying user velocity commands, PLATO learns a policy that outperforms prior methods while minimizing training crashes.Training commands vary across specified sideways and forward velocity ranges; test commands are sampled uniformly at random.Source: §VI · p. 7

Comparison with training on full state

  • The observation-based policy substantially outperforms a policy mapping full state to action, even when the latter uses oracle SLAM.The comparison suggests that mapping observations directly to actions can outperform going through full state estimation in some cases.Source: §VI · pp. 7–8, Figure 4 · p. 7

Comparison with MPC-GPS

  • On a fixed-length episodic forest task, PLATO substantially outperforms MPC-GPS in mean time to failure (MTTF).MPC-GPS requires deterministic resets during training, unlike PLATO.Source: Figure 4 · p. 7, §VI · p. 8

VII. DISCUSSION

PLATO uses an adaptive trajectory-optimization teacher to train complex perception-and-control policies without executing the learned policy during training. Simulated quadrotor results show better final-policy effectiveness and robustness, alongside safer training than prior methods.

PLATO

  • PLATO adapts its trajectory-optimization teacher to the neural policy, keeping training state distributions close enough to support a bound on long-term performance.The method is continuous and reset-free.Source: §VII · p. 8

Training robustness

  • Because the learned policy is not executed during training, PLATO can retain MPC robustness and automatically shift toward off-policy behavior in novel observations.This is particularly relevant when the training state distribution is non-stationary; methods executing the learned policy can suffer catastrophic failure in novel observations.Source: §VII · p. 8

Observation requirements

  • The learned policy can use raw onboard sensor inputs different from MPC's full-state observations, enabling use in uninstrumented natural environments after training.MPC may rely on instrumentation such as motion capture or a known map with relocalization.Source: §VII · p. 8

Future work

  • Applying PLATO on real physical platforms is identified as an avenue for future work.The discussion frames real-world policies using rich sensory inputs as an appealing prospect.Source: §VII · p. 8
Loading 1603.00622v4…