PaperlayerView exact revision on arXiv

Source-linked AI summary

Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework

Wanxin Jin, Zhaoran Wang, Zhuoran Yang, Shaoshuai Mou

arXiv:1912.12970v5cs.LGcs.ROcs.SYeess.SYmath.OC

TL;DR

Learning and control methods offer complementary benefits, but the paper seeks a unified framework for a broad range of learning and control tasks. PDP differentiates through PMP and uses an auxiliary control system to compute trajectory derivatives for end-to-end learning. The authors report higher efficiency and capability across tasks, and 1000x faster running time than differentiable MPC.

  • Problem

    Learning and control offer complementary benefits, but a unified framework for solving a broad range of learning and control tasks is sought.

    Source: §1 · p. 1

  • Method

    PDP differentiates through PMP and uses an auxiliary control system in the backward pass to compute analytical trajectory derivatives for learning system parameters.

    Source: §5.2 · p. 6, §7 · p. 9

  • Results

    PDP is reported to achieve higher efficiency and capability across many learning and control tasks, and to run 1000x faster than differentiable MPC.

    Source: §7 · p. 9, §8 · p. 9

  • Takeaways & Limitations

    The authors present PDP as a framework for solving high-dimensional, continuous-space learning and control problems.

    Source: §8 · p. 9

    PDP is a first-order framework for non-convex bi-level optimization and can generally achieve only local minima.

    Source: §7 · p. 9

Abstract

from arXiv · show

This paper develops a Pontryagin Differentiable Programming (PDP) methodology, which establishes a unified framework to solve a broad class of learning and control tasks. The PDP distinguishes from existing methods by two novel techniques: first, we differentiate through Pontryagin's Maximum Principle, and this allows to obtain the analytical derivative of a trajectory with respect to tunable parameters within an optimal control system, enabling end-to-end learning of dynamics, policies, or/and control objective functions; and second, we propose an auxiliary control system in the backward pass of the PDP framework, and the output of this auxiliary control system is the analytical derivative of the original system's trajectory with respect to the parameters, which can be iteratively solved using standard control tools. We investigate three learning modes of the PDP: inverse reinforcement learning, system identification, and control/planning. We demonstrate the capability of the PDP in each learning mode on different high-dimensional systems, including multi-link robot arm, 6-DoF maneuvering quadrotor, and 6-DoF rocket powered landing.

1 Introduction

The introduction frames learning and control as complementary ways to obtain unknown aspects of decision-making systems. PDP aims to combine control structure with learning from data across a range of tasks.

1 Introduction

  • PDP integrates optimal-control theory as an inductive bias into end-to-end learning for tasks spanning inverse reinforcement learning, system identification, and control or planning.The paper presents rocket landing control and quadrotor dynamics and control-objective learning as examples.Source: §1 · p. 1, Figure 1 · p. 1

2 Background and Related Work

The background relates learning tasks to control problems, while identifying limitations in existing approaches. PDP’s contribution is to differentiate through PMP and use an auxiliary control system to obtain trajectory derivatives for learning.

Learning dynamics

  • For learning dynamics, existing approaches trade off data efficiency against long-term prediction accuracy, while physically informed learning is limited to mechanical systems.The paper surveys approaches including regression, neural networks, and Gaussian processes.Source: §2 · p. 2

Learning optimal policies

  • Model-free RL can have significant data complexity, while model-based RL and optimal control face challenges involving imperfect models and deployment robustness.PMP methods optimize trajectories but are open loop; MPC repeatedly solves finite-horizon problems to produce closed-loop controls.Source: §2 · p. 2
  • PDP complements open-loop optimal-control methods by learning feedback policies, and its policy parameterization can also support motion planning.The paper presents this mode as a perspective for model-based RL or MPC.Source: §2 · p. 2

Learning control objective functions

  • Existing IRL methods commonly learn feature weights in a selected feature space, while optimality-violation methods do not assure reproduced-trajectory or objective-value closeness.PDP instead supports learning complex, including neural, objective functions by minimizing loss between reproduced trajectories and demonstrations.Source: §2 · pp. 2–3

Claim of contributions

  • PDP differentiates through PMP to compute analytical trajectory derivatives with respect to tunable parameters, enabling end-to-end learning of dynamics, policies, or control objectives.An auxiliary control system in the backward pass produces these derivatives and can be iteratively solved using standard control tools.Source: §2 · p. 3

3 Problem Formulation

The formulation treats learning and control as parameter tuning in an optimal-control system, evaluated through a differentiable loss. Different tasks specialize the system and loss to the data or objective at hand.

Base problem

  • The base problem parameterizes system dynamics and control objectives, then tunes parameters to minimize a differentiable loss evaluated on the resulting trajectory.The formulation assumes twice-differentiable dynamics and cost functions.Source: §3 · p. 3

Inverse reinforcement learning

  • In IRL, the expert system is represented by the parameterized model, and the loss penalizes mismatch between its reproduced trajectory and demonstrations.Even when demonstrations are not optimal, the formulation selects the parameterized system with the smallest reproduced-trajectory distance to them.Source: §3 · p. 4

System identification

  • For system identification, externally supplied inputs drive a parameterized dynamics model, and the loss measures prediction error against observed trajectories under the same inputs.The objective function is set to zero so the system represents dynamics alone.Source: §3 · p. 4

Control/Planning Mode

  • In control or planning, learned dynamics are paired with a parameterized policy and a given cost; state-dependent policies yield feedback control, while time-dependent policies yield open-loop planning.This mode can also serve as a component in the IRL/IOC formulation.Source: Control/Planning Mode · p. 4

4 An End-to-End Learning Framework

The framework updates parameters by differentiating a trajectory-based loss. It obtains the trajectory in a forward pass and requires the trajectory’s parameter derivative for the backward pass.

End-to-End Learning Framework

  • End-to-end learning updates tunable parameters by gradient descent on the loss evaluated over the system trajectory.Each iteration solves the system at the current parameters, computes the loss, and updates the parameters.Source: §4 · p. 4

Forward and backward passes

  • The forward pass solves the optimal-control problem to obtain a trajectory, while the loss gradient with respect to that trajectory is obtained from the loss function.The forward solution may use available optimal-control methods; in system-identification and control/planning modes, it reduces to integrating difference equations.Source: §4 · p. 4
  • The central backward-pass challenge is computing the trajectory’s derivative with respect to the system parameters.This derivative is needed alongside the loss gradient to update parameters.Source: §4 · p. 4

5 Key Contributions: Differential PMP & Auxiliary Control System

The section derives a differential PMP by differentiating the original optimality conditions with respect to system parameters, then constructs an auxiliary control system to solve for trajectory sensitivities.

5.1 Differential PMP

  • Differentiating the original PMP conditions with respect to θ yields a differential PMP whose unknowns are state and control trajectory sensitivities.The forward-pass trajectory and costates provide the quantities needed to form its coefficient matrices.Source: §5.1 · pp. 5–6

5.2 Auxiliary Control System

  • The auxiliary system’s PMP conditions match the differential PMP, and its resulting state and control trajectory are the original trajectory’s parameter derivatives.Its state and control variables are matrix-valued, with linear dynamics and a quadratic objective determined by the forward-pass trajectory.Source: §5.2 · p. 6
  • The auxiliary system can be solved with a standard finite-time LQR procedure: solve for matrices backward, then integrate the feedback-control system forward.This procedure forms the backward pass used to compute the trajectory derivative in PDP.Source: §5.2 · pp. 6–7

6 Applications to Different Learning Modes and Experiments

Across imitation learning, system identification, and control/planning, the experiments compare PDP with baseline or state-of-the-art methods. PDP reports lower losses or faster convergence in several comparisons, while its planning result depends on policy expressiveness and trades convergence speed against runtime.

Experiment scope

  • The experiments cover three learning modes across four environments; results for 6-DoF rocket landing are reported in Appendix I.The paper compares PDP with a baseline and a state-of-the-art method in each mode.Source: §6 · p. 7, Table 2 · p. 7

Experiment: imitation learning

  • PDP achieves lower training loss and faster convergence than policy cloning and inverse-KKT, and lower imitation loss than GAIL when learning a neural objective function.With demonstrations not strictly realizable under the parameterized objective, PDP’s final loss is small but nonzero; it finds the closest reproduced trajectory within that function set.Source: §6 · p. 7, Figure 3 · p. 7

Experiment: system identification

  • PDP has lower training loss and faster convergence than the NN baseline and DMDc, and also outperforms Adam when learning the same robot-arm neural dynamics model.The authors attribute these advantages to injecting an optimal-control inductive bias into learning.Source: §6 · p. 8, Figure 4 · p. 8

Experiment: control and planning

  • PDP outperforms GPS on feedback-policy control loss and achieves competitive quadrotor planning performance against iLQR, with lower final loss for degree-35 than degree-5 polynomials.iLQR converges faster, while PDP has a large per-iteration runtime advantage; PDP’s planning loss depends on polynomial expressiveness.Source: §6 · p. 8, Figure 5 · p. 8

7 Discussion

PDP differentiates through the Pontryagin Maximum Principle instead of relying on an unrolled optimization graph or intermediate LQR approximations. Its runtime comparison favors PDP, while its general non-convex optimization formulation has a local-minimum limitation.

7 Discussion

  • Unlike unrolling, PDP stores only the resulting trajectory and differentiates through PMP without intermediate LQR approximations.The paper contrasts this with unrolled graphs, which require storing intermediate results and whose gradient accuracy depends on graph length.Source: §7 · p. 9
  • PDP is reported to be 1000x faster than differentiable MPC in the per-iteration runtime comparison for varying pendulum horizons.Figure 6 compares the two methods across horizons.Source: Figure 6 · p. 9, §7 · p. 9
  • PDP is a first-order gradient-descent framework for non-convex bi-level optimization and can generally achieve only local minima.The authors identify convexity and smoothness assumptions as sufficient for global convergence, but consider them too restrictive for dynamical control systems.Source: §7 · p. 9

8 Conclusions

The paper proposes Pontryagin differentiable programming as an end-to-end framework for learning and control. It incorporates optimal control theory as an inductive bias and targets inverse reinforcement learning, system identification, and control/planning.

8 Conclusions

  • Pontryagin differentiable programming provides an end-to-end learning framework for a range of learning and control tasks.The paper describes it as incorporating optimal control theory as an inductive bias.Source: §8 · p. 9
  • The framework addresses inverse reinforcement learning, system identification, and control/planning.Source: §8 · p. 9
  • The paper reports higher efficiency and capability than existing learning and control methods on many tasks.The stated scope includes high-dimensional continuous-space problems.Source: §8 · p. 9

Broader Impact

The paper presents PDP as integrating learning and control concepts and as a way to formulate control tasks using learning techniques. It also identifies safety during physical-system training as an unresolved concern when system bounds are not considered.

Broader Impact

  • PDP unifies concepts from reinforcement learning, backpropagation/deep learning, and control theory in one learning framework.The authors describe the framework as integrating optimal control into end-to-end learning.Source: Broader Impact · p. 10
  • The framework formulates challenging control tasks as learning problems solvable with techniques such as neural networks and backpropagation.The paper cites nonlinear system identification and optimal control as areas where it shows significant advantage over state-of-the-art control methods.Source: Broader Impact · p. 10
  • Because the formulation omits system bounds or constraints, physical-system use may raise safety issues during training.The authors suggest adding safety bounds as soft constraints and identify formal treatment within a safety framework as future work.Source: Broader Impact · p. 10

A Proof of Lemma 5.1

The supplied supplementary material outlines proofs for the auxiliary-system PMP and the discrete-time PMP, gives implementation and experiment details, and describes validation procedures for imitation learning and system identification.

A Proof of Lemma 5.1

  • The proof of Lemma 5.1 identifies the auxiliary control system’s PMP with the differential PMP equations.The derivation defines an auxiliary-system Hamiltonian and concludes that the resulting PMP equations match the differential PMP.Source: A Proof of Lemma 5.1 · p. 15

B Proof of Lemma 5.2

  • The proof of Lemma 5.2 derives the auxiliary-system equations by substituting the control solution and using an induction argument for matrices P_t and W_t.The terminal condition initializes the induction, which then establishes the recursions used in the derivation.Source: B Proof of Lemma 5.2 · pp. 15–16

C Proof of the Discrete-Time Pontryagin’s Maximum Principle

  • The discrete-time PMP proof treats the optimal-control problem as constrained optimization and derives its conditions from KKT conditions.The dynamics multipliers are identified as costates, and the resulting first-order conditions are stated to be the PMP equations.Source: C Proof of the Discrete-Time… · p. 16

D Algorithms Details for Different Learning Modes

  • The learning-mode algorithms compute auxiliary-system coefficient matrices and iteratively integrate the resulting systems; the modes can also be combined in one task.For example, the paper proposes learning dynamics in SysID Mode before using them in Control/Planning Mode.Source: D Algorithms Details for Different… · pp. 17–18

E Experiment Details

  • The authors released the PDP code and simulation environments as standalone packages and made experiment video demonstrations available.Source: E Experiment Details · p. 18

E.1 System/Environment Setup

  • Experimental continuous-time dynamics are discretized with Euler steps of 0.05s or 0.1s.The quadrotor setup specifies full position and attitude on SE(3), with state and input vectors defined in the supplied passages.Source: §E.1 · pp. 18–19

E.2 Experiment of Imitation Learning

  • Imitation-learning validation plans with learned models from new initial states and horizons, then compares the resulting trajectories with the expert’s.The reported detailed loss experiments use five trials per system with random initial guesses.Source: §E.2 · pp. 19–20

E.3 Experiment of System Identification

  • System-identification validation tests learned dynamics by predicting motion in unactuated conditions, with u_t = 0.Detailed SysID-loss results cover cart-pole, robot arm, and quadrotor, with five trials per system.Source: §E.3 · p. 20

E.4 Experiment of Control/Planning

In control/planning experiments, PDP lowers feedback-policy control loss than GPS and produces quadrotor plans near optimal, though iLQR converges faster. The framework differentiates through PMP and uses an auxiliary control system, with linear time-horizon complexity and a reported 1000x runtime advantage over differentiable MPC.

E.4 Experiment of Control/Planning

  • The control/planning experiments use identified dynamics, a weighted goal-distance objective, and time horizons T from 20 to 40.Cart-pole and robot-arm feedback policies use neural networks; quadrotor planning uses Lagrange polynomial policies of different degrees.Source: §E.4 · pp. 20–21
  • PDP lowers feedback-policy control loss versus GPS and yields quadrotor plans near optimal; higher polynomial degree improves accuracy, while iLQR converges faster.iLQR's high-accuracy solutions come at higher computation expense.Source: §E.4 · p. 21, Figure S3 · p. 24, Figure S6 · p. 25

F Related End-to-End Learning Frameworks

  • Unlike differentiable MPC's LQR approximation, PDP differentiates directly through PMP and solves trajectory gradients using an auxiliary control system.The auxiliary system's trajectory is the gradient of the forward system trajectory with respect to parameters.Source: F Related End-to-End Learning Frameworks · p. 26
  • PDP runs 1000x faster than differentiable MPC per iteration across pendulum time horizons.The comparison averages 100 CPU iterations per method, implemented in Python on the same machine.Source: Figure S7 · p. 26
  • PDP stores the final optimal-control solution rather than unrolling solver steps, accepts external solvers, and has memory and computation complexity linear in time horizon T.Unrolling methods store and traverse intermediate gradient-descent results, incurring memory and computation costs.Source: F Related End-to-End Learning Frameworks · pp. 26–27, G Complexity of PDP · p. 27

H Limitation of PDP

The rocket experiments apply PDP to imitation learning, system identification, and powered-landing control, with reported validation on unseen conditions and against ground truth. The paper also identifies first-order optimization and parameterization-sensitive local convergence as limitations.

H Limitation of PDP

  • PDP uses exact analytical gradients without second-order derivatives, but its first-order nature can make planning converge more slowly than higher-order methods such as iLQR.The slower convergence is reported empirically for PDP planning mode.Source: H Limitation of PDP · p. 28
  • For general non-convex problems, PDP can achieve only local minima; global convergence results require assumptions such as model convexity and smoothness.The paper describes those assumptions as too restrictive for dynamical control systems.Source: H Limitation of PDP · p. 28
  • Convergence depends on parameterization: more complex models can increase local-minimum risk, and standard polynomials can be poorly conditioned.The paper contrasts neural objective and policy parameterizations with simpler alternatives, and uses Lagrange rather than standard polynomials for planning.Source: H Limitation of PDP · p. 28

I PDP to Solve 6-DoF Rocket Powered Landing Problems

Related papers

Loading 1912.12970v5…