Guided Policy Search as Approximate Mirror Descent

1 Introduction

• Prior guided policy search guarantees typically hold only at convergence when the constraint is enforced exactly, although practical applications often use few iterations.This motivates analyzing improvement before convergence.Source: §1 · p. 1
• The new method interprets guided policy search as approximate mirror descent, with supervised learning acting as projection onto the policy constraint manifold.Under linear dynamics and convex policy spaces, the proposed variant corresponds exactly to mirror descent; outside those assumptions, projection error can be bounded in terms of step size.Source: §1 · p. 1
• The method alternates trajectory-centric reinforcement learning with supervised learning to train complex, high-dimensional policies.This simplifies the guided policy search formulation and uses fewer hyperparameters than prior methods.Source: §1 · pp. 1–2
• On simulated robotic navigation and manipulation tasks, the method is stable and achieves similar or better performance than prior guided policy search methods.The paper also reports a simpler formulation and fewer hyperparameters.Source: §1 · p. 2

2 Guided Policy Search Algorithms

• Guided policy search uses local policies for different initial states, then trains a global policy to match them with supervised learning; local optimization can use trajectory optimization or local reinforcement learning.The approach can train policies with nearly a hundred thousand parameters for vision-based control using under 200 episodes.Source: §2 · p. 2

2.1 Efficiently Optimizing Local Policies

• Time-varying linear-Gaussian local controllers support efficient model-based optimization, while a KL-divergence constraint limits policy changes where fitted dynamics are locally valid.The local dynamics are fitted from real-system samples, then used to improve the controller with a modified LQR algorithm.Source: §2.1 · p. 3

2.2 Prior Convergence Results

• Prior methods enforce agreement between local and global policies at convergence, but do not characterize global-policy performance at intermediate iterations.Practical constraints may preclude running the method to full convergence.Source: §2 · p. 2, §2.2 · p. 3

3 Mirror Descent Guided Policy Search

• With linear dynamics, quadratic cost, and convex supervised learning, MDGPS matches mirror descent and inherits monotonic improvement of the global policy.The C-step improves a trajectory distribution; the S-step projects it onto the parameterized policy set.Source: §3 · p. 4

3.1 Implementation for Nonlinear Global Policies and Unknown Dynamics

• For nonlinear systems with unknown dynamics, the C-step uses local quadratic costs and fitted linear-Gaussian dynamics, while the global policy is locally linearized around each local policy.The policy linearization can be obtained by differentiation or regression; the implementation uses regression and Gaussian mixture model priors for dynamics and policy fits.Source: §3.1 · p. 4
• The nonlinear S-step minimizes reverse policy KL on sampled states; unlike exact trajectory KL projection, this is approximate, and the authors describe its weighting as more informative.Samples for fitting the dynamics and policy can come from either the latest local policy or the latest global policy.Source: §3.1 · pp. 4–5
• The nonlinear S-step minimizes local policy KL rather than trajectory-distribution KL, so MDGPS no longer corresponds exactly to mirror descent in that setting.The C-step can still be evaluated exactly; the projection error is discussed in the following section.Source: §3.1 · p. 5

3.2 Analysis of Prior Guided Policy Search Methods as Approximate Mirror Descent

• Compared with prior methods’ dual-descent constraint adjustment, MDGPS enforces the constraint at each iteration, taking bounded trajectory-space steps; it is somewhat more stable but can be slower.The authors characterize MDGPS as simpler and requiring less tuning, while prior methods can sometimes be faster because their step sizes are not constrained exactly.Source: §3.2 · p. 5

4 Analysis in the Nonlinear Case

• The nonlinear S-step is not an optimal projection, but its additional cost can be bounded in terms of the KL divergence between local and global policies.The analysis also explains why prior methods with asymptotic guarantees can perform well after only a few iterations.Source: §4 · p. 5

4.1 Bounding the Global Policy Cost

• Reducing local-policy cost or its KL divergence from the global policy can eventually reduce global-policy cost; choosing a sufficiently small constraint can close the gap.Directly optimizing the bound is too slow in practice because it is loose; it can instead guide step-size reductions near the end or when global performance lags.Source: §4.1 · p. 6

4.2 Step Size Selection

• The cost-change model treats additional cost as quadratic in step size from inaccurate dynamics and linear in step size from KL divergence.The coefficients are used to solve for a new optimal step size.Source: §4.2 · p. 6
• The classic rule adapts the earlier step-size method by measuring local-policy cost against the previous global policy rather than the previous local policy.It uses the expected cost under the previous global policy, estimated using its linearization.Source: §4.2 · p. 6
• The global step size uses a linearized estimate of global-policy cost and reduces the step when that policy fails to reproduce local-policy performance.This discrepancy can reflect either overly local dynamics or an unsuccessful supervised-learning step.Source: §4.2 · pp. 6–7

• Unlike DAGGER, MDGPS adapts the teacher to the global policy’s limitations instead of assuming the learner can imitate an arbitrary teacher with bounded error.This matters when the global learner has computational or observational limits, including partial observability and blind peg insertion.Source: §5 · p. 7
• Policy-gradient methods with KL constraints linearize complex policies in parameter space, whereas MDGPS linearizes time-varying dynamics and trains the policy through supervised learning.The passage presents this as better suited to quickly training highly nonlinear, high-dimensional policies.Source: §5 · p. 7

• The evaluation compares methods on simulated obstacle navigation, peg insertion, and blind peg insertion tasks.The blind variant withholds target-relative end-effector positions from the global policy, requiring it to search for the hole; task results are plotted in Figure 1.Source: §6 · p. 7, Figure 1 · p. 8
• MDGPS variants achieve similar performance to BADMM on the easier point-mass and peg tasks, while requiring far fewer hyperparameters.BADMM’s reported results use the best setting found through a hyperparameter sweep; MDGPS’s adaptive step-size rules make its initial step-size choice less important.Source: §6 · p. 8
• On blind peg insertion, MDGPS with off-policy sampling consistently outperforms BADMM for both classic and global step sizes.The success-rate comparison reports that MDGPS inserts the peg more often and succeeds across more conditions.Source: §6 · p. 8, Table 1 · p. 8

7 Discussion and Future Work

• MDGPS corresponds to mirror descent under linearity and convexity assumptions, while prior guided policy search methods approximate mirror descent.The paper also reports a nonlinear global-policy return bound and empirical results on simulated robotic manipulation with less tuning than prior guided policy search methods.Source: §7 · p. 8
• The paper proposes exploring trajectory-space KL-constrained policy-gradient steps followed by projection as a route toward new model-free policy-gradient methods.This connection is described for sampling directly from the global policy.Source: §7 · p. 9

A KL Divergence Between Gaussian Trajectory Distributions

• The appendix derives KL divergence between Gaussian trajectory distributions induced by time-varying linear-Gaussian dynamics and policies.Its derivation uses cancellation of the shared dynamics and initial-state distribution, linearity of expectations, and properties of conditional Gaussian entropy.Source: A KL Divergence Between Gaussian… · p. 10

B Details of the MDGPS Algorithm

• Each iteration gathers samples, fits local dynamics and a linearized global policy, updates local policies in a constrained C-step, trains the global policy in an S-step, and adjusts ϵ.The C-step bounds trajectory-distribution KL divergence from the linearized global policy; the S-step minimizes local-to-global policy KL through supervised learning.Source: B Details of the MDGPS Algorithm · p. 11

B.1 C-Step Details

• The C-step fits local dynamics from sampled trajectories, uses LQR with a quadratic cost approximation to optimize the local policy, and adjusts its dual variable by bracketing line search.Dynamics are fit by linear regression with a Gaussian mixture prior incorporating samples from other time steps and previous iterations; the dual function is convex.Source: §B.1 · p. 11

B.2 S-Step Details

• The S-step minimizes KL divergence between the global and local policies; with state-independent global covariance, the covariance is solved in closed form and the mean is optimized by stochastic gradient descent.Both policies are assumed conditionally Gaussian; state-dependent covariance could instead be optimized by stochastic gradient descent.Source: §B.2 · p. 12

B.3 Step Size Adjustment

• Step-size adjustment uses expected costs under local and linearized global policies, which can be computed analytically by propagating Gaussian state-action marginals and integrating the quadratic cost.The fitted dynamics and policies are linear-Gaussian, enabling the analytic calculations.Source: §B.3 · p. 12

C Global Policy Cost Bounds

• The policy-cost bound combines results from Ross et al. and Schulman et al. and extends them to finite-horizon episodic tasks.The appendix presents this derivation as support for the result discussed in Section 4.1.Source: C Global Policy Cost Bounds · p. 12

C.1 Policy State Distribution Bound

• The proof uses stepwise action-agreement probabilities to bound the difference between the policies’ state distributions.It considers the probability that all actions through time t would have matched, then uses that to bound state-distribution divergence.Source: §C.1 · pp. 12–13
• Lemma C.2 links a per-step total-variation bound between policies to their probability of taking the same action: at most √2ϵt divergence gives agreement probability 1 − √2ϵt.The bound assumes maxxt ∥p(ut|xt) − q(ut|xt)∥1 ≤ √2ϵt.Source: §C.1 · p. 13
• The state-distribution argument can be extended from discrete distributions to continuous densities by taking the limit of an infinitely fine discretization.Source: §C.1 · p. 13

C.2 Total Policy Cost Bound

• The proof of Lemma 4.2 builds on Ross et al. and extends the result to time-varying finite-horizon systems.The appendix restates the lemma using the notation of the preceding appendix.Source: §C.2 · p. 13
• The proof defines Qmax,t as the maximum total cost from time t through the end of the horizon T.Source: §C.2 · p. 13
• The per-step cost under q is decomposed into state-distribution and action-policy differences, plus the expected cost under p.The proof sums this quantity over time and then rewrites the result.Source: §C.2 · p. 14