Source-linked AI summary
Fast Policy Learning through Imitation and Reinforcement
Ching-An Cheng, Xinyan Yan, Nolan Wagener, Byron Boots
TL;DR
IL can yield policies inferior to RL when its expert is suboptimal, while RL can require many interactions. The paper formulates RL and IL in a common mirror-descent framework and proposes LOKI, which randomizes an IL-to-policy-gradient switch; with proper randomization, it can outperform a suboptimal expert and improve faster than common RL methods.
Problem
IL depends strongly on expert quality and can yield inferior policies to direct policy-gradient RL when the available expert is suboptimal.
Source: §1 · p. 1Method
LOKI first performs a randomly chosen number of online IL steps, then switches to policy-gradient updates, within a paper-wide mirror-descent account of RL and IL.
Source: §1 · p. 1Results
With K properly randomized, LOKI can significantly outperform a suboptimal expert and improve faster than common RL methods.
Source: §1 · p. 2Takeaways & Limitations
The paper reports that LOKI is straightforward to implement and has stronger theoretical guarantees than prior combined RL/IL methods.
Source: §1 · p. 1Online imitation gradients generally cannot improve beyond a one-step policy improvement over the expert, motivating a reinforcement phase when the expert is suboptimal.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.
1 INTRODUCTION
The introduction motivates combining reinforcement learning (RL) and imitation learning (IL): RL can require many interactions, while IL can be limited by expert quality. It presents LOKI, which randomizes the number of online IL steps before policy-gradient updates, and reports theoretical and experimental benefits.
1 INTRODUCTION
- RL can demand enormous numbers of interactions, making learning costly and slow in real-world problems such as robotics.The introduction cites prior combined RL/IL approaches but notes limited theoretical understanding and practical requirements such as state resetting.Source: §1 · p. 1
- IL can yield inferior policies to direct policy-gradient RL when the available expert is suboptimal.IL leverages expert demonstrations, but its performance depends strongly on expert quality.Source: §1 · p. 1
- LOKI performs a random number of online IL steps before switching to policy-gradient updates.The introduction describes it as straightforward to implement compared with prior methods combining RL and IL.Source: §1 · p. 1
- With K properly randomized, LOKI can outperform a suboptimal expert and improve faster than common RL methods.The paper also reports that its simulated-environment results corroborate the theoretical findings.Source: §1 · p. 2
2 PROBLEM DEFINITION
The paper defines a discounted infinite-horizon reinforcement-learning objective over policies and a trajectory distribution. It also introduces a surrogate objective using an advantage function with respect to a reference policy.
2 PROBLEM DEFINITION
- The objective J(π) is the expected accumulated discounted cost over trajectories generated by policy π.The trajectory distribution starts from the initial-state distribution and includes states and actions over time.Source: §2 · p. 2
- The paper defines state-value and Q-functions, then uses a discounted state-time distribution dπ(s,t) for expectations.The policy class may contain deterministic or stochastic policies.Source: §2 · p. 2
- Instead of optimizing J(π) directly, the paper uses a surrogate objective involving the advantage relative to a fixed reference policy.The paper states that solving the surrogate is equivalent to solving the original objective.Source: §2 · p. 2
3 FIRST-ORDER RL AND IL
The section places first-order RL and IL methods in a shared mirror-descent framework, distinguishing them mainly by their first-order oracles. It describes the policy-gradient and imitation-gradient approaches within that framework.
3 FIRST-ORDER RL AND IL
- First-order RL and IL methods fit a common mirror-descent framework and mainly differ in the first-order stochastic oracle that supplies their update direction.The section summarizes the comparison of the two oracle types in Table 1.Source: §3 · p. 2, Table 1 · p. 3
3.1 MIRROR DESCENT
- Mirror descent updates a parameterized policy using an oracle vector, a chosen strictly convex regularizer, and a prox-map step.The framework includes choices such as negative entropy, squared Euclidean norm, and a Fisher-information-based regularizer.Source: §3.1 · p. 2
3.2 FIRST-ORDER ORACLES
- The RL and IL oracle vectors can be viewed as stochastic approximations to derivatives of the same surrogate objective, but with different reference policies.The section compares the first-order oracles in Table 1.Source: §3.2 · p. 2, Table 1 · p. 3
3.2.1 Policy Gradients
- Policy-gradient RL uses an estimate of ∇θJ(π) as its oracle, with the current policy as the reference policy.The derivation treats the current policy as fixed when computing the gradient; practical function approximation can produce biased gradient estimators.Source: §3.2.1 · p. 3
3.2.2 Imitation Gradients
- Online IL alternates data collection and policy updates using expert demonstrations, with its mirror-descent oracle derived from a surrogate loss.DAGGERED uses a policy-matching loss, while AGGREVATED uses the expert-relative advantage; online IL minimizes an upper bound on the objective.Source: §3.2.2 · pp. 3–4
4 THEORETICAL COMPARISON
Under the shared mirror descent update, policy-gradient and imitation-gradient methods differ in their behavior: policy gradients can improve monotonically, while imitation gradients improve on average. The comparison characterizes the conditions and limitations of each approach.
4 THEORETICAL COMPARISON
- Policy-gradient and imitation-gradient mirror descent share an update rule but produce, respectively, monotonically improving and on-average-improving policy sequences.The comparison specializes existing optimization techniques to contrast their performance and motivate LOKI.Source: §4 · p. 4
4.1 POLICY GRADIENTS
- Monotonic policy-gradient improvement requires proper smoothness assumptions, a small step size, and sampling noise comparably small to the gradient; performance remains sensitive to the initial policy.A randomly initialized policy may have poor initial performance.Source: §4.1 · p. 4
- The policy-gradient surrogate gradient norm generally converges only to a size proportional to sampling noise, though quadratic regularization with unconstrained parameters can guarantee average convergence to zero.The general result reflects the nonlinear prox-map for general regularizers and parameter sets.Source: §4.1 · p. 4
4.2 IMITATION GRADIENTS
- Under strong convexity, imitation-gradient performance can converge on average close to the expert's, with additional error proportional to ϵclass and ϵregret.ϵregret is less than ˜O(1/N) for a sufficiently large step size, which contrasts with policy gradients' requirement of small enough steps for local improvement.Source: §4.2 · p. 4
- Imitation-gradient updates need not improve monotonically, and online imitation with a suboptimal expert can yield suboptimal policies rather than exceed one-step improvement.Whether performance improves monotonically is problem-dependent, even for deterministic problems; the imitation direction is not necessarily a descent direction, even when using the expert's advantage.Source: §4.2 · p. 5
5 IMITATE-THEN-REINFORCE
LOKI randomly chooses how long to imitate before switching to policy-gradient updates. Its analysis says this can approximate starting policy gradient from the expert while retaining the possibility of outperforming that expert.
5.1 ALGORITHM: LOKI
- LOKI randomly selects K, takes K mirror-descent steps with imitation gradients, then uses policy gradients for the remaining steps.The switching distribution is controlled by d, Nm, and NM.Source: §5 · p. 5, §5.1 · p. 5
- The imitation phase moves the policy toward the expert but is not itself guaranteed to outperform it; the later reinforcement phase can use an available advantage estimate.The imitation gradient can use KL divergence without a value-function estimator, and can reduce variance through multiple learner-action samples per expert query.Source: §5.1 · p. 5
5.2 ANALYSIS
- The expected performance gap after K imitation steps converges at rate Õ(d/NM) when NM ≫ d, with additional terms for policy-class expressiveness and the mirror-descent decision-space measure.The decision-space term’s influence can be reduced by increasing d, while the analysis favors larger imitation-phase step sizes.Source: §5.2 · p. 6
- With suitable randomization, LOKI’s final policy can outperform the expert and converge on average to a locally optimal policy under Proposition 1’s assumptions.Compared with starting policy gradients directly from the expert, the stated additional gap is O(∆ + K∥∇̂θJ(π)∥²).Source: §5.2 · p. 6
- Against policy gradients from random initialization, LOKI is favorable when K such updates would remain worse than the expert’s performance plus ∆.The stated performance difference is O(J(π*) − J(π0) + ∆ + K∥∇̂θJ(π)∥²).Source: §5.2 · p. 6
6 RELATED WORK
The section contrasts LOKI’s randomized imitation-to-policy-gradient approach with methods that rely on expert value-function estimates. It also introduces SLOLS, a linear-combination oracle with LOLS’s performance guarantee.
6 RELATED WORK
- AGGREVATE(D) can potentially improve on the expert using Aπ*, but off-policy estimation may suffer covariate shift and on-policy estimation requires state resets or about 1/(1−γ) times more iterations.The section says these requirements can be impractical for typical RL problems.Source: §6 · p. 6
- THOR uses an H-step truncated advantage with the expert value function as terminal loss, but its theoretical properties remain unclear and it relies on multiple advantage estimators.The section contrasts this with LOKI’s stronger guarantees and straightforward implementation.Source: §6 · pp. 6–7
- LOLS uses a mixed current-and-expert policy to estimate Q, but that estimate relies on the state-resetting assumption.Its mixture runs the current policy with probability 1−λ and the expert with probability λ at each step.Source: §6 · p. 7
- SLOLS uses a convex combination of policy-gradient and AGGREVATE gradients as its first-order oracle and has the same performance guarantee as LOLS.The section also states that LOLS or SLOLS can perform on average better than the expert under the stated analysis.Source: §6 · p. 7
7 EXPERIMENTS
The experiments compare LOKI with RL, IL, and combined-method baselines on simulated robotic-control tasks. LOKI starts with fast imitation-like improvement and continues improving, while comparisons with SLOLS and THOR vary with task complexity.
7.1 TASKS
- The benchmark includes Inverted Pendulum, Hopper, 2D Walker, 3D Walker, and Reacher tasks, with a discount factor of γ = 0.99.The tasks cover pendulum balancing, locomotion, and reaching with a 5-DOF arm.Source: §7.1 · p. 7
7.2 ALGORITHMS
- The comparison uses LOKI, TRPO, DAGGERED, THOR, SLOLS, and an Ideal setup initialized from the expert, with shared parameters except for algorithm-specific ones.Results are averaged across 25 random seeds; policy and value networks use two hidden layers with 32 tanh units per layer.Source: §7.2 · pp. 7–8
- All methods use an on-policy component in constructing their first-order oracle, while the accumulated rewards provided by OpenAI Gym measure performance.The expert is a TRPO policy stopped before convergence.Source: §7.2 · p. 8, §7.3 · p. 8
7.3 EXPERIMENTAL RESULTS
- TRPO improves considerably and almost monotonically from random initialization, while DAGGERED reaches the suboptimal expert’s performance quickly but does not surpass it.The paper gives 15 iterations on 2D Walker as an example of DAGGERED’s speed.Source: §7.3 · p. 8
- Across the reported tasks, LOKI improves as quickly as DAGGERED initially, continues improving, and eventually matches Ideal; it outperforms SLOLS and THOR by a large margin on Hopper and both Walker tasks.The on-policy value function learned during imitation, although unused in that phase, is reported to smooth the transition to reinforcement.Source: §7.3 · p. 8
8 CONCLUSION
The conclusion presents LOKI as an algorithm combining properties of reinforcement and imitation learning. It identifies randomized switching as part of the theoretical contribution.
8 CONCLUSION
A Task Details
The task details list Pendulum, Hopper, 2D Walker, 3D Walker, and Reacher, and report expert value-estimator training and explained-variance values.
A Task Details
- The listed tasks are Pendulum, Hopper, 2D Walker, 3D Walker, and Reacher.Source: A Task Details · p. 11
- Expert value estimators for SLOLS and THOR were trained on 50 times the samples used in each later policy-learning batch.Source: A Task Details · p. 11
- Explained variance was 0.972 for Pendulum, 0.989 for Hopper, 0.975 for 2D Walker, 0.983 for 3D Walker, and 0.973 for Reacher.The passage defines explained variance as 1 minus the variance of error divided by the variance of prediction.Source: A Task Details · p. 11
B.1 Proof of Proposition 1
These proofs establish bounds using mirror-descent arguments, with assumptions and stochastic-gradient conditions specifying when the bounds apply.
B.1 Proof of Proposition 1
- Under convex K and strongly convex R, the special case recovers expected average convergence to stationary points for stochastic smooth nonconvex optimization.Source: §B.1 · pp. 11, 13
B.2 Proof of Proposition 2
- The weighted-cost online mirror-descent lemma is introduced for later Theorem 2, though unnecessary for Proposition 2; unbiased stochastic gradients yield its performance bound in expectation.Source: §B.2 · pp. 13–14
C.1 Proof of Theorem 1
- With ∥g_n∥*≤G almost surely, weighted accumulated loss is bounded, and sampling K according to w_s transfers the bound to J(π_K).Source: §C.1 · p. 15
C.2 Proof of Theorem 3
The proof casts policy updates as online learning, derives a bound under strong convexity and a sufficiently large step size, then relates regret to performance error.
C.2 Proof of Theorem 3
- The proof treats deterministic problems directly and states that stochastic problems extend to expected performance, as in Proposition 2.Source: §C.2 · p. 15
- Each policy update uses the gradient of the per-round cost defined from the policy’s λ-loss at the current policy.Source: §C.2 · p. 15
- Under strong convexity and a sufficiently large step size, the proof obtains a bound and relates performance to the class error plus regret error.Source: §C.2 · pp. 15–16