Source-linked AI summary
Mirror Descent Policy Optimization
Manan Tomar, Lior Shani, Yonathan Efroni, Mohammad Ghavamzadeh
TL;DR
The paper addresses a gap between mirror-descent theory for tabular RL and practical trust-region algorithms. It develops MDPO by approximately solving trust-region policy updates with gradient steps, and reports performance on par with or better than TRPO, PPO, and SAC across benchmark tasks.
Problem
A gap remains between theoretically analyzed tabular mirror-descent algorithms and practical trust-region RL methods such as TRPO and PPO.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Off-policy MDPO provides an optimization perspective on SAC, which the paper derives through slight modifications of MDPO.
Source: §1 · p. 2, §7 · p. 9The paper identifies evaluation in more complex and realistic problems and discrete-action settings, as well as further work on alternative Bregman divergences and exploration, as future directions.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as an important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called {\em mirror descent policy optimization} (MDPO). MDPO iteratively updates the policy by {\em approximately} solving a trust-region problem, whose objective function consists of two terms: a linearization of the standard RL objective and a proximity term that restricts two consecutive policies to be close to each other. Each update performs this approximation by taking multiple gradient steps on this objective function. We derive {\em on-policy} and {\em off-policy} variants of MDPO, while emphasizing important design choices motivated by the existing theory of MD in RL. We highlight the connections between on-policy MDPO and two popular trust-region RL algorithms: TRPO and PPO, and show that explicitly enforcing the trust-region constraint is in fact {\em not} a necessity for high performance gains in TRPO. We then show how the popular soft actor-critic (SAC) algorithm can be derived by slight modifications of off-policy MDPO. Overall, MDPO is derived from the MD principles, offers a unified approach to viewing a number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous control tasks. Code is available at \url{https://github.com/manantomar/Mirror-Descent-Policy-Optimization}.
1. Introduction
The paper addresses a gap between mirror-descent theory for tabular RL and practical trust-region algorithms. It introduces MDPO as a practical method and uses it to connect and compare popular policy-optimization algorithms.
1. Introduction
- TRPO enforces its trust-region constraint with line search, whereas PPO clips policy ratios in an unconstrained optimization problem.The cited studies report that clipping reduces, but does not prevent, ratios from leaving their bounds.Source: §1 · p. 1
- Mirror descent is a first-order method for constrained convex optimization, but a gap remains between theoretically analyzed tabular RL algorithms and practical methods such as TRPO and PPO.The paper motivates its approach as a way to develop practical algorithms from MD theory and better understand trust-region methods.Source: §1 · p. 1
- MDPO approximately solves trust-region policy updates with multiple gradient steps, and the paper derives on-policy and off-policy variants from mirror-descent principles.The variants are used to study connections to TRPO and PPO and compare performance on continuous-control tasks.Source: §1 · p. 1
- The paper shows that off-policy MDPO coincides with SAC when the trust region is defined relative to the uniform policy rather than the old policy.It further studies this connection and compares the algorithms on continuous-control problems.Source: §1 · p. 2
- On-policy and off-policy MDPO achieve state-of-the-art performance across benchmark tasks; MDPO does not require a hard trust-region constraint, and TRPO consistently outperforms PPO in both vanilla and code-optimized versions.The paper also presents MDPO as an excellent alternative to TRPO, PPO, and SAC.Source: §1 · p. 2
2. Preliminaries
The preliminaries specify the discounted MDP setting and introduce entropy-regularized objectives alongside standard value and advantage functions. They then describe mirror descent as a trust-region update that combines a linearized objective with a proximity term.
2. Preliminaries
- The paper models interaction as a γ-discounted MDP and defines state-value, action-value, and advantage functions for a policy.The discounted state-visitation frequency is also introduced.Source: §2 · p. 2
- Entropy regularization adds an entropy-related term to the reward; setting its regularization parameter λ to zero recovers the hard MDP.The paper names Shannon, Tsallis, and relative entropy as possible terms and uses “regularized” and “soft” interchangeably.Source: §2 · p. 2
2.1. Mirror Descent in Convex Optimization
- Mirror descent minimizes a linear approximation of the objective plus a proximity term that keeps consecutive estimates close.Its stated setting is constrained convex optimization with a convex objective and a convex compact constraint set.Source: §2.1 · p. 2
- The proximity term is a Bregman divergence; with negative Shannon entropy it becomes KL divergence, yielding an exponentiated-gradient update on the unit simplex.With a Euclidean potential, the divergence instead yields projected gradient descent.Source: §2.1 · p. 2
3. Mirror Descent in RL
The section adapts mirror-descent updates to RL despite the policy objectives’ nonconvexity. It reports convergence guarantees for the tabular algorithms and relates their rates to those in convex optimization.
3. Mirror Descent in RL
- The RL formulations optimize either value over the full state space or the value at an initial state drawn from μ; the latter is more common in policy optimization.The state-frequency measure induced by the current policy is used in the latter MD-style update, while the other formulation is uniform over states.Source: §3 · p. 3
- Although the RL policy objectives are not convex, prior work applies the general mirror-descent update rule to derive MD-style algorithms for both formulations.The section presents these algorithms as an adaptation of the update rule to the two RL optimization problems.Source: §3 · p. 3
- MD-style RL algorithms converge to the global optimum in the tabular case, with K denoting total iterations; their rate can be improved to e O(1/K) in soft MDPs.These rates match those for mirror descent in convex optimization.Source: §3 · p. 3
4. Mirror Descent Policy Optimization
MDPO approximates mirror-descent trust-region updates with multiple stochastic-gradient steps, yielding on-policy and off-policy variants. The section connects on-policy MDPO to TRPO and PPO and derives SAC from off-policy MDPO through related KL-based updates.
4. Mirror Descent Policy Optimization
- MDPO approximates trust-region updates with multiple SGD steps on their objectives, with on-policy and off-policy variants.The on-policy variant estimates advantages from current-policy Monte Carlo data; the off-policy variant uses replay-buffer samples.Source: §4 · p. 3
4.1. On-Policy MDPO
- A single SGD step on the on-policy objective reduces to vanilla policy gradient, so MDPO uses multiple steps to retain the trust-region effect.Source: §4.1 · p. 3
- On-policy MDPO approximately satisfies the trust-region constraint through multiple SGD steps, using simple SGD rather than TRPO’s natural gradient.It uses KL(π, πk) and anneals its step size from 1 to 0 over iterations.Source: §4.1 · p. 4
4.2. Off-Policy MDPO
- Off-policy MDPO samples replay-buffer states to approximate uniform state sampling, making updates less dependent on the current policy.It estimates the current policy’s value functions from replay data and uses the resulting advantage estimate in multiple SGD steps.Source: §4.2 · p. 5
- The paper derives SAC through slight modifications to off-policy MDPO and relates both algorithms to follow-the-regularized-leader.For soft MDPs, the off-policy update modifies the value target and actor loss to include entropy terms.Source: §4.2 · pp. 5–6
- Off-policy MDPO’s KL projection follows from its Bregman-divergence choice; SAC’s loss omits the current policy and can be viewed as trust-region optimization relative to a uniform policy.The paper reports off-policy MDPO performs better than or on par with SAC in six MuJoCo domains, while noting that further investigation of KL direction is needed.Source: §4.2 · p. 6
5. Experimental Results
The experiments compare on- and off-policy MDPO with TRPO, PPO, and SAC on continuous-control tasks. MDPO is reported to perform better than PPO and on par with or better than TRPO and SAC, with outcomes varying by update design and policy variant.
5. Experimental Results
- Experiments compare MDPO with TRPO, PPO, and SAC on continuous-control tasks, reporting final training scores averaged over five runs with 95% confidence intervals.The reported scores are final training scores; the paper boldfaces the best mean scores in its tables.Source: §5 · p. 6
5.1. On Multiple SGD Steps
- On-policy performance favors m = 10 over m = 1, while TRPO gains nothing from multiple steps; off-policy m = 1000 is reasonable across tasks, but larger m is costly.The off-policy experiments used an m-step old policy copy and a single gradient update per iteration to reduce computation; multi-step updates were omitted at scale because of wall-clock time.Source: §5.1 · p. 7
5.2. On Code-level Optimizations
- Code-level optimizations improve TRPO and PPO; their loaded+GAE versions match the state-of-the-art results reported in the literature.The study reports vanilla, loaded, and loaded+GAE versions of the on-policy algorithms.Source: §5.2 · p. 7, §5.3 · p. 7
- Across experiments, MDPO performs significantly better than PPO and on par with or better than TRPO and SAC, while remaining simpler to implement.The authors characterize MDPO as more general because it is derived from mirror descent theory in RL.Source: §5.2 · p. 7
5.3. On-policy Results
- On-policy MDPO performs better than or on par with TRPO and better than PPO across tasks, without requiring TRPO's extra line-search step.The authors report that explicitly enforcing the trust-region constraint is not necessary for these performance gains.Source: §5.3 · p. 7, Table 1 · p. 8
- TRPO consistently outperforms PPO, whose performance decreases on some tasks after 1M time steps; on Ant-v2, TRPO continues improving after both reach about 1000.Adding code-level optimizations and GAE improves PPO but does not make it outperform TRPO.Source: §5.3 · p. 8, Table 1 · p. 8
5.4. Off-policy Results
- Off-policy MDPO-KL performs on par with SAC, while MDPO-Tsallis can outperform SAC across tasks with tuned q values in [1.0, 2.0].Off-policy MDPO also improves sample efficiency and final performance over on-policy MDPO in most tasks.Source: §5.4 · p. 8
- Preliminary SAC-Tsallis experiments showed little improvement over SAC from tuning q, unlike the reported MDPO-Tsallis results, leaving the effect of Tsallis entropy in need of further investigation.For Humanoid-v2 and Ant-v2, prior SAC-Tsallis results cited in the paper reported only the 1M time-step mark rather than the standard 3M mark.Source: §5.4 · p. 8
6. Related Work
Related work connects mirror descent theory to trust-region RL, but practical empirical study of that connection remains limited. The section contrasts prior ECPO results with this paper's off-policy MDPO findings and calls for further investigation of the discrepancy.
6. Related Work
- Several works establish theoretical convergence guarantees for trust-region algorithms through connections to mirror descent, while empirical study of the connection has been limited to one cited work.The cited empirical work focuses on off-policy methods and proposes Exploratory Conservative Policy Optimization (ECPO).Source: §6 · p. 8
- ECPO uses forward KL rather than off-policy MDPO's reverse KL, adding normalization-term estimation overhead; its authors report poor performance for Policy Mirror Descent.The paper notes that ECPO's authors argue the forward-KL direction improves performance.Source: §6 · p. 8
- Despite differing code and configurations, this paper reports off-policy MDPO at or above SAC on six MuJoCo domains, and says further work is needed to explain the discrepancy with prior PMD results.Source: §6 · p. 8
7. Conclusions
The paper concludes that MDPO derives on- and off-policy algorithms from mirror descent by approximately optimizing trust-region objectives with multiple gradient steps. It relates these methods to TRPO, PPO, and SAC, reports competitive experimental performance, and identifies broader evaluation and algorithmic directions for future work.
7. Conclusions
- On- and off-policy MDPO formulate policy updates as trust-region optimization problems and approximate their solutions with multiple gradient steps on the objectives.Source: §7 · pp. 8–9
- The paper relates on-policy MDPO to TRPO and PPO, and shows that SAC can be derived through slight modifications of off-policy MDPO.Source: §7 · p. 9
- Future work includes evaluation on more realistic and discrete-action problems, other Bregman divergences, and incorporating exploration into MDPO.The paper specifically mentions comparing on discrete-action problems with DQN and PPO; it says more Tsallis-related algorithmic and empirical work is needed.Source: §7 · p. 9
A. Pseudocodes
- The appendix provides pseudocodes for the on-policy and off-policy MDPO algorithms.Source: A. Pseudocodes · p. 11
B. Experimental Details
Experiments cover six OpenAI Gym continuous-control tasks, with results averaged over five random seeds and reported with 95% confidence intervals. The appendix describes off-policy settings and compares minimal and code-optimized implementations.
B.1. Setup
- Results use Hopper-v2, Walker2d-v2, HalfCheetah-v2, Ant-v2, Humanoid-v2, and HumanoidStandup-v2, averaged over five seeds with 95% confidence intervals.Tabular results report mean final training performance; bolding marks best means with overlapping confidence intervals.Source: §B.1 · p. 12
- Off-policy experiments fix λ = 0.2, use a fixed Bregman stepsize of 1/tk, and hold the old policy for m gradient steps to match environment and gradient step counts.The λ setting is reported as working equally well for Humanoid as the SAC paper’s 0.05.Source: §B.1 · p. 12
B.2. Code-level Optimization Techniques
- On-policy comparisons include minimal methods, code-optimized loaded methods, and loaded methods with GAE; the appendix lists their hyperparameters in Table 2.The loaded PPO implementation includes value-function and reward clipping, reward and observation normalization, orthogonal initialization, and an annealed Adam learning rate.Source: §B.2 · p. 12, Table 2 · p. 13
- Off-policy experiments include minimal and loaded versions, branched according to neural-network and batch sizes, to compare on-policy and off-policy methods because their standard values differ.Table 3 lists off-policy hyperparameters, and Table 4 gives domain-specific Bregman stepsizes for off-policy MDPO.Source: §B.2 · p. 12, Table 3 · p. 13, Table 4 · p. 13
C. Tsallis-based Bregman Divergence
The section develops a Tsallis-based alternative to the KL Bregman divergence for off-policy MDPO. Its implementation is used off-policy, while the on-policy Tsallis variant is left for future work because its closed-form divergence is cumbersome.
C. Tsallis-based Bregman Divergence
- A strongly convex potential function induces the Bregman divergence, and choosing negative Tsallis entropy gives the Tsallis-based divergence used by off-policy MDPO.The paper writes the off-policy Tsallis objective using a convenient definition and omits a term independent of the optimized policy from its loss.Source: C. Tsallis-based Bregman Divergence · p. 14
- Tsallis entropy generalizes Shannon entropy as q approaches 1, while q = 2 gives the sparse Tsallis entropy.The passage identifies Shannon entropy as the q → 1 limit.Source: C. Tsallis-based Bregman Divergence · p. 14
- The on-policy Tsallis variant was not pursued because its closed-form Bregman divergence is cumbersome to implement, though the paper says it is feasible in principle.The implemented on-policy policies are Gaussian, for which the KL divergence has a closed form.Source: C. Tsallis-based Bregman Divergence · p. 14
D. On-policy Results
On-policy results compare minimal, loaded, and loaded-plus-GAE versions of TRPO, PPO, and MDPO. Code-level optimizations generally raise performance, while the relative performance trend remains consistent with the main results.
D. On-policy Results
- The three on-policy variants are minimal, loaded with code-level optimizations, and loaded+GAE with those optimizations plus GAE.The loaded+GAE comparison is against corresponding PPO and TRPO implementations.Source: D. On-policy Results · p. 15
- Overall performance increases in most cases over minimal versions, while the trend among algorithms remains consistent with the main results.The loaded-plus-GAE and loaded comparisons cover six MuJoCo tasks; the reported figures average five runs with shaded 95% confidence intervals.Source: D. On-policy Results · p. 15, Figure 3 · p. 16, Figure 4 · p. 17
- The minimal comparison evaluates MDPO-M against PPO-M and TRPO-M on six MuJoCo tasks, with results reported in Figure 2 and Table 5.The table reports averages over five runs with 95% confidence intervals.Source: Table 5 · p. 15, Figure 2 · p. 15
E. Off-policy Results
Off-policy results compare MDPO’s KL and Tsallis versions with SAC in minimal and loaded settings. In the loaded comparison, mean scores overlap across methods, while MDPO has a higher mean in five of six domains.
E. Off-policy Results
- Across off-policy minimal and loaded versions, overall performance increases in most cases relative to minimal versions, while trends remain consistent with the main results.The loaded comparison uses MDPO-LOADED and SAC-LOADED.Source: E. Off-policy Results · p. 18
- The minimal comparison covers KL- and Tsallis-based MDPO-M against SAC-M on six MuJoCo tasks, as shown in Figure 5 and Table 7.The figure plots performance against time steps in millions and averages five runs with shaded 95% confidence intervals.Source: Figure 5 · p. 18, Table 7 · p. 18
- Despite overlap in performance, MDPO has a higher mean score in 5 of 6 domains in the loaded comparison with SAC.Figure 6 compares KL- and Tsallis-based MDPO-LOADED with SAC-LOADED across six MuJoCo tasks; Table 8 reports their performance.Source: Figure 6 · p. 19, Table 8 · p. 19
F. Additional Experiments
These experiments find that multiple gradient updates improve both off-policy MDPO and SAC, while matching the Tsallis entropy settings performs best for MDPO-Tsallis. Preliminary SAC-Tsallis experiments show little improvement from tuning q over SAC.
F.1. Multi-step Update
- Multiple gradient updates improve scores for both off-policy MDPO and SAC on Hopper-v2.Figure 7 compares single and multiple updates; the main experiments used single-step SAC updates for a fair comparison.Source: §F.1 · p. 20, Figure 7 · p. 20
F.2. Different Tsallis Entropies for Bregman and MDP Regularization
- Matching the q values for the Bregman divergence and MDP regularizer gives the best performance across Walker2d-v2, Humanoid-v2, and Ant-v2.Table 9 reports results averaged over five runs with 95% confidence intervals.Source: §F.2 · p. 20, Table 9 · p. 20
F.3. Tsallis-based SAC
- Preliminary SAC-Tsallis experiments show little improvement over SAC from tuning q, unlike the reported MDPO-Tsallis results.Figure 8 reports SAC-Tsallis performance averaged over five runs with 95% confidence intervals.Source: §F.3 · p. 21, Figure 8 · p. 21