PaperlayerView exact revision on arXiv

Source-linked AI summary

Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning

Felix Leibfried, Jordi Grau-Moya

arXiv:1909.05950v1cs.LGcs.AIstat.ML

TL;DR

Mutual-information regularization had been studied in tabular MDPs and high-dimensional discrete-action settings, leaving continuous-action applications as a focus. The paper analyzes its Bellman operator and develops MIRACLE, an actor-critic method with an adaptive marginal prior. In Mujoco, MIRACLE is competitive with contemporary methods and improves over SAC in Ant.

  • Problem

    Mutual-information regularization had been studied for tabular MDPs and high-dimensional discrete-action RL, while continuous-action actor-critic learning remained to be addressed.

    Source: §1 · p. 1, §4 · p. 7

  • Method

    The paper analyzes the mutual-information-regularized Bellman operator and develops MIRACLE, an actor-critic algorithm for continuous actions with an adaptive marginal prior.

    Source: §3.2 · p. 5, §4 · p. 7

  • Results

    MIRACLE is competitive with contemporary Mujoco methods, consistently outperforms SAC on lower-dimensional environments, and can outperform SAC on high-dimensional ones.

    Source: §5 · p. 8, §6 · p. 8

  • Takeaways & Limitations

    MIRACLE may not always help because it favors actions close to past actions, which may differ from those identified by the optimization procedure.

    Source: §5 · p. 8

    The policy-dependent marginal action distribution couples optimization across states, so standard tools for establishing unique optimal values and convergence do not apply straightforwardly.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it tackles the value overestimation problem. It also leads to a significant performance increase in tabular and high-dimensional settings, as demonstrated via algorithms such as soft Q-learning (SQL) and soft actor-critic (SAC). Cumulative entropy regularization has been extended to optimize over the reference marginal policy instead of keeping it fixed, yielding a regularization that minimizes the mutual information between states and actions. While this has been initially proposed for Markov Decision Processes (MDPs) in tabular settings, it was recently shown that a similar principle leads to significant improvements over vanilla SQL in RL for high-dimensional domains with discrete actions and function approximators. Here, we follow the motivation of mutual-information regularization from an inference perspective and theoretically analyze the corresponding Bellman operator. Inspired by this Bellman operator, we devise a novel mutual-information regularized actor-critic learning (MIRACLE) algorithm for continuous action spaces that optimizes over the reference marginal policy. We empirically validate MIRACLE in the Mujoco robotics simulator, where we demonstrate that it can compete with contemporary RL methods. Most notably, it can improve over the model-free state-of-the-art SAC algorithm which implicitly assumes a fixed reference policy.

1 Introduction

The introduction motivates mutual-information regularization as an extension of entropy regularization and presents a theory-and-algorithm contribution for continuous-action reinforcement learning. It reports that MIRACLE is competitive in Mujoco and improves over SAC in Ant.

1 Introduction

  • Entropy regularization has improved exploration and robustness and alleviated value overestimation, with performance gains in tabular and high-dimensional settings.The cited high-dimensional examples include soft Q-learning and soft actor-critic.Source: §1 · p. 1
  • Optimizing the reference policy yields mutual-information regularization, previously studied in tabular MDPs and high-dimensional discrete-action RL, where it improved over fixed-reference SQL.The high-dimensional work uses function approximators.Source: §1 · p. 1
  • The paper analyzes the mutual-information-regularized Bellman operator and develops continuous-action MIRACLE, reporting improved Ant performance over SAC in Mujoco.The introduction also describes the Mujoco results as competitive with contemporary methods.Source: §1 · p. 1

2 Background

The background introduces MDP and reinforcement-learning notation, then sets up mutual-information regularization as a constrained reward-maximization approach for non-sequential decisions.

2.1 Markov Decision Process (MDP) and Reinforcement Learning (RL)

  • An MDP specifies states, actions, transitions, rewards, and discounting; policies map states to actions, and optimal policies maximize expected cumulative reward.In RL, agents lack prior knowledge of transitions and rewards and learn through environment interaction; actor-critic methods are suited to high-dimensional continuous-action settings.Source: §2.1 · p. 2

2.2 Mutual-Information Regularization for Non-Sequential Decision-Making

  • The non-sequential formulation constrains mutual information between states and actions, with a marginal action distribution and a trade-off parameter for reward maximization versus information minimization.The marginal action distribution averages the policy over states.Source: §2.2 · p. 2
  • The agent is treated as an information channel that balances reward-based reconstruction against an upper bound on state-action information.The framework asks the agent to discard state information with little impact on reward and is related to rate-distortion theory and the information bottleneck [24].Source: §2.2 · pp. 2–3

3 Mutual-Information Regularization in MDPs

The paper recasts mutual-information regularization in MDPs as inference with an optimizable action prior, then analyzes the Bellman operator that results. Its application uses alternating policy and prior updates, which converge to an optimal policy at rate O(1/M) under the stated conditions.

3.1 Problem Formulation from an Inference Perspective

  • Because the policy-induced state distribution depends on the policy, the paper instead uses a policy-independent distribution; stronger transition-consistency assumptions are left for future work.The marginal action distribution depends on the behavioral policy across states, so the Bellman operator cannot be applied independently state by state and standard uniqueness and convergence tools do not apply straightforwardly.Source: §3.1 · p. 4

3.2 Theoretical Analysis of the Bellman Operator

  • O(1/M) convergence to an optimal policy is established for alternating Blahut-Arimoto updates, given a bounded value function and an initial policy with support in every action.The Bellman optimization is solved by alternating the one-step optimal policy and prior updates.Source: §3.2 · pp. 5–6

3.1 Problem Formulation from an Inference Perspective

  • Optimizing the prior action distribution yields the marginal policy and turns the expected KL penalty into mutual information between states and actions.The formulation also applies to discounted infinite-horizon settings with the specified policy-induced state distribution.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Theoretical Analysis of the Bellman Operator

  • With a fixed prior, the mutual-information-regularized Bellman operator generalizes the soft Bellman operator and cumulative entropy regularization as special cases.The paper reports grid-world validation of convergence and compares mutual-information-regularized values with soft values using a fixed prior.Source: §3.2 · p. 6, Figure 1 · p. 6

4 Mutual-Information Regularization with Continuous Actions

The section presents MIRACLE, a mutual-information-regularized actor-critic method for high-dimensional continuous actions, inspired by the analyzed Bellman operator. It learns a state-averaged marginal action policy and uses it to regularize the actor.

4 Mutual-Information Regularization with Continuous Actions

  • MIRACLE extends mutual-information regularization to high-dimensional continuous actions, using the analyzed Bellman operator as inspiration.The algorithm optimizes over the reference prior rather than keeping it fixed, in contrast to SAC's implicit fixed prior.Source: §4 · p. 7
  • The actor is trained by gradient ascent, with a logarithmic penalty for deviations between its policy and the learned marginal action policy.The marginal policy approximates the actor policy averaged over states sampled from the replay buffer.Source: §4 · p. 7
  • A separate V-critic supports Q-critic learning, with critic training based on replay-buffer samples.The section introduces a V-critic and describes gradient-based losses for the Q-critic and V-critic.Source: §4 · p. 7
  • The marginal action policy is approximated by a probabilistic map from Gaussian samples to actions, trained by maximum log likelihood.The policy's marginal action distribution is fitted using actions from the replay buffer.Source: §4 · p. 7
  • For Mujoco, replay-buffer states approximate the policy-induced marginal, since a fixed state distribution is unsuitable for admissible states.Experiments with a variational autoencoder for the marginal policy did not yield better results; training is off-policy with minibatch updates and reparameterization.Source: §4 · p. 7

5 Experiments in Mujoco

MIRACLE is evaluated on Mujoco v2 environments against SAC, DDPG, and PPO, with ten seeds per experiment. It consistently outperforms SAC in lower-dimensional environments and can outperform SAC in high-dimensional tasks, though it may not always help.

5 Experiments in Mujoco

  • The comparison with SAC isolates the reference-policy choice: the SAC and MIRACLE implementations otherwise differ only in this aspect.Experiments use ten seeds and the latest Mujoco v2 environments.Source: §5 · p. 8
  • MIRACLE outperforms DDPG and PPO in high-dimensional environments.DDPG from RLlib did not work properly in the Ant setup.Source: §5 · p. 8, Figure 2 · p. 8
  • MIRACLE may not always help because it favors actions close to past actions, which can differ from those identified by the optimization procedure.The passage relates this behavior to experiments in discrete-action environments.Source: §5 · p. 8, Figure 2 · p. 8
  • MIRACLE consistently outperforms SAC in lower-dimensional environments and can outperform SAC in high-dimensional tasks, including Ant.The figure reports average episodic rewards over the last 100 episodes with standard errors; the Ant curve includes a kink caused by one seed.Source: Figure 2 · p. 8, §5 · p. 8

6 Conclusion

The conclusion presents mutual-information regularization as an inference-motivated extension of the soft Bellman operator and introduces MIRACLE for continuous actions. Mujoco results are competitive with contemporary methods, with improvements over SAC reported in Ant.

6 Conclusion

  • The mutual-information-regularized Bellman operator generalizes the soft Bellman operator and receives a theoretical analysis.The analysis motivates a practically applicable algorithm.Source: §6 · p. 8
  • MIRACLE is an actor-critic algorithm with an adaptive marginal prior for high-dimensional continuous domains.It is inspired by the mutual-information-regularized Bellman operator.Source: §6 · p. 8
  • MIRACLE demonstrates competitive Mujoco results and improves over SAC in Ant.SAC implicitly assumes a fixed marginal prior.Source: §6 · p. 8

A Details Regarding the Proof of Lemma 2

The proof details derive the result of Lemma 2 by forming a Lagrangian, solving its stationary conditions for the policy and multiplier, and substituting the resulting expression into the earlier equation.

A Details Regarding the Proof of Lemma 2

B Details Regarding the Proof of Lemma 3

The proof derives an upper value bound for Lemma 3 from Equation (18), using the policy-independence of rewards and values.

C Details Regarding the Proof of Lemma 4

The proof averages a rearranged bound over iterations to obtain Equation (19); the accompanying remark recommends uniform initialization, and the appendix specifies the grid-world setup.

C Details Regarding the Proof of Lemma 4

D Remark Regarding Lemma 4

  • A uniform initial policy minimizes the upper bound in Equation (19).The remark gives U(A) = arg minπ maxa log (π(a))−1 over admissible action distributions.Source: D Remark Regarding Lemma 4 · p. 12

E Grid World Setup

  • The grid-world setup uses a deterministic 16 × 16 grid, five actions, a +9 terminal goal reward, and a −1 step penalty.Value iteration and the inner Blahut-Arimoto scheme both stop when their specified differences fall below 5 · 10−3.Source: E Grid World Setup · p. 12

F Experiment Details for Mujoco

The Mujoco experiments use shared network and optimization settings, Gaussian policy networks with bounded actions, and an additional variational-autoencoder alternative for marginal actions.

  • All function approximators use Adam with a 3·10−4 learning rate, 256-sample minibatches, two 256-unit ReLU layers, and a replay buffer of up to one million transitions.The discount is γ = 0.99, reward scale β = 10, and the V-target network uses τ = 0.01.Source: F Experiment Details for Mujoco · p. 12
  • The policy and marginal-policy networks output Gaussian means and log standard deviations, with tanh adjusting actions to Mujoco bounds.Source: F Experiment Details for Mujoco · p. 12
  • The variational-autoencoder alternative models marginal actions with an ELBO and adds a recognition network that maps actions to latent-Gaussian parameters.Source: F Experiment Details for Mujoco · p. 12

G Experiment Results in Mujoco

On the best-so-far episodic-reward measure, MIRACLE improves over SAC in five of eight Mujoco environments, most notably Ant; variational-autoencoder experiments perform worse than the main-paper setup.

  • In one of ten Ant runs, episodic rewards dropped around 800,000–900,000 steps, while the other runs kept improving.The authors also report no DDPG baseline in Ant and say their RLlib implementation failed to recover under the tested hyperparameters.Source: G Experiment Results in Mujoco · p. 12
  • MIRACLE improves over SAC in five of eight environments under best episodic reward, most notably Ant.The measure is the best average episodic reward over the last 100 episodes obtained so far during training.Source: Figure 3 · p. 13
  • Variational-autoencoder marginal-action experiments were worse than the main-paper results under the same setup; the authors suggest more training samples might help.Their best-reward plots are presented as analogous to the main best-reward figure.Source: G Experiment Results in Mujoco · p. 12, Figure 4 · p. 13, Figure 5 · p. 14
Loading 1909.05950v1…