Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Sergey Levine

Source-linked AI summary

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Sergey Levine

arXiv:1805.00909v3May 2, 2018cs.LGcs.AIcs.ROstat.ML

TL;DR

The paper addresses how reinforcement learning and optimal control can be connected to probabilistic inference, a connection valuable for algorithm design and flexible modeling. It develops a maximum entropy control-as-inference framework, deriving exact inference for deterministic dynamics and variational inference for stochastic dynamics, while reviewing related algorithms and future directions. The paper also identifies risk-seeking behavior in a standard probabilistic graphical model for stochastic dynamics and notes that robust-control connections remain insufficiently explored.

Problem
The connection between reinforcement learning or optimal control and inference in probabilistic models is not immediately obvious, despite its potential value for approximate inference, model extension, compositionality, and partial observability.
Method
The paper gives a unified tutorial treatment that embeds maximum entropy reinforcement learning in probabilistic graphical models and derives exact, variational, and approximate inference procedures.
Results
Maximum entropy reinforcement learning is equivalent to exact probabilistic inference for deterministic dynamics and variational inference for stochastic dynamics, with related update equations recovered through structured variational inference.
Takeaways & Limitations
The control-as-inference framework supports entropy-based exploration, inverse reinforcement learning, approximate inference, probabilistic reward interpretation, and improved stability, exploration, and model-free performance in cited algorithms.
Takeaways & Limitations
The standard probabilistic graphical model is often not well suited to stochastic dynamics because its optimistic soft-max backup can create risk-seeking behavior, while the connection to robust control remains insufficiently explored.

Abstract

from arXiv · show

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

1 Introduction

The article reframes reinforcement learning and optimal control as probabilistic inference in graphical models, connecting decision making with approximate inference and flexible probabilistic modeling. It develops a maximum entropy formulation that supports exact inference for deterministic dynamics and variational inference for stochastic dynamics.

1 Introduction: The article treats decision making, reinforcement learning, and optimal control as inference problems in probabilistic graphical models rather than fundamentally separate problems from probabilistic inference.This reframing can support approximate inference tools, flexible model extensions, and reasoning about compositionality and partial observability.
1 Introduction: Maximum entropy reinforcement learning is equivalent to exact probabilistic inference with deterministic dynamics and variational inference with stochastic dynamics.The article presents this connection as a unifying framework for control and reinforcement learning.
1 Introduction: The framework connects several related approaches, including Kalman duality, maximum entropy reinforcement learning, KL-divergence control, and stochastic optimal control.Despite differing derivations, these methods share an underlying framework and optimization objective involving probabilistic graphical models and inference methods.
1 Introduction: Formulating reinforcement learning as inference provides entropy-based exploration, tools for inverse reinforcement learning, and approximate inference methods for solving reinforcement learning problems.It also gives rewards a probabilistic interpretation and relates them to the optimal policy.
1 Introduction: The article offers a unified tutorial and review rather than claiming a fundamentally novel interpretation of the control–inference connection.It derives the probabilistic model, discusses exact and variational inference, relates function approximation and deep reinforcement learning to the framework, reviews recent literature, and outlines future directions.

2 A Graphical Model for Control as Inference

This section embeds maximum-entropy reinforcement learning into a probabilistic graphical model by adding reward-dependent optimality variables to states, actions, and dynamics. Conditioning on optimality enables inference of action sequences or policies, with exact inference under deterministic dynamics and an entropy-augmented objective.

2.1 The Decision Making Problem and Terminology: A reinforcement-learning task is specified by a reward function, and solving it recovers a policy that distributes actions conditioned on states.The policy may be parameterized by a vector θ, such as neural-network weights, or represent an open-loop action sequence.
2.1 The Decision Making Problem and Terminology: The graphical model augments state-action dynamics with binary optimality variables whose true values encode reward-dependent optimality.The resulting model treats optimality variables like observations in an HMM-style framework.
2.2 Control as Inference: Conditioning on all optimality variables being true yields a trajectory distribution proportional to dynamics probability multiplied by exp of the total reward.Under deterministic dynamics, dynamically feasible trajectories share a constant dynamics term, so the highest-reward trajectory is most probable.
2.3 Policy Search as Probabilistic Inference: The optimal policy can be recovered from the posterior action conditional given future optimality variables using HMM-style sum-product inference and backward messages.The messages represent the probability that the remaining trajectory can be optimal from a given state or state-action pair.
2.3 Policy Search as Probabilistic Inference: For stochastic dynamics, the standard graphical-model backup uses a soft maximum over next-state values, producing optimistic and potentially risk-seeking policies.The framework is therefore often unsuitable for stochastic dynamics without the variational correction developed in the next section.
2.4 Which Objective does This Inference Procedure Optimize?: The resulting objective maximizes expected reward together with expected conditional entropy, defining maximum-entropy reinforcement learning or control.A uniform action prior can be used for simplicity, while non-uniform action priors can be absorbed into the reward-dependent optimality factor.

3 Variational Inference and Stochastic Dynamics

This section corrects the stochastic-dynamics formulation by fixing the transition and initial-state distributions and optimizing only the policy. The resulting procedure is maximum-entropy control under true stochastic dynamics and corresponds to structured variational inference.

3 Variational Inference and Stochastic Dynamics: The stochastic-dynamics problem arises because the posterior can differ from the true dynamics, effectively allowing the agent limited control over system transitions.This mismatch explains why unlikely but severely bad outcomes can be removed from risky actions.
3.1 Maximum Entropy Reinforcement Learning with Fixed Dynamics: Freezing the dynamics and initial-state distribution prevents the inference procedure from modifying transitions, while leaving the policy as the optimization variable.The fixed distributions are q(st+1|st,at) = p(st+1|st,at) and q(s1) = p(s1).
3.1 Maximum Entropy Reinforcement Learning with Fixed Dynamics: With fixed stochastic dynamics, the objective still maximizes reward and entropy, but it is optimized under the true transition process.The policy optimization can be derived through dynamic programming rather than by changing the dynamics.
3.1 Maximum Entropy Reinforcement Learning with Fixed Dynamics: The terminal policy is proportional to exp of the immediate reward normalized by exp of the state value.This follows from minimizing the KL-divergence between the policy and the reward-weighted action distribution.
3.1 Maximum Entropy Reinforcement Learning with Fixed Dynamics: The recursive policy has the form π(at|st) = exp(Q(st,at) − V(st)), and fixed dynamics produce a Bellman backup based on expected next-state value rather than an optimistic estimate.The KL-divergence vanishes when the policy matches this normalized exponential form.
3.2 Structured Variational Inference: The resulting optimization is structured variational inference that approximates the maximum-entropy trajectory posterior while changing the policy but not the dynamics.The variational distribution retains the original initial-state and transition factors and optimizes the action policy under those constraints.

4 Approximate Inference with Function Approximation

The framework turns maximum-entropy reinforcement learning into approximate inference, yielding practical policy-gradient, actor-critic, and soft Q-learning algorithms through parameterized policies and messages.

4.1 Maximum Entropy Policy Gradients: Structured variational inference optimizes a parameterized policy qθ(at|st) while fixing the initial-state and dynamics factors to their model distributions.The policy may be represented by a neural network producing categorical or continuous-action distribution parameters, with gradients estimated from samples.
4.1 Maximum Entropy Policy Gradients: Maximum-entropy policy gradients retain the standard policy-gradient form after adding −log qθ(at′|st′) to each timestep’s reward.The resulting estimator can use standard advantage estimators and trajectory sampling from the current policy.
4.2 Maximum Entropy Actor-Critic Algorithms: Backward message passing produces soft value and Q-functions, with the optimal policy recovered from these messages and the entropy term.At convergence, the messages correspond to the optimal soft quantities; before convergence, they represent the current policy’s values.
4.2 Maximum Entropy Actor-Critic Algorithms: Actor-critic algorithms approximate Q(st,at) and V(st) with parameterized functions and use the backward Q-message instead of a Monte Carlo advantage estimate.This treatment generally provides lower-variance gradient estimates and motivates maintaining both value and Q networks.
4.3 Soft Q-Learning: Soft Q-learning replaces the hard maximization in standard Q-learning with a soft maximum, approaching the hard update as reward magnitude increases.For discrete actions, the integral becomes a sum and the policy is obtained by normalizing the Q-function; continuous actions require further approximation.
4.3 Soft Q-Learning: The maximum-entropy framework exposes a connection between soft Q-learning and policy gradients that is not apparent in the standard formulation.The policy-gradient and value-gradient terms can exactly match the soft Q-learning update for a particular baseline choice.

5 Review of Prior Work

Prior work develops multiple formulations connecting control and reinforcement learning with probabilistic inference, including path-integral control, linearly solvable MDPs, policy search, and entropy-based methods. These frameworks support tractable inference, exploration, off-policy learning, policy matching, and diverse behavior modeling, while differing in assumptions and objectives.

5.1 Frameworks for Control as Inference: Control-as-inference frameworks include Kalman duality, linearly solvable MDPs, path-integral control, and graphical-model planning methods.These approaches formulate control or decision making through graphical models and inference procedures, with some arriving at equivalent frameworks through different derivations.
5.1 Frameworks for Control as Inference: Linearly solvable MDPs improve tabular tractability by marginalizing actions or treating actions as equivalent to next states.This removes the nonlinear maximization operator from the standard reinforcement learning formulation.
5.2 Related but Distinct Approaches: Entropy regularization prevents policies from becoming prematurely deterministic, while KL-divergence constraints bound policy changes for smoother convergence.These techniques are related to maximum-entropy updates but do not necessarily optimize the maximum-entropy objective overall.
5.2 Related but Distinct Approaches: Reward-weighted regression and related variational policy search methods weight samples by return, then fit policies more closely to high-return actions.Some variants use importance weights, exponential reward transformations, or explicit trajectory optimization to construct target actions.
5.3 Reinforcement Learning Algorithms: Maximum-entropy methods support off-policy learning, model-based-to-global policy matching, improved exploration, and model-free performance and stability.For speed-only quadrupedal locomotion, the maximum-entropy policy runs in all directions with equal probability rather than choosing an arbitrary direction.
5.3 Reinforcement Learning Algorithms: Latent-variable extensions represent time-correlated exploration, multiple skills, complex multimodal policies, and higher-level control through latent action spaces.The latent variable may remain fixed throughout an episode or be selected independently at each timestep, producing different exploration and representation properties.
5.4 Inverse Reinforcement Learning: Maximum-entropy reinforcement learning is also used for inverse reinforcement learning to infer intent, recover reward functions, and predict observed agent behavior.This extends the framework beyond forward control and planning into probabilistic modeling of goal-directed behavior.

6 Perspectives and Future Directions

The paper presents control as structured variational inference, with maximum expected reward recovered as a limiting case and maximum entropy as the general formulation. It highlights extensions to latent-variable models, robust control, and reward design as important future directions.

6 Perspectives and Future Directions: Structured variational inference recovers value-function-like update equations for a graphical model of reward maximization.The classical maximum expected reward formulation appears as a limiting case, while the general formulation also maximizes policy entropy.
6 Perspectives and Future Directions: Latent-variable extensions model time-correlated exploration and higher-level control through learned latent action spaces.The paper identifies graphical models' extensibility and compositionality as resources for developing more sophisticated reinforcement learning methods.
6 Perspectives and Future Directions: The connection between maximum-entropy reinforcement learning and robust control remains insufficiently explored theoretically.The paper notes that policies trained with injected noise may be robust to unexpected perturbations, domain shift, and model errors, but broader analysis is lacking.
6 Perspectives and Future Directions: A probabilistic interpretation of rewards as log probabilities could inform more interpretable, effective, and easier-to-specify reward functions.The paper emphasizes reward design as practically consequential because applications often depend substantially on selecting a suitable reward function.