PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy Regularization for Mean Field Games with Learning

Xin Guo, Renyuan Xu, Thaleia Zariphopoulou

arXiv:2010.00145v2math.OCcs.LGstat.ML

TL;DR

Finite-horizon entropy regularization in multiagent reinforcement learning lacks theoretical study, while exploration scheduling also lacks theoretical validation. This paper analyzes entropy-regularized mean field games and derives policy implications. It finds time-dependent Gaussian exploration and reports that exploration costs rise toward the terminal time.

  • Problem

    The role of entropy regularization in multiagent reinforcement learning is theoretically understudied, and finite-horizon mean field games remain an open case.

    Source: §1 · p. 2

  • Method

    The paper analyzes Shannon and enhanced entropy regularization in finite-horizon mean field games and derives explicit Nash equilibria for a class of linear-quadratic games.

    Source: §1 · pp. 2–3

  • Results

    The optimal policies suggest Gaussian exploration under entropy regularization, with both variances time-dependent and exploration costs increasing toward the terminal time.

    Source: §3.3 · p. 11

  • Takeaways & Limitations

    Entropy regularization yields time-dependent policies and helps stabilize and accelerate convergence to game equilibrium.

    Source: §1 · p. 3

    The iterative procedure is not guaranteed to yield an MFG solution because its first step may have multiple solutions under relaxed controls.

    Source: §2 · p. 7

Abstract

from arXiv · show

Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.

1 Introduction

The paper addresses limited theoretical analysis of entropy regularization in multiagent reinforcement learning, especially for finite-horizon mean field games. It studies Shannon and enhanced entropy and derives implications for time-dependent policies and exploration scheduling.

1 Introduction

  • Exploration must balance opportunities to improve suboptimal solutions against its cost and the risk that over-exploration impairs convergence.Pure exploitation is easy to implement but tends toward suboptimal global solutions.Source: §1 · p. 1
  • Finite-horizon entropy regularization in multiagent reinforcement learning remains theoretically understudied, despite empirical reports of convergence improvement and variance reduction.The cited theoretical work addresses infinite-horizon stationary discrete-time mean field games, leaving the finite-horizon case open.Source: §1 · p. 2
  • The paper studies Shannon entropy and enhanced entropy, a combination of Shannon entropy and cross-entropy, in finite-horizon MFG with learning.It frames MFG as a large-population setting for MARL, which is critical to avoiding MARL's curse of dimensionality.Source: §1 · pp. 2–3
  • For a class of linear-quadratic stochastic games, the authors derive explicit Nash equilibria and theoretically justify time-dependent policies under entropy regularization.They also report that entropy regularization helps stabilize and accelerate convergence to game equilibrium.Source: §1 · p. 3
  • The paper proposes a model-free policy-gradient algorithm for agents to learn exploration scheduling in unknown environments with large competing populations.Convergence to game equilibrium is described as stable and fast when appropriate exploration rates are chosen.Source: §1 · p. 3
  • The algorithm is inspired by single-agent LQ policy gradients [13] and differs from concurrent MFG policy-gradient work [38] by including exploration.The concurrent work is described as addressing global convergence without exploration.Source: §1 · p. 3

2 Mathematical Formulation

The formulation reduces a large-population game to a representative agent interacting through population state and action distributions. It models exploration with randomized policies and entropy terms, then describes an iterative fixed-point procedure for solving the game.

2 Mathematical Formulation

  • The MFG aggregation framework treats agents as identical and interchangeable, with interactions through population state and action distributions, allowing analysis through one representative agent.The population distributions can be considered as limits of empirical distributions when the number of agents tends to infinity.Source: §2 · p. 3
  • Learning and exploration are modeled by randomized policies, also called relaxed controls or mixed strategies, represented by probability densities over actions.The action space is a closed subset of a Euclidean space.Source: §2 · pp. 3–4
  • The formulation uses Shannon entropy and cross-entropy to encourage exploration, with separate temperature parameters for self-exploration and exploration of other agents' actions.The text associates these terms with information gain about the unknown environment and other agents' policies, respectively.Source: §2 · pp. 4–5
  • The controlled process is posed over a finite horizon with population state and action distribution flows and a representative agent following a randomized policy.The process uses an initial state distribution and Brownian motion.Source: §2 · p. 4
  • The MFG solution procedure alternates between optimizing the representative agent's policy for fixed population flows and updating those flows until convergence.The three-step procedure reduces the fixed-flow problem to stochastic control with randomized policies, updates population action and state distributions, and repeats.Source: §2 · p. 7
  • This procedure is not guaranteed to yield an MFG solution because Step 1 may have multiple solutions under relaxed controls.For the paper's linear-quadratic MFG family, verification arguments can establish that the explicit fixed-point solution is a solution.Source: §2 · p. 7
  • Uniqueness of MFG solutions with relaxed controls generally requires additional convexity properties of the value function, which fail in the linear-quadratic framework when entropy regularization is included.The passage cites regarding the convexity condition.Source: §2 · p. 7

3 Shannon Entropy and Enhanced Entropy for MFG with Learning

This section analyzes Shannon and enhanced entropy regularization in finite-horizon LQ mean field games, deriving equilibrium and optimal feedback policies while examining how entropy shapes exploration. It also establishes uniqueness for equation (3.31) and its associated controlled state process, using a fixed-point and verification approach.

3.2 Game with Enhanced Entropy (Linear Combination of Shannon Entropy and Cross-entropy)

  • Enhanced entropy, a linear combination of Shannon entropy and cross-entropy, also has an explicit Nash equilibrium with a unique optimal state process and time-independent population mean.The formulation assigns temperature parameters λSE and λCE to the two entropy terms.Source: §3 · p. 7, §3.2 · pp. 10–11

3.1 Game with Shannon Entropy

  • The LQ games yield explicit Nash equilibria: Shannon-regularized policies are Gaussian, mean-reverting, and have time-dependent variance, while the population mean is time-independent.With Shannon entropy alone, the population action distribution does not interact with the representative agent’s policy.Source: §3.1 · pp. 8–9

3.3 Discussion.

  • Across both games, optimal policies are Gaussian; their means depend on mean-field interaction and the agent’s state, and their variances vary over time.The strength of mean reversion is quantified by B D^2; smaller variance indicates less uncertainty and a faster mean-reverting policy.Source: §3.3 · p. 11
  • Exploration incentives change over time: agents have more incentive to explore when early-time exploration costs are low, but explore less as the game approaches its terminal reward.When A ≡ Q ≡ 0, Shannon-policy variance decreases over time, implying more exploration at the beginning and less near the end.Source: §3.3 · p. 11
  • Cross-entropy changes the optimal policy differently from Shannon entropy: its mean depends on λCE relative to λSE, and adding it increases exploration and speeds learning convergence.The discussion states that λSE and λCE also affect the enhanced-entropy policy’s variance.Source: §3.3 · p. 11

3.4 Derivations and Proofs of Main Results

  • The proofs first find a candidate solution using a fixed-point approach, then verify it with a verification theorem.For the Shannon game, the derivation uses an HJB equation and forward equation to obtain the mean-field fixed point and candidate policy.Source: §3.4 · pp. 12–13
  • The proof establishes a unique solution to equation (3.31), derives its associated optimal feedback policy, and identifies the controlled state process as the unique solution of SDE (3.13).Verification is omitted because it is similar to the verification of Theorem 2.Source: §3.4 · pp. 19–20

4 Experiment

The experiment tests whether entropy regularization supports optimal exploration scheduling and faster convergence to the Nash equilibrium. It presents a discrete-time policy-gradient approach and evaluates stability, convergence speed, and learned equilibrium accuracy.

4.1 Set-up

  • The discrete-time setup focuses on a smaller policy class characterized by the mean-state process, with step size δ=T/N and initial-state distribution ν.The experiment uses Shannon entropy only; the authors note that adding cross-entropy could be studied similarly.Source: §4 · p. 21, §4.1 · p. 22

4.2 Mean Field Policy Gradient with Exploration

  • The learning setting assumes agents do not know model parameters and instead have access to a simulated reward function.Source: §4.2 · p. 22
  • The policy-gradient algorithm alternates agent updates under the previous iteration's mean field with a population mean-field update after agents improve their policies.Each agent update takes I gradient-descent rounds using n simulated reward samples for perturbed policies; the gradient is estimated by a bounded, biased zeroth-order method.Source: §4.2 · pp. 22–23

4.3 Results

  • With Shannon entropy, runs are stable and converge; without it, the algorithm is unstable and does not converge, while λSE=3 reaches equilibrium in three iterations versus five for λSE=1.Figure 3 compares relative errors across λSE; Figures 1 and 2 show performance for λSE=1 and λSE=3.Source: Figure 3 · p. 24, Results · pp. 24–25, Figure 1 · p. 24
  • For λSE=1 and λSE=3, agents learn the decreasing time-dependent exploration variance with error below 5%, and the learned mean-field information also has error below 5%.The figures report learned cM values of 0.732 and 0.736 against true M=0.75 for λSE=1.0 and λSE=3.0, respectively.Source: Figure 1 · p. 24, Figure 2 · p. 24, Results · p. 25

Related papers

Loading 2010.00145v2…