PaperlayerView exact revision on arXiv

Source-linked AI summary

Extreme Q-Learning: MaxEnt RL without Entropy

Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon

arXiv:2301.02328v2cs.LGcs.AIcs.RO

TL;DR

Continuous-action RL needs maximal Q-values for Bellman backups, but policy-based estimates can be difficult to compute and problematic for offline data. Extreme Q-Learning uses Gumbel modeling to estimate optimal soft-values directly, and the paper reports strong offline benchmark results and moderate online gains over SAC and TD3.

  • Problem

    In continuous action spaces, maximal Q-values and the MaxEnt log-partition needed for optimal Bellman backups are difficult to compute, while offline policy-sampled actions can be out of distribution.

    Source: §1 · p. 1

  • Method

    Extreme Q-Learning uses Gumbel-based maximum-likelihood estimation to fit optimal soft-values and Bellman backups without requiring policy sampling or explicit policy entropy.

    Source: §1 · p. 2

  • Results

    The method improves over prior methods by a large margin on D4RL Franka Kitchen tasks and performs moderately better than SAC and TD3 in online RL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework provides simple, performant online and offline RL algorithms while avoiding explicit access to a policy for the optimal Bellman backup.

    Source: §1 · p. 2, §6 · p. 10

    Shared parameters can correlate errors across states and actions, potentially weakening the Gumbel character of the errors.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.

1 INTRODUCTION

Extreme Q-Learning uses Gumbel error modeling to estimate optimal soft-values directly, without policy sampling or explicit entropy access. The paper reports strong offline D4RL results and moderate online improvements over SAC and TD3.

1 INTRODUCTION

  • Computing max Q for Bellman targets remains difficult in continuous control, and policy-based estimates can use out-of-distribution actions in offline RL.Many off-policy methods estimate the current policy's Q-function rather than the optimal Q-function.Source: §1 · p. 1
  • In MaxEnt RL, the optimal Bellman backup requires a log-partition over Q-values that is usually intractable in high-dimensional action spaces.Methods such as SAC use auxiliary policy networks and therefore do not estimate the optimal Bellman backup directly.Source: §1 · p. 1
  • The framework uses Gumbel error modeling to estimate LogSumExp in continuous spaces and derive a maximum-likelihood RL loss.The motivation draws on extreme-value results and the connection between Gumbel-distributed utility errors and soft-optimal choice.Source: §1 · p. 2
  • The formulation extends soft-Q learning to offline and continuous-action settings, estimating optimal soft-values and Bellman updates without policy entropies.It also links soft and conservative Q-learning, and frames MaxEnt RL as conservatism within the framework.Source: §1 · p. 2
  • Offline results improve over prior methods by a large margin on Franka Kitchen, while online results are moderately better than SAC and TD3.The framework is also described as theoretically avoiding actor-critic formulations.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define the MDP and offline-data setting, then introduce entropy-regularized soft values and the extreme-value properties used to motivate the framework.

2 PRELIMINARIES

  • An infinite-horizon MDP is specified by its state and action spaces, dynamics, rewards, and discount factor; offline RL uses a fixed behavior-policy dataset without further environment interaction.The notation also distinguishes the state distribution induced by a policy.Source: §2 · p. 2

2.1 MAXIMUM ENTROPY RL

  • The generalized MaxEnt objective adds a KL term between the policy and a reference distribution; offline learning uses the behavior policy as that reference to keep the learned policy close to it.A uniform reference recovers the standard online MaxEnt objective up to a constant.Source: §2.1 · pp. 2–3
  • The optimal soft value is the log-sum-exp of Q-values, and the soft-Bellman operator uses it to define the optimal soft-Q function.The approach seeks to fit these optimal soft-values directly.Source: §2.1 · p. 3

2.2 EXTREME VALUE THEOREM

  • Under exponential-tailed i.i.d. samples, maxima converge asymptotically to a Gumbel distribution, which is max-stable.The paper uses this property to motivate Gumbel-distributed errors through chains of max operations.Source: §2.2 · p. 3
  • Adding i.i.d. Gumbel noise to a discrete set makes its maximum Gumbel-distributed with a location set by LogSumExp, while the maximizing index follows a softmax distribution.The passage identifies this as the Gumbel-Max Trick.Source: §2.2 · p. 3
  • In the McFadden-Rust model, additive i.i.d. Gumbel reward noise that is conditionally independent across state-action pairs yields soft-Bellman equations over observed rewards.The stated converse connects softmax policies and these noise conditions to Gumbel-distributed reward noise.Source: §2.2 · p. 3

3 EXTREME Q-LEARNING

Extreme Q-learning uses Gumbel error modeling to estimate soft-optimal values directly, deriving an entropy-free MaxEnt Bellman update and practical online and offline algorithms.

3.1 GUMBEL ERROR MODEL

  • Bellman errors are skewed and better fit by Gumbel than Gaussian distributions; repeated maximization propagates errors toward a max-stable Gumbel distribution in the limit.The authors note that shared parameters can correlate errors across states and actions, weakening the Gumbel pattern in practice.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 GUMBEL REGRESSION

  • Gumbel regression derives a likelihood-based objective whose minimizer estimates the LogSumExp over sampled Q-values.Its optimum is β log E[exp(x_i/β)], with the empirical expectation replacing the sum.Source: §3.2 · pp. 4–5

3.2.1 THEORY

  • The operator Lβ interpolates between expectation and maximum, and the finite-sample empirical loss consistently estimates the log-partition function.The empirical loss can be minimized with stochastic gradient descent.Source: §3.2.1 · p. 5

3.3 MAXENT RL WITHOUT ENTROPY

  • Gumbel regression estimates soft-optimal values so the MaxEnt Bellman backup can be solved without explicitly using policy entropy.The framework substitutes Gumbel-regressed values into the optimal Bellman backup.Source: §3.3 · p. 6

3.4 LEARNING POLICIES

  • Forward-KL policy fitting recovers the offline AWR objective without action sampling, while reverse KL gives a SAC-like actor update and can impose a trust region relative to the prior policy.The trust-region connection applies when the reference distribution is the previous policy.Source: §3.4 · pp. 6–7

3.5 PRACTICAL ALGORITHMS

  • For stochastic dynamics, X-QL separates value fitting from unbiased Q estimation; offline learning uses behavior data, while online backups sample actions from a previous policy.The value network fits optimal soft-values, and Q is trained toward the expectation of single-sample targets.Source: §3.5 · p. 7

4 EXPERIMENTS

The experiments evaluate Extreme Q-Learning in offline D4RL tasks and online DM Control, with strong offline results and modest online gains over SAC and TD3.

4 EXPERIMENTS

  • The evaluation compares X-QL with state-of-the-art algorithms across continuous-control tasks in both online and offline settings.The authors note that the exponential Gumbel regression creates optimization challenges.Source: §4 · p. 8

4.1 OFFLINE RL

  • X-QL reaches state-of-the-art on Franka Kitchen and X-QL-C surpasses prior work on five of eight Adroit tasks.The reported offline results also describe consistently high performance across environments and fast convergence on some tasks.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 ONLINE RL

  • Across DM Control tasks, ExtremeQ variants match or surpass vanilla SAC and TD3, with particularly large Hopper gains and modest overall online improvements.X-QL-DQ, the X-TD3 variant without Double-Q, beats standard TD3 in all environments except Cheetah Run.Source: §4.2 · pp. 8–9, Figure 3 · p. 9

5 RELATED WORK

The related-work discussion positions Extreme Q-Learning against MaxEnt and off-policy RL methods, as well as offline methods based on constraints, regularization, or behavior policies.

Online RL

  • Unlike prior MaxEnt methods, Extreme Q-Learning models optimal soft-values with a Gumbel loss rather than requiring explicit entropy estimation or a policy network.The authors link their objective to KL divergence and distinguish it from most off-policy methods by directly modeling the optimal Bellman backup.Source: §5 · p. 9

Offline RL

  • Offline methods commonly constrain or regularize Q-learning or extract greedy policies from behavior policies; IQL fits behavior-policy Q-function expectiles.The passage contrasts IQL's approach with CQL's conservative lower-bound motivation.Source: §5 · p. 9
  • Like IQL, Extreme Q-Learning avoids evaluating Q on out-of-distribution actions; like CQL, it is conservative, using a behavior-policy KL constraint.It extends soft-Q learning offline without a policy or explicit entropy values, and, unlike BRAC, avoids learning a policy during training.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents Extreme Q-Learning as a policy-independent method for estimating the optimal Bellman backup. The appendix relates soft-Bellman optimality to Gumbel reward noise under specified hidden-state assumptions.

6 CONCLUSION

  • Extreme Q-Learning directly estimates the optimal Bellman backup without explicit policy access and bridges regular, soft, and conservative Q-learning formulations.The authors identify stability with the exponential Gumbel loss and automatic tuning of β as future directions.Source: §6 · p. 10

A THE GUMBEL ERROR MODEL FOR MDPS

A.1 RUST-MCFADDEN MODEL OF MDPS

  • Under additive separability and conditional independence, i.i.d. Gumbel reward noise in a hidden-state MDP recovers the soft-Bellman equations.The passage presents this as an equivalence between a MaxEnt soft-MDP and an MDP with hidden state introducing Gumbel reward noise.Source: §A.1 · p. 13
  • The appendix relates the soft-Bellman optimality equation and its optimal policy to both entropic regularization and the Gumbel-error view.Conversely, a softmax policy implies i.i.d. Gumbel reward noise under the stated AS+CI conditions.Source: §A.1 · p. 13

A.2 GUMBEL ERROR MODEL (GEM) FOR MDPS

This section connects functional-approximation errors to Gumbel noise and soft-optimal policies, while noting that maximization over noisy estimates can create overestimation bias.

A.2 GUMBEL ERROR MODEL (GEM) FOR MDPS

  • Taking the maximum of noisy Q estimates generally exceeds the true maximum, allowing even initially zero-mean errors to propagate overestimation bias through Bellman updates.Source: §A.2 · p. 14
  • Treating functional-approximation parameters as hidden state makes the errors reward noise; under the model's conditions, a soft-optimal policy implies Gumbel-distributed errors.Source: §A.2 · p. 14
  • Conversely, i.i.d. Gumbel errors yield soft-Bellman equations and a softmax policy, making Gumbel errors an optimality condition under functional bootstrapping.Source: §A.2 · p. 14

A.2.1 TIME EVOLUTION OF ERRORS IN MDPS UNDER DETERMINISTIC DYNAMICS

Under deterministic dynamics, a Gumbel model of Q-value distributions evolves through Bellman updates toward the optimal soft-Bellman solution while its scale contracts. The appendix also bounds LogSumExp overestimation and notes that error characterization under stochastic dynamics remains nontrivial.

A.2.1 TIME EVOLUTION OF ERRORS IN MDPS UNDER DETERMINISTIC DYNAMICS

  • The Q-distribution remains Gumbel under deterministic Bellman evolution, with location following the optimal soft-Bellman equation and scale shrinking by γ each step.The distribution eventually collapses to a delta distribution over the unique contraction Q*; propagated Gumbel errors decay at rate γ.Source: §A.2.1 · p. 15
  • Under stochastic dynamics, error characterization is nontrivial because Gumbel distributions are not mean-stable; a Gumbel-Gaussian mixture is left as a future direction.Source: §A.2.1 · p. 15

B GUMBEL REGRESSION; B.1 OVERESTIMATION BIAS

  • The Gumbel Regression analysis bounds both LogSumExp bias and finite-sample estimation error; for unbiased bounded Q estimates, expected LogSumExp lies above V(s) but has a stated upper bound.The upper bound is the action-average Q plus β log cosh(Qmax/β).Source: §B.1 · pp. 15–16, B GUMBEL REGRESSION · p. 15

B.2 PAC LEARNING BOUNDS FOR GUMBEL REGRESSION

The appendix establishes finite-sample properties for Gumbel Regression and describes deterministic-dynamics variants of Extreme Q-Learning. These variants use fixed-point Bellman objectives to estimate values or Q-functions without the stochastic-dynamics B* estimation step.

B.2 PAC LEARNING BOUNDS FOR GUMBEL REGRESSION

  • With finite samples, the Gumbel Regression estimate is consistent for the log-partition function, and its exponentiated form is an unbiased partition-function estimator with a concentration bound that tightens as β increases.The bound is reported to tighten with increasing β.Source: §B.2 · p. 16

C EXTREME Q-LEARNING

  • Under deterministic dynamics, X-QL avoids estimating B* because no next-state expectation is needed, yielding simpler Bellman equations and value-iteration or Q-iteration objectives.Source: §C.1 · p. 17
  • The deterministic X-TD3 objective directly estimates Qθ without learning values.β is a hyperparameter and may be rescaled as β′ = γβ.Source: §C.1 · p. 17
  • Optimizing the deterministic objective by fixed point is equivalent to Q-iteration with the Bellman operator.The value-iteration approach is described as converging to optimal V(s) up to sampling error.Source: §C.1 · pp. 17–18

C.2 BRIDGING SOFT AND CONSERVATIVE Q-LEARNING

The appendix relates X-QL’s conservatism to KL regularization and contrasts it with CQL’s χ2 regularization. Additional experiments illustrate value fitting, Bellman-error distributions, and the numerical stabilization needed for training.

C.2 BRIDGING SOFT AND CONSERVATIVE Q-LEARNING

  • X-QL’s conservatism corresponds to KL regularization, while CQL’s conservatism is described as χ2 regularization; the paper characterizes X-QL’s update as a reverse-KL soft-Bellman update.Source: §C.2 · pp. 18–19
  • The reverse-KL objective is described as mode-seeking: X-QL fits the behavior policy’s mode and is more robust to random dataset actions, whereas CQL can be affected by noisy actions.Source: §C.2 · p. 18

D.1 A TOY EXAMPLE

  • In the toy maze, MSE produces poor early policy behavior and failed value propagation, while increasing β makes learned values better approximate the optimal max-Q policy.Source: Figure 4 · p. 19

D.2 BELLMAN ERROR PLOTS

  • For SAC errors, Gumbel fits Cheetah and Walker better than Hopper, yet the approach still improves performance in Hopper.Source: Figure 5 · p. 19

D.3 NUMERIC STABILITY

  • The Gumbel loss can be numerically unstable from its exponential term; the implementation uses max-normalization, gradient scaling, and input clipping.Source: §D.3 · p. 20

D.4 OFFLINE EXPERIMENTS

Offline experiments report fast X-QL convergence, especially when tuned, but also identify numerical stability as a concern. The untuned method diverges on Antmaze, and some runs with β < 1 could diverge; Layer Normalization helped mitigate this issue.

D.4 OFFLINE EXPERIMENTS

  • X-QL exhibits extremely fast offline convergence, particularly when tuned, while the untuned version diverges on Antmaze.Figure 7 plots D4RL returns against training iterations, averaged over six seeds, using batch size 1024 for each method.Source: §D.4 · p. 20, Figure 7 · p. 21
  • The implementation follows IQL’s network architecture, Double-Q trick, and data preprocessing, while tuning β and gradient clipping per environment.Source: §D.4 · p. 20
  • Offline AntMaze runs could occasionally diverge for β < 1, and adding Layer Normalization to value networks helped mitigate the issue.The experiments report no stability issues in MuJoCo locomotion environments.Source: §D.4 · p. 21

D.5 OFFLINE ABLATIONS

The offline ablations examine temperature and batch-size sensitivity, while the online-experiment appendix describes training settings, instability, and return curves across D4RL benchmarks.

D.5 OFFLINE ABLATIONS

  • Temperature β controls KL penalization: small values suit noisy datasets, while high values favor expert-like datasets; extremes can hurt performance or destabilize training.The ablation notes that environments differ in their sensitivity to β.Source: §D.5 · p. 21, Figure 8 · p. 21
  • Larger batches tended to stabilize Gumbel-regression training and sometimes improved final performance; a fixed batch size of 1024 was also compared with IQL to test whether batch size explained X-QL's advantage.The implementation normalizes gradients at the batch level.Source: §D.5 · p. 21
  • Increasing value updates per Q update reduces value-update variance on some hard environments, and variance-reduction tricks help stabilize tuned offline X-QL on some environments.Table 4 reports untuned hyperparameters first and tuned values in parentheses.Source: Table 4 · p. 22

D.6 ONLINE EXPERIMENTS

  • Online X-variants sometimes became unstable from outliers in collected policy rollouts, especially in Hopper and Quadruped; about one in six runs destabilized after roughly 100k gradient steps, and unstable runs were restarted.The reported instability was rarer for Cheetah and Walker.Source: §D.6 · p. 22
  • The appendix plots D4RL returns over training iterations averaged across six seeds, comparing per-environment tuned X-QL with a shared-hyperparameter version; the shared settings did best on some Androit environments.The plots cover Mujoco, AntMaze, Franka Kitchen, and Androit.Source: Figure 10 · p. 23, Figure 11 · p. 23, Figure 13 · p. 24

Related papers

Loading 2301.02328v2…