PaperlayerView exact revision on arXiv

Source-linked AI summary

EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu

arXiv:2007.11091v2cs.LGstat.ML

TL;DR

Offline RL must learn from fixed data while avoiding unsupported actions that can distort value estimates. This paper introduces EMaQ, a BCQ simplification whose operator accounts for behavior-policy support, proposal distribution, and sample count. EMaQ matches or outperforms prior state-of-the-art on D4RL benchmarks, while remaining competitive with SAC online and using fewer moving parts.

  • Problem

    Offline RL methods can underperform because value estimation and policy updates encounter action distribution shift when learned actions fall outside the fixed dataset.

  • Method

    EMaQ simplifies BCQ by removing its perturbation network and introducing a backup operator that incorporates behavior-policy support, proposal distribution, and sample count.

  • Results

    EMaQ matches and outperforms prior state-of-the-art on D4RL benchmarks and is competitive with SAC online.

  • Takeaways & Limitations

    EMaQ provides a simple, interpretable offline RL baseline with fewer moving parts and highlights the importance of careful behavior-policy generative modeling.

  • Takeaways & Limitations

    The tabular sub-optimality result assumes inf_s µ*(s) > 0 and bounded rewards.

Abstract

from arXiv · show

Off-policy reinforcement learning holds the promise of sample-efficient learning of decision-making policies by leveraging past experience. However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform. Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions. In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy. Importantly, in contrast to their original theoretical considerations, we derive this simplified algorithm through the introduction of a novel backup operator, Expected-Max Q-Learning (EMaQ), which is more closely related to the resulting practical algorithm. Specifically, in addition to the distribution support, EMaQ explicitly considers the number of samples and the proposal distribution, allowing us to derive new sub-optimality bounds which can serve as a novel measure of complexity for offline RL problems. In the offline RL setting -- the main focus of this work -- EMaQ matches and outperforms prior state-of-the-art in the D4RL benchmarks. In the online RL setting, we demonstrate that EMaQ is competitive with Soft Actor Critic. The key contributions of our empirical findings are demonstrating the importance of careful generative model design for estimating behavior policies, and an intuitive notion of complexity for offline RL problems. With its simple interpretation and fewer moving parts, such as no explicit function approximator representing the policy, EMaQ serves as a strong yet easy to implement baseline for future work.

1. Introduction

Offline RL reuses fixed interaction data but must prevent learned policies from drifting into unsupported actions. EMaQ simplifies BCQ with an operator that models behavior-policy support, sample count, and proposal distribution while achieving strong benchmark results.

  • 1. Introduction: Offline RL methods constrain learned policies near the dataset because out-of-distribution actions can produce erroneous values and uncovered states.Actor-critic updates can amplify this error by repeatedly selecting actions assigned falsely high values.
  • 1. Introduction: The paper simplifies BCQ by removing a heuristic while keeping extracted policies exactly within the behavior policy’s support.The resulting EMaQ operator explicitly accounts for the proposal distribution and number of samples, yielding sub-optimality bounds and a complexity notion.
  • 1. Introduction: EMaQ uses no explicit policy function approximator, leaving an estimated behavior policy and Q functions as its two standard practical components.This removes one fitted function approximator relative to prior approaches.
  • 1. Introduction: EMaQ matches and outperforms prior state-of-the-art on D4RL offline RL benchmarks and is competitive with SAC online.It surpasses SAC in the deployment-efficient setting.
  • 1. Introduction: The paper identifies behavior-policy modeling as important and presents EMaQ as a simple, interpretable baseline for future offline RL work.Across benchmark settings, surprisingly little modification to a base behavior policy was needed for a performant policy.

2. Background

Offline RL faces action distribution shift because value targets may rely on actions absent from the dataset. Existing approaches constrain policies toward the behavior distribution, including BCQ’s sampled-and-perturbed action construction.

  • 2. Background: Offline RL’s central background challenge is action distribution shift: evaluating unsupported actions can corrupt Q-values and trigger hazardous policy updates.State distribution shift is absent during fixed-dataset training, but action distribution shift remains significant.
  • 2. Background: Constraining learned policies near the behavior policy aims to keep state-action visitation close to the offline data distribution.The cited formulation expresses this goal as dπ(s, a) ≈ dµ(s, a).
  • 2. Background: BCQ constrains continuous-control Q-learning by sampling actions from an estimated behavior policy and perturbing them with a bounded network.The perturbation bound is controlled by Φ, intended to keep modified actions near dataset actions.

3. Expected-Max Q-Learning

EMaQ simplifies BCQ by sampling actions from an estimated behavior policy and selecting high-value actions, while its operator interpolates between behavior-policy evaluation and support-constrained optimal control as sample count increases. Theoretical results establish contraction, fixed-point convergence, monotonic policy improvement with N, and sub-optimality bounds tied to behavior-policy coverage and problem difficulty.

  • 3.2. Dynamic Programming Properties in the Tabular MDP Setting: Repeated EMaQ updates converge because the operator is an L∞ contraction with a unique fixed point.This property holds for every N in the tabular setting.
  • 3.2. Dynamic Programming Properties in the Tabular MDP Setting: The EMaQ operator interpolates between evaluating µ and learning the support-constrained optimal Q-function as N increases.With full action support, this interpolates between standard Q-evaluation and Q-learning.
  • 3.2. Dynamic Programming Properties in the Tabular MDP Setting: For N > M, the induced policy using N samples is at least as good as the policy using M samples on actions supported by µ.The result formalizes monotonic improvement with sample count within the behavior-policy support.
  • 3.3. Sub-optimality Bounds: EMaQ’s sub-optimality bounds depend on behavior-policy mass over optimal supported actions and can define an intuitive offline-RL difficulty measure.The proposed measure tracks how the bound decreases with N, while experiments suggest the effective N may be surprisingly small.
  • 3.4. Offline RL Setting with Function Approximators: The practical method first fits a generative behavior model, then trains an ensemble of Q-functions whose targets use the best of N sampled actions.This design makes behavior-model quality central to the resulting value estimates and policy.
  • 4. Offline RL Setting with Function Approximators: EMaQ samples N actions from µ(a|s) and chooses the action with maximum ensemble Q-value without training an explicit policy network.The same sampled-action mechanism forms both training targets and the test-time implicit policy.

4. Related Work

EMaQ belongs to offline-RL methods that constrain learned behavior to the data distribution, but it avoids an additional proposal-modifying policy or reward modification. The paper also positions EMaQ as a new backup-operator family with empirical validation for offline RL with function approximators.

  • Offline RL: Unlike BCQ, BEAR, and related methods, EMaQ constrains actions through behavior-policy sampling without an additional proposal policy or reward modification.Prior methods use perturbation, divergence penalties, or reward changes to keep learned policies near the data distribution.
  • Offline RL: Figure 1 evaluates EMaQ across standard D4RL Mujoco domains and dataset types while varying the action-sample count N.The figure also provides base-policy values and BEAR reference lines for comparison.
  • Modified Backup Operators: EMaQ adds a backup-operator family to reinforcement-learning literature and validates its simple operator modification empirically in offline RL with function approximation.The related-work discussion connects EMaQ to prior modified backup operators and approximate action-search methods.

5. Experiments

Experiments show that EMaQ is competitive with or outperforms prior offline RL methods on D4RL, while its performance depends strongly on the behavior-policy generative model and sample count.

  • 5. Experiments: Online EMaQ is competitive with SAC and surpasses SAC in the deployment-efficient setting, although online results are secondary to the offline evaluation.The implementation uses the Wu et al. codebase and D4RL datasets for benchmarking.
  • 5.1. Practical Effect of N and the Choice of Generative Model: With an autoregressive behavior model, EMaQ’s constrained backup matches and often exceeds prior algorithmic choices, including reported BEAR performance.Figure 2 compares EMaQ, BCQ, and BEAR across D4RL domains while varying deviation-related hyperparameters for BCQ and BEAR.
  • 5.1. Practical Effect of N and the Choice of Generative Model: Replacing the autoregressive model with a VAE significantly reduces EMaQ performance, often below reported BEAR results and without a monotonic trend in N.The two behavior estimates have almost identical standalone evaluation results, indicating that model choice matters specifically for EMaQ’s performance.
  • 5.2. Comparison on D4RL Offline RL Benchmark: EMaQ matches and, on Table 1, outperforms prior state-of-the-art methods on D4RL benchmark tasks.Many D4RL domains remain effectively unsolved by all considered algorithms.
  • 5.1. Practical Effect of N and the Choice of Generative Model: N = 5 significantly improves upon the behavior estimate in almost all standard MuJoCo settings, often matching or exceeding previously reported results.In HalfCheetah-Random, selecting the best of five sampled actions raises return from 0 for the random policy to 2000.
  • 5.1. Practical Effect of N and the Choice of Generative Model: The experiments suggest that EMaQ’s complexity measure depends on the number of sampled actions and the quality of the behavior-policy estimate.The authors connect this empirical observation to future theoretical investigation of Δ(s, N).

6. Conclusion

The paper concludes that EMaQ simplifies BCQ by removing its perturbation network while retaining tractable theoretical properties and strong online and offline RL performance. Its theory explicitly incorporates the behavior distribution and sample count, under stated support assumptions.

  • 6. Conclusion: EMaQ removes BCQ’s heuristic perturbation network and one function approximator while matching or outperforming prior state-of-the-art in online and offline RL.The authors position the method as a simple, tractable foundation for future offline RL work.
  • 6. Conclusion: The EMaQ theory explicitly incorporates the proposal distribution μ(a|s) and the number of action samples N.This setup is intended to more closely match the practical algorithm and supports new notions of offline RL complexity.
  • 6. Conclusion: The proofs assume that μ(a|s) has full support; otherwise, they are transferred to an MDP restricted to μ-reachable states and supported actions.The restricted MDP uses Sμ for reachable states and Aμ for actions in the behavior-policy support.
  • 6. Conclusion: In tabular MDPs, repeated application of the EMaQ operator converges to a unique fixed point because the operator is a contraction in the L∞ norm.The contraction result is stated for any N ∈ N.
  • 6. Conclusion: For N > M, the resulting Q-value function is at least as good as the one obtained with M samples for actions within the support of μ(·|s).The comparison is stated for all states and supported actions.

A.4. Bounds

The appendix develops bounds and implementation details for EMaQ, including its behavior-policy proposals, action sampling, and online adaptation.

  • Implementation: The autoregressive behavior model generates actions sequentially from state embeddings and preceding action components, trained with cross-entropy.Separate MLPs predict discretized action dimensions, and sampling proceeds one index at a time.
  • Implementation: The appendix describes EMaQ training with a pretrained behavior model, Q-function ensembles, target networks, and sampled action-value targets.The algorithm uses an offline dataset and repeatedly samples actions from the behavior estimate for target construction.
  • Proposal distribution: The proposed second proposal distribution was tested as a mixture with the behavior model but produced no noticeable gains in the reported experiments.The authors suggest this may reflect the relative simplicity of the Mujoco domains.
  • Limitation: Autoregressive models made EMaQ slower to train because the method requires many generated samples.The authors suggest better generative models and engineering could address this cost.
  • Online RL: EMaQ can adapt to online RL by smoothing the proposal distribution, sampling boundary actions more often, and selecting the highest-Q exploratory action.The online variant uses a temperature τ > 1 and samples N actions before choosing with Q_explore.

G.1. Comparing Offline RL Methods

This section specifies the offline-RL comparison setup, including BCQ and BEAR hyperparameters, deployment-efficient evaluation, and implementation settings.

  • Comparison setup: BCQ and BEAR permit greater policy deviation from the behavior estimate as their respective clipping or divergence hyperparameters increase.The evaluated BCQ clipping factors are Φ ∈ {0.005, 0.015, 0.05, 0.15, 0.5}, while BEAR uses ϵ ∈ {0.015, 0.05, 0.15, 0.5, 1.5}.
  • Comparison setup: Deployment-efficient evaluation concerns reducing the number of distinct policies executed in the environment, which can benefit constrained domains.The section defines deployment-efficient settings in terms of fewer different policies being run.
  • Caveat: The online EMaQ variant has more hyperparameters to tune, whereas ensemble SAC adds only the ensemble-mixing parameter λ.The authors note that relative performance depends on EMaQ hyperparameter choices.

H.1. Implementation

The implementation evaluates VAE-based behavior models and compares them with the autoregressive models used elsewhere in the study.

  • Implementation: The VAE uses matched encoder and decoder capacities, a latent space twice the action dimension, and a tanh-transformed decoder output.The architecture is designed to approximate the parameter count of the autoregressive models.
  • Results: EMaQ has a harder time improving upon the behavior estimate when the behavior model uses the described VAE architecture.The accompanying comparisons examine autoregressive and VAE results side by side.

I. EMaQ Medium-Expert Setting Results

In the medium-expert setting, EMaQ’s training behavior depends on the backup design, sample count, and generative-model choices, with large N creating convergence challenges.

  • EMaQ Medium-Expert Setting Results: Large N values slowed HalfCheetah convergence, although the curves continued improving and were not trained long enough to converge.
  • Comparison with Softmax Backup Operators: The soft backup adds an additional hyperparameter and, in the limited HalfCheetah settings considered, did not provide a practical advantage over regular EMaQ.
  • Comparison with Softmax Backup Operators: Soft backups approach EMaQ as α increases, but matching EMaQ may require large α values and offered limited mitigation in the medium-expert setting.

K. Qualitative Differences in Training Curves

Training-curve differences were especially pronounced in the antmaze-umaze and antmaze-umaze-diverse domains, where EMaQ appeared more stable than competing methods.

  • Qualitative Differences in Training Curves: EMaQ training curves were sometimes significantly more stable than those of BEAR and BCQ.
  • Qualitative Differences in Training Curves: The antmaze-umaze and antmaze-umaze-diverse domains showed particularly striking differences in training behavior.
  • Qualitative Differences in Training Curves: Figure 4 reports performance throughout training across multiple hyperparameters and random seeds for these small antmaze domains.

L. Larger Plots for Visibility

The appendix provides larger visualizations of EMaQ evaluations, ablations, model comparisons, and training-return stability across benchmark domains.

  • Larger Plots for Visibility: Table 4 compares agent returns throughout training across hyperparameters and random seeds in small antmaze domains, reporting greater EMaQ stability than BCQ.
  • Larger Plots for Visibility: Figure 11 evaluates EMaQ on D4RL domains across N values from 5 to 400 and compares results with base behavior policies and reported BEAR performance.
  • Larger Plots for Visibility: Figure 12 compares EMaQ, BCQ, and BEAR on D4RL domains while varying the allowed deviation from the behavior-policy estimate.
Loading 2007.11091v2…