PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song

arXiv:2110.01548v2cs.LGcs.AI

TL;DR

Offline RL must control erroneous Q-values for OOD actions without relying on difficult behavior-policy estimation or OOD sampling. This paper uses ensemble uncertainty through clipped Q-learning and diversifies the ensemble to reduce its size. EDAC achieves state-of-the-art performance across the considered D4RL benchmarks while using substantially fewer networks.

  • Problem

    Offline RL faces extrapolation errors from OOD state-action pairs, while prior constraints or penalties can require behavior-policy estimation or OOD sampling.

  • Method

    The paper uses clipped Q-learning with Q-function ensembles to penalize uncertain predictions and introduces ensemble diversification to reduce the required ensemble size.

  • Results

    EDAC achieves state-of-the-art performance on various D4RL datasets while requiring up to 90% fewer ensemble networks than the vanilla Q-ensemble.

  • Takeaways & Limitations

    Uncertainty-aware clipping can penalize OOD actions according to prediction confidence without explicit data-distribution estimation or OOD sampling.

  • Takeaways & Limitations

    For Adroit tasks, the authors could not completely reproduce the reported CQL results, and some SAC-N performance falls behind on small datasets such as pen-human.

Abstract

from arXiv · show

Offline reinforcement learning (offline RL), which aims to find an optimal policy from a previously collected static dataset, bears algorithmic difficulties due to function approximation errors from out-of-distribution (OOD) data points. To this end, offline RL algorithms adopt either a constraint or a penalty term that explicitly guides the policy to stay close to the given dataset. However, prior methods typically require accurate estimation of the behavior policy or sampling from OOD data points, which themselves can be a non-trivial problem. Moreover, these methods under-utilize the generalization ability of deep neural networks and often fall into suboptimal solutions too close to the given dataset. In this work, we propose an uncertainty-based offline RL method that takes into account the confidence of the Q-value prediction and does not require any estimation or sampling of the data distribution. We show that the clipped Q-learning, a technique widely used in online RL, can be leveraged to successfully penalize OOD data points with high prediction uncertainties. Surprisingly, we find that it is possible to substantially outperform existing offline RL methods on various tasks by simply increasing the number of Q-networks along with the clipped Q-learning. Based on this observation, we propose an ensemble-diversified actor-critic algorithm that reduces the number of required ensemble networks down to a tenth compared to the naive ensemble while achieving state-of-the-art performance on most of the D4RL benchmarks considered.

1 Introduction

Offline RL learns from fixed datasets, avoiding costly or unsafe environment interaction, but faces extrapolation errors on insufficiently covered or OOD state-action pairs. The paper proposes uncertainty-based penalization that avoids behavior-policy estimation and OOD sampling while preserving confident actions.

  • Offline RL learns policies from previously collected data without further environment interaction, avoiding potentially prohibitive exploratory actions.The motivation includes applications such as autonomous driving and healthcare, where exploration can cause critical damage.
  • Insufficient dataset coverage causes vanilla offline RL to overestimate Q-values for OOD state-action pairs.
  • Prior methods constrain policies toward the behavior policy or penalize OOD Q-values, but may require behavior-policy estimation or OOD sampling.
  • The proposed method quantifies Q-value uncertainty with an ensemble and uses clipped Q-learning without estimating or sampling the data distribution.
  • The ensemble-diversified method reduces required ensemble size while achieving state-of-the-art performance across various offline RL tasks.

2 Preliminaries

The preliminaries formulate offline actor-critic learning over a static dataset and identify the central failure mode: policy improvement can select OOD actions with erroneously high Q-values. Existing regularization addresses this by biasing policy or Q-function estimates toward the dataset, with corresponding estimation or sampling requirements.

  • The environment is modeled as an MDP with state and action spaces, transitions, rewards, an initial-state distribution, and a discount factor.
  • Q-learning represents expected cumulative discounted reward with a neural-network state-action value function.
  • Offline actor-critic learning samples transitions from a static dataset and updates the policy to maximize expected Q-values.
  • Policy improvement can favor OOD actions with erroneously high Q-values, whose errors cannot be corrected through online environmental feedback.
  • Offline RL methods regularize either the policy or Q-function toward the dataset, while policy regularization typically requires accurate behavior-policy estimation and CQL requires suitable action-distribution sampling.

3 Uncertainty penalization with Q-ensemble

Clipped Q-learning can act as uncertainty-aware pessimism: taking the ensemble minimum penalizes high-variance predictions, especially for OOD actions. Increasing ensemble size strengthens this relative penalty and enables SAC-N to outperform CQL on the reported D4RL experiments.

  • Uncertainty interpretation: Clipped Double Q-learning uses the minimum of parallel Q-networks as the Bellman target.
  • SAC-N evaluation: SAC-N increases SAC’s Q-ensemble from 2 networks to N and outperforms CQL by a large margin on the reported halfcheetah-medium and hopper-medium experiments.
  • Uncertainty interpretation: The clipping procedure favors dataset-seen actions because Bellman-residual training aligns their Q-predictions and yields lower variance than OOD actions.
  • Uncertainty interpretation: Under the ensemble interpretation, the clipped value penalizes the ensemble mean by a standard-deviation term whose coefficient depends on N.
  • Empirical uncertainty: Clipping penalizes OOD random state-action pairs more strongly than in-distribution behavior pairs, whose Q-value estimates have lower variance.
  • Empirical uncertainty: Increasing N further separates OOD and in-distribution penalties by strengthening the relative penalty on OOD samples.

4 Ensemble gradient diversification

Ensemble gradient diversification addresses weak OOD penalization caused by aligned Q-function input gradients. EDAC increases ensemble variance by diversifying these gradients, using a tractable objective related to maximizing the smallest variance eigenvalue.

  • Aligned Q-function input gradients can create low-variance OOD directions, weakening ensemble penalization and requiring many Q-networks.The smallest-eigenvalue direction bounds OOD Q-value variance when gradients are well aligned.
  • Gradient diversification increases Q-value variance for near-distribution OOD actions by maximizing variance through a regularizer.The proposed regularizer targets near-distribution OOD actions, where insufficient variance can reduce uncertainty-based penalization.
  • The method minimizes pairwise cosine similarity among Q-network input gradients on dataset state-actions as a tractable proxy for maximizing total variance.Maximizing total variance avoids the computational burden of directly optimizing the smallest eigenvalue while remaining closely related to it.
  • EDAC is the resulting actor-critic algorithm; setting η=0 reduces it to SAC-N, and additionally setting N=2 reduces it to vanilla SAC.

5 Experiments

Experiments on D4RL MuJoCo Gym and Adroit tasks evaluate EDAC against offline RL baselines, including performance, action distances, ensemble requirements, and computational cost. EDAC achieves strong benchmark performance while using fewer ensembles and running faster than CQL.

  • 5.1 Evaluation on D4RL MuJoCo Gym tasks: EDAC and SAC-N outperform or match the previous state-of-the-art across the considered D4RL Gym tasks.Both methods achieve average normalized scores over 80 and reduce the gap with the online expert by 40% compared to CQL.
  • 5.1 Evaluation on D4RL MuJoCo Gym tasks: EDAC achieves performance marginally better than SAC-N while using fewer Q-networks on hopper tasks.SAC-N requires 200 to 500 Q-networks, whereas EDAC requires fewer than 50.
  • 5.1 Evaluation on D4RL MuJoCo Gym tasks: EDAC and SAC-N select actions from a more diverse range than CQL on the evaluated medium datasets.The comparison uses distances between policy actions and dataset actions, supporting uncertainty-based penalization that does not penalize all OOD actions equally.
  • 5.2 Evaluation on D4RL Adroit tasks: On Adroit pen tasks, EDAC outperforms or matches the previous state-of-the-art, while both EDAC and SAC-N score 75% higher than CQL on pen-cloned.SAC-N falls behind on some datasets such as pen-human, whereas ensemble diversification overcomes this difficulty for the proposed method.
  • 5.3 Computational cost comparison: EDAC runs faster than CQL with comparable memory consumption on hopper-medium-v2.The ensemble inference is embarrassingly parallelizable, and EDAC's gradient diversification term has linear computational complexity.

6 Related Works

Related work addresses offline RL through behavior-policy regularization, Q-function regularization, ensemble methods, and uncertainty estimation. These approaches target distribution shift, overestimation bias, or uncertainty in learned value functions.

  • Model-free offline RL: Behavior-policy regularization methods constrain learned actions toward actions represented by the offline dataset.BCQ uses a generative model and a restricted policy, while BEAR and BRAC stabilize policy learning through related regularization strategies.
  • Model-free offline RL: CQL directly regularizes Q-functions by lowering values for OOD actions and raising values for in-distribution actions.REM instead uses a random convex combination of Q-network ensembles for discrete action spaces.
  • Estimation bias in Q-learning: Q-learning can suffer overestimation bias from the maximum operation in Bellman updates, with bootstrapping potentially amplifying errors.TD3 addresses this issue with a clipped version of Double Q-learning.
  • Uncertainty measures in RL: Ensemble-based uncertainty estimates support exploration, Q-learning, and planning in reinforcement learning.Bootstrapped DQN uses Q-function ensembles for uncertainty, while UCB uses ensemble means and standard deviations to form an upper confidence bound.

7 Conclusion

The paper concludes that clipped Q-learning can provide uncertainty-based penalization in offline RL. EDAC reduces ensemble requirements while maintaining state-of-the-art performance across various datasets.

  • Clipped Q-learning constructs an uncertainty-based offline RL method that outperforms previous methods on various datasets.
  • EDAC reduces the required number of ensemble networks by up to 90% compared to the vanilla Q-ensemble.It does so while respecting each data point's epistemic uncertainty during penalization.
  • EDAC does not require explicit estimation of the data-collecting policy or sampling from OOD data.

A.1 Proofs

The proofs analyze variance and gradient geometry in diversified Q-ensembles. They show that ensemble-gradient diversification promotes an approximately uniform spherical distribution, maximizing the smallest variance eigenvalue.

  • Proof of Proposition 1: The proof represents each input-action gradient as q_j and the ensemble-average gradient as q̄ when analyzing the variance matrix.The total variance equals the matrix trace.
  • Proposition 1: Proposition 1 bounds the variance of Q-values for nearby OOD actions along the minimum-eigenvalue direction under local linearity and coincident Q-values.The bound uses the smallest eigenvalue and its normalized eigenvector for the gradient variance matrix.
  • Asymptotic behavior: As the ensemble size grows, the gradient variance matrix converges to 1/|A|I and its smallest eigenvalue reaches its maximum under the stated variance bound.The total variance is bounded by 1, so the smallest eigenvalue cannot exceed 1/|A|.
  • Gradient geometry: Maximizing total gradient variance is equivalent to minimizing pairwise cosine similarity, distributing gradients uniformly on the unit sphere.This connects ensemble-gradient diversification to isotropic gradient directions.
  • Spherical variance: The variance matrix of a uniform spherical distribution is 1/|A|I, with all eigenvalues equal to 1/|A|.The result follows from symmetry and the unit-norm constraint of the spherical distribution.

B Implementation details

The implementation details specify baseline configurations, ensemble-size sweeps, and method-specific hyperparameter searches for MuJoCo Gym and Adroit experiments.

  • Baselines: SAC uses rlkit with default parameters, except both policy and Q-function networks increase from 2 to 3 layers.This follows the CQL experimental protocol.
  • Baselines: REM is implemented on top of SAC with a modified Bellman residual and simplex-based ensemble mixing.The ensemble size is swept over {2, 5, 10, 20, 50, 100, 200, 500, 1000}, reporting the best value.
  • CQL: CQL uses the authors’ official implementation, the codebase hyperparameters, and fixed α values searched over α ∈ {5, 10}.The codebase settings outperformed the original-paper settings on the updated D4RL datasets.
  • Ours: SAC-N tunes ensemble size by environment, using up to N = 100 for walker2d-expert and N ∈ {100, 200, 500, 1000} for hopper.Halfcheetah and walker2d generally use smaller ranges.
  • Ours: EDAC tunes N ∈ {10, 20, 50} and η ∈ {0.0, 1.0, 5.0} on MuJoCo Gym, with larger η ranges for Adroit.Pen-cloned uses η = 10.0.

C Experimental settings

The experiments evaluate offline RL methods across D4RL MuJoCo Gym and Adroit settings using normalized average return, extended baselines, and ensemble-size analyses. The authors report that their methods outperform most considered baselines, while EDAC uses a smaller ensemble than SAC-N.

  • Experimental settings: MuJoCo Gym experiments use v2 datasets, 3 million training steps, and normalized average return.The longer training schedule was intended to help convergence on more complex datasets.
  • Experimental settings: Adroit experiments use v1 datasets, max Q backup, reward normalization, and 200,000 training steps with normalized average return.The shorter schedule provides an early-stopped comparison because CQL performance degrades after additional training.
  • Ensemble-size analysis: EDAC reduces the minimum ensemble size substantially when the original requirement is high, including on hopper and walker2d-expert.The minimum N is selected from sweeps spanning 2 through 1000.
  • Additional analyses: Action-distance histograms use 500,000 random state-action pairs per dataset and compare policy actions with dataset actions using ℓ2 distance.The actions come from each fully trained policy.
  • Adroit reproduction: On pen-∗ tasks, CQL performance degraded after about 200,000 steps, so Adroit comparisons used early stopping and also included reported CQL numbers.The cause of the performance gap was uncertain and possibly related to an unspecified parameter or a code modification.
  • Results: The methods outperform all baseline methods on most considered datasets, while EDAC is marginally better than SAC-N with a much smaller Q-ensemble.Extended comparisons include BCQ, BEAR, BRAC, MOReL, and UWAC, with normalized average returns averaged over 4 seeds.
  • CQL ablation: Increasing Q-networks or adding gradient diversification improves CQL on some datasets, but remains far behind SAC-N and EDAC.The ablation evaluates CQL-N and CQL with ensemble diversification.

G Comparison to variance regularization

The variance-regularization comparison tests whether directly increasing Q-estimate variance can replace EDAC’s diversification strategy. On walker2d-expert, stronger variance regularization causes severe underestimation, while weaker regularization causes exploding Q-values.

  • Comparison setup: The comparison evaluates EDAC against a Q-estimate variance regularizer on walker2d-expert using the same number of Q-networks.The regularizer is added to SAC-N and weighted by coefficient c.
  • Results: c = 50 causes exploding Q-values, whereas c = 100 and c = 200 cause severe Q-value underestimation.These are the two observed failure modes of the variance-enhancing regularizer.
  • Interpretation: At c = 200, the Q-estimate standard-deviation gap is 4.5, about half EDAC’s, despite much higher absolute standard deviations for both action types.The gap between behavior and OOD actions grows more slowly than the absolute variances.

H Hyperparameter sensitivity

EDAC’s sensitivity to the gradient-diversification weight η is evaluated across D4RL hopper datasets with 50 Q-networks. Except on the random dataset, EDAC achieves expert-level performance across a broad hyperparameter range, while larger η can hurt performance on datasets containing suboptimal-policy trajectories.

  • Experimental setup: η is swept over {0.0, 0.5, 1.0, 2.0, 5.0} on hopper datasets while fixing the ensemble size at N = 50.The results are presented in Table 9.
  • Results: Except for the random dataset, EDAC achieves expert-level performance across a large well of hyperparameters.
  • Results: Increasing η sometimes degrades performance on random, medium, and medium-replay datasets containing trajectories from suboptimal policies.The gradient diversification term favors in-distribution actions over OOD actions, so larger η can make the learned policy more conservative.
Loading 2110.01548v2…