PaperlayerView exact revision on arXiv

Source-linked AI summary

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Ilya Kostrikov, Jonathan Tompson, Rob Fergus, Ofir Nachum

arXiv:2103.08050v1cs.LG

TL;DR

Offline RL must learn from fixed data while controlling policy deviation from the behavior distribution. Fisher-BRC parameterizes the critic as behavior-policy logits plus a learned offset and applies a gradient penalty equivalent to Fisher divergence regularization. On standard benchmarks, it reports strong and consistent performance with faster convergence and lower computational cost than CQL.

  • Problem

    Offline RL learns from static datasets, motivating methods that keep learned policies close to the data-generating behavior policy.

  • Method

    Fisher-BRC parameterizes critic values as behavior-policy logits plus a state-action offset and regularizes the offset with a gradient penalty connected to Fisher divergence.

  • Results

    Fisher-BRC delivers state-of-the-art and consistent performance across standard offline RL benchmarks while converging faster and requiring less training time than CQL.

  • Takeaways & Limitations

    The approach offers a simple, implementable form of behavior regularization with reported performance and computational advantages across offline RL tasks.

  • Takeaways & Limitations

    An alternative Fisher-BRC divergence direction showed no large empirical differences in initial experiments, but the paper retains its original formulation because of its closer connection to the actor loss.

Abstract

from arXiv · show

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data. In this work, we propose an alternative approach to encouraging the learned policy to stay close to the data, namely parameterizing the critic as the log-behavior-policy, which generated the offline data, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. We propose using a gradient penalty regularizer for the offset term and demonstrate its equivalence to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature. We thus term our resulting algorithm Fisher-BRC (Behavior Regularized Critic). On standard offline RL benchmarks, Fisher-BRC achieves both improved performance and faster convergence over existing state-of-the-art methods.

1. Introduction

Offline RL learns from static data rather than new environment interaction, motivating behavior regularization to keep learned policies near the data-generating policy. Fisher-BRC instead regularizes a critic offset, connecting the approach to Fisher divergence and achieving strong benchmark performance with improved efficiency.

  • Motivation: Offline RL uses a static dataset collected by another behavior policy when online environment interaction is expensive or risky.The offline setting removes the ability to generate new trial-and-error experience during learning.
  • Motivation: Existing offline RL methods commonly penalize divergence between the learned policy and the data-generating behavior policy.These penalties are typically added to model-free actor-critic objectives.
  • Method: Fisher-BRC parameterizes critic values as behavior-policy logits plus a state-action offset, then regularizes that offset to keep the learned policy near behavior.The actor maximizes the critic, so constraining the offset constrains deviations from the behavior policy.
  • Method: A gradient penalty on the offset is mathematically connected to Fisher divergence regularization and energy-based model literature.The derivation interprets critic values as the energy function of a Boltzmann distribution.
  • Results: Fisher-BRC achieves state-of-the-art performance across standard offline RL benchmarks and is more computationally efficient than more sophisticated algorithms.The authors report both better policies and faster convergence as practical benefits.

2. Related Work

Fisher-BRC relates its offset regularization to Fisher divergence and energy-based models while situating gradient penalties among prior regularization techniques.

  • Behavior regularization: Behavior regularization in offline RL has been implemented through divergence penalties, network initialization, policy parameterization, and critic modification.These approaches all aim to keep learned policies close to offline data.
  • Energy-based connections: Fisher-BRC connects its critic parameterization to Fisher divergence regularization and energy-based models.Related energy-based policy work targets multimodal distributions, whereas this paper focuses on return-maximizing reinforcement learning.
  • Gradient penalties: The method uses a gradient penalty on the critic offset, extending a technique previously used in Wasserstein GANs, imitation learning, disentanglement, and uncertainty estimation.The paper places this use of gradient penalties within a broader machine-learning literature.

3. Background

Offline RL learns policies from fixed transition datasets, where incomplete state-action coverage creates extrapolation problems. Prior methods constrain policies or penalize critics, but these strategies face sampling, learning-signal, convergence, or computational limitations.

  • Reinforcement Learning: The paper models environments as discounted Markov Decision Processes with continuous action spaces and seeks policies maximizing cumulative returns.The MDP includes state and action spaces, initial-state distribution, dynamics, rewards, and a discount factor.
  • Offline Reinforcement Learning: Offline RL learns from a fixed dataset of transitions generated by a behavior policy, without generating new experience.The behavior policy is denoted µ.
  • Offline Reinforcement Learning: Incomplete state-action coverage causes standard SAC and DDPG methods to suffer from extrapolation errors on offline datasets.The observed state-action set is typically only a small subset of the full space.
  • Policy Regularization: Behavior regularization constrains offline RL through policy regularization or critic penalties.Policy constraints may be hard, as in BCQ, or soft, using divergences such as KL divergence.
  • Policy Regularization: BCQ samples perturbed actions from a fitted behavior model and selects the highest-valued action, but competitive performance may require many samples.The perturbation is constrained to [−Φ, Φ], and n controls the number of sampled actions.
  • Policy Regularization: Policy divergence penalties do not prevent critic extrapolation because the critic receives no learning signal for unseen actions yet is queried on out-of-distribution actions.This issue can dominate policy divergence regularizers.
  • Critic Regularization: AlgaeDICE and CQL regularize the critic, but AlgaeDICE converges more slowly while CQL requires numerical integration for continuous-action log-sum-exp terms.Both methods provide critic learning signals across the action space, but their computational or convergence properties differ.

4. Fisher-BRC

Fisher-BRC represents Q-values as the fixed log behavior density plus a learnable state-action offset, then regularizes the offset through a gradient penalty equivalent to Fisher divergence. This design aims to balance staying near offline behavior with richer value representations while avoiding normalization-cost issues associated with KL-based alternatives.

  • 4.1. Conceptual Derivation: The critic is parameterized as Qθ(s, a) = Oθ(s, a) + log µ(a|s), where µ is the fixed behavior policy approximated from offline data.The algorithm first learns approximate µ through behavioral cloning, then alternates critic and actor updates.
  • 4.1. Conceptual Derivation: Fisher-BRC replaces the overly restrictive state-value term with a state-action offset, allowing the learned policy to generalize beyond simply copying behavior.The offset must remain constrained to avoid extrapolation errors and policy divergence.
  • 4.1. Conceptual Derivation: Because the offset affects policy updates through its action gradients, Fisher-BRC applies a gradient penalty to control the balance between behavior matching and richer Q-values.The penalty coefficient λ controls its contribution to the critic objective; λ = 0.1 unless otherwise noted.
  • 4.2. Fisher Divergence Derivation: The gradient penalty is derived as Fisher divergence regularization between a Boltzmann policy and the behavior policy, connecting Fisher-BRC to score matching and energy-based models.Fisher divergence requires sampling access and score gradients, so normalized distributions and their log-normalizers need not be computed.
  • 4.2. Fisher Divergence Derivation: Unlike CQL’s continuous-action normalization term, Fisher-BRC avoids expensive numerical integration through its critic representation and Fisher-divergence formulation.The paper interprets CQL as KL-divergence regularization between the Boltzmann policy and behavior policy, while noting a divergence-direction variant for Fisher-BRC.

5. Experiments

Experiments evaluate Fisher-BRC on a toy continuous bandit and D4RL benchmarks, examining policy landscapes, gradient-penalty sensitivity, benchmark performance, and convergence speed. Fisher-BRC generally performs consistently, converges faster, and avoids the computational cost associated with CQL’s numerical integration.

  • Toy Continuous Bandit Problem: BRAC’s KL coefficient α is difficult to choose because low regularization can over-generalize while high regularization can move optima away from the best actions.The comparison identifies either optima outside [−0.25, 0.25] or optima far from {−0.25, 0.25}.
  • Convergence Speed: Fisher-BRC achieves better final performance and faster convergence in most tasks than BRAC and CQL.For 1 million steps, Fisher-BRC takes 1.4 hours of behavioral-cloning pretraining plus 6.2 hours of policy training, compared with CQL’s 16.3 hours total.
  • Deep Offline RL Benchmarks: Fisher-BRC performs comparably to or surpasses prior methods on most D4RL tasks and maintains more consistent performance across tasks than many baselines.The baselines include BRAC-vp, BRAC-pr, MBOP, and CQL.
  • Gradient-Penalty Sensitivity: Fisher-BRC’s gradient penalty is essential: λ = 0.0 causes dramatic performance degradation, whereas λ = 1.0 over-constrains the learned policy.A high λ improves expert-dataset performance but limits medium-dataset performance because the behavior policy is highly suboptimal there.

6. Conclusions

The conclusion presents Fisher-BRC as a simple critic representation and regularization technique that connects offline RL training to Fisher divergence, score matching, and energy-based models. Across varied domains, it reports better and more consistent performance than existing offline RL algorithms.

  • 6. Conclusions: Fisher-BRC combines a critic representation with regularization connected through derivations to Fisher divergence, score matching, and energy-based models.The authors characterize the method as simple, easy to implement, and highly performant.
  • 6. Conclusions: Fisher-BRC exhibits better and more consistent performance across a variety of domains than existing offline RL algorithms.

A. Implementation Details

Implementation uses a behavior-density model and a Soft Actor-Critic-based actor-critic setup, with added reward handling for fair comparison. Network and optimization choices follow stated CQL- and SAC-related configurations.

  • Implementation Details: The behavior model is a five-component conditional Mixture of Gaussians with tanh squashing, trained with Adam for 10^6 steps.Its learning rate starts at 10^-3 and is reduced by factors of 10 at 8·10^5 and 9·10^5 steps.
  • Implementation Details: Actor and critic learning is based on Soft Actor-Critic, using default SAC hyperparameters and 3-layer MLPs with 256 hidden units.The critic loss is modified to include the additional regularization term, and entropy is not added to rewards.
  • Implementation Details: The implementation adds a reward bonus to match the survival-bonus treatment used for environments with early termination.The paper explains that terminal-state Q-targets are replaced with 0 in practice.

B. Gradient Penalty Ablation

The ablation evaluates F-BRC across gradient penalty settings, showing that regularization strength materially affects stability and performance across dataset types.

  • λ = 1 over-constrains the learned policy toward the behavior policy, stabilizing expert-dataset performance but worsening medium-dataset performance.
  • Without regularization, λ = 0.0, Fisher-BRC collapses on most tasks, with plot cutoffs indicating NaN values from at least one training seed.

C. Critic Regularization Ablation

The ablation tests the gradient penalty without the proposed critic representation and finds that the penalty alone does not improve the underlying offline RL algorithm.

  • The experiment evaluates gradient-penalty effects in standard Soft Actor-Critic without the paper’s critic representation.
  • Without the critic representation, the gradient penalty alone fails to improve the underlying reinforcement learning algorithm on offline datasets.
Loading 2103.08050v1…