PaperlayerView exact revision on arXiv

Source-linked AI summary

Offline RL Without Off-Policy Evaluation

David Brandfonbrener, William F. Whitney, Rajesh Ranganath, Joan Bruna

arXiv:2106.08909v3cs.LGstat.ML

TL;DR

Offline RL commonly relies on iterative actor-critic methods with off-policy evaluation. This paper studies a simpler one-step alternative using the behavior policy’s Q estimate and finds strong benchmark performance, while noting unresolved conditions and limitations.

  • Problem

    Prior offline RL methods primarily rely on iterative actor-critic algorithms and off-policy evaluation, motivating a simpler alternative.

  • Method

    The paper performs one constrained or regularized policy-improvement step using an on-policy estimate of the behavior policy’s Q function.

  • Results

    One-step policy improvement beats prior iterative-algorithm results on much of the D4RL benchmark and frequently outperforms tested iterative algorithms.

  • Takeaways & Limitations

    Practitioners should run the substantially simpler one-step algorithm as a baseline before trying more elaborate offline RL methods.

  • Takeaways & Limitations

    The paper lacks a clear theoretical characterization of which environments and behavior policies favor one-step over multi-step methods, or vice versa.

Abstract

from arXiv · show

Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

1 Introduction

The paper challenges iterative actor-critic offline RL, showing that one-step policy improvement using the behavior Q function can outperform more complicated methods across many benchmark tasks.

  • Motivation: Offline RL learns a new policy from data collected by another behavior policy without further environment interaction.The paper frames offline RL as an avenue for improving sample efficiency in real-world reinforcement learning.
  • Dominant paradigm: Prior deep offline RL primarily alternates policy evaluation and improvement, relying heavily on off-policy evaluation.These methods include actor-critic algorithms designed to keep learned policies near the behavior distribution.
  • Proposed challenge: A simple one-step baseline performs one policy-improvement step using the behavior Q function instead of repeatedly evaluating policy iterates.All methods constrain policies to a neighborhood of safe policies around β, but one-step uses on-policy bQβ while multi-step methods use off-policy estimates.
  • Contributions: The paper demonstrates that one-step policy improvement outperforms more complicated iterative algorithms on a broad set of offline RL problems.The contributions also examine iterative off-policy-evaluation failure modes and identify conditions favoring one-step methods.

2 Setting and notation

The paper studies offline RL in discounted continuous-control MDPs, where a policy must be learned solely from a fixed dataset collected by a behavior policy.

  • MDP setting: The environment is modeled as a discounted infinite-horizon MDP M = {S, A, ρ, P, R, γ}.The continuous-control focus generally assumes bounded continuous state and action spaces.
  • Offline data: The offline dataset DN contains N tuples (si, ai, ri) collected by a behavior policy β, with no environment interaction during learning.The initial state distribution is ρ, and expected rewards are represented by r(s, a).
  • Objective: For any policy π, Qπ(s, a) is its expected discounted return after starting from state s and action a.The learning objective is to maximize the expected return J of the learned policy.
  • Evaluation protocol: The experimental protocol permits environment access to tune a small set of fewer than 10 hyperparameters.This follows prior offline RL evaluation practice.

3 Related work

Related offline RL methods primarily use iterative algorithms that constrain policies near the behavior distribution, while one-step behavior-value approaches provide a contrasting alternative.

  • Iterative methods: Prior iterative offline RL methods fall broadly into policy constraints or regularization, imitation-learning modifications, and Q regularization.These categories share the goal of preventing learned policies from selecting unsupported actions.
  • Policy constraints/regularization: Policy-directed methods impose support constraints or regularize the learned policy toward the behavior policy using divergences such as KL or MMD.These methods remain iterative and rely on off-policy evaluation.
  • Imitation modifications: Imitation-based methods filter observed actions or weight them using Q values to encourage policy improvement.Weighted variants use exponentiated Q values.
  • One-step methods: One-step behavior-value methods have performed well in both continuous-control and discrete-action offline RL settings.The paper distinguishes its continuous-control setting from discrete settings where policy improvement can be computed exactly from Q.
  • Failure analysis: The paper separates distribution shift from iterative error exploitation, which can combine to cause overestimation.This distinction contrasts with attributing poor iterative performance only to overestimation.
  • Connections: The one-step approach is connected to conservative policy improvement, discussed further in the paper’s appendix.

4 Defining the algorithms

The paper unifies offline RL algorithms as approximate modified policy iteration and compares one-step, multi-step, and iterative schemes with several behavior-constrained improvement operators.

  • 4.1 Algorithmic template: The OAMPI template alternates estimating the current policy’s Q function and improving the policy using the estimate, behavior model, and dataset.Policy evaluation may be warm-started from the previous Q estimate, while policy improvement may be warm-started from the previous policy.
  • 4.1 Algorithmic template: The one-step algorithm sets K = 1, estimates β by maximum likelihood, estimates Qβ, and performs one constrained or regularized policy-improvement step.It completely avoids off-policy evaluation.
  • 4.1 Algorithmic template: The multi-step algorithm sets K > 1 and evaluates off-policy policies πk−1 ≠ β, training each evaluation and improvement step to convergence.
  • 4.1 Algorithmic template: The iterative actor-critic method uses many iterations, each with one gradient step for Q estimation and one for policy improvement.It does not train either step to convergence at every iteration.
  • 4.2 Policy evaluation operator: Policy evaluation uses fitted Q evaluation with TD-style learning and a target network, without double Q learning or Q ensembles.One-step and multi-step procedures train evaluation to convergence per iteration, unlike the iterative procedure.
  • 4.3 Policy improvement operators: The tested improvement operators include behavior cloning, Easy BCQ-style constrained updates, reverse-KL regularization, and exponentially weighted imitation.Each operator has a hyperparameter controlling deviation from the behavior policy.
  • 4.3 Policy improvement operators: The experiments combine one-step, multi-step, or iterative templates with Easy BCQ, reverse-KL regularization, or exponentially weighted imitation.

5 Benchmark Results

One-step policy improvement performs strongly across the D4RL benchmark, often surpassing iterative offline RL methods while using a substantially simpler procedure.

  • One step of policy improvement is sufficient to beat state-of-the-art results on much of the D4RL benchmark suite.Unlike prior work, the method uses one step derived from the behavior policy's Q estimate rather than iteratively estimating the current policy's Q function.
  • One-step algorithms usually outperform the best iterative algorithms tested by Fu et al. across the benchmark.The notable exception is random data, especially in HalfCheetah, where iterative algorithms have a clear advantage.
  • The reverse-KL comparison evaluates one-step, multi-step, and iterative algorithms under the same regularization approach.Each variant uses six hyperparameters and reports mean and standard error over ten seeds with 100 evaluation episodes.
  • The results recommend running the one-step algorithm as a baseline before attempting more elaborate methods.The paper describes it as substantially simpler than prior work while frequently achieving better performance.

6 What goes wrong for iterative algorithms?

Iterative offline RL can become unstable because off-policy evaluation suffers from distribution shift, while repeatedly optimizing against dependent Q estimates exploits their errors. One-step improvement avoids off-policy evaluation and is more robust to regularization, especially when the behavior policy has limited coverage but is already good.

  • Learning curves and hyperparameter sensitivity: Iterative algorithms can learn initially and then crash, while stronger regularization toward the behavior policy can prevent this instability.Strong regularization makes evaluation nearly on-policy.
  • Learning curves and hyperparameter sensitivity: Figure 2 compares learning curves and final performance across algorithms and reverse-KL regularization settings on halfcheetah-medium.Error bars show the minimum and maximum over three seeds.
  • Learning curves and hyperparameter sensitivity: The one-step approach is more stable across regularization settings, whereas iterative and multi-step performance can degrade rapidly with poor hyperparameters.The one-step method usually has a lower optimal regularization setting.
  • Distribution shift: Off-policy evaluation becomes harder as the evaluation policy diverges from the behavior policy because distribution shift reduces effective sample size and increases estimate variance.Distribution shift is the difference between behavior and evaluation state-action distributions.
  • Iterative error exploitation: Iterative error exploitation arises when policies are optimized against Q estimates and those estimates are re-estimated from the same data, creating dependence between steps.Warm-starting Q estimation passes approximation errors between iterations.
  • Iterative error exploitation: Each improvement step can select overestimated actions, after which dynamic programming propagates their overestimation through policy evaluation.Under a fixed error approximation, repeated optimization can move policies farther from the behavior distribution and increase error magnitude.
  • Behavior-policy dependence: The one-step algorithm is robust to errors when β has small magnitude, while behavior-policy structure determines whether one-step or multi-step improvement is preferable.A modified behavior policy with better coverage can reduce errors and make multi-step improvement preferable.
  • Iterative error exploitation: In a noisy gridworld, one step often outperforms multiple steps because reused data lets overestimated bad states persist and propagate across the state space.The example uses 100 trajectories of length 100 and many stochastic bad states with rewards drawn from N(−0.5, 1).

7 When are multiple steps useful?

Multiple policy-improvement steps can help when behavior data provides sufficient coverage to reduce estimation noise, but one-step methods often remain preferable when errors are large or coverage is limited.

  • Multi-step and iterative methods become more useful when sufficient behavior-policy coverage reduces estimation noise.Lower noise allows these methods to propagate useful signal more broadly.
  • D4RL experiments generally favor one-step methods because estimation errors are large enough to outweigh the benefits of wider signal propagation.
  • A modified behavior policy can make multi-step methods dramatically better by improving coverage of noisy states and weakening one-step reward propagation.

8 Discussion, limitations, and future work

The paper presents one-step offline RL as a strong practical baseline, while leaving unresolved when one-step or multi-step approaches should win and whether other solutions could avoid off-policy evaluation problems.

  • The authors recommend trying the simple one-step baseline before more elaborate offline RL methods.
  • A main limitation is the lack of a clear theoretical characterization of environments and behaviors favoring one-step versus multi-step methods.
  • The paper does not consider all potentially more elaborate algorithmic solutions for avoiding off-policy evaluation problems.
  • The authors suggest that better offline RL methods could affect applications including robotics and healthcare.

A Gridworld example where multi-step outperforms one-step

A modified gridworld behavior policy favors multi-step planning by improving coverage of noisy states while reducing propagation of reward information to the behavior value function.

  • A Gridworld example where multi-step outperforms one-step: The modified gridworld uses β = 0.2 · π− + 0.8 · u, replacing the optimal component with a maximally suboptimal policy mixed with uniform behavior.The suboptimal policy always moves down and left; the experiment uses 100 trajectories of length 100, discount 0.9, and α = 0.1.
  • A Gridworld example where multi-step outperforms one-step: The example provides a case where multi-step methods clearly outperform one-step methods under the same data-generating and learning processes.
  • A Gridworld example where multi-step outperforms one-step: The modified behavior has better coverage of noisy states, reducing evaluation error and enabling more aggressive multi-step planning.
  • A Gridworld example where multi-step outperforms one-step: Because the behavior is less likely to reach the rewarding state, its Q function propagates the reward signal less far, harming the one-step method.
  • Connection to policy improvement guarantees: The one-step objective is motivated as optimizing a lower bound on performance difference after replacing Qβ with an estimate and total variation with KL divergence.

C.1 Benchmark experiments (Tables 1 and 2, Figure 2)

The benchmark experiments use D4RL datasets and standardized neural-network training, evaluation, and comparison procedures across one-step, multi-step, and iterative algorithms.

  • Data and evaluation: The experiments use the latest D4RL dataset versions: v2 for MuJoCo and v1 for Adroit.
  • Hyperparameter tuning: Hyperparameters are tuned through simulator interaction using 3 seeds, then the selected setting is evaluated on 7 additional training seeds.
  • Models and training: All one-step models use two-hidden-layer ReLU MLPs of width 1024, with truncated-normal policies and behavior and Q estimators trained before policy optimization.
  • Implementation details: Reverse-KL weights are clipped at 100 for numerical stability, and reverse KL is estimated using 10 current-policy samples.
  • Training comparisons: Multi-step and iterative methods share one-step initialization and policy-update budgets, while their Q updates use off-policy actions from the current policy.
  • Data and evaluation: Policies are evaluated with 100 environment trajectories, reporting means and standard errors over 10 training seeds.

C.2 MSE experiment (Figure 3)

The experiment evaluates Q-function estimation and overestimation using controlled gridworld data, varied dataset mixtures, and independently sampled evaluation trajectories. It compares learned Q functions with evaluation-dataset estimates under the stated training and evaluation procedures.

  • Data: 1000 behavior-cloned-policy trajectories provide an independently sampled dataset, with checkpointed policies saved every 5000 gradient steps across three seeds.
  • Evaluation procedure: MSE is computed over 1000 state-action pairs with 3 rollouts per pair, while reverse KL uses 10 samples per state and averages across each batch.
  • Environment: The 15 x 15 deterministic gridworld uses deterministic rewards at the top-right corner and stochastic rewards on the left and bottom walls.Initial states are uniformly random, with discount 0.9.
  • Data: 100 trajectories of length 100 are collected from a policy mixing uniform behavior with optimal behavior at probabilities 0.8 and 0.2.
  • Training procedure: The empirical-MDP procedure estimates rewards from data, applies 100 dynamic-programming steps, and solves regularized policy updates exactly.The policy update is proportional to β(a|s) exp(...), as stated in the procedure.
  • MSE experiment: The overestimation analysis compares Q functions learned during training with Q functions trained on the independently sampled evaluation dataset.It uses the same setup as the MSE experiment.

D Learning curves

This section reproduces learning curves and hyperparameter plots for one-step, multi-step, and iterative algorithms with reverse KL regularization across medium, medium-expert, and random datasets.

  • Learning curves: The reproduced plots compare one-step, multi-step, and iterative algorithms using reverse KL regularization.
  • Medium datasets: Figure 8 presents learning curves on the medium datasets.
  • Medium-expert datasets: Figure 9 presents learning curves on the medium-expert datasets.
  • Random datasets: Figure 10 presents learning curves on the random datasets.
Loading 2106.08909v3…