PaperlayerView exact revision on arXiv

Source-linked AI summary

When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

Aviral Kumar, Joey Hong, Anikait Singh, Sergey Levine

arXiv:2204.05618v1cs.LG

TL;DR

The paper asks when offline RL can outperform behavioral cloning (BC) on demonstration-like data. It characterizes relevant conditions theoretically and empirically, finding that offline RL can beat BC in practical settings, including with equal amounts of expert data.

  • Problem

    The paper asks whether offline RL can outperform BC when both learn from expert or near-expert demonstrations.

    Source: §1 · p. 1, §6 · p. 10

  • Method

    The paper theoretically characterizes conditions favoring offline RL over BC and tests them across diagnostic and practical benchmark domains and data sources.

    Source: §1 · p. 2

  • Results

    Across practical domains, offline-tuned offline RL can outperform BC, including expert-data cloning with equal amounts of data; noisy-expert data also gives more favorable guarantees than expert-data BC.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Sparse rewards or a small fraction of critical states are conditions under which offline RL may gain an advantage over reward-agnostic BC.

    Source: §1 · p. 2, §4.2 · p. 5

    The expert-data structural comparison considers environments where the total density of critical states is bounded by pc.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additionally, we show that policies trained on sufficiently noisy suboptimal data can attain better performance than even BC algorithms with expert data, especially on long-horizon problems. We validate our theoretical results via extensive experiments on both diagnostic and high-dimensional domains including robotic manipulation, maze navigation, and Atari games, with a variety of data distributions. We observe that, under specific but common conditions such as sparse rewards or noisy data sources, modern offline RL methods can significantly outperform BC.

1 INTRODUCTION

The paper asks when offline RL can outperform behavioral cloning (BC) on expert or near-expert data, a setting where practitioners may naturally choose BC. It proposes a theoretical and empirical characterization of favorable conditions and tests it across practical domains.

1 INTRODUCTION

  • Offline RL can work with highly suboptimal data, while BC uses supervised learning to imitate dataset behavior.The paper asks whether offline RL can also be preferable when available data consists of expert or near-expert demonstrations.Source: §1 · p. 1
  • The work addresses mixed prior comparisons and the lack of a rigorous account of when offline RL beats imitation learning.Prior studies reported mixed outcomes, leaving practitioners unsure whether to use offline RL or BC on demonstrations.Source: §1 · p. 1
  • The paper characterizes conditions under which offline RL outperforms BC, including with equal amounts of expert data.The theoretical guarantees identify environment and dataset conditions; the empirical evaluation tests these conclusions across domains.Source: §1 · p. 2

2 RELATED WORK

Prior work reports mixed empirical comparisons of offline RL and BC, while theoretical results do not yet provide a general MDP characterization of when offline RL is better. This paper analyzes existing methods rather than proposing a new algorithm.

2 RELATED WORK

  • Prior offline RL methods address distribution shift through policy constraints or conservative return estimates, but this paper does not propose a new algorithm.Policy constraints may be explicit, implicit, or based on importance sampling.Source: §2 · p. 2
  • Existing empirical findings conflict: offline RL can outperform BC on stitching tasks, while other studies report stronger BC results.The reported discrepancy includes D4RL Adroit results versus results reported by Florence et al.; prior work also notes evaluation-budget and tuning considerations.Source: §2 · p. 2
  • A conservative offline RL method has a guarantee against BC in contextual bandits, but its extension to MDPs is nontrivial because RL suffers compounding errors [52, 41, 42, 64].Safe policy-improvement bounds compare against the behavior policy but do not clearly establish when offline RL is better or worse.Source: §2 · p. 2
  • The analysis combines error propagation and horizon-free offline Q-learning tools, and uses tighter Bernstein bonuses in a lower-confidence-bound-style analysis [7] [36] [53] [73] [1] [52].The analysis relaxes strict coverage assumptions used in prior horizon-free bounds.Source: §2 · p. 2

3 PROBLEM SETUP AND PRELIMINARIES

The setup defines the discounted-reward MDP objective and models offline data as transitions sampled from a distribution over state-action pairs. The analysis introduces coverage and bounded-return conditions for reasoning about suboptimality.

3 PROBLEM SETUP AND PRELIMINARIES

  • An MDP is specified by state and action spaces, transition dynamics, reward, and discount factor; policies are evaluated by expected cumulative discounted reward.The section defines Q- and value functions, including their optimal-policy forms.Source: §3 · p. 3
  • The offline dataset contains N transitions sampled i.i.d. from a state-action distribution that induces the effective behavior policy.Empirical dynamics and rewards may differ from the true distributions because of stochasticity.Source: §3 · p. 3
  • The concentrability coefficient C* measures how well the dataset covers state-action pairs visited by the optimal policy; C*=1 corresponds to data from that policy.The setup also assumes every trajectory's infinite-horizon discounted return is bounded by 1, a condition that holds in sparse-reward tasks.Source: §3 · p. 3
  • The section introduces notation for polylogarithmic factors and vector entries used in the analysis.It defines ι as a context-dependent polylogarithmic quantity and x(i) as the i-th entry of a vector.Source: §3 · p. 3

4 THEORETICAL COMPARISON OF BC AND OFFLINE RL

The section compares BC and offline RL guarantees, then identifies data and environment conditions under which offline RL can outperform BC. It also examines how full offline RL compares with reward-informed BC variants.

4.1 IMPROVED PERFORMANCE GUARANTEES OF BC AND OFFLINE RL

  • BC’s bound has an Õ(H) rather than Õ(H^2) suboptimality term under Condition 3.2, while its other bound term reflects behavior-policy mismatch.The section’s BC analysis matches the empirical behavior policy on observed states and uses uniform random actions outside dataset support.Source: §4.1 · p. 4
  • Conservative offline RL uses pessimistic value estimates, and its bound avoids strict coverage assumptions while improving on prior bounds in several ways.The analysis also reports savings in horizon and state-space factors relative to prior methods.Source: §4.1 · p. 4

4.2 COMPARISON UNDER EXPERT DATA

  • With expert data and no assumptions about environment structure, offline RL and BC perform equally poorly; critical-state structure can instead favor offline RL.The critical-state comparison assumes low average occupancy of critical states, many good actions at other states, and a controlled gap between good and bad actions.Source: §4.2 · pp. 5–6
  • Even when C* is near 1, distribution shift can impose BC suboptimality on states present in the dataset, allowing MDP structure to favor offline RL.The paper notes that a changed initial-state distribution can make C* exceed 1 even when the data-collection policy follows the expert.Source: §4.2 · p. 5

4.3 COMPARISON UNDER NOISY DATA

  • Offline RL with covered noisy data can outperform BC trained on expert data, particularly on long-horizon tasks.The comparison relies on coverage conditions for the noisy dataset.Source: §4.3 · p. 7

4.4 COMPARISON OF GENERALIZED BC METHODS AND OFFLINE RL

  • Multiple policy-improvement steps can improve performance over one step when average exponentiated empirical advantage is sufficiently large, especially on long-horizon tasks.The paper connects this setting to stitching segments from poor-performing trajectories into better-performing ones.Source: §4.4 · p. 8

5 EMPIRICAL EVALUATION OF BC AND OFFLINE RL

The evaluation compares offline RL and behavioral cloning across gridworlds and high-dimensional tasks, testing expert and noisy-expert data. Offline-tuned RL can outperform BC, with noisy-data gains especially evident in longer-horizon settings.

Diagnostic experiments on a gridworld

  • In gridworlds, BC performs best with optimal data, while RL methods perform better when initial-state distributions differ; with random-policy data mixed in, RL outperforms BC methods.The left comparison covers different numbers of critical states, and the right examines increasing random-data proportions in Multiple Critical.Source: §5 · p. 8, Figure 3 · p. 9
  • With noisy data, BC with multiple policy-improvement steps performs better than BC with one step.The paper reports this as validating Theorem 4.4.Source: §5 · p. 9

Evaluation in high-dimensional tasks

  • The high-dimensional evaluation includes manipulation, antmaze, and seven Atari games, with sparse rewards and expert or noisy-expert data.The study uses scripted, human, and RL-trained experts, and CQL as its representative offline RL method.Source: §5 · p. 9

Answers to questions (1) to (3)

  • After offline tuning, CQL outperforms BC on expert data, although naïve CQL is often comparable to or worse than BC.The reported tuning uses no additional online rollouts; BC performs better or comparably on Antmaze large with expert data.Source: §5 · p. 9, Table 1 · p. 10

Evaluation in high-dimensional tasks

  • CQL trained on noisy-expert data outperforms BC trained on equal-size expert data in manipulation and Atari, and outperforms expert-only CQL in manipulation.Table 2 reports the manipulation comparison; Figure 4 summarizes seven Atari games and reports that noisy-expert CQL significantly outperforms cloning the expert policy.Source: Table 2 · p. 9, §5 · p. 9, Figure 4 · p. 10
  • The performance gap grows with horizon, from manipulation tasks with H = 40 and H = 80 to Atari domains with H = 27000.On noisy-expert Atari data, CQL also significantly outperforms BC-PI, even though BC-PI was tuned using online rollouts.Source: §5 · p. 10

6 DISCUSSION

The discussion concludes that offline RL can be favorable to BC even with expert or near-expert data, and that noisy suboptimal data can have more favorable guarantees than expert data for BC. The authors identify function approximation, other real-world data distributions, and cases favoring BC as areas for further study.

Discussion

  • Offline RL can have an advantage over BC on expert data under additional assumptions about the environment.The discussion says the two approaches attain similar worst-case performance on expert data without those additional assumptions.Source: §6 · p. 10
  • Noisy suboptimal data can yield more favorable guarantees for RL than expert data yields for BC, given equal amounts of data.Source: §6 · p. 10
  • The analysis leaves open function approximation, other real-world data distributions, and settings where BC outperforms RL.Source: §6 · p. 10

Appendices

The appendixed pseudocode gives setup and update steps for a conservative offline RL algorithm and a policy-constraint offline RL algorithm.

A PSEUDOCODE FOR ALGORITHMS

  • Algorithm 1 takes an offline dataset, discount factor, and confidence level, then estimates counts, rewards, and transitions and initializes Q and V estimates.Its iterative procedure calculates a bonus and a policy-related quantity.Source: A PSEUDOCODE FOR ALGORITHMS · p. 16
  • Algorithm 1’s loop calculates a bonus and then a policy-related quantity at each iteration.The excerpt does not specify the formulas for either calculation.Source: A PSEUDOCODE FOR ALGORITHMS · p. 16
  • Algorithm 2 takes an offline dataset, discount factor, and threshold, estimates counts, rewards, transitions, and behavior-policy probabilities, then initializes a policy and Q estimates for nested updates.It defines a state-action indicator from whether the estimated behavior probability meets the threshold.Source: A PSEUDOCODE FOR ALGORITHMS · p. 16

B.1 PROOF OF THEOREM 4.1

The appendix proves the paper’s BC and conservative offline-RL guarantees. Its analysis establishes pessimistic value estimates and bounds policy suboptimality through value-difference and variance arguments.

B.1 PROOF OF THEOREM 4.1

  • BC on the behavior policy has an expected-error bound, which the proof combines with a bound on behavior-policy suboptimality.The proof uses Lemma B.1 and the definition of C∗ to obtain the Theorem 4.1 result.Source: §B.1 · p. 17

B.2 PROOF OF THEOREM 4.2

  • The conservative algorithm uses empirical value iteration with a penalty in each Q-update, and returns a policy greedy with respect to its final Q-values.It initializes Q and V at zero and runs m = H log N iterations.Source: §B.2 · p. 17

B.2.1 TECHNICAL LEMMAS

  • The passages state Bernstein’s inequality for i.i.d. [0,1]-valued variables, Theorem 4 of Maurer & Pontil [39] for such variables with n ≥ 2, and Ren et al.’s [53] recursion lemma for bounded f.The Bernstein passage also assumes δ > 0; Ren et al.’s lemma assumes λ1, λ2 > 0 and f: Z≥0 → R bounded above by H and satisfying a recursion.Source: §B.2.1 · pp. 17–18

B.2.2 PESSIMISM GUARANTEE

  • On the good event, each estimated value is no greater than the value of the learned policy, which is itself no greater than the optimal value.The proof establishes that the good event occurs with high probability using concentration bounds, including an adaptation of Agarwal et al. [1].Source: §B.2.2 · pp. 18–19

B.2.3 VALUE DIFFERENCE LEMMA

  • The value-difference analysis bounds the gap between optimal Q-values and pessimistic estimates by propagated value error plus the algorithm’s penalty.The proof recursively applies the one-step inequality in Lemma B.8 to derive Lemma B.9.Source: §B.2.3 · pp. 19–20

B.2.4 BOUND ON ∆2

  • For the second term, the proof uses ρ(s) ≤ Hd∗(s, π∗(s)) and the bound max_{p∈[0,1]} p(1−p)N ≤ (4/9)N.Source: §B.2.4 · p. 21

B.2.5 BOUND ON ∆3

  • The ∆3 contribution is bounded by summing its penalty terms, applying variance recursion, and combining the resulting bounds.The derivation uses Cauchy–Schwarz, Condition 3.2, and Ren et al. [53].Source: §B.2.5 · pp. 21–22

B.3 PROOF OF COROLLARY 4.1

This appendix proves the comparison between conservative offline RL and BC when critical states have low occupancy. It bounds performance separately at critical and non-critical states, where the proof gives RL a faster error reduction under stated conditions.

B.3 PROOF OF COROLLARY 4.1

  • RL controls non-critical states using reward information, while BC mimics the dataset; at non-critical states, the modified RL algorithm can collapse exponentially fast to a good-enough action.The fast-convergence guarantee applies with sufficient state coverage and the stated condition on the expert’s action distribution.Source: §B.3 · p. 23
  • At critical states, the proof reuses the general offline-RL bound, while the non-critical-state contribution is controlled by separating good and bad actions.The critical-state term is bounded using the machinery from Appendix B.2; Lemma B.10 controls the bad-action term.Source: §B.3 · pp. 23, 25
  • Under Condition B.1, if ∆(s) ≥ ∆0 for all s ∉ C and p_c ≲ 1/H (for an appropriate ∆0), conservative offline RL’s worst-case suboptimality is upper bounded by BC’s lower bound on performance from Theorem 4.3.The bound for BC is tight.Source: §B.3 · pp. 24, 26

B.4 PROOF OF COROLLARY 4.2

  • For Corollary 4.2, the ∆2 bound is adjusted to account for a lower bound on state-action data density, and the resulting ∆2 and ∆3 bounds yield the corollary.The proof changes the bound on ∆2 and then uses it in the ∆3 recursion.Source: §B.4.1 · pp. 26–27

B.5 PROOF OF THEOREM 4.4

The proof analyzes repeated advantage-weighted policy improvement and compares it with methods that make only one or a few updates. It connects the benefit of multiple updates to settings where useful improvements can be made repeatedly.

B.5 PROOF OF THEOREM 4.4

  • When k = O(H) high-advantage policy updates are possible, methods making O(H) improvement steps attain higher performance than counterparts making only one update.Maze navigation can provide this setting when updates stitch suboptimal trajectory segments into a better trajectory.Source: §B.5 · p. 28

C GUARANTEES FOR POLICY-CONSTRAINT OFFLINE RL

The policy-constraint analysis bounds performance against the best policy that stays mostly within high-density state-action support. The accompanying illustration explains critical states through how many actions preserve a route to the goal.

C GUARANTEES FOR POLICY-CONSTRAINT OFFLINE RL

  • The policy-constraint algorithm restricts action selection to safe, previously observed high-density actions, and its guarantee compares it with the best ζ-covered policy.The guarantee matches the conservative offline-RL bound up to a slower-growing horizon term when U(b) ≤ O(H^0.5−ε).Source: C GUARANTEES FOR POLICY-CONSTRAINT… · pp. 28–30
  • The guarantee retains a U(b) term; the authors conjecture it could be removed under assumptions linking support indicators at consecutive state-action tuples.One example is that support at the next tuple implies support at the current tuple.Source: C GUARANTEES FOR POLICY-CONSTRAINT… · p. 30

D INTUITIVE ILLUSTRATIONS OF THE CONDITIONS ON THE ENVIRONMENT AND PRACTICAL GUIDELINES FOR VERIFYING THEM

  • The paper presents practical illustrations of its conditions and notes that sparse end-of-episode success or failure rewards satisfy Condition 3.2 in common settings.The section focuses on Conditions 4.1 and 4.2.Source: D INTUITIVE ILLUSTRATIONS OF THE… · p. 30

D.1 CONDITION 4.1

  • In the navigation illustration, B is critical because few actions there preserve a route to the goal, while many actions work at A and C.The wide regions around A and C are non-critical, so the task has only a small volume of critical states.Source: Figure 5 · p. 30, §D.1 · p. 31

D.2 CONDITION 4.2

Condition 4.2 explains how expert-state coverage can be improved by exploration in the expert policy or noisy data, enabling offline RL to learn from outcomes BC cannot use. The practical guidance is qualitative, and the gridworld experiments vary how many critical states each route contains.

D.2 CONDITION 4.2

  • Condition 4.2 is met when frequently visited expert state-action pairs have sufficient dataset density, through expert exploration or noisy data that explores less-visited states.Failed or counterfactual trajectories can show offline RL what not to do, while BC cannot use this negative data.Source: §D.2 · p. 31

D.3 PRACTICAL GUIDELINES FOR VERIFYING THESE CONDITIONS

  • Practitioners are advised to consider whether many trajectory states allow recovery or multiple good actions, and whether noisy or augmented data are available.If either condition seems to hold, the paper advises trying offline RL, while noting these are non-rigid guidelines.Source: §D.3 · p. 32

E.1 TABULAR GRIDWORLD DOMAINS

  • The 10 × 10 gridworlds use sparse goal rewards, zero reward after entering lava, and a 10% chance of moving in a direction other than the commanded one.The three domains contain one critical state, three critical states, or make every state critical, respectively.Source: §E.1 · p. 32
  • Figure 7 renders Single Critical, Multiple Critical, and Cliffwalk from left to right, with states colored by type.The gridworlds differ in the number of critical points encountered per trajectory.Source: §E.1 · p. 32, Figure 7 · p. 33

E.2 MULTI-STAGE ROBOTIC MANIPULATION DOMAINS

The experimental details describe multi-stage robotic manipulation, maze navigation, Adroit control, and Atari tasks, along with their datasets and reward setups. In Atari, the reported table captions distinguish tuned CQL with DR3 from naïve CQL and report that offline RL generally outperforms BC-PI on noisy-expert data.

E.2 MULTI-STAGE ROBOTIC MANIPULATION DOMAINS

  • The WidowX manipulation tasks require opening a drawer, grasping an object, and placing it in a tray, with obstructions varying across three task variants.Each task dataset contains 5,000 trajectories; expert data has 100% success, while the noisy scripted data has mixed task-stage success rates.Source: §E.2 · p. 33

E.3 ANTMAZE DOMAINS

  • Antmaze evaluates an 8-DoF ant navigating to a target in two maze layouts, using 500,000 transitions from scripted trajectories with differing start-state coverage.The task uses a sparse binary reward provided only at the end of a trajectory.Source: §E.3 · p. 34

E.4 ADROIT DOMAINS

  • Adroit evaluates four 24-DoF Shadow Hand tasks using human demonstrations, with offline RL trained on a binary reward derived from D4RL goal-achievement flags.The tasks involve hammering, opening a door, twirling a pen, and relocating a ball.Source: §E.4 · p. 34

E.5 ATARI DOMAINS

  • The Atari experiments use seven games and DQN-Replay data, including the final buffer as expert data and a mixture of final-buffer and early-buffer random data as noisy-expert data.Evaluations use sticky actions and retain sparse rewards.Source: §E.5 · pp. 34–35
  • Tuned CQL with DR3 performs much better than BC on Atari expert data, whereas naïve CQL performs similarly to BC.On noisy-expert data, offline RL generally significantly outperforms BC-PI.Source: Table 3 · p. 35, Table 4 · p. 35

F TUNING AND HYPERPARAMETERS

The tuning section describes offline CQL and BC-PI procedures, regularization and policy selection, and a single shared BC-PI temperature despite variation across games. The lower-bound discussion distinguishes policy evaluation outside data support from conservative in-support policy learning.

F TUNING AND HYPERPARAMETERS

  • CQL tuning uses Q-value trends to diagnose overfitting or underfitting, then applies dropout for overfitting or DR3 for underfitting and selects checkpoints accordingly.For Adroit, the authors use dropout; for Atari and Antmaze, they use DR3.Source: F TUNING AND HYPERPARAMETERS · pp. 35–36
  • BC-PI trains a Q-function with SARSA and extracts a policy using advantage-weighted behavior-policy actions.Its behavior-policy model follows the BC tuning protocol, while the method uses a single temperature across states.Source: F TUNING AND HYPERPARAMETERS · p. 36
  • For Atari, the best average BC-PI setting uses η = 0.005 and advantage clipping [−10, 2], although performance varies substantially across games.BC-PI reaches 118% normalized return on Qbert, 78% on Enduro, and 31.7% on SpaceInvaders; the authors use one η across games.Source: F TUNING AND HYPERPARAMETERS · p. 36
  • The authors note that per-state temperature tuning may be needed, but it is outside their BC-PI definition and they use a single η across states.Source: F TUNING AND HYPERPARAMETERS · p. 36

G REGARDING OFFLINE RL LOWER BOUNDS

  • The cited lower bound concerns evaluation of some policies that leave offline-data support, not every policy or in-support policies produced by conservative offline RL.The authors also state that the Wang et al. lower bound applies to non-pessimistic algorithms that do not address distribution shift.Source: G REGARDING OFFLINE RL LOWER BOUNDS · p. 37
Loading 2204.05618v1…