PaperlayerView exact revision on arXiv

Source-linked AI summary

The Generalization Gap in Offline Reinforcement Learning

Ishita Mediratta, Qingfei You, Minqi Jiang, Roberta Raileanu

arXiv:2312.05742v2cs.LGcs.AI

TL;DR

Offline methods are often evaluated on the same environment, leaving their generalization to new environments underexplored. The paper introduces Procgen and WebShop benchmarks and compares online and offline approaches. Offline methods generally trail online RL, while BC is a strong offline baseline and greater data diversity improves generalization.

  • Problem

    Offline learning’s generalization to new environments is underexplored, in part because existing datasets predominantly draw trajectories from singleton environments.

    Source: §1 · p. 1

  • Method

    The paper introduces datasets from Procgen and WebShop and benchmarks offline learning algorithms on unseen levels and instructions.

    Source: §1 · p. 2

  • Results

    Offline methods underperform online RL on Procgen, BC generally leads offline RL and sequence modeling, and greater data diversity improves generalization while greater dataset size does not.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Existing offline methods developed without generalization in mind are not enough for these challenges, motivating further work on offline-learning generalization.

    Source: §6 · p. 12

    WebShop evaluations omit DT and BCT because 512-token states limit transformer context, and IQL because its action-space implementation does not fit the state-dependent action sizes.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Despite recent progress in offline learning, these methods are still trained and tested on the same environment. In this paper, we compare the generalization abilities of widely used online and offline learning methods such as online reinforcement learning (RL), offline RL, sequence modeling, and behavioral cloning. Our experiments show that offline learning algorithms perform worse on new environments than online learning ones. We also introduce the first benchmark for evaluating generalization in offline learning, collecting datasets of varying sizes and skill-levels from Procgen (2D video games) and WebShop (e-commerce websites). The datasets contain trajectories for a limited number of game levels or natural language instructions and at test time, the agent has to generalize to new levels or instructions. Our experiments reveal that existing offline learning algorithms struggle to match the performance of online RL on both train and test environments. Behavioral cloning is a strong baseline, outperforming state-of-the-art offline RL and sequence modeling approaches when trained on data from multiple environments and tested on new ones. Finally, we find that increasing the diversity of the data, rather than its size, improves performance on new environments for all offline learning algorithms. Our study demonstrates the limited generalization of current offline learning algorithms highlighting the need for more research in this area.

1 Introduction

Offline learning’s generalization to new environments remains underexplored because datasets commonly cover a single environment, despite the importance of generalization for safe use. The paper introduces Procgen and WebShop datasets and benchmarks offline methods, finding that behavioral cloning performs strongly and data diversity matters more than dataset size for generalization.

1 Introduction

  • Offline learning generalization remains underexplored because existing datasets predominantly contain trajectories from singleton environments.The paper frames generalization to new scenarios as important for safe deployment, especially where online data collection is costly or dangerous.Source: §1 · p. 1
  • The paper introduces offline-learning datasets of varying sizes and skill levels from Procgen and WebShop to evaluate generalization.The datasets support tests on unseen Procgen levels and WebShop instructions.Source: §1 · p. 2
  • Across Procgen and WebShop, behavioral cloning generally outperforms offline RL and sequence modeling, while offline methods underperform online RL on Procgen.On WebShop human demonstrations, BC also exceeds offline methods in final score and success rate.Source: §1 · p. 2
  • Increasing training-data diversity significantly improves generalization, whereas increasing dataset size does not.The study examines both factors in its Procgen experiments.Source: §1 · p. 2

2 Background

The paper frames offline learning in contextual Markov decision processes, where contexts select the MDP encountered by an agent. It evaluates policies across unobserved contexts after training on fixed datasets generated from training contexts.

2 Background

  • A contextual MDP maps each context to an MDP with context-dependent transitions, rewards, and initial-state distributions.The context set is split into training, validation, and test subsets.Source: §2 · p. 2
  • RL seeks to maximize a policy’s expected reward over the time horizon, with r_t denoting reward at time t and T the horizon.The objective is expressed as an expected sum of rewards.Source: §2 · p. 2
  • Test contexts represent different environment instances: Procgen levels vary initial states and transitions, while WebShop instructions vary initial states and rewards.The contexts are not observable to the agent.Source: §2 · pp. 2–3
  • Offline agents learn from fixed trajectories generated by a behavior policy on training contexts, without online interaction during learning.Because the dataset may not cover the environment’s full state-action distribution, offline methods must address distribution shift or avoid out-of-distribution actions.Source: §2 · p. 3

3 Experimental Setup

The experimental setup builds offline datasets across Procgen and WebShop, compares behavioral cloning with offline RL and sequence-modeling baselines, and evaluates performance on held-out contexts. WebShop evaluations exclude some methods because their architectures do not fit the environment’s action-space or context-length constraints.

3.1 Datasets

  • The benchmark includes Procgen expert and mixed expert-suboptimal datasets at 1M transitions, expert data at 10M, and suboptimal data at 25M.WebShop datasets include 452 human trajectories and suboptimal sets of 100, 1K, 1.5K, 5K, and 10K trajectories.Source: §3.1 · p. 3
  • For Procgen, data are collected from 200 different levels for offline training; hyperparameters are selected via online validation on another 50 levels, and agents’ online performance is evaluated on remaining levels (level_seed ∈[250, ∞)).Source: §3.1 · p. 3

3.2 Baselines

  • BC predicts dataset actions; BCQ constrains actions to those in the data, CQL regularizes Q-values, and IQL uses expectile regression to estimate state values.These methods address different offline-learning challenges, including distribution drift, value overestimation, and evaluating out-of-distribution actions.Source: §3.2 · pp. 3–4
  • On Procgen’s 1M expert dataset, BC outperforms offline RL and sequence-modeling approaches, while all offline methods lag online RL on train and test environments.Figure 2 aggregates min-max-normalized returns over 16 games; evaluation uses 100 test episodes and IQM across five model seeds.Source: Figure 2 · p. 4
  • BCT predicts actions from state and episode history, while DT additionally conditions on return-to-go and predicts it alongside the action.Both use causal-transformer policies.Source: §3.2 · p. 4
  • Procgen results report per-game mean and standard deviation plus aggregate IQM and mean normalized return; WebShop reports average scores and success rates.WebShop excludes DT and BCT because 512-token states limit transformer context, and IQL is out of scope because action-space size varies by state.Source: §3.2 · p. 4

4 Experimental Results

Across Procgen and WebShop, offline methods often struggle on unseen environments or instructions, while behavioral cloning (BC) is a strong baseline. Increasing dataset diversity improves Procgen generalization more than increasing dataset size alone.

4.1 Generalization to New Environments using Expert Data

  • On Procgen expert demonstrations, offline methods underperform online RL at test time, while BC outperforms offline RL and sequence-modeling methods on train and test environments.The reported comparison aggregates performance across all 16 games.Source: §4.1 · p. 5

4.2 Generalization to New Environments using Mixed Expert-Suboptimal Data

  • With mixed expert-suboptimal Procgen data, BC beats other offline RL and sequence-modeling baselines on train and test, while offline methods still lag online RL.Methods have similar average generalization gaps, and IQL struggles to train and generalize across multiple levels despite its singleton-environment results.Source: Figure 3 · p. 5, §4.2 · pp. 5–6

4.3 Training and Testing on a Single Environment

  • When trained and tested on one Procgen level, offline methods perform well; with suboptimal data, offline RL is comparable to or better than BC on most games.The single-environment pattern differs from the multi-environment setting, where BC tends to outperform other offline methods.Source: §4.3 · pp. 6–7

4.4 The Effect of Data Diversity on Generalization

  • At a fixed dataset size of 1M transitions, increasing the number of Procgen training levels improves test performance and reduces the generalization gap.Training performance changes little as the number of levels increases.Source: Figure 5 · p. 7, §4.4 · p. 8

4.5 The Effect of Data Size on Generalization

  • With expert data and 200 training levels fixed, scaling Procgen data from 1M to 10M transitions slightly raises train and test returns but leaves the generalization gap nearly unchanged.For suboptimal data, algorithms remain poor on train and test even at 10M transitions; increasing size alone does not significantly improve test performance.Source: §4.5 · pp. 8–9, Figure 6 · p. 8, Figure 7 · p. 9

4.6 Generalization to New Instructions using Human Demonstrations

  • On WebShop human demonstrations, BC achieves higher scores than CQL and BCQ on train and test instructions, and the highest test success rate.Longer training improves training performance but test performance begins to decrease, suggesting overfitting.Source: §4.6 · pp. 9–10, Figure 8 · p. 10

4.7 Generalization to New Instructions using Suboptimal Demonstrations in WebShop

  • On WebShop suboptimal demonstrations, all three baselines improve substantially once the dataset reaches 500 episodes; BC outperforms BCQ and CQL across dataset sizes.Beyond 500 episodes, scores change little, while BCQ and CQL show a slight upward trend in success rates.Source: §4.7 · p. 10, Figure 9 · p. 10

5 Related Work

Prior work on generalization largely targets online RL, while existing offline RL datasets generally lack train–test splits across distinct environment instances. This paper situates its benchmark against those gaps and earlier offline generalization studies.

5 Related Work

  • Online-RL generalization research explores regularization, representation learning, and other strategies, but these works consider online rather than offline RL.Prior offline generalization studies include contrastive-learning offline RL (Mazoure et al., 2022) and offline goal-conditioned RL.Source: §5 · p. 11
  • Prior offline generalization studies do not combine benchmarks across transition and reward functions with comparisons to sequence-modeling or transformer-policy approaches.The cited contrastive-learning study reports better generalization than other baselines in some cases, while the cited goal-conditioned study finds failures on unseen goals.Source: §5 · p. 11
  • Existing offline RL datasets generally contain trajectories from a single environment instance, unlike this benchmark’s related-but-distinct train and test scenarios.V-D4RL targets continuous-control robustness; NetHack supports procedural generalization but lacks a clear train–test split and also varies only transition functions.Source: §5 · p. 11

6 Conclusion

The paper finds that current offline learning methods generalize less well than online RL and introduces Procgen and WebShop datasets for studying that gap. It also finds that data diversity, rather than size alone, is associated with better generalization, and identifies future research directions and resource details.

6 Conclusion

  • Existing offline learning algorithms perform significantly worse on new environments than online RL methods.The conclusion says current offline algorithms developed without generalization in mind are not enough to tackle these challenges.Source: §6 · p. 12
  • The paper introduces offline-learning generalization datasets containing Procgen and WebShop trajectories to help address the lack of a benchmark.The authors state that the absence of such a benchmark has limited understanding of these algorithms’ real-world applicability.Source: §6 · p. 12
  • Increasing dataset diversity can improve generalization even without increasing dataset size, whereas size alone does not significantly improve it.The conclusion contrasts this with prior offline RL work in singleton environments.Source: §6 · p. 12
  • The authors propose combining offline RL with online generalization techniques, while noting that transformer policies’ limited context lengths impede direct use in complex environments such as WebShop.They also leave open the possibility that methods tailored to offline generalization will be needed.Source: §6 · p. 12

Appendix

  • The Appendix reports that code and datasets are open-sourced on GitHub under a CC-by-NC license, with READMEs describing reproduction and use.The repositories are separated into Procgen and WebShop folders.Source: Appendix · p. 20

A Limitations and Future Work

The authors identify limits in dataset scale, action-space coverage, WebShop generalization scope, and theoretical validation, while discussing the benchmark’s broader implications and related work. They suggest that risk-averse offline RL and fixed, insufficiently diverse data may help explain its generalization gap relative to behavioral cloning and online RL.

A Limitations and Future Work

  • The Procgen datasets contain 1 million transitions, a lighter alternative to larger datasets that reduces computational demands but leaves larger datasets and more efficient training strategies for future work.The authors note that scaling to 25 million transitions would require substantial compute, especially for sequence modeling.Source: A Limitations and Future Work · p. 20
  • All datasets use discrete actions, limiting coverage of continuous-action offline RL algorithms whose losses are nontrivial to adapt.The authors leave this conversion problem for future work.Source: A Limitations and Future Work · p. 20
  • WebShop experiments do not test cross-product generalization to product categories absent from training instructions.The authors identify this as a direction for subsequent studies.Source: A Limitations and Future Work · p. 20
  • The study evaluates generalization empirically rather than explaining it theoretically, and the cited quadratic-horizon error bound had only been empirically validated on singleton environments.The authors call for theory that better models generalization to new environments.Source: A Limitations and Future Work · p. 20

B Broader Impact

  • The benchmark’s simulated Procgen and WebShop settings are simplified relative to real-world applications, and the authors recommend extra safety precautions for high-stakes deployment.They do not foresee direct societal impact from results based on these simulations.Source: B Broader Impact · pp. 20–21

C Extended Related Work

  • Unlike multi-task and meta-RL approaches that fine-tune at test time, this benchmark targets zero-shot generalization and intra-game, inter-level transfer in Procgen.Its memory-efficient datasets are intended to support broader academic participation.Source: C Extended Related Work · p. 21

D Discussion

  • The authors hypothesize that offline RL’s risk aversion limits action selection in unfamiliar states, while BC may use learned representations to select actions from similar training states.They also suggest online RL benefits from broader state coverage through data collection; multiple PPO checkpoints alone may remain too sparsely sampled to cover the state space.Source: D Discussion · p. 21

E Dataset Details

The paper builds Procgen and WebShop datasets for offline generalization evaluation, then describes behavior-policy collection and algorithm-specific training and hyperparameter selection. Procgen varies procedurally generated levels, while WebShop uses text instructions and a restricted choice task.

E.1 Procgen

  • Procgen comprises 16 procedurally generated games with varying levels, and its collection, validation, and online evaluation use levels [0, 200), [200, 250), and [250, ∞), respectively.Each game can generate new levels with differing layouts, dynamics, and appearances.Source: §E.1 · pp. 21–22

E.2 WebShop

  • WebShop experiments focus on the choice task and use human demonstrations plus imitation-policy trajectories, with dataset sizes varying across numbers of episodes and instructions.The human dataset has 398 training episodes and 54 evaluation episodes; imitation-policy datasets range from 100 to 10,000 episodes.Source: §E.2 · pp. 22–23

F.1 Behaviour Policy

  • For Procgen behavior data, PPO policies train for 25 million steps on 200 levels, and expert and suboptimal datasets are formed from selected checkpoints.The suboptimal checkpoint reaches 50% of expert performance.Source: §F.1 · pp. 23–24

WebShop

  • The WebShop imitation-learning behavior policy is a 110M-parameter BERT model that scores available actions and achieves a test score of 59.9 and success rate of 29.1%.Its mean reward is 5.99.Source: WebShop · p. 24

F.2 Model Training

  • Offline baselines and BC share encoder types within each environment, and experiments run on a single NVIDIA V100 32GB GPU.Procgen uses ResNet encoders and WebShop uses BERT encoders.Source: §F.2 · p. 24

F.2.1 Procgen

  • Procgen experiments sweep algorithm-specific hyperparameters, select configurations using training and validation returns, and generally apply early stopping when validation fails to improve for 10 epochs.The 5M- and 10M-transition size experiments instead use three fixed epochs.Source: §F.2.1 · pp. 24–25

F.2.2 WebShop

  • WebShop experiments sweep BC, BCQ, and CQL hyperparameters, select the best configuration using validation and training goal levels, and use one trial per configuration.Final configurations are listed in Table 4.Source: §F.2.2 · p. 26, Table 4 · p. 26

G Training and Testing on Single Level in Procgen

On a single Procgen level, offline methods can learn from high-return demonstrations, and offline RL sometimes matches or exceeds BC and sequence modeling on suboptimal data.

G Training and Testing on Single Level in Procgen

  • In 9 of 16 games, offline RL on suboptimal level-1 data performs comparably to or better than BC and sequence modeling.The authors also report that offline RL can learn well from high-return demonstrations on one level, though some games remain difficult.Source: G Training and Testing on Single Level… · p. 26

H Effect of Data Diversity on Generalization: More Results

With dataset size fixed, adding Procgen training levels lowers the generalization gap for additional offline algorithms. The authors also compare validation-level selection strategies and recommend a consistent level-range practice.

H Effect of Data Diversity on Generalization: More Results

  • Increasing training levels while holding dataset size fixed lowers the generalization gap for BCT, CQL, and IQL.The reported trend is consistent with the earlier diversity experiment.Source: H Effect of Data Diversity on… · p. 27

I Scaling number of validation levels in Procgen

  • A proportional validation-level allocation correlates robustly with using 50 validation levels, and the authors recommend reserving [n, n + 50) when training uses [0, n).The comparison evaluates validation-level selection practices in Procgen.Source: I Scaling number of validation levels… · p. 27

J Effect of Data Size on Generalization: More Results

Scaling expert data from 1M to 10M transitions leaves the generalization gap almost constant for BCT, CQL, and IQL. On suboptimal data, performance remains poor even at 10M transitions, and the resulting policies do not generalize to the 200 training levels.

J Effect of Data Size on Generalization: More Results

  • On the 10M expert dataset, BCT, CQL, and IQL show an almost constant generalization gap as dataset size increases.The scaling experiment covers 10M expert and 25M suboptimal datasets.Source: J Effect of Data Size on… · p. 27
  • On the suboptimal dataset, all methods have poor aggregate train and test performance, even with 10M transitions.The passage contrasts this with prior works and describes the data as a subset of behavioral-policy training-log episodes.Source: J Effect of Data Size on… · p. 27
  • In this setup, offline policies trained on sampled suboptimal episodes do not generalize and do not perform well even on the 200 training levels.Source: J Effect of Data Size on… · p. 27

K More scaling analysis

This experiment holds episodes per level at 10 while varying the number of Procgen training levels, allowing datasets to exceed one million transitions. The generalization gap decreases slightly for BC, BCQ, DT, and BCT, but the trend resembles fixed-transition scaling.

K More scaling analysis

  • The experiment collects 10 episodes per level across 200, 400, 800, or 1000 levels, so datasets can exceed one million transitions.Each episode has at least three transitions.Source: K More scaling analysis · p. 27
  • The experiment is capped at 1000 seeds because larger datasets became too large to fit on a single GPU.Source: K More scaling analysis · p. 27
  • BC, BCQ, DT, and BCT show a slight decrease in generalization gap, following the trend observed with fixed 1M-transition datasets.Source: K More scaling analysis · p. 27
  • The authors conclude that the amount of data per level does not affect offline RL generalization performance in this setup.Source: K More scaling analysis · p. 27

L Per-game scores in Procgen

Across Procgen games, offline methods generally lag online PPO on unseen levels, while BC is often competitive with or better than offline RL and sequence modeling methods. Results also show that increasing data diversity improves test performance, whereas scaling dataset size generally leaves the generalization gap unchanged.

L Per-game scores in Procgen

  • BC is generally competitive with or better than offline RL and sequence modeling methods on test levels, while sequence-modeling methods perform similarly or slightly worse than offline RL on test.DT and BCT outperform offline RL on training environments, but their training gap over BC is small while their test gap is large.Source: L Per-game scores in Procgen · p. 27, Figure 22 · p. 40
  • On training levels, most methods match the training-data average in most games, while BC is the most robust across individual games.Figure 21 plots per-game training returns and Table 5 reports train performance for the 1M expert dataset.Source: L Per-game scores in Procgen · p. 27, Table 5 · p. 28, Figure 21 · p. 39
  • Across unseen Procgen levels, offline methods perform worse on average than PPO, and for more than half the games offline RL methods do not reach PPO’s test performance.The test comparison covers 16 Procgen games and both expert and mixed expert-suboptimal data regimes; Figure 22 plots per-game test returns, and Table 6 reports test performance for the 1M expert dataset.Source: L Per-game scores in Procgen · p. 28, Table 6 · p. 29, Figure 22 · p. 40
  • All sequence-modeling and offline RL methods fail to generalize on Miner, one of Procgen’s most stochastic games.Source: L Per-game scores in Procgen · p. 28
  • Increasing the number of training levels improves test performance and reduces the generalization gap, while scaling expert data from 1M to 10M transitions leaves the gap about the same.The diversity comparison uses 1M expert datasets; the size comparison fixes training levels at 200.Source: Figure 16 · p. 36, Figure 17 · p. 37
  • For suboptimal data, increasing dataset size from 1M to 10M transitions leaves test performance largely unchanged while train performance generally rises.All three methods have poor train and test performance, even at 10M transitions.Source: L Per-game scores in Procgen · p. 38

Related papers

Loading 2312.05742v2…