PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarks for Deep Off-Policy Evaluation

Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, Tom Le Paine

arXiv:2103.16596v1cs.LGstat.ML

TL;DR

Off-policy evaluation lacks a unified benchmark with difficult tasks, making comparisons between methods challenging. The paper presents DOPE, a benchmark combining challenging tasks, varied policies, and datasets, and evaluates existing algorithms. No evaluated method performs consistently well across tasks, and performance depends on the metric and data setting.

  • Problem

    Off-policy evaluation lacks a comprehensive, unified benchmark and difficult evaluation tasks for comparing methods and measuring progress.

    Source: §1 · pp. 1–2

  • Method

    DOPE combines challenging high-dimensional continuous-control tasks with offline datasets and varied policies, and evaluates methods across multiple metrics and data-coverage settings.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    No existing algorithm consistently performs well across tasks, and performance varies by metric; rank correlation is 0.69 on DOPE RL Unplugged versus 0.25 on DOPE D4RL.

    Source: §5.2 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    DOPE provides a platform for studying policy evaluation and selection across tasks and datasets, including how data distributions affect performance.

    Source: §7 · p. 9

    The benchmark focuses on the purely offline setting without logged behavior-policy propensities, estimating propensities with behavior cloning for methods that require them.

    Source: §2 · p. 2

Abstract

from arXiv · show

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between papers is difficult because currently there is a lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets can be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of state-of-the-art algorithms and provide open-source access to our data and code to foster future research in this area.

1 INTRODUCTION

Offline reinforcement learning depends on evaluating policies from data when online collection may be costly, dangerous, or time-consuming. The paper introduces DOPE, a standardized benchmark designed to compare OPE methods on challenging tasks with varied policies and data conditions.

1 INTRODUCTION

  • Offline policy evaluation estimates policy performance solely from offline data, supporting high-confidence guarantees before deployment and policy improvement or model selection.Online data collection may be costly, dangerous, or time-consuming in real-world settings.Source: §1 · p. 1
  • The benchmark targets the lack of broadly accepted evaluation tasks combining complex, high-dimensional problems with standardized protocols and metrics.It emphasizes large datasets, long horizons, and task complexity to support development of scalable methods for high-dimensional problems.Source: §1 · pp. 1–2
  • DOPE benchmarks OPE methods on challenging control tasks, varied policy values, and ideal and adversarial data-coverage settings, with datasets, target policies, an evaluation API, and recorded algorithm results.The benchmark includes 10 to 96 policies per task and covers policy evaluation, ranking, and selection.Source: §1 · p. 2

2 BACKGROUND

The paper frames off-policy evaluation as estimating a policy's value when data come from another behavior policy, and focuses on the purely offline case. The setup also distinguishes estimating value from selecting among candidate policies.

2 BACKGROUND

  • A policy's value is its expected sum of discounted rewards, while OPE estimates that value using data collected from a separate behavior policy.The offline case uses a historical dataset.Source: §2 · p. 2
  • The paper studies the purely offline setting, using trajectories from a behavior policy and focusing on methods that do not require logged behavior-policy propensities.For methods requiring propensities, the benchmark estimates them with behavior cloning.Source: §2 · p. 2
  • OPE may estimate one policy's value or select the best policy from a candidate group; Figure 1 contrasts these related tasks.Policy selection reflects the use of OPE to choose among strategies when online evaluation is too expensive.Source: §2 · pp. 2–3, Figure 1 · p. 2

3 DOPE: DEEP OFF-POLICY EVALUATION

DOPE is designed as an accessible benchmark of challenging OPE tasks, with properties intended to expose method shortcomings. Its protocol evaluates value accuracy as well as policy ranking and selection.

3 DOPE: DEEP OFF-POLICY EVALUATION

  • The benchmark is designed to challenge existing OPE methods while keeping tasks simulatable and accessible to evaluate.Source: §3 · p. 3

3.1 TASK PROPERTIES

  • DOPE includes high-dimensional continuous-space tasks, where value estimation can require high-capacity models and large datasets with broad state coverage.The benchmark targets settings where feature engineering is difficult, including robotics and autonomous driving.Source: §3.1 · p. 3
  • Long-horizon tasks challenge OPE through credit assignment, and increasing state or action dimensions can worsen that difficulty.Source: §3.1 · p. 3
  • Sparse-reward tasks add credit-assignment and exploration challenges that may interact with offline data coverage; DOPE includes difficult robotics and navigation tasks.Source: §3.1 · p. 3
  • Two navigation tasks require hierarchical decisions, combining high-level planning with low-level control.Source: §3.1 · p. 3

3.2 EVALUATION PROTOCOL

  • DOPE provides policies with varying values and metrics for policy evaluation, ranking, and selection, addressing the limits of evaluating methods only by point-estimate error.Source: §3.2 · p. 3
  • The protocol measures estimate accuracy with absolute error, policy-ranking quality with rank correlation, and selection quality with Regret@k.Absolute error is used instead of MSE to be robust to outliers; Regret@k compares the best estimated top-k policy with the best policy overall.Source: §3.2 · pp. 3–4

4 DOMAINS

DOPE combines RL Unplugged and D4RL tasks to examine OPE across different settings. RL Unplugged controls for ideal data coverage, while D4RL includes varied coverage and greater distribution shift.

4 DOMAINS

  • DOPE combines RL Unplugged and D4RL to provide OPE evaluation across task settings, including controlled coverage and more practical data conditions.Source: §4 · p. 4

4.1 DOPE RL UNPLUGGED

  • RL Unplugged uses online-RL-generated data and policies trained on the evaluation dataset, supporting study of task complexity while controlling for ideal data coverage.Source: §4 · p. 4

4.2 DOPE D4RL

  • D4RL draws on varied data sources and online-RL policies, creating varying coverage and less similar behavior and evaluation-policy distributions than RL Unplugged.Source: §4 · p. 4

4.1 DOPE RL UNPLUGGED

  • Its DeepMind Control Suite tasks span Cartpole swingup to Humanoid run, with state observations ranging from 5 to 67 dimensions.Source: §4.1 · p. 4
  • Four offline RL algorithms and behavior cloning are trained across tasks and hyperparameters, with three seeds and policy snapshots recorded at increasing intervals.Source: §4.1 · p. 4

4.2 DOPE D4RL

  • D4RL includes Gym-MuJoCo, maze-navigation, and sparse-reward Adroit tasks, with datasets and policies spanning different sources and performance levels.Source: §4.2 · pp. 4–6

5 BASELINES AND RESULTS

The evaluation compares six OPE methods across DOPE tasks and finds that none reaches near-oracle performance across the reported metrics. Performance varies by benchmark setting, metric, task, and dataset.

5 BASELINES AND RESULTS

  • The evaluation aims both to establish reference results for existing algorithms and to identify shortcomings that suggest future research directions.Source: §5 · p. 6

5.1 BASELINES

  • The six evaluated baselines span FQE, learned-dynamics model-based evaluation, importance sampling, doubly robust estimation, DICE, and the Variational Power Method.FQE bootstraps from Q(s′, π(s′)); model-based evaluation uses simulated policy trajectories, while IS and doubly robust methods use a learned behavior policy.Source: §5.1 · pp. 6–7

5.2 RESULTS

  • No evaluated algorithm attains near-oracle performance on absolute error, regret, or rank correlation across DOPE RL Unplugged and D4RL.The overall mean results are shown in Figures 4 and 5; the authors note that finite datasets make oracle performance unlikely.Source: §5.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
  • On RL Unplugged, model-based methods MB-AR and MB-FF and direct-value methods FQE-D and FQE-L2 outperform VPM, DICE, and IS across all metrics.Performance also differs among MB and FQE variants on some tasks, including by model architecture and implementation.Source: §5.2 · pp. 7–8
  • On D4RL, FQE-L2 performs best on absolute error and Regret@1, while IS and DR outperform it on rank correlation; rank correlation is 0.69 on RL Unplugged versus 0.25 on D4RL.The authors characterize RL Unplugged as having more ideal conditions than D4RL.Source: §5.2 · p. 8
  • Task-level results include exceptions: IS is among the best methods on humanoid run, and MB-AR and FQE-D differ across selected tasks.Scatter plots show MB-AR assigning similar values across policies when it performs worse, while FQE-D severely overestimates poor policies.Source: §5.2 · p. 8, Figure 7 · p. 8
  • D4RL dataset choice significantly changes OPE performance in Gym-MuJoCo, Adroit, and AntMaze, and no tested method is reliably robust to dataset variation.On AntMaze, no algorithm achieves good absolute error; IS is the only method with correlation consistently above zero.Source: §5.2 · p. 8

6 RELATED WORK

Prior OPE work spans several method families and applications, but evaluation metrics and tasks are not standardized. DOPE builds on offline-RL benchmarks and differs from an earlier OPE benchmark in task types, data-support settings, metrics, and standardized datasets.

6 RELATED WORK

7 CONCLUSION

DOPE provides a platform for studying policy evaluation and selection across challenging tasks and datasets. Existing methods do not perform consistently across tasks, and their relative performance depends on the metric and use case.

7 CONCLUSION

  • DOPE combines multiple datasets and policies to study how data distributions affect OPE and to evaluate metrics relevant to offline policy selection.Source: §7 · p. 9
  • No existing algorithms consistently perform well across all tasks, and methods that perform poorly on absolute error may do better on correlation.The metric-dependent results can inform method choice for policy evaluation versus policy selection.Source: §7 · p. 9
  • Reproducible and accessible protocols for evaluating with real-world datasets remain an important challenge for future work.Source: §7 · p. 9

A.1 METRICS

The section defines three complementary OPE metrics: absolute error, Regret@k, and rank correlation.

A.1 METRICS

  • Absolute error measures the difference between a policy’s true value and estimated value, and is used for robustness to outliers.The true value is V π and the estimate is ˆV π.Source: §A.1 · p. 14
  • Regret@k compares the best policy overall with the best policy among the top-k policies selected by estimated values.The top-k set is chosen using estimated policy values.Source: §A.1 · p. 14
  • Rank correlation, also called Spearman’s ρ, measures correlation between the rankings induced by estimated and true policy values.Source: §A.1 · p. 14

A.2 DETAILED RESULTS

The appendix presents detailed results by task in tables and charts, alongside scatter plots comparing estimated and ground-truth returns for every policy.

A.2 DETAILED RESULTS

  • Results are organized by task in both tabular and chart form, with scatter plots comparing each policy’s estimated and ground-truth returns.Source: §A.2 · p. 14
  • The scatter plots compare estimated returns against ground-truth returns for every policy.Source: §A.2 · p. 14
  • The results presentation includes both tables and charts.Source: §A.2 · p. 14

A.2.1 CHART RESULTS

The chart-results section presents normalized results by algorithm and task, with separate figures for absolute error, rank correlation, and Regret@1.

A.2.1 CHART RESULTS

  • Figure A.1 reports absolute error for each baseline algorithm across the RL Unplugged tasks considered.Source: Figure A.1 · p. 14
  • Figure A.2 reports rank correlation for each baseline algorithm across the RL Unplugged tasks considered.Source: Figure A.2 · p. 15
  • Figure A.3 reports Regret@1 for each baseline algorithm across the RL Unplugged tasks considered.Source: Figure A.3 · p. 15
  • Figure A.4 reports absolute error for each baseline algorithm across the D4RL task domains considered.Source: Figure A.4 · p. 15
  • Figure A.5 reports rank correlation for each baseline algorithm across the D4RL task domains considered.Source: Figure A.5 · p. 15
  • Figure A.6 reports Regret@1 for each baseline algorithm across the D4RL task domains considered.Source: Figure A.6 · p. 16

A.2.2 TABULAR RESULTS

The tabular results report means and standard deviations across three seeds, while the following scatter plots compare estimated and true policy returns.

A.2.2 TABULAR RESULTS

  • The tabular results report means and standard deviations across three seeds.Source: §A.2.2 · p. 16
  • Tables report average absolute error, Spearman rank correlation, and normalized Regret@5 at a discount factor of 0.995.Values not significantly different from the best are bolded in each table, and methods are ordered by median.Source: Table A.1 · p. 16, Table A.2 · p. 17, Table A.3 · p. 17

A.2.3 SCATTER PLOTS

Related papers

Loading 2103.16596v1…