Source-linked AI summary
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
Cameron Voloshin, Hoang M. Le, Nan Jiang, Yisong Yue
TL;DR
Off-policy policy evaluation asks how to estimate a target policy’s value from logged data produced by other policies, a problem relevant to safety-critical reinforcement learning. The paper introduces COBS, a controllable and diverse benchmark for stress-testing OPE methods, and uses it to derive practical guidance. Its findings indicate that no single method consistently wins, because estimator accuracy depends on environmental factors and method class.
Problem
OPE lacks standardized empirical analysis despite its importance for evaluating reinforcement-learning policies from logged data in safety-critical settings.
Method
COBS systematically benchmarks OPE methods by controlling data-generation factors and testing diverse environments, including different policy mismatches, dimensionalities, and stochasticities.
Results
No single OPE method or method class consistently performs best; accuracy varies with environmental factors, while FQE and Qπ(λ)-based MAGIC are typically among the most reliable methods in the studied settings.
Takeaways & Limitations
OPE method selection should account for data availability, horizon, policy mismatch, domain complexity, and the underlying direct method rather than rely on a universal winner.
Takeaways & Limitations
The benchmark lacks short-horizon, high-dimensional settings and natural stochastic environments, leaving complex medium-horizon domains such as dialogue for future work.
Abstract
from arXiv · showhide
We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.
1 Introduction
The paper addresses off-policy policy evaluation (OPE), which estimates a target policy’s value from historical data generated by other policies. It introduces COBS, a diverse and controllable benchmark designed to evaluate OPE methods systematically and support reproducible comparisons.
- OPE estimates a target policy’s value using pre-collected data generated by other policies.
- COBS benchmarks OPE techniques through experimental designs that systematically vary factors influencing method performance.
- COBS enables fine-grained control over data generation, including the divergence between behavior and target policies.
- Its domains span diverse dimensionality, stochasticity, and representations to expose when and why OPE methods work well.
- The open software package interfaces with new environments and methods and supports large-scale OPE experiments.
- Compared with prior complementary benchmarks, COBS emphasizes reproducibility, nuanced experimental control, and systematic analyses across diverse environments.
2 Benchmarking Design & Methodology
COBS benchmarks OPE methods by varying the environmental, policy, data, and representation factors that influence performance. Its protocol combines diverse controllable domains, parameterized policies, repeated evaluation, and multiple comparison metrics.
- Design Factors: COBS varies horizon length, reward sparsity, stochasticity, behavior-policy knowledge, policy mismatch, and model misspecification as core OPE design factors.These factors can interact, so the benchmark does not treat them as independent determinants of performance.
- Domains: The suite includes eight environments spanning deterministic and stochastic settings, flexible horizons and sparsity, partial observability, and tabular, coordinate-based, and pixel-based representations.Examples include Graph, Graph-POMDP, Gridworld, Pixel-Gridworld, Mountain Car, Pixel Mountain Car, Tabular Mountain Car, and Atari Enduro.
- Experiment Protocol: COBS controls policy behavior through state-independent policies and ε-Greedy policies whose deviation from a learned policy is parameterized by ε.Policies remain stochastic so the required absolute-continuity condition is maintained.
- Experiment Protocol: Each experiment specifies an environment, behavior policy, evaluation policy, and trajectory count, estimates true value from 10,000 evaluation-policy rollouts, and repeats conditions 10 times.Method quality is assessed with Relative MSE and Near-top Frequency, which counts results within 10% of the best method.
- Implementation: COBS includes reference implementations and supports scalable experiments and integration of new domains and OPE techniques.Hyperparameters are held consistently across experimental conditions, while OPE tuning remains difficult because no proper validation signal is available.
- Baselines: The benchmark evaluates importance-sampling, direct, doubly robust, and hybrid OPE approaches, including eight direct estimators and 21 hybrid approaches.Recent methods can blur traditional category boundaries, such as Retrace(λ) and MAGIC.
3 Empirical Evaluation
COBS evaluates OPE methods across diverse conditions, showing that performance depends on interacting factors rather than one universally best estimator. The results support condition-specific method selection based on horizon, mismatch, stochasticity, representation, data, and behavior-policy knowledge.
- Benchmark design: 33 OPE methods were evaluated in thousands of experiments across 8 domains to systematically test factors influencing estimator performance.The benchmark varies environmental conditions and provides detailed results in the appendix.
- Overall findings: No single method or method class is consistently best because multiple environmental factors influence estimator accuracy.The authors therefore frame recommendations as aggregate top-performance guidelines rather than a universal ranking.
- Method classes: Among IPS estimators, PDWIS tends to perform best, while FQE, Qπ(λ), and IH are generally strongest among direct methods.FQE is more data-efficient when data is limited; Qπ(λ) benefits from more data, and IH is competitive for long horizons and high tabular policy mismatch.
- Method classes: Hybrid performance depends strongly on the underlying direct method: MAGIC often outperforms DR and WDR, with MAGIC using FQE or Qπ(λ) typically among the most reliable.WDR often outperforms standard DR, while MAGIC introduces additional partial-trajectory hyperparameters.
- Horizon and task structure: COBS distinguishes fixed finite-, infinite-, and indefinite-horizon tasks because estimator assumptions and required reward padding differ across these settings.Applying hybrid methods to indefinite-horizon tasks requires proper reward padding for their fixed-length trajectory assumptions.
- Environmental factors: High-dimensionality, long horizons, estimated behavior policies, and stochasticity can make direct methods outperform hybrid methods, sometimes substantially.In contrast, sufficient data or severe model misspecification can give hybrid methods consistent gains over direct methods.
- Environmental factors: Policy divergence can matter more than horizon length: extending horizon caused 10× accuracy degradation, whereas comparable policy-divergence growth caused 100× degradation.Direct methods generally hold up better than IPS as the policy gap increases, although importance-weighted direct methods also weaken.
- Representation mismatch: Model-based AM can be among the worst direct methods in high-dimensional settings because transition-model errors compound over long horizons.AM performs well in tabular large-data settings but tends to perform poorly with function approximation and complex domains.
4 Discussion and Future Directions
The discussion identifies scope gaps in current OPE benchmarking and outlines future directions, including broader benchmark settings, support for new estimators, and systematic method selection.
- Benchmark limitations: COBS lacks a natural short-horizon, high-dimensional benchmark and currently lacks natural stochastic environments in high-dimensional reinforcement learning.The authors suggest medium-horizon complex domains such as dialogue as potential future benchmarks.
- Other OPE settings: The discussion also highlights missing-data coverage, confounding variables, and strategic environmental behavior as practically relevant OPE settings for future benchmarking.These settings challenge full-support, observed-decision, and single-agent MDP assumptions, respectively.
- Estimator coverage: Current benchmarks do not include all newly proposed OPE estimators, although COBS is modular enough to accommodate new estimators and environments.The discussion mentions DICE-family and double reinforcement learning estimators as examples of methods for future evaluation.
- Method selection: COBS can distill general method-selection guidelines, but systematic selection remains difficult because decision criteria such as misspecification, stochasticity, and data sufficiency are hard to judge.The authors identify systematic model selection as an important missing piece given variability among existing techniques.
- Terminology: The paper points readers to a glossary of terminology used throughout the study.Table 2 provides the terms used in the paper.
B Ranking of Methods
The supplementary ranking aggregates how often estimators are within 10% of the lowest Relative MSE across experimental conditions and organizes results by decision-tree regimes.
- Ranking criterion: A method within 10% of the lowest Relative MSE is counted as a top method, and these near-top occurrences are aggregated across experiments.Table 3 lists how often methods appear within 10% of the best method.
- Decision-tree regimes: Tables 4–11 provide numerical support for the decision tree by covering combinations of specification, representation, horizon, policy mismatch, and behavior-policy estimation.Each table corresponds to a child node ordered from left to right in the decision tree.
- Properly specified settings: The ranking includes properly specified settings with short or long horizons and small or large policy mismatch.These regimes are represented by Tables 4–8, including separate deterministic and stochastic long-horizon conditions.
- Potentially misspecified settings: The ranking also includes potentially misspecified settings with insufficient representation or with sufficient representation paired with poor or good behavior-policy estimates.These conditions are represented by Tables 9–11.
C Supplementary Folklore Backup
The supplementary analysis compares horizon and policy difference while holding policy mismatch constant, using baseline and modified table conditions to support the paper’s empirical guidelines.
- Observed comparison: Despite identical policy mismatch, increasing πe to 0.9 affects error more than increasing the horizon from T = 10 to T = 100 in the reported comparison.The longer-horizon condition is compared against the baseline with fixed πe, while the policy-difference condition fixes the horizon.
- Experimental comparisons: Tables 12–14 compare relative MSE under a baseline, a longer horizon, and a larger evaluation-policy difference.The baseline uses T = 10 and πe(a = 0) = 0.1246; the variants use T = 100 or πe(a = 0) = 0.9.
- Guideline support: The supplementary tables provide numerical support for the decision tree and its model-selection guidelines.Table 15 is identified as the model-selection-guidelines table, while Table 3 defines the near-top-frequency support elsewhere.
E Methods
The methods appendix introduces the tested-method descriptions after defining the adjusted horizon quantity ˜T = T −1.
- Method notation: The appendix begins its method descriptions by defining the adjusted horizon quantity ˜T = T −1.This definition precedes the descriptions of the methods tested.
E.1 Inverse Propensity Scoring (IPS) Methods
Table 16 presents the inverse propensity scoring (IPS) methods used in the study and cites their sources.
- Table 16 identifies the study’s IPS methods.
- The IPS-method list is presented in tabular form.
- The table attributes the IPS methods to references [15, 26].
E.2 Hybrid Methods
The hybrid-method section describes methods that combine action-value estimates with direct or doubly robust estimation. MAGIC selects weighted blends to trade off these approaches.
- Hybrid methods use an action-value function bQ to obtain an estimate of bV(x).
- Weighted Doubly-Robust is included as a hybrid-method variant.
- MAGIC is a weighted average of blends combining direct methods with Hybrid estimation.
- MAGIC chooses weights that trade off direct-method estimates against Hybrid estimates across trajectory steps.For some i, the first i steps use DR or WDR and the remaining steps use bQ.
E.3 Direct Methods (DM)
Direct methods estimate policy value by modeling dynamics and rewards or by approximating action values. The benchmark applies these methods across varied environments, horizons, representations, and policy-observation settings.
- Approximate Model fits transition dynamics, rewards, and terminal conditions, then estimates policy value from simulated rollouts.
- The direct-method estimators approximate Q with a parameterized function bQ(·; θ).
- Direct Model Regression is presented as a direct-method approach.
- MRDR and State Density Ratio Estimation are included among the evaluated estimators.
- Environment design: Experiments use fixed horizons with absorbing states after goal achievement or at the horizon limit.
- Environment design: The benchmark spans graph, partially observable, gridworld, Mountain Car, pixel-based, and Enduro environments.
G Experimental Setup
The experiments vary policy structure, environment parameters, representations, implementation choices, and evaluation conditions across a Cartesian-product benchmark. The setup also documents practical constraints, including unavailable validation for OPE hyperparameters and compute exclusions.
- Policies: Graph-family policies use state-independent probabilities for the two actions, with p varied across experiments.
- Policies: ε-greedy policies vary deviation from a trained Q* policy through ε.
- Parameterization: Each environment’s experiments are formed as the Cartesian product of its listed parameters.
- Implementation: Simpler environments use tabular representations, while function-approximation settings fit state-difference dynamics for AM.
- Implementation: Pixel-based environments use convolutional neural networks, while MC also includes linear and dense neural-network function classes.
- Reproducibility: Datasets are omitted because COBS is simulation-based, with datasets recreated by selecting environment parameters.
- Practical constraints: Hyperparameter search is infeasible because OPE lacks proper validation, creating a trade-off between computational cost and accuracy.
- Evaluation conditions: Multi-step rollouts for AM, Retrace(λ), Qπ(λ), and Tree-Backup(λ) exceeded the compute budget in the Enduro comparison.
I Complete Results
Complete experiment results are available on the COBS GitHub page.
- The COBS GitHub page provides tables containing the complete experimental results.