Source-linked AI summary
Hyperparameter Selection for Offline Reinforcement Learning
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, Nando de Freitas
TL;DR
Offline RL needs hyperparameter selection that preserves its logged-data-only assumption, because online evaluation of candidate policies is often infeasible. This paper empirically studies offline ranking of policies across hyperparameter settings and finds that careful choices of algorithm, Q estimator, and ranking statistic support strong selection across challenging tasks.
Problem
Offline RL hyperparameter selection must choose among candidate policies using logged data rather than online environment interaction.
Method
The paper conducts a broad empirical study of offline hyperparameter selection using offline statistics to rank policies across hyperparameter settings.
Results
Carefully choosing the offline RL algorithm, Q estimator, and ranking statistic yields a strong selection strategy across challenging tasks, including visual humanoid control.
Takeaways & Limitations
Algorithms close to the behavior policy, FQE Q estimates, and initial-state value estimates can support offline hyperparameter selection in the studied tasks.
Takeaways & Limitations
The study does not establish how much data is sufficient, how data-generating policy type affects selection quality, or how to generally tune FQE hyperparameters, and its experiments use simulation.
Abstract
from arXiv · showhide
Offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. However, existing hyperparameter selection methods for offline RL break the offline assumption by evaluating policies corresponding to each hyperparameter setting in the environment. This online execution is often infeasible and hence undermines the main aim of offline RL. Therefore, in this work, we focus on \textit{offline hyperparameter selection}, i.e. methods for choosing the best policy from a set of many policies trained using different hyperparameters, given only logged data. Through large-scale empirical evaluation we show that: 1) offline RL algorithms are not robust to hyperparameter choices, 2) factors such as the offline RL algorithm and method for estimating Q values can have a big impact on hyperparameter selection, and 3) when we control those factors carefully, we can reliably rank policies across hyperparameter choices, and therefore choose policies which are close to the best policy in the set. Overall, our results present an optimistic view that offline hyperparameter selection is within reach, even in challenging tasks with pixel observations, high dimensional action spaces, and long horizon.
1. Introduction
Offline RL is motivated by real-world settings where online interaction is challenging, risky, or expensive, but selecting hyperparameters still requires an offline way to rank policies. This study empirically evaluates such selection using logged data and identifies choices that support reliable ranking across challenging domains.
- Offline RL learns policies from logged data for applications where arbitrary environment interaction is challenging, risky, or expensive.
- Offline hyperparameter selection must rank policies produced by different hyperparameter settings without evaluating them through online environment interaction.
- The study evaluates scalable offline selection metrics across domains with high-dimensional actions and observations, long horizons, and hyperparameters spanning architectures, optimizers, and loss functions.
- Experiments confirm that offline RL algorithms are not robust to hyperparameter choices, motivating reliable offline hyperparameter selection.
- Algorithms that stay near the behavior policy, FQE-based Q estimates, and average critic values for initial states improve policy evaluation and ranking.
- Combining these choices yields a strong offline hyperparameter selection strategy across several challenging tasks.
2. Offline Hyperparameter Selection
Offline hyperparameter selection chooses among policies trained with different hyperparameters using only logged data, rather than evaluating each policy through environment interaction. The paper distinguishes this goal from offline policy evaluation and studies statistics and metrics for ranking policies and selecting those close to the best.
- Motivation: Offline hyperparameter selection must choose among policies using only logged data, preserving offline RL’s no-interaction assumption.The practical workflow trains policies under several hyperparameter settings, computes offline summary statistics, and selects the top k for possible execution.
- Motivation: Unlike offline policy evaluation, offline hyperparameter selection primarily needs to rank policies or identify one close in value to the best policy.OPE targets accurate value estimation, whereas OHS can use rank correlation or low regret as its objective.
- Scope: The paper defines hyperparameter selection as choosing the best policy from a given set, whereas hyperparameter tuning also searches the hyperparameter space.This distinction limits the study to selecting among already trained policies.
- Offline statistics: Policy-ranking statistics are computed from critics obtained either during offline RL training or by retraining with Fitted Q Evaluation.The resulting critic and dataset are used to calculate scalar statistics for each policy without environment interaction.
- Offline statistics: The main statistics are the estimated value from initial states, V̂(s0), and Soft OPC, which contrasts critic values on successful versus all dataset transitions.V̂(s0) estimates the evaluation policy’s value from the initial-state distribution; Soft OPC uses a success threshold and critic-value difference.
- Evaluation metrics: Evaluation uses Spearman rank correlation, Regret @ k, and absolute error to measure ranking quality, near-best selection, and value-estimation accuracy.The actual discounted return is estimated by running each policy in the environment and serves as ground truth for these metrics.
3. Experimental Setup
The experiments compare three offline RL algorithms, offline policy evaluation with FQE, and hyperparameter settings across challenging tasks. Policies are assessed using learned critics and environment returns, with Figure 4 illustrating overestimation patterns.
- Tasks and algorithms: BC and CRR regress toward logged actions, whereas D4PG directly optimizes critic estimates without behavior-policy regularization.Because the algorithms share the same value objective and update, their policy objectives isolate the effect of behavior-policy proximity.
- Hyperparameters and evaluation: FQE re-evaluates each fixed policy using the same value-function updates and dataset, providing offline value estimates for hyperparameter selection.FQE is chosen for simplicity and scalability and is run 256 times per task.
- Hyperparameters and evaluation: The study varies model architecture, optimizer, and loss-function hyperparameters, producing 256 policies per task.The search includes hidden size, number of blocks, learning rate, learner steps, algorithm, and loss-term beta.
- Hyperparameters and evaluation: Actual policy values are obtained from 100 environment episodes and compared with both ORL and FQE statistics.Figure 4 shows that ORL estimates generally overestimate values, while FQE reduces overestimation, although D4PG remains the most divergent.
4. Results
Hyperparameter choices substantially affect offline RL performance, while policy ranking improves when evaluation uses FQE-derived estimates and initial-state value statistics. Policies from behavior-constrained algorithms are generally easier to rank than D4PG policies.
- Hyperparameter choice materially changes offline RL policy performance across the evaluated tasks.The experiments examine the resulting range of actual policy values and assess when offline statistics rank those choices well.
- Overestimation: ORL initial-state value estimates overestimate policy quality, sometimes making D4PG appear best when it is actually worst.The strongest failures occur in humanoid environments, where the ranking reverses the actual policy ordering.
- Overestimation: Overestimation is greatest for D4PG, followed by CRR and BC, and is generally highest in DM Locomotion.BC and CRR remain closer to the behavior policy, which may make their values easier to estimate from logged data.
- Overestimation: FQE re-evaluation significantly reduces overestimation across algorithms and tasks, although substantial overestimation remains for D4PG on DM Locomotion.The same reduction is summarized across domains and algorithms in Figure 5.
- Ranking Quality: OPE initial-state values generally rank policies better than ORL values, especially for BC and CRR, while D4PG and cross-algorithm rankings remain difficult on DM Locomotion.For BC and CRR, OPE initial-state-value rank correlation exceeds 0.9 on most tasks.
- Ranking Quality: Initial-state value ranking outperforms Soft OPC in both rank correlation and regret, with OPE values especially improving D4PG regret.BC and CRR usually have low regret, often near zero and below the regret of selecting the median-value policy.
- FQE Sensitivity: FQE appears relatively stable to learner-step count, but its hyperparameter sensitivity was not exhaustively investigated and tuning remains open.The reported stability does not extend to a complete account of FQE hyperparameter selection.
5. Related Work
Related work spans offline RL, hyperparameter tuning, offline model selection, and off-policy evaluation. FQE is highlighted as a simple evaluation approach that avoids importance sampling and learned transition models, though prior studies used simpler settings or did not directly address model selection.
- Offline RL methods learn policies from logged data, motivated by settings where environment interaction is challenging, risky, or expensive.Examples include robotics, self-driving, healthcare, and dialogue.
- Hyperparameter Tuning: General hyperparameter-tuning methods include random search, Bayesian optimization, and population-based evolutionary optimization.These methods have been studied across machine learning and difficult application domains.
- Offline Model Selection: Offline hyperparameter tuning has received relatively little attention, with prior work addressing offline model selection, policy classification, or evaluation protocols on separate tasks.These approaches differ in scope and evaluation setting from direct selection among policies trained on the available task.
- Off-Policy Evaluation: Importance-sampling OPE methods can have very high variance and are difficult to apply in continuous action spaces because they depend on action-probability ratios.Model-based OPE avoids those ratios but can be biased when learned dynamics do not accurately capture the environment, especially with pixel observations.
- Off-Policy Evaluation: FQE evaluates a policy by applying the policy-evaluation step of a policy-improvement algorithm without importance sampling or a learned transition model.It can be viewed as a special case of Retrace with T = 0 and c = 1.
- Off-Policy Evaluation: Prior empirical work found FQE surprisingly effective, but used simpler environments and datasets and focused mainly on MSE rather than model selection.This paper extends the evaluation setting toward challenging tasks and direct policy selection.
6. Future Work
Future work must address FQE hyperparameter selection and broader reliability conditions for offline hyperparameter selection. The current study also evaluates only FQE and simulated environments.
- Selecting FQE hyperparameters remains an open problem despite reported robustness to the number of learner steps.The paper does not exhaustively investigate FQE sensitivity.
- The study evaluates a single OPE method, FQE, leaving importance-sampling, model-based, and other OPE methods for future scaling to these challenging problems.The choice of FQE was motivated by its value-function-only formulation and expected scalability.
- Future studies should examine whether data quantity and the policy generating the logged data affect reliable hyperparameter selection.These factors were not investigated in the current work.
- The experiments use only simulated environments because ground-truth values for many robot-learned policies are difficult to obtain.This limits direct evaluation on physical systems.
7. Conclusions
Careful choices of offline RL algorithm, Q estimator, and ranking statistic support offline hyperparameter selection across challenging tasks. Error patterns vary by algorithm, task domain, and whether statistics come from ORL or OPE critics.
- Conclusions: Using behavior-policy-constrained algorithms, FQE, and ˆV (s0) provides a strong offline hyperparameter-selection strategy across the considered tasks.This includes DM Locomotion, which controls a 56-degree-of-freedom humanoid from egocentric visual input.
- Conclusions: Statistics have highest absolute error on D4PG, followed by CRR and BC, and on DM Locomotion, followed by Manipulation Playground and DM Control Suite.OPE statistics have lower error than ORL statistics.
B. Additional Ranking Results
Additional policy statistics show that avg_q behaves similarly to ˆV (s0), while td_err performs poorly for ranking policies. These patterns appear in both rank correlation and normalized regret@5.
- Additional statistics: avg_q performs similarly to ˆV (s0), whereas td_err performs quite poorly for offline policy ranking.avg_q is computed across dataset states, while td_err averages temporal-difference error across dataset transitions.
- Rank correlation: avg_q follows similar trends to ˆV (s0) but is slightly worse, while td_err remains poor overall in rank-correlation comparisons.The comparisons use actual values and additional policy statistics from ORL and OPE critics.
- Regret@5: Rank correlation trends for avg_q and td_err are mirrored by normalized regret@5 across policies.The regret@5 comparison uses actual values and statistics from ORL and OPE critics.
C. Fitted Q Evaluation without Distributional Critic
FQE estimates with and without distributional critics generally fall within a similar range. Distributional critics produce a few high-value outliers associated with experiments terminated early.
- Distributional critics: FQE estimates with and without distributional critics generally fall within a similar range.The comparison was conducted on DM Control Suite tasks.
- Distributional critics: High-value outliers occur only for FQE with a distributional critic and correspond to experiments terminated early.FQE without a distributional critic does not show these outliers.
D. Fitted Q Evaluation Code
The FQE implementation iterates through dataset transitions, computes critic losses and gradients, applies optimizer updates, and periodically updates target parameters. The listing is a simplified but functional version of the code.
- FQE update: The FQE code processes dataset transitions and computes a categorical critic loss using current and target critic values.Terminal states use zero discount during target construction.
- FQE update: The implementation computes gradients and applies them through the optimizer to update trainable variables.The listing explicitly marks gradient computation and optimizer application steps.
- Target updates: Target-network parameters are updated periodically by iterating over source and destination variables.The update is triggered when num_steps is divisible by target_update_period.
- Implementation: The listing is a simplified, functional version of the FQE code written using TensorFlow 2 and Acme.The copyright and Apache-2.0 license identifiers are included in the listing.
E. Dataset Details
The paper uses datasets spanning DM Control Suite, manipulation tasks, and DM Locomotion, with episodes retained across full training runs to include diverse outcomes. Dataset sizes are reported in Table 3.
- Dataset sizes: Dataset sizes are measured and reported as numbers of episodes in Table 3.
- DM Control Suite: DM Control Suite datasets come from three independent D4PG runs, retaining episodes from entire training runs for greater diversity.Unlike Gulcehre et al. (2020), successful episodes were not filtered out, and the datasets are larger.
- Manipulation tasks: Manipulation-task datasets contain 8,000 episodes per task from three independent D4PGfD runs assisted by 100 human demonstrations.The episodes include both successful and unsuccessful outcomes.
- DM Locomotion: DM Locomotion datasets retain episodes from three policies trained per task, subsampled to include both successful and failed episodes.The datasets are larger than those in Gulcehre et al. (2020) because successful episodes were not filtered out.
- Dataset-generation agents: D4PGfD is a DDPGfD variant augmented with distributional critics.