PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Value Learning Really the Main Bottleneck in Offline RL?

Seohong Park, Kevin Frans, Sergey Levine, Aviral Kumar

arXiv:2406.09329v2cs.LGcs.AI

TL;DR

Offline RL is expected to learn effectively from lower-quality data than imitation learning, yet often underperforms it, leaving its performance bottlenecks unclear. The paper empirically studies value learning, policy extraction, and policy generalization, finding that extraction choices and test-time generalization often constrain performance. It reports practical improvement from behavior-regularized policy gradients and test-time policy improvement approaches.

  • Problem

    Offline RL should be able to learn effective policies from suboptimal data, but often performs worse than imitation learning and its limiting factors are unclear.

    Source: §1 · p. 1

  • Method

    The paper empirically analyzes value learning, policy extraction, and test-time generalization using data-scaling properties across algorithmic choices.

    Source: §1 · p. 2

  • Results

    Policy extraction and test-time generalization often limit offline RL performance more than value learning, with behavior-regularized policy gradients such as DDPG+BC often outperforming value-weighted regression such as AWR.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The authors recommend diverse data and policy procedures that use the learned value function effectively, including appropriate extraction objectives and possible test-time improvement strategies.

    Source: §7 · p. 13

    Test-time policy accuracy cannot be guaranteed without additional coverage or structural assumptions because the test-time state distribution may be arbitrarily adversarial.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.

1 Introduction

Offline RL is expected to learn from lower-quality data than imitation learning, but often underperforms it, leaving its practical bottleneck unclear. The study finds that policy extraction and test-time generalization often limit performance more than value learning.

1 Introduction

  • Imitation learning requires high-quality demonstrations, whereas offline RL can in principle learn effective policies from suboptimal data, yet often performs worse in practice.The paper frames this performance gap as a motivation for identifying what holds back offline RL.Source: §1 · p. 1
  • The study examines value estimation, policy extraction, and test-time generalization through data-scaling analyses that vary data size, quality, and coverage.It studies three value-learning methods and three policy-extraction methods across diverse environments.Source: §1 · p. 2
  • Policy extraction choice often affects performance more than value learning, with behavior-regularized policy gradients such as DDPG+BC [14] outperforming value-weighted regression such as AWR [45] [57].The paper reports better performance and data scaling for DDPG+BC across many settings.Source: §1 · p. 2
  • Offline RL is often bottlenecked by policy generalization to test-time states, rather than policy performance on training states.The study proposes high-coverage datasets and test-time policy extraction techniques, including methods that distill value-function information into the policy during evaluation rollouts.Source: §1 · p. 2
  • The paper concludes that improving policy extraction and test-time generalization, alongside value learning, is important for translating offline RL advances into performant policies.It recommends suitable policy extraction and generalization strategies, including test-time training with the value function.Source: §1 · p. 2

2 Related work

The paper studies practical bottlenecks in offline RL rather than proposing a new algorithm. Its holistic, diverse-environment analysis extends prior component studies by considering interactions among value learning, policy extraction, and generalization.

2 Related work

  • Offline RL methods estimate value functions from static data and then use policy extraction methods such as behavior-regularized policy gradients [14] [33], weighted behavioral cloning [45], or sampling-based action selection [7] [15] [20].Value-learning and policy-extraction stages may be interleaved or decoupled, depending on the algorithm.Source: §2 · p. 3
  • The paper aims to analyze current offline RL techniques to inform future methodological development, rather than propose a new algorithm.The authors position their analysis as informing practitioner choices about which components to improve.Source: §2 · p. 3
  • Unlike prior studies of individual components, this analysis examines interactions among value learning, policy extraction, and generalization across diverse environments.The authors contrast their scope with Fu et al., whose conclusions use D4RL locomotion tasks and do not consider data-scaling properties or identify policy generalization as a bottleneck.Source: §2 · p. 3

3 Main hypothesis

The paper hypothesizes that policy learning is often a greater offline RL bottleneck than value-function accuracy alone. It tests this by separately analyzing value learning and policy extraction, then test-time generalization.

3 Main hypothesis

  • The paper identifies three possible bottlenecks: imperfect value estimation, imperfect policy extraction from the learned value, and imperfect generalization to evaluation-time states.The authors state that perfect value learning, extraction, and test-time generalization would yield optimal performance under a dataset.Source: §3 · p. 3
  • Policy learning—how a policy is extracted from the value function and generalizes at deployment—is hypothesized to often limit offline RL performance and scalability more than value accuracy.The paper tests value learning and policy extraction in Section 5, and test-time generalization in Section 6.Source: §3 · p. 3

4 Preliminaries

The paper formulates offline RL as learning a return-maximizing policy solely from a static dataset, and also considers goal-conditioned variants.

4 Preliminaries

  • Offline RL seeks a policy maximizing discounted return using only a static dataset of trajectories, without online interactions.The formulation defines an MDP with state and action spaces, rewards, initial-state distribution, and transition dynamics.Source: §4 · p. 4
  • In goal-conditioned experiments, policy and reward depend on a goal state, with a sparse reward of 1 when the state equals the goal.Episodes end upon reaching the goal; the goal is sampled from a goal distribution.Source: §4 · p. 4

5 Empirical analysis 1: Is it the value or the policy? (B1 and B2)

The study varies data available to value learning and policy extraction to identify which component limits offline RL. It finds that extraction choices can matter more than value-learning objectives, while weighted behavioral cloning may underuse the learned value function.

5.1 Analysis setup

  • Varying value-training and policy-extraction data in controlled experiments helps identify which component is the larger bottleneck.The study focuses on decoupled methods, where policies are extracted separately from a trained value function; policy learning is often the main bottleneck even there.Source: §5 · p. 4

5.1.1 Value learning objectives

  • The study compares SARSA and CRL behavioral Q functions with IQL’s optimal Q function, using CRL for goal-conditioned tasks and SARSA elsewhere.IQL fits an optimal Q function through in-sample expectile regression; SARSA and CRL fit behavioral Q functions.Source: §5.1.1 · pp. 4–5

5.1.2 Policy extraction objectives

  • AWR weights cloned dataset actions by advantage, DDPG+BC combines value maximization with behavioral cloning, and SfBC selects the highest-value sampled behavior action.The DDPG+BC regularizer controls deviation from behavior, while SfBC selects among actions sampled from a learned behavior-cloning policy.Source: §5.1.2 · p. 5

5.1.3 Environments and datasets

  • The eight environments span state- and pixel-based control, locomotion and manipulation, goal-conditioned and single-task settings, and varying data suboptimality.The exorl locomotion datasets are highly suboptimal and diverse; gc-roboverse is pixel-based robotic manipulation.Source: §5.1.3 · pp. 5, 7

5.2 Results: Policy extraction mechanisms substantially affect data-scaling trends

  • DDPG+BC is better than or as good as AWR in 15 of 16 settings, and often has the best policy-extraction performance.Table 1 aggregates performance over the data-scaling matrix and eight random seeds per setting; extraction hyperparameters are individually tuned.Source: Table 1 · p. 7, §5.2 · p. 7
  • Policy-extraction choices often affect performance more than value-learning objectives, although antmaze-large is an exception.Different extraction methods can perform differently even when applied to the same value function; the exception uses sparse-reward, suboptimal data with long-horizon trajectories.Source: §5.2 · p. 7

5.3 Deep dive 1: How different are the scaling properties of AWR and DDPG+BC?

  • On gc-antmaze-large, AWR remains policy-bounded across BC strengths, while DDPG+BC at α = 1.0 performs strongly across the matrix, even with 0.1M data.AWR also fails to achieve strong performance on exorl-walker at α = 100.Source: §5.3 · p. 8

5.4 Deep dive 2: Why is DDPG+BC better than AWR?

  • DDPG+BC can hillclimb the learned value function, whereas AWR’s weighted cloning can overfit when a few high-advantage samples dominate or data coverage is limited.AWR actions remain within the dataset-action convex hull; with limited coverage and expressive policies, it can memorize state-action pairs and revert toward unweighted cloning. The authors recommend behavior-constrained policy gradients over weighted cloning.Source: §5.4 · pp. 8–9

6 Empirical analysis 2: Policy generalization (B3)

The analysis finds that offline RL performance often hinges on policy generalization to test-time states, not just policy accuracy on the training distribution. It examines this bottleneck through online interaction, dataset coverage, and test-time policy improvement.

6.1 Analysis setup

  • The analysis tracks training, validation, and evaluation policy MSE during offline-to-online interaction, using expert policies to approximate the optimum across six D4RL tasks.Validation MSE evaluates states from the training distribution; evaluation MSE evaluates states visited at test time, which may differ from that distribution.Source: §6.1 · p. 10, Figure 5 · p. 10

6.2 Results: Test-time generalization is often the main bottleneck in offline RL

  • Online interaction often improves evaluation MSE while training and validation MSE remain flat, and evaluation MSE is most predictive of offline RL performance.This suggests test-time policy generalization can be a larger bottleneck than learning on in-distribution states.Source: §6.2 · p. 10, Figure 6 · p. 11
  • Test-time policy generalization cannot be guaranteed without additional coverage or structural assumptions when the test-time state distribution may be arbitrarily adversarial.The authors argue practical improvement is possible when test-time information is used or offline datasets can be designed.Source: §6.2 · p. 11

6.3 Solution 1: Improve offline data coverage

  • In gc-antmaze-large and adroit-pen, performance generally improves with better dataset state coverage despite increased suboptimality.In low-data gc-antmaze-large, DDPG+BC improves with better value functions while AWR does not fully leverage them.Source: §6.3 · p. 11, Figure 7 · p. 12

6.4 Solution 2: Test-time policy improvement

  • OPEX adjusts sampled actions along the frozen Q-function gradient, while TTT updates the policy on test-time data with Q and the offline policy fixed.Both methods are proposed to improve policy accuracy on test-time states without changing offline data collection.Source: §6.4 · p. 12
  • Across eight tasks, OPEX and TTT improve performance over vanilla IQL and non-gradient SfBC in many tasks, often by significant margins.The authors attribute these gains to mitigating the test-time policy generalization bottleneck.Source: §6.4 · p. 12, Figure 8 · p. 13

7 Conclusion: What does our analysis tell us?

Offline RL is often limited less by value quality than by policy extraction and test-time generalization. The analysis recommends diverse data, value-using extraction, and potential test-time policy improvement, while identifying two open research questions.

7 Conclusion: What does our analysis tell us?

  • The analysis finds offline RL often limited by policy extraction fidelity and test-time generalization rather than value-function quality.It identifies the best way to extract a policy and how to train one that generalizes at test time as open questions.Source: §7 · p. 13
  • Practitioners are advised to train value functions on diverse data and let policies use them through strong extraction objectives such as DDPG+BC or potential test-time improvement.The conclusion frames these as an empirical recipe, not a universal guarantee.Source: §7 · p. 13
  • The authors argue that progress requires research on policy extraction and test-time generalization alongside value-function learning.They note that the generalization question challenges the prevailing emphasis on pessimism, with few works explicitly addressing it [36] [37].Source: §7 · p. 13

A Limitations

The analysis relies on policy MSE as a proxy for policy accuracy, and its policy-extraction study covers continuous-action environments only.

A Limitations

  • Policy MSE is only a proxy and may poorly indicate performance when optimal actions differ substantially or the expert policy is not sufficiently optimal.The authors report strong correlation between evaluation MSE and performance, while leaving more sophisticated metrics for future work.Source: A Limitations · p. 18
  • The policy-extraction analysis covers continuous-action environments only, so its takeaways may not directly transfer to discrete actions.DDPG+BC is not straightforwardly defined for discrete actions, and a Q function may define a policy without a separate policy network.Source: A Limitations · p. 18

B Policy generalization: Rethinking the role of state representations

A continuous, invertible change to antmaze state representation produced different performance despite nearly identical training and validation MSE, with the difference reflected in test-time evaluation MSE. The section presents representation choice as a practical route to improving test-time generalization.

C Experimental details

B Policy generalization: Rethinking the role of state representations

  • Different homeomorphic representations of gc-antmaze-large produced very different performance, while their training and validation MSEs were nearly identical and evaluation MSE differed.The comparison used the original state and a continuous, invertible representation transforming x-y coordinates with invertible tanh kernels.Source: B Policy generalization: Rethinking… · p. 18, Figure 9 · p. 18
  • A good state representation should naturally enable test-time generalization, although designing one may require task knowledge or inductive biases.The authors report that representation choice significantly affects offline RL performance through test-time policy generalization.Source: B Policy generalization: Rethinking… · p. 18

C.1.1 Data-scaling analysis

The appendices specify datasets, training and evaluation procedures, MSE metrics, test-time improvement settings, state transformations, and implementation choices used in the experiments.

C.1.1 Data-scaling analysis

  • Data-scaling experiments use antmaze, locomotion, kitchen, and pixel-based goal-conditioned robotic datasets, including datasets collected by noisy, partially trained, or exploratory policies.The datasets range from 1M to 10M transitions where stated; ExORL data are described as highly suboptimal and diverse.Source: §C.1.1 · p. 19

C.1.2 Policy generalization analysis

  • Policy-generalization experiments use D4RL antmaze, kitchen, and adroit tasks, together with ExORL walker-run and cheetah-run.The passage lists the environments and dataset suites used, without reporting results.Source: §C.1.2 · p. 19

C.2 Data-scaling matrices

  • Agents are trained for 1M steps, or 500K on gc-roboverse, evaluated every 100K steps, and reported over the last three evaluations and eight seeds.For Figures 1 and 7, extraction hyperparameters are tuned per cell; multiple policies can be extracted from the same value function.Source: §C.2 · p. 19

C.3 MSE metrics

  • Training and validation sets are split by trajectory at 95% and 5%, and expert policies for MSE evaluation come from D4RL or saturated offline-to-online training.For antmaze expert training, arbitrary maze locations are used as initial states, while other experiments retain the original initial-state distribution.Source: §C.3 · p. 20

C.4 Test-time policy improvement methods

  • Test-time training extends policy training up to 2M gradient steps; antmaze evaluation compares deterministic and fixed-standard-deviation stochastic policies and reports the better performance.The fixed standard deviation is 0.4 for stochastic evaluation.Source: §C.4 · p. 20

C.5 State representation experiments

  • The antmaze representation transforms x and y coordinates with 32 continuous, invertible tanh kernels while retaining the 27-dimensional proprioceptive state.The passage describes this representation as similar to discretization with 32 bins, but continuous and invertible.Source: §C.5 · p. 20

C.6 Implementation details

  • The implementation is based on jaxrl_minimal and the official HIQL implementation, with experiments run on A5000 GPUs in no more than 18 hours each.The authors report these implementation details for their experiments.Source: §C.6 · p. 20

C.6.1 Data-scaling analysis; C.6.2 Policy generalization analysis

  • Data-scaling experiments mostly follow IQL, goal-conditioned IQL, and CRL hyperparameters, while policy extraction uses Gaussian action distributions with specified standard-deviation choices and clipped DDPG+BC actions.Policy-generalization experiments use Gaussian distributions with learnable standard deviations; the passage provides separate hyperparameter tables for these setups.Source: §C.6.1 · p. 20, §C.6.2 · p. 21, Table 2 · p. 21

D Additional results

The appendix supplies data-scaling matrices for AWR, DDPG+BC, and SfBC across policy-extraction hyperparameters. It also identifies a table of hyperparameters used for policy-generalization analysis.

D Additional results

  • Table 4 is identified as listing hyperparameters for the policy-generalization analysis.Source: Table 4 · p. 22
  • The matrices label data scales of 0.1M, 0.3M, 1M, and 10M.The extracted figure labels these scales as Value Data.Source: D Additional results · p. 23
  • Figure 11 presents full data-scaling matrices for AWR, DDPG+BC, and SfBC with different policy-extraction hyperparameters.The accompanying description specifies α for AWR and DDPG+BC, and N for SfBC.Source: D Additional results · p. 21, Figure 11 · p. 24

Related papers

Loading 2406.09329v2…