PaperlayerView exact revision on arXiv

Source-linked AI summary

Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, Xianyuan Zhan

arXiv:2303.15810v1cs.LGcs.AI

TL;DR

Offline RL must improve policies using fixed data despite risks from OOD action values, and the paper addresses how in-sample learning handles distribution shift. It develops IVR and the SQL and EQL algorithms, reporting benchmark performance and robustness in noisy and small-data regimes.

  • Problem

    Offline RL faces a trade-off between improving beyond the behavior policy and constraining policy deviation, while it remains unclear how in-sample methods handle distribution shift in value learning.

    Source: §1 · p. 1

  • Method

    The paper formulates in-sample learning as Implicit Value Regularization and proposes SQL and EQL using existing value regularizers in a complete in-sample manner.

    Source: §1 · p. 2

  • Results

    SQL and EQL obtain state-of-the-art benchmark results, with reported robustness in noisy and small-data regimes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that sparsity filters low-Q actions and benefits inferior-quality datasets, while in-sample learning is beneficial in small-data comparisons with CQL.

    Source: §1 · p. 2

    EQL’s exponential term causes unstable gradients when learning V and makes it more vulnerable to hyperparameter choices than SQL.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing $Q$-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recently proposed \textit{In-sample Learning} paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the \textit{Implicit Value Regularization} (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse $Q$-learning (SQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes.

1 INTRODUCTION

The paper frames offline RL as a way to learn from fixed data when online interaction is costly or dangerous, but OOD action values create a trade-off between policy improvement and reliable value estimation. It proposes Implicit Value Regularization (IVR) as an in-sample framework and develops SQL and EQL, reporting benchmark performance and robustness in noisy and small-data regimes.

1 INTRODUCTION

  • OOD actions can cause Q-function extrapolation errors, while behavior or value regularization trades accurate value estimation against maximum policy performance.Offline RL learns from fixed data without additional online interaction, which is useful where exploration is costly or dangerous.Source: §1 · pp. 1–2
  • IVR adds behavior regularization to policy learning, yielding a closed-form policy weighted by behavior and value functions that can be learned from dataset samples.The state-value function normalizes the policy, and a mathematical transformation makes its learning equivalent to a convex optimization problem.Source: §1 · p. 2
  • The framework interprets IQL as applying implicit value regularization, with τ controlling its strength; theory and practice differ on the preferred τ.The paper says theory favors τ close to 1 for an optimal policy, while a larger τ may perform worse in practice.Source: §1 · p. 2
  • SQL and EQL achieve state-of-the-art benchmark results, with reported robustness in noisy and small-data regimes.The paper also reports state-of-the-art performance for SQL on D4RL, especially on suboptimal AntMaze and Kitchen datasets.Source: §1 · p. 2
  • SQL and EQL use value regularizers from existing methods in a complete in-sample manner, and their sparsity filters actions with Q-values below a threshold.The authors report that sparsity benefits inferior-quality and noisy datasets, and compare in-sample learning with CQL in small-data regimes.Source: §1 · p. 2

2 RELATED WORK

Prior offline RL work addresses distribution shift through behavior regularization or imitation-like filtering, each with trade-offs. The paper positions SQL and EQL as combining implicit shift control with in-sample generalization.

2 RELATED WORK

  • Behavior regularization has appeared as divergence penalties, weighted behavior cloning, policy parameterization, critic regularization, uncertainty, and distance functions.These approaches aim to stay near the behavioral distribution or be pessimistic about unknown state-action pairs.Source: §2 · pp. 2–3
  • Imitation-style methods can clone or filter dataset actions and transitions, but their single-step dynamic programming loses the ability to stitch suboptimal trajectories.The passage describes these methods as useful when datasets are good enough or contain high-performing trajectories.Source: §2 · p. 3
  • SQL and EQL combine implicit distribution-shift control with in-sample generalization, which the authors contrast with IQL’s different motivation.IQL approximates the optimum in-support policy using an upper expectile of the behavior policy’s action-value function, but is not motivated by pessimism about distribution shift.Source: §2 · p. 3
  • Unlike online entropy-regularized methods, this offline method jointly learns state- and action-value functions along with behavior regularization.The passage notes that online entropy terms involve only the policy and can be directly computed.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define the offline RL setting and describe IQL’s in-sample value learning and policy extraction. They also identify unresolved questions about IQL’s hyperparameter, distribution-shift handling, and policy extraction.

3 PRELIMINARIES

  • An MDP comprises state and action spaces, transition and reward functions, an initial-state distribution, and a discount factor; RL seeks a policy maximizing expected discounted return.The passage defines trajectories through an initial state-action pair followed by transitions and policy-sampled actions.Source: §3 · p. 3
  • Offline RL learns from a fixed dataset of trajectories that may be heterogeneous and suboptimal, with the behavior policy denoting the dataset’s observed conditional action distribution.The dataset is collected by different policies.Source: §3 · p. 3
  • IQL learns its value function using only in-sample actions and asymmetric ℓ2 expectile regression for V, which estimates a maximum Q-value over dataset-supported actions.The passage describes this as enabling implicit Q-learning.Source: §3 · p. 4
  • After learning Q and V, IQL extracts its policy through advantage-weighted regression.The passage cites Peters et al. (2010), Peng et al. (2019), and Nair et al. (2020) for this technique.Source: §3 · p. 4
  • The preliminaries raise unresolved questions about τ’s theory-practice gap, IQL’s handling of distribution shift, and whether policy extraction matches the learned Q-function’s implied policy.The passage notes that τ should be close to 1 in theory, whereas a larger value may perform worse in practice.Source: §3 · p. 4

4 OFFLINE RL WITH IMPLICIT VALUE REGULARIZATION

The IVR framework adds behavior regularization to the policy objective, yielding a softened maximum over behavior-policy actions and an in-sample learning scheme. SQL and EQL instantiate this framework with different regularizers.

4.1 BEHAVIOR-REGULARIZED MDPS

  • Behavior regularization smooths the greedy maximum over the policy into a softened maximum over behavior-policy actions, enabling in-sample learning.The induced Q-function regularization is difficult to compute directly when the behavior policy is unknown.Source: §4.1 · p. 4

4.2 ASSUMPTIONS AND SOLUTIONS

  • Under the stated assumptions, optimal Q- and V-values can be learned using only the normalization term and behavior-policy action samples; suitable regularizers can make the policy sparse.Repeated application of the resulting operator converges to the optimal policy’s Q-values.Source: §4.2 · p. 5

4.3 SPARSE Q-LEARNING (SQL)

  • SQL uses the χ2-divergence regularizer and learns Q and V iteratively; its mode-seeking policy assigns zero probability to actions below a Q-value threshold.The paper also describes a three-network SQL-U variant as computationally expensive.Source: §4.3 · p. 6

4.4 EXPONENTIAL Q-LEARNING (EQL)

  • EQL uses the reverse-KL choice and an exponential learning objective, which avoids approximation but can produce unstable value-learning gradients and greater hyperparameter sensitivity than SQL.The paper notes that XQL derives the same objective from a different perspective.Source: §4.4 · pp. 6–7

4.5 DISCUSSIONS

  • SQL and EQL apply CQL- and AWR-like regularization implicitly during policy evaluation, learning from dataset samples without sampling OOD actions.SQL’s residual derivative stays unchanged below a threshold, whereas IQL’s keeps decreasing for increasingly negative residuals; IQL also uses two hyperparameters while SQL uses one.Source: §4.5 · pp. 7–8
  • 14 of 18 tasks: SQL or EQL achieves the highest averaged normalized score.Scores average the final 10 evaluations across 5 seeds.Source: Table 1 · p. 8

5 EXPERIMENTS

The experiments compare SQL and EQL with offline-RL baselines on D4RL and Atari, then test their behavior as data become noisier or smaller.

5.1 BENCHMARK DATASETS

  • On D4RL, SQL and EQL are competitive with the best prior methods on saturated MuJoCo tasks and outperform all baselines by a large margin on AntMaze and Kitchen.The authors note that AntMaze and Kitchen have few or no near-optimal trajectories and require value learning for policy stitching.Source: §5.1 · p. 8
  • SQL and EQL also achieve superior performance to baselines on high-dimensional image-based Atari datasets.Aggregated results and performance profiles are reported in Appendix D.Source: §5.1 · p. 8

5.2 NOISY DATA REGIME

  • Across all expert/random mixing settings, SQL and EQL outperform IQL, while IQL drops sharply as the expert ratio falls from 30% to 1%.At a 5% expert ratio, SQL and EQL reach near-100 performance on walker2d; IQL is affected on halfcheetah even at 30%.Source: §5.2 · p. 9, Figure 1 · p. 9

5.3 SMALL DATA REGIME

  • The small-data comparison pairs SQL’s in-sample learning with CQL’s out-of-sample learning while holding their inherent value regularization the same.The authors create Easy, Medium, and Hard AntMaze tasks by controlling the transition-discarding ratio.Source: §5.3 · p. 9
  • As AntMaze task difficulty increases, CQL’s performance drops and its Bellman error grows exponentially, while SQL and EQL remain stable and SQL’s error is much smaller.The custom tasks remove transitions with higher probability when they are closer to the target location, simulating reduced data and limited state coverage near the target.Source: §5.3 · p. 9, Table 2 · p. 9

6 CONCLUSIONS AND FUTURE WORK

The paper presents Implicit Value Regularization as a bridge between behavior-regularized and in-sample offline RL, and proposes two practical algorithms. It reports evaluations across benchmark, noisy, and small-data regimes, while identifying extensions beyond offline RL as future work.

  • Implicit Value Regularization bridges behavior-regularized and in-sample learning methods in offline RL.Source: §6 · p. 9
  • The proposed practical algorithms use value regularization from existing works in a complete in-sample manner and are evaluated on D4RL and custom noisy and small-data regimes.Source: §6 · p. 9
  • Future work includes scaling the framework to online RL, offline-to-online RL, or offline imitation learning, and constraining state-action distributions rather than only action distributions.Source: §6 · p. 9

A A STATISTICAL VIEW OF WHY SQL AND EQL WORK

The SQL and EQL value objectives estimate state-conditional extrema as α decreases, while α controls a trade-off between optimality and distributional shift. The section also examines how SQL sparsity relates to performance across tabular and continuous-action settings.

A A STATISTICAL VIEW OF WHY SQL AND EQL WORK

B HOW DOES SPARSITY BENEFIT VALUE LEARNING IN SQL?

B.1 SPARSITY IN THE TABULAR SETTING

In Four Rooms, SQL learns values closer to the optimum and reaches the goal despite suboptimal actions corrupting IQL’s value learning. In continuous-action datasets, the best sparsity ratio depends on dataset composition.

B.1 SPARSITY IN THE TABULAR SETTING

  • Unlike IQL, SQL keeps the derivative of its value-learning objective unchanged below a residual threshold; IQL’s derivative keeps decreasing as the residual becomes more negative.The paper links this IQL behavior to over- or underestimation of the value function by bad actions with small Q-values.Source: §B.1 · p. 14
  • SQL’s values and policy are closer to optimal than IQL’s in Four Rooms, where suboptimal dataset actions corrupt IQL’s values and prevent it from reaching the goal.Incorrect IQL Q-values at (1, 1) and (5, 9) make the agent fail; SQL succeeds in reaching the goal.Source: §B.1 · p. 14

B.2 SPARSITY IN THE CONTINUOUS ACTION SETTING

  • MuJoCo’s best sparsity ratios are around 0.1, compared with around 0.4 for AntMaze, where more transitions are described as useless.The best ratio depends on dataset composition; a larger α typically gives less sparsity.Source: §B.2 · p. 15

C.1 PROOF OF THEOREM 1

The proof derives the behavior-regularized MDP’s optimality condition using KKT conditions. It identifies a unique solution for u* and uses the irreducibility assumption to eliminate the stationary state distribution.

C.1 PROOF OF THEOREM 1

  • The proof uses KKT conditions and states that the resulting optimality condition is necessary and sufficient.It starts from the Lagrangian of the constrained problem and identifies u and β as multipliers for equality and inequality constraints.Source: §C.1 · p. 15
  • Assuming every policy induces an irreducible Markov chain lets the proof eliminate dπ(s) from the KKT conditions.Source: §C.1 · p. 15
  • For a positive-probability action, complementary slackness sets its multiplier β(a|s) to zero and requires Q(s,a) > u(s) + αh′f(0).Source: §C.1 · p. 15
  • Equation (23) has a unique solution u* because its left-hand side is continuous and monotonic.The proof denotes the corresponding policy as π*.Source: §C.1 · p. 16

C.2 PROOF OF THEOREM 2

The proof begins by comparing two state-value functions and their corresponding maximizing policies, then invokes symmetry to extend a statewise result.

C.2 PROOF OF THEOREM 2

  • For two value functions V1 and V2, the proof defines πi as the policy maximizing T*f Vi for each i.Source: §C.2 · p. 16
  • The proof next states a relation for every state s in S.The supplied passage does not specify the relation itself.Source: §C.2 · p. 16
  • By symmetry, the proof extends the stated relation to any state s in S.Source: §C.2 · p. 16

D EXPERIMENTAL DETAILS

The experiments specify implementation and evaluation settings across D4RL, Atari, noisy-data, and small-data regimes. On Atari, SQL achieves the best performance in 10 of 12 games.

D EXPERIMENTAL DETAILS

  • SQL and EQL use in-sample policy extraction based on Q − V residuals, with residual scaling in EQL and removal of the “1+” term in SQL.The section describes the scaling as reward normalization that does not change the optimal policy; α values are listed for both algorithms.Source: D EXPERIMENTAL DETAILS · p. 17, Table 5 · p. 17
  • D4RL returns are averaged over five seeds, with task-specific evaluation frequencies and reward standardization; SQL and EQL use 2-layer, 256-unit MLPs and task-specific learning rates.EQL uses exp-normalize for the V loss and clips its input at 5.0; the section also points to α sensitivity and algorithm runtime tables.Source: D EXPERIMENTAL DETAILS · p. 17, Table 6 · p. 18, Table 7 · p. 18
  • The Atari experiments use 5% or 10% of mixed, medium, or expert datasets for Breakout, Qbert, and Seaquest, with D-SQL, D-IQL, D-CQL, and D-BCQ evaluated using consistent preprocessing and network structures.D-IQL, D-CQL, and D-BCQ use selected hyperparameter values; D-SQL uses α = 1.0 for all datasets.Source: D EXPERIMENTAL DETAILS · pp. 17–18
  • The noisy-data experiment mixes expert and random MuJoCo locomotion data into datasets of 100,000 transitions, with different expert ratios.IQL scores use the best result from τ values in [0.5, 0.6, 0.7, 0.8, 0.9].Source: D EXPERIMENTAL DETAILS · p. 18, Table 8 · p. 18
  • Small AntMaze datasets are generated by discarding transitions according to task difficulty, with transitions closer to the goal more likely to be removed.The experiments report task-difficulty settings and select CQL scores across a sweep of min-q-weight values.Source: D EXPERIMENTAL DETAILS · p. 18, Table 9 · p. 19
  • SQL achieves the best performance in 10 of 12 games on the 10% and 5% Atari dataset settings.Table 10 reports the performance for both dataset sizes.Source: Table 10 · p. 19
  • Supplementary figures show SQL and EQL learning curves on D4RL MuJoCo, AntMaze, and Kitchen tasks, plus score-distribution profiles comparing CQL, IQL, and SQL.The performance profiles include average score distributions and pointwise 95% bootstrap confidence bands.Source: Figure 5 · p. 19, Figure 8 · p. 20, Figure 6 · p. 20
Loading 2303.15810v1…