PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting the Minimalist Approach to Offline Reinforcement Learning

Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, Sergey Kolesnikov

arXiv:2305.09836v2cs.LGcs.AI

TL;DR

Offline RL has many algorithms with additional design choices, while the effect of those choices on established baselines remains understudied. The paper proposes ReBRAC, an extension of TD3+BC, and reports state-of-the-art performance among ensemble-free methods across proprioceptive and visual benchmark settings.

  • Problem

    The effects of recent design choices on established offline-RL baselines remain understudied, despite their use beyond core algorithmic innovations.

    Source: §1 · p. 1

  • Method

    The paper retrospectively analyzes recent offline-RL design choices and incorporates them into ReBRAC, a minimalistic extension of TD3+BC.

    Source: §1 · p. 1

  • Results

    ReBRAC demonstrates state-of-the-art performance among ensemble-free methods across proprioceptive and visual D4RL and V-D4RL settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present ReBRAC as a strong baseline under different hyperparameter budgets, with competitive results on offline and offline-to-online benchmarks.

    Source: §6 · p. 10

    The study covers only one approach and a subset of possible design changes, leaving their impact on other offline-RL methods and choices unexplored.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments.

1 Introduction

The paper asks whether recently emerged design choices can improve the minimalistic TD3+BC offline-RL baseline, and proposes ReBRAC as an extension. It reports state-of-the-art performance among ensemble-free methods across proprioceptive and visual benchmark settings.

1 Introduction

  • Offline-RL algorithms often add design and implementation choices beyond core innovations, complicating reproduction, hyperparameter tuning, and attribution of performance gains.The paper notes similar complexity concerns in online RL.Source: §1 · p. 1
  • TD3+BC became a de-facto baseline, but applying newer design and implementation choices to it remained unexamined.TD3+BC adds behavioral-cloning regularization to TD3.Source: §1 · p. 1
  • The paper asks how much newly emerged design choices can advance the minimalistic offline-RL algorithm TD3+BC.ReBRAC is proposed as an extension that adds recent design decisions to TD3+BC.Source: §1 · p. 1
  • ReBRAC achieves state-of-the-art performance among ensemble-free methods across proprioceptive and visual D4RL and V-D4RL settings.Figure 1 also presents performance profiles and probability of improvement across D4RL Gym-MuJoCo, AntMaze, and Adroit datasets.Source: Figure 1 · p. 2, §1 · p. 2

2 Preliminaries

The preliminaries define the offline-RL setting and introduce behavior-regularized actor-critic methods, including TD3+BC as a minimalistic baseline.

2.1 Offline Reinforcement Learning

  • An MDP specifies state and action spaces, transitions, rewards, and a discount factor; standard RL seeks a policy that maximizes cumulative discounted return through environment interaction.Offline RL instead uses a static dataset and cannot interact with the environment, making values for unseen state-action pairs difficult to estimate.Source: §2.1 · p. 2

2.2 Behavior Regularized Actor-Critic

  • BRAC penalizes actor-critic methods through actor penalization, critic penalization, or both, using a divergence between dataset and policy action distributions.The actor and critic objectives are given separately.Source: §2.2 · p. 2
  • BRAC evaluated KL divergence, Kernel MMD, and Wasserstein distance as regularizers but found no consistent advantage among them.The original actor and critic regularization coefficients had the same weight.Source: §2.2 · p. 2
  • TD3+BC uses mean squared error as the actor’s behavior-cloning regularizer and is presented as a simple offline-RL baseline with little added computational overhead.It modifies an existing RL algorithm by adding a behavior-cloning term to the actor loss.Source: §2.2 · p. 3

3 ReBRAC: Distilling Key Design Choices

ReBRAC extends TD3+BC into a broader BRAC-style method by combining recent design choices while keeping the algorithm simple. The section specifies choices spanning discounting, network architecture, normalization, batch size, and regularization.

3 ReBRAC: Distilling Key Design Choices

  • ReBRAC builds on TD3+BC as a more general BRAC version, adding design modifications while keeping the method simple.The authors say comparisons with TD3+BC are most important for assessing performance differences, rather than attributing them only to additional hyperparameter search.Source: §3 · pp. 3–4

Deeper Networks

  • ReBRAC uses three hidden layers for both actor and critic networks, matching a common choice in recent offline RL algorithms.The authors provide an ablation study of layer count in Section 4.5.Source: §3 · p. 3

LayerNorm

  • ReBRAC applies LayerNorm between each layer of its critic networks.The authors adopt it following prior work on normalization in RL and offline RL.Source: §3 · p. 3

Larger Batches

  • Large batches can improve performance in some domains but have little effect or reduce it in others.For D4RL Gym-MuJoCo, the authors use batches of 1024 and scale the learning rate.Source: §3 · pp. 3–4

Actor and critic penalty decoupling

  • ReBRAC independently sets actor and critic penalties, uses MSE as the divergence, and normalizes Q values.It omits state normalization because the authors intend to run the algorithm online and observe that this adjustment typically has negligible impact.Source: §3 · p. 4

Discount factor γ value change

  • Increasing γ from 0.99 to 0.999 is vital for improved performance on AntMaze tasks.The authors motivate the increase by sparse rewards, which may make low γ values propagate the training signal poorly.Source: §3 · p. 4

4 Experiments

Across D4RL and V-D4RL, ReBRAC achieves leading results among ensemble-free methods, with performance varying by design choice, network depth, and policy-evaluation budget.

4.1 Evaluation on offline D4RL

  • ReBRAC achieves state-of-the-art average results on D4RL Gym-MuJoCo, AntMaze, and Adroit, except that SAC-RND is slightly better on Gym-MuJoCo.The comparisons are reported in Tables 2–4.Source: Table 2 · p. 5, Table 3 · p. 5, §4.1 · p. 5

4.2 Evaluation on offline V-D4RL

  • On V-D4RL, ReBRAC is state-of-the-art or close on most tasks and is the only method notably above naive Behavioral Cloning on average.The evaluation covers available single-task datasets without distractors.Source: Table 5 · p. 6, §4.2 · p. 6

4.3 Evaluation on offline-to-online D4RL

  • In offline-to-online D4RL, ReBRAC surpasses four of six AntMaze datasets, leads Adroit final scores, and beats Cal-CQL on average; only Cal-QL edges its regret in AntMaze.Tables 6 and 7 report normalized performance and cumulative regret, respectively.Source: Table 6 · p. 7, Table 7 · p. 7, §4.3 · p. 7

4.4 Ablating Design Choices

  • Ablations identify LayerNorm, actor penalty, Gym-MuJoCo batch size, and increased AntMaze discount as consequential choices, while critic penalty and decoupling matter less on most problems.Disabling LayerNorm halves average performance overall; disabling the actor penalty prevents learning most tasks, and the proposed configuration performs best on average.Source: §4.4 · p. 8, Table 8 · p. 8

4.5 Stacking Even More Layers

  • AntMaze performance can improve through five network layers but drops or stalls at six; four critic and three actor layers perform best on average.Reducing critic size gives the worst performance on most datasets, while changing only the actor is more stable on average.Source: Figure 2 · p. 9, §4.5 · p. 9

4.6 Penalization Sensitivity Analysis

  • Under uniform policy selection, ReBRAC’s Expected Online Performance exceeds TD3+BC at every evaluation-policy budget and beats IQL across domains when the budget exceeds two policies.Approximately ten policies are required for ReBRAC to attain ensemble-free state-of-the-art performance.Source: Table 9 · p. 9, §4.6 · p. 9

5 Related Work

Offline RL work spans ensemble-free methods that constrain policies or critics and ensemble-based methods that use uncertainty estimation. Prior ablations of design choices on established baselines remain limited, particularly for pure offline RL.

Ensemble-free offline RL methods

  • TD3+BC adds behavioral-cloning regularization to the actor loss, while CQL penalizes out-of-distribution values and IQL avoids sampling out-of-distribution actions.These are examples of ensemble-free offline RL methods.Source: §5 · p. 10
  • Other ensemble-free methods include latent-action encoder pre-training, SPOT's VAE-based uncertainty penalty, and SAC-RND's random-network-distillation penalties.These approaches add mechanisms beyond the minimalist baseline.Source: §5 · p. 10

Ensemble-based offline RL methods

  • Ensemble-based approaches use uncertainty estimation, but SAC-N and EDAC require ensembles of up to 500 and 50, respectively, and neither solves AntMaze.SAC-N's hopper task required an ensemble of 500; EDAC reduced the size to 50.Source: §5 · p. 10
  • RORL uses ensembles of up to 20, while MSG uses four members for Gym-MuJoCo and 64 for state-of-the-art AntMaze performance.RORL perturbs state-action pairs with out-of-distribution actions to make the Q function more robust and smooth.Source: §5 · p. 10

Design choices ablations

  • Ablations of design choices on established baselines are very limited in offline RL, with prior evidence from Fujimoto & Gu (2021) and a parallel online-with-offline-data study by Ball et al. (2023).The latter setting differs from pure offline RL.Source: §5 · p. 10

6 Conlusion, Limitations, and Future work

The paper revisits recent offline RL advances and adds a modest set of improvements to TD3+BC. It reports competitive results across offline and offline-to-online D4RL and offline V-D4RL benchmarks, while its scope remains one approach and a subset of design changes.

Conclusion

  • Modest updates to TD3+BC achieve more than competitive results on offline and offline-to-online D4RL and offline V-D4RL benchmarks under different hyperparameter budgets.The paper describes this as revisiting recent offline RL advancements.Source: §6 · p. 10

Limitations and Future Work

  • The study is limited to one approach and a subset of possible design changes.It identifies testing these modifications on IQL, CQL, and MSG, and examining other choices such as learning-rate schedules, dropout, wider networks, or stochastic policies as future work.Source: §6 · p. 10

A Experimental Details

The experimental details specify implementations and task-specific dataset, evaluation, and hyperparameter choices across D4RL and V-D4RL. AntMaze uses a modified reward and discount factor to address sparse rewards over long episodes.

Experimental setup

  • D4RL uses a JAX implementation, while V-D4RL uses PyTorch with the Clean Offline RL TD3+BC implementation; experiments run on V100 and A100 GPUs.V-D4RL uses single-task datasets without distraction at 84 × 84 resolution, with β1 and β2 selected from specified ranges.Source: A Experimental Details · p. 14

Gym-MuJoCo and Adroit tasks

  • Gym-MuJoCo uses v2 datasets and Adroit uses v1; agents train for one million steps and are evaluated over ten episodes.Source: A Experimental Details · p. 14
  • For ReBRAC, actor β1 was selected from {0.001, 0.01, 0.05, 0.1} and critic β2 from {0, 0.001, 0.01, 0.1, 0.5}; best settings were reported for each dataset.For TD3+BC here and in AntMaze, the ReBRAC actor-regularization grid was used with the default α value of 0.4.Source: A Experimental Details · p. 14
  • For Adroit, SAC-RND's β1 and β2 are tuned over separate specified ranges.Source: A Experimental Details · p. 14

AntMaze tasks

  • AntMaze experiments use v2 datasets, train for 1 million steps, evaluate over 100 episodes, and multiply rewards by 100.ReBRAC's discount factor is increased to 0.999 because episodes can reach 1000 steps and sparse rewards arrive at the end; the passage contrasts minimum discount values of 4 · 10−5 and 0.36.Source: A Experimental Details · p. 14
  • For AntMaze, ReBRAC's actor and critic penalties and learning rates are optimized over specified ranges, with dataset-specific best hyperparameters reported in Table 11.Source: A Experimental Details · p. 14

Offline-to-offline

  • In offline-to-online experiments, β2 is set to zero and β1 is linearly decayed to half its initial value during online training.The same parameters are otherwise used as for the offline-to-online setup, as stated in the passage.Source: A Experimental Details · p. 14

B.1 ReBRAC

The appendix identifies tables for ReBRAC's general hyperparameters and its benchmark-specific best hyperparameters.

B.1 ReBRAC

  • Table 10 presents ReBRAC's general hyperparameters.Source: Table 10 · p. 15
  • Table 11 presents ReBRAC's best hyperparameters used in the D4RL benchmark.Source: Table 11 · p. 15
  • Table 12 presents ReBRAC's best hyperparameters used in the V-D4RL benchmark.Source: Table 12 · p. 16

B.2 IQL

The three tables report best hyperparameters used for IQL, TD3+BC, and SAC-RND on their respective D4RL benchmarks.

B.2 IQL

  • Table 13 lists IQL’s best hyperparameters used in the D4RL benchmark.Source: Table 13 · p. 17

B.3 TD3+BC

  • Table 14 lists TD3+BC’s best hyperparameters used in the D4RL benchmark.Source: Table 14 · p. 18

B.4 SAC-RND

  • Table 15 lists SAC-RND’s best hyperparameters used in D4RL Adroit tasks.Source: Table 15 · p. 19

C Comparison to Ensemble-based Methods

ReBRAC is competitive with ensemble-based methods on Gym-MuJoCo, strong among ensemble-free methods on AntMaze, and outperforms both method families on Adroit. The accompanying sections report feature-normalization scores, computational costs, and expected online performance comparisons.

C Comparison to Ensemble-based Methods

D Feature Normalization

  • Table 19 reports final-evaluation and ten-seed average normalized scores for LayerNorm and Feature Norm, with hyperparameters tuned using the same grid.Source: Table 19 · p. 21

E Computational costs

F Expected Online Performance

G D4RL tasks ablation

The ablation tables report final normalized scores averaged over four unseen training seeds across eight D4RL task groups.

G D4RL tasks ablation

Loading 2305.09836v2…