Source-linked AI summary
RvS: What is Essential for Offline RL via Supervised Learning?
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine
TL;DR
The paper asks when offline RL can be solved effectively with supervised learning and which RvS components are essential. It studies conditional imitation across diverse offline RL tasks using simple models and compares goals, rewards, capacity, and regularization. RvS can match strong prior methods broadly, while conditioning choices and random-data settings expose important limitations.
Problem
Prior work disagreed about which components make RvS effective and how well it works on tasks with few near-optimal trajectories.
Method
The paper empirically studies outcome-conditioned behavior cloning across diverse offline RL problems, using simple feedforward policies and analyzing conditioning, capacity, and regularization.
Results
Across suites, simple fully connected RvS methods can match or outperform strong prior methods, including on Franka Kitchen and AntMaze.
Takeaways & Limitations
Careful capacity, regularization, and conditioning choices are central to making simple RvS methods perform well.
Takeaways & Limitations
Reward-target selection remains difficult in a truly offline setting, and RvS methods perform worse than TD learning on random datasets.
Abstract
from arXiv · showhide
Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin "RvS learning"). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.
1 INTRODUCTION
RvS reframes offline RL as conditional supervised learning, using experience that may be suboptimal for one task as demonstrations for another outcome. The paper asks which RvS components matter and reports that simple feedforward implementations can match more complex methods across suites.
- Motivation: RvS converts offline RL into conditional, filtered, or weighted imitation learning by conditioning behavior cloning on goals, rewards, or related information.The framework exploits experience that is suboptimal for one task but optimal for another task.
- Headline findings: Across the considered suites, supervised RvS with a feedforward MLP matches methods using TD learning and Transformer sequence models.Figure 1 averages each bar over many tasks within each suite.
- Research questions: The paper studies whether online data, advantage weighting, large Transformers, or temporal-difference learning are essential for effective RvS.It also examines performance when datasets contain few near-optimal trajectories, including Kitchen and AntMaze.
- Headline findings: The study finds that pure likelihood maximization and simple feedforward models can perform as well as conservative TD learning and complex sequence models.It also reports that conditioning on reward values versus goals has domain-dependent effects.
2 RELATED WORK
The paper situates RvS among offline RL approaches that avoid value functions by using conditional behavior cloning. Prior methods vary in what information they condition on and whether they use filtering or reweighting.
- Approach landscape: Offline RL methods include value functions, dynamics estimation, and uncertainty quantification, whereas this paper focuses on conditional behavior cloning without value functions.Common forms are goal-conditioned and reward-conditioned behavior cloning.
- RvS variants: Prior RvS methods condition on goals, reward values, previous timesteps, or other information, and may reweight or filter demonstrations.
- Study focus: The paper studies which conditioning and modeling choices matter for RvS performance across offline RL tasks.
3 REINFORCEMENT LEARNING VIA SUPERVISED LEARNING
RvS formulates offline RL as learning a policy conditioned on an outcome sampled from trajectories. The implementation samples trajectories, timesteps, and outcomes, then maximizes the likelihood of observed actions.
- Framework: RvS places existing supervised offline RL methods under a common outcome-conditioned policy framework rather than proposing a new method.The paper uses the framework to analyze important design decisions.
- Outcome choices: An outcome can be a future visited state for RvS-G or an average future return for RvS-R.The outcome distribution is defined from the future portion of a trajectory.
- Policy: Offline RvS takes a dataset of trajectories and learns an outcome-conditioned policy πθ(at | st, ω).The dataset is written as D = {τ}.
- Analysis: The empirical analysis compares goals and rewards and examines the design choices needed for strong performance across benchmark tasks.
- Training: Training samples a trajectory, a timestep, and a corresponding future outcome, then updates policy parameters using the log-likelihood of the observed action.
4 TASKS AND DATASETS
The evaluation spans diverse offline RL suites with different conditioning needs, data quality, and requirements for composing or imitating behavior. These include random, mixed, suboptimal, and human-demonstration datasets.
- Scope: The task selection covers domains suited to goal or reward conditioning and datasets with varying proportions of near-optimal data.The study aims to include both near-expert and substantially suboptimal data.
- GCSL: GCSL tasks are adapted to offline RL by collecting suboptimal trajectories with a random policy across navigation, robotics, and game environments.
- Gym Locomotion: Gym Locomotion evaluates HalfCheetah, Hopper, and Walker using random, medium, medium-expert, and medium-replay D4RL datasets.The datasets mix policies with different optimality levels.
- Franka Kitchen: Franka Kitchen requires composing multi-step manipulation behaviors from component skills and includes complete and partial human-demonstration datasets.
- AntMaze: AntMaze tests temporal compositionality by navigating an 8-DoF quadruped through mazes using diverse and play datasets.The benchmark includes umaze, medium, and large mazes.
- Conditioning: For Kitchen and AntMaze, RvS-G conditions on achieved-task states or maze goal locations, requiring knowledge of how the reward function is defined.
5 ARCHITECTURE, CAPACITY, AND REGULARIZATION
RvS performance depends strongly on policy capacity, regularization, output-distribution expressiveness, and conditioning choices. Validation loss is not consistently reliable for selecting these hyperparameters, motivating task-specific tuning.
- Capacity and regularization: Simple feedforward RvS models can match complex sequence models, but capacity and regularization require careful tuning.The paper identifies overfitting and underfitting as major challenges, with task-dependent effects from architectural choices.
- Capacity and regularization: Larger architectures generally perform better, while dropout improves kitchen-complete but can worsen hopper-medium-expert and antmaze-medium-play.The authors relate these differences to dataset size, demonstration complexity, and the need to generalize across conditioning variables.
- Output distributions: Categorical distributions over discretized actions match or outperform unimodal Gaussians across the GCSL suite.Discretized categorical policies can represent more complex multimodal action distributions, increasing effective policy capacity.
- Tuning with validation loss: Validation loss correlates with final performance inconsistently across Franka Kitchen datasets and is generally unreliable for hyperparameter tuning.The experiments compare regularized and unregularized settings with different batch sizes, while fully automated tuning remains open.
- A recipe for practitioners: The recommended tuning recipe is to increase network width until performance saturates, then try modest dropout such as p = 0.1.Batch size can also be adjusted when validation loss suggests underfitting or overfitting.
6 COMPARING RVS WITH PRIOR OFFLINE RL METHODS
Across the evaluated suites, RvS implementations are generally competitive with prior offline RL methods, but performance depends strongly on the conditioning variable and has clear limits on random data and reward interpolation.
- RvS-G is state-of-the-art in AntMaze, Kitchen, and GCSL, while RvS-R performs worse on those suites.
- In Gym, RvS-R matches Decision Transformer performance while using an MLP.
- RvS-G or RvS-R attains results comparable to the best prior method in every evaluated suite.
- Subtrajectory stitching: RvS-G performs as well as dynamic-programming-based TD3+BC and CQL on tasks requiring subtrajectory stitching, and outperforms RvS-R there.
- Reaching arbitrary goals: RvS-G successfully reaches many different goals from random offline data in the GCSL suite.
- TD learning performs better than RvS methods on random datasets, while RvS-R cannot interpolate between the medium and expert modes in walker2d-medium-expert.
- Reward-target tuning is required for the reported results, creating an important challenge for fully offline use.
7 DISCUSSION AND FUTURE WORK
The discussion concludes that simple RvS policies can be highly competitive when capacity, regularization, and conditioning are chosen appropriately. It also identifies validation-based tuning and conditioning-variable selection as open problems.
- With appropriate capacity and regularization, simple fully connected RvS architectures can match or outperform the best prior methods.
- Choosing the conditioning variable, such as goals or rewards, is critical to RvS performance.
- RvS remains competitive on Franka Kitchen and AntMaze despite limited optimal data.
- Future work: Automatically tuning capacity and regularization remains open because validation error is not a reliable metric.
- Future work: Automating the choice of conditioning variable is another proposed direction for improving RvS applicability.
A EXPERIMENT DETAILS
The experiments use repeated seeds and rollouts across several offline RL suites, with suite-specific dataset versions, baselines, and data-collection protocols. Franka Kitchen additionally tests whether validation loss tracks evaluation return.
- Each task uses 5 random training seeds and 200 evaluation rollouts.
- GCSL: GCSL data comes from random rollouts with a 50-action episode limit and suite-dependent experience budgets.
- Gym Locomotion: Gym Locomotion uses v2 datasets and reports replicated CQL alongside published DT, TD3+BC, and Onestep results.
- Franka Kitchen: Franka Kitchen uses v0 datasets and includes CQL-p, replicated CQL, BC, and Filtered BC baselines.
- AntMaze: AntMaze uses v2 datasets for most experiments but includes some comparison numbers based on v0 datasets.
- Validation and evaluation: In Franka Kitchen, hyperparameter settings have nearly identical validation loss while evaluation return differs by up to 1.4x.
- Timeout flags contain errors, but TD-learning methods appear fairly robust to those errors.
B THE IMPACT OF MODEL CAPACITY AND REGULARIZATION
Model capacity and regularization affect RvS performance in ways not explained by validation loss alone. Across the tested settings, returns varied substantially despite nearly identical validation losses.
- Return can vary by as much as 1.4x between hyperparameter settings while validation loss remains nearly identical.Evaluation return also continues increasing after validation loss has mostly leveled off.
- The observed return differences refute the hypothesis that regularization improves performance solely through better validation loss.
- Architecture decisions can affect downstream performance beyond validation loss, a phenomenon also observed in neural machine translation and contrastive representation learning.
C COMPARISON OF CONDITIONING STRATEGIES
The effectiveness of goal versus reward conditioning depends on the task and how goals are selected. Reward conditioning outperforms goal conditioning on the evaluated Gym tasks, while specific goal commands work best in kitchen-complete.
- Across all tested Gym goal-selection strategies, RvS-G cannot match RvS-R because the tasks are not inherently goal-conditioned.The strategies include Reward Scalar, Reward Goal, Length Goal, and Optimized Goal.
- In kitchen-complete, oracle dynamic conditioning and commanding all subtasks completed perform best.Dynamic conditioning updates the goal as successive subtasks are completed.
- Commanding the microwave subtask outperforms commanding the other individual subtasks in kitchen-complete.The authors speculate that the microwave subtask may appear most often in demonstrations.