URLB: Unsupervised Reinforcement Learning Benchmark

Michael Laskin; Denis Yarats; Hao Liu; Kimin Lee; Albert Zhan; Kevin Lu; Catherine Cang; Lerrel Pinto; Pieter Abbeel

Source-linked AI summary

URLB: Unsupervised Reinforcement Learning Benchmark

Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel

arXiv:2110.15191v1Oct 28, 2021cs.LGcs.AIcs.RO

TL;DR

RL agents often remain specialized to the tasks used for training, while generalist agents that adapt quickly remain difficult to develop. This paper introduces URLB, a standardized benchmark with unified baselines for unsupervised pre-training and downstream adaptation. The baselines make progress, but none solve the benchmark.

Problem
Deep RL commonly produces task-specific policies, leaving fast adaptation to new tasks an outstanding challenge.
Method
URLB provides twelve continuous-control tasks across three domains, standardized pre-training and evaluation, and eight open-source baselines sharing one optimization backbone.
Results
No implemented unsupervised RL algorithm solves URLB, although the baselines make progress.
Takeaways & Limitations
URLB offers a unified basis for comparing and developing unsupervised RL agents while exposing substantial room for improvement.
Takeaways & Limitations
URLB tasks remain far less visually and combinatorially complex than real-world robotics, and the benchmark does not evaluate dataset diversity as an alternative pre-training measure.

Abstract

from arXiv · show

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Yet training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. Building on the DeepMind Control Suite, we provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods. We find that the implemented baselines make progress but are not able to solve URLB and propose directions for future research.

1 Introduction

Deep RL agents often remain narrow and poorly generalize beyond their training task, motivating unsupervised pre-training for faster downstream adaptation. URLB provides a standardized benchmark and unified baselines, but the implemented methods do not yet solve it.

Current RL algorithms often produce brittle policies that solve only the task they were trained on, limiting broader applicability.
Unsupervised RL combines reward-free exploration with downstream fine-tuning, whose effectiveness depends on the intrinsic reward used during pre-training.
URLB standardizes evaluation procedures and releases eight leading baselines trained with the same optimization backbone for fairer comparison.
URLB evaluates adaptation efficiency across three domains and twelve continuous-control tasks of varying difficulty.
No existing unsupervised RL algorithm can solve URLB, although the implemented baselines make progress.

2 Preliminaries and Notation

URLB treats unsupervised RL as reward-free intrinsic-reward pre-training followed by extrinsic-reward fine-tuning. Agents may learn from state or pixel observations, with pixel inputs embedded before intrinsic rewards are computed.

In unsupervised RL, agents pre-train with self-supervised intrinsic rewards and fine-tune on downstream tasks using extrinsic rewards.
Some algorithms condition the agent on a learned task vector denoted w.
URLB supports both proprioceptive state and RGB-image observations, enabling separate evaluation of behavior and representation learning.
For pixel inputs, an encoder maps observations to lower-dimensional embeddings used to compute intrinsic rewards; state inputs normally use the identity encoder.

3 URLB: Evaluation and Environments

URLB standardizes pre-training and downstream adaptation to measure how quickly unsupervised agents learn new tasks. It uses DeepMind Control Suite domains spanning state- and pixel-based settings with varied task difficulty.

Pre-training checkpoints at 100k, 500k, 1M, and 2M steps measure downstream performance as a function of pre-training duration.
URLB evaluates adaptation by initializing downstream agents with pre-trained parameters, fine-tuning for 100k steps, and measuring task performance.
URLB benchmarks both state-based and pixel-based agents to separate unsupervised behavior learning from representation learning.
The benchmark uses three DeepMind Control Suite domains with twelve tasks: Walker, Quadruped, and Jaco Arm.
The domains vary in difficulty, from Walker balance and locomotion to Quadruped control and challenging Jaco Arm manipulation.

4 URLB: Algorithmic Baselines for Unsupervised RL

URLB standardizes unsupervised RL evaluation with a shared optimization backbone, fixed pre-training and fine-tuning procedures, and eight intrinsic-reward baselines spanning three algorithm categories.

URLB provides a unified codebase for eight unsupervised RL algorithms, with every algorithm trained using the same optimization backbone for fair comparison.
The benchmark evaluates algorithms through unsupervised pre-training followed by supervised fine-tuning on downstream tasks.Pre-training uses intrinsic rewards, while downstream adaptation uses extrinsic rewards.
The shared backbone uses DrQ-v2 for image observations and DDPG, as implemented in DrQ-v2, for state observations.The choice is motivated by the use of off-policy optimization across most evaluated algorithms.
The implemented baselines are grouped into knowledge-based, data-based, and competence-based categories according to their intrinsic-reward objectives.Knowledge-based methods maximize prediction error, data-based methods maximize entropy, and competence-based methods maximize mutual information with an explicit skill vector.
Knowledge-based baselines include ICM, Disagreement, and RND, which use prediction error or uncertainty to increase knowledge about the environment.Their predictive functions operate on encoded observations z.
Data-based baselines APT and ProtoRL maximize entropy, while competence-based baselines such as DIAYN maximize mutual information between encoded observations and skills.ProtoRL extends APT by clustering replay-buffer data into prototypes.
Aggregate results average normalized scores across tasks and methods after 2M pre-training steps and 100k fine-tuning steps, using ten random seeds per algorithm.The Random Init baseline trains DrQ-v2 and DDPG from random initialization for 100k steps.

5 Experiments

The experiments show meaningful but incomplete progress: unsupervised RL often falls short of the benchmark, with performance varying by observation type, algorithm category, and pre-training duration. The benchmark also exposes several concrete directions for improving representation learning, competence-based methods, scaling with pre-training, and fine-tuning.

The experiments evaluate intrinsic-reward pre-training followed by downstream fine-tuning, using fixed procedures and ten random seeds for performance estimates.The reported results average scores and display standard errors across the evaluation seeds.
None of the implemented algorithms solve URLB; the best methods achieve 40–70% normalized return, below the near-100% threshold for solving the benchmark.This conclusion holds despite up to 2M pre-training steps and 100k fine-tuning steps.
Random initialization is competitive with most baselines for state-based tasks, whereas it degrades for pixel-based tasks.The pixel-based result indicates that representation learning contributes importantly to unsupervised pre-training.
Most algorithms lose 20–50% when learning from pixels rather than states, especially on harder environments such as Quadruped and Jaco Arm.The results identify better representation learning during pre-training as an important research direction.
Competence-based approaches underperform knowledge-based and data-based approaches, with none achieving state-of-the-art mean performance on any URLB task.The authors identify competence-based unsupervised RL as a research direction with substantial room for improvement.
No single unsupervised RL algorithm leads on both states and pixels: APT and ProtoRL lead state-based URLB, while ICM leads pixel-based URLB at 2M pre-training steps.The leading method therefore depends on the observation modality.
Performance fails to improve monotonically with pre-training: 9 of 18 experiments show no improvement or degradation as pre-training steps increase.The benchmark uses pre-training checkpoints to expose this scaling behavior, and the authors identify monotonic improvement as an open research goal.
The benchmark’s fine-tuning strategy initializes the actor-critic with pre-trained weights and continues training with an extrinsic reward.The authors note that alternative strategies may be especially relevant for competence-based approaches conditioned on skill.

6 Related work

Related work situates URLB at the intersection of established RL benchmarks, unsupervised representation learning, and fully unsupervised behavioral learning. The paper argues that differing procedures and environments have made behavioral methods difficult to compare, motivating a standardized benchmark.

Established benchmarks such as Atari ALE, OpenAI Gym, and DeepMind Control Suite accelerated progress in supervised RL across observation and action spaces.
Other benchmarks target specialized settings including navigation, procedural generalization, offline RL, preference-based RL, and multi-task performance.
Unlike computer vision and NLP, unsupervised RL must learn both representations and policies without extrinsic rewards.
Representation learning for RL transforms image observations into latent vectors, whereas coordinate-state settings supply the representation directly through the task design.
Fully unsupervised behavioral learning uses intrinsic rewards and includes prediction-, entropy-, and mutual-information-based approaches.
These behavioral methods previously used different pre-training, evaluation, optimization, and environment procedures, motivating URLB’s standardized comparisons.

7 Conclusion

URLB standardizes evaluation of unsupervised RL across twelve tasks and three domains, but the implemented baselines do not solve the benchmark. The authors identify substantial room for improvement while noting important scope and safety limitations.

URLB evaluates unsupervised RL using twelve tasks from three domains with standardized pre-training and evaluation procedures.
Eight leading unsupervised RL algorithms are open-sourced and compared using the same optimization method to reduce confounding factors.
None of the implemented baselines solve URLB, although many make substantial progress and suggest directions for future research.
URLB tasks remain less visually and combinatorially complex than real-world robotics, limiting how directly benchmark performance transfers to that setting.
Unsupervised RL agents may develop policies misaligned with human intent because pre-training uses no extrinsic reward interactions.

Checklist

The checklist records affirmative responses for claim accuracy, limitations, ethics, reproducibility materials, and reporting practices, while several categories are marked not applicable. The paper also documents code, training details, error bars, and compute resources for its experiments.

The authors report that the main claims accurately reflect the paper’s contributions and scope, and that the work’s limitations are discussed in Section 7.
The paper reports discussing potential negative societal impacts and confirms conformity with ethics review guidelines.
The theoretical-results checklist is marked not applicable because the paper does not present theoretical results requiring assumptions or complete proofs.
The experiments provide reproducibility materials, training details, repeated-seed error bars, and compute-resource information.
The paper reports citing existing assets, documenting licenses, and marking human-subject and personally identifiable-information categories as not applicable.

A.2 Data-based Baselines

The data-based baselines use state or embedding distributions to drive exploration through entropy estimation, representation learning, or prototype-based density estimation. The experiments specify shared and per-algorithm hyperparameters, while continuous-control APS omits Atari’s generalized policy improvement.

Entropy-maximization approaches provide simple unsupervised pre-training algorithms, and URLB implements two leading variants.
APT estimates entropy with a K-nearest-neighbor particle estimator and uses forward or inverse dynamics losses to obtain latent representations.
ProtoRL decouples representation learning from exploration using contrastive clustering, learned prototypes, and KNN entropy estimation.
Competence-based methods maximize mutual information between encoded states and skills, using alternative entropy decompositions.
SMM combines target state-distribution matching with skill-conditioned density modeling, while DIAYN estimates skill information using a discriminator.
APS estimates H(z) with a particle estimator and H(z|w) with successor features; generalized policy improvement is not implemented for continuous-control experiments.

C Per-domain Individual Results

The appendix reports individual fine-tuning results across methods, tasks, domains, input modalities, and pre-training durations. Figure 5 aggregates domain-level performance after 100k fine-tuning steps and normalizes it to optimal performance.

Individual fine-tuning results are reported for each method, task, and pre-training duration across state- and pixel-based settings.
Figure 5 measures performance after 100k fine-tuning steps following different amounts of pre-training.
Figure 5 aggregates results across all tasks within each domain and normalizes performance relative to the optimal performance.
Figures 6 and 7 show fine-tuning efficiency as a function of pre-training steps for states-based and pixels-based learning.
Figure 8 provides fine-tuning curves for each evaluated unsupervised algorithm and task after intrinsic-reward pre-training.

E Individual Numerical Results

The paper presents individual fine-tuning results across tasks and methods for both states-based and pixels-based learning settings.

The reported results are organized by task and method rather than summarized as a single aggregate score.
Table 4 reports individual fine-tuning results for states-based learning.The evaluations use 1 × 10^5 frames after different levels of pre-training.
Table 5 reports individual fine-tuning results for pixels-based learning.The evaluations use 1 × 10^5 frames after different levels of pre-training.

F Compute Resources

URLB is designed for accessible single-GPU training, while its results indicate that data-based and knowledge-based methods are competitive and competence-based methods underperform. The paper attributes this gap to small skill spaces and limited discriminators in current competence-based approaches.

Compute Resources: Each state- and pixel-based algorithm requires a single GPU, with training taking roughly 30 minutes to 12 hours depending on snapshot and input.Large-scale experiments used NVIDIA Tesla V100 GPU instances.
Method Comparisons: RND and ProtoRL achieve similar fine-tuning scores despite maximizing data diversity through prediction error and entropy maximization, respectively.
Method Comparisons: Competence-based methods perform much worse than data-based and knowledge-based methods, possibly because current methods support only small skill spaces.
Competence-based Methods: The competence-based mutual-information objective uses a variational lower bound involving the skill variable z and discriminator q(z|τ).The discriminator is interpreted as a classifier, motivating small skill representations in practice.
Competence-based Methods: Fixed-length DeepMind Control episodes expose a larger behavior set than terminating environments, making small skill spaces insufficient and motivating more powerful discriminators.