PaperlayerView exact revision on arXiv

Source-linked AI summary

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum

arXiv:2010.13611v3cs.LG

TL;DR

Offline RL must learn from undirected data when online environment access is limited. OPAL extracts recurring temporally extended primitives before downstream learning, improving offline policy optimization and extending benefits to imitation and online transfer settings.

  • Problem

    Practical RL applications may severely limit continuous environment interaction, motivating methods that leverage unlabeled, undirected offline data.

  • Method

    OPAL uses unsupervised learning to extract recurring, temporally extended primitive behaviors from an offline dataset, then uses them for downstream task learning.

  • Results

    OPAL improves offline policy optimization without additional data and improves downstream task-learning speed and quality across offline RL, few-shot imitation, online RL, and multi-task transfer.

  • Takeaways & Limitations

    Primitive learning can provide temporal abstraction and a constrained action space that benefit downstream offline and online learning across benchmark tasks.

  • Takeaways & Limitations

    The paper assumes offline data is generated from mixtures of behavior policies and initial states, and focuses on simple auto-encoding models rather than image-based tasks.

Abstract

from arXiv · show

Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning. Primitives extracted in this way serve two purposes: they delineate the behaviors that are supported by the data from those that are not, making them useful for avoiding distributional shift in offline RL; and they provide a degree of temporal abstraction, which reduces the effective horizon yielding better learning in theory, and improved offline RL in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning as well as exploration and transfer in online RL on a variety of benchmark domains. Visualizations are available at https://sites.google.com/view/opal-iclr

1 INTRODUCTION

OPAL addresses offline RL settings where environment interaction is limited by extracting temporally extended primitives from diverse, undirected data. These primitives constrain downstream learning while improving offline optimization and other task-learning settings.

  • Motivation: Offline RL is motivated by practical settings where agents have large past datasets but limited ability to continuously interact with the environment.Real-world robot learning can require costly supervision, safety checks, and resets.
  • Method: OPAL uses unsupervised learning to extract recurring, temporally extended primitive skills from unlabeled and undirected experience data.The primitives distill behaviors that are useful and natural in environments such as undirected maze navigation.
  • Method: The learned primitives form a compact constrained action space, allowing offline task policies to focus on task-relevant behavior rather than relearning locomotion and navigation jointly.A point-to-point navigation policy can invoke locomotion primitives while concentrating on navigation.
  • Offline RL: OPAL uses one offline dataset for both primitive learning and subsequent task-directed offline policy optimization, without additional data.The authors report dramatic improvement over optimizing directly on the raw dataset.
  • Offline RL: OPAL’s hierarchy provides temporal abstraction that reduces compounding-error effects in offline RL, contrasting with online hierarchical RL’s emphasis on exploration.The paper presents this benefit as theoretically justified and experimentally verified.
  • Broader applications: OPAL also improves the speed and quality of downstream learning in online RL, few-shot imitation, and multi-task transfer learning.These applications use OPAL as a preprocessing step.

2 RELATED WORK

Prior work studies offline RL, online skill discovery, and hierarchical policy learning, whereas OPAL targets primitive discovery from static diverse datasets when environment access is restricted.

  • Offline RL: Offline RL methods learn from fixed transition-and-reward datasets and commonly constrain policies toward the data distribution to avoid out-of-distribution actions.Prior constraints include penalties based on KL divergence, MMD, and related distributional measures.
  • Online skill discovery: Online skill-discovery methods learn skills for planning, few-shot imitation, or online RL while assuming access to the environment.OPAL instead addresses settings with large static datasets and restricted environment access.
  • Hierarchical policy learning: Hierarchical policy learning combines low-level primitive policies with a high-level policy that directs them toward a task.Existing approaches may learn either discrete primitive sets or continuous primitive spaces.

3 PRELIMINARIES

The paper formalizes the MDP and data-generation setting for OPAL, then defines primitive extraction from unlabeled trajectories and their use in reward-labeled offline RL.

  • MDP formulation: The environment is modeled as an MDP M = (S, A, P, µ, r, γ), with state and action spaces, transitions, initial-state distribution, rewards, and discount factor.A policy maps states to action distributions and induces a discounted future-state distribution.
  • MDP formulation: The return is the discounted expected reward under the policy’s induced state distribution, and every-k-step state distributions describe states sampled at temporal intervals of k.The reward- and discount-agnostic environment is represented separately as E = (S, A, P, µ).
  • Offline data: OPAL assumes an unlabeled dataset D of sub-trajectories generated by sampling behavior policies and initial states, then executing each policy for c time steps.The policy and initial-state distributions are unknown and may represent diverse behaviors and starting locations.
  • Primitive extraction: Unsupervised learning extracts a continuous latent-conditioned primitive policy πθ(a|s, z), with z inferred in a latent space Z.The same sub-trajectories are also available in a reward-labeled dataset Dr for downstream offline RL.
  • Downstream learning: For downstream offline RL, OPAL labels Dr with latents, separates high- and low-return data, and trains a task policy to select primitives while fine-tuning the primitive policy with behavioral cloning.The overview describes autoencoding pretraining followed by offline RL and behavioral-cloning stages.

4 OFFLINE RL WITH OPAL

OPAL extracts continuous, temporally extended primitives from offline trajectories and uses them as a latent action space for downstream offline RL. Its analysis relates performance to data representativeness, primitive quality, temporal abstraction, and latent-space distribution shift.

  • 4.1 Extracting Temporally-Extended Primitives from Data: OPAL distills offline data into a continuous space of latent-conditioned primitive policies, then learns a high-level policy that selects primitives every c steps.The pipeline uses an encoder qφ(z|τ), primitive policy πθ(a|s,z), and prior ρω(z|s0) before downstream offline RL.
  • 4.1 Extracting Temporally-Extended Primitives from Data: The unsupervised objective combines action reconstruction with a KL constraint encouraging encoder and prior consistency for each c-step sub-trajectory.The constraint is implemented in practice as a weighted penalty, making the objective interpretable as a sequential β-VAE with a state-conditioned prior.
  • 4.2 Offline RL with OPAL: The learned primitive policy acts as a low-level controller, while an off-the-shelf offline RL algorithm learns a high-level latent policy from relabeled temporally extended transitions.In the experiments, CQL learns the high-level policy and the primitive policy is fine-tuned with latent-conditioned behavioral cloning.
  • 4.3 Suboptimality and Performance Bounds for OPAL: OPAL's performance analysis bounds primitive suboptimality using trajectory divergence and connects it to how representative the offline dataset is of the optimal policy.A policy is ζ-common when its c-length trajectory behavior is close, in total variation, to behaviors represented in the policy class.
  • 4.3 Suboptimality and Performance Bounds for OPAL: The theoretical bounds improve with more representative data and better primitives, but increasing primitive length c creates a trade-off by making autoencoding error harder to control.The resulting suboptimality depends on dataset mismatch ζ and autoencoding loss ϵc.
  • 4.3 Suboptimality and Performance Bounds for OPAL: Compared with the standard offline-RL bound, OPAL reduces horizon dependence through γc and error propagation through the latent action-space size |Z| rather than |A|.The bound also depends on divergence between the learned latent policy and the primitive distribution, controlled by conservative offline RL and the latent-space size.

5 EVALUATION

OPAL is evaluated on diverse offline datasets and downstream imitation, online-RL, and transfer settings. CQL+OPAL generally improves offline task learning, while temporal abstraction and learned primitives are also associated with gains in few-shot imitation, exploration, and multi-task transfer.

  • 5.1 Offline RL with OPAL: The evaluation uses D4RL Antmaze and Franka kitchen datasets containing varied, undirected or partially task-directed behavior for downstream task learning.Antmaze tests point-to-point navigation, while kitchen tests object rearrangement with sparse completion rewards.
  • 5.1 Offline RL with OPAL: CQL+OPAL outperforms nearly all offline-RL baselines on Antmaze and kitchen tasks, except for similar EMAQ performance on kitchen mixed.The ablation with c = 1 performs similarly to standard CQL, supporting temporal abstraction as an important factor in the reported result.
  • 5.2 Few-Shot Imitation Learning with OPAL: BC+OPAL clearly outperforms BC and BC+SVAE in few-shot imitation, indicating benefits from temporal abstraction and the learned primitives.The comparison evaluates average success rate over expert demonstrations in Antmaze environments.
  • 5.3 Online RL and Multi-Task Transfer with OPAL: SAC+OPAL outperforms the online-RL baselines on Antmaze, while most baselines make no progress and SAC+OPAL fails only on Antmaze large with sparse rewards.The method is evaluated against HIRO, SAC+BC, and DDQN+DDCO.
  • 5.3 Online RL and Multi-Task Transfer with OPAL: PPO+OPAL outperforms PPO and SAC on MT10 and MT50, where OPAL primitives support online multi-task transfer.The transfer setup learns primitives from expert pick-and-place data and applies them to simultaneous robotic manipulation tasks.

6 DISCUSSION

OPAL extracts recurring primitive behaviors from diverse, unlabeled offline data and uses them as a preprocessing representation for downstream learning. The paper derives conditions for offline-RL improvements, demonstrates gains across several settings, and identifies image-based scaling as future work.

  • OPAL extracts recurring primitive behaviors from undirected, unlabeled datasets containing diverse behaviors.The method is presented as a preprocessing step for downstream learning.
  • The paper derives theoretical conditions under which OPAL improves downstream offline-RL learning.
  • OPAL produces significant improvements on complex manipulation tasks in practice.
  • The paper also demonstrates OPAL for few-shot imitation learning, online RL, and online multi-task transfer learning.
  • The work uses simple auto-encoding models and identifies scaling the paradigm to image-based tasks as future work.

Appendices

The appendices describe OPAL applications to few-shot imitation learning and online RL using offline-learned primitives. Experiments report strong performance relative to baselines, while online RL relies on environment access and task-specific evaluation settings.

  • Few-Shot Imitation Learning: Few-shot imitation uses OPAL primitives as a low-level controller and learns a high-level latent policy from expert demonstrations.Expert sub-trajectories are labeled with the OPAL encoder and used for behavioral cloning.
  • Few-Shot Imitation Learning: The few-shot imitation evaluation uses 10 successful expert trajectories in Antmaze environments.
  • Few-Shot Imitation Learning: BC+OPAL clearly outperforms the evaluated baselines, supporting the importance of temporal abstraction and the quality of learned primitives.SVAE performs slightly worse, which the authors associate with directly decoding state trajectories rather than predicting actions.
  • Online RL: For online RL, OPAL fixes the learned low-level primitives and trains a high-level policy that selects one primitive every c steps.The high-level policy can be trained with an off-the-shelf online RL algorithm; the experiments use SAC.
  • Online RL: SAC+OPAL outperforms all baselines on the evaluated online-RL comparisons.All baselines except HIRO on Antmaze large with dense rewards fail to make progress; SAC+OPAL fails on Antmaze large with sparse rewards.

A.3 ONLINE MULTI-TASK TRANSFER LEARNING WITH OPAL

For online multi-task transfer, OPAL learns state-agnostic action primitives from an offline dataset and trains task-conditioned high-level policies in new MDPs. PPO+OPAL outperforms PPO and SAC on the Metaworld evaluation.

  • Setup: The multi-task setting assumes additional MDPs sharing the original action space and discount factor.
  • Method: OPAL transfers pretrained primitives while learning high-level policies conditioned on the state and task index.
  • Method: Because new MDPs may have different state spaces, the transferred primitives are made blind to state and decode action sub-trajectories.This transfers action sub-trajectory structure rather than the offline dataset's state distribution.
  • Evaluation: The Metaworld offline dataset comes from an expert pick-and-place task containing basic operations such as moving, grasping, and placing.
  • Results: PPO+OPAL clearly outperforms PPO and SAC, demonstrating the importance of temporal abstraction for online multi-task transfer.

B.1 BOUNDING THE SUBOPTIMALITY OF THE LEARNED PRIMITIVES

The appendix develops theoretical bounds for OPAL's learned primitives and downstream policy performance. The analysis relates performance to primitive approximation, data representativeness, latent-policy divergence, and the effective horizon created by temporal abstraction.

  • A general performance-difference lemma bounds return differences using total-variation divergence between action policies.
  • The corresponding temporally abstracted bound measures divergence over c-length sub-trajectories and uses an effective discount factor γc.The abstraction treats each c-step sub-trajectory as one action with rewards accumulated across the segment.
  • Theorem 4.1 states that, with high probability, a ζ-common policy has a latent-primitives representation whose quality is bounded under the OPAL objective.
  • Theorem 4.2 bounds the CQL-based hierarchical policy's return relative to the data-generating policy by a quantity κ.The bound is stated with high probability and compares the learned policy using πθ with πβ.
  • The bound's worst-case deterioration depends on latent-policy divergence and latent-space size, while temporal abstraction reduces the effective task horizon by a factor of c.

C.1 OPAL EXPERIMENT DETAILS

OPAL combines trajectory encoding, state-conditioned priors, and latent-conditioned primitive policies, then trains task policies over latent primitives using offline data. The section specifies the sequential-VAE objective and experimental configurations.

  • Architecture: The encoder maps length-c trajectories of states and actions to the mean and log standard deviation of a latent vector.It uses processed states, concatenated actions, and a bidirectional GRU.
  • Architecture: The prior maps the current state to the mean and log standard deviation of the latent vector.Its architecture is a fully connected network with two hidden layers and ReLU activation.
  • Architecture: The primitive policy conditions action distributions on the current state and latent vector.Kitchen environments use an autoregressive primitive policy.
  • Training configuration: OPAL uses H = 200 for antmaze and H = 256 for kitchen, training for 100 epochs with learning rate 1e −3, β = 0.1, Adam, and batch size 50.These settings apply to OPAL training in both environment families.
  • Task policy: The task policy is a three-hidden-layer fully connected network that outputs the mean and log standard deviation of the latent vector.The hidden layers have size 256 and use ReLU activation.
  • Objective: The sequential-VAE objective fits length-c sub-trajectories from an undirected, unlabelled, diverse dataset using an encoder, prior, and decoder.The derivation expands trajectory likelihood across timesteps and yields an autoencoding-style lower-bound objective.

E ABLATION STUDIES

The ablations examine latent dimensionality, temporal abstraction, and alternative discrete skill-discovery methods. They support selecting dim(Z) = 8 and c = 10 while identifying limitations of discrete alternatives.

  • Latent dimensionality: dim(Z) = 8 and 16 gave similar performances, while dim(Z) = 4 performed slightly worse; dim(Z) = 8 was selected for simplicity.The comparison uses antmaze-medium (diverse) with c = 10.
  • Temporal abstraction: c = 1 with dim(Z) = 8 performed similarly to CQL at 55.3 ± 3.8, supporting temporal abstraction as a source of OPAL’s gain.The ablation compares temporal abstraction against a latent action space without extended primitives.
  • Scope and tuning: dim(Z) = 8 and c = 10 worked across environments, but CQL+OPAL could improve with environment-specific tuning or offline hyperparameter selection.The authors identify this optimization as future work.
  • Alternative skill discovery: Offline CARML and offline DADS extract discrete skills by clustering trajectories, then use behavioral cloning to learn the corresponding primitive policy.The clustering objective maximizes mutual information between state trajectories and latent skill labels.

F.1 RESULTS

The results compare OPAL with offline DADS and CARML on antmaze-medium (diverse), using success and dense-reward metrics alongside trajectory and visitation visualizations. OPAL outperforms both discrete alternatives and avoids their observed maze failures, while those alternatives also ignore much of the state space.

  • Discrete skill discovery: k = 10 was selected for the discrete skill-discovery baselines because k = 10 and 20 worked best and k = 10 was simpler.Offline CARML effectively used 6 skills, while offline DADS used all skills.
  • Quantitative results: CQL+OPAL outperformed CQL combined with offline DADS and offline CARML on success rate, cumulative dense reward, and last-5-step dense reward.Table 7 reports these comparisons on antmaze medium (diverse), with k = 10 for DADS and CARML.
  • Mechanistic comparison: Offline CARML’s spatial-location clustering fails to separate control modes sharing starting locations, while offline DADS separates modes more effectively through state-transition clustering.CQL+offline DADS slightly improves upon CQL but remains limited by its discrete number of skills.
  • Behavioral visualization: Figure 4 colors subsets of dataset trajectories by their assigned clusters for offline DADS and offline CARML using k = 10.The figure visualizes the resulting discrete-skill trajectory clusters.
  • Behavioral visualization: Offline CARML and offline DADS became stuck at maze corners, whereas OPAL found a path to the goal location in the top right.The comparison is shown through state-visitation heatmaps for antmaze-medium policies.
  • Limitations: Both offline CARML and offline DADS consider only the ant’s global x-y pose and fail when the full state space is used.Their clustering ignores other state dimensions.
Loading 2010.13611v3…