PaperlayerView exact revision on arXiv

Source-linked AI summary

COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, Sergey Levine

arXiv:2010.14500v1cs.LGcs.RO

TL;DR

Robotic reinforcement-learning policies trained on task-specific data may not handle unfamiliar starting conditions, and prior approaches often rely on explicit skill hierarchies or decompositions. This paper combines prior interaction data with task data through offline reinforcement learning to compose behaviors. The authors report generalization to new initial conditions, including multi-stage tasks whose stages were never seen together in one episode.

  • Problem

    Task-specific robotic RL data may not cover the initial conditions needed for a skill, while approaches using pre-collected data often rely on explicit hierarchies or skill decomposition.

    Source: §1 · pp. 1–2

  • Method

    COG applies model-free offline RL to combine unlabeled prior interactions with task-specific data, using conservative Q-learning to incorporate prior data.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    COG produces previously unseen skill combinations and generalizes to new initial conditions, including multi-stage tasks whose stages never appeared together in a single episode.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Model-free reinforcement learning can stitch prior behaviors into task performance without explicit planning or skill hierarchies.

    Source: §7 · p. 9

Abstract

from arXiv · show

Reinforcement learning has been applied to a wide variety of robotics problems, but most of such applications involve collecting data from scratch for each new task. Since the amount of robot data we can collect for any single task is limited by time and cost considerations, the learned behavior is typically narrow: the policy can only execute the task in a handful of scenarios that it was trained on. What if there was a way to incorporate a large amount of prior data, either from previously solved tasks or from unsupervised or undirected environment interaction, to extend and generalize learned behaviors? While most prior work on extending robotic skills using pre-collected data focuses on building explicit hierarchies or skill decompositions, we show in this paper that we can reuse prior data to extend new skills simply through dynamic programming. We show that even when the prior data does not actually succeed at solving the new task, it can still be utilized for learning a better policy, by providing the agent with a broader understanding of the mechanics of its environment. We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task, with our hardest experimental setting involving composing four robotic skills in a row: picking, placing, drawer opening, and grasping, where a +1/0 sparse reward is provided only on task completion. We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands, and present results in both simulated and real world domains. Additional materials and source code can be found on our project website: https://sites.google.com/view/cog-rl

1 Introduction

The paper asks how unlabeled robot interaction data can extend a newly learned skill to unfamiliar starting conditions without explicit skill hierarchies. It proposes combining prior and task-specific data through offline reinforcement learning, and demonstrates skill composition in simulation and on a real robot.

1 Introduction

  • The motivating gap is that a skill trained to remove an object from an open drawer may fail when the drawer is closed, while explicit hierarchies or skill decomposition might seem necessary.The paper asks whether unlabeled interaction data and offline RL can combine behaviors automatically instead.Source: §1 · p. 1
  • Prior data can enable multi-stage behavior in unfamiliar conditions, even when the stages never appeared together in one episode.For example, a robot can open a drawer before removing an object, without collecting new drawer-opening data for that task.Source: §1 · p. 2
  • The method uses prior data even when it does not succeed at the new task, because it can provide information about environment mechanics.The authors initialize the replay buffer with prior data, assign its transitions zero reward, and train CQL on it alongside task-specific data.Source: §1 · p. 2
  • COG combines task-specific data with prior data through model-free offline RL, composing previously unseen skills to meet task preconditions and generalize to new initial conditions.The system trains policies from high-dimensional images with sparse binary rewards and is evaluated in simulation and on a real-world low-cost robotic arm.Source: §1 · p. 2

2 Related Work

The related work distinguishes COG from approaches that generalize actions across objects or use explicit prediction, hierarchy, or imitation to compose behavior. It adapts existing offline RL methods to combine data from a different domain for new initial conditions.

2 Related Work

  • Unlike prior offline-RL robotics work focused on new objects or greater variability [11] [22], this work targets new initial conditions requiring different skills, such as opening a drawer before grasping.Most other robotic manipulation RL works cited rely on online data collection rather than prior datasets.Source: §2 · p. 3
  • Where much data-driven robotics repeats actions on novel objects [25], this work uses prior interactions to learn new behavior through model-free RL.Other cited approaches use video prediction for temporally extended tasks [29] [30] [9] [31] [32], hierarchical policies [33], or offline imitation to compose demonstrations.Source: §2 · p. 3
  • The paper adapts existing offline deep RL methods rather than proposing a new algorithm, integrating prior data from a different domain to learn a task that succeeds across conditions.Offline deep RL trains on static, previously collected datasets without environment interaction [1] [3] [2] [4] [5] [38] [39] [40].Source: §2 · p. 3

3 Incorporating Prior Data into Robotic Reinforcement Learning

The problem is to learn a new robotic task from sparse-reward task data while using diverse prior interactions to generalize across initial conditions. The prior data describes environment transitions without reward labels, while the task data carries rewards for the objective.

3 Incorporating Prior Data into Robotic Reinforcement Learning

  • The goal is to use diverse previously collected data with reinforcement learning so new robotic skills generalize when test-time conditions differ from task-training data.The paper uses “common sense” for handling small task variations, such as opening a drawer before removing an object, without explicit precondition reasoning or hierarchical planning.Source: §3 · p. 3
  • The setting combines task-agnostic, unlabeled prior transitions Dprior with task-specific transitions DT that include rewards for the target task.Prior data alone cannot train a policy for a particular objective, but it informs the dynamics; task data can train a policy to maximize observed reward.Source: §3 · pp. 3–4
  • The input is unlabeled Dprior plus sparsely rewarded DT, and the output is a policy for task T that should generalize to new initial conditions.Figure 2 depicts Q-learning propagating information backward and using Bellman backups over prior data to learn actions from prior-data initial conditions.Source: §3 · p. 4, Figure 2 · p. 4

4 Connecting New Skills to Past Experience via Dynamic Programming

Q-learning can propagate task reward backward through task data and connect it to prior trajectories, allowing a policy to act from states absent from task-specific data. COG uses conservative Q-learning to incorporate prior data offline, with optional online fine-tuning.

4 Connecting New Skills to Past Experience via Dynamic Programming

  • Task-only training can assign a new test state zero or arbitrary value, while training across prior data can provide paths into states where the task is solvable.Without prior data, covering every possible starting state would make training large skill repertoires prohibitively costly.Source: §4 · pp. 4–5
  • Q-learning can use prior trajectories to connect unseen initial states to states from which task-specific behavior succeeds, even without a trajectory that performs both behaviors.A non-zero overlap between prior and task data is sufficient for Q-learning to learn from the prior data; no explicit planning or skill decomposition is required.Source: §4 · p. 5
  • COG uses conservative Q-learning, which penalizes Q-values on out-of-distribution actions and trains values to lower-bound policy value.The objective combines a log-sum-exp penalty with maximizing Q-values on dataset actions and standard Bellman-error training.Source: §4 · p. 5
  • The policy can be trained offline with CQL for 1m gradient steps, then fine-tuned online using additional environment data.During fine-tuning, the method continues using CQL but only the newly collected online data.Source: §4 · p. 5

5 End-to-End Robotic Learning with Prior Datasets

The system combines task-specific learning with prior data in an image-based robotic RL setup. It uses sparse rewards, scripted prior-data collection, and convolutional networks for the policy and Q-function.

System overview

  • The section describes how the proposed learning approach is instantiated in a practical robotic system.Source: §5 · p. 5

MDP for robotic skills

  • The MDP observes camera images and robot state, uses six continuous and two discrete actions, and rewards successful task completion with +1 and all other states with zero.Simulated images are 48 × 48 and real-robot images are 64 × 64; the robot state includes end-effector pose and gripper opening.Source: §5 · p. 5

Data collection

  • Prior data is collected before task-specific learning using weak scripted policies that bias interaction toward interesting behaviors.These policies typically succeed 30–50% of the time, depending on task complexity.Source: §5 · pp. 5–6

Neural network architectures

  • Convolutional networks represent both the policy and Q-function, allowing behaviors to be learned directly from visual observations.The shared eight-layer architecture combines convolutional and max-pooling layers with fully connected layers.Source: §5 · p. 6

6 Experiments

Experiments evaluate whether offline RL can use prior robotic data to solve multi-step tasks from novel initial conditions, in simulation and on a real robot. COG performs well across simulated conditions, outperforms the listed baselines on novel conditions, and also succeeds in the real-world closed-drawer test.

6.1 Experimental Setup

  • The pick-and-place task rewards placing an object in a tray, with a harder condition requiring grasping first; its unlabeled prior set has 10K grasp attempts and its rewarded task set has 5K placing attempts.No trajectory solves the complete task, although the two datasets have overlapping state distributions.Source: §6.1 · p. 6, Figure 3 · p. 6
  • The drawer task trains grasping from an open drawer but tests prerequisite chains including opening a closed drawer, clearing a blocked handle, and moving an obstruction.The prior dataset has 1.5m transitions and the task-specific dataset has 300K; prior data includes no interactions with the object inside the drawer.Source: §6.1 · pp. 6–7, Figure 4 · p. 7

6.2 Empirical Results

  • COG succeeds on both simulated tasks in the majority of cases across initial conditions and significantly outperforms behavioral-cloning baselines, ablations, and SAC on novel conditions.SAC diverged in the reported runs.Source: Table 1 · p. 7, §6.2 · p. 7

6.1 Experimental Setup

  • Comparisons include BC-init, BC, BC-oracle, SAC, and a no-prior-data ablation.Source: §6.1 · p. 7

6.2 Empirical Results

  • COG learns to open closed drawers and move obstructions before grasping, despite having no reward labels in prior data or complete episodes combining drawer opening and object removal.Source: §6.2 · p. 7
  • Behavioral cloning performs far below COG and BC-oracle cannot complete the task; without prior data, the ablation solves only task-specific initial states, while BC-init fails on unseen ones.The authors attribute BC's weakness to difficulty distinguishing task-supporting behavior from meaningful but currently unhelpful behavior, and suggest catastrophic forgetting for BC-init.Source: §6.2 · p. 8
  • Online fine-tuning raises success above 90% for all novel drawer-task initial conditions using 500–4,000 additional episodes, depending on the task.The figure reports that only a small amount of additional data is required.Source: §6.2 · p. 8, Figure 5 · p. 8
  • In the real-world closed-drawer test, the learned policy succeeds in 7/8 trials, while BC-oracle never succeeds.Training used grasping data for an open drawer and prior drawer-opening data, without a complete opening-and-grasping trajectory.Source: §6.2 · p. 8, Figure 6 · p. 8

7 Discussion

Model-free RL can stitch prior behaviors into skills that work from new initial conditions, without an explicit hierarchy or world model. The authors identify offline-only training and naive reward labeling as limitations and point to online fine-tuning and reward inference as directions for improvement.

7 Discussion

  • Prior data lets model-free RL stitch together behaviors that handle test-time initial conditions different from those used to learn a task.The example is a robot learning to open a drawer at test time before grasping an object, despite never seeing the complete sequence in training.Source: §7 · p. 9
  • The authors suggest these multi-stage behaviors can emerge without explicit hierarchy or model representation when training data captures relevant behaviors.Source: §7 · p. 9
  • The approach is limited to offline learning, and the authors say online fine-tuning and more sophisticated prior-data reward inference could improve results.Source: §7 · p. 9

A Experimental Setup Details

The simulated and real environments use the same eight-dimensional control scheme, specifying end-effector position and orientation changes alongside gripper and neutral-position commands.

A Experimental Setup Details

A.1 Data Collection Policies

The appendix describes noisy scripted policies for collecting manipulation data, neural network choices for simulated and real experiments, and CQL training settings.

A.1 Data Collection Policies

  • Scripted policies use approximate object localization and noisy actions for grasping, random workspace drop points for placing, and handle-directed motions for drawer opening or closing.Grasping success rates were 50% in simulated pick-and-place, 70% in the simulated drawer environment, and 30% in the real drawer environment.Source: §A.1 · p. 12
  • Scripted trajectories return the robot to its starting configuration, which the authors believe increases dataset state-distribution overlap and enables relevant trajectory stitching.Source: §A.1 · p. 12

A.2 Neural Network Architectures

  • The selected real and simulated network architectures map image observations to robot commands; the Q-function additionally receives actions, while the simulated design omits gripper position and orientation.The authors report these architectures performed best among tested networks with 2–4 convolutional layers and 4–16 filters per layer.Source: Figure 7 · p. 14, Figure 8 · p. 14, §A.2 · p. 14

A.3 Hyperparameters for Reinforcement Learning

  • The method uses CQL(H) with α=1.0, one negative sample for logsumexp estimation, and a 10K-gradient-step behavior-cloning warm start.Using one rather than the default ten negative samples substantially reduces training wall-clock time for image observations.Source: §A.3 · pp. 14–15

B Comparison to BC + SAC for online fine-tuning

CQL fine-tuning substantially outperforms fine-tuning a behavior-cloned policy with SAC in the closed-drawer grasping comparison. SAC begins with low performance and struggles to perform well on harder, longer-horizon tasks from BC initialization.

B Comparison to BC + SAC for online fine-tuning

C Learning Curves

COG achieves high performance across the reported initial conditions and substantially outperforms behavioral-cloning comparisons. Figure 10 shows learning curves across four settings where prior data is essential.

C Learning Curves

  • COG achieves high performance across all initial conditions for both tasks, substantially outperforming behavioral-cloning baselines, including an oracle using high-reward trajectories.Figure 10 covers the place-in-tray task with the object starting in the tray and grasp-from-drawer cases with a closed or blocked drawer.Source: Figure 10 · p. 16, C Learning Curves · p. 16
  • Figure 10 compares COG with three behavioral-cloning baselines across four settings where prior data is essential.The settings are object starting in the tray, and grasping from a closed drawer or either of two blocked-drawer conditions.Source: Figure 10 · p. 16
  • The learning-curve x-axis counts policy and Q-function update steps, not the amount of data available.All data is available to the methods from the start of offline training.Source: C Learning Curves · p. 16
Loading 2010.14500v1…