PaperlayerView exact revision on arXiv

Source-linked AI summary

Jump-Start Reinforcement Learning

Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, Karol Hausman

arXiv:2204.02372v2cs.LG

TL;DR

Efficiently learning RL policies from scratch is difficult, and naïvely initializing value-based methods with prior policies can fail. The paper introduces JSRL, which rolls in a guide-policy to create a curriculum for an exploration-policy. JSRL outperforms existing imitation and reinforcement learning approaches in reported tasks, while its analysis gives a polynomial rather than exponential-in-horizon sample-complexity bound for non-optimism exploration methods.

  • Problem

    Early RL may require randomly encountering rewards, while value-based methods struggle to bootstrap effectively from a prior policy without a pre-trained Q-function.

  • Method

    JSRL uses a guide-policy to roll in toward useful states, then hands control to a self-improving exploration-policy whose guide influence diminishes over time.

  • Results

    JSRL significantly outperforms previously proposed imitation and reinforcement learning approaches on benchmark and vision-based robotic tasks.

  • Takeaways & Limitations

    With a guide-policy, non-optimism exploration methods improve from exponential to polynomial dependence on task horizon in the paper’s sample-complexity analysis.

  • Takeaways & Limitations

    JSRL is susceptible to biases and quality problems in the guide-policy or training data, and an adversarial guide-policy can make learning slower than random exploration.

Abstract

from arXiv · show

Reinforcement learning (RL) provides a theoretical framework for continuously improving an agent's behavior via trial and error. However, efficiently learning policies from scratch can be very difficult, particularly for tasks with exploration challenges. In such settings, it might be desirable to initialize RL with an existing policy, offline data, or demonstrations. However, naively performing such initialization in RL often works poorly, especially for value-based methods. In this paper, we present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy, and is compatible with any RL approach. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks: a guide-policy, and an exploration-policy. By using the guide-policy to form a curriculum of starting states for the exploration-policy, we are able to efficiently improve performance on a set of simulated robotic tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms, particularly in the small-data regime. In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.

1. Introduction

The paper addresses how to bootstrap value-based RL from a reasonable but sub-optimal prior policy despite difficult early exploration. JSRL rolls in a guide-policy to create starting-state curricula, then lets an exploration-policy improve autonomously.

  • Motivation: RL can improve behavior through trial and error, but early learning often requires randomly encountering rewards.Prior policies, demonstrations, offline data, or manual engineering can provide guidance, yet value-based methods are difficult to initialize from a policy alone.
  • Method: JSRL gradually rolls in a guide-policy that provides a curriculum of starting states for the exploration-policy.The exploration-policy begins closer to the goal, while the guide-policy’s influence diminishes as exploration improves.
  • Contribution: JSRL is designed to accelerate the early phases of RL while remaining compatible with prior policies and existing offline or online RL methods.The approach is presented as a generic method for downstream RL algorithms that require environmental exploration.
  • Contribution: Experiments report that JSRL significantly outperforms previously proposed imitation and reinforcement learning approaches on benchmark and vision-based robotic tasks.The paper also derives an upper bound on JSRL’s sample complexity compared with classic RL alternatives.

2. Related Work

Related work combines prior policies, data, curricula, or theoretical exploration methods with RL, but these approaches differ in assumptions and compatibility. JSRL is positioned as using only a prior policy while supporting value-based exploration.

  • Imitation learning combined with reinforcement learning: IL+RL methods include sequence modeling, behavioral-cloning initialization, replay-buffer data, and offline-RL pretraining before fine-tuning.Behavioral cloning can initialize policy-search methods but is generally ineffective with actor-critic or value-based methods when the critic is not initialized.
  • Imitation learning combined with reinforcement learning: Some prior approaches require prior datasets, whereas JSRL requires only a prior policy and can perform comparably or better.The cited comparison emphasizes weaker data assumptions for JSRL.
  • Curriculum learning and exact state resets for RL: Curriculum-based exploration methods often assume exact resets to arbitrary states or use demonstration states as starting points.Other methods construct start-state curricula by beginning at the goal and expanding the start-state distribution.
  • Provably efficient exploration techniques: Provably efficient exploration methods use confidence intervals such as UCB or Thompson sampling, but these can be difficult to approximate with neural networks or have exponential worst-case sample complexity.The paper instead leverages a pre-trained guide-policy for exploration.
  • “Rolling in” policies: Prior work also uses ensembles, learned dynamics for look-ahead, or fixed controllers to provide bootstrap data or guide useful actions.These approaches form the broader literature on rolling in policies for RL initialization and exploration.

3. Preliminaries

The paper formulates RL in a Markov decision process and focuses on using a prior policy in value-based learning. A motivating experiment shows that naïve actor initialization can fail when the critic starts untrained.

  • MDP formulation: The environment is modeled as an MDP M = (S, A, P, R, p0, γ, H), with states, actions, transitions, rewards, initial-state distribution, discount factor, and horizon.The RL objective is to find a policy maximizing expected discounted trajectory reward.
  • Naïve policy initialization: Naïve actor-critic fine-tuning can cause the initially useful policy to be forgotten when the critic is randomly initialized.The untrained critic supplies a poor learning signal; later experiments allow the critic to warm up before fine-tuning.
  • Prior-policy initialization: Offline RL methods commonly use pre-trained, regularized Q-functions that can be improved with online data.This differs from the setting considered here, where only a prior policy is available and the Q-function is fresh.
  • Prior-policy initialization: When a pre-trained Q-function is unavailable, value-based RL struggles to incorporate a prior policy effectively.The experiment starts with a pre-trained actor and a fresh Q-function to isolate this setting.

4. Jump-Start Reinforcement Learning

JSRL initializes RL with a guide-policy that progressively hands control to an exploration-policy, creating a curriculum of starting states. Its analysis shows that guide-policy coverage can make non-optimism-based exploration polynomially sample-efficient, whereas exploration from scratch can require exponential samples.

  • Two-policy approach: JSRL combines a fixed guide-policy with an online exploration-policy, allowing the latter to learn from states reached by the former.The guide-policy may have any form, while the exploration-policy adapts through online experience.
  • Two-policy approach: The curriculum starts with the guide-policy acting for a long prefix, then progressively reduces that prefix as the exploration-policy improves.At each stage, the exploration-policy masters a smaller portion of the state space needed to reach states covered earlier.
  • Algorithm: At each episode, JSRL rolls out the guide-policy for h steps and the exploration-policy for the remaining H − h steps, then updates the latter with collected data.The combined policy is evaluated after training, and the algorithm can use standard procedures such as ϵ-greedy Deep Q-Network training.
  • Algorithm: JSRL supports curriculum and random-switching guide-step schedules; the paper calls these variants JSRL and JSRL-Random, respectively.The curriculum variant decreases guide steps based on evaluations, while random switching samples guide steps uniformly and independently.
  • Theoretical analysis: Without a guide-policy, ϵ-greedy can require exponential samples in horizon, while JSRL with suitable guide-policy coverage achieves polynomial sample complexity.The informal theorem gives suboptimality O(CH^5/2S^1/2A/T^1/2) for tabular MDPs and a near-optimal bound up to C · poly(H) with general function approximation.

5. Experiments

The experiments evaluate JSRL against IL+RL baselines on D4RL benchmarks and simulated robotic grasping tasks, including sensitivity to data quality, curriculum design, guide-policy quality, and task generalization. Across these settings, JSRL is especially effective in low-data regimes and improves sample efficiency and final performance on robotic tasks.

  • 5.1. Comparison with IL+RL baselines: IQL+JSRL is competitive with IL+RL baselines using full D4RL datasets but performs significantly better in the small-data regime.The comparison uses averaged normalized scores after 1m online fine-tuning steps.
  • 5.1. Comparison with IL+RL baselines: The experiments test JSRL against competitive IL+RL baselines on challenging D4RL and vision-based robotic manipulation tasks.D4RL experiments use offline pre-training followed by online fine-tuning, while robotic experiments address complex vision-based grasping.
  • 5.2. Vision-Based Robotic Tasks: QT-Opt+JSRL is more sample efficient and attains higher final performance than baseline IL+RL methods on two simulated robotic grasping tasks.Each plotted line reports the mean and standard deviation over three random seeds.
  • 5.3. Initial Dataset Sensitivity: Only QT-Opt+JSRL learns with 20 demonstrations, 100x fewer than the standard 2,000 demonstrations, on the difficult robotic grasping tasks.The experiments show that limiting initial demonstrations is challenging for IL+RL baselines.
  • 5.3. Initial Dataset Sensitivity: JSRL outperforms baselines in low-data settings, while AW-Opt and BC become more competitive with 20,000 demonstrations.This pattern suggests JSRL’s benefits are most prominent when offline data does not densely cover good state-action pairs.
  • 5.4. JSRL-Curriculum vs. JSRL-Random Switching: JSRL outperforms JSRL-Random in early sample efficiency, although their converged performance is comparable.The same trend holds when the quality of the guide-policy is limited by constraining the initial dataset.
  • 5.5. Guide-Policy Generalization: A guide-policy trained for indiscriminate grasping still improves instance grasping over vanilla QT-Opt, although it performs worse than an instance-specific guide.Both JSRL variants outperform vanilla QT-Opt in this cross-task transfer setting.

6. Conclusion

The paper presents JSRL as a generic way to use a prior policy to improve exploration and sample efficiency across offline RL benchmarks and simulated vision-based robotic tasks. Its theoretical analysis gives an upper bound showing a change from exponential to polynomial horizon dependence for non-optimism exploration methods.

  • 6. Conclusion: JSRL uses a pre-existing guide-policy to create a starting-state curriculum for an exploration-policy that eventually improves autonomously.The guide-policy’s influence diminishes as the exploration-policy improves.
  • 6. Conclusion: JSRL is generic because it can combine with any RL method, including value-based approaches that have traditionally struggled with exploration.The method was evaluated on offline RL benchmarks and challenging vision-based robotic simulation tasks.
  • 6. Conclusion: JSRL is more sample efficient than more complex IL+RL approaches while remaining compatible with their benefits.The reported experiments cover benchmark tasks and vision-based robotic simulation tasks.
  • 6. Conclusion: The sample-complexity analysis shows a from-exponential-to-polynomial improvement in time-horizon dependence for non-optimism exploration methods.The result is obtained with the help of a guide-policy.

7. Limitations

JSRL’s usefulness is constrained by the quality and biases of the pre-existing policy or training data. Poor or adversarial guide-policies can reduce safety and effectiveness, and may even slow learning relative to random exploration.

  • 7. Limitations: JSRL can inherit biases from the training data or guide-policy, affecting the policy it discovers.The limitation is especially important in high-risk domains such as robotics.
  • 7. Limitations: Poor or misguided guide-policies can harm the safety and effectiveness of JSRL in high-risk settings.The authors emphasize the need for carefully curated training data and guide-policies.
  • 7. Limitations: An adversarial guide-policy can make learning slower than random exploration by constraining the agent until the curriculum is complete.The paper gives a static guide-policy in a small-maze task as an example.

A.1. Imitation and Reinforcement Learning (IL+RL)

Prior IL+RL work commonly initializes reinforcement learning with policies trained through imitation or offline data, then improves them through online reinforcement learning. These approaches motivate comparison with JSRL but generally rely on prior data.

  • A.1. Imitation and Reinforcement Learning (IL+RL): IL+RL methods commonly pre-train policies on offline data and then fine-tune them online.The paper contrasts this workflow with its use of demonstrations or sub-optimal pre-existing policies to accelerate RL.
  • A.1. Imitation and Reinforcement Learning (IL+RL): Prior IL+RL approaches include sequence-modeling methods, behavioral-cloning initialization, and offline RL followed by fine-tuning.These approaches differ in implementation but use prior data to initialize or improve RL.

A.1.1. D4RL

AWAC, CQL, and IQL are offline-to-online reinforcement learning baselines with different strategies for handling critic estimation and out-of-distribution actions.

  • AWAC updates its critic with dynamic programming while constraining the actor toward the offline behavior policy.
  • CQL regularizes Q-values to reduce estimation errors from Bellman updates involving out-of-distribution actions.
  • IQL avoids estimating values for actions absent from the offline dataset and represents a recent state-of-the-art IL+RL baseline.

A.1.2. SIMULATED ROBOTIC GRASPING

AW-Opt combines AWAC and QT-Opt to fine-tune offline-trained policies, while QT-Opt targets high-dimensional robotic control from pixels.

  • AW-Opt combines AWAC and QT-Opt into a distributed actor-critic algorithm for fine-tuning policies trained offline.
  • QT-Opt has been demonstrated on complex, high-dimensional robotic control from pixels, beyond common simulation benchmarks such as D4RL.

A.2. Experiment Implementation Details

The experiments evaluate JSRL across challenging D4RL and robotic settings, while the appendix details implementations, initialization choices, theoretical rates, and analysis limitations.

  • IQL+JSRL: IQL+JSRL uses IQL pretraining to obtain a guide-policy, then applies JSRL during online fine-tuning with the IQL update as its training procedure.
  • Initialization: Cold-starting generally performs better with poor initial datasets, whereas warm-starting tends to perform better with dense, high-quality datasets.
  • Simulated robotic grasping: The simulated grasping tasks use a 7 DoF arm, RGB-image states, continuous gripper-position and yaw actions, discrete gripper controls, and sparse lifting rewards.
  • JSRL hyperparameters: JSRL introduces H1, n, and β to control initial guide steps, curriculum stages, and the threshold for advancing stages.
  • Theory: The theory establishes polynomial rates for tabular exploration with ϵ-greedy and FALCON+, including O(CH7/3S1/3A1/3/T 1/3) and O(CH5/2S1/2A/T 1/2).
  • Theory: With a guide-policy, non-optimism exploration methods can achieve polynomial sample complexity instead of suffering a curse of horizon.
  • Limitations: The analysis relies on visitation assumptions and contextual-bandit exploration, while the Q-learning-based exploration analysis remains future work.
Loading 2204.02372v2…