PaperlayerView exact revision on arXiv

Source-linked AI summary

AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine

arXiv:2006.09359v6cs.LGcs.ROstat.ML

TL;DR

The paper addresses the difficulty of continuing online RL improvement after learning from offline data, a challenge that matters for costly real-world robotics. It introduces AWAC, combining dynamic-programming critics with maximum-likelihood-style constrained actor updates, and reports rapid fine-tuning across simulated and real robotic tasks. The authors also identify threshold selection, limited theoretical understanding, and evaluation on isolated setups as limitations.

Abstract

from arXiv · show

Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings such as robotic control. If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill. Such prior data could either constitute expert demonstrations or sub-optimal prior data that illustrates potentially useful transitions. While a number of prior methods have either used optimal demonstrations to bootstrap RL, or have used sub-optimal data to train purely offline, it remains exceptionally difficult to train a policy with offline data and actually continue to improve it further with online RL. In this paper we analyze why this problem is so challenging, and propose an algorithm that combines sample efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of RL policies. We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. We demonstrate these benefits on simulated and real-world robotics domains, including dexterous manipulation with a real multi-fingered hand, drawer opening with a robotic arm, and rotating a valve. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.

I. INTRODUCTION

AWAC addresses the difficulty of using prior offline data for online RL by combining offline pre-training with online fine-tuning. Its constrained actor update and dynamic-programming critic support sample-efficient learning for challenging robotic tasks.

Motivation

  • Offline datasets can contain demonstrations, suboptimal-policy trajectories, related-task demonstrations, or random exploration, providing information about dynamics and tasks.Effective methods should pre-train on such data while continuing to collect online data.Source: §I · p. 1

Problem

  • Prior demonstration methods can bias learning toward data that may be non-optimal, while fully offline methods do not directly address continued online improvement.The paper seeks one algorithm that handles varied prior-data sources without privileged treatment.Source: §I · p. 1

Method

  • AWAC trains a critic with dynamic programming and updates a constrained actor using a supervised-learning-style objective.The critic leverages off-policy data, while the actor constraint mitigates offline distribution shift without overly conservative updates.Source: §I · p. 2

Evaluation

  • AWAC is evaluated on simulated dexterous manipulation and three real-world robots performing drawer opening, object pickup, and valve rotation.The tasks include high-dimensional action spaces and sparse, uninformative rewards.Source: Figure 1 · p. 1, §I · p. 2

Preliminaries

  • Actor-critic methods alternate between critic policy evaluation and actor policy improvement, using value estimates to maximize expected returns.The critic can be learned by Bellman updates, while the actor is updated to optimize the estimated action value.Source: §II · p. 2

Problem

  • Replay-buffer access to prior transitions alone is insufficient for the paper’s offline-pretraining and online-improvement setting.Standard actor-critic algorithms can use off-policy data in principle, but the paper analyzes why this does not solve the problem by itself.Source: §II · p. 2

III. CHALLENGES IN OFFLINE RL WITH ONLINE FINE-TUNING

Offline RL followed by online fine-tuning requires reusing arbitrary prior data while avoiding the failures of standard off-policy methods. The paper identifies bootstrapping error and inaccurate behavior-model constraints as central challenges.

Problem Setup

  • The setting provides a static dataset from an arbitrary policy mixture or human expert, then uses limited online interaction to learn an optimal policy.It covers demonstrations, random data, prior RL experiments, and multi-task data.Source: A. Problem Definition · p. 3

Existing Approaches

  • Imitation pre-training followed by on-policy RL is data-inefficient because it does not reuse prior data during fine-tuning.It also assumes the prior data may be optimal, which need not hold.Source: B. Data Efficiency · p. 3

Observed Failures

  • Standard off-policy actor-critic methods gain little from prior data: SAC with prior data performs similarly to SAC trained from scratch.Behavior-cloning pre-training can additionally cause an initial performance decrease before behavior returns near scratch-level learning.Source: C. Bootstrap Error in Offline Learning… · p. 3

Bootstrap Error

  • Off-policy bootstrapping accumulates error when target actions fall outside the static dataset’s distribution.Inaccurate Q-values for out-of-distribution actions propagate through actor-critic updates.Source: C. Bootstrap Error in Offline Learning… · p. 3

Conservatism

  • Offline RL methods constrain policy improvement using an estimated behavior distribution, but online fine-tuning makes that estimate difficult to maintain.The behavior distribution mixes offline and online policies, and inaccurate modeling can make constrained optimization overly conservative.Source: C. Bootstrap Error in Offline Learning… · p. 3

D. Excessively Conservative Online Learning

Standard constrained offline RL methods can perform well before online fine-tuning yet improve slowly afterward because explicit behavior models become inaccurate as new data arrives. AWAC addresses this with off-policy critic estimation and an implicit, advantage-weighted actor update that reuses replay data without explicitly modeling behavior.

Failure modes

  • Offline RL methods with constraints perform well initially but improve slowly during online fine-tuning.Their purely offline performance is much better at the start, yet additional online iterations produce only gradual gains.Source: D. Excessively Conservative Online… · p. 3
  • Behavior-model accuracy decreases during online fine-tuning as incoming data changes the replay distribution.The resulting mismatch can make constrained optimization overly conservative and limit improvement.Source: D. Excessively Conservative Online… · pp. 3–4

AWAC design

  • AWAC uses off-policy temporal-difference learning to estimate the critic while training the actor with an implicit policy constraint.This design combines data reuse with protection against offline instability and bootstrap error accumulation.Source: §IV · pp. 4–5
  • The policy improvement step keeps the updated policy close to observed data while maximizing the critic's estimated value.AWAC enforces this constraint implicitly rather than through an explicitly learned behavior model.Source: §IV · p. 4
  • AWAC's actor update is weighted maximum likelihood, reweighting replay-buffer state-action pairs by critic-predicted advantages without fitting a parametric behavior model.The update uses samples directly from the current dataset, avoiding explicit behavior modeling during fine-tuning.Source: §IV · p. 5

V. RELATED WORK

AWAC builds on prior work in off-policy learning, demonstration bootstrapping, maximum-likelihood policy optimization, and robotic learning from prior data. Its distinguishing scope is to use arbitrary suboptimal data and continue improving beyond it.

Off-policy and offline RL

  • Standard off-policy methods reuse training data but can struggle with previously collected data because distribution shift causes error accumulation.Offline RL methods address this issue by constraining policy updates.Source: §V · p. 5

Demonstration bootstrapping

  • Demonstration-based methods often initialize policies by behavioral cloning and then use on-policy fine-tuning, which is less sample-efficient than off-policy value estimation.Other methods add demonstrations to replay buffers, but these strategies can produce a large performance dip during online fine-tuning.Source: §V · pp. 5–6

Maximum-likelihood actor methods

  • AWAC differs from AWR by estimating the current policy's Q-function with bootstrapping rather than estimating the behavior policy's value function with Monte Carlo or TD-λ methods.The paper relates this choice to improved efficiency and final performance.Source: §V · p. 6

Robotic learning

  • Prior robotic-learning work used initial rollouts, motion primitives, reward shaping, or reward inference to bootstrap on-policy reinforcement learning.These approaches established bootstrapping from prior data as a recurring strategy for real-world robotic learning.Source: §V · p. 6
  • AWAC extends robotic bootstrapping beyond demonstrations to arbitrary suboptimal data while supporting continued improvement beyond that data.This broadens the type of prior experience that can participate in online reinforcement learning.Source: §V · p. 6

VI. EXPERIMENTAL EVALUATION

The evaluation tests whether AWAC combines prior data with online experience across challenging simulated and real-world robotic settings. Results emphasize rapid learning in sparse-reward dexterous manipulation and broad performance across locomotion tasks.

Evaluation goals

  • The evaluation asks whether AWAC improves complex robotic control efficiently, learns from suboptimal or random data, and bootstraps practical real-world reinforcement learning.The study uses challenging high-dimensional simulated tasks and three real-world robotic platforms.Source: §VI · p. 6, A. Simulated Experiments · p. 6

Real-world evaluation

  • The study includes real-world drawer opening, object pickup with a multi-fingered hand, and valve rotation, alongside simulated dexterous manipulation.The experiments target high-dimensional action spaces and sparse or uninformative rewards.Source: §VI · p. 6

Simulation tasks

  • The simulated dexterous tasks use a 28-DoF five-fingered hand and involve pen rotation, door opening, and object relocation under binary rewards.The dataset contains 25 human demonstrations per task plus 500 trajectories generated from a behavioral-cloned policy.Source: 1) Comparative Evaluation When… · pp. 6–7

Simulation results

  • 120K timesteps, equivalent to 20 minutes of online interaction, suffice for AWAC to solve the simulated pen task.Alternative off-policy and offline methods largely fail to solve the door and relocation tasks within the evaluated time frame.Source: 1) Comparative Evaluation When… · p. 7

Locomotion results

  • AWAC matches or exceeds the best prior method on every evaluated locomotion task, while no other single prior method performs well across all tasks.SAC+BC and BRAC are competitive on HalfCheetah, ABM is competitive on Ant, and AWAC leads on Walker2D.Source: 1) Comparative Evaluation When… · p. 7

2) Fine-Tuning from Random Policy Data:

AWAC is evaluated on real-world robotic tasks using prior data and online fine-tuning, including a random-data pushing experiment and dexterous manipulation. It preserves useful offline performance and improves during online interaction, while the paper identifies remaining tuning and generalization limitations.

Fine-Tuning from Random Policy Data

  • The simulated pushing experiment uses 500 trajectories generated by a random process to test whether AWAC can learn from suboptimal prior data.The task uses a Sawyer robot in a tabletop pushing environment.Source: 2) Fine-Tuning from Random Policy Data · p. 7

Real-World Evaluation

  • The real-world evaluation covers valve rotation with a 3-fingered claw, object repositioning with a dexterous hand, and drawer opening with a Sawyer arm.The tasks involve high-dimensional control, fine finger coordination, or precise hook insertion.Source: B. Real-World Robot Learning with… · pp. 7–8

Results

  • AWAC begins online fine-tuning with performance comparable to the best prior method trained offline alone, then improves more quickly during online interaction.The value at timestep 0 marks the beginning of online fine-tuning.Source: B. Real-World Robot Learning with… · p. 8
  • On the real-world tasks, prior methods struggle to improve online because sparse rewards and challenging dynamics make progress difficult from poor initialization.The figure reports that other methods fail to use prior data effectively offline and show slow or no online improvement.Source: B. Real-World Robot Learning with… · p. 8, Figure 6 · p. 8

Limitations

  • AWAC has limitations involving constrained-optimization threshold selection, incomplete theoretical understanding of fine-tuning, and broader transfer across robots, labs, and tasks.The authors propose adaptive threshold tuning and broader data integration as future directions.Source: §VII · p. 8

APPENDIX

The appendix details AWAC’s constrained policy optimization, implementation choices, and supporting analyses. It also reports that including the normalization factor Z(s) worsens fine-tuning performance in the tested settings.

Algorithm Derivation

  • AWAC derives a closed-form constrained policy update by applying KKT conditions to the optimization problem defined by the advantage estimate and replay-buffer distribution.The derivation proceeds through the Lagrangian and its derivative with respect to the policy.Source: A. Algorithm Derivation Details · p. 12

Policy Projection

  • Reverse KL projection enables maximum-likelihood optimization over replay-buffer states and avoids sampling potentially out-of-distribution actions for the Q function.The authors report this choice as vital for stable off-policy learning.Source: A. Algorithm Derivation Details · p. 12

Implementation

  • AWAC is implemented on a twin soft actor-critic skeleton, with the policy update replaced while off-policy baselines use the same underlying framework.The appendix also lists the base hyperparameters and compares variants in Figure 8.Source: B. Implementation Details · p. 12

Normalization Analysis

  • Table I compares success rates after online fine-tuning with and without Z(s) weighting for pen, door, and relocate tasks.The caption states that weighting by Z(s) performs worse despite the possibility of estimating it.Source: Table I · p. 12

Implementation

  • The Lagrange multiplier λ is fixed at 0.3 for manipulation environments and 1.0 for MuJoCo benchmarks.Adaptive learning of λ would require access to the behavior policy πβ.Source: B. Implementation Details · p. 13

C. Environment-Specific Details

The experiments span dexterous manipulation, Sawyer manipulation, and MuJoCo benchmark environments, with details provided in subsequent sections.

Environment Domains

  • AWAC is evaluated in three domains: dexterous manipulation environments, Sawyer manipulation environments, and MuJoCo benchmark environments.The paper states that following sections describe domain-specific experimental details.Source: C. Environment-Specific Details · p. 13

1) Dexterous Manipulation Environments:

The paper specifies dexterous manipulation tasks, benchmark rewards, datasets, and baseline implementations for offline pretraining followed by online fine-tuning. Comparisons show task-dependent baseline performance and a failure case for the tested MPO implementation.

Dexterous Tasks

  • The dexterous benchmark includes pen spinning, door opening after latch twisting, and object relocation to a goal location.The tasks use high-dimensional action spaces and binary-style success rewards.Source: 1) Dexterous Manipulation Environments · p. 13

Sawyer Environment

  • The SawyerPush environment requires moving a puck to a goal in a 40cm x 20cm workspace using a distance-based reward and Ornstein-Uhlenbeck random prior data.Hindsight experience replay is used for goal-conditioned reinforcement learning.Source: 2) Sawyer Manipulation Environment · p. 13

Data Comparison

  • Table III reports expert-data performance and behavior cloning on expert-only versus combined expert and behavior-cloning data.For Gym benchmarks, the reported metric is average return, and expert data comes from a trained SAC policy.Source: Table III · p. 13

Algorithm Comparisons

  • Figure 8 compares algorithms that incorporate prior datasets and motivates avoiding behavior-policy estimation, retaining a constraint, and using Qπ for sample-efficient learning.The comparison is framed around mixed-policy datasets and offline-to-online training.Source: Figure 8 · p. 14

Baselines

  • SAC, BRAC, AWR, MARWIL, MPO, BEAR, DAPG, and behavioral cloning are implemented or adapted as comparison methods.The appendix describes their use of prior data, policy regularization, value estimation, or online-training modifications.Source: D. Baseline Implementation Details · p. 14

Baseline Results

  • The tested MPO implementation showed no offline or online improvement in most environments and was omitted from the reported comparison.Its update samples actions from the policy and uses Q-function rather than advantage weights.Source: D. Baseline Implementation Details · p. 14

MuJoCo Results

  • On MuJoCo tasks, SAC+BC and BRAC match AWAC on HalfCheetah, ABM matches it on Ant, and AWAC outperforms all other methods on Walker2D.The figure caption notes that these tasks are easier than dexterous manipulation and isolate offline pretraining followed by online fine-tuning.Source: Figure 9 · p. 15

E. Gym Benchmark Results From Prior Data

Across MuJoCo tasks, AWAC is designed to improve efficiently from prior data, while several alternatives either improve slowly, suffer poor initialization, or lag during online fine-tuning.

F. Extra Baseline Comparisons (CQL, AlgaeDICE)

Additional comparisons test offline pretraining followed by online fine-tuning across methods and data qualities. AWAC remains competitive offline and generally improves online, unlike the reported limitations of CQL, AlgaeDICE, and BEAR.

H. Hardware Experimental Setup

The hardware experiments use task-specific robot state and action spaces, sparse or shaped rewards, and prior demonstrations supplemented by behavior-cloned trajectories.

Dexterous Manipulation with a 3 Fingered Claw

Drawer Opening with a Sawyer Arm

Dexterous Manipulation with a Robotic Hand

Related papers

Loading 2006.09359v6…