PaperlayerView exact revision on arXiv

Source-linked AI summary

AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine

arXiv:2006.09359v6cs.LGcs.ROstat.ML

TL;DR

The paper addresses the difficulty of continuing online RL improvement after learning from offline data, a challenge that matters for costly real-world robotics. It introduces AWAC, combining dynamic-programming critics with maximum-likelihood-style constrained actor updates, and reports rapid fine-tuning across simulated and real robotic tasks. The authors also identify threshold selection, limited theoretical understanding, and evaluation on isolated setups as limitations.

  • Problem

    Continuing to improve policies online after offline training remains difficult, limiting practical use of prior data in robotics where data collection is costly.

  • Method

    AWAC combines dynamic programming for critic training with a supervised-learning-style constrained actor update to use offline data and fine-tune online.

  • Results

    AWAC rapidly learns and fine-tunes successful policies across simulated dexterous manipulation and three real-world robotic tasks using prior data and online experience.

  • Takeaways & Limitations

    Incorporating prior data can reduce the time needed to learn robotic skills to practical time-scales.

  • Takeaways & Limitations

    AWAC requires an appropriate constrained-optimization threshold, lacks a complete theoretical account of fine-tuning, and has so far been applied mainly to isolated setups.

Abstract

from arXiv · show

Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings such as robotic control. If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill. Such prior data could either constitute expert demonstrations or sub-optimal prior data that illustrates potentially useful transitions. While a number of prior methods have either used optimal demonstrations to bootstrap RL, or have used sub-optimal data to train purely offline, it remains exceptionally difficult to train a policy with offline data and actually continue to improve it further with online RL. In this paper we analyze why this problem is so challenging, and propose an algorithm that combines sample efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of RL policies. We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. We demonstrate these benefits on simulated and real-world robotics domains, including dexterous manipulation with a real multi-fingered hand, drawer opening with a robotic arm, and rotating a valve. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.

I. INTRODUCTION

AWAC addresses the difficulty of using prior offline data for online RL by combining offline pre-training with online fine-tuning. Its constrained actor update and dynamic-programming critic support sample-efficient learning for challenging robotic tasks.

  • Motivation: Offline datasets can contain demonstrations, suboptimal-policy trajectories, related-task demonstrations, or random exploration, providing information about dynamics and tasks.Effective methods should pre-train on such data while continuing to collect online data.
  • Problem: Prior demonstration methods can bias learning toward data that may be non-optimal, while fully offline methods do not directly address continued online improvement.The paper seeks one algorithm that handles varied prior-data sources without privileged treatment.
  • Method: AWAC trains a critic with dynamic programming and updates a constrained actor using a supervised-learning-style objective.The critic leverages off-policy data, while the actor constraint mitigates offline distribution shift without overly conservative updates.
  • Evaluation: AWAC is evaluated on simulated dexterous manipulation and three real-world robots performing drawer opening, object pickup, and valve rotation.The tasks include high-dimensional action spaces and sparse, uninformative rewards.
  • Preliminaries: Actor-critic methods alternate between critic policy evaluation and actor policy improvement, using value estimates to maximize expected returns.The critic can be learned by Bellman updates, while the actor is updated to optimize the estimated action value.
  • Problem: Replay-buffer access to prior transitions alone is insufficient for the paper’s offline-pretraining and online-improvement setting.Standard actor-critic algorithms can use off-policy data in principle, but the paper analyzes why this does not solve the problem by itself.

III. CHALLENGES IN OFFLINE RL WITH ONLINE FINE-TUNING

Offline RL followed by online fine-tuning requires reusing arbitrary prior data while avoiding the failures of standard off-policy methods. The paper identifies bootstrapping error and inaccurate behavior-model constraints as central challenges.

  • Problem Setup: The setting provides a static dataset from an arbitrary policy mixture or human expert, then uses limited online interaction to learn an optimal policy.It covers demonstrations, random data, prior RL experiments, and multi-task data.
  • Existing Approaches: Imitation pre-training followed by on-policy RL is data-inefficient because it does not reuse prior data during fine-tuning.It also assumes the prior data may be optimal, which need not hold.
  • Observed Failures: Standard off-policy actor-critic methods gain little from prior data: SAC with prior data performs similarly to SAC trained from scratch.Behavior-cloning pre-training can additionally cause an initial performance decrease before behavior returns near scratch-level learning.
  • Bootstrap Error: Off-policy bootstrapping accumulates error when target actions fall outside the static dataset’s distribution.Inaccurate Q-values for out-of-distribution actions propagate through actor-critic updates.
  • Conservatism: Offline RL methods constrain policy improvement using an estimated behavior distribution, but online fine-tuning makes that estimate difficult to maintain.The behavior distribution mixes offline and online policies, and inaccurate modeling can make constrained optimization overly conservative.

D. Excessively Conservative Online Learning

Standard constrained offline RL methods can perform well before online fine-tuning yet improve slowly afterward because explicit behavior models become inaccurate as new data arrives. AWAC addresses this with off-policy critic estimation and an implicit, advantage-weighted actor update that reuses replay data without explicitly modeling behavior.

  • Failure modes: Offline RL methods with constraints perform well initially but improve slowly during online fine-tuning.Their purely offline performance is much better at the start, yet additional online iterations produce only gradual gains.
  • Failure modes: Behavior-model accuracy decreases during online fine-tuning as incoming data changes the replay distribution.The resulting mismatch can make constrained optimization overly conservative and limit improvement.
  • AWAC design: AWAC uses off-policy temporal-difference learning to estimate the critic while training the actor with an implicit policy constraint.This design combines data reuse with protection against offline instability and bootstrap error accumulation.
  • AWAC design: The policy improvement step keeps the updated policy close to observed data while maximizing the critic's estimated value.AWAC enforces this constraint implicitly rather than through an explicitly learned behavior model.
  • AWAC design: AWAC's actor update is weighted maximum likelihood, reweighting replay-buffer state-action pairs by critic-predicted advantages without fitting a parametric behavior model.The update uses samples directly from the current dataset, avoiding explicit behavior modeling during fine-tuning.

V. RELATED WORK

AWAC builds on prior work in off-policy learning, demonstration bootstrapping, maximum-likelihood policy optimization, and robotic learning from prior data. Its distinguishing scope is to use arbitrary suboptimal data and continue improving beyond it.

  • Off-policy and offline RL: Standard off-policy methods reuse training data but can struggle with previously collected data because distribution shift causes error accumulation.Offline RL methods address this issue by constraining policy updates.
  • Demonstration bootstrapping: Demonstration-based methods often initialize policies by behavioral cloning and then use on-policy fine-tuning, which is less sample-efficient than off-policy value estimation.Other methods add demonstrations to replay buffers, but these strategies can produce a large performance dip during online fine-tuning.
  • Maximum-likelihood actor methods: AWAC differs from AWR by estimating the current policy's Q-function with bootstrapping rather than estimating the behavior policy's value function with Monte Carlo or TD-λ methods.The paper relates this choice to improved efficiency and final performance.
  • Robotic learning: Prior robotic-learning work used initial rollouts, motion primitives, reward shaping, or reward inference to bootstrap on-policy reinforcement learning.These approaches established bootstrapping from prior data as a recurring strategy for real-world robotic learning.
  • Robotic learning: AWAC extends robotic bootstrapping beyond demonstrations to arbitrary suboptimal data while supporting continued improvement beyond that data.This broadens the type of prior experience that can participate in online reinforcement learning.

VI. EXPERIMENTAL EVALUATION

The evaluation tests whether AWAC combines prior data with online experience across challenging simulated and real-world robotic settings. Results emphasize rapid learning in sparse-reward dexterous manipulation and broad performance across locomotion tasks.

  • Evaluation goals: The evaluation asks whether AWAC improves complex robotic control efficiently, learns from suboptimal or random data, and bootstraps practical real-world reinforcement learning.The study uses challenging high-dimensional simulated tasks and three real-world robotic platforms.
  • Real-world evaluation: The study includes real-world drawer opening, object pickup with a multi-fingered hand, and valve rotation, alongside simulated dexterous manipulation.The experiments target high-dimensional action spaces and sparse or uninformative rewards.
  • Simulation tasks: The simulated dexterous tasks use a 28-DoF five-fingered hand and involve pen rotation, door opening, and object relocation under binary rewards.The dataset contains 25 human demonstrations per task plus 500 trajectories generated from a behavioral-cloned policy.
  • Simulation results: 120K timesteps, equivalent to 20 minutes of online interaction, suffice for AWAC to solve the simulated pen task.Alternative off-policy and offline methods largely fail to solve the door and relocation tasks within the evaluated time frame.
  • Locomotion results: AWAC matches or exceeds the best prior method on every evaluated locomotion task, while no other single prior method performs well across all tasks.SAC+BC and BRAC are competitive on HalfCheetah, ABM is competitive on Ant, and AWAC leads on Walker2D.

2) Fine-Tuning from Random Policy Data:

AWAC is evaluated on real-world robotic tasks using prior data and online fine-tuning, including a random-data pushing experiment and dexterous manipulation. It preserves useful offline performance and improves during online interaction, while the paper identifies remaining tuning and generalization limitations.

  • Fine-Tuning from Random Policy Data: The simulated pushing experiment uses 500 trajectories generated by a random process to test whether AWAC can learn from suboptimal prior data.The task uses a Sawyer robot in a tabletop pushing environment.
  • Real-World Evaluation: The real-world evaluation covers valve rotation with a 3-fingered claw, object repositioning with a dexterous hand, and drawer opening with a Sawyer arm.The tasks involve high-dimensional control, fine finger coordination, or precise hook insertion.
  • Results: AWAC begins online fine-tuning with performance comparable to the best prior method trained offline alone, then improves more quickly during online interaction.The value at timestep 0 marks the beginning of online fine-tuning.
  • Results: On the real-world tasks, prior methods struggle to improve online because sparse rewards and challenging dynamics make progress difficult from poor initialization.The figure reports that other methods fail to use prior data effectively offline and show slow or no online improvement.
  • Limitations: AWAC has limitations involving constrained-optimization threshold selection, incomplete theoretical understanding of fine-tuning, and broader transfer across robots, labs, and tasks.The authors propose adaptive threshold tuning and broader data integration as future directions.

APPENDIX

The appendix details AWAC’s constrained policy optimization, implementation choices, and supporting analyses. It also reports that including the normalization factor Z(s) worsens fine-tuning performance in the tested settings.

  • Algorithm Derivation: AWAC derives a closed-form constrained policy update by applying KKT conditions to the optimization problem defined by the advantage estimate and replay-buffer distribution.The derivation proceeds through the Lagrangian and its derivative with respect to the policy.
  • Policy Projection: Reverse KL projection enables maximum-likelihood optimization over replay-buffer states and avoids sampling potentially out-of-distribution actions for the Q function.The authors report this choice as vital for stable off-policy learning.
  • Implementation: AWAC is implemented on a twin soft actor-critic skeleton, with the policy update replaced while off-policy baselines use the same underlying framework.The appendix also lists the base hyperparameters and compares variants in Figure 8.
  • Normalization Analysis: Table I compares success rates after online fine-tuning with and without Z(s) weighting for pen, door, and relocate tasks.The caption states that weighting by Z(s) performs worse despite the possibility of estimating it.
  • Implementation: The Lagrange multiplier λ is fixed at 0.3 for manipulation environments and 1.0 for MuJoCo benchmarks.Adaptive learning of λ would require access to the behavior policy πβ.

C. Environment-Specific Details

The experiments span dexterous manipulation, Sawyer manipulation, and MuJoCo benchmark environments, with details provided in subsequent sections.

  • Environment Domains: AWAC is evaluated in three domains: dexterous manipulation environments, Sawyer manipulation environments, and MuJoCo benchmark environments.The paper states that following sections describe domain-specific experimental details.

1) Dexterous Manipulation Environments:

The paper specifies dexterous manipulation tasks, benchmark rewards, datasets, and baseline implementations for offline pretraining followed by online fine-tuning. Comparisons show task-dependent baseline performance and a failure case for the tested MPO implementation.

  • Dexterous Tasks: The dexterous benchmark includes pen spinning, door opening after latch twisting, and object relocation to a goal location.The tasks use high-dimensional action spaces and binary-style success rewards.
  • Sawyer Environment: The SawyerPush environment requires moving a puck to a goal in a 40cm x 20cm workspace using a distance-based reward and Ornstein-Uhlenbeck random prior data.Hindsight experience replay is used for goal-conditioned reinforcement learning.
  • Data Comparison: Table III reports expert-data performance and behavior cloning on expert-only versus combined expert and behavior-cloning data.For Gym benchmarks, the reported metric is average return, and expert data comes from a trained SAC policy.
  • Algorithm Comparisons: Figure 8 compares algorithms that incorporate prior datasets and motivates avoiding behavior-policy estimation, retaining a constraint, and using Qπ for sample-efficient learning.The comparison is framed around mixed-policy datasets and offline-to-online training.
  • Baselines: SAC, BRAC, AWR, MARWIL, MPO, BEAR, DAPG, and behavioral cloning are implemented or adapted as comparison methods.The appendix describes their use of prior data, policy regularization, value estimation, or online-training modifications.
  • Baseline Results: The tested MPO implementation showed no offline or online improvement in most environments and was omitted from the reported comparison.Its update samples actions from the policy and uses Q-function rather than advantage weights.
  • MuJoCo Results: On MuJoCo tasks, SAC+BC and BRAC match AWAC on HalfCheetah, ABM matches it on Ant, and AWAC outperforms all other methods on Walker2D.The figure caption notes that these tasks are easier than dexterous manipulation and isolate offline pretraining followed by online fine-tuning.

E. Gym Benchmark Results From Prior Data

Across MuJoCo tasks, AWAC is designed to improve efficiently from prior data, while several alternatives either improve slowly, suffer poor initialization, or lag during online fine-tuning.

  • DAPG, AWR, and MARWIL are somewhat worse offline than the best methods and improve steadily but very slowly during fine-tuning.
  • SAC with prior replay data learns faster than training from scratch, but AWAC is generally faster at solving the tasks.
  • A characteristic early learning dip produces a poor initial policy and lack of steady improvement, which can hinder real-world use and raise safety concerns.
  • On more difficult dexterous manipulation tasks, the compared algorithms do not show significant learning.
  • BEAR can outperform prior methods offline on Ant and Walker2d, but its online improvement is slow and final performance is much lower than AWAC.

F. Extra Baseline Comparisons (CQL, AlgaeDICE)

Additional comparisons test offline pretraining followed by online fine-tuning across methods and data qualities. AWAC remains competitive offline and generally improves online, unlike the reported limitations of CQL, AlgaeDICE, and BEAR.

  • CQL and AWAC perform similarly offline, but CQL does not improve during online fine-tuning, while AlgaeDICE performs poorly for offline pretraining.
  • The D4RL experiment evaluates random, medium, medium-expert, and expert datasets, then adds 500,000 online fine-tuning steps.
  • AWAC is competitive with BEAR offline and improves through online fine-tuning even when initialized from random or medium-quality data.
  • In almost all tested data-quality settings, AWAC online is the best-performing method or tied with BEAR.
  • In four of six random or medium-quality settings, AWAC online is significantly better than BEAR.
  • For HalfCheetah random data, the reported values are 2.2, 52.9, and 25.5 across the compared methods.
  • For Walker2D medium-expert data, the reported values are 42.7, 78.3, and 10.8 across the compared methods.

H. Hardware Experimental Setup

The hardware experiments use task-specific robot state and action spaces, sparse or shaped rewards, and prior demonstrations supplemented by behavior-cloned trajectories.

  • Dexterous Manipulation with a 3 Fingered Claw: The three-fingered claw setup uses a 22-dimensional state and 9-dimensional action space for valve rotation.
  • Dexterous Manipulation with a 3 Fingered Claw: Valve rotation receives reward −1 when the valve is within 0.25 radians of the target and 0 otherwise.
  • Dexterous Manipulation with a 3 Fingered Claw: The valve task uses 10 kinesthetic-teaching demonstrations and 200 behavior-cloned trajectories.
  • Drawer Opening with a Sawyer Arm: The Sawyer drawer-opening setup uses a 4-dimensional state and 3-dimensional velocity-control action space.
  • Drawer Opening with a Sawyer Arm: Drawer opening receives reward −1 when the motor rotates more than 15 radians from reset and 0 otherwise.
  • Dexterous Manipulation with a Robotic Hand: The three-fingered hand setup uses a 25-dimensional state, 19-dimensional action space, and a distance-based reward weighted toward the target location.
  • Dexterous Manipulation with a Robotic Hand: The hand task uses 19 kinesthetic-teaching demonstrations and 50 behavior-cloned trajectories.
Loading 2006.09359v6…