PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives

Murtaza Dalal, Deepak Pathak, Ruslan Salakhutdinov

arXiv:2110.15360v1cs.LGcs.AIcs.CVcs.RO

TL;DR

Robotic RL is challenged by exploration in continuous action spaces and by learning both task choices and robot control. The paper introduces manually specified, argument-parameterized robot action primitives controlled by an RL policy. Across three environment suites, the authors report stronger performance than prior methods in most settings, while noting limits on dynamic and fully general motion.

  • Problem

    Robotic RL remains difficult because exploration in continuous action spaces is challenging and agents must learn both what to do and how to control the robot.

    Source: §1 · p. 1

  • Method

    RAPS uses manually specified robot action primitives parameterized by arguments learned and selected by a high-level RL policy.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across Kitchen Suite, Metaworld, and Robosuite, parameterized actions significantly outperform prior state-of-the-art methods in most settings.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    RAPS demonstrates the utility of incorporating agent-centric structure into the robot action space across a wide range of environments, though it is not a general solution for every task.

    Source: §6 · p. 10

    Primitives run to the end of a fixed horizon and cannot express every motion, limiting dynamic behavior and trajectories that require changing actions at each low-level step.

    Source: §3.2 · p. 4, §6 · p. 10

Abstract

from arXiv · show

Despite the potential of reinforcement learning (RL) for building general-purpose robotic systems, training RL agents to solve robotics tasks still remains challenging due to the difficulty of exploration in purely continuous action spaces. Addressing this problem is an active area of research with the majority of focus on improving RL methods via better optimization or more efficient exploration. An alternate but important component to consider improving is the interface of the RL algorithm with the robot. In this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. These parameterized primitives are expressive, simple to implement, enable efficient exploration and can be transferred across robots, tasks and environments. We perform a thorough empirical study across challenging tasks in three distinct domains with image input and a sparse terminal reward. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance irrespective of the underlying RL algorithm, significantly outperforming prior methods which learn skills from offline expert data. Code and videos at https://mihdalal.github.io/raps/

1 Introduction

Robotic RL exploration is difficult in continuous action spaces, and agents must learn both task choices and robot control. This work introduces RAPS, manually designed parameterized primitives controlled by a high-level policy, and evaluates them across three manipulation domains.

1 Introduction

  • Random joint-space motions rarely produce useful object interactions, while robotic RL can require extensive reward engineering, data, demonstrations, or simulation setup.Prior examples use dense rewards estimated with AR tags, large-scale robot infrastructure, or simulation with engineered rewards for transfer.Source: §1 · p. 1
  • Robotic RL jointly requires learning what task action to take and how to control the robot, making task learning and motion execution difficult together.The agent must both infer which object to manipulate and learn accurate motions such as reaching and grasping.Source: §1 · p. 1
  • RAPS uses manually specified, argument-parameterized robot behaviors that a high-level policy selects and controls, with primitives reusable across tasks without modification.Examples include lift, push, and top-grasp; policy-learned arguments specify behavior such as grasp position, movement depth, and gripper closure.Source: §1 · p. 2
  • The paper evaluates RAPS through questions about action parameterizations, offline-learned skills, RL-algorithm independence, multi-task sequencing, and exploration without extrinsic rewards.The empirical study spans Kitchen Suite, Metaworld, and Robosuite.Source: §1 · p. 2
  • Across Kitchen Suite, Metaworld, and Robosuite, the authors report that parameterized actions outperform prior state-of-the-art methods by a significant margin in most settings.The paper also points readers to its code and performance videos online.Source: §1 · p. 2

2 Related Work

Related work covers higher-level action structures, hierarchical skill learning, and parameterized actions. The paper distinguishes RAPS by applying manually specified, expressive primitives to image-based manipulation with sparse rewards.

Higher Level Action and Policy Spaces in Robotics

  • Task-and-motion planning over primitive arguments depends on accurate state estimation, while this paper uses RL to learn from trial and error with image input and sparse rewards.Related approaches include dynamical-system skills that can be sensitive to hyperparameter tuning and often remain in behavior cloning, and Neural Dynamic Policies evaluated with state and dense rewards.Source: §2 · p. 3

Hierarchical RL and Skill Learning

  • Prior action-structure methods include manually designed hierarchies, trajectory-organized skills, and skill libraries learned from offline datasets, but may be sample-inefficient, poorly scalable, or task-data-dependent.RAPS uses robot-state-dependent manually specified primitives that can reuse implementation details across robots, tasks, and domains.Source: §2 · p. 3

Parameterized Actions in RL

  • Prior parameterized-action RL work largely targets state-based domains or simulated robot soccer, whereas this paper studies household-object manipulation and image-input policies with sparse terminal rewards.The paper describes its contribution as complementary to Chitnis et al. [10], which studies state-input policies for bimanual manipulation.Source: §2 · p. 3

3 Robot Action Primitives in RL

RAPS divides task choice from robot motion: an RL policy selects a primitive and its continuous arguments, while a fixed controller executes the resulting behavior. The primitives operate over fixed horizons and return the resulting observation and accumulated reward to the policy.

3 Robot Action Primitives in RL

  • RAPS assigns task-level choices to an environment-centric RL policy and low-level motion to fixed, manually defined, agent-centric primitives parameterized by continuous arguments.The policy chooses a primitive and its arguments while leaving low-level control to the primitive.Source: §3 · p. 3

3.1 Background

  • The background formulates RL as maximizing expected discounted reward over trajectories, and vision-based RL uses image observations and a policy conditioned on observations.The action-space modifications in this work leave the other MDP or POMDP components unchanged.Source: §3.1 · pp. 3–4

3.2 Parameterized Action Primitives

  • A primitive computes a target robot state from its arguments, then uses a feedback controller for a fixed horizon to drive the robot state toward that target.The primitive-specific controller, error metric, and target computation determine the motion; the same primitive implementation can be reused across robots when suitable controllers are available.Source: §3.2 · p. 4
  • For the lifting primitive, the argument sets upward displacement, the target is the current end-effector position plus that displacement, and the controller runs for 100–300 steps.The example uses end-effector position as the robot state and an end-effector controller.Source: §3.2 · p. 4
  • Because a fixed primitive library cannot express every motion, RAPS includes a raw-action dummy primitive, but its fixed argument over a horizon is less expressive than a feedback policy that updates every low-level step.For example, producing a circle would require many higher-level calls with the dummy primitive, whereas a feedback policy could update the target iteratively.Source: §3.2 · p. 4
  • The policy outputs primitive-selection and argument distributions, then the environment executes the selected primitive and returns its final observation and accumulated intermediate reward.A fixed output dimension is maintained by outputting argument distributions for all primitives; a 10-primitive, three-argument example uses a 40-dimensional action representation.Source: §3.2 · p. 5

4 Experimental Setup

The evaluation compares action spaces and learned-skill baselines on 16 tasks across Metaworld, Kitchen, and Robosuite. Tasks use sparse success rewards and visual input, with performance assessed using task success, wall-clock time, and training steps.

Tasks and Environments

  • The study evaluates 16 tasks across Metaworld, Kitchen, and Robosuite, chosen to cover varied manipulation difficulty, task diversity, and simulation realism.Kitchen includes seven subtasks and sequences of up to four; Robosuite adds more realistic simulation and torque-based control relative to the other two domains.Source: §4 · p. 5

Sparse Reward and Image Observations

  • Each task uses a sparse success reward of 1 when achieved and 0 otherwise, and methods receive visual input rather than world state.The evaluation uses mean task success rate as its interpretable performance measure.Source: §4 · pp. 5–6

Evaluation Metrics

  • The evaluation uses wall-clock training time and gradient training steps because either metric alone misses relevant costs or efficiency information across action spaces.Wall-clock time includes interaction and agent updates but can vary with controller speed and implementation; training steps serve as a data-efficiency proxy.Source: §4 · p. 6

Baselines

  • Baselines include environment Raw Actions, VICES controller-parameter tuning, Dyn-E skills from random-policy data, and SPIRL and PARROT skills learned from demonstrations or offline data.SPIRL learns a latent skill model from demonstrations, while PARROT learns an observation-conditioned flow model mapping raw to latent actions.Source: §4 · p. 6

5 Experimental Evaluation of RAPS

The evaluation tests RAPS in single-task learning, sequential multi-task control, and reward-free exploration, as well as across RL algorithms and learned-skill baselines. Across these settings, RAPS performs robustly and efficiently relative to the compared action parameterizations and learned skills.

5.1 Accelerating Single Task RL using RAPS

  • RAPS quickly solves every evaluated single-task task from sparse rewards, while Raw Actions and VICES struggle with difficult manipulation.The comparisons span Kitchen, Metaworld, and Robosuite; RAPS also outperforms baselines on challenging tasks in Figure 3.Source: §5 · p. 6, Figure 3 · p. 7, §5.1 · pp. 7–8
  • SPIRL and PARROT compete with or exceed RAPS on easier Kitchen tasks, but only RAPS solves top-left-burner and hinge-cabinet; Dyn-E makes no progress.The learned-skill methods are limited by tasks in their demonstration data, whereas RAPS primitives are reused across domains.Source: Figure 5 · p. 8, §5.1 · p. 8
  • Across Dreamer, SAC, and PPO, RAPS maintains similarly high final Kitchen performance, unlike Raw Actions, whose results vary substantially across algorithms.Table 1 reports final success rates from sparse rewards, using five evaluation trials and three training seeds.Source: Table 1 · p. 8, §5.1 · p. 8

5.2 Enabling Hierarchical Control via RAPS

  • Across SAC, Dreamer, and PPO, RAPS solves three of four sequential Kitchen subtasks and continues improving, while SPIRL and PARROT plateau after one or two.Raw Actions eventually approach three subtasks but require significantly more wall-clock time and training steps; offline methods struggle outside tasks in their demonstration data.Source: Figure 7 · p. 9, §5.2 · p. 9

5.3 Leveraging RAPS to enable efficient unsupervised exploration

  • After reward-free exploration, RAPS enables rapid fine-tuning, solving hinge-cabinet in one hour, while raw-action exploration does not fine-tune quickly.The reported results concern hinge-cabinet and top-left-burner in Kitchen.Source: Figure 8 · p. 10, §5.3 · p. 10

6 Discussion

RAPS quickly solves tasks across three environment suites, but it is not a general solution to every task. Its limitations include fixed-horizon execution, incomplete motion expressivity, and possible design demands for more complex robot morphologies.

6 Discussion

  • A primitive runs to the end of its fixed horizon, preventing the dynamic feedback behavior available to policies acting on raw actions.The authors suggest combining primitives with standard feedback policies as a possible route to more dynamic tasks.Source: §6 · p. 10
  • The considered primitives cannot express every needed motion, so the dummy primitive lets the policy fall back to the default action space.Source: §6 · p. 10
  • Designing primitives for more complex robot morphologies may require significant domain knowledge; learned skills are proposed as a possible complement.The authors suggest combining agent-centric structure in learned and manually defined skills.Source: §6 · p. 10
  • RAPS uses parameterized actions to learn high-level policies that quickly solve robotics tasks across three environment suites, while primitives are not a general solution to every task.The paper presents agent-centric structure in the robot action space as useful across a wide range of environments.Source: §6 · p. 10

A Additional Experimental Results

Additional experiments examine transfer between robot morphologies, comparison with a DMP-based method, and execution time on a real robot. They report successful zero-shot transfer and faster trajectories with RAPS.

Cross Robot Transfer

Comparison against Dynamic Motion Primitives

  • Across nearly every Kitchen task, RAPS outperforms Neural Dynamic Policies from visual input.Neural Dynamic Policies incorporate Dynamic Motion Primitives with deep RL.Source: A Additional Experimental Results · p. 14

Real Robot Timing Results

  • 16.49 seconds per episode with raw actions versus 0.51 seconds with RAPS, a 32x speed up on the real xArm 6.The comparison averaged ten trajectories; a RAPS trajectory contained twice as many low-level actions.Source: A Additional Experimental Results · p. 14

B Ablations

Ablations examine primitive use and two design choices for the dummy primitive. The results indicate that it improves performance, while changing its control from 3DOF to 6DOF leaves overall performance largely unchanged.

Primitive Usage Experiments

  • At convergence, every primitive is used; the policy selects about 2.69 unique primitives per episode against a path length of 5.Move-delta-ee-pose and angled-xy-grasp are favored across tasks.Source: B Ablations · p. 14

Evaluating the Dummy Primitive

  • The dummy primitive improves performance significantly, indicating that hand-designed primitives alone are not always sufficient to solve the task.Source: B Ablations · p. 14

Using a 6DOF Control Dummy Primitive

  • Changing the dummy primitive from 3DOF position control to full 6DOF pose control leaves overall performance largely the same on Kitchen tasks.Source: B Ablations · p. 14

C Environments

The described environments use the MuJoCo simulator. Kitchen tasks share one setup, with the figure showing several tasks and omitting a bottom-burner task that differs only in which dial is turned.

C Environments

  • All described environments use the MuJoCo simulator.Source: C Environments · p. 14
  • The Kitchen tasks are contained within the same setup.Source: Figure 9 · p. 15
  • The Kitchen depiction includes top-left burner, microwave, light-switch, hinge-cabinet, kettle, and slide-cabinet tasks; the omitted bottom-burner task has the same goal as the top-burner task with a different dial.Source: Figure 9 · p. 15

C.1 Kitchen

The appendix describes the Kitchen, Metaworld, and Robosuite task setups, including their reward and control choices. It also specifies how the environments are configured for evaluation.

C.1 Kitchen

  • The Kitchen suite uses sparse rewards over seven tasks with a Franka Panda arm; its sequential multi-task episodes require completing four subtasks.The two multi-task environments group the seven tasks differently, and the setup uses 6DOF delta-position control with a mocap constraint for the multi-task version.Source: §C.1 · p. 15

C.2 Metaworld

  • Metaworld experiments use six tasks spanning easy to difficult, with sparse success-metric rewards and no rotation primitives.The tasks run in single-task mode with fixed target positions, using the V2 environments and 3DOF end-effector control with grasp control.Source: §C.2 · p. 16, Figure 10 · p. 16

C.3 Robosuite

  • Robosuite experiments evaluate block lifting and door opening in randomized initial states, using its existing sparse rewards and 3DOF end-effector control with grasp control.The setup adds workspace limits; it also raises the lift controller frequency to 40Hz and increases the door task's maximum action magnitude to .1.Source: §C.3 · pp. 16–17, Figure 11 · p. 17

D Primitive Implementation Details

The implementation uses an end-effector pose controller to execute parameterized primitives, with a fixed horizon and domain-specific control choices.

D Primitive Implementation Details

  • For each primitive, the controller target is the argument state plus the primitive arguments, and its error is the target minus the argument state.For lifting, the relevant state is the end-effector's z position, and the error is the difference between target and current z positions.Source: D Primitive Implementation Details · p. 17
  • Primitives use a fixed horizon, which runs faster than looping until a tolerance threshold is reached; the learned policy accounts for primitive inaccuracies.The same primitive library is used across tasks within each domain, but not necessarily across domains.Source: D Primitive Implementation Details · p. 17
  • Table 2 lists the Dreamer hyper-parameters used in the experiments.Source: Table 2 · p. 18

E RL Implementation Details

The appendix details RL and baseline implementations, then examines primitive selection and compares RAPS with competing action and skill methods. Its comparisons report advantages over baselines across updates, while SPIRL remains competitive on easier tasks.

E RL Implementation Details

  • Dreamer uses REINFORCE for actor training in imagination and a discount predictor for imagined states beyond episode end; its single-task imagination trajectory length is reduced to five.The implementation also samples full primitive trajectories and sets the batch size to 2500.Source: E RL Implementation Details · p. 18
  • For primitive experiments, RAPS uses horizon five and discount 0.8 in single-task settings, and horizon 15 and discount .93 in hierarchical control.The authors report that this horizon-based discounting improves RAPS performance and stability; the implementation also modifies SAC and uses Kostrikov's PPO implementation.Source: E RL Implementation Details · p. 18
  • The baseline implementations use original or author-provided code when possible, including vision-based SPIRL-CL; Table 5 describes primitive parameters, search spaces, low-level actions, and environment usage.Source: E RL Implementation Details · p. 18, Table 5 · p. 20
  • During training, policies shift toward the dummy and angled xy grasp primitives, and the average number of unique primitives falls from four early to around 2.69 at convergence.Removing the dummy primitive generally harms performance and prevents solving hinge-cabinet; a 6DOF dummy primitive yields largely similar performance.Source: Figure 12 · p. 19, Figure 13 · p. 20, Figure 14 · p. 20
  • RAPS dramatically outperforms NDP on nearly every visual-input task in both wall-clock time and training steps.Source: Figure 16
  • RAPS outperforms all baselines against number of updates, while SPIRL is competitive on easier tasks but makes no progress on more challenging tasks.The full comparisons include additional environments omitted from the main figures.Source: Figure 17 · p. 22, Figure 18 · p. 23
  • Across 50 Metaworld tasks with sparse rewards, RAPS solves or makes progress on up to 43, while raw actions struggle on most environments.Source: Figure 18 · p. 27

Related papers

Loading 2110.15360v1…