PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks

Soroush Nasiriany, Huihan Liu, Yuke Zhu

arXiv:2110.03655v3cs.LGcs.AIcs.RO

TL;DR

Long-horizon manipulation tasks remain difficult for deep reinforcement learning because of the exploration burden and the challenge of composing heterogeneous behavior primitives. MAPLE combines such primitives with low-level actions in a hierarchical policy; across simulated manipulation tasks, it achieves the highest average success rate among the reported baselines.

  • Problem

    Deep reinforcement learning faces an exploration burden in long-horizon manipulation, while heterogeneous behavior primitives are difficult to compose because they use different inputs, temporal resolutions, and behaviors.

    Source: §I · p. 1

  • Method

    MAPLE augments reinforcement learning with pre-built behavior primitives and low-level actions, using a hierarchical policy to select primitives and their input parameters.

    Source: §I · p. 1, A. Decision-Making with Parameterized… · p. 3, C. Composing Primitives via… · p. 3

  • Results

    90% average task success rate across all baselines, versus 19% for Atomic, 36% for Flat, 41% for Open Loop, 11% for DAC, and 79% for MAPLE (Non-Atomic).

    Source: B. Quantitative Evaluation · p. 5

  • Takeaways & Limitations

    Policies trained in simulation transferred to real-world Stack and Cleanup tasks, achieving average success rates of 93% and 83%, respectively.

    Source: D. Real-World Evaluation · p. 6, K. Real-World Experiments · p. 12

    Atomic actions cannot fill large gaps requiring a significant number of low-level action executions when the available primitives are insufficient.

    Source: §V · p. 6

Abstract

from arXiv · show

Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces Manipulation Primitive-augmented reinforcement Learning (MAPLE), a learning framework that augments standard reinforcement learning algorithms with a pre-defined library of behavior primitives. These behavior primitives are robust functional modules specialized in achieving manipulation goals, such as grasping and pushing. To use these heterogeneous primitives, we develop a hierarchical policy that involves the primitives and instantiates their executions with input parameters. We demonstrate that MAPLE outperforms baseline approaches by a significant margin on a suite of simulated manipulation tasks. We also quantify the compositional structure of the learned behaviors and highlight our method's ability to transfer policies to new task variants and to physical hardware. Videos and code are available at https://ut-austin-rpl.github.io/maple

I. INTRODUCTION

Long-horizon manipulation remains difficult for DRL because of exploration burden, while existing temporal abstraction approaches can require extensive learning and generalize brittly. MAPLE combines reusable behavior primitives with RL, and the paper reports stronger task performance and transfer to new variants and hardware.

I. INTRODUCTION

  • Long-horizon DRL faces an exploration burden, while learned temporal abstractions can have high sample complexity, limited interpretability, and brittle generalization.The agent must explore a prohibitively large space of possible behaviors; prior hierarchical methods learn low- and high-level controllers through trial and error.Source: §I · p. 1

II. RELATED WORK

  • Robotics planning and neural-programming approaches compose modules such as motion planners and pick-and-place controllers, but rely on domain knowledge or strong human supervision.Related DRL work using pre-built primitives is limited by narrow primitive sets, rigid or hard-coded composition, or evaluation in simpler domains.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • MAPLE uses pre-built, robust behavior primitives alongside atomic actions, letting RL learn how to compose them without strong domain knowledge.The policy selects a primitive or atomic action and supplies input parameters that instantiate the selected behavior.Source: §I · pp. 1–2
  • MAPLE achieved a 70% increase in task success rate over atomic-action DRL and was the only method to consistently solve all single-arm robosuite tasks.The evaluation covered eight simulated manipulation tasks and compared MAPLE with standard DRL, option-learning, and open-loop-schema baselines.Source: §I · p. 2
  • MAPLE’s modularity and abstraction facilitate transfer of learned policies to new task variants and physical hardware.The paper also reports quantifying the compositional structure of learned behaviors.Source: §I · p. 2

III. METHOD

MAPLE frames manipulation as RL over parameterized behavior primitives, including an atomic-action option. Its primitive library and compositionality metric support learning and evaluation across varied manipulation tasks.

III. METHOD

  • MAPLE evaluates compositional behavior by pairing a library of behavior primitives with an RL algorithm that composes them to solve manipulation tasks.The paper also proposes a metric to quantify how compositional the resulting learned behavior is.Source: §III · p. 2

A. Decision-Making with Parameterized Behavior Primitives

  • In a Parameterized Action MDP, each decision selects a primitive and its parameters; the primitive executes a finite, variable sequence of atomic actions specified by those parameters.Primitive input dimensions vary across the heterogeneous library.Source: A. Decision-Making with Parameterized… · p. 3

B. Behavior Primitives: Building Blocks for Manipulation

  • The library treats primitives as implementation-agnostic functional APIs, with semantically meaningful inputs, and adds atomic actions to fill gaps the other primitives cannot cover.The paper’s five primitives are reaching, grasping, pushing, gripper release, and atomic action.Source: B. Behavior Primitives: Building… · p. 3
  • The policy architecture separates the choice of which primitive to execute from how to instantiate it, accommodating the library’s heterogeneous parameterizations.Figure 2 depicts the high-level task policy and low-level parameter policy.Source: Figure 2 · p. 3, B. Behavior Primitives: Building… · p. 3

C. Composing Primitives via Reinforcement Learning

MAPLE uses a hierarchical policy to select a primitive and then produce its parameters. The design supports heterogeneous primitives and is implemented by adapting SAC’s actor and critic.

C. Composing Primitives via Reinforcement Learning

  • A high-level task policy selects the primitive, while a low-level parameter policy supplies that primitive’s inputs using dedicated sub-networks for heterogeneous parameterizations.The parameter networks output a shared maximum-size parameter distribution, truncated to the selected primitive’s required input length.Source: C. Composing Primitives via… · p. 3
  • The hierarchical design reduces the total number of policy outputs relative to a flat policy and delegates primitive selection and parameter instantiation to separate policies.The paper notes that prior work explored this hierarchy, but demonstrates it on complex manipulation domains with heterogeneous primitives.Source: C. Composing Primitives via… · p. 3
  • MAPLE adapts SAC’s critic and actor to represent primitive choices and their parameters.The losses include separate maximum-entropy terms for the task policy and parameter policy.Source: C. Composing Primitives via… · p. 4

D. Facilitating Exploration with Affordances

MAPLE uses affordance scores to represent whether primitive parameters are appropriate for a state, supplementing exploration with analytically defined guidance. The evaluation includes eight manipulation tasks, including two designed multi-stage, contact-rich tasks.

D. Facilitating Exploration with Affordances

  • Affordance scores measure how suitable a primitive’s parameters are for the current state, adding guidance to exploration beyond temporal abstraction.The paper notes that affordances could come from learned models using robot-interaction or human data, or be defined analytically.Source: D. Facilitating Exploration with… · p. 4
  • For reach, grasp, and push, the score encourages reaching parameters near keypoints; atomic and gripper-release actions receive a score of 1.Keypoints are objects to push or grasp, or targets for reaching.Source: D. Facilitating Exploration with… · p. 4
  • The evaluation covers eight tasks: six from robosuite and two designed for multi-stage, contact-rich manipulation.The designed tasks are Cleanup and Peg Insertion.Source: Figure 3 · p. 4

E. Quantifying Compositionality

The paper hypothesizes that manipulation tasks have compositional structure and that MAPLE can express it through recurring primitive-use patterns. It measures those patterns by comparing task sketches.

E. Quantifying Compositionality

  • Temporal abstraction can yield greater progress and average reward per timestep, encouraging the agent to prefer high-level primitives when appropriate.The authors hypothesize that manipulation tasks have intrinsic compositional structure, though they describe no explicit mechanism for discovering recurring primitive patterns.Source: E. Quantifying Compositionality · p. 4
  • Compositionality is measured as the average pairwise normalized similarity of task sketches, using Levenshtein distance to compare them.The sketches capture high-level task semantics and recurring behavior patterns.Source: E. Quantifying Compositionality · p. 4
  • Each non-atomic primitive type is one token in a sketch, while every atomic-action occurrence receives a unique token to discourage low-level actions.Source: E. Quantifying Compositionality · p. 4

IV. EXPERIMENTS

The experiments evaluate MAPLE across eight simulated manipulation tasks, comparing it with atomic-action, flat, open-loop, hierarchical, and non-atomic baselines. MAPLE achieves the highest average final success rate among the reported baselines.

IV. EXPERIMENTS

  • The experiments examine primitive composition, learned-behavior compositionality, and transfer to task variants and physical hardware.Source: §IV · p. 4

A. Experimental Setup

  • The evaluation used eight robosuite tasks with a Franka Emika Panda arm, operational-space control, dense task-progress rewards, and robot and object observations.Source: A. Experimental Setup · p. 5

B. Quantitative Evaluation

C. Model Analysis

MAPLE’s learned sketches combine temporally extended primitives with atomic actions, and can be reused to accelerate related tasks. The paper also reports transfer to real-world tasks and identifies limits when primitives leave large behavioral gaps.

C. Model Analysis

  • Learned sketches combine temporally extended primitives with atomic actions; peg insertion uses atomic actions for its contact-rich insertion phase.Peg insertion and wiping had lower compositionality scores than prehensile tasks such as pick-and-place and stacking.Source: C. Model Analysis · p. 5, Figure 5 · p. 6
  • Transferring a can-placement sketch to bread placement achieved over 5× sample efficiency versus learning from scratch.The ablations also report that removing affordances, reaching, or grasping leaves the agent unable to solve the task.Source: C. Model Analysis · p. 6, Figure 6 · p. 6

D. Real-World Evaluation

  • Sim-trained policies transferred to real-world Stack and Cleanup, with average success rates of 93% and 83%, respectively.The team re-implemented the behavior primitives on the robot and used an off-the-shelf pose-estimation model.Source: D. Real-World Evaluation · p. 6, Figure 7 · p. 6

V. CONCLUSION

  • MAPLE combines behavior primitives with low-level actions, which improve exploration while retaining flexibility for intricate behaviors.The conclusion notes that atomic actions cannot fill large gaps requiring many low-level executions, and calls for further exploration and credit-assignment work.Source: §V · p. 6

APPENDIX

The appendix describes MAPLE’s behavior primitives, learning procedure, affordance score, and flat-policy baseline. Its primitives use distinct controllers and parameters, while the hierarchical policy selects a primitive and then its parameters.

A. Behavior Primitives

  • Non-atomic primitives execute variable sequences of atomic actions, while the reaching, grasping, and pushing skills are hard-coded closed-loop controllers.The gripper-release primitive uses a fixed number of atomic actions; all atomic actions interface with the five-degree-of-freedom OSC controller.Source: A. Behavior Primitives · p. 9

B. Algorithm

  • MAPLE alternates environment data collection with off-policy Soft Actor-Critic updates to its Q network, task policy, and parameter policy.The parameter-policy loss uses the reparameterization trick; discrete parameters can use Gumbel-Softmax.Source: B. Algorithm · p. 9

C. Affordance Score

  • The affordance score uses keypoints determined by the primitive and state, takes the maximum over applicable keypoints, and is zero when none applies.Default distance thresholds are 0.03 for grasping, 0.06 for reaching, and 0.12 for pushing, with some larger reaching regions for large objects.Source: C. Affordance Score · p. 9

D. Flat Baseline

  • The adopted flat baseline samples primitive type and parameters independently, which can yield sub-optimal performance when primitive parameter structures differ.A variant with outputs scaling linearly with all primitive parameters made no progress despite extensive hyperparameter tuning.Source: D. Flat Baseline · p. 9

E. Open Loop Baseline

This appendix describes the open-loop and options baselines and the manipulation environments used for evaluation. The open-loop baseline differs from MAPLE in whether its task policy receives the current observation or only the episode timestep.

E. Open Loop Baseline

  • The open-loop baseline uses episode timestep rather than current observation as the task-policy input, and did not enable more effective learning on the suite’s more complex tasks.Its implementation otherwise matches MAPLE; the authors note that the cited method uses a different update rule and is designed for sparse rewards.Source: E. Open Loop Baseline · pp. 9–10

F. DAC Baseline

  • Double Actor-Critic was selected as the representative options baseline after outperforming alternatives on HalfCheetah-v2 and the lift task.Deep Skill Chaining was not applicable to these manipulation domains because it is designed primarily for goal-based navigation agents.Source: F. DAC Baseline · p. 10

G. Environments

  • The evaluation suite contains eight manipulation tasks, including six from the standard robosuite benchmark and two designed for multi-stage, contact-rich behavior.The two designed tasks are cleanup and peg insertion.Source: G. Environments · p. 10

H. Training

Training uses fixed atomic-action episode limits and an initial high-entropy period. Evaluation reports training rewards and final success rates, and separately estimates compositionality from successful task sketches.

H. Training

  • Episodes are limited to 150 atomic timesteps, and target entropy for both policies is set high during the first 600k environment steps.The high-entropy setting encourages greater exploration early in training.Source: H. Training · p. 10

I. Evaluation

  • Reward curves average episodic task rewards over 20 episodes at checkpoints every 30k exploration steps, normalize them from 0 to 100, and show a 150k-step moving average.Final task success rates are measured over 20 episodes using the final checkpoint.Source: §I · pp. 10–11
  • Task success is defined by task-specific criteria, such as insertion past a distance threshold for peg insertion and clearing all debris for wiping.Other criteria include object placement, lifting above a height threshold, opening the door past an angle threshold, and fitting the nut fully onto its peg.Source: §I · p. 11
  • Compositionality scores average five seeds, sampling 50 sketches per seed from the last checkpoint and discarding sketches that did not solve the task.The remaining sketches are scored according to Equation (1).Source: §I · p. 11

J. Task Sketch Transfer Experiments

The transfer procedure extracts a representative task sketch from source-task sketches, then trains on the target with that sketch fixed. Each primitive can be repeated until it earns high affordance reward, followed by atomic actions for uncovered behavior.

K. Real-World Experiments

The authors evaluated simulation-trained policies on real-world stacking and cleanup tasks whose simulated variants matched the real tasks’ characteristics. Success was 93% for stack and 83% for cleanup, below the respective simulation rates.

  • Simulated task variants were designed to match real-world object sizes, table size, and workspace layout; policies were trained to convergence before transfer.Source: K. Real-World Experiments · p. 11
  • Stacking required placing the butter box on the popcorn box, while cleanup required putting the butter box in a bin and pushing popcorn to a marked table area.Source: K. Real-World Experiments · p. 11
  • Perception combined robot proprioception with Kinect-based object poses, using a lift-and-recompute routine for occlusions and fixed pose constraints to reduce estimation errors.Objects were lifted to a location with minimized occlusions, their poses were recomputed, and the robot returned to its initial location; pitch, yaw, and table height constraints supported reliable estimates.Source: K. Real-World Experiments · p. 12
  • 93% stack success (97% in simulation) and 83% cleanup success (93% in simulation) across 30 trials per task, with up to 20 primitive calls per episode.Source: K. Real-World Experiments · p. 12
Loading 2110.03655v3…